If your company runs on computers....

[SlowToady]

Don't go cheap. Seriously, if your business, if you making money, depends on computers and related equipment, why skimp on it? I have a client whose entire operation gets done with computers (which isn't too odd these days). All order taking/tracking/processing, account settling, credit card running, communication, EVERYTHING. We had a server failure awhile back and it cost us $20K in lost productivity in a single day. Now, that may not be a lot to those of you that work at big companies, but in our business...that's a lot of bread. So anyway, my point is this:

All our "mission-critical" stuff runs on garden variety PC hardware! The ONE server we have, that handles DNS, AD, accounting, credit card processing, internet access...is a homebrew white box. Intel mobo, P4 3.0Ghz, SATA RAID. The same thing you likely have sitting at your desk. I know that these days anything with a big disk, lots of RAM and a fast CPU is a "Server" but there's a DAMN good reason SUN, IBM, HP, SGI, etc make good money selling *real* servers. A homebrew white box running Windows 2003 hardly counts, in my book, when the business depends on it. Spend a few extra dollars and get something rock solid. The ONE this server went down would have more than paid for an entry level IBM/SUN/HP box. And I wouldn't have to field calls at all different times and days.

Oh yea, this pisses me off, too: they splurged on a 50 user license 3DES equipped Cisco PIX 501 firewall, and then went with a run of the mill Linksys switch. Why not get a *good* switch? Hell, get one off ebay...I have a Cisco 1900 on my desk.

This whole rant came about because I got a text message from my client saying that the server was down, and they need the password to reboot it. Last week I had to install a new HSF because the stock one SUCKED, and the CPU was ideling @55c. So anyway, I thought it was weird the server was down; I tested it before and it was fine. Temps were down 25c at idle. Oh yea, the original "server" kill -9ed itself due to overheating. So, anyway, I go in to check on it, and they hired some new kid to assembly parts that is "a computer geek." He decided to bounce the server, since the accounting software locked up. That's how he "knew" the server was "down." I check through event logs and what not, and can't find anything. Nada. Zip. Only a message that the last shutdown was unexpected. Duh. Then I start showing him the network gear and stuff. Suddenly the lights go out for about 3/4second on the switch and came back on. Moved it some more and they went off again. Maybe it was a switch issue, and not a "server" issue? MAYBE HE SHOULDN'T HAVE PULLED THE ****ING PLUG NOT KNOWING WHAT WAS WRONG! God damn it, he didn't even try and troubleshoot anything!

Anyway, I'm pissed off, some kid now has the Administrator password to the "server" and he pulled the plug on it due what was probably nothing more than a network glitch. Thank God for RBAC, I'm going to give him an account but lock it down pretty well. And change the admin password. And why the hell can't we buy good ****? Why can't I have a GOOD switch back there? Seriously, this is the same switch you'd find in any Dick, Jane or Joe's house; not a problem if it reboots itself there or looses a connection. But at work? Big problem.

IF YOU INSIST ON RUNNING YOUR ENTIRE OPERATION OFF COMPUTERS, DON'T BUY CHEAP ****!

rant over, Nomex on, flame away.

[Neilk]

Well if you bill your client by the hour, then all these service calls aren't so bad. As long as they know the issue is with the hardware and not their main IT Consultant.

[legion]

Do all the zOS boxes I'm looking at qualify as "servers"?

[widebody911]

Right now I'm learning my way around Superdomes. We have two of them, with 128 CPUs and close to a TB of RAM - each.

[Scott R]

You need to move on from these mom and pop shops and get into the real data centers. We just spent 20 million alone this week on new servers to replace our aging four year old servers. I spared no expense, and I never do on new enterprise class hardware. I know SA's have it rough, so I try and by the best technology possible every time I refresh a series of systems. Everything business critical should be N+1 for maximum availability.

One other thing bothers me about your client, he has credit card processing on servers that are accessible by the office "computer geek." That's borderline illegal, not to mention just plain reckless from a information security standpoint. Anything that handles peoples personal information should be locked in raised floor space and secured behind some sort of audited system.

I'm sorry that you have to go though this, I would have pulled my hair out by this point.

[djmcmath]

What a nightmare -- that's just plain awful.

If it's any consolation, the US Navy isn't doing a whole lot better. My submarine spent 4 months at the pier for a whole lot of shiny new computers, including all of sonar, most of fire control (the stuff that takes sonar information and does useful interpretation of the data) and the entire ship's LAN (which contains some remarkably useful stuff, and without which all ship's operations come to a halt).

The fire control computers don't talk to the sonar computers, completely obviating the point of having fire control computers in the first place. The sonar computers overheat and shut down, because nobody apparently bothered running any kind of thermal analysis as to what happens if ambient temperature rises above 65F. That means that anytime we do anything unexpected with ventilation in the sonar computer space, we can't hear ("see") anything. The ship's LAN is no better -- most of it is installed in a space that was never meant for that many servers (at least we have decent gear), so overheating is a routine problem. We've performed a variety of our own modifications and have managed to keep temperatures within specification in there. But that doesn't fix the problem of the other server rack that they installed which was heavy enough to slightly alter the shape of the ship, disrupting the weapons handling system ever so slightly. Insubstantial? Sure, until you need to load a torpedo into a port-side tube, at which point the whole system binds up and bends lots of bits of expensive metal.

Good stuff. I'll stop ranting now -- suffice it to say that I sympathize.

[campbellcj]

We are a very small company but in the software biz so the infrastructure is mission-critical. We have decent equipment and very few issues in general. My main worry is the infrastructure of our space - like anything built more than a few years ago, it is short on electricity, cooling, cable management, and physical security (windows+walls). Not a showstopper, yet.

We do have a few "white box" Wintel machines that we use for dev/QA sandboxes - no critical files or apps there. Everything else is racked servers, dual power supplies, dual NICs, etc. The small cost delta it is not worth risking the downtime from cheapskate stuff dying - the commodity hardware isn't even very expensive these days. A few grand buys a pretty robust name-brand box.

The HP 9000 server that was one of our main workhorses for many years was like $85,000 retail when new in 1996. We moved most of its load to a $6,000 Sun in 2003, then I finally replaced the HP in 2004 with another HP machine bought on ebay for $1,600...boy how things change

[id10t]

Wayne - test it anyway. Sounds like you've made it part of a disaster/backup plan. Make sure it really works. That way, if it doesn't, you can find something that will work and have it setup and waiting *before* you need it

[legion]

The zOS boxes we run are super-robust.

They've been running for over 20 years with an average of one hardware issue that affects applications every 3 years, and I can count on one hand the number of times any of those boxes have been rebooted (IPLed in mainframe lingo) during that time period. They can be upgraded while still running. The memory on the boxes can be partitioned to isolate applications from each other.

http://www-03.ibm.com/systems/z/hardware/

[stomachmonkey]

Originally posted by id10t
Wayne - test it anyway. Sounds like you've made it part of a disaster/backup plan. Make sure it really works. That way, if it doesn't, you can find something that will work and have it setup and waiting *before* you need it

True. "Crashing" your environment is a good idea. You don't really crash it, just take it off line to see what your redundent systems will do.

For real disaster recovery you should have boxes off site. All the extra hardware in the world won't help if it's all in the same spot that had a fire, got flooded, etc....

At minimum you should do a scheduled tape rotation and always keep one set off site.

[Super_Dave_D]

I work for a major company and all data is backed up nightly and store off site.

We just completed a huge ordeal for disaster recovery - making sure we have all primary systems available off site.

Katrina really opened our eyes.

[Scott R]

Originally posted by Wayne at Pelican Parts
Those are all good points, however I tend to think that most all of that computer equipment is not too well built these days. The way to get around this is to build multiple servers. We just bought new machines here for Pelican. We bought nine servers at about $3300 a piece. All with the same hardware configuration, so that if anything happened to one of the super-critical ones, then one of the not-so-critical ones could be scavenged for parts. All the disks are mirror 1 arrays too, so that if a machine takes a complete dump, we should be able to simply swap one drive into another box, and have it boot up. Theoretically. I don't want to have to test that theory.

-Wayne

All that like hardware, you should look into clustering, very nice redundancy in that scenario.

[Tim Walsh]

I have to agree with most of what's said here. We(a university of 15,000 students) run almost exclusively dell and sun machines not just because of their reliability but also their support. Both companies provide 4 hour onsite service, all of our major systems are redundant with SAN storage, etc, etc. The biggest problem we're running into currently is thermal. We've been using one of our two machines rooms for 30 years and our A/C when on generator power is marginal. It's one thing to think about when designing a datacenter.. if the power goes out and you go to UPS/Generator will your A/C units keep up?

Not to hijack the thread, but what is everyone using in their major data centers for monitoring? We've been on BMC Patrol for the last year.

[Scott R]

Originally posted by Tim Walsh
I have to agree with most of what's said here. We(a university of 15,000 students) run almost exclusively dell and sun machines not just because of their reliability but also their support. Both companies provide 4 hour onsite service, all of our major systems are redundant with SAN storage, etc, etc. The biggest problem we're running into currently is thermal. We've been using one of our two machines rooms for 30 years and our A/C when on generator power is marginal. It's one thing to think about when designing a datacenter.. if the power goes out and you go to UPS/Generator will your A/C units keep up?

Not to hijack the thread, but what is everyone using in their major data centers for monitoring? We've been on BMC Patrol for the last year.

IBM had issues with cooling in the centers for a different reason, the raised floor areas in Boulder were installed in the 60's. By the 2000's we had so much copper under the floor it was reducing airflow. We suggested getting bids from contractors to help us clean up these football field sized under floor areas, and no one would take the job (too much risk involved disturbing production systems). Before I left the plan to vacate the raised floors and demo them, what a mess.

As far as data center monitoring tools, we use ECM by Configuresoft, SiteScope, Landesk, a few top secret internal jobs that our command centers use, MOM, and I can't remember what we use on the Sun/Unix/Linux/AIX side. (I'm a Windows Architect)

[HardDrive]

You can go overboard. We have a pair of dual processor 64-bit boxes with 4 gb of memory acting as domain controllers for out company......of 25 people

That 4 terabyte SAN we bought last year ain't exactly filling up to quickly either.

We had a bit of cash lying around last year.....

[SlowToady]

Thanks for the sympathy, I figured I might find kindred spirits here:-P I'm going to reply to a few posts..and then I have some other good stories..heh.

Neilk, most of the hours I bill for, but some I can't. Especially since they actually think it's mostly my fault! For example, when I installed the new HSF on the "server," a few days later they couldn't find the link/shortcut to the credit card processing system on their desktops. Their conclusion? I must have broken it (the CC software) while "rebuilding the server" (their words, not mine). That means I had to stop fishing/hunting, pack up my laptop and notebook, and find somewhere with an internet connection, then get on the VPN and poke around. On my vacation...

Legion, widebody, I'm jealous. Not sure what else to say there...Big iron gives me big...


Wayne, I've already thought of, and proposed, that exact solution. I was going to build (or repaier/get repaired) the old "server," throw a SATA RAID controller in their and 2 drives, and install 2003. Fault Tolerant Cluster setup. So if the main "server" goes down or stops responding, this one would take right over. They rejected my plan as unneeded, and too expensive. Keep in mind this is AFTER the server failure that cost us $20K in productivity. I would have even preferred going with a SAN or something, but I *knew* that wouldn't fly. I figured a 2 server cluster might, though.

HAHAHAHAHAHAHAHAHAHAH. BACKUPS?!?!?! ROFL!!! Ahahahahaha...man..that was a good one. Backups, lol. The only backups we have are the accounting software backups, and they only make one copy (against my recommendation) and KEEP IT ON SITE IN THE OPEN! I've stressed multiple times that 2 copies of each backup should be made, and one should be stored offsite, preferably in a safe. The "server" doesn't even have a tape drive, so we have backups of the accounting stuff, but if the hardrives take a ****, we have to hope only one of the two dies. Drives are mirrored. A tape drive "isn't in the budget."

Raised floors? Locked anything? Proper cooling? The current setup is in a backroom, with no AC, regular room. Door doesn't lock properly. I believe one of the key reasons the old "server" died was due to an extreme heat environment; it was in at attic that, by my bosses admission, you would sweat in if you sat in there. Contributing to that is a CPU core idle temp. of 55deg. C. And they wonder why it failed? Didn't help that they kept trying to turn it back on before I got there, and that it just kept overheating.

Oh, well hell, here's a good one! Let's talk security policy! Windows NT has this wonderful idea called RBAC (Role Based Access Controls)*. It's a dead sexy idea. Except it's not implemented in our environment...so users can do things they shouldn't be able to do. Oh and how about a password policy. The passwords are TRIVIAL. I didn't even have to ASK for passwords, I guessed all of them. For email, accounting, CC processing, AD, you name it. I'm dying to implement a new password policy, but they aren't too receptive. I don't think they understand the true costs that a breach would entail, especially as 60% of business is conducted via email.

Scott R, the office computer geek already has access to the accounting system, full access, so, hell, why not CC? Geeezz... And didn't you used to post on the 944 board when I first registered here? I seem to remember your nick, and haven't seen it in awhile. I also remember you worked with computers.

*I'm just ranting, not assuming anyone here doesn't know what RBAC is.

Thanks for letting me rant guys..feels good to just get this off my chest.

For the SAs out here,
a day in the life of a SysAdmin. I like to read it when I hate work:-P Gives me ideas...

[Tim Walsh]

AHH yes BOFH.. it's brought me much comfort when all was lost in ITland.

[Scott R]

Scott R, the office computer geek already has access to the accounting system, full access, so, hell, why not CC? Geeezz... And didn't you used to post on the 944 board when I first registered here? I seem to remember your nick, and haven't seen it in awhile. I also remember you worked with computers.

*I'm just ranting, not assuming anyone here doesn't know what RBAC is.

Thanks for letting me rant guys..feels good to just get this off my chest.

For the SAs out here,
a day in the life of a SysAdmin. I like to read it when I hate work:-P Gives me ideas... [/B]

Yes, I started here on the 944 board with a supercharged NA, then later a 951 which I sold to a fellow Pelican. After that I went to aircooled, on my second 911 now, what an amazing addiction p-car's are.

I was but a lowly Sys Admin back in the 944 days, I moved into management, then to architecture.

[Bill Douglas]

I like Waynes semi manual recovery system. I worked for a place with a "fail-over" system that connected different servers with different disks depending on what failed. The fail-over system was the weakest link in the whole setup. I think a number of exactly the same severs and disks (so there are no startup problems) then swapping in the bits required, and some know how is the best solution. And brand name equipment is easier to get working than no name stuff.

[schnellfahrer]

RAID is one thing, but what do you do in the event of a break in?
A customer of our lost all their computer hardware in a break in wednesday. It's a small company and all of their servers and all laptops/desktops were stolen. They started backing up all their data 2 months ago (after we were hired to go over their system). Luckily we have all their data on tape. Without backup, they would now be bankrupt. No kidding!