If your company runs on computers.... - Page 2

Scott R · Scott R's Garage

Quote:

Originally posted by schnellfahrer
RAID is one thing, but what do you do in the event of a break in?
A customer of our lost all their computer hardware in a break in wednesday. It's a small company and all of their servers and all laptops/desktops were stolen. They started backing up all their data 2 months ago (after we were hired to go over their system). Luckily we have all their data on tape. Without backup, they would now be bankrupt. No kidding!

This was a problem at one time, not so much now however. We have in-line encryption devices that encrypt the data as it moves in and out of the servers. We also use a product called "SafeBoot" on the laptops that encrypts the hard drive. All of this basically renders the machins useless without the passkey.

schnellfahrer · 02:01 PM

In this case the problem would be that all their data would have been lost without backup.
You can have all the redundancy in the world (RAID, cluster etc.) but it only takes a break-in and all your data (and probably your business) is gone.

Scott R · Scott R's Garage

Quote:

Originally posted by schnellfahrer
In this case the problem would be that all their data would have been lost without backup.
You can have all the redundancy in the world (RAID, cluster etc.) but it only takes a break-in and all your data (and probably your business) is gone.

Then the answer would be to co-loc to a hot site.

schnellfahrer · 03-23-2007, 02:49 PM

We'll I guess if a company chooses not to invest in a decent backup solution, they are unlikely to spend the cash to replicate their data to a different location.

We had to talk this customer into invest in a backup solution.
Now just a few weeks later the see the full benefit of backups.

jeffgrant · 03-23-2007, 04:33 PM

Personally, I could care less if the gear is cheap... I'd spend my money on people and process.

I've seen some pretty amazing things being done by 2 smart guys with next to no budget; fully redundant, backed up off-site nightly, etc., etc.

Not a name brand in the place.

I've also seen a whole crew of people with a zillion bucks worth of name brand hardware not have a clue about what/how things work and kill a business because they didn't test a single database backup routine to see if they could actually recover with it.

Now, if it's a big operation, and you need high-end performance/reliability from your hardware, and the 4-hour response time from support contracts, then the premium suppliers are the way to go.

If it's a small to medium sized requirement, you can use something more reasonably priced.

As with anything computer related, I think it's always a matter of compromise... maximum performance, reliability, fault tolerance, and manageability/support for a set budget (which is NEVER enough, it seems).

Personally, for smaller installations I'm a big fan of buying a ton of the same gear, with hot-swappable drives where appropriate, so that a broken box can be immediately reimplemented using another box beside it, with minimal downtime. (Assuming the disk controller didn't FUBAR the drives, etc).

Bigger stuff I prefer reprovisioning with blades or larger "multi-domain" boxes.

It's all about determining and managing risk, and balancing that with the business requirements. And making the owners/operators of said business aware of the implications of their decisions.

$0.02

(FYI, I'm a Technical/Systems Architect, specializing in Oracle RAC and distributed Java apps, in large, global, secure systems, such as banks, governments, gaming companies, etc).

campbellcj · 03-23-2007, 07:35 PM

Quote:

Originally posted by schnellfahrer
RAID is one thing, but what do you do in the event of a break in?
A customer of our lost all their computer hardware in a break in wednesday. It's a small company and all of their servers and all laptops/desktops were stolen. They started backing up all their data 2 months ago (after we were hired to go over their system). Luckily we have all their data on tape. Without backup, they would now be bankrupt. No kidding!

My wife had her office computers stolen a couple years ago. It was a huge hassle as she and her staff are not 'computer people' and had been very lax on backups. They literally had to recover a lot of data by re-keying from paper records.

I have set up backup hardware and software for them but it is still a big training and habit-development issue to actually USE it.

This is one of many reasons why hosted solutions / software as a service kinds of models are appealing for certain applications and user communities.

Bill Douglas · 03-24-2007, 02:21 PM

LA and where I live are eathquake towns and data (backups) has to not just be offsite but outa town too. Like you say the hardware is the easy stuff, the data and doing business is staying alive.

Scott R · Scott R's Garage

Quote:

Originally posted by jeffgrant
Personally, I could care less if the gear is cheap... I'd spend my money on people and process.

I've seen some pretty amazing things being done by 2 smart guys with next to no budget; fully redundant, backed up off-site nightly, etc., etc.

Not a name brand in the place.

I've also seen a whole crew of people with a zillion bucks worth of name brand hardware not have a clue about what/how things work and kill a business because they didn't test a single database backup routine to see if they could actually recover with it.

Now, if it's a big operation, and you need high-end performance/reliability from your hardware, and the 4-hour response time from support contracts, then the premium suppliers are the way to go.

If it's a small to medium sized requirement, you can use something more reasonably priced.

As with anything computer related, I think it's always a matter of compromise... maximum performance, reliability, fault tolerance, and manageability/support for a set budget (which is NEVER enough, it seems).

Personally, for smaller installations I'm a big fan of buying a ton of the same gear, with hot-swappable drives where appropriate, so that a broken box can be immediately reimplemented using another box beside it, with minimal downtime. (Assuming the disk controller didn't FUBAR the drives, etc).

Bigger stuff I prefer reprovisioning with blades or larger "multi-domain" boxes.

It's all about determining and managing risk, and balancing that with the business requirements. And making the owners/operators of said business aware of the implications of their decisions.

$0.02

(FYI, I'm a Technical/Systems Architect, specializing in Oracle RAC and distributed Java apps, in large, global, secure systems, such as banks, governments, gaming companies, etc).

You and I are in the same exact line of work. But I always give thumbs down to blade servers, been burned to many times on boot from SAN, and failed shared PCI backplanes. My philosophy now is that if the app is small enough for a 2 proc blade, I'm going to put it on VMware first.

jeffgrant · 03-24-2007, 06:32 PM

I'm a big fan of the IBM Blades, and XCAT. Done some serious benchmarking/testing against all other major blade vendors, and they are hands-down the leaders in the field, IMO.

I've rolled out 18 racks of them for a render farm, as well as some for a global gaming company. In that case, they are all behind load balancers and we could care less if they work or not... they are just one instance/node in a giant pool. Most of those also net boot, and when they're not, then they're booting from mirrored local drives.

Never had a problem. (Well, one bad blade, but that was replaced that same day by IBM).

If it comes to big-box reliability/performance for multiple apps, I'm a big fan of Solaris Containers/Zones. Between that and DTrace, Solaris is my OS of choice.

Haven't had too much call to use VMWare, but I've heard some pretty good things. (What is old is new again!)

I HAVE used VMWare to test/train on Oracle RAC's though... that was VERY slick.

When it comes to SAN's, I don't trust them as far as I can throw them. I've had too much experience with Toshiba, Fujitsu, EMC, etc., etc., that have totally messed up entire systems because they've messed up a firmware update, etc. (Just had a big client be down for 9 days due to this... what a mess! Got me a free golf trip from the vendor, though, as they knew they'd messed up and appreciated that we could come in and fix things)

For true reliability, I always implement 2 separate SAN vendors for the same system. That way, if one "goes away", odds are the other one will still be around.

Scott R · Scott R's Garage

Quote:

Originally posted by jeffgrant
I'm a big fan of the IBM Blades, and XCAT. Done some serious benchmarking/testing against all other major blade vendors, and they are hands-down the leaders in the field, IMO.

I've rolled out 18 racks of them for a render farm, as well as some for a global gaming company. In that case, they are all behind load balancers and we could care less if they work or not... they are just one instance/node in a giant pool. Most of those also net boot, and when they're not, then they're booting from mirrored local drives.

Never had a problem. (Well, one bad blade, but that was replaced that same day by IBM).

If it comes to big-box reliability/performance for multiple apps, I'm a big fan of Solaris Containers/Zones. Between that and DTrace, Solaris is my OS of choice.

Haven't had too much call to use VMWare, but I've heard some pretty good things. (What is old is new again!)

I HAVE used VMWare to test/train on Oracle RAC's though... that was VERY slick.

When it comes to SAN's, I don't trust them as far as I can throw them. I've had too much experience with Toshiba, Fujitsu, EMC, etc., etc., that have totally messed up entire systems because they've messed up a firmware update, etc. (Just had a big client be down for 9 days due to this... what a mess! Got me a free golf trip from the vendor, though, as they knew they'd messed up and appreciated that we could come in and fix things)

For true reliability, I always implement 2 separate SAN vendors for the same system. That way, if one "goes away", odds are the other one will still be around.

My last job was with IBM! In fact I've only been with my new company (First Data Corp) for two months. The major issue I had with blades was the lack of internal drives, we were forced to use (this was IBM On Demand) boot from SAN. Now that's a interesting technology when it comes to Microsoft. When the SAN folks do a change, or a microcode update they shut down a controller and that prompts a path failover. Normally that's a non issue, except where blades are concerned. This momentary hiccup halts the pagefile and causes a BOSD instantly.

So I optioned out about 30 blade chassis last year for Coca Cola to run the bottle games, first SAN maintenance window we had killed the entire environment. Needless to say there was an angry mob of SA's at my doorstep the following Monday. Since them I have sworn them off.

jeffgrant · 03-24-2007, 07:12 PM

Well, THERE'S your problem! Windows? meh

I don't touch the stuff, personally.

(Joys of being the boss... get to choose what you get to work on!)

We've tried running some Windows installs on the blades, and there are quite a few issues (or were, last year, when we tried it).

Not just SAN stuff, but their IP failover on the internal blade switches is FUBAR.

We had one minor "failure" that caused our ENTIRE internal network to completely come to a halt. We'd set up IP failover on the blade's 2 NICs, and brought the live one down, expecting to be able to test the failover. 120 machines over 8 vlans, all lost their routing info as a result of that driver going nuts and just spewing arp/spanning-tree stuff everywhere. Caused all of our network gear to feeze up. Powered off the windows blade (gotta love out-of-band management!), and everything came back up within 2 minutes as everything re-learned the internal networking info.

Thankfully it was still in the "testing" phase, so no harm, no foul. It's just a major "moment" when everything around you just STOPS. Racks of gear just stop blinking, etc. Surreal.

Scott R · Scott R's Garage

Quote:

Originally posted by jeffgrant
Well, THERE'S your problem! Windows? meh

I don't touch the stuff, personally. (Joys of being the boss... get to choose what you get to work on!)

We've tried running some Windows installs on the blades, and there are quite a few issues (or were, last year, when we tried it).

Not just SAN stuff, but their IP failover on the internal blade switches is FUBAR.

We had one minor "failure" that caused our ENTIRE internal network to completely come to a halt. We'd set up IP failover on the blade's 2 NICs, and brought the live one down, expecting to be able to test the failover. 120 machines over 8 vlans, all lost their routing info as a result of that driver going nuts and just spewing arp/spanning-tree stuff everywhere. Caused all of our network gear to feeze up. Powered off the windows blade (gotta love out-of-band management!), and everything came back up within 2 minutes as everything re-learned the internal networking info.

Thankfully it was still in the "testing" phase, so no harm, no foul. It's just a major "moment" when everything around you just STOPS. Racks of gear just stop blinking, etc. Surreal.

Lucky bastard! I'm principle in Windows discipline, I was a SUN architect prior, but now I'm stuck in the seventh level of hell.

jeffgrant · 03-24-2007, 07:42 PM

Sorry to hear it!

I firmly believe in "right tool for the job", but I don't think Windows has a place in the server room.

Desktop? Debatable. (I'm an OS X fan-boy, and we do all of our development on OSX, and run Parallels if we have to have Linux/Solaris X86 or Windoze for anything on our desktop)

My friends and family still don't get the blank stare I give them when they ask me to fix their XP box... "Uhh... I don't know how to".

They laugh and say something to the effect of "I thought you did computer stuff?".

*sigh*

My sister was funny, and got me one of those "no, I will not fix your computer" T-Shirts for X'Mas a few years ago, as my "go home to see the folks" trips seemed to include 2-3 days of "free" tech support for my folks and their friends.

SlowToady · 03-26-2007, 06:23 AM

Here's another gold nuggest..

Boss calls today to ask me if I can check on the accounting software, it isn't letting her log in, and the list of companies it shows...aren't ours. For various reasons, there are several folders (each containing a company), and she can't find the one we are currently using. So I stumble over to my laptop, log in to the VPN, Remote Desktop to someone's computer, fire up the acct. software..and I can find it, but when I go to connect, it says there is an operation being performed that only allows single user access. Most probably, our accountant forgot to logout. There is no admin control panel or anything for the acct. software we use, so I was going to force a logoff on the Domain Controller. Not the prettiest fix but...So anyway, apparently now also my boss is having trouble logging into the VPN and the "office computer geek" text messages me (I'm in class now) asking how many people can log in to the VPN. Tell him 50. I get back, connect, log into the server, and as I'm looking around for some reason I threw the system info command at the cmd prompt. One thing I noticed is....server up time is 20 minutes! Hmmmm...open up Event Viewer and this is what I find:

The previous system shutdown at 7:59:08 AM on 3/26/2007 was unexpected.

and also,

The reason supplied by user VITALMOOSE\Administrator for the last unexpected shutdown of this computer is: Other Failure: System Unresponsive
Reason Code: 0x8000005
Bug ID:
Bugcheck String:
Comment: System did not allow logon for vpn

Check that last line again...

"System did not allow for logon for vpn"

THE SERVER DOESN'T CONTROL VPN LOGIN!!!!!!!! THE PIX FIREWALL DOES!

So apparently, this kid bounced the server, either by pulling the plug or just holding the power button, and when it comes backup he gives that lame ass reason.

Next time I go into the office, that power button is getting disconnected. Now if I could just stop him from pulling the plug...

Really good for data integrity, too. Guess I'll have to schedule a disk check...

SlowToady · 03-26-2007, 06:27 AM

Jeffgrant,

I have the "I READ YOUR EMAIL" shirt.

Don't wear that one to work....

jeffgrant · 03-26-2007, 07:29 AM

Yeah... some people's "admin" skills consist of the MS-nurtured concept of "hmmm... try rebooting, see if that fixes it..."

*sigh*

Right now I'm consulting at a big gaming company, so can wear all those "not appropriate for real client" shirts that I've got socked away in the closet.

Meetings scheduled with annoying people? --> STFU

Grumpy? --> NSFW (For some reason, I REALLY live up to this shirt... not really known for being too politically correct)

I'm almost twice the age of most of the people at the job... I'm now the old "go-to" guy if they have issues. "He's worked on MAINFRAMES!".

*sigh*

SlowToady · 03-26-2007, 09:27 AM

Those shirts are great, good for you:-P Must be nice to work a place where not everything is deathly serious. Should see the stares I get when people figure out email *isn't* private. If I have the power to add, modify, and delete email accounts, as well as complete Admin control over the server, what makes you think I can't read your email?

User, sort of concerned: Can you really read my email?
Me, tired of questions: Yes
User, extremely scared look; like when you discipline a puppy.

Here's a good one, a followup to the story I posted earlier today. I just text messaged the "office computer geek" to determine if everything is working, because I haven't heard from them in a few hours. He says:

"As far as I know. Error messages that look weird."

Really? No kidding? Error messages that look weird, huh? Well those are just my favorite. Come to think of it...I wonder if it could have anything to do with you PULLING THE PLUG ON A RUNNING SERVER. Hmmmm....

I guess I'm off to check the event log viewer.

jeffgrant · 03-26-2007, 09:35 AM

LOL! Yeah... I don't suffer fools easily... and have short patience when dealing with them.

Reminds me of a client meeting I had with a government agency a while back. After 6 weeks of planning, a major Oracle upgrade didn't go well, due to some pretty incompetent actions on the local "support" staff's part.

We're in the postmortem meeting after about 28 hours on the go, and my patience is beyond thin. The main culprit is spouting off endlessly about how it's not his fault, yada yada yada. I'm just sitting at the table, trying to stay awake after giving my initial assessment, when the Big Kahuna from the client asks for my opinion of Idiot's assessment.

Without thinking or pausing, I tell them what I was thinking "I think he's too fscking stupid to know he's stupid."

There's silence in the room.

"oh... was that my out loud voice?"

EVERYONE started laughing their asses off, except for the Idiot.

He storms out (never did see him again), and his boss catches up to me later and thanks me for my comments... he's been dying to say that for months, but was afraid of the union fallout.

Sometimes it's fun being a contractor.