AT&T sucks. Data center fire, no Service for 48 Hours!?!?!?

[Neilk]

Local At&t Data Center had a fire and rumor has it that it may take 48 hours to get Dallas area customers back online? That s crazy. What kind of redundancy or disaster recovery plan did they have in place? Apparently a lousy one.

Link to article.

Spectrum sounds better and better, but I have heard bad things about them too. Good thing we can tether our phones, but with everyone doing the same thing, it’s been rather slow.

P.S. definitely a first world problem and thankful for what I have

[KFC911]

Could be incompetence, bad execution, or by design (too expensive to do it "right...so just roll the dice) ....

[masraum]

It's a balance of cost vs risk. The cost to do it right was too high, so it was done right enough for most things but not a fire.

And it could have been lots of things. Maybe there was network redundancy, but inadequate power redundancy or inadequate cooling redundancy.

Either way, most residential customers aren't paying enough for them to be worried about a 2 day outage. If you want SLAs, you'll likely pay WAY more for your service and will get better service. But even then, there can be long outages. If you want as close to 100% uptime as possible, you'll have to get multiple circuits from multiple providers and have them come into your house on opposite sides of the house, and get maps of how the circuits get to your house to ensure that they don't meet up 5 miles from your house in the same fiber bundle that will eventually get hit by a truck or a backhoe or a "fire in the man hole" (I hear that one a lot, it always makes me chuckle and think of Mexican or Indian food).

[stomachmonkey]

Originally Posted by masraum

It's a balance of cost vs risk. The cost to do it right was too high, so it was done right enough for most things but not a fire.

And it could have been lots of things. Maybe there was network redundancy, but inadequate power redundancy or inadequate cooling redundancy.

Either way, most residential customers aren't paying enough for them to be worried about a 2 day outage. If you want SLAs, you'll likely pay WAY more for your service and will get better service. But even then, there can be long outages. If you want as close to 100% uptime as possible, you'll have to get multiple circuits from multiple providers and have them come into your house on opposite sides of the house, and get maps of how the circuits get to your house to ensure that they don't meet up 5 miles from your house in the same fiber bundle that will eventually get hit by a truck or a backhoe or a "fire in the man hole" (I hear that one a lot, it always makes me chuckle and think of Mexican or Indian food).

Pretty much.

100% SLA for consumers would be cost prohibitive to offer consumers.

Spectrum rep offered me 100% SLA for my office, at $1,500.00 per month.

I think I laughed so long he just hung up.

[osidak]

I have spectrum at home - Internet in my neighborhood went out around 9pm Friday night - came back around 7 pm Sunday

[DanielDudley]

Must have been a heck of a fire.

[KFC911]

Network redundancy and data center reliability are two completely different animals imo. They are being cross bred in this thread...

[stomachmonkey]

Seems they are back up in under 12 hours.

Not terrible.

[stomachmonkey]

Originally Posted by KC911

Network redundancy and data center reliability are two completely different animals imo. They are being cross bred in this thread...

Two parts of the whole so perfectly fine to conflate them.

[Neilk]

I think lack of communication is what peeved me. If AT&T had said that it typically takes 12 hours to get back online, I would have been fine with the issue, but open ended, no details and rumors it could take days was the issue.

[KFC911]

Originally Posted by stomachmonkey

Two parts of the whole so perfectly fine to conflate them.

Respectfully disagree....redundancy to the periphery of a network is expensive and rare imo. If you paid for this level of service, and were out of commission due to a data center outage, then what? A 12 hr. DC outage (the ea$ier part imo) is freakin' huge. My former employers literally spent millions each year on DR, testing, etc. Two did it for "real" (megabanks), another International corp did it for insurance purposes only...."rolling the dice"...an expensive farce .

I haven't read about this particular outage though. IMO it just depends on "which part" out of mamy parts were the "cause", and failed to work as designed (or failed as designed due to cost).

I don't think about this crap anymore though...so I'm done here

[stomachmonkey]

Originally Posted by KC911

Respectfully disagree....redundancy to the periphery of a network is expensive and rare imo. If you paid for this level of service, and were out of commission due to a data center outage, then what? A 12 hr. DC outage (the ea$ier part imo) is freakin' huge. My former employers literally spent millions each year on DR, testing, etc. Two did it for "real" (megabanks), another International corp did it for insurance purposes only...."rolling the dice"...an expensive farce .

I haven't read about this particular outage though. IMO it just depends on "which part" out of mamy parts were the "cause", and failed to work as designed (or failed as designed due to cost).

I don't think about this crap anymore though...so I'm done here

Well it depends on your frame of reference.

From your side it matters, from the consumers side (the OP), they could care less. All they know is they are paying for something they are not getting and the why is inconsequential.

It was a lightning strike that caused a fire in a data center.

[MBAtarga]

The Atlanta Jackson-Hartsfield airport lost complete power one afternoon within the last year. There was a fire in one of the tunnels where power was fed that knocked out the service. Interesting thing was that the primary and secondary power cables went through the same tunnel, within INCHES of each other. Of course the fire damaged both sets. It had been deemed too expensive to route sources via different paths.

[KFC911]

I have personally experienced the "backhoe" gotcha that Steve mentioned earlier. Back in the early days of fiber....on paper it looked great...until a backhoe cut both fat trunks (different carriers, SUPPOSEDLY totally different paths through the city too...but nope ). Money was no object...but it happened. We limped along on copper and other data centers....took daze to fix fiber back then...others learned from us....not in a good way though

[stealthn]

Should look at The Bunker in Woodlands, ex nuclear shelter, great guys and good pricing