IT dudes, Co-lo screw up from hell

[masraum]

I'm a "network guy". I support the network gear, mostly Cisco(although that's changing), switches, firewalls, load balancers, that sort of thing.

We have several cages in an Equinix Data Center in the North-East. Today at 1300 ET ±2 mins, 3 bundles of fiber, 36 links, 72 strands between 2 of our cages were cut. These were mostly layer 3 switch/switch or switch/firewall links, but also a couple of circuits to other parts of the world.

We are all dying to know WTF someone was doing when they cut 72 strands of fiber in the middle of the day in the middle of the week in a major co-location facility.

I would say that I can't wait to hear the explanation, but I suspect I'll never hear the explanation.

And, of course, Equinix can't fix this fiber, they will have to call in a specialist company to fix it.

But in the grand scheme of things, it wasn't that bad. We had enough redundancy to get almost everything up except one item that was designed without redundancy, and I was only on the bridge for 6 hours which isn't too bad.

[Icemaster]

Knocked a rack over, pulled down a wireway.

Or zombies with a chain saw. Someone forgot to write mem. Take your pick.

[Scott R]

I have some bad Equinix stories as well, mostly with their "helping hands" service. But what really kills me is our NW Equinix facility can't take a full sized truck at their loading dock. Something they didn't tell us until after we inked the deal.

[stealthn]

Wow crappy, sounds like disgruntled employee

[eastbay]

LOL, backhoe alert!

[mattdavis11]

A good glass blower ought to have it back up in no time.

[HardDrive]

Someone with a backhoe who can't read the 'call before you dig' signs.

I'm pretty sure we work for the same company. I didn't notice any ill effects

[Vipergrün]

Uggh, you didn't get off too bad. Although 72 strands won't be fun to repair.

When I worked at UCSB, someone had run a bundle of cat5 cables down a loooong hallway, and attached them to a large plumbing pipe, on the top, out of site. Everything was fine until there was a water leak and they broke out the Sawzall. Yep, it sawed everything alright.

[KFC911]

Originally Posted by eastbay

LOL, backhoe alert!

Absolutely true story...back when fiber was just hitting the scene on corporate backbone networks, and wasn't easily repaired. On paper...we were redundent...different carriers even. Backhoe showed us the fiber trunks were passing through the same culvert in a different area of the city one afternoon... A.mega-data center in limp mode for a day or so...fun times

[Halm]

And this is one of the reasons why I focused our IT business on small/medium businesses. No data centers, no third parties. . .

[wayner]

If anyone needs a process/risk/ organizational change /ITIL guy let me know

[oldE]

They took my stapler.





Best
Les

[MBAtarga]

Originally Posted by oldE

They took my stapler.


Best
Les

the RED one?

[John Rogers]

Is your gear center/room secured with a secure lock system and is everyone who goes in logged? Hummm, should be I'd say? If the fiber lines are between two cages in the same room, why not get replacement fiber and just replace it. As you noted, no one would admit to it so.......

[flipper35]

Originally Posted by eastbay

LOL, backhoe alert!

This. When we were building an addition here on campus the contruction company was told absolutely do not break ground until the new fiber loop was in place. Well, they broke ground anyway on a Friday night and cut 24 strands from the building with our datacenter in to the main care facility and the apartments. That was a long night but we did get an all new loop in place and the company we had help us was awesome. I was just glad the horizontal boring was already done. I know our maintenance guy told the construction crew he was taking all the keys until we were ready for them. We had no phones, no networking and the fire panels used some of the fiber as well so they were all down. I was not here when the original fiber was first put in so I don't know why there was no redundancy prior.

Funny thing is, they knew that they would break the fiber when they started because the strands went where the new building was going.

[masraum]

Originally Posted by eastbay

LOL, backhoe alert!

My boss said something along the lines of "how did they get a backhoe into the data center?" hahahah.

My favorite is always "fire in the manhole"

[masraum]

Originally Posted by HardDrive

Someone with a backhoe who can't read the 'call before you dig' signs.

I'm pretty sure we work for the same company. I didn't notice any ill effects

Crazy thing is that this was all fiber inside the building between 2 cages INSIDE the data center.

[masraum]

Originally Posted by KC911

Absolutely true story...back when fiber was just hitting the scene on corporate backbone networks, and wasn't easily repaired. On paper...we were redundent...different carriers even. Backhoe showed us the fiber trunks were passing through the same culvert in a different area of the city one afternoon... A.mega-data center in limp mode for a day or so...fun times

Yeah, most of our redundancy includes getting maps of the exact routes of the fiber so we can see anyplace that they intersect or are close to each other.

[masraum]

Originally Posted by John Rogers

Is your gear center/room secured with a secure lock system and is everyone who goes in logged? Hummm, should be I'd say? If the fiber lines are between two cages in the same room, why not get replacement fiber and just replace it. As you noted, no one would admit to it so.......

Yeah, it's not fiber patch cables. It's structured fiber in the building. I'm pretty sure the company that comes in will be running new fiber to replace it, but it's not going to be a onesy twosy affair. I assume it's from the back of one patch panel to the back of another in the 3 bundles. I'm not sure if it's overhead or in the floor.

[stomachmonkey]

Run of the mill F'UP.

Someone in your cage backing into and knocking over your rack. Check.

Someone installing a power strip at the bottom of a rack so it gets stepped on and kills the rack when someone is working behind it. Check.

Someone not plugging the dial out line on the building UPS into the phone system so when another idiot places a ceiling tile on top of the exhaust vent on said UPS causing it to over heat and shut down the entire building and center of worldwide ops, Check.

None of those or the other dozen or so f'ups were me, i just got to deal with them.