Biggest Screw Up At Work (No Deaths Plz)

[John Rogers]

Here is a post to allow the old timers or other "engineers to reflect back on something that really caused some issues and as usual I'll start.

In 1967 as a Navy nuclear machinist I was stationed at the General Electric/Navy training site near Saratoga Springs NY as an instructor and we shut down to do a reactor refueling and other upgrades such as switching to transistor based electronics. At that time the fuel cells were removed one at a time until all were gone then new fuel modules which were about 5.5 inches square and 48 inches long slid into the vessel. They were then under water and could go critical if the control rods came out! Anyways they were installed in a certain order from outside to the final one in the center which was zirconium to help pass those little neutrons along. Soooooooo the center cell went to go in after about a week and oh dear it was TOO BIG! So investigation time and this problem was finally traced to a warehouse worker who had been labeling the wooden boxes the fuel cells were in and he wrote the wrong code on the end of the crate! Seems he was in the wrong row so the item should have a label like 5 - 3 - D1G which meant row 5, stack 3 and D1G for New York reactor plant D1G. He was in row 4!

There was a lot of worry about which cell was wrong and it was about half way in so the cells were pulled, kept shielded and the correct one installed and the finished up. It was found the warehouse worker was a minimum wage fellow without a high school degree even so the fur really flew! Fortunately no reactor accident occurred and we finally got completed.

Lets hear yours?

[varmint]

Was going to say that my boss put the wrong year on the cover of a sport fishing almanac. And we had to pulp six thousand copies.

But the nuke thing is probably worse.

[GH85Carrera]

None of my jobs involved life and death situations or serious injury possibilities. Fortunately.

About the worst was only indirectly my fault. I was the production manager at a full service professional photo lab. That meant, do my production, and hold the hands of the other employees if they have a problem or difficult issue, and train the new employees. One new guy started and he wanted to run the black and white darkroom. Step one is show him the right way to process the black and white film. We went through the steps with the lights on and junk, already processed film, to show him what is going on. Then off go the lights and it is 100% cave dark. No hint of light at all, except some small glow in the dark spots to show you the edge of the processor sink. Everything else is done by feel.

He had processed film before as a hobby so he was a quick study. I waited in the corner listening to him do a film run and I was there if he had an problem. It went smooth so I left him to do the job.

The next day we discovered he was one of the people that closes their eyes in the dark since there is nothing to see.

The light in the B&W film room was just one little 60 watt bulb to help prevent light fog if the covers were on the tanks and we needed to turn the light on. He walked in, closed his eyes and started loading film on the racks with the lights on!

It ruined one photographers entire day of shooting. The good news it was all product photos shot in a studio so it could be re-shot but it wasted his day. We had to pay his costs and process the re-shoot for free. It could easily have been a once in a lifetime photo.

Since the one time that I was in there with him he did turn the light out, I could not see his eyes were closed. We put up two little glow in the dark spots over by the light switch which was on the wall 6 feet up. He learned to turn off the lights and make sure he saw the two dots before he opened up the film. And we added instructions to new employees, keep your eyes open all the time!

[legion]

I ran some SQL like this in our production environment:

Update table
Set column = 1;
Where ...;

Took our whole application down for a day. The application that my company essentially exists to use.

[id10t]

Originally Posted by legion

I ran some SQL like this in our production environment:

Update table
Set column = 1;
Where ...;

Took our whole application down for a day. The application that my company essentially exists to use.

I've done similar - like synch a list of faculty/courses/sections to our course management system and it removes all of the students since I said "this upload is the final result"

No damage done, was easy to fix (simply re-process students) but it did cause quite a few phone calls.

[KFC911]

Wasn't there already a similar thread recently? My screwups, though fairly rare, brought world wide corporations to an absolute screeching halt...I was a systems/networking guy and held the keys to the kingdom so to speak. Nobody bats 1.000 if you're in the batter's box for any length of time

I did hit a LOT of grand slams too fwiw

[Schrup]

Our crew had adjusted a Clayton valve for a hot refueling station at Ramstein AB. Dumbass closes the drain, but not the bypass. A little later they hook up to an F16 that is still running. When the crew chief pulls the deadman switch to began fueling, it starts the pump & the fuel hits the jet unrestricted. The whole jet rocked violently. The crew chief couldn't believe the wing tank seals didn't fail. If they had it would have been a giant fire & the pilot would have had to eject from the tarmac. We got really lucky. Dumbass was relegated to tool boy after that.

[JavaBrewer]

I led a team of folks building a prototype web app (this was in the late 90's). We were a small fish in a big pond...so to speak. I made the mistake of trusting a person I knew, who was on the big fish side, to stand up the services we relied our stuff to work on. I travelled weekly from SD to IAD to make this happen. Day of the critical demo the services, we relied on, were not online. I spent the next 53 days straight working to resolve the issue. Too little too late. The customer went with a competitor who later outsourced the entire project back to us for reduced cost. Still bitter.

[Evans, Marv]

I was a beginning rebar detailer for a rebar fabricator (1962). Rebar detailers take a set of plans; take out the rebar for each phase; make a list of sizes, dimensions, bends; make drawings; prepare placement instructions; schedule jobsite delivery; and make fabrication instructions and tickets for lots and numbers of bars. We did freeway bridges, parking structures, & other concrete reinforced structures. I switched the lot number with the number of bars on one fabrication ticket to the shop. The lot number was something like 967 & the number of bars was something like 50. It was number nine rebar - 1 1/8 in. diameter. The shop cut a pretty large amount before the shear operator decided to double check. That amounted to several times my yearly pay.

[John Rogers]

I may have had a post like this one before......as my memory is slowly fading away except at night when bunches of people and places seem to come together to make really interesting dreams? My doctor thinks that is because of my cholesterol medicine. She said as long as the dreams do not turn into nightmares I am okay then she'll give me something to make me sleep w/o dreams! Scary!!!

[wdfifteen]

My biggest screwup at work got me in trouble with both my bosses.
As an aerospace structural materials scientist, I was performing a set of stress corrosion cracking tests on a new alloy of aluminum and lithium. The protocol was to submerge the test coupon in a corrosive liquid after initiating a fatigue crack in it. To envision the coupon, imagine a square of metal about 2"x2" an 1/2" thick with a 1/2 'hole near the upper right corner and a 1/2" hole near the lower right corner. There was a notch cut in the coupon halfway between the holes. The coupon was attached to a hydraulic actuator via the holes that pulled on the coupon in such a way as to create a fatigue crack in the notch after about 100k cycles. After spending 10 days or so generating the proper fatigue crack, it was time to put the coupon on the creep frame for the constant stress corrosion test. I was setting up the stress test when I got called out for parade duty (I was in the Air Force) and I hurriedly set the test up, trying not to get into trouble with either of my bosses. To do it properly, you needed to put a load on the crack (using pull rods on the two holes) that was just under its rupture limit. But you had to submerge it in the corrosive fluid before applying the stress, so the separation of the fracture faces as the load was applied would draw the fluid in, and into contact with the fatigue crack. In my rush to get the test started before I had to go stand and salute I forgot to submerge it before applying the load. The test was trash because he new fatigue fracture face was not in contact with the corrosive fliud, all because some damn general showed up and they needed bodies to salute him. I got yelled at by my tech area manager for screwing up a critical test and wasting $20k, and I got yelled at by some Major for showing up late for parade duty. I was always in a tough spot. The civilian people i worked with thought my job was a research engineer and my civilian tech area manager was on my case to get work done, but the AF brass was always telling me,"Your JOB, is not science. Your JOB is to be a lieutenant in the United States Air Force." and dragging my ass out of the lab to play soldier at critical times. I hated the hell out of that job,.

[IROC]

I'm in charge of replacing a multi-million dollar component next January/February. If the installation doesn't go well, it will shut down a billion dollar DOE research facility for years. This replacement has never been done before. Radiation dose rates will be "uncomfortable" to say the least (~10,000 R/hr worst case). Here's a pic of the original component installation during facility construction (it's about 19' tall and weighs roughly 65,000 lbs):



Stay tuned. We do this kind of stuff all the time, but this is new challenge...

[KFC911]

^^^^
"Oops" - The word IROC don't wanna hear while at work

[John Rogers]

Working around anything that concerns radiation is scary, not like in the movies stuff but the fact you might end up like a rat in the microwave oven!

There was a New Port News cleaner/prep man who lost his balance while cleaning the vessel cover seating surface and fell into the "hot" water. This was the 1B reactor on the Big E (my baby). The residual stuff cooked him!

A freeze plug nearly melted (weeping a bit) on the #2 reactor of the Long Beach CGN9 while tied up at NAS North Island. If it had failed, the coolant would have drained into San Diego Bay and the fuel would have gotten a tad hot!

While off the coast of Iran during the hostage crisis, the CGN35, USS Truxtun, had a bad electrical fire in the #2 main switch gear, both reactors scrammed and neither diesel would start and come on line right away, no cooling pumps. After an hour #1 diesel finally did and the meters started to work and both reactors were hot, very hot due to no water circulation! We decided to start the pumps and it gave things a good shock but GE had made things pretty stout! The review board once we got to Subic Bay was lively to say the least!!!!

That's a different sort of PTSD than being blown up I imagine, sort of a slow long term stress.............

[VincentVega]

Originally Posted by legion

I ran some SQL like this in our production environment:

Update table
Set column = 1;
Where ...;

Took our whole application down for a day. The application that my company essentially exists to use.

ha

I've had a few copy/paste errors or similar. Why are we dropping traffic? Huh? Oh shoot, did apply something to the wrong interface/module/whatever? Surely not nuclear but in my world it was damn close.

[HardDrive]

Originally Posted by legion

I ran some SQL like this in our production environment:

Update table
Set column = 1;
Where ...;

Took our whole application down for a day. The application that my company essentially exists to use.

Oh yeah, a poorly placed semi-colon can ruin your day.

My colleague needed to do some testing, and instead of connecting to his local db schema, he connected to the main development schema that all of our devs use. He made some changes that he had not meant to do. Instead of just rolling them back, he forgot what he was attached to, and dropped the whole database! We work on a huge enterprise system, and it took us 2-3 days just to get our dev environment back. His name is Adam, and to date, we still talk about the Adam-bomb. Poor guy....

[masraum]

Originally Posted by VincentVega

ha

I've had a few copy/paste errors or similar. Why are we dropping traffic? Huh? Oh shoot, did apply something to the wrong interface/module/whatever? Surely not nuclear but in my world it was damn close.

Yep, copy/paste error into a firewall that didn't surface until the next day after active sessions had expired.

Device reload midday because I thought I was reloading the lab router, but had my windows mixed up.

Not me, but someone that I know who was providing support once turned on a debug that was too active that killed her remote session. She asked if the folks from the company that she was supporting still had access, they didn't. She then asked if they could reload the device. The said that they would call back, because the device was in an unmanned location 400 miles away, and they would have to send someone to do the reload.

I've also discovered several problems that had been traps waiting to spring for a long time. Someone configures something wrong or initially deploys something in a way that under the right circumstances, a failure is guaranteed. Eventually the failure occurs. I've been on calls where folks say "so when so-in-so did XXX, that caused the failure," and I've explained, "no, the misconfiguration of yyy that has existed for months or years caused the failure, but when so-in-so did XXX, that revealed the misconfiguration that has existed waiting for the right circumstances to fail."

Inevitably, "they" want to find a scapegoat. That person may not (usually not, IME) get fired, but they'll probably get a stern talking to or some sort of disciplinary action and everyone will be scared and a bit gun shy for a while afterwards. In some cases, someone's head will roll.

I learned about this and read it about 3 years ago. There are times that I'd like to send it to various levels of management. It should probably be required reading for management.
http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf

[Craig T]

In my early 30's I was a sales rep in Los Angeles for a East Coast big-pharma company. At an annual sales meeting in Chicago, after the meeting ended, a bunch of us went out partying on Rush Street. I ended up sleeping with a hot sales chick from a Dallas territory who told me she was single. It all seemed OK the next day when we said goodbye at the airport

Three days later the girl shows up at our LA office looking for me I get a call from my sales manager demanding I get into the LA office ASAP...and he's PISSED. Turns out the girl was married (no ring???) She told her husband about the affair when she got home. He booted her azz out and she shows up in LA wanting to stay with me! HR got involved...what a mess. Luckily I was performing and well over quota or I'd have been canned.

[widebody911]

Originally Posted by VincentVega

ha

I've had a few copy/paste errors or similar. Why are we dropping traffic? Huh? Oh shoot, did apply something to the wrong interface/module/whatever? Surely not nuclear but in my world it was damn close.

I did that once; I was configuring a PIX for a test environment and reversed the interfaces, which turned configured the PIX as the proxy ARP for the entire network.

[masraum]

Originally Posted by Craig T

In my early 30's I was a sales rep in Los Angeles for a East Coast big-pharma company. At an annual sales meeting in Chicago, after the meeting ended, a bunch of us went out partying on Rush Street. I ended up sleeping with a hot sales chick from a Dallas territory who told me she was single. It all seemed OK the next day when we said goodbye at the airport

Three days later the girl shows up at our LA office looking for me I get a call from my sales manager demanding I get into the LA office ASAP...and he's PISSED. Turns out the girl was married (no ring???) She told her husband about the affair when she got home. He booted her azz out and she shows up in LA wanting to stay with me! HR got involved...what a mess. Luckily I was performing and well over quota or I'd have been canned.

Wow, she must have been one of the hot ones that is good in bed because she was clearly bat-schit crazy. What happened to the girl?