So...A Coworker Screwed Up Big Today...

[legion]

A little background...

I am an application developer, and am responsible for servicing the applications I have developed. Last March, I sent some SQL to our Data team. It had a mistake in it. Specifically, through the wonders of copy and paste, my update statement had an extra semicolon BEFORE the where clause. For those of you who don't know SQL, that means update every row in the table. It's bad. Well, someone on the data team noticed my mistake and corrected it in my request by putting corrected SQL below my bad SQL. It was run successfully in test environments. When the person who picked it up to run in production environments, they didn't notice the corrected SQL and ran the bad SQL, taking down a big application for about 2 hours. (BIG DEAL.) Fortunately the recovery was straight-forward and nothing was corrupted.

Because of this and a few other similar screw-ups, we were rewarded with more process. All SQL must be tested, reviewed three times, can't be run during the day. A big hassle as suddenly a significant amount of my work had to be done after 6 p.m.. The guy who actually made the screw up was "reassigned" (demoted) to a job.

A coworker friend that I have lunch with daily, who happens to be on the data team but is not the guy who ran the wrong SQL, was pretty merciless. He made it clear that he thought me an idiot and that his team member should have been fired. He made this known to his manager. Never mind that that the guideline in place at the time from his team was simply to tell them what they wanted done and they would write the SQL. I had quickly put some SQL together as a courtesy. (See above, now his team only accepts tested, reviewed SQL.) That's not to say I'm not culpable in the screw up, but I didn't have his "throw the book at 'em' attitude about the screw up.

So I come in to work this morning to a flurry of e-mails. Major outage. Recovery has been in process for 15 hours, should be done in 17 hours. Real bad stuff. My coworker friend sends us lunch buddies an e-mail saying that he has been working for 26 hours straight, and hopes to be able to go home at 9:00 a.m..

It turns out that he ran some SQL in production yesterday that caused the outage. Through the wonders of copy and paste, he had run an update statement that had an extra semicolon BEFORE the where clause. It didn't bring down the application, but stuff kept running causing further data corruption. He realized his mistake right away, but it took another two hours to identify the affected processes and stop them. It then took another 17 hours to formulate a backout plan, identify the corrupted records, and execute the backout plan.

I expect that he will avoid me for awhile. After the 9 months of (sometimes mean-spirited) teasing about my screw-up, I expect he feels pretty embarrassed. But that's not the worst part. Being a vocal advocate for strong punishment for people who make such mistakes, I imagine he will have some uncomfortable conversations with his boss. And his boss's boss. And probably his boss's boss's boss.

[URY914]

You lost me by the third sentence.

[legion]

Summary:

I make mistake 9 months ago.

Coworker demands blood.

Coworker who demands blood makes same mistake today with MUCH worse consequences.

[Aurel]

From your summary, I would demand coworkers head now. [will read the details later].

[onewhippedpuppy]

Originally Posted by legion

Summary:

I make mistake 9 months ago.

Coworker demands blood.

Coworker who demands blood makes same mistake today with MUCH worse consequences.

Much better. Glass houses, black kettle, pick your metaphor. Don't knock someone's mistakes unless you yourself are perfect.

[cmccuist]

Chris, I have the philosophy that if you never made a mistake, you never did much in your career. And I have the same attitude about co- workers. That guy needs to check himself before he starts running his mouth again.

Oh, and it sounds like semi-colon placement is pretty important!!

[Zeke]

I say he goes down. Mainly because he was a dick. Once a dick, always a dick.

[legion]

Originally Posted by Aurel

From your summary, I would demand coworkers head now. [will read the details later].

Nope, he painted himself into a corner on this one. I'll make fun of him when this has blown over, assuming he still has a job.

I'm more realistic and believe that everyone screws up sometimes. When I demand consequences is when the same screw-ups are made repeatedly and they aren't learned from.

[msk1986911]

Life's too short...were lessons learned? Can they be used moving forward? Did anything happen that cannot be undone/corrected, albeit at a cost? My view from the comfort of my home on the east coast is to get over it and press on.

Be as kind to your co-worker as you would have wanted him to have been to you when you f'ed up; show that you are the better person.

[notfarnow]

Originally Posted by legion

Nope, he painted himself into a corner on this one. I'll make fun of him when this has blown over, assuming he still has a job.

I dunno. In my previous life in IT, I had the misfortune of working with a couple people very similar to your cohort. A funny thing I learned about people who are VERY hard on other's mistakes... they tend to never make mistakes themselves. If ever there's a situation where others may percieve that they made a mistake (ie, they DID make a mistake), they feel cornered and run a campaign of excuse making, finger pointing, and back stabbing. Watch yourself!

[Icemaster]

Karma's a *****.

So what happened to the change management process?

[legion]

Originally Posted by Icemaster

Karma's a *****.

So what happened to the change management process?

It now takes longer to navigate the change management process than to make the change.

[Icemaster]

It usually does....no matter where you go. There's a reason for that.

Sounds like whatever policy that was in place got violated if I understand it correctly.

[legion]

No. Everything was followed correctly. He had to modify the SQL to run in a test environment and when he changed it again to run in production the copy/paste error occurred.

More process can't prevent all possible mistakes. I also firmly believe that just because a mistake was made it shouldn't always create more process.

[aap1966]

Mistakes happen.
As long as they only happen once, I can live with it UNLESS it's rank stupidity rather than a mistake. Sounds like the technical issue was a mistake, but your colleague's response to your team's mistake earlier was stupidity. So he has to live up to the standards HE has set, or take the consequences.

Exceptions? Turn up for work pissed, the door's that way. I used to work in an ICU where one of the doctors turned up for work "tired". He wasn't sacked, so I quit.

[red-beard]

Originally Posted by onewhippedpuppy

Much better. Glass houses, black kettle, pick your metaphor. Don't knock someone's mistakes unless you yourself are perfect.

He who is without sin, cast the first stone...

[stealthn]

Take the high road, find allnthe job ads for Access prgrammers and put them on his chair before e comes into work.

Don't you have some sort of syntax teting app?

[john70t]

Mr. stealthn, you are not a nice fellow.

[stealthn]

I'm glad someone appreciates my humor

[jyl]

Seems like there is a pattern here of production code being repeatedly screwed up by copy/paste mistakes.