Pelican Parts
Parts Catalog Accessories Catalog How To Articles Tech Forums
Call Pelican Parts at 888-280-7799
Shopping Cart Cart | Project List | Order Status | Help



Go Back   Pelican Parts Forums > Miscellaneous and Off Topic Forums > Off Topic Discussions


Reply
 
LinkBack Thread Tools Rate Thread
Author
Thread Post New Thread    Reply
Moderator
 
Z-man's Avatar
 
Join Date: Feb 2001
Location: NJ, USA
Posts: 8,560
Garage
Lousy day - and it's not over! (Work rant)

Got a call at 4:30am about a production server that had a hardware failure that effected the SAN Storage it was attached to. (I'm the SAN guy here at work). So I grab my clothes, jump in the car and come into work to verify that nothing has changed on the SAN environment. This after not being able to convince the guy on the other side of the phone that the SAN doesn't simply 'break' by itself - if there are no warning lights or if the box doesn't "phone home" to IBM, there simply is no problem. We're talking about enterprise level storage here - 5 9's stuff - Raid five with dynamic sparing, dual fibre connected servers -- there's so much redundancy built into the SAN fabric. I mean, if a cooling fan in the storage device (IBM ESS 2105-800) stops, the other cooling fans actually increase their RPM's in order to keep the box at the optimum temps.

Oh, and by the way, I'm supposed to coordinate a disaster recovery of an important Linux server at an IBM sight about an hour from work this morning. Hoping that I can verify the SAN, and then jump over to IBM and kick off the DR test without skippnig a beat. Ah the best laid plans of men...

Ok, so anyway, I get my butt into work early - 5:30/6-ish, and verify the system. Sure enough, everything's fine on the SAN side, but the lousy Microsoft servers cannot connect. The operating system on DB9 is completely corrupt and has taken a non-releasable reserve on a LUN (logical disk) that DB10 needs to see. (Note: I hate Microsoft Clustering - it's about the worst server/SAN methodology I have ever worked with). We come to this conclusion around 10:00am. (I was supposed to kick off the DR test around 8:00am. Four people are anxiously awaiting my arrival - but I gotta work on produciton first. Two of them decide to come into work instead. Good call.)

Oh, and in the meantime, I'm getting slammed left and right with a million other production related problems. BTW: A little less than a year ago, my college was fired. Why? Harassment. (Yeah, she was harassing me - to the point that as soon as she came back from her appointment with HR, where she was told explicitly to leave me alone, she ripped me a new hole. Infront of our boss. That was the last straw.) So half the staff was eliminated, but the work has increased ever since then. And they will not fill her position with someone else. Why? The company has decided to freeze all open positions to save money. Thank you very much. Remind me NOT to be a company man.... but I digress...

By around 1:00pm they call off the DR test. At least I didn't have to worry about going up there AFTER work to get the test done, though there was talk of doing so, since, after all, we have the site until midnight tonight!

So I'm putting out fires left and right - trying to get work done, and all the while, I'm supporting this Microsoft Cluster-F***k. Removing logical luns, adding luns, verifing the zones in the McData SAN fabric, removing disks on the IBM storage box, adding them....basically turning the disks upside down to try to get these freakin' servers to both see all the disks.

Meanwhile, I contact IBM to assist. They dial into the box, and as I suspected, the box is running just fine. The reserve is on the software level. and there are two ways to resolve:
1. Get the original system that had the reserve back online
2. Do a POR on the storage box. (POR = POWER ON RESET, IE: REBOOT the box.) Mind you, this is a production box - with about 17 TB (that's terabytes) of production data on it. With DB9/DB10 down, it's only effecting about 300-400 users. Bring this box down, and the whole company infastructre suffers. Nation wide. All our important stuff. Basically business would grind to a halt and the company would loose money. Big time.
So we all wisely chose option #1. Problem is, it is going to take a long time to recover the bad operating system - we were hoping the Raid 5 disks could rebuild the two failed drives (it didn't) so plan two was to recovery from Tivoli backups. (that finally worked...but I'm getting head of myself here...)

So, back to our saga - as all this LUN masking/ Zoning / bouncing disks left and right is happening, another LINUX system desperately needs a backup put in place. No biggie - it's an FTP from VM to MVS to initial a 5 job process that backs up about 15GB worth of disk, and an FTP back to the VM/LINUX server. Got the JCL set up easy (since the DR test was a very similar setup and we got all that ironed out last week). Problem is, I'm having trouble putting the code to production, via our new change control package, SCLM.

SCLM - oh yeah, I was 'deemed' the administrator of that product too. Went live a couple of weeks ago with it. It was not a smooth transition. But the System manager insists that I take on all responsibilities of this new product (that incidentally, wasn't fully and properly installed by he and his staff - typical install - there's always a shakedown period and fallout that needs resolution. Problem is that I'm the one who's getting dumped on)

So anyway, I'm using SCLM to move this code to production, and hit a snag. I ask the system manager guy to help me resolve the issue and politely note that it is a production issue. He refuses to even look at my change record. I speak to my boss, and we speak to his boss, who calls Mr. System Manager. When my boss' boss asks this guy to help, there is silence on the other end. So my boss-boss says "This is BS - I'm coming over and we're resolving this now." Long story short (too late), Mr. System Manager reluctantly moves my change into production. Of course we missed the backup window, and will have to wait till tomorrow to back a clean backup. This guy has been giving others major grief as well - I just don't understand why people do malicious things, especially at work - I mean really - it's just freakin' work. Get the job done, go home, and live your life. No reason to make everyone else miserable just because you're a miserable SOB...

Here a LUN, there a LUN, everywhere a LUN-LUN - still chasing the problem with good -ol DB9/DB10 - finally the DB9 system is restored about 3pm - but the persistent problem won't go away - DB10 still can't see half the disks, and DB9 can't see the other half. So we start bouncnig the servers, removing disks here, adding them there....till finally, about an hour ago, both systems have come up and can see all disks! But we're not out of the woods yet...

Now all that is needed is to get Microsoft Clustering reinstalled on both systems, and have them come up again without problems. The NT/Windows server guy has been working on it for the past hour. It's not supposed to take this long...afriad to ask what the hold up is...

It's 8:15pm. I'm at work, I'm frustrated beyond normal - especially since Mr. Manager was such an idiot - taking a personal vendetta to the point where it jeopardizes production...I am tired, I probably smell, and I just want this day to end...

All I know is - this coming weekend is the first DE of the year - three days at Pocono International Raceway. I don't want to know nothing about work starting Thursday afternoon...

I could deal with being over-worked. Deadlines for me are now a figment of the imagination - "if you don't give me backup, I can't promise you I'll me the deadlines you impose on me." And so far, I've been keeping my head above the water. But when I'm getting slammed with more work, and not getting the help I need when I ask for it due to a personal vendetta, that's just wrong. And unacceptable.

Rant over, if you've read all this, I am amazed by your persistence. Thank you.
-Z-man.

__________________
1989 944S2 - June, 2001 - January, 2014

Visit my homepage!
<><
Old 04-03-2006, 05:18 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #1 (permalink)
canna change law physics
 
red-beard's Avatar
 
Join Date: Jul 2000
Location: Houston, Tejas
Posts: 38,648
Garage
14 Hours? Not too bad....I've been on site for 4 days straight, no shower, no shave, can't afforrd for the plant to go down during the acceptance test. Catch sleep here and there on cardboard on the floor.

Or the time I put in a Full day in Rhode Island then drove to Hartford only to pull an all nighter to restore the data recording system (A Microvax) for a power plant. I think I finally got to a hotel room about 10am the second day (Up about 40 hours including a 2 hour drive).

Yep, the good ole days. I don't miss them at all.
__________________
James
The pessimist complains about the wind; the optimist expects it to change; the engineer adjusts the sails.- William Arthur Ward (1921-1994)
Red-beard for President, 2020
Old 04-03-2006, 05:31 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #2 (permalink)
Registered User
 
Join Date: Sep 2001
Location: Tucson AZ USA
Posts: 8,229
(Jeez, Z.....Be quiet or snowman will think you did not choose wisely in your career path....)
__________________
Bob S. former owner of a 1984 silver 944
Old 04-03-2006, 05:33 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #3 (permalink)
Banned
 
fastpat's Avatar
 
Join Date: Mar 2005
Location: Travelers Rest, South Carolina
Posts: 8,796
Yeah, I spent 17 hours in surgery, beginning at about 9Pm, five different patients.

Don't miss that either.
Old 04-03-2006, 05:38 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #4 (permalink)
Moderator
 
Z-man's Avatar
 
Join Date: Feb 2001
Location: NJ, USA
Posts: 8,560
Garage
The workload, though overwhelming, I can bear. I've done the 2-3 days of work with little or no sleep before. It's the personality BS and the political crap and personal vendettas that I can't stand.

-Z.
__________________
1989 944S2 - June, 2001 - January, 2014

Visit my homepage!
<><
Old 04-03-2006, 05:56 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #5 (permalink)
Detached Member
 
Hugh R's Avatar
 
Join Date: May 2003
Location: southern California
Posts: 25,010
Z

I don't have a clue what you wrote, but I hope you get some rest.

My plane arrived an hour late in New Orleans and I spent almost an hour getting my internet connection to work. Ended up changing hotel rooms. Ordered steak and got fish, with nothing else, just the f'n slab of fish, called room service 45 minutes ago, still waiting.
__________________
Hugh

Last edited by Hugh R; 04-03-2006 at 06:28 PM..
Old 04-03-2006, 06:25 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #6 (permalink)
 
Unconstitutional Patriot
 
turbo6bar's Avatar
 
Join Date: Apr 2000
Location: volunteer state
Posts: 5,618
It could always be worse. Your sig line could read 2004 Toyota Prius.
Old 04-03-2006, 06:25 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #7 (permalink)
canna change law physics
 
red-beard's Avatar
 
Join Date: Jul 2000
Location: Houston, Tejas
Posts: 38,648
Garage
I think you misspeled that. I think it's "Toyota Pious"
__________________
James
The pessimist complains about the wind; the optimist expects it to change; the engineer adjusts the sails.- William Arthur Ward (1921-1994)
Red-beard for President, 2020
Old 04-03-2006, 06:27 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #8 (permalink)
Registered User
 
williecoyote's Avatar
 
Join Date: Jun 2004
Location: outer banks,n.c.
Posts: 466
Quick everybody post nekkid pictures in OT while Z-man's asleep.
__________________
Will

85 carrera
Old 04-03-2006, 06:34 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #9 (permalink)
canna change law physics
 
red-beard's Avatar
 
Join Date: Jul 2000
Location: Houston, Tejas
Posts: 38,648
Garage
Shaaddduuppppp! You're gonna ruin it for all of us!
__________________
James
The pessimist complains about the wind; the optimist expects it to change; the engineer adjusts the sails.- William Arthur Ward (1921-1994)
Red-beard for President, 2020
Old 04-03-2006, 06:35 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #10 (permalink)
1977 Targa ROW
 
M.D. Holloway's Avatar
 
Join Date: Aug 2004
Location: Rolling Hills Estates, HV TX
Posts: 22,126
Z, I feel your pain! Got into work about 10:30 and there was no more half-n-half for my coffee. Man I hate that! Back to my desk I got a paper cut opening up my pay stub. Took a a nap before lunch and was woken up by a wrong number - at least I think it was. Maybe it was a cell call. I could n't understand anything past 'Mike, Ineedyour help figueringoft oyu bla bla bla" I have time for that nonesense!.

After my normal 2 hour lunch which gave me heartburn (dang triple servings of those beef frajetas and beer!) and took another nice nap in my car in the parking lot. What a lovely 85 degree day - windows down and some soft jazz playing. A gentle breeze blowing through the car when I was rudely woken up by some inconsiderate naive trying to trim the grass around our building. about 4:45 I ended up going back up to my office where I had to talk to somebody about something - can't really remember what it was about but they said that we had a meeting and they were waiting in my office for 45 minutes. Oh well, CEO's really have no sense of timing or consideration. I really have to slow down this pace before I burn out!

OK - my rant is over...
__________________
Michael D. Holloway
MLA, MLT, OMA, CLS
DOUOSVAVVM
I'm not your Rumpelstiltskin, spin your own Gold.
"Ya can't be Pablo if ya work ain't selln'..."
Old 04-03-2006, 09:12 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #11 (permalink)
B58/732
 
BlueSkyJaunte's Avatar
 
Join Date: Feb 2000
Location: Hot as Hell, AZ
Posts: 12,311
I really wish I didn't understand your post.

Microshaft screws everybody in the end...
__________________
ΜΟΛΩΝ ΛΑΒΕ
I don't always talk to vegetarians--but when I do, it's with a mouthful of bacon.
Old 04-03-2006, 09:51 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #12 (permalink)
Super Moderator
 
cstreit's Avatar
 
Join Date: Feb 2000
Location: Naperville, IL USA
Posts: 13,321
Garage
God I remember being on call for production systems... After reading your nightmare Z-man I think I got sweaty palms like I was there all over again...

Cobol, JCL, and VSAM, oh my!
__________________
Chris
----------------------------------------------

2001 GT3 Cup - "Pepin"
1999 Spec Miata
1996 993 RS Replica
2010 Harley 48
2013 Honda 250 CRF
1971 Norton 750 Commando
Alcon Brake Kits
Old 04-03-2006, 10:21 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #13 (permalink)
Registered User
 
Join Date: Sep 2001
Location: Tucson AZ USA
Posts: 8,229
Older than that..FORTRAN and PL-1 !!!

(working from decks of punch cards in the late 60s..)

The jobs would line up and be run at night. On one, which I remember was a simple modulus of elasticity program, the engineer forgot to put an "end of run" card. Thing ran for hours. In real life, what would have been a bending problem would have been a coil!! Held up other jobs and led to a 36 hour marathon. Thank heavens for modern computers!!
__________________
Bob S. former owner of a 1984 silver 944
Old 04-03-2006, 10:29 PM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #14 (permalink)
Kantry Member
 
oldE's Avatar
 
Join Date: Feb 2001
Location: N.S. Can
Posts: 4,007
Zoltan,
I read your post. Then I re-read a bit of it. I was going to c&p it to my son (a computer techie), but then realized I understood enough of it to understand your day was **** through and through.
Sorry to hear that. Hope today is better and the support issue gets resolved.
Les
__________________
"Emerson, there's an easy way to get rid of the understeer. Just use more throttle." Jochen Rindt
Old 04-04-2006, 05:07 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #15 (permalink)
Cars & Coffee Killer
 
legion's Avatar
 
Join Date: Sep 2004
Location: State of Failure
Posts: 29,934
I love Z/OS!

That, and we test for a minimum of a year before hitting production. Just installing something on a box around here is the quickest way to the unemployment line.

Tell us if you experience anymore in-SAN-ity...
__________________
Some Porsches long ago...then a wankle...
5 liters of VVT fury now
-Chris

"There is freedom in risk, just as there is oppression in security."
Old 04-04-2006, 06:07 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #16 (permalink)
Too big to fail
 
widebody911's Avatar
 
Join Date: Jan 2002
Location: Carmichael, CA
Posts: 32,038
Garage
Send a message via AIM to widebody911 Send a message via Yahoo to widebody911
Re: Lousy day - and it's not over! (Work rant)

Quote:
Originally posted by Z-man
Rant over, if you've read all this, I am amazed by your persistence. Thank you.
The burning question for me is: did you get to keep the stapler?
__________________
"You go to the track with the Porsche you have, not the Porsche you wish you had."
'03 E46 M3
'57 356A
Various VWs
Old 04-04-2006, 08:22 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #17 (permalink)
Moderator
 
Z-man's Avatar
 
Join Date: Feb 2001
Location: NJ, USA
Posts: 8,560
Garage
Well, after a good night's sleep, I am much better. The DB9/DB10 systems are now stable. So it's back to the grid.

Good to know there are others here at Pelican with an 'old school' IT background. Crazy thing is, as this storage environment is growing, I've been responsible for open systems stuff as well - UNIX/AIX & SUN Oracle, as well as crappy Microsoft. And ya know what? The most stable platform is still the mainframe, those the midrange stuff comes pretty close. As far as Microsoft - it works, unless it breaks, then it's a really pain to get back, like yesterday.

legion: in-SAN-ity: I like that!

-Z.
__________________
1989 944S2 - June, 2001 - January, 2014

Visit my homepage!
<><
Old 04-04-2006, 08:48 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #18 (permalink)
 
Registered User
 
pwd72s's Avatar
 
Join Date: Apr 2001
Location: Linn County, Oregon
Posts: 38,097
A line from a B grade movie (poolhall junkies) comes to mind here. Delivered by Rod Steiger, his last acting roll:
"Every day is a good day. Try missing one once."
__________________
"Now, to put a water-cooled engine in the rear and to have a radiator in the front, that's not very intelligent."
-Ferry Porsche (PANO, Oct. '73) (I, Paul D. have loved this quote since 1973. It will remain as long as I post here.)
Old 04-04-2006, 08:52 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #19 (permalink)
Registered User
 
}{arlequin's Avatar
 
Join Date: Jun 2003
Location: DC/Boston
Posts: 5,500
Garage
grid girls make everything allright

__________________
dave
1973,5: one two thweeee!
no.don't.stop.
Old 04-04-2006, 08:52 AM
  Pelican Parts Catalog | Tech Articles | Promos & Specials    Reply With Quote #20 (permalink)
Reply

Thread Tools
Rate This Thread
Rate This Thread:

 


All times are GMT -8. The time now is 03:58 AM.


 
Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Search Engine Optimization by vBSEO 3.6.0
Copyright 2020 Pelican Parts, LLC - Posts may be archived for display on the Pelican Parts Website -    DMCA Registered Agent Contact Page
 

DTO Garage Plus vBulletin Plugins by Drive Thru Online, Inc.