slight work issue...how to handle it...

[mikester]

doink

[HardDrive]

Why hasn't the equipment been updated?

[oldE]

This sounds remarkably like a scenario my son described to me a few weeks ago. He too is an IT guy for a company which has had :a/ a system in place for a few years and :b/ turn-over in their IT dept.
That combination tends to create a back-log of problems which get 'bandaid' fixes which, in themselves create more problems.
A bigger problem can be managers who have some emotional baggage in their historical involvement with the system setup and pressure from their bosses to support the system for performance and cost targets.
Then along comes a new guy who sees the problems the system has got and feels hesitant to say what is wrong and what has to be fixed.

So, I'll give you the same advice I gave my son. Be proactive with this. If you like the company/site and would like a future there, try to come up with solutions and work with others to implement them. Remember the emotional attachment some people will have with the system currently in place. There will be a lot of inertia to overcome, but if you are not too proud to plant ideas (and, from time to time, let others think they are their ideas), you can make the changes happen and have a better work experience.
If they don't want to hear it, maybe you should think about moving on.

Good luck
Les

[RickM]

I agree with Les. A state of the network needs to be done. Assess the health of the entire environment, document and present your findings (along with recommendations) to both your boss and the director. Be clear on what needs to be done to maintain 100% availability.

BTW, a 5 min downtime max is unreasonable....$hit happens no matter how dilligent maint and support are.

[Paul_Heery]

Originally posted by RickM
BTW, a 5 min downtime max is unreasonable....$hit happens no matter how dilligent maint and support are.

I could not disagree more. Five minutes is easily achieveable if (big if) the infrastructure is there to support it. Properly configured and maintained load balancers could have avoided this outage automatically, assuming there is redundancy on the web tier.

I do agree that performing a RCA to identify the souce of the outage would be the best tact here. However, it sounds like Mike might not have the time to do one since he is fighting other fire.

Find blame in the problem, not the people.

[930addict]

+2 for Les' advice.

[RickM]

Originally posted by Paul_Heery
I could not disagree more. Five minutes is easily achieveable if (big if) the infrastructure is there to support it.

My take is that all involved know that software and hardware are outdated with reluctant support from the vendors.

On re-reading I see an audit was done but not 100% complete.

[Joeaksa]

Give them a plan on what needs to do to bring the system into today's world. They need to clear out the deadwood and upgrade where possible and shore up otherwise.

Be proactive and give them the good and bad news but also give them a solution on how to make it work, then do it if they give you the green light. Go into any meeting like this with the facts AND a solution.

You will look good and your career will get better along the way.

Joe

[azasadny]

You need to have SLA's (service level agreements) agreed upon by all and maintenance windows. Nobody ever wants their network down, but you've got to apply updates, test, troubleshoot, etc...

[tabs]

Mikester knows what the Tech problems and solutions are. His problem is a Political one, of how to get your solutions implemented so that the system works fine.

So Joeaksa gets my +1...Problems and solutions...just the facts Jack...Don't make it Personal.

If they are not willing to address these problems with Realistic solutions, then a nice memo duly documented to memorialize your postion would be warranted. Then the next time a failure occurs you can have a nice warm feeling that your head is safe. YOu might even want to send a memo about the solutions that were agreed upon in this upcoming meeting to keep their memory fresh if need be.

[tabs]

BTW Paper Trails scar the begeezus out of Bureaucrats.

[Moses]

Did you fill this out? You gotta put a cover sheet on the TPS report. Didn't you get the memo?

[Vipergrün]

Agreed that SLO's and SLA's are needed. Typically these are tied to up-time and un-plannned downtime, such as you experienced.

Maintenance windows, or planned outages are critical to any enterprise environment. If upper management does not 'get it', you are in a bad situation. If your environment is setup correctly, you can have donwtime and have minimal impact on customers.

Only in rare cases should you not upgrade software/firmware. Just wait until they are hacked due to some security hole being exploited.

You should be able to enable debugging on your routers/switches/etc and look at that after the fact.....

Good luck..