Electronic health record system crash

[Tobra]

This has been inconvenient.

The “network outage” began around 10:30 p.m. Monday and affected all 24 hospitals in the Sutter system. It was caused by the activation of the fire- suppression system in Sutter Health’s data center in Sacramento, according to an internal message sent by text to Sutter employees.

They call it a "Code Lime" Code Red is fire, they have a kaleidoscope of colors. They canceled all the elective surgeries, did everything on paper. Worst inconvenience has been the inability to access records. Was still out last night.

I specifically asked them about this very thing happening prior to purchasing the software. What if you have a fire or something at your data hub and it all crashes? "That can never happen, there are redundant systems and backups." Have not checked to see if it is on line yet today.

[Noah930]

That could never happen, says the salesman.

"Inconceivable," agrees the movie character.

[id10t]

https://www.theregister.co.uk/2017/06/16/bofh_2017_episode_7/





Buddy of mine manages teh Epic system for UFHealth/Shands - he says it keeps him busy

[KFC911]

Originally Posted by Tobra

...

I specifically asked them about this very thing happening prior to purchasing the software. What if you have a fire or something at your data hub and it all crashes? "That can never happen, there are redundant systems and backups." Have not checked to see if it is on line yet today.

Well that sux Doc...unless redundancy is provided at another location...preferably in a different part of the US...then they were FOS....as you have discovered. Back in the day...disaster recovery was a small part of "my game" for some heavyweight corporations...it was a high $ expenditure just to meet Fed requirements, audits, and insurance requirements. Though literally millions were spent on the "farce"... it was only on paper and for show . If the shtf they were gonna be down for the count...and "they" knew it...or they foolishly bought into some serious bs....

[GH85Carrera]

The Medical software systems are messed up. My dad was in the local heart hospital, and right next door is the hospital for regular medical care. They are two different hospitals, both under the same name. The use completely incompatible. They can't even export and import the other systems data.

And of course across town at the Stroke center thay use yet another system, and across town another large complex on a incompatible system. Dad was in his 80s and hitting all the hospitals for a while. He carried a sysops of his medical issues with him because every system had to strat from scratch and none could see the other systems date.

That should be criminal to make some poor nurse have to enter the same data at every hospital. You know each software vendor purposely make each package incompatible and will not use a standard format for the records.

[Rtrorkt]

a result of the fight between Epic and Cerner. Competing EMR's that do not talk to each other. My wife worked briefly for a lab reporting company whose software that was less expensive and more comprehensive than that offered in the EMR platforms were frozen out as both companies didn't want to give up the revenue or allow a third party to access their software.

The administration is doing nothing to force interoperable translation. The Affordable Care Act had provisions to try to force common EMR platforms and sharing of information but the current bozos don't see the need. There is no open source platform now though both Apple and Microsoft say they are working on a solution.

Think Beta vs VHS with life threatening potential outcomes.

[KFC911]

It seems to be happening here in NC, based upon my parents' care. Many facilities are now part of the UNC healthcare system, and the sharing of records, etc. has come a LONG way in recent years.

[Tobra]

Originally Posted by Rtrorkt

The Affordable Care Act had provisions to try to force common EMR platforms and sharing of information

two things

1.

2.Thanks for going political

Originally Posted by Rtrorkt

There is no open source platform now though both Apple and Microsoft say they are working on a solution.

Actually, there are a few free, open source systems, one of which is fairly prevalent, though primarily for office based practitioners.

Sutter and UC Davis both use Epic, but the two systems don't communicate with one another well

Looks like it is up, bye

[Rtrorkt]

Originally Posted by Tobra

two things



2.Thanks for going political

apologies for that. didn't really mean to get political

Just trying to make the point that the market won't solve the problem. Though I could see some of those open source versions you mention bulk up. With Sutter, Dignity, Kaiser, Mayo, etc all invested, primarily in Epic, that may be the VHS solution. Interestingly, Epic was started by a woman in her kitchen apparently.

[red-beard]

Originally Posted by Noah930

That could never happen, says the salesman.

"Inconceivable," agrees the movie character.

I don't that means what you think it means...

[flipper35]

HL7 has been out for years and no one wanted to use it.

We had decent a decent working relationship with Cerner and getting them to talk with our other system. We recently "upgraded" our EHR to a different version of what we had from the same company. One step forwards and slide back down the cliff.

[Tobra]

When these guys came to me in the first place, it was with Centricity, a GE product. Epic was not interested in selling to doctor's offices, until they figured out the amount of money that could be had. They just got that running okay and switched over to Epic. I told those mugs if they had a system that would work with all the other EHR's, they would crush everything else on the market. I used the VHS vs Betamax example at the time.

The EHR stuff was getting pushed prior to the ACA coming along, if I recall correctly. Nothing meaningful has ever been in place to get the various systems to communicate effectively. All this EHR stuff is just a vehicle to make it easier to audit providers and take money back anyway, so maybe it is not so bad the various systems won't communicate with one another.

[rockfan4]

Originally Posted by flipper35

HL7 has been out for years and no one wanted to use it.

We had decent a decent working relationship with Cerner and getting them to talk with our other system. We recently "upgraded" our EHR to a different version of what we had from the same company. One step forwards and slide back down the cliff.

We use HL7 a lot, to get the data out of the crap little systems that departments buy because they saw it at a trade show, and into first our own EHR and now Epic.

Flipper where are you? If by "Cheese Country" you mean WI then we aren't that far apart.

Tobra you may not know the answer to this, but does your Epic provider have a DR site? We run from ours for 24hrs once every 6 months.

EHRs were driven by Medicare, to cut down on Medicare fraud. When we went live with Epic a few of our providers retired rather than have to deal with the system.

I had no idea there were that many people on this site in the healthcare field.

[KFC911]

As a former systems guy for the "big fish", I can assure you that the "best" technical solution (Beta...used by small fish), will disappear most of the time...size DOES matter, and it's always....follow the money

[John Rogers]

I am very surprised that this hospital system does not have a dual or mirrored system in place where each system is in a different and very safe location. Having one system as you have found out is a pretty risky way to operate but then even the US Navy does that (or did) on their warships like carriers and support ships "as it can never happen to us"!

Our hospital system (Sharp Healthcare) has double systems so if one goes out for any reason....no one will even recognize it has happened! When I went to get my quad bypass surgery done at UCSD Heart Center 5 years ago I asked the heart surgeon the question......"what would happen about the time you put my circulation onto the little machine while you wrench around on my heart" and he said "everything is doubled"! I was happy to hear that as I would hate like hell to be in heaven looking down at a stupid computer blackout!

On a more serious I would hope the hospital's lawyers are at full throttle after those A-Holes who sold a system like that!?

[Noah930]

Originally Posted by rockfan4

EHRs were driven by Medicare, to cut down on Medicare fraud. When we went live with Epic a few of our providers retired rather than have to deal with the system.

Exactly. EMR is all about money (billing). Initially mandated to cut down on Medicare fraud. Now used to maximize billing, from the provider perspective. It has nothing to do with actually making health care any better. [/my political rant]

[flipper35]

Originally Posted by rockfan4

We use HL7 a lot, to get the data out of the crap little systems that departments buy because they saw it at a trade show, and into first our own EHR and now Epic.

Flipper where are you? If by "Cheese Country" you mean WI then we aren't that far apart.

Tobra you may not know the answer to this, but does your Epic provider have a DR site? We run from ours for 24hrs once every 6 months.

EHRs were driven by Medicare, to cut down on Medicare fraud. When we went live with Epic a few of our providers retired rather than have to deal with the system.

I had no idea there were that many people on this site in the healthcare field.

Platteville is the largest town near me. Dubuque is on the Iowa side.

[Por_sha911]

I wonder if "network outage" may be a veil for something more disturbing like a security breach or data corruption. If they have "redundant systems and backups" shouldn't they be able to have the data available?
When my company's system crashes and we can't access our client's data we asked to advise them we are "updating our system" and it should be back up in an hour...

[sammyg2]

[KFC911]

Originally Posted by Por_sha911

I wonder if "network outage" may be a veil for something more disturbing like a security breach or data corruption. If they have "redundant systems and backups" shouldn't they be able to have the data available?
When my company's system crashes and we can't access our client's data we asked to advise them we are "updating our system" and it should be back up in an hour...

Well, based upon my experience, it might truly have been a "network outage". Though I was a systems guy, networking & communucations was my area of expertise. Redundent systems and backup data is often the easy part....accessing it is a whole 'nuther matter .
Odds are, if an organization is not actually running off the DR platforms on a regular basis (as some have noted), then it's just a paper drill, an no one really has a clue if it will work as designed when needed...jmnsho.