Massive Health IT Outage: But, Of Course, Patient Safety Was Not Compromised

Having been 'Down Under' in Sydney addressing the Health Informatics Society of Australia on the need to slow down their national health IT program - and on the need to think critically about HIT seller public relations exaggerations and hubris - and being very busy, I missed this quite stunning story of a major health IT outage.

Just a typical "glitch":

Some lessons from a major outage
Posted on July 31, 2012
By Tony Collins

Last week Cerner had a major outage across the US. Its international customers might also have been affected.

InformationWeek Healthcare reported that Cerner’s remote hosting service went down for about six hours on Monday, 23 July. It hit “hospital and physician practice clients all over the country”. Information Week said the unusual outage “reportedly took down the vendor’s entire network” and raised “new questions about the reliability of cloud-based hosting services”.

A Cerner spokesperson Kelli Christman told Information Week,

“Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety.  [Meaning, for the most part, blank paper - ed.] The issue has been resolved and clients are back up and running. A human error caused the outage.  [I don't think they mean human error as in poor disaster recovery and business continuity engineering - ed.]  As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made.”

Christman did not respond to a question about how many Cerner clients were affected. HIStalk, a popular health IT blog, reported that hospital staff resorted to paper [if that was true, that paper was OK in an unplanned workflow disruption of major proportions, then why do we need to spend billions on health IT, one might ask? - ed.] but it is unclear whether they would have had access to the most recent information on patients.

One Tweet by @UhVeeNesh said “Thank you Cerner for being down all day. Just how I like to start my week…with the computer system crashing for all of NorCal [Northern California].”

Tony Collins is a commentator for ComputerWorldUK.com.  He's quoted me, as I wrote in my May 2011 post Key lesson from the NPfIT - The Tony Collins Blog.

This incident brings to life longstanding concerns about hospitals outsourcing their crucial functions to IT companies.  

Quite simply, I think it's insane, at least in the foreseeable future, as this example shows.

It also brings to mind the concerns that health IT, as an unregulated technology, causes dangers in hospitals with inadequate internal disaster and business continuity functions aside from fresh sheets of paper.  Such capabilities would likely be mandatory if health IT were meaningfully regulated.

The Joint Commission, for example, likely issued its stamp of approval for the affected hospitals, hospitals who had outsourced their crucial medical records functions to an outside party that sometimes went mute.  If someone was injured or died due to this outage, they would not care very much about the supposed advantages.

There's this in the article:

... “Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption. Outage was across all North America clients as well as some international clients.”

Of course, patient safety was not compromised.

Finally:

Imagine being a patient, perhaps with a complex history, in extremis at the time of this outage.  

I, for one, do not want my own medical care nor that of my relatives and friends subject to cybernetic recordkeeping unreliability and incompetence like this, and the risk it creates.

-- SS

Aug. 8, 2012 addendum:

The Los Angeles Times covered this outage in a story aptly entitled "Patient data outage exposes risks of electronic medical records."

They write:

Dozens of hospitals across the country lost access to crucial electronic medical records for about five hours during a major computer outage last week, raising fresh concerns about whether poorly designed technology can compromise patient care.

My only comment is that the answer to this question is rather axiomatic.

They also quote Jacob Reider, acting chief medical officer at the federal Office of the National Coordinator for Health Information Technology, who said:

"These types of outages are quite rare and there's no way to completely eliminate human error"

This is precisely the type of political spin and misdirection I cautioned the Australian health authorities to evaluate critically.

Paper, unless there is a mass outbreak of use of disappearing ink, or locally hosted clinical IT, do not go blank en masse across multiple states and countries for any length of time, raising risk across multiple hospitals greatly, acutely and simultaneously.  (Locally hosted IT outages only cause "local" mayhem; see my further thoughts on this issue here).

-- SS

Forbes: Obamacare Billionaire: What One Entrepreneur's Rise Says About The Future Of Medicine

An article in Forbes appeared today (4/18/12) entitled "Obamacare Billionaire: What One Entrepreneur's Rise Says About The Future Of Medicine" by Matthew Herper.

The article is largely a hagiography of Cerner founder Neal Patterson:

North Kansas City is an unlikely place to launch a revolution in American health care. Yet here, amid the dilapidated grain elevators, fast food joints and vast green plains, the dream of using computers to keep you alive at a reasonable cost is battling onward. In a bunkerlike building built to withstand a direct hit by a category five tornado, 22,000 servers handle 150 million health care transactions a day, roughly one-third of the patient data for the entire U.S. Records of your blood pressure, cholesterol, lab test results, that gallbladder surgery last year—and how much you paid for it—may sit there right now. Armed guards stand watch.

This is a data center at the headquarters of Cerner, the world’s largest stand-alone maker of health IT systems—and company number 1,621 on FORBES’ Global 2000 list—where the blood-and-guts realities of medicine meet the ­sterile speed and exactitude of the computer revolution.

Omitted are some pertinent negative accounts, such as Cerner's role in the failed £12.7bn ($20bn U.S.) National Programme for IT in the NHS (NPfIT), as described here and here.

Patterson is quoted with the standard industry bellicose grandiosity and hysterics about computers "revolutionizing" (as opposed to facilitating) medicine:

... In 1999 a report from the prestigious Institute of Medicine gave Patterson hope that the rest of medicine was ready to follow in Mayo’s footsteps. Titled “To Err Is Human,” the report detailed how between 44,000 and 98,000 people die every year in hospitals from preventable mistakes, like getting the wrong medicine or the wrong dose of the right one. The ­report specifically prescribed better computer systems as a way to prevent these deadly mistakes. Patterson cites that study as the moment when health IT entered the mainstream. But it was still slow going, and that drove him nuts. His customers at that time were more worried about the Y2K bug than they were about revolutionizing health care.

I first heard him and other HIT CEO's uttering the "revolution" line at a Microsoft Healthcare Users Group meeting ca. 1997 and attended largely by IT technicians. I was probably the only Medical Informatics-trained professional in attendance, and perhaps the only physician there.

When I then asked those in the room how many had healthcare experience or had even read the Merck Manual, few hands went up. The CEO's could not then satisfactorily explain how medicine would be "revolutionized" by such a crowd. (One of those CEO's, of erstwhile HIT vendor HBOC, was later found to be seriously cooking the books after acquisition by McKesson. See "Former McKesson HBOC Chairman Convicted of Securities Fraud; Defrauded Investors Lost in Excess of $8 Billion" at FBI.gov. Some revolution...)

Unfortunately, most medical errors have little to do with documentation, either paper or electronic, as I wrote in Dec. 2010 at "Is Healthcare IT a Solution to the Wrong Problem?". The latter, however, has introduced many new errors modes and other social-technical problems, permitted massive security breaches (it's very hard to haul away 10,000 paper charts without being noticed) and opportunities for medical records evidence spoliation simply not possible with paper.

There are a few Ddulite-ish quotes from Eric Topol and David Bates in an article with a clear tone of exalting health IT.

I am quoted in the Forbes article in about the only comment critical of health IT, and the only comment concerning the ethics of human subjects experimentation conducted with this technology, as it exists in 2012:

... the Hippocratic oath says nothing about breaking eggs to make omelets. “We’re kind of headed in the wrong direction,” says Scot Silverstein, a health IT expert at Drexel University who believes that the current systems are too prone to randomly losing data [I had cited this study - ed.] and complicating doctors’ lives.

While the "breaking eggs to make omelets" metaphor was not mine, it reminded me that I just carried out my final earthly duty for my mother, injured in mid 2010 by a health IT-related error and deceased since June 2011. I filed her final IRS tax return yesterday, the due date.

I just hope my mother is enjoying her omelet in the Pearly Gates Diner. Scrambled eggs were about all she could eat well after the health IT accident.

As a result of that experience, my new business card (phone # redacted):

(Click to enlarge)

I no longer merely write about health IT risks. I find past involvement with attorneys early in my medical career, as Manager of Medical Programs and Medical Review Officer (drug testing officer) for one of the largest public transit authorities in the United States, quite helpful in this new role.

The "eggs that get broken", as in a well known old nursery rhyme from the early 1800's, cannot be reassembled. Those injured or killed in the unregulated, thoughtless, cavalier journey to some mystical medical cybernetic utopia deserve justice, and the eggheads responsible for their harms have earned the privilege of explaining themselves in the courtroom and being properly penalized where appropriate.

The future of medicine - if it is to not become a nightmare ignoring the lessons of history, repeating the mistakes of the past - belongs to those willing to take an ethical stand in defense of medicine's core values, and in defense of patients.

-- SS

The Very Latest Health IT "Glitch" - Britsh MP Says No to Cerner

It's just a glitch:

Bacon calls for halt on Millennium
e-Health Insider.com
19 January 2012

Conservative MP Richard Bacon has called for a halt to all Cerner Millennium deployments following appointment problems and delays at the latest trusts to go-live with the system - North Bristol and Oxford.

Bacon, who has followed the progress of the National Programme for IT in the NHS [NPfIT - ed.] for many years, said the two hospitals had been “brought to their knees” by the implementation of the new electronic patient record system.

“These deployments need to be stopped until we are sure that they can be managed safely,” he said; adding that the system should be "switched off" if it was not working for patients.

North Bristol NHS Trust and Oxford University Hospitals NHS Trust said they are working through some deployment issues, but denied that patient safety has been compromised.

[There's that 'safety has not been compromised by major IT system disruptions' line again - see here - ed.]

However, Oxford University Hospitals told eHealth Insider that it has had to bring in extra staff to help it overcome some “temporary problems while the new system beds in.”

Bacon said local news reports indicated that the trust was having serious difficulties booking patients in for treatment.

The Oxford Mail has reported that problems were so bad before Christmas that the trust had to suspend its parking charges as clinics over-ran by hours.

... Bacon, who was instrumental in triggering last year’s National Audit Office and Commons’ public accounts committee inquiries into the programme, said the NHS should never have been locked into buying software that was “unreliable” and “unreasonably expensive."

“Effective, affordable and robust IT systems are vital to the future of the NHS, but it is clear that the fiasco that is the national programme cannot deliver them,” he said this morning.

He called for “a halt” to new Cerner Millennium deployments, including that at Imperial College Healthcare NHS Trust, which is being undertaken by BT as the local service provider for London, and that at Royal Berkshire NHS Foundation Trust, which went outside the national programme more than two years ago.

[More typical hospital executive-style excuses and spin control follow]

Readers, I won't bore you with the rest of the excuses, pleadings for special accommodation, etc. You've heard them all already on this blog.

However you can read them at the link above if you so desire.

-- SS

North Bristol Hits Appointment Problems: Another "Our Lousy IT Systems Screwed Up, But Patient Safety Was Never Compromised" Story

At my Dec. 2011 post "IT Malpractice? Yet Another "Glitch" Affecting Thousands of Patients. Of Course, As Always, Patient Care Was "Not Compromised" referencing prior posts, I wrote:

... At my Nov. 2011 post "Lifespan (Rhode Island): Yet another health IT glitch affecting thousands - that, of course, caused no patient harm that they know of - yet" I wrote:

There's been yet another health IT "glitch" that, of course, caused no patients to be harmed. See other "glitches" here, here, here and at other posts which can be found by searching this blog on the banal term 'glitch'.

Another "our clinical IT crapped out , BUT ... patient care/safety was never compromised" story just arose:

North Bristol hits appointment problems
E-Health Insider
11 January 2012
Rebecca Todd

Clinicians working at North Bristol NHS Trust have expressed concern about disruption to patient care, which they say is caused by appointment problems following the go-live of a new Cerner Millennium electronic patient record system.

I would have entitled the article "North Bristol hit by IT-created appointment problems."

Reported problems include patients being booked into non-existent clinic appointments or not being told about scheduled operations, resulting in some operations being cancelled.

No patient care compromise possible there. Who, after all, needs a timely operation? It frees up a lot of money for IT golf tournaments to let those of no value to society (i.e., the old, and those who will not admit computers in healthcare with deterministically revolutionize medicine because, well, they're magic) simply die due to delayed or cancelled surgery...

Ehealth Insider understands that some of the problems relate to the way the trust configured the EPR system; including setting up dummy clinics for which appointment letters were subsequently sent out.

It's never the software or computer's fault.

As a matter of fact, I have not seen any official response to the work of Dr. Jon Patrick at U. Sydney on the many software engineering flaws of another product of the same company. His work is entitled "A study of an Enterprise Health information System" and is at this link: http://sydney.edu.au/engineering/it/~hitru/index.php?option=com_content&task=view&id=91&Itemid=146. Do they have Class Action lawsuits in Australia?

In a regional BBC news report, aired on Monday evening, anonymous hospital clinicians called the implementation a “complete shambles” and said it represented a “potential danger” to patients.

According to the BBC report, the problems meant patients were being booked for impossible appointment times, such as 12.05 am, and quoted correspondence saying staff and the system were both on the “verge of meltdown."

The clinician comments are anonymous since non-anonymous reporting would get the clinicians declared health IT apostates, and then excommunicated. Non-anonymous 'whistleblowers' could also fear being sued due to possible gag clauses - the kind of clause hospital executives sign in violation of their fiduciary responsibilities to their staff and to patients. (See my 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards"at this link and the much-expanded essay on the same themes "Health Care Information Technology Vendors' Hold Harmless and Keep Defects Secret Clauses" at this link.)

Martin Bell, director of IM&T at the trust, confirmed to EHI that North Bristol had experienced some “unexpected problems” in the past few weeks with some of the outpatient appointments and theatre lists.

Bell stressed, however, that patient safety had not been compromised and that this continued to be the top priority.

There's that line again. Perhaps it's part of some hospital administrator JournoList-recommended catchphrase for describing how safety was not compromised during a major workflow disruption?

He said the problems were not down to the software itself, but due to “implementation and data migration difficulties in some clinics."

Right. Quite credible.

“Our information management and technology team, supported by our suppliers BT and Cerner, have been working very hard to sort out any initial issues as quickly as possible and we are already seeing improvements,” he said.

Congratulations are due. They are seeing "improvements" in dangerous clinical IT malfunctions that should never have to have been seen in the first place, if the statement is true.

“Many wards, our two minor injuries units and the Emergency Department, are successfully using the new system." The trust is one of the largest in the South of England, with more than 1,000 beds.

Just give them time.

EHI understands that as part of the Millennium implementation, dummy clinics were set up. Patients were then sent appointment letters for these clinics in error.

EHI also understands that some patients had also not turned up for scheduled operations because they had not been informed about the booking.

Bell apologised to patients who had been “inconvenienced during this transition period” and said staff had shown real dedication to continue to deliver patient care.

What if someone had been inconvenienced into their grave, or ends up there as a result of delays? On what wavelength will the apology be transmitted?

“We firmly believe that the new system, once fully implemented, will improve services for our patients and provide real value,” he said.

That seems to be the mantra, but delivery on such promises are rare. See "Pessimism, Computer Failure, and Information Systems Development in the Public Sector" as here, Public Administration Review 67;5:917-929, Sept/Oct. 2007, Shaun Goldfinch, University of Otago, New Zealand. The article is a cautionary article on IT that should be read by every healthcare executive documenting the widespread nature of IT difficulties and failure, the lack of attention to the issues responsible, and recommending much more critical attitudes towards IT.

A £69m contract for BT to deliver Cerner Millennium to three new, or ‘greenfield’ sites in the South of England was agreed in April 2010, under the auspices of the National Programme for IT in the NHS.

That would not be the failed National Programme for IT in the NHS, the NPfIT what went PfffT, would it?

North Bristol was the last of these three sites to go-live with the system in December last year.

It followed Oxford University Hospitals NHS Trust, which went live a week earlier, and Royal United Hospital Bath NHS Trust, which was the first to go-live in July.

Cerner said it was working closely with North Bristol and BT on the recent implementation of Millennium.

“In complex and large deployments, especially when migrating from two different systems, it is always anticipated that it would take time for the new system to bed-in,” it said in a statement.

The patients are given full informed consent on this issue, right? Right?

“Across much of the trust, the deployment has worked well. However, this is a major change management project and there have been some difficulties with outpatient appointments.

“Although this is not a problem with the software, Cerner is working in partnership with BT and trust staff to resolve any issue as quickly as possible.”

Link: BBC News

Right. Perhaps this software and claim needs testing - in a court of law.

The only thing missing is the word "glitch", though I am including that term in this posting's index, since I consider it another story in the ever-growing health IT "glitch" series.

-- SS

Addendum:

A reader sent me this comment:

How can anyone claim the problems at N. Bristol are unexpected. they are EXACTLY the same problems encountered in Taunton five years ago.

The Somerset Trust had sixteen cancelled go live dates and when Cerner 'Millennium" (note: they never defined which Millennium...) was switched on the whole hospital went into slow-motion.

Appointments could not be made at out-patient reception desks while patients waited and therefore had to be posted on. Twenty-four whole time equivalent clerks had to be employed to manage the back-up of appointment requests. So much for enhanced efficiency and cost savings.

The only possible response to this news is again to remind people of Einstein's famous definition of insanity: "repeating the same thing again and again and expecting a different result."

As for other Trusts, why no news of transformed performance by Cerner's systems at other Cerner implemented sites, Berkshire, Newcastle, Kingston, Oxford etc. The only 'good' news we get is that the system has been switched on.

If any of this expensive activity had really produced data, efficiency or cost gains, we would be drowning in Cerner press releases, the silence can only mean one thing, that their system is performing as poorly at other sites as it has in the South West.

Contrast this with the output and data produced openly by Birmingham University Hospital from its in-house created IT system.

Unfortunately one can only draw one serious conclusion about the whole Cerner/ NHS debacle - to paraphrase Mr. Clinton - "It's the (imho substandard) software, stupid!

This story needs serious investigation ... Recently US news items have started to discount the supposed efficiency gains for e-Health implementations and started to emphasize data capture and patient safety as the imperative for switch on. Unfortunately for Cerner supporters (and other vendors) the US Institute of Medicine's recent report stated unequivocally that there was (to everyone's apparent surprise) no quality evidence that e-Health improved patient safety.

I would contend no drug, therapeutic equipment or operation would or could be implemented in secondary care in the absence of critical and peer reviewed evidence of benefit [emphasis mine - ed.] that has characterized the rush to switch-on substandard IT solutions in English NHS hospitals.

I note that critical, peer-reviewed evidence, especially based on prospective randomized clinical trials as opposed to anecdotal, weak retrospective observational studies, have been deemed unnecessary in health IT.

Yet serious case reports of risk and injury from credible sources are deemed the true "anecdotes" and discounted. As I've written before, the science of medicine is nearly entirely lacking in the domain of health IT.

To put it in the words of James Le Fanu (channeling Sherlock Holmes) in his very apropos essay entitled "The Case of the Missing Data: The Dog That Didn't Bark", details on contrary strands of evidence that could reasonably have been expected to appear in evidential text are absent.

-- SS

IT Malpractice? Yet Another "Glitch" Affecting Thousands of Patients. Of Course, As Always, Patient Care Was "Not Compromised."

At my Nov. 2011 post "Lifespan (Rhode Island): Yet another health IT glitch affecting thousands - that, of course, caused no patient harm that they know of - yet" I wrote:

There's been yet another health IT "glitch" that, of course, caused no patients to be harmed. See other "glitches" here, here, here and at other posts which can be found by searching this blog on the banal term 'glitch'.

Add another case to the health IT glitch file, under the "do we feel lucky today?" patient risk category.
From the Pittsburgh Post-Gazette (I am quoted):

Computer outage at UPMC called 'rare' Systemwide disruption potentially dangerous, expert warns Saturday, December 24, 2011 By Jonathan D. Silver, Pittsburgh Post-Gazette

UPMC's electronic medical records system for inpatients went offline for more than 14 hours at nearly all its hospitals in the region, marking what the health system called a "rare" outage, but one that it claims did not harm patients.

First, as my aforementioned Nov. 2011 post and its contained links point out, these events are not as "rare" as they should be. (The asteroid colliding with Earth that caused the extinction of the dinosaurs - now that's a "rare" event.)

Second, as multiple posts on this blog have pointed out, the claims that "no patients were harmed" is both misleading and irrelevant:

Such claims of 'massive EHR outage benevolence' are misleading, in that medical errors due to electronic outages might not appear for days or weeks after the outage, depending on what information was corrupted/lost/misindentified/or otherwise mishandled after it is 'backloaded' once the system is up. All it takes is one med lost to cause misery and death. (I can speak about that from unfortunate personal experience.)

Claims of 'massive EHR outage benevolence' are also irrelevant in that, even if there was no catastrophe directly coincident with the outage, their was greatly elevated risk. Sooner or later, such outages will maim and kill.

The outage affected a system designed by Cerner Corp., a global electronic records company, and customized by UPMC that doctors and nurses rely on for communication about patient records, medical orders and prescriptions.

It was unavailable from about 8:45 p.m. Thursday to 11 a.m. Friday at almost all of UPMC's hospitals except for Children's and UPMC Hamot in Erie, spokeswoman Wendy Zellner said.

"This is rare. This kind of widespread, extensive downtime would be rare," Ms. Zellner said.

Doctors and nurses continued to have access to patients' electronic records through backup systems, she said. They also had to resort to using old-fashioned paper records for documentation and orders.

"These things happen. They have really well spelled-out procedures for what to do when something goes down," Ms. Zellner said.

She acknowledged that doctors and nurses faced some challenges.

Faced 'some challenges?' In other words, care was compromised by the outage and the 'challenges' were to avoid medical error (and, of course, to make sure billing was unaffected):

Compromised -

a. To expose or make liable to danger, suspicion, or disrepute

b. To reduce in quality, value, or degree; weaken or lower.

Thousands of patients were affected, again reinforcing my point about how IT can and does greatly amplify the risks of paper -- as in my Rhode Island post -- such as errors and confidentiality breaches.

I cannot, for example, think of a single instance where thousands of paper records went unavailable simultaneously (unless, that is, someone lost the key to the Medical Records department), were made available to identity thieves en masse, or where thousands of medical orders were scrambled or truncated in a relatively short period of time as in Rhode Island.

These amplified risks could wipe out any advantages of EHR's over paper in a microsecond.

A partial list of facilities apparently affected in this latest episode of EHR mayhem, from this list:

• Magee-Womens Hospital of UPMC
• UPMC Bedford Memorial
• UPMC East
• UPMC Horizon
• UPMC McKeesport
• UPMC Mercy
• UPMC Montefiore
• UPMC Northwest
• UPMC Passavant
• UPMC Presbyterian
• UPMC Shadyside
• UPMC St. Margaret
• Western Psychiatric Institute and Clinic of UPMC

That accounts for several thousand active patients, I am sure.

(12/28 Addendum: Bed counts of PA hospitals are here. Searching on "University of Pittsburgh Medical Center", it can be seen that thousands of beds were indeed involved.)

"Whenever people aren't working in their native system and workflow I have to believe that is more cumbersome for the clinicians, but these folks are well-trained in what to do when these things happen."

This seems at best an insensitive and perhaps even inhumane bit of P.R. More "cumbersome" for the clinicians? What about the poor patients? How would Ms. Zellner feel, I wonder, if it were her mother, child or significant other on the Operating Room table or having an acute MI when the EHR/CPOE systems went down?

Ms. Zellner said UPMC's public relations staff was unaware of the outage until contacted by a reporter.

It appears P.R. is not very high on the list for receiving information when a crisis arises. I may have known of the outage before they did.

The outage was caused by a "bug" or glitch in software designed by a vendor affiliated with Cerner, Ms. Zellner said. She refused to identify the company.

"We're not trying to point fingers at different vendors. It's a database bug, that's all I can tell you."

(That is, it's not our fault, it's the fault of the database vendor. Hospitals, I regret to inform you - you are responsible for unapproved medical devices used in your facilities, no matter what the source.)

And there's that word "glitch" again, accompanied by the equally banal "bug."

It's just a "bug." Cute little critter!

Me again in the Post-Gazette:

Scot M. Silverstein, a doctor and assistant professor Healthcare Informatics at Drexel University in Philadelphia, disagreed with the use of the terms "bug" and "glitch."

"What occurred here was a disruptive, potentially dangerous major malfunction of a life-critical enterprise medical device," he said.

Somehow, when a clinician makes a mistake, the terms "bug" and "glitch" are never used. In fact, when clinicians fail to meet accepted professional standards of healthcare practice, it is called "malpractice."

I think we can all agree that a major near-full-day outage of an enterprise EHR affecting multiple hospitals and thousands of patients does not meet accepted professionals standard of life-critical computing practice. Yet, all this merits is the word "glitch." It seems to me that if patients are harmed by, in reality, what is (on its face) IT malpractice during such events, not only the clinicians affected should be held liable.

Ms. Zellner said the problem was not a "crash" of the system because there were alternate methods used to cope that prevented patient care from being compromised.

The usual refrain. Let me repeat my definition of "compromised:"

Compromised -

a. To expose or make liable to danger, suspicion, or disrepute

b. To reduce in quality, value, or degree; weaken or lower.

A simple question - if extended EHR outages like this never seem to "compromise" care, then why not eliminate health IT entirely and spend the hundreds of millions saved on patient care?

"This is not a crash of Cerner either," Ms. Zellner said. "I think a crash is, 'Oh my God, the sky is falling,' nobody can get anything."

I leave it to the readers to ascertain the computer expertise levels and reasonableness of what Ms. Zellner thinks a "crash" is.

Technicians from UPMC, Cerner and the third company [the 'mystery' database company? - ed.] worked together on-site to identify and fix the problem. Ms. Zellner said she did not know why it took 14 hours to fix and the underlying cause was still unclear.

"They know what the problem is and I believe it's been fixed, but we really don't know what triggered it," Ms. Zellner said. "I think the next step would be some actual software upgrades."

They "don't know what triggered the 'problem'" - is a proper translation that they have no idea what went wrong?

In fact, regarding another Cerner EHR system which was extensively studied (see "A Study of an Enterprise Information System" at this link), Dr. Jon Patrick came to the conclusion that one of the sources of catastrophic failures is poor software engineering that has made the behavior of the studied system "non-deterministic." Further, software upgrades are not protected from incremental changes made by maintenance and customization staff, and may introduce even more instability.

A software upgrade without clearly understanding "what triggered the problem" is simply asking for more trouble. (My bet, however, is that they attempt it anyway.)

A Cerner representative could not be reached for comment.

What's to say?

How about this:

Dr. Silverstein said based on what he was told about the computer outage, it means that hospital medical staff would have been unable to update patient charts and probably would not have been able to issue any orders through the system during the time it was off line.

He also questioned how up-to-date the hospital's redundant records were.

Repeating UMPC's statement from the article that appeared after I gave my quotes to the reporters: "Doctors and nurses continued to have access to patients' electronic records through backup systems, [the UPMC spokesperson] said. They also had to resort to using old-fashioned paper records for documentation and orders."

My stated fears of disruption and increased risk due to compromised care seem well-grounded.

In May, Allegheny General Hospital had to shut its electronic medical records computer system down because of problems with the vendor's hardware.

The hospital used backup procedures to continue care for patients, including using paper orders and record-keeping.

Wait ... I thought I'd heard these events were "rare." Two in the same city within six months?

---------------------------

Truth be told:

The primary rule in computing is:

Either you are in control of your information systems, or they are in control of you.

Clearly the latter was the case here.

The following questions arise:

• Was the software containing the "bug" properly vetted before being used on live patients? This is not just the vendor's obligation.
  
• If it was not vetted properly, why not?
• Was it an "upgrade" or patch? (If so, the same vetting rules apply.)

Further, the soft-selling of these incidents must end. The use of terms such as "bug" and "glitch" must also end. What occurred here, echoing my newspaper quote, was a disruptive, potentially catastrophic major malfunction of a life-critical enterprise medical device.

System-wide EHR crashes are not merely ‘glitches’ or ‘bugs.’ They need to be considered, as in medicine itself, as 'never events.' From AHRQ:

The term "Never Event" was first introduced in 2001 by Ken Kizer, MD, former CEO of the National Quality Forum (NQF), in reference to particularly shocking medical errors (such as wrong-site surgery) that should never occur. Over time, the list has been expanded to signify adverse events that are unambiguous (clearly identifiable and measurable), serious (resulting in death or significant disability), and usually preventable.

Further, re: "patient care was never compromised." How do they know that? In fact, this is 'spin' and word games on its face. By definition, if CPOE and chart updating was unavailable, patient care was compromised, where "compromised" means "increased levels of risk for error were created, requiring workarounds."

Further, as mentioned earlier, harms might not show up for some time. Lost orders, corrupted data, errors of omission or commission transcribing backup paper records into the computer ("backloading"), etc. can take their toll later. Post-outage vigilance is essential, putting even more stress on clinicians that increases likelihood of further error and that they certainly do not need. Clinicians are stressed enough already.

Finally:

IT personnel have not only deliberately inserted themselves into clinical affairs (e.g, via the HITECH Act of 2009), they have also done so with a stunning arrogance and unproven braggadocio about their systems "revolutionizing" medicine (whatever that means).

Indeed, they need to accept the medical responsibility and obligations this territorial intrusion entails.

On its face, this massive outage was the result of issues that did not meet accepted professional standards of IT practice for life-critical environments. Res ipsa loquitur.

Something was not vetted properly, there was a lack of redundancy, the IT personnel were NOT in control of their systems.

Just as when physicians don't provide care that meets accepted professional standards of healthcare, this incident and others like it are, by definition, a result of IT malpractice.

If patients are harmed, IT personnel and their management (often non-IT C-level officers) involved in this system need to be held accountable.

If they can't take the clinical heat (as clinicians do daily since the time they enter medical or nursing school), then they need to get out of the clinical kitchen.

-- SS

Note: see this take on these matters at the HIStalk blog:

UPMC’s Cerner systems go down for 14 hours at most campuses last Thursday and Friday, forcing them to go back to paper. The PR person blamed “a database bug,” which makes the above Oracle press release from this past summer a particularly fun read. Cerner and UPMC have an atypical vendor-customer relationship since they’ve invested big money together in innovation projects and UPMC runs a Cerner implementation business overseas.

Now we know who the unnamed "mystery database vendor" is...

-- SS

Dec. 29, 2011 Addendum:

Was UPMC acting as a "proving ground" for some Oracle-Cerner-UPMC experimental health IT technology that resulted in the crash? The claim of being an IT "proving ground" has been made in the past:

Pittsburgh Tribune
May 2, 2006
UPMC partners with technology provider

The University of Pittsburgh Medical Center is taking another step in a quest to commercialize new medical technology.

UPMC on Monday signed a three-year deal with health care information technology provider Cerner Corp. to develop and market medicine-related technological advances. Both parties will contribute $10 million in cash, services and intellectual property to the effort.

The deal is a smaller version of an April 2005 deal between UPMC and information technology behemoth IBM.

As is the case in the IBM deal, UPMC will serve as a built-in proving ground for jointly developed technologies and products, with Cerner marketing the products and UPMC awarded a share of profits.

As I wrote at "Proving Ground for IT Tests On Children: Pioneers in Health IT, or Pioneers in Ignoring the Past?":

"A hospital and patients are not a learning lab for HIT vendors. The appropriate "proving ground" for new medical technology is the controlled clinical trial where participants (in this case, patients and healthcare professionals alike) have freedom of choice whether or not to participate, and a chance to give (or deny) consent after being fully informed of potential risk."

This is a fundamental human rights issue.

-- SS

The EHR Mission Hostile User Experience, Part 10: Cerner Powerchart - via FDA's MAUDE Database

"You should not have to work around something that is not in the way" - SS
Some time ago, I'd written an eight part (later expanded to nine part) series on health IT mission hostile user interfaces and experiences.

(Note: Part 1 of this series is here, part 2 is here, part 3 is here, part 4 is here, part 5 is here, part 6 is here, part 7 is here, and part 8 is here. April 2011 addendum: a post that can be considered part 9 is here.)

Here is an addendum, part 10, from the FDA Maude (Manufacturer and User Facility Device Experience) Database. I do not know who wrote it or where it was from. It was received by FDA as indicated on 3 April 2011:

FDA MAUDE REPORT

CERNER POWERCHART

Event Type Other

Event Description

I am writing to report a problem in the design of the cerner powerchart computer provider order entry (cpoe) product. As i will describe below, these design flaws are largely responsible for approximately 100 medication errors per day in our 240 bed hosp. The (b)(6) hosp in (b)(6) is a 240 bed teaching hosp. We are owned by the (b)(6). We have to be one of the only hospitals in the country to fully implement cpoe with two entirely different electronic medical records (emr).

In 2007, we completed a comprehensive cpoe implementation using idx 3. 1 (which was later purchased by general electric). Our self-reported medication error rate was <2 error reports per day. In (b)(6) of 2010, we transitioned our emr to cerner millennium - powerchart (b)(4). This implementation was done as a first step to converge onto a single electronic medical record across the (b)(6) enterprise.

Despite extensive nursing informatics support (approx. 7000 hrs per month for the first six months after implementation), our error rate went from approx. 176 medication errors per day one month after implementation to approx. 100 medication errors per day currently. It has been stable at this level for at least the past 3 months. Many of these errors involve high risk medications (e. G. Heparin, morphine). In order to understand the etiology of the errors, i need to explain how cerner processes cpoe orders.

With cerner, orders are grouped into plans ("powerplans"). These plans can include sub-plans ("phases") and sub-sub-plans ("sub phases"). A typical orders display screen appears as follows: (b)(4). Under the plans is an order tab which displays other orders.

This design allows two distinct sources of error.

First, there is no consistent way to view orders for medications. A medication order can display either in a plan, sub-plan, or under the orders tab. In order to find a given medication, cerner mandates that you click through each and every plan. This facilitates duplication of both medication and non-medication orders (there is a medication checker which is so poorly designed and does little to aid the pharmacists in detecting duplicate medications).

Second and more dangerous, high risk intravenous medications can be run either inside or outside of the plan. Cerner programming [i.e., the user interface design -ed.] does not keep high risk medications in a consistent spot. It is very easy to stop monitoring for heparin (which is included in the heparin plan) while continuing the heparin (which is outside the plan) or vice-versa.

As you can probably guess, we formed multiple interdisciplinary teams to address these deficiencies. [Workarounds - ed.]

I was the lead physician on the orders team. Although we were unable to reduce the error rate, it is not close to an acceptable level. Again, i believe this is because of the design of the cerner product which is why i chose to bring this to your attention. With issues this complex, i am a firm believer that "seeing is believing. " if desired, i would be happy to arrange a web conference to demonstrate my concerns.

Brand Name CERNER POWERCHART COMPUTER
Type of Device NONE
Manufacturer (Section F) CERNER
Manufacturer (Section D) CERNER
Device Event Key 2081342
MDR Report Key 2045867
Event Key 1942844
Report NumberMW5020161
Device Sequence Number 1
Product Code LNX
Report Source Voluntary
Reporter Occupation Other
Type of Report Initial
1 Device Was Involved in the Event
1 Patient Was Involved in the Event
Date FDA Received 04/03/2011
Is This An Adverse Event Report? No
Is This A Product Problem Report? No
Device Operator Service Personnel
Was The Report Sent To Manufacturer? No
Is the Device an Implant? No
Is this an Explanted Device?
Page Last Updated: 11/30/2011

I will let the FDA report speak for itself, with only one question:

When will the designers, developers, purchasers and implementers of medical devices like this start to be held criminally liable for patient injuries that occur from the risk these devices pose to patient safety?

criminal negligence - (law) recklessly acting without reasonable caution and putting another person at risk of injury or death (or failing to do something with the same consequences)

It's not as if the issues related to good user interface design are a mystery, nor have those issues been a mystery for at least a few decades.

An EHR design as described, if accurate, while perhaps "nifty" in some way from the computer-techie perspective, would require significant recklessness to design and to actually implement in a life-critical setting.

Those who try to point out these issues internally are sometimes subject to retaliation (for not being a "team player", of course, which in today's parlance means someone who is silent, or silenced, or a co-conspirator regarding managerial mediocrity, malfeasance, or madness). An example is here. We at Healthcare Renewal simply refuse membership on that team.

(Did I mention this deterministic miracle-making technology is slated via the HITECH Act for rollout in the U.S., unless the medical professional or organization is willing to accept progressive cuts to their Medicare reimbursement?)

-- SS