Vermont: Despite $70 million investment, health IT systems a long way from prime time - "Problems are appropriate"

Preliminary note:  This post is rich with hyperlinks.  At minimum I recommend opening them in another window and at least scanning their contents - SS

No surprises in this article, including an amorality alien to the healing professions, but common in technology circles:

Despite $70 million investment, health IT systems a long way from prime time
VTDigger.org
Andrew Nemethy
July 18, 2012

The state’s efforts to digitize the world of health information, a costly multi-year endeavor that is approaching a $70 million pricetag, got a lousy diagnosis Tuesday.

Instead of creating cost efficiency and improving payment flow to doctors and treatment for patients, it’s creating stress and a lot of headaches for physicians, according to both lawmakers and state officials overseeing the effort. [It's also creating increased risk for patients, a factor - the most crucial factor - rarely mentioned in articles such as this - ed.]

Money, of course, grows on trees, and physicians, hospitals and government have nothing better to do with $70 million than conduct experiments on patients with alpha and beta software ...

There's the usual excuses from the usual actors:

But Health Information Technology (HIT) coordinator Hunt Blair said that’s to be expected considering the difficulty of the “incredibly challenging” task of getting such disparate groups as doctors, hospitals, other health care providers, insurance companies, the state and federal government on the same digital page.

Let alone (per Social Informatics) the organizational and sociological challenges of implementing any new information or communications technology (ICT), that's somewhat akin to saying it's hard to get people to consume arsenic as an aphrodisiac.  Not mentioned is the deplorable state of health IT in terms of quality, safety, usability, unregulated nature etc.

“We’re talking about an extremely complex undertaking and I think it’s important to recognize the state of Vermont was way out in front,” Blair said.

“We’re on the bleeding edge,” he told a legislative Health Care Oversight Committee Tuesday at the Statehouse.

That prompted Sen. Claire Ayer, D-Addison, the panel’s chairwoman, to ask him to clarify if he meant “leading.”

He stuck with “bleeding.”

"Bleeding edge?"

Aside from the very poor choice of terms, this attitude is the polar opposite of the culture of "first, do no harm."  It is not a clinician's attitude.  It is an attitude of someone who seems to forget that patients are at the receiving end of the "bleeding edge" (which usually implies a rocky course) technology:

Bleeding edge technology is a category of technologies incorporating those so new that they could have a high risk of being unreliable and lead adopters to incur greater expense in order to make use of them.  The term bleeding edge was formed as an allusion to the similar terms "leading edge" and "cutting edge". It tends to imply even greater advancement, albeit at an increased risk of "metaphorically cutting until bleeding" because of the unreliability of the software or other technology.The phrase was originally coined in an article entitled "Rumors of the Future and the Digital Circus" by Jack Dale, published in Editor & Publisher Magazine, February 12, 1994.

Wonderful.

Considering the risks to patients, this claim brings to mind the definition I posted of the health IT Ddulite ("Luddite", the common canard against cautious doctors, with the first four letters reversed):

Ddulite:  Hyper-enthusiastic technophiles who either deliberately ignore or are blinded to technology's downsides, ethical issues, and repeated local and mass failures.

On the other hand, did critic VT Sen. Kevin Mullin read my Drexel website on health IT failure and mismanagement?

That doesn’t surprise Sen. Kevin Mullin, R-Rutland, who had tough questions about the state’s effort to oversee and promote use of electronic medical health records and a statewide health information exchange.

“I hear genuine frustration from providers who are spending time and resources trying to modernize and make their offices more efficient, and prepare for the future, and yet every one of them feels like they’ve been burned,” he said.

“Basically we’re not getting any results for these millions and millions of dollars that have been pumped into IT (information technology),” he said after the meeting.

“We should be a lot further along,” he said. “I just don’t think the leadership’s in place.”

He's on the right track, but I'm not sure Vermont (or any state government) really understands what levels of leadership are truly needed, e.g., as outlined by ONC a few years ago here.

More excuses:

... Mark Larson, commissioner of the Vermont Health Access Department and a former House representative from Burlington, oversees management of Vermont’s publicly funded health insurance programs and the effort on digital medical records and a new medical information exchange.

... Larson told lawmakers he hears the same message they do, that there’s “a lot of confusion in the field.” He said that is an inevitable part of the complex process.

“These are not systems where you just plug that in and they work perfectly on day one,” he said. “Problems are appropriate along the path to get where we want." [I note this is an implicit admission that experimentation is being performed - ed.]


“We just have to work through that,” he said.

"Problems are appropriate?"  ... Really?  In a mission critical field such as medicine?  That's a maddeningly reckless and cavalier ideology, to put it mildly.   In what other safety-critical domain would such a happy-go-lucky attitude that "problems are appropriate", outside of the laboratory, be tolerated? 

"We just have to work through that?" ... "Just work through that?"  Really?

That should be really easy,  just like the failed £12.7bn (~ US $20 billion) National Programme for IT (NPfIT) in the NHS, a program that went "Pffft" last year.

... Based on testimony Tuesday, the issues that medical practitioners and the industry face in digitizing information are familiar ones for anyone who deals with technology: Software that is problematic, digital files that don’t translate and can’t be read by other systems, lost time spent on technological issues that detract from what doctors are paid to do, which is treat their patients. [And create increased risk that leads to maimed or dead patients - ed.]

"Familiar to anyone who deals with technology?"  As in, say, mercantile/management computing that runs Walmart's stock inventory system, or the Post Office?  Is that an appeal to common practice of some type?  This brings to life my observation that there is an utter lack of recognition (either due to ignorance, or opportunism) that HIT systems are not business management systems that happen to be used by clinicians, they are virtual clinical tools (with all that implies) that happen to reside on computers.

How utterly amoral and alien to the ethics of medicine these attitudes are.  No regulation of the technology is in place.  No systematic postmarket surveillance of patient harm or death is conducted. 

Further, the literature on health IT is not entirely as optimistic and tolerant as the VT government.

Is the VT government aware of that literature?

Just a few specific examples - the National Research Council wrote that  "Current Approaches to U.S. Healthcare Information Technology are Insufficient", do not support clinician cognitive processes (then what, exactly, is health IT supposed to do?) and may result in harm. 

The ECRI Institute indicates health IT systems are among the top ten technology risks in healthcare.

Reports from the Joint Commission, FDA and AHRQ indicate that risk is known, but magnitude of risk unknown ("tip of the iceberg" per FDA).

A report from NIST indicates that usability is lacking and promotes "use error" (technology-promoted error, as opposed to "user error").

A report from IOM on safety of health IT states that:

... While some studies suggest improvements in patient safety can be made, others have found no effect. Instances of health IT–associated harm have been reported. However, little published evidence could be found quantifying the magnitude of the risk.

Several reasons health IT–related safety data are lacking include the absence of measures and a central repository (or linkages among decentralized repositories) to collect, analyze, and act on information related to safety of this technology. Another impediment to gathering safety data is contractual barriers (e.g., nondisclosure, confidentiality clauses) that can prevent users from sharing information about health IT–related adverse events. These barriers limit users’ abilities to share knowledge of risk-prone user interfaces, for instance through screenshots and descriptions of potentially unsafe processes. In addition, some vendors include language in their sales contracts and escape responsibility for errors or defects in their software (i.e., “hold harmless clauses”).

The committee believes these types of contractual restrictions limit transparency, which significantly contributes to the gaps in knowledge of health IT–related patient safety risks. These barriers to generating evidence pose unacceptable risks to safety.

(Institute of Medicine, 2012. Health IT and Patient Safety: Building Safer Systems for Better Care. PDF.  Washington, DC: The National Academies Press, pg. S-2.)

Human subjects protections?  Not needed here ... it's just one big, unconsented, happy $70 million medical experiment.

What is not mentioned, and either deliberately ignored or lost on these politicians, is the effects of this turmoil on patient care in terms of risk and adverse outcomes, and that being on the "bleeding edge" in this experimental technology is not desirable.  

Slow, skeptical and cautious approaches are essential.  Physician education on health IT risks and patient informed consent might be nice, too.

It seems mass social re-engineering experiments are fine, even if the consequences include wasted resources the healthcare delivery system can ill afford, physicians being distracted by major defects, and outcomes like baby deaths, adults suffocating at the bedside, and graveyards.

-- SS

University of Arizona Medical Center, $10 million in the red in operations, to spend $100M on new EHR system

In my Oct. 2006 post "$70 million for an Electronic Medical Records system?" I wrote:

... healthcare doesn’t have the capital for clinical IT misadventures, and I believe when the issues become more public in this industry sector and information flows about mismanagement and abuses (as is happening in the UK ’s Connecting for Health project) [now abandoned as described here - ed.], the fallout won’t be pretty.

Here's an example of an organization in profound ardent technophile-driven Ddulite mode:

UA Medical Center to spend $100M on new records system

Tucson's largest health-care organization expects to spend upward of $100 million on getting its two hospitals talking to each other.

Right now, the inpatient medical record systems at the University of Arizona Medical Center's two campuses aren't speaking to each other.

The lack of communication is resulting in more work for healthcare providers in the University of Arizona Health Network.

[How much more work, exactly, and how much would a non-cybernetic solution cost? These issues seem never to be mentioned - ed.]

The $1.2 billion, nonprofit company employs nearly 7,000 people.

The network is installing a new, uniform electronic medical records system for all patients at its two hospitals - UA Medical Center - University Campus and UA Medical Center - South Campus - and at outpatient centers as well.

... Project leaders predict it will result in a more efficient organization with fewer medication errors and better patient care.

... The new system's benefits will certainly trickle down to patients, said Clint Hinman, an experienced pharmacy director within the network who is directing the computer upgrade program.

[Note once again the absolutist statements of deterministic benefit and beneficence, based on scant supportive evidence and increasing contradictory evidence, that I bolded above - ed.]

I note that $100 million+ is probably enough to pay for AN ENTIRE NEW HOSPITAL or hospital wing ... or a lot of human medical records professionals.

Executives and project leaders have probably never read any of the literature at the reading list here or at my academic site here, or if they have, choose to be blind to it and trusting of literature such as ONC's sloppy-science "should not have been published in its present form" health IT cheerleading here.

Most important of all:

It's not advisable to gamble with $100 million in that fashion, especially under these conditions:

... BUDGET ISSUES

Spending $100-million-plus on electronic medical records is a lot of money for a network that as of mid-January was at $10 million in the red in operations, BUT network spokeswoman Katie Riley said the electronic medical records are not to blame.

[Capitalization and emphasis of the "but" mine - ed.]

This statement is both representative of a healthcare system gone overboard - you don't spend on luxuries when you're $10 million in the red - and is a non-sequitur.

Who cares if the EHR's are not to blame for the system being $10 million in the red? That does not seem like a good reason to go ahead and spend $100 million (which will probably balloon to several times that figure, hence I will use $100 million++) on a very risky gamble.

Further, all it will take is a few of these mishaps to put the system further in the red.

I should also ask: will medical and other staff be laid off to afford the new systems, in effect trading people for computers?

To spend $100 million++ on HIT when you're already $10 million in the red on operations is, in my view, financially reckless.

-- SS

EHRs and test ordering: Health Affairs authors reply to ONC

At my March 9, 2012 post "Increased Lab Ordering with EHR's?" I refuted ONC's response to a Harvard-based research study "Giving Office-Based Physicians Electronic Access To Patients’ Prior Imaging And Lab Results Did Not Deter Ordering Of Tests" that may contradict the notion that HIT reduces medical costs.

Now the authors themselves have responded to ONC. There are many similar themes in their response. I've bolded them below:

The Effect Of Physicians’ Electronic Access To Tests: A Response To Farzad Mostashari

March 12th, 2012
by Danny McCormick, David Bor, Stephanie Woolhandler, and David Himmelstein

Our recent Health Affairs article linking increased test ordering to electronic access to results has elicited heated responses, including a blog post by Farzad Mostashari, National Coordinator for Health IT. Some of the assertions in his blog post are mistaken. Some take us to task for claims we never made, or for studying only some of the myriad issues relevant to medical computing. And many reflect wishful thinking regarding health IT; an acceptance of deeply flawed evidence of its benefit, and skepticism about solid data that leads to unwelcome conclusions.

Dr. Mostashari’s critique of our paper, will, we hope, open a fruitful dialogue. We trust that in the interest of fairness he will direct readers to our response on his agency’s site. [If not, HC Renewal posts get highly ranked by Google - ed.]

Our study analyzed government survey data on a nationally representative sample of 28,741 patient visits to 1187 office-based physicians. We found that electronic access to computerized imaging results (either the report or the actual image) was associated with a 40% -70% increase in imaging tests, including sharp increases in expensive tests like MRIs and CT scans; the findings for blood tests were similar. Although the survey did not collect data on payments for the tests, it’s hard to imagine how a 40% to 70% increase in testing could fail to increase imaging costs.

Dr. Mostashari’s statement that “reducing test orders is not the way that health IT is meant to reduce costs” is surprising, and contradicts statements by his predecessor as National Coordinator that electronic access to a previous CT scan helped him to avoid ordering a duplicate and “saved a whole bunch of money.” A Rand study, widely cited by health IT advocates including President Obama, estimated that health IT would save $6.6 billion annually on outpatient imaging and lab testing. Another frequently quoted estimate of HIT-based savings projected annual cost reductions of $8.3 billion on imaging and $8.1 billion on lab testing.

We focused on electronic access to results because the common understanding of how health IT might decrease test ordering is that it would facilitate retrieval of previous results, avoiding duplicate tests. Indeed, it’s clear from the extensive press coverage that our study was seen as contravening this “conventional wisdom”.

Nonetheless, Dr. Mostashari criticizes us for analyzing the impact of physicians’ electronic access to imaging and test results, but not other aspects of electronic health record (EHR) use. We did, however, analyze the relationship of EHRs to test ordering in a subsidiary analysis. While physicians use of a full EHR was associated with a 19% increase in image ordering, as we noted in the paper this finding was not statistically significant. While we cautiously (and properly) interpreted this as a “null” finding, these data are inconsistent with Mostashari’s optimistic view that use of a full EHR reduces costs.

He asserts that our 2008 data are passe, and that health IT meeting today’s “meaningful use” criteria definitely saves money. The data we analyzed were the latest available data when we initiated the study. While the proportion of outpatient physicians utilizing health IT has grown since 2008, we are unaware of any “game changing” health IT developments in the past four years that are would produce substantially different results if the study were repeated today. The EHR vendors that dominated the market in 2008 remain, by and large, today’s market leaders, and their products have undergone mostly modest tweaks. Mostashari’s contention that 2012 EHRs – incorporating decision support and electronic information exchange – save money in ways not possible in 2008 should be tested through additional research but remains merely a hypothesis. We hope that some day his predicted savings can be achieved.

Dr. Mostashari offers his own explanation for our findings, suggesting that doctors who are inclined to order more tests are also inclined to purchase health IT for viewing test results electronically rather than on paper. He offers no evidence for this assertion and ignores the fact that we explored (and rejected) this explanation by analyzing subgroups of doctors who are unlikely to be the decision maker for IT purchases – e.g. employed physicians, those working in an HMO setting etc. In other words, electronic access to results predicted more test ordering whether or not the ordering physician was responsible for health IT purchases.

He incorrectly states that our analysis did not take into account patients’ severity of illness, physicians’ level of training, and the nature of physicians’ financial arrangements. In fact, we reported subsidiary multivariate analyses that included several serious diagnoses; all of our models included physician specialty (which we specified in several different ways); and all models included adjustment for an extensive list of indicators of financial arrangements (e.g. whether the physician owned the practice or was an employee; the type of office; whether the practice was owned by a hospital; whether the physician was a solo practitioner; whether the physician’s compensation was based, in part or whole on “profiling”; and whether the practice was predominantly prepaid). We also performed a series of subsidiary analyses that explored whether physicians with a proclivity to “self refer” patients for imaging tests accounted for our finding; they didn’t.

Dr. Mostashari criticizes us for failing to assess whether health IT improved the quality or appropriateness of care. Of course, these were not the topic of our research. Those are different studies for a different time. However, we would note that other large-scale studies have found no, or trivial quality improvements associated with HIT outside of a few flagship institutions^4-6.

Dr. Mostashari’s strongest claim is that observational studies like ours (and most other health policy studies, including some by Dr. Mostashari himself) cannot prove causation. This is surely true. As long time teachers of evidence based medicine we took care to couch our conclusions in cautious terms, stating only that “Computerization, whatever its other benefits, remains unproven as a cost control strategy.”

But Dr. Motashari is less cautious, asserting that the case for HIT is closed. The paper he cites to buttress this claim (authored by members of his own agency) culled studies reporting any impact of HIT on virtually any aspect of care, and accepted authors’ claims of benefit without regard to study quality or statistical niceties. Thus, a focus group’s impressions of benefit are accorded the same weight as nationwide studies of Medicare data showing virtually no impact of computerization on quality measures. Reports of a reduction from 70% to 38 % in “missed billing opportunities” or a $7,000 reduction in office supply costs are among the 92% of studies judged “positive”. While the literature review he cites is interesting, nothing in it contradicts our findings.

Dr. Mostashari is also correct in reiterating that randomized trials are the best way to assess health IT. In fact, no randomized trial has ever been published that examines patients’ outcomes or costs associated with off-the-shelf health IT systems that dominate the U.S. market. No drug or new medical device could pass FDA review based on such thin evidence as we have on health IT. Yet his agency is disbursing $19 billion in federal funds to stimulate the adoption of this inadequately evaluated technology. Dr. Mostashari is perhaps the only person in our nation who commands the resources needed to mount a well done randomized controlled trial to fairly assess the impact of health IT, and the comparative efficacy of the various EHR options.

Finally, Dr. Mostashari’s unbridled faith in technology is mirrored by his belief that ACOs are the next panacea for health costs and quality. That health policy flavor-of-the-month also remains wholly unproven.

I think readers comparing my response to the authors', and who are familiar with the many posts at HC Renewal going back to 2004, will recognize the themes I have bolded in the author's response to HHS.

-- SS

Another Health IT Mythbuster: Doctors order more X-rays, not fewer, with computer access

This piece is in the Washington Post about a just-released piece in Health Affairs (at this link) by Danny McCormick, David H. Bor, Stephanie Woolhandler and David U. Himmelstein of Harvard Medical School.

(Woolhandler and Himmelstein have written on such topics before, such as in the Nov. 2009 article “Hospital Computing and the Costs and Quality of Care: A National Study”, Amer J Med 123:1; 40-46.)

In the abstract of the new article, the researchers note:

Giving Office-Based Physicians Electronic Access To Patients’ Prior Imaging And Lab Results Did Not Deter Ordering Of Tests

Health Aff March 2012 vol. 31 no. 3 488-496

Abstract

Policy-based incentives for health care providers to adopt health information technology are predicated on the assumption that, among other things, electronic access to patient test results and medical records will reduce diagnostic testing and save money.

To test the generalizability of findings that support this assumption, we analyzed the records of 28,741 patient visits to a nationally representative sample of 1,187 office-based physicians in 2008. Physicians’ access to computerized imaging results (sometimes, but not necessarily, through an electronic health record) was associated with a 40–70 percent greater likelihood of an imaging test being ordered. The electronic availability of lab test results was also associated with ordering of additional blood tests.

The availability of an electronic health record in itself had no apparent impact on ordering; the electronic access to test results appears to have been the key. These findings raise the possibility that, as currently implemented, electronic access does not decrease test ordering in the office setting and may even increase it, possibly because of system features that are enticements to ordering. We conclude that use of these health information technologies, whatever their other benefits, remains unproven as an effective cost-control strategy with respect to reducing the ordering of unnecessary tests.

The WaPo offered these observations in their article:

Doctors order more X-rays, not fewer, with computer access

By Lena H. Sun, Published: March 5

In the debate over the high cost of health care, federal policymakers have always claimed that one way to cut costs is for doctors to use electronic medical records and other information technology. Doing so, they say, avoids duplication and saves money.

But new research suggests that may not be the case.

Doctors who have easy computer access to results of X-rays, CT scans and MRIs are 40 to 70 percent more likely to order those kinds of tests than doctors without electronic access, according to a study to be published in the March issue of the journal Health Affairs.

This is not surprising to me. Click a button, order an test, click a button, get the results. What could be easier?

What the pundits failed (and continually fail) to take into account is the social context of health IT. It is not used on a robotic assembly line. There are incentives, both proper and perverse (e.g., "defensive medicine", profits, patient demands) that figure into how any IT will "play out" in healthcare.

Making the blanket statement that "IT in healthcare will save money" without solid, sustained evidence (as opposed to the anecdotal evidence that now exists), while simultaneously ignoring contrary evidence is simply promoting an industry-created meme, a.k.a. wishful thinking.

“On average, this is comparing doctors who had electronic medical records and those who didn’t,” said lead author Danny McCormick, a physician and assistant professor of medicine at Harvard Medical School.

Researchers say the findings challenge a key premise of the nation’s multibillion-dollar effort to promote the widespread adoption of health information technology.

“This should give pause to those making the argument,” McCormick said. Instead of saving money, that effort could drive costs higher, he said.

For the study, researchers at the Cambridge Health Alliance, a health system in Cambridge, Mass., and the City University of New York analyzed data from a 2008 federal government survey. The data included information collected from 28,741 patient visits to 1,187 office-based doctors. The information included the type of doctor, their office computerization and the tests ordered at each visit. About half of the doctors’ offices surveyed had computerized access to results of X-rays and other imaging tests.

Researchers found that doctors who did not have computerized access ordered imaging tests in 12.9 percent of visits, while doctors with electronic access ordered imaging in 18 percent of visits, a 40 percent greater likelihood. Doctors with computerized access were even more likely — about 70 percent more likely — to order advanced imaging tests, such as PET scans, which experts said are most commonly used to detect cancer, heart problems, brain disorders and other central nervous system disorders.

The study found the results hold true even after taking into account other factors, such as patient demographics, doctor specialty and physician self-referral.

Unfortunately, nothing up to and including HIT-related patient deaths will give pause to health IT pundits pushing national rollout. They have too much skin in the game in the health IT bubble, psychologically (link) and financially.

The pundits need to be opposed by medical and legal professionals in an organized, systematic fashion until such time as this industry follows the National Academies' direct recommendations on getting health IT "right."

... Researchers were not able to determine why physicians ordered the imaging tests, or whether in those cases, physicians had looked at patients’ prior chest X-rays. Nor were they able to assess whether the increased imaging helped or harmed patients.

That needs to be followed up upon, obviously. Will it? Probably not.

At some flagship hospitals with customized health information technology systems, for example, doctors get specific feedback about diagnostic tests that can result in ordering fewer tests.

Note the word "can." (After decades there is no rigorous, unopposed proof, yet we are embarking on a national rollout with penalties for non-adopters?) This rationalization is both the "it will be better in version 2.0" and wishful thinking combined.

Michael Furukawa, a health economist in the Office of the National Coordinator for Health Information Technology, the administration’s health IT czar, said the researchers’ focus was not deep enough to support the study’s conclusions.

“The data are sound, the methods are appropriate, but the focus is limited,” he said. “They only looked at one piece of health IT.”

"But the focus is limited?"

So what?

That was all they intended to look at, and the result was concerning. Moving the goalposts is not a valid refutation of research.

On the other hand, coming from a political office that clearly does not understand how to conduct qualitative research and creates political promotion pieces masquerading as "research", such a statement is not surprising. See "ONC: "The Benefits Of Health Information Technology: A Review Of The Recent Literature Shows Predominantly Positive Results" at this link, where essential research methodologies were thrown under the bus for publication in Health Affairs.

At least the deviations from rigorous research methodologies were admitted:

“... Our findings must be qualified by two important limitations: the question of publication bias [e.g., bias in evidence selection - ed.], and the fact that we implicitly gave equal weight to all studies regardless of study design or sample size.”

Unfortunately, the media, politicians, financial decisionmakers and others are likely not to really comprehend, in-depth, the full significance of that sentence.

I can only imagine the reaction to such an excuse made by pharma for the mainstreaming of a risk-prone drug.

I personally take with the greatest of skepticism anything coming from ONC, an increasingly politicized government office in my observation.

You should, too, in my opinion.

-- SS

A Critical Review of a Critical Review of e-Prescribing ... Or Is It CPOE?

In PLoS medicine, the following article was recently published by researchers at the University of New South Wales in Australia:

Westbrook JI, Reckmann M, Li L, Runciman WB, Burke R, et al. (2012) Effects of Two Commercial Electronic Prescribing Systems on Prescribing Error Rates in Hospital In-Patients: A Before and After Study. PLoS Med 9(1): e1001164. doi:10.1371/journal.pmed.1001164

The section I find most interesting is this:

We conducted a before and after study involving medication chart audit of 3,291 admissions (1,923 at baseline and 1,368 post e-prescribing system) at two Australian teaching hospitals. In Hospital A, the Cerner Millennium e-prescribing system was implemented on one ward, and three wards, which did not receive the e-prescribing system, acted as controls. In Hospital B, the iSoft MedChart system was implemented on two wards and we compared before and after error rates. Procedural (e.g., unclear and incomplete prescribing orders) and clinical (e.g., wrong dose, wrong drug) errors were identified. Prescribing error rates per admission and per 100 patient days; rates of serious errors (5-point severity scale, those ≥3 were categorised as serious) by hospital and study period; and rates and categories of postintervention “system-related” errors (where system functionality or design contributed to the error) were calculated.

Here is my major issue:

Unless I am misreading, this research took place in hospitals (i.e., "wards" in hospitals) and does not seem to focus (if even refer to) discharge prescriptions.

I think it would be reasonable to say that what are referred to as "e-Prescribing" systems are systems used at discharge, or in outpatient clinic/offices to communicate with a pharmacy selling commercially and not involved in inpatient care.

From the U.S. Centers for Medicare and Medicaid Services (CMS), for example:

E-Prescribing - a prescriber's ability to electronically send an accurate, error-free and understandable prescription [theoretically, that is - ed.] directly to a pharmacy from the point-of-care

I therefore think the terminology used in the article as to the type of system studied is not well chosen. I believe it could mislead readers not experienced with the various 'species' of health IT.

This study appears to be of an inpatient Computerized Practitioner Order Entry (CPOE) system, not e-Prescribing.

Terminology matters. For example, in the U.S. the HHS term "certification" is misleading purchasers about the quality, safety and efficacy of health IT. HIT certification as it exists today (granted via ONC-Authorized Testing and Certification Bodies) is merely a features-and-functionality "certification of presence." It is not like an Underwriter Labs (UL) safety certification of an electrical appliance that the appliance will not electrocute you.

(This is not to mention the irony that one major aspect of Medical Informatics research is to remove ambiguity from medical terminology, e.g., via the decades-old Unified Medical Language System project or UMLS. However, as I've often written, the HIT domain lacks the rigor of medical science itself.)

I note that if this were a grant proposal for studying e-Prescribing, I would return it with a low ranking and a reviewer comment that the study proposed is actually of CPOE.

That said, looking at the nature of this study:

The conclusion of this paper was as follows. I am omitting some of the actual numbers such as confidence intervals for clarity; see the full article available freely at above link for that data:

Use of an e-prescribing system was associated with a statistically significant reduction in error rates in all three intervention wards. The use of the system resulted in a decline in errors at Hospital A from 6.25 per admission to 2.12 and at Hospital B from 3.62 to 1.46. This decrease was driven by a large reduction in unclear, illegal, and incomplete orders. The Hospital A control wards experienced no significant change. There was limited change in clinical error rates, but serious errors decreased by 44% across the intervention wards compared to the control wards.

Both hospitals experienced system-related errors (0.73 and 0.51 per admission), which accounted for 35% of postsystem errors in the intervention wards; each system was associated with different types of system-related errors.

I note that "system related errors" were defined as errors "where system functionality or design contributed to the error." In other words, these were unintended adverse events as a result of the technology itself.

The authors conclude:

Implementation of these commercial e-prescribing systems resulted in statistically significant reductions in prescribing error rates. Reductions in clinical errors were limited in the absence of substantial decision support, but a statistically significant decline in serious errors was observed.

The authors do acknowledge some limitations of their (CPOE) study:

Limitations included a lack of control wards at Hospital B and an inability to randomize wards to the intervention.

Thus, this was mainly a pre-post observational study, certainly not a randomized controlled clinical trial.

Not apparently accounted for, either, were potential confounding variables related to the CPOE implementation process (as in this comment thread).

In that thread I wrote to a commenter [a heckler, actually, apparently an employee of HIT company Meditech] with a stated absolute faith in pre-post studies that:

... A common scenario in HIT implementation is to first do a process improvement analysis to improve processes prior to IT implementation, on the simple calculus that "bad processes will only run faster under automation." There are many other changes that occur pre- and during implementation, such as training, raising the awareness of medical errors, hiring of new support staff, etc.

There can easily be scenarios (I've seen them) where poorly done HIT's distracting effects on clinicians is moderated to some extent by process and other improvements. Such factors need to be analyzed quite carefully, datasets and endpoints developed, and data carefully collected; the study design and preparation needs to occur before the study even begins. Larger sample sizes will not eliminate the possible confounding effects of these factors and many more not listed here.

The belief that simple A/B pre-post test that look at error rate comparisons are adequate is seductive, but it is wrong.

Stated simply, in pre-post trials the results may be affected by changes that occur other than the intervention. HIT implementation does not involve just putting computers on desks, as I point out above.

In other words, the study was essentially anecdotal.

The lack of RCT's in health IT are, in general, one violation of traditional medical research methodologies for studying medical devices. That issue is not limited to this article, of course.

Next, on ethics:

CPOE has already been demonstrated in situ to create all sorts of new potential complications, such in at Koppel et al.'s "Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors", JAMA. 2005;293(10):1197-1203. doi: 10.1001/jama.293.10.1197 that concluded:

In this study, we found that a leading CPOE system often facilitated medication error risks, with many reported to occur frequently. As CPOE systems are implemented, clinicians and hospitals must attend to errors that these systems cause in addition to errors that they prevent.

CPOE technology, at best, should be considered experimental in 2012.

In regards to e-Prescribing proper, there's this: Errors Occur in 12% of Electronic Drug Prescriptions, Matching Handwritten and this: Upgrading e-prescribing system can bump up error risk to consider; in other words, the literature is conflicting, confirming the technology remains experimental.

This current study confirmed some (CPOE) errors that would not have occurred with paper did occur with cybernetics, amounting to "35% of postsystem errors in the intervention wards."

In other words, patient Jones was now subjected to a cybernetic error that would not have occurred with paper, in the hopes that patients Smith and Silverstein would be spared errors that might have occurred without cybernetic aid.

Even though the authors observe that "human research ethics approval was received from both hospitals and the University of Sydney", since patient Jones did not provide informed consent to the experimentation with what really are experimental medical devices as I've written often on this blog [see note 1], I'm not certain the full set of ethical issues have been well-addressed. It's not limited to this occasion, however. This phenomenon represents a pervasive, continual world-wide oversight with regard to clinical IT.

Furthermore, and finally: of considerable concern is another common limitation of all health IT studies, which I believe is often willful.

What really should be studied before justifications are given to spend tens of millions of dollars/Euros/whatever on CPOE or other clinical IT is this:

The impact of possible non-cybernetic interventions (e.g., additional humans and processes) to improve "medication ordering" (either CPOE, or ePrescribing) that might be FAR LESS EXPENSIVE, and that might have far less IT-caused unintended adverse consequences, than cybernetic "solutions."

Instead, pre-post studies are used to justify expenditures of millions (locally) and tens or hundreds of billions (nationally), with results sometimes like this affecting an entire country.

There is something very wrong with this, both scientifically and ethically.

-- SS

Note:

[1] If these devices are not experimental, why are so many studying them to see if they actually work, to see if they pose unknown dangers, and to try to understand the conflicting results in the literature? More at this query link: http://hcrenewal.blogspot.com/search/label/Healthcare%20IT%20experiment

Addendum Feb. 10, 2012:

An anonymous commenter points out an interesting issue. They wrote:

The study was flawed due to its failure to consider delays in care and medication administration as an error caused by these experimental devices.

Delays are widespread with CPOE devices. One emergency room resorted to paper file cards and vacuum tubes to communicate urgency with the pharmacy. Delays were for hours.

I agree that lack of consideration of a temporal component, i.e., delays due to technology issues, is potentially significant.

I, for example, remember a more than five-minute delay in getting sublingual nitroglycerin to a relative with apparent chest pain due to IT-related causes. The problem turned out to be gastrointestinal, not cardiac; however, in another patient, the hospital might not be so lucky.

Addendum Feb. 12, 2012:

A key issue in technology evaluation studies is to separate the effects of the technology intervention from other, potentially confounding variables which always exist in a complex sociotechnical system, especially in a domain such as medicine. This seems uncommonly done in HIT evaluation studies. Not doing so will likely inflate the apparent contribution of the technology.

A "control ward" where the same education and training, process re-engineering, procedural improvements, etc. were performed as compared to the "intervention ward" (but without actual IT use) would probably be better suited to pre-post studies such as this.

A "comparison ward" where human interventions were implemented, as opposed to cybernetic, would be a mechanism to determine how efficacious and cost-effective the IT was compared to less expensive non-cybernetic alternatives.

-- SS

EHR Lowers Costs By 33% ... Oh ... Wait A Minute ... We Retract That Statement

There's this:

Blue Cross Blue Shield of RI Pilot Lowered Monthly Costs By Up to 33% Through EHRs
December 13, 2011
Becker's Hospital Review.com
Sabrina Rodak

Health plan members that received care from physician practices using electronic health records had an average of 17-33 percent lower monthly healthcare costs compared to members receiving care at non-EHR physician practices, according to a news release by Blue Cross & Blue Shield of Rhode Island.

BCBSRI conducted a three-year pilot program that partially funded 79 primary care physicians to purchase and use EHRs.

Then, there's this:

Blue Cross backtracks on claim that EHRs lower care costs
December 22, 2011
FierceEMR.com
Dan Bowman

A week after reporting that use of electronic health records lowered the cost of care for its customers between 17 and 33 percent, Blue Cross Blue Shield of Rhode Island has backtracked on those claims. In an email sent to FierceEMR, BCBS/RI says that while the program significantly improved healthcare quality, its cost data had not been risk adjusted and did not include costs related to infrastructure spending. "At this time we are unable to accurately ascertain the cost implications of the pilot and are retracting the news release," the email reads. BCBSRI updated its announcement, which can be found here.

I note the following:

• Healthcare organizations, HIT pundits and vendors seem happy to promote EHR's as saving a great deal of money. I do not believe they ever will. (I have company, such as at Wharton.)

• One can only wonder what led to the retraction of a press release based on a curbstone data analysis, as cited above. Internal dissent, perhaps?

• Healthcare IT, as in this pilot study, can improve healthcare quality metrics (outcomes is another matter as yet undecided as illustrated by this reading list) - but only if done well, a remarkably complex undertaking. That undertaking starts at conception and design, to production, to implementation, to customization, to maintenance of software and hardware during the system lifecycle. See this report by the U.S. National Research Council.

• When healthcare IT is done poorly, it will neither reduce costs, nor improve outcomes, and will increase risk to patients.

• The current ecosystem of commercial health IT and its lack of true regulatory framework for these experimental devices can be counted upon to "do health IT poorly."
  

-- SS