Computerised physician order entry-related medication errors: analysis of reported errors and vulnerability testing of current systems.

IMPORTANCE: Medication computerised provider order entry (CPOE) has been shown to decrease errors and is being widely adopted. However, CPOE also has potential for introducing or contributing to errors.
OBJECTIVES: The objectives of this study are to (a) analyse medication error reports where CPOE was reported as a 'contributing cause' and (b) develop 'use cases' based on these reports to test vulnerability of current CPOE systems to these errors.
METHODS: A review of medication errors reported to United States Pharmacopeia MEDMARX reporting system was made, and a taxonomy was developed for CPOE-related errors. For each error we evaluated what went wrong and why and identified potential prevention strategies and recurring error scenarios. These scenarios were then used to test vulnerability of leading CPOE systems, asking typical users to enter these erroneous orders to assess the degree to which these problematic orders could be entered.
RESULTS: Between 2003 and 2010, 1.04 million medication errors were reported to MEDMARX, of which 63 040 were reported as CPOE related. A review of 10 060 CPOE-related cases was used to derive 101 codes describing what went wrong, 67 codes describing reasons why errors occurred, 73 codes describing potential prevention strategies and 21 codes describing recurring error scenarios. Ability to enter these erroneous order scenarios was tested on 13 CPOE systems at 16 sites. Overall, 298 (79.5%) of the erroneous orders were able to be entered including 100 (28.0%) being 'easily' placed, another 101 (28.3%) with only minor workarounds and no warnings. CONCLUSIONS AND RELEVANCE: Medication error reports provide valuable information for understanding CPOE-related errors. Reports were useful for developing taxonomy and identifying recurring errors to which current CPOE systems are vulnerable. Enhanced monitoring, reporting and testing of CPOE systems are important to improve CPOE safety. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Computerised provider order entry (CPOE) has long been considered and demonstrated to be a high-leverage tool for preventing medication errors, and incentives are being provided to accelerate its adoption.1–3 However, there is a growing awareness and increasing documentation of concerns that CPOE can also introduce or facilitate new errors.4–6 The Institute of Medicine Committee report ‘Health IT and Patient Safety: Building Safer Systems for Better Care’ recognised that Health Information Technology (HIT) is part of a complex sociotechnical system and recommended investing in efforts to uncover and understand the vulnerabilities of HIT systems to errors and unintended consequences.7 More recently, the US Food and Drug Administration Safety and Innovation Act (FDASIA) similarly recommended developing similar approaches for reporting with a key recommendation advocating compilation of reports of errors across multiple systems.8

In 1999, the United States Pharmacopeia (USP) launched a pioneering online medication error reporting system that has now collected more than two million medication errors.9 In 2003, in response to the growing number of reports suggesting that CPOE was playing a role in the medication errors being reported, USP added a coded field for reporters to check off ‘CPOE’ as a contributing cause of the error. Shortly thereafter, the USP's MEDMARX annual report stated that computer entry and CPOE errors had become the third leading contributing cause being checked off in medication error submissions.10 However, since that initial report there has been no detailed investigation or analysis of CPOE-related MEDMARX error reports. Furthermore, the report narratives have not been assessed previously. To better understand how and why the errors were occurring, as well as ways they could have been prevented, we undertook a study of the USP MEDMARX CPOE medication error reports, subjecting these reports to a detailed review as well as testing the vulnerability of current CPOE systems to the types of errors identified.

The aims of this study are to (1) analyse the USP MEDMARX medication error reports where CPOE was checked off as a ‘contributing cause’ of the error by performing in-depth review of 10 000 of the error report narratives to understand details of each error and develop a new taxonomy for CPOE-related errors and (2) develop and test ‘use cases’ based on these actual error reports and assess the vulnerability of leading CPOE systems to these errors.

Methods

Phase I: MEDMARX data analysis—taxonomy development and coding

We queried the USP MEDMARX (now part of Quantros Safety and Risk Management suite) for all medication error reports from January 2003 to April 2010 that were coded by the error reporters as having ‘CPOE’ as one of the ‘contributing causes’ of the errors. These spontaneous error reports were submitted from institutions subscribing to the MEDMARX medication error reporting system. Reporters typically included a mix of centralised quality assurance staff (who collected reports from front-line staff and then entered reports for their institution into MEDMARX) and in a minority of institutions, front-line staff directly entering reports.

We identified a total of 63 040 medical error reports having CPOE checked off as a contributing cause in 1.04 million total reports. These reports served as the data for analysis of CPOE errors and taxonomy development. A total of 10 060 error reports were then manually reviewed, representing all 191 of the reports categorised as National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) outcome categories E–I (ie, categories where patient harm occurred, which we designated as highest priority to review in detail) plus a random sample of the remaining A–E category reports.11 Each of these 10 060 error reports was reviewed and coded by one of three clinical pharmacists (MGA, JJB and ACS) with an emphasis on detailed review of the free text narrative description of the error. For cases where there were questions, or consideration of new codes, the cases were re-reviewed in detail by the entire team of three pharmacists and a general internist (GDS). These codes grounded in the data and served as the basis for our taxonomy development (grounded qualitative analysis).12

The coding was done using a customised qualitative-coding software tool developed in Microsoft Access with codes progressively developed or added based on the error report narratives and iteratively refined via weekly meetings of the clinical review team. Each report was coded to categorise three elements of the error: (a) what happened, (b) why it happened and (c) potential prevention strategies. Pharmacist reviewers were instructed to code what and why exclusively based on information in the narrative and accompanying report. For potential prevention strategies, pharmacist investigators were encouraged to suggest ways the error could have been prevented based on the report and their knowledge of medication safety and information technology. To ensure conservative coding, when reports lacked sufficient information to determine a what, why and prevention classification, reviewers were instructed to assign them as ‘unknown’. Each case could be assigned one or more codes in each category. Inter-reviewer reliability was assessed using a 1% random sample of reviews to calculate a kappa statistic. Once the coding was completed, the codes were re-reviewed and reorganised using several card sorting and iterative team consensus exercises to group and refine the final taxonomy.

Phase II: CPOE vulnerability testing based on reported error scenarios

During the above qualitative review of the error reports, reviewers were instructed to flag cases that might serve as representative ‘test cases’ to assess whether errors identified could be replicated in current CPOE systems. A weighted scoring system based on error frequency, severity, generalisability and testability was used to narrow this list down to key error scenarios for testing. Based on this prioritisation, 21 test scenarios were chosen. Scenarios included erroneous or problematic CPOE orders related to wrong units, major overdoses, drug allergies, order element omission errors, wrong frequency and drug–disease contraindications, as well as three ‘correct’ but ‘complex’ test orders (eg, prednisone tapers, alternate-day dosing, non-formulary drug) that reports often suggested led to problem-prone workarounds such as potentially confusing free text comments in the drug order (see online supplementary appendix 1 for list of test case scenarios).

We identified a convenience sample of leading vendor and homegrown CPOE system test sites and obtained institutional and institutional review board permission to enter these problematic orders on test patients at these sites. To test each of these error scenarios, we recruited one to two typical users (mostly medical residents or primary care attending physicians) with at least 1-year experience with the CPOE system (range 1–8) and instructed them to enter the erroneous orders. Users understood that these orders were problematic but were instructed to proceed with placing these orders as they typically would, using any methods they might routinely use to enter a desired order. Outcomes of whether orders were successfully entered and behaviours of medical doctors and CPOE systems were recorded by physicians or pharmacists (GDS, TE, MGA and ACS) and research assistant (DLW) observers who rated the ease or difficulty of entering the erroneous or complex orders using predefined operational definitions (table 1).

Table 1

Operational definitions used to classify ease of entry of ‘error scenario’ test orders

How easy was it to place the order?
1	2	3	4	5
Easy	Minor workarounds	Some protections	Difficult	Impossible
1. Easy ▸ End user successfully and quickly entered the erroneous order ▸ No alerts/warnings ▸ No workarounds or additional mouse clicks required ▸ Order ‘sailed through’ (order simply accepted as if was a normal error)
2. Minor workarounds ▸ End user is able to enter the order fairly easily ▸ No alerts/warnings ▸ Requires some kind of additional workarounds (eg, needed to adjust default dosing or enter all or part of the order in free text, or use of comments field to complete order)
3 Some protections ▸ End user is able to enter the order ▸ ‘Passive’ alerts/warnings appear  – Warning appears but it can be ignored (no over-ride required)  – Warning appears but can over-ride with single mouse-click (this includes selecting a reason for over-ride from pull-down menu) ▸ Typical response from the provider is to say ‘I usually just blow through these [warnings]’ or equivalent.
4 Difficult ▸ End user is able to enter the order, but doing so requires a conscious, concerted effort ▸ ‘Active’ alerts/warnings appear that require additional action from provider (eg, typed reason for over-ride) ▸ Often, typed workarounds and extra mouse clicks are required to over-ride ▸ Order often does not go through on first attempt ▸ Significant time and thought required to enter successfully ▸ Articulated end-user frustration
5 Impossible ▸ Order could not be entered, despite attempted workarounds ▸ No way to enter order in free text comments field ▸ Hard-stop warnings appear or significant changes are required to send to pharmacy (eg, required to d/c order or remove drug/diagnosis) ▸ System is completely ‘bulletproof’, at least in regard to this particular order

Results

Phase I: qualitative review

Of 1.04 million reported errors, 63 040 (6.1%) were classified by the reporters as CPOE related. Our pharmacists reviewed and coded 10 060 (15.7% sample) of these 63 040 reports and derived a taxonomy that included 101 codes describing what occurred, 67 codes describing why errors occurred as well as 73 codes describing potential prevention strategies (see online supplementary appendix 2 for full taxonomy). Tables 2–4 summarise findings for the top 25 most frequent codes assigned for the what, why and prevention codes. Many reports lacked sufficient detail describing the error to permit adequate coding, particularly to classify why the error occurred. Although all of these reports had ‘CPOE’ checked off by the reporter as a contributing cause, our reviewers could determine the role of the CPOE system in only 5004 (49.8%) of the reports based on report content alone. Pharmacists’ inter-rater agreement rates and kappa scores for the taxonomy coding of what occurred and why the error occurred were 66%, kappa 0.56 (95% CI 0.39 to 0.72) and 64%, kappa 0.58 (95% CI 0.42 to 0.73), respectively.

Table 2

Top 25 what happened? Codes

Code	N
Missing or incorrect directions/patient instructions	2088
Ordered wrong dose or strength	877
Missing quantity or wrong number ordered	877
Unknown	680
Wrong schedule entered	566
Duplicate order: same exact drug	510
Overdose or potential overdose	376
Ordered wrong formulation/dosage form	363
Order not processed/delayed	361
Extra dose potential	337
Ordered wrong drug	302
Routing issue	275
Comment field issue	267
Missed does potential	256
Nursing administration issues	240
Wrong time selected	234
Ordered/entered for wrong patient	229
Discontinuation issues	216
Not processed/delayed: order confusing/needed clarification	203
Patient missed dose	203
Omitted drug	167
Ordered wrong PO formulation, (eg, ER, XR, etc)	160
Patient given extra dose	152
Telephonic/verbal order issues	139
Correct drug ordered/wrong drug processed	138

ER, extended release; PO, by mouth; XR, extended release.

Table 3

Top 25 why did it happen? Codes

Code	N
Unknown	5326
Multiple systems (two or more electronic systems)	1211
Use of system or SIG abbreviations	494
Failure to follow established procedures or protocol	480
Profiling Issues: failure to perform or use correctly	443
Inexperienced end user	415
Lack of computer training/system knowledge	325
Typing error	206
Hybrid system (electronic and paper)	205
Communication issues	200
System limitations/inadequacy: routing/mapping issue	186
Lack of clinical knowledge	186
Medication reconciliation issue	184
Alert ignored/overridden	153
Nursing administration issues	149
Pharmacy order entry problems/issues	134
Transcriptions (copy/paste)	133
Comments field free text confusing/confusion	121
eMAR/MAR issues	119
Order set/template/protocol issues	113
Drug dictionary miscode/out-of-date drug information	109
Patient identification issue	95
Initial vs continuing order issue	93
Patient transferred (within hospital)	91
Misinterpretation of order(s)	72

eMAR, electronic medication administration record; MAR, medication administration record; SIG, sig code for directions on how to take a medication.

Table 4

Top 25 prevention codes

Code	N
Unknown	3021
Systems integration	1520
Standardised constructs for dosing regimens	1142
Enhanced education/training	791
Standardised SIGs	775
Autocalculation for prescription quantities	604
Duplicate order checking/support	403
Default dosing selections	318
Drug database improvements/enhancements	232
Dose range checking	212
Individual dosing calculations	207
Improved design/functionality	201
Medication reconciliation support	200
Scheduling feedback	169
Blank field checking	141
Standardised constructs for dose form-route	136
System for reconciling new/now with continuing dosing	116
Enhanced allergy entry for drugs not included in allergy list	109
Direct order entry to minimise verbal/telephonic issues	106
Route-formulation checking	93
Include time in checklist (12, 24, etc)	91
Duplicate therapy checklist/support	89
Formulary status and restrictions warnings	89
Medication handoff/transfer standardisation	86
Better testing of order sets/updates	69

SIG, sig code for directions on how to take a medication.

Phase II results: CPOE vulnerability testing

Our pharmacist reviewers identified 338 error reports as potential candidate scenarios for testing the vulnerability of current systems. This list was narrowed to 21 scenarios by combining similar scenario types (ie, various drug overdosages; orders for drug to which patient was allergic) and prioritised based on preselected criteria of (a) frequency, (b) seriousness and (c) testability. These scenarios included five inpatient-only scenarios (eg, intravenous orders) that were not tested on outpatient systems, and three ‘complex’ orders (not errors per se, but designed based on reports that repeatedly arose from similar potentially error-prone orders reported). We recruited a convenience sample of 13 representative systems (four homegrown, one open source, eight commercial, including each of the leading inpatient and outpatient vendors) at 16 test sites. Not all tests could be performed on all systems (because of formulary and other design limitations). Excluding these scenarios, we entered a total of 375 erroneous orders during 24 testing sessions on 13 systems at 16 test sites.

Overall, 298 (79.5%) erroneous orders were able to be placed, including 100 (28.0%) being ‘easily’ placed (order simply accepted with no extra steps or warnings), with another 101 (28.3%) placed with only ‘minor workarounds’ (eg, adjusting default dosage, with no warnings). Thus, 201 (56.3%) of the errors could be relatively easily replicated (entered easily or with minor workarounds) with no warning or blocking of potentially dangerous orders. Table 5 lists the frequencies of how often erroneous orders were prevented versus went through easily or with some difficulty. Only 26.6% of orders generated specific warnings related to the erroneous order. Of these, 69% were passive alerts (information display only or easily over-ridden and/or ignored). Another 29% required workarounds but nonetheless, could still be entered. Notable failures included erroneous orders for pioglitazone accepted for patients with congestive heart failure in 87.5%, orders for insulin 60 ‘mL’ (rather than ‘units’) going through in 75.0% and no specific warnings for a 1000-fold levothyroxine overdose in 37.5% of attempts. Figure 1 illustrates a breakdown of which of the erroneous order scenarios had greater protection (ie, were more difficult to enter; higher mean scores overall) versus those where systems were generally more vulnerable (lower mean scores overall). Finally, for 40 of 72 (55.6%) of the error-prone more complex (eg, variable daily warfarin, prednisone taper) test orders, prescribers encountered problems or ordering difficulties, creating potentials for error-prone workarounds.

Table 5

Frequency distribution of erroneous orders going through, ease with which they went through, and whether there was a warning

	N	Per cent
Did order go through?
Yes	298	79.5
No	59	15.7
Untestable	18	4.8
Likert scale: difficulty
Easy	100	28.0
Minor workarounds	101	28.3
Some protections	69	19.3
Difficult	28	7.8
Impossible	59	16.5
Warnings?
Yes	95	26.6
No	216	60.5
Irrelevant warnings only	44	12.3
Uncertain/maybe	2	0.6

Figure 1

Radar plot showing mean score for each test scenario across all tested computerised provider order entries in difficulty of entering erroneous orders. To maximise safety, the plot ideally should occupy the most outer grid (score 5); that is, impossible to enter the erroneous orders. For example, greatest protection was against 1000-fold overdose of levothyroxine; however, drug–disease contraindication checking had the lowest mean score indicating least protection, hence making it easier to enter this erroneous order.

Discussion

We analysed a large medication error database for errors that were reported as being related to CPOE and developed a new taxonomy of the types, causes and prevention strategies that we could identify in these reports, resulting in a number of useful insights regarding the frequency of specific error types. We then performed vulnerability testing to examine whether these errors could be replicated in current CPOE systems with the worrisome finding that the majority (overall, 56.3%) of the selected erroneous orders could be readily entered.

Report narratives provided both rich details of the types of errors that occur in CPOE systems and served as the basis for the development of a taxonomy that facilitated classification of the types. Leading CPOE-related errors included missing or erroneous sig (label directions) or patient instructions, wrong dose or strength, problems with wrong quantity or strength, scheduling problems (particularly related to inpatient orders and timing of stat vs continuing orders), delays in medication processing or administration due to confusing orders and wrong drug or wrong patient errors.

Many of these problems are not unique to CPOE and could also occur with handwritten ordering, although some, in theory (eg, drug overdosages), should be preventable with properly designed electronic systems.2 13–15 Reasons for these errors were discernible for roughly half of the error narrative reports and included problems with miscommunication between multiple electronic or hybrid paper-electronic systems, user issues such as failure to follow established protocols, inexperience or lack of training in using the CPOE system, typing and pull-down menu errors, medication reconciliation issues, ignoring or over-riding alerts and confusion related to or arising from comments fields.

Although it may be argued that many errors were isolated occurrences or perhaps based on the vulnerability of older systems, when we tested current systems, we found that current systems had high degrees of susceptibility to many of these errors. Nearly 80% of the erroneous orders could be entered, with more than half entered with little or no difficulty or warnings. More than a quarter (28.0%) of the orders were easily entered (in the words of our test physicians and research pharmacists, ‘sailed right through’), with no warnings or additional efforts on the part of the ordering physicians.

Systems that overalert or frustrate busy physicians attempting to enter appropriate orders are also an important problem and can lead to the so-called ‘alert fatigue’. Thus, systems need to balance ease of ordering with appropriate protections. Our study was not designed to determine the best balance, although better designed systems have the potential to achieve both better efficiency (eg, well-designed order sets) and improved signal-to-noise ratio (better ratio of appropriate to nuisance alerts).15–18 Shockingly, one of our test systems had zero warning alert fire in response to our erroneous test orders; we discovered that all alerts had been turned off for a system upgrade several months earlier and it was not until we performed our testing that it was discovered they had not been reinstated. We documented a high degree of variability of vulnerability and alerting from system to system. We even observed variations in different implementations of the same system or even different users entering orders in different ways at the same site. This is similar to the findings when the Leapfrog tool was used to test potential errors, revealing wide variations in detection of test orders that would cause adverse events with varied local implementations, even of the same vendor's systems.19

From the policy perspective, one approach would be to regulate electronic health records and/or clinical decision support. Another would be to allow vendors to continue the current approach in which there is relatively little regulation but to improve postmarket surveillance. The FDASIA committee has recommended the latter approach to the Food and Drug Administration (FDA), Office of the National Coordinator and Federal Communication Commission.8 The data from our study suggest that it is possible to aggregate large numbers of reports across multiple vendors and draw useful conclusions.

Our study was limited by the fact that the reported CPOE-related errors were based on spontaneous self-reporting of medication errors. Thus, no conclusions can be drawn about the actual incidence or relative frequency of these errors and problems. In addition to the well-known problem of under-reporting from spontaneous reporting systems, there are problems with the quality and non-verifiability of the reports we studied.20 21 Many reports were incomplete, lacked details of the role CPOE played in the error (ie, only that they had the box checked that CPOE was contributing factor). Thus, as instructed, our reviewers conservatively coded these reports as ‘unknown’, for what happened and why, making this a leading category. Despite these limitations of the quantity and quality of these spontaneous reports, the errors speak for themselves as noteworthy and likely (particularly based on our vulnerability testing results) representative CPOE safety issues. This points to the need to improve the quality of such reports and perhaps standardise collection of contributing causal factors. Additional potential limitations are that, reports and subscribers to MEDMARX may not be representative of all prescribing systems, and outpatient reports were under-represented as subscribers were mainly hospitals and hospital systems. Many of the reports were nearly a decade old, although our efforts to replicate these errors demonstrate that vulnerabilities also exist in current systems. Our study did not examine the likelihood that the erroneous orders placed would be intercepted by pharmacists or other staff and hence, not cause harm. Nonetheless, the MEDMARX reports contained many examples of where errors did reach or harm patients. Finally, our qualitative pharmacist coding and rating of the error scenarios was based on subjective reviewer judgement. To offset this, we assigned clear operational definitions to the codes as they were developed, had a consensus process for adjudicating questions or disagreement and achieved reasonably good inter-reviewer reliability scores in assessment of the reports. We have continued to refine this taxonomy for a white paper to be published by the US FDA that can help guide future research as well as organisations analysing CPOE-HIT error reports in the future.

In conclusion, we reviewed error reports to identify patterns of CPOE-related errors and used them to develop a new taxonomy and recurring error scenarios. We then tested current systems and found areas of noteworthy vulnerability. Developers and users need to be aware of this potential for error and should build in protection strategies at multiple levels to learn from and protect patients by continuously improving the safety of CPOE systems.22–25 Efforts that permit both better reporting and awareness of medication errors as well as testing the vulnerabilities of local CPOE systems are needed; such efforts are crucial to safe prescribing, ongoing postimplementation monitoring and improvement of CPOE systems.