How effective are electronic medication systems in reducing medication error rates and associated harm among hospital inpatients? A systematic review and meta-analysis.

OBJECTIVE: To conduct a systematic review and meta-analysis to assess: 1) changes in medication error rates and associated patient harm following electronic medication system (EMS) implementation; and 2) evidence of system-related medication errors facilitated by the use of an EMS.
MATERIALS AND METHODS: We searched Medline, Scopus, Embase, and CINAHL for studies published between January 2005 and March 2019, comparing medication errors rates with or without assessments of related harm (actual or potential) before and after EMS implementation. EMS was defined as a computer-based system enabling the prescribing, supply, and/or administration of medicines. Study quality was assessed.
RESULTS: There was substantial heterogeneity in outcomes of the 18 included studies. Only 2 were strong quality. Meta-analysis of 5 studies reporting change in actual harm post-EMS showed no reduced risk (RR: 1.22, 95% CI: 0.18-8.38, P = .8) and meta-analysis of 3 studies reporting change in administration errors found a significant reduction in error rates (RR: 0.77, 95% CI: 0.72-0.83, P = .004). Of 10 studies of prescribing error rates, 9 reported a reduction but variable denominators precluded meta-analysis. Twelve studies provided specific examples of system-related medication errors; 5 quantified their occurrence. DISCUSSION AND
CONCLUSION: Despite the wide-scale adoption of EMS in hospitals around the world, the quality of evidence about their effectiveness in medication error and associated harm reduction is variable. Some confidence can be placed in the ability of systems to reduce prescribing error rates. However, much is still unknown about mechanisms which may be most effective in improving medication safety and design features which facilitate new error risks.

INTRODUCTION

In an effort to rapidly amplify and focus activities leading to improvements in medication safety, in 2017 the World Health Organization (WHO) announced the third Global Patient Safety Challenge (GPSC) as “Medication Without Harm.” To meet the WHO’s target of reducing the prevalence of medication errors causing severe patient harm by half within 5 years, it is essential for hospitals to introduce interventions with clear evidence of effectiveness. The Institute of Medicine has promoted the adoption of electronic medication systems (EMS) (also called electronic prescribing systems or computerized provider order entry systems) for well over a decade, and, in the US, incentive payments have been available for the meaningful use of electronic health records.

Systematic reviews of the impact of EMS across inpatient settings have reported significant reductions in medication error rates, as well as improvements to practitioner workflow and adherence to guideline-based care. However, these reviews have typically focused only on prescribing errors and omitted administration errors. Existing systematic reviews have also failed to draw attention to system-related errors (SREs), which have been cited to be frequent with electronic systems yet difficult to detect. Moreover, reviews have consistently highlighted the low quality of individual studies.,

Several systematic reviews have assessed the impact of EMS on medication errors that cause actual patient harm (ie, preventable adverse drug events or pADEs) or potential harm to patients. The assessed outcomes have included reported rates of pADEs;,,,, average error severity ratings (describing potential harm); average lengths of patient stay (LOS);,,, and/or hospital mortality.,,,, However, these reviews often include studies relying upon incident reports, which have been shown to underestimate error rates.

Further, many previous reviews have only included studies that were specific to a ward type (eg, ICU). Only 3 reviews have provided a meta-analysis of study results, and all focused on intensive care settings.,, Two of these 3 reviews reported no significant change in measures of medication-related patient harm., The remaining review included studies published prior to 2014 and reported a significant reduction in the rate of pADEs among adult inpatients (pooled risk ratio 47%; based on 6 studies, 2 of which were conducted prior to 2000).

Synthesized evidence about the impact of introducing EMS on the rate of harmful medication errors across hospital inpatient groups (eg, for those on different wards, with different clinical conditions or age ranges) is unclear from the available evidence in systematic reviews. This highlights the need to collate quality research to determine how effective these systems may be at contributing to the current WHO GPSC of Medication Without Harm. Thus, our aims were to conduct a systematic review and meta-analysis of studies to assess: 1) changes in the rates of medication errors and any associated patient harm following the introduction of an electronic medication system; and 2) evidence of medication errors facilitated by the use of an EMS.

MATERIALS AND METHODS

Search strategy

We searched the Embase and Medline (via Ovid), and the Cumulative Index to Nursing and Allied Health Literature (CINAHL; via EBSCOhost) databases for English-language literature published between January 2005 and March 2019. Our search strategy used a combination of MeSH terms and keywords related to the intervention (computerized provider/physician order entry and clinical decision support [CDS]) and outcomes of interest (medication errors, LOS, mortality, and patient harm). We also “hand-searched” the reference lists of full-text articles that were assessed for potential inclusion. The complete search strategy was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, including registration with PROSPERO (registration number CRD42018080778), and is provided in Supplementary Appendix SA.

Study selection

We executed our search strategy on 25 March 2019 and found 2936 citations. After removing duplicates, 2 researchers (PG, RH) screened the titles and abstracts of citations. All potential articles identified from each independent assessment were assigned to full-text review. The same 2 researchers then reviewed the full-text articles and discussed reasons for inclusion or exclusion and met to discuss disagreements and gain consensus. Multiple reports of the same study were assessed for unique data. No single report of multiple studies was found. The study selection process is presented in Figure 1.

Figure 1.

Study selection process.

We defined an electronic medication system (EMS) as a computer-based system for electronic prescribing and/or administration of medicines. Systems with or without detailed CDS and those integrated or not with larger hospital electronic health record systems were included. All types of study designs conducted in single or multiple wards were included if they reported on: changes in medication error rates and/or medication error-related harm before and after the introduction of an EMS in hospitals that previously used paper-based ordering; or cross-sectional studies of a single hospital that randomly allocated patients to a ward using paper-based ordering or to an EMS.

Studies were required to report data in at least 1 of 4 outcomes:

rate of medication errors (ME), defined as the combination of error subtypes including, at a minimum, prescribing errors (PE) and medication administration errors (MAE);

rate of PE, where the study definition of PE included a minimum of the 2 most frequent clinical error types of wrong dose and wrong time;

rate of MAE, where the study definition of MAE included multiple types of administration errors that met Shawahna et al’s definition of: “any deviation from the prescriber’s medication order as [directly] observed or [indirectly identified] on the patient’s chart, manufacturers’ preparation/administration instructions, or relevant institutional policies”;

rate of potential or actual patient harm associated with ME, PE, and/or MAE. These studies reported on harm outcomes including error severity ratings (typically associated with potential harm, for example the Index published by the National Co-ordinating Council for Medication Error Reporting and Prevention [NCC-MERP]), number of pADEs (actual harm), LOS, and/or mortality rate (actual harm).

Exclusion criteria were:

studies solely using incident reports to identify errors (due to the known underreporting of errors by incident reports). Studies identifying PE by chart review and MAE by observation were included;

studies where EMS was introduced in conjunction with an additional intervention at the same time which may have confounded results. Some examples of these excluded studies were EMS implemented in parallel with an automated unit-dose trolley, or dispensing system, patient barcoding, and education-based stewardship programs. We included studies which introduced system training or workflow optimizations to compliment EMS implementation and describe these where relevant in Supplementary Appendix SB;

studies that were highly selective or specialized. For example, studies which focused only on specific medications,, or medication groups such as chemotherapy; as well as studies of a specific medication error type;

studies only available as abstracts, posters, editorials, or opinion pieces;

studies conducted outside of inpatient settings (eg, outpatients, simulation, or primary care clinics);

studies without data from a paper-based setting (eg, the introduction of clinical decision support to an existing electronic system).

Data extraction

We extracted study details including study author(s), year of publication, country in which the study was conducted, type of hospital, study design, full description of the EMS, duration, and timing of data collection and system implementation. For studies that measured medication errors and/or prescribing errors, we extracted error definitions, number of patients, patient days, orders, and administrations, as well as number of errors or error rates at baseline and intervention. For studies that measured administration errors, we also extracted number of opportunities for error (OE, defined as all doses administered plus any doses omitted that could be classified as either correct or incorrect), and errors per OE, at baseline and intervention. For studies that measured mortality, we extracted number of patient deaths in hospital at baseline and post-EMS. For studies that measured LOS, we extracted mean or median LOS before and after EMS. For studies that measured patient harm, we extracted the number of preventable adverse drug events, or mean severity ratings, before and after intervention. We also extracted any quantitative or qualitative data regarding SREs. Specifically, quantitative data included the rate, or proportion, of prescribing errors, or administration errors, that were reportedly facilitated by the EMS. Qualitative data included information presented about the types of SRE identified or specific descriptive accounts of these errors. As the majority of EMS operate with differing levels of functionality and CDS, from basic alerts for duplicate orders to patient-specific algorithms, we extracted all information describing the EMS and CDS provided in the reviewed articles. As shown in Table 1, the CDS was classified using the language used by study authors as: “absent” (3 studies), “basic” (2 studies), “rudimentary” (1 study), “moderate” (2 studies) and “extensive” (1 study). In addition, some studies did not specifically describe the level of CDS but indicated that the EMS had capabilities for alert notifications (9 studies). Details of the type of EMS and CDS in the reviewed studies is provided in Supplementary Appendix SB.

Table 1.

Characteristics of the included studies

Author (year)	Country	Hospitals, Study wards	Patient population	EMS vendor	Study description of CDSa	Data collection	Outcomes reported	Quality rating
Al-Sarawi et al (2009)41	Australia	3 Public, Multiple-ward	NR	EPAS	Alertsb	PCR	PE	2: Weak
Armada et al (2014)42	Spain	1 Teaching, ICU	Adult, Cardiac	FarmaTools Dominion	“Moderate”a	PCR	PE, LOS, Mo, Hm, SRE	6: Moderate
Colpaert et al (2006)43	Belgium	1 Teaching, ICU	Adult	Centricity Critical Care Clinisoft	“Moderate”a	PCR	PE, LOS, Hm	10: Strong
Han et al (2016)44	USA	1 Teaching, ICU	Adult	EPIC Systems Corp	None	Obs.	Mortality, LOS	7: Moderate
Hernandez et al (2015)45	France	1 Teaching, General	Adult, Orthopaedic	NR, Commercial	Alertsb	Obs.	PE, MAE	6: Moderate
Hinjosa-Amaya et al (2016)46	Mexico	1 Teaching, General	Adult + Paediatric	NR	NR	RCR	ME	7: Moderate
Holdsworth et al (2007)47	USA	1 Public, Multiple ward	Paediatric	Eclipsys System 2000	”Extensive”a	PCR	Hm	8: Moderate
Jheeta and Franklin (2017)48	UK	1 Teaching, General	Adult, Elderly	NR	Alertsb	Obs.	MAE	2: Weak
Joy et al (2012)49	USA	1 Public, Multiple ward	Adult + Paediatric	EPIC Systems Corp	NR	PCR	ME, Hm	5: Moderate
Jozefczyk et al (2013)50	USA	1 Teaching, ICU	Paediatric, Neonatal	McKesson’s Horizon Expert Orders	”Rudimentary”a	RCR	ME	4: Weak
Mills et al (2017)51	UK	1 Public, Multiple ward	Adult	NR, Commercial	Alertsb	RCR	LOS, PE, SRE	4: Weak
Oliveros et al (2016)52	Spain	1 Teaching, General	NR	Prescriwin	“Basic”a	RCR	MAE, SRE	4: Weak
Pontefract et al (2018)53	UK	3 NR, General	Adult	NR, Commercial	Variedc	PCR	PE	4: Weak
Shawahna et al (2011)54	Pakistan	1 Teaching, Multiple ward	Adult + Paediatric	NR	None	PCR	PE	5: Moderate
Van Doormaal et al (2009)55	Netherlands	2 Teaching, General	Adult	iSoft Medicator + Theriak	“Basic”a	PCR	LOS, ME, Hm	8: Moderate
Walsh et al (2008)56	USA	1 Public, Multiple ward	Paediatric	Sunrise Clinical Manager	Alertsb	PCR	ME, Hm, SRE	8: Moderate
Warrick et al (2011)57	USA	1 Public, ICU	Paediatric	ICIP	Alertsb	PCR	PE	4: Weak
Westbrook et al (2012)58	Australia	2 Teaching, General	Adult	CMPO + iSoft MedChart	Alertsb	PCR	PE, SRE	10: Strong

Abbreviations: CDS, Clinical Decision Support; CMPO, Cerner Millennium Power Orders; EMS, Electronic Medication System; EPAS, The Enterprise Patient Administration System; General, the study included only general wards; Hm, Harm; Multiple-ward, study included both general and intensive care wards; ICU/ED, the study included only intensive care or complicated patient wards (eg, oncology, neonate); ICIP, Intellivue Clinical Information Portfolio; PCR, Prospective Chart Review (charts were assessed during the normal course of daily work); PE, Prescribing Error; Public, the hospital was nonteaching, RCR, Retrospective Chart Review (archived charts were assessed in retrospect); LOS, length of stay; Mo, mortality; NR, not reported; Obs., direct observations conducted pre- and post-EMS implementation; SRE, quantitative reporting of system-related errors.

The terms “basic,” “rudimentary,” “moderate,” and “extensive” are taken verbatim from study author. For a complete description of the EMS and CDS, see Supplementary Appendix SB.

Many studies did not provide a specific description of CDS, though they did mention some alert functionality, such as potential drug–allergy and drug–drug interactions.

This study assessed the impact of 5 levels of CDS. For a complete description, please see Supplementary Appendix SB.

Quality assessment

Two researchers (PG, RH) independently assessed the methodological quality of the included studies and met to discuss disagreements and gain consensus. This quality assessment comprised application of a combination of a previously described 10-point Methodological Quality Assessment and the Effective Public Health Practice Project (EPHPP) quality assessment tool for quantitative studies. These tools were selected for their applicability to both randomized and nonrandomized studies and suitability for use in systematic reviews through expert judgement or previous use.,,, We adapted these tools to best reflect the complexity of conducting these “real-world” intervention studies and removed the criterion requiring control groups as—while highly desirable—opportunities for obtaining control groups may be very limited for real-world studies of this type of organization-wide IT intervention., Thus, our quality assessment focused on 6 components: 1) allocation bias, 2) unit of allocation bias, 3) data collection methods, 4) completeness of the samples, and 5) assessment of confounders. A further component, 'clarity of reporting', was added in order to score how clearly the electronic system being assessed, and the patient population, were reported. Using these 6 components, each with a score of 0–12, we describe study quality as weak (score 0–4), moderate (score 5–8), and strong (score 9–12). Full details of this assessment by individual study is presented in Supplementary Appendix SC.

Statistical analysis

Authors of articles to be included in the meta-analyses, but containing insufficient information (n = 18), were sent emails requesting additional data to be provided, and, in 4 cases, data were received. Meta-analyses were performed if there were at least 3 studies with sufficient data reported in published studies or provided by authors, using random effects models to pool results examining the impact of EMS on medication error rates and associated harm. Risk ratios and their 95% confidence intervals (CIs) were calculated. To be conservative, the Knapp-Hartung approach was applied to account for heterogeneity between studies. Forest plots were used to present meta-analyses results. Between-study heterogeneity was evaluated using I2 statistics. The potential for publication bias for each meta-analysis was assessed by inspection of funnel plots and statistical tests based on weighted linear regression of the intervention effect on its standard error. Sensitivity analyses were conducted by grouping studies by their quality rating, and also for studies with and without CDS, provided there were at least 3 studies with sufficient information. All statistical tests were 2-sided and were evaluated at a significance level of 0.05. Analyses were carried out using R version 3.6.1.

RESULTS

Study characteristics

Eighteen studies met our inclusion criteria. Characteristics of the included studies are provided in Table 1. Study publication dates ranged from 2006 to 2019. All studies used a before–after design, except for 2 interrupted time series without control,, and 1 controlled cross-sectional trial. One before–after study included a control group. Seven studies were rated as being of weak quality,,,, 9 moderate,,,, and 2 studies were rated as strong., Individual study results for each outcome are presented in Supplementary Appendix SD.

Changes in medication error (ME) rates pre- and post-EMS

Five studies assessed the impact of EMS on the incidence of ME,,,,, with 4 studies rated to be of moderate quality.,,, Differences in the reported numerators and denominators across studies prevented meta-analysis examining the association between the introduction of EMS and ME rates. A significant reduction in the rate of ME was reported in 2 of the 4 moderate quality studies., One study, conducted in a pediatric hospital, reported no significant change in error rates at 9 months post-EMS, and 1 study, conducted in the intensive care wards of an adult hospital, reported a significant increase in ME rates at 3 months post-EMS.

Changes in prescribing error (PE) rates pre- and post-EMS

Ten studies assessed the impact of EMS on PE rates, with a range of denominators reported across studies.,,,,, Heterogeneity in error rate reporting and study settings prevented meta-analysis of the association between EMS and rates of PE. EMS, with or without CDS, was associated with a significant reduction in PE rate in all but 1 study. That study was rated to be of weak quality and found no significant change in PE per 100 orders. Only 1 study compared before–after changes in a control group. That study identified no statistically significant change in the PE rate in the control group.

Changes in medication administration error (MAE) rates pre- and post-EMS

Three studies assessed the impact of EMS on MAE rates, each assessing the rate of errors per OE. All were included in our meta-analysis.,, Two studies reported a significant reduction in MAE rates after the intervention., Overall, the meta-analysis showed a significant reduction in MAE rates post-EMS implementation (pooled RR: 0.77, 95% CI: 0.72–0.83, P = .004) (Figure 2). There was little evidence of heterogeneity among these studies (P < .73, I2<0.1%). There was evidence of publication bias (P = .5).

Figure 2.

Relative risk of a medication administration error following the introduction of EMS.

Changes in the rate of ME, PE, or MAE with potential for harm or causing harm pre- and post-EMS

Nine studies assessed the impact of introducing an electronic medication system (EMS) on the rate of ME, PE, or MAE with potential for harm or causing actual patient harm using a range of denominators.,,,, Five studies assessing the impact of EMS on the proportion of ME (n = 2), and PE (n = 3),, to cause actual harm were included in the meta-analysis. All 5 studies classified actual harm severity using the NCC-MERP index. Two studies reported results at 1 and 3 months post-EMS,, however; as there were no statistically significant changes in harm across these periods, the results from both were combined for this analysis. The 5 studies were found to be significantly heterogeneous (heterogeneity between studies: I2=97.0%, P < .001). There was no evidence of publication bias (P = .8). Overall, the meta-analysis results indicated that the introduction of EMS had no significant effect on patient harm (pooled RR: 1.22, 95% CI: 0.18–8.38, P = .8) (Figure 3). Sensitivity analysis was conducted for 3 studies of moderate quality,,, and there was no significant effect (pooled RR: 0.83, 95% CI: 0.01–78.26, P = .8).

Figure 3.

Relative risk of actual patient harm related to medication errors following the introduction of EMS.

Results were mixed in the 4 studies not included in the meta-analysis. One of 2 moderate quality studies assessing the rates of patients experiencing at least 1 pADE, reported a significant reduction post-EMS. One strong quality study reported a reduction in the rate of potentially harmful PE per admission The remaining study assessed the rate of PE that reached the patient with the potential to cause any harm, was rated to be of strong quality, and reported a significant reduction post-EMS, from 1.0 per 100 orders to 0.15 per 100 orders.

Changes in length of stay and mortality pre- and post-EMS

Differences in statistical parameters (ie, mean, median) and patient groups across studies prevented meta-analysis examining the association between introduction of EMS and LOS. Six studies assessed the association between an EMS and LOS; 5 in adult hospitals,,, and 1 in a pediatric hospital. Overall, there was little evidence of change up to 1 year post-EMS implementation in the adult hospitals. The largest moderate quality study, reported a significant change in the LOS on general wards in 2 adult hospitals 2 months post-EMS. That study also reported the highest baseline average LOS, with large variation across patients. The only pediatric study included over 1000 patients in each group, was rated as moderate quality, and found no significant reduction in mean LOS 15 months post-EMS using a logistic regression model which included all variables found to be significantly different between the 2 study periods.

Only 2 studies assessed the association between an EMS and hospital mortality rates, both in adult ICUs., The first, a small study of poor quality, found no significant reduction in mortality 3 months post-EMS with “moderate” CDS. The second, a large, moderate quality study, showed a significant increase in mortality in the first 4 months following system introduction using data adjusted for patient mortality risk, though reported an overall significant reduction in mortality at 5 months postintervention.

System-related errors (SREs) introduced post-EMS

Twelve studies provided qualitative examples of SREs,,,,,, though only 5 reported on SRE rates.,,,, We report the full qualitative data extraction in Supplementary Appendix SD. Regarding the quantitative data, 3 studies reported rates of SRE as 0.4% of prescribed drugs, 1.6 SREs per 1000 patient-days, and 0.57 SREs per admission. Four studies reported the percentage of medication errors that were related to system use,, which ranged from 1.2% to 34.8% of all errors, with heterogeneity in study quality and in the hospitals and EMS implemented.

The 5 studies providing quantitative data also assessed SREs in relation to the overall effect of EMS on medication error rates.,,,, These studies assessed the extent to which the SRE associated with the introduction of EMS moderated any identified change in the rate of medication errors following EMS introduction. Three weak quality studies reported an overall significant decrease in medication error rates despite new errors associated with system use and advocated that the impact of new errors can be minimized with additional training and adherence to EMS use guidelines.,, One moderate quality study of a pediatric hospital post-EMS reported no significant change in the rate of ME and that the rate of SRE was low and unlikely to have moderated this lack of effect. The remaining study, of strong quality, reported a significant reduction in procedural prescribing error rate despite new SRE. That study also reported a significant reduction in the clinical error rate, though no statistically significant change was found when new SRE were included. However, the overall proportion of prescribing errors classified as “potentially serious” was lower post-EMS.

DISCUSSION

This systematic review and meta-analyses examined the extent to which introducing an electronic medication system (EMS) in hospital wards which previously used paper-based medication ordering was associated with changes in ME rates (including both PE and MAE at a minimum) and any associated potential or actual patient harm associated with those errors. Although half of the 18 studies published between January 2005 and March 2019 assessed potential or actual patient harm, the impact of EMS on patient harm remains unclear as studies were heterogeneous and results mixed. Meta-analysis found no evidence of a significant overall effect on patient harm related to medication error rates (combining data on ME, PE, and MAE), with studies showing positive, negative, and no change following the introduction of EMS. In contrast, there was consistent evidence across 14 studies that changing from a paper-based medication system to an EMS is likely to reduce medication error rates, particularly prescribing errors.,,,,,,,Meta-analysis results also indicated a significant reduction in administration error rates, though only 3 studies were included in this analysis.,, The very few studies that measured the relationship between changes in medication error rates and the introduction of SREs, found that new SREs are likely to marginally limit overall reductions in medication errors. Those studies reporting SREs consistently reported that system implementations which included ongoing support and the flexibility to adapt systems to respond to user input could reduce the overall rate of SRE and improve the safety benefits of these systems.

Consistent with other interventions designed to reduce the rate of medication error, such as the participation of a clinical pharmacist, or the use of checklists and guidelines,, the jury is still out as to whether introducing EMS will significantly impact rates of medication-related harm. These results support observations of a continuing trend where decisions to implement health information technology such as EMS are based on motivators other than research evidence of effectiveness. The lack of consistent evidence of effects on patient harm may be explained in several ways. Firstly, previous reviews,, and the present review, have identified an absence of high-quality studies. We identified that the use of quasi-experimental study designs with no control group, inadequate reporting of data collection methods, and failure to report between group differences and consider potential confounding variables were the main contributors to poor study quality. Given that actual patient harm is reasonably infrequent,, studies with large samples are required to assess the impact on harm. We identified only 1 strong quality study which reported outcomes on actual patient harm.

Secondly, EMS represented in the review may not have significantly impacted rates of medication-related patient harm due to CDS functionality. CDS in EMS is an important feature shown to be effective in targeting particular types of medication errors yet was limited in many systems studied and poorly described by others. Only 4 studies reported CDS functionality beyond simple system alert notifications.,,, One of these studies was included in our meta-analysis, though that study did not report any actual patient harm before or after system implementaion. Among the 4 studies with advanced CDS, 3 assessed changes to the rate of prescribing error and all reported a significant reduction,,, and 3 assessed actual patient harm, and, excluding the study which found no harm, the remaining 2 studies reported a significant reduction postimplementation., One study assessed 3 hospitals which introduced 2 different EMS with differing levels of CDS to paper-based ordering and measured the change in the rate of prescribing errors. In that study, although the authors reported a significant reduction in prescribing errors that had targeted CDS, no linear association was found between CDS functionality and error reduction, with different results generated across the 2 sites with the same system. These authors concluded that the manner of system implementation may have significantly moderated this association.

The successful adoption of EMS requires significant adaption of work practices, acceptance of innovation, and good organizational fit, and the methods of implementation and ongoing improvements in response to user-needs can significantly impact system effectiveness. We found that details of the implementation methods were only briefly reported in studies or entirely omitted (see Supplementary Appendix SB for details), thus making it difficult to determine how these factors may moderate system effectiveness. Ignoring the importance of the organizational and social context in which EMS is to be implemented may have significant impact on the system effectiveness. Review of factors associated with successful system implementation and studies included in this review have identified the importance of adequate system training,,, and the use of support staff. Though the level of training or support that would be considered adequate is unclear.

Finally, interpretation of results relating to reductions in medication-related patient harm should consider the overall mix of medication error types included in the calculation of rates pre- and post-EMS. EMS implementation brings immediate benefits in terms of the standardization of orders, almost eliminating “procedural” errors such as illegible orders, incomplete orders, incorrect units, etc. These errors are highly prevalent at baseline, yet usually of low potential severity. Consequently, comparison of the rates of medication errors causing harm pre- and post-EMS should report medication-related harm rates for “clinical’ and “procedural” medication errors separately.

Our results highlight the difficulties faced by those measuring the effectiveness of interventions designed to reduce medication error-related harm and meet the goals of the WHO Global Patient Safety Challenge. This review identified several ways to improve future research as others have called for. In particular, outcomes should be reported as error rates: medication errors per order, or if this is not feasible, per 1000 patient days; the proportion of errors with the potential to cause patient harm; the proportion of patients who experienced actual harm; and the level of harm severity. Actual harm resulting from medication errors is understudied and should be an important focus of future studies.

Limitations

This review focused on evidence of the impact of EMS on patient harm and medication errors; however, we acknowledge that the use of electronic systems in hospital settings are more widely transformative and provide other benefits that we did not assess. In addition, all systematic reviews are limited by the number and quality of the included studies. We utilized 2 quality assessment tools to describe the quality of included study methods as weak, moderate, or strong., We found only 2 strong quality studies. In addition, pooling study outcomes was limited by the small number of studies and variation in the chosen denominator to measure outcomes and in the chosen methods for describing patient harm. Our review was limited by the poorly reported, heterogenous, and often short follow-up periods of less than 7 months across studies postimplementation of EMS. This is important, as factors that would impact on the outcomes we assessed, such as changes in staff perceptions of or proficiencies in using these systems, required system improvements or the development of workarounds that would not be captured. A strength of this review was its wide criteria for study inclusion which were not limited by patient group or hospital setting. Moreover, by excluding studies using incident reports, the included studies were less likely to underreport error rates. Finally, there was heterogeneity in the definitions of medication error and harm across studies; however, we attempted to address this limitation by including only studies that met our minimum criteria.

CONCLUSION

Our review identified evidence indicating that introducing EMS is likely to significantly reduce rates of medication errors, particularly prescribing errors. However, evidence of changes in the rate of harmful medication errors is less clear. Further, insufficient studies have investigated the effects on medication administration error rates. We identified few methodologically strong studies, particularly of advanced systems with substantive clinical decision support. Future controlled trials should be powered to be able to detect changes in medication errors resulting in harm but also need to focus on the identification of specific mechanisms and system features most effective in improving medication safety along with those that may facilitate new error risks.

EMS systems have been used in many hospitals around the world for well over 2 decades. The limited and poor-quality effectiveness evidence base is somewhat surprising given the size and costs associated with these interventions and their extensive reach in terms of potentially impacting the care of all hospital inpatients. The methodological challenges in conducting studies of this type is 1 reason for this deficit, but this situation is also likely a reflection of complacency regarding the necessity for evaluation of large-scale information technology (IT) systems. Many still view these systems as IT interventions despite their integral role and desired impact on clinical care. While few would argue with the great potential of clinical IT systems, it is critical that we assess their actual impact in real-world settings and identify system, user, and organizational factors which will allow this potential to be realized in terms of improved clinical care now and into the future.

FUNDING

This work was supported by the National Health and Medical Research Council, grant numbers [APP1094878, APP1143941, and APP1174021] and by Macquarie University, grant number [MQRDG IRIS # 9201601531].

AUTHOR CONTRIBUTIONS

All authors have participated in the concept and design or analysis and interpretation of data and drafting or revising of the manuscript. All have approved the manuscript as submitted and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Click here for additional data file.