Impact of a commercial order entry system on prescribing errors amenable to computerised decision support in the hospital setting: a prospective pre-post study.

BACKGROUND: In this UK study, we investigated the impact of computerised physician order entry (CPOE) and clinical decision support (CDS) implementation on the rate of 78 high-risk prescribing errors amenable to CDS.
METHODS: We conducted a preintervention/postintervention study in three acute hospitals in England. A predefined list of prescribing errors was incorporated into an audit tool. At each site, approximately 4000 prescriptions were reviewed both pre-CPOE and 6 months post-CPOE implementation. The number of opportunities for error and the number of errors that occurred were collated. Error rates were then calculated and compared between periods, as well as by the level of CDS.
RESULTS: The prescriptions of 1244 patients were audited pre-CPOE and 1178 post-CPOE implementation. A total of 28 526 prescriptions were reviewed, with 21 138 opportunities for error identified based on 78 defined errors. Across the three sites, for those prescriptions where opportunities for error were identified, the error rate was found to reduce significantly post-CPOE implementation, from 5.0% to 4.0% (P<0.001). CDS implementation by error type was found to differ significantly between sites, ranging from 0% to 88% across clinical contraindication, dose/frequency, drug interactions and other error types (P<0.001). Overall, 43/78 (55%) of the errors had some degree of CDS implemented in at least one of the hospitals.
CONCLUSIONS: Implementation of CPOE with CDS was associated with clinically important reductions in the rate of high-risk prescribing errors. Given the pre-post design, these findings however need to be interpreted with caution. The occurrence of errors was found to be highly dependent on the level of restriction of CDS presented to the prescriber, with the effect that different configurations of the same CPOE system can produce very different results. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Introduction

Computerised physician order entry (CPOE) with integrated clinical decision support (CDS) software is associated with a reduction in medication errors1–4 and adverse events.2 5 The impact of CPOE on medication errors is strongly influenced by the complexity of the system(s) in use and the baseline error rate of the study site.6–8 Unfortunately, owing to variations in the definitions for error and error types, as well as the data collection techniques adopted by healthcare researchers, direct comparison of error rates between sites (and settings) is difficult.2 8–11 In those studies that do conduct multisite analyses, researchers are required to adopt inter-rater reliability testing to assess the classification of error types, and often the severity.12 This approach can be time-consuming, which reduces the feasibility of the methodology as a regular and ongoing strategy for monitoring error rates.

In 2012, an eDelphi study was conducted to identify high-risk prescribing errors for the medical inpatient hospital setting that were amenable to CDS if prescriptions were generated via a CPOE system.13 A total of 80 prescribing errors were identified by 20 experts that could be used to standardise the review of prescriptions for errors preimplementation and postimplementation of CPOE in a hospital. Targeting the capture of prescription error data in a structured and objective manner using a clear definition of what constitutes an error can help to minimise uncertainty over whether or not an error has actually occurred.9 14 The approach can provide a powerful insight into the incidence and prevalence of defined higher risk errors that occur across healthcare settings. In this case, the data can be used to identify how factors such as the complexity of systems and the level of CDS may impact on error rates, to better inform quality improvement initiatives. While it is important to review and monitor for actual harms (or adverse events) that occur during the medication process, such as with retrospective review of ‘trigger’ events,15 such methods do not identify potential harms. The use of triggers has also been found to identify a lower proportion of adverse events relating to prescribing in the UK hospital setting compared with prospective review of prescriptions.16

This research aimed to investigate the rate at which high-risk prescribing errors occurred preimplementation and postimplementation of CPOE in hospital sites in England, and consider whether the level of CDS had an impact on the error rates observed. To date, evidence for the impact of CPOE and CDS technology in the hospital setting has largely been generated from studies conducted outside the UK, since the digital maturity of hospitals in this country is still emerging and evolving.17 This study was the first of its kind to measure the impact of commercial systems in the UK hospital setting using a predefined list of high-risk prescribing errors. Considering the pace of change relating to digitisation across the National Health Service (NHS),18 the findings from this study are vital for identifying potential high-risk errors that may impact on patient safety, and so inform the safe implementation and use of systems across a complex system of care.

Materials and methods

Setting

Acute NHS hospitals in England that were planning on implementing CPOE were identified as part of a National Institute for Health Research programme grant: ‘Investigating the adoption, implementation and effectiveness of electronic prescribing systems in English Hospitals’ (RP-PG-1209-10099). Hospitals were eligible for inclusion if (1) they were at least 6 months from implementation; and (2) on invitation to participate agreed that they had the necessary staff resource (ie, a pharmacist or experienced pharmacy technician) to capture prescribing data for the defined sample size. It was estimated that it would take no more than 10 min to capture the necessary data for each patient reviewed. Four hospitals were identified based on the first inclusion criteria relating to their implementation plans and were invited to participate in the study via email, with a follow-up meeting to explain what the study would entail and to discuss any approvals required from their local research and development departments. Sites were informed that their participation must remain confidential so that participants remained blinded to the other sites involved. All four hospitals agreed to participate in the study, but only three implemented CPOE during the study period of the programme grant and so were able to collect both preimplementation and postimplementation data.

Data collection

Each hospital was given access to the iMPACT (Investigating Medication Prescribing Accuracy for Critical error Types) audit tool to facilitate the capture of data relating to the presence of 80 high-risk prescribing errors in the general inpatient setting, all of which have the potential to be prevented by CDS13 (see online supplementary appendix 1). iMPACT is a stand-alone Windows application designed to capture data on the presence of each of the 80 errors through a process of medication chart review and answering a series of objective questions relating to the medicines a patient is prescribed. For every medicine (including regular, when-required and one-off doses), the drug was compared against the list of 80 trigger drugs and drug classes from the list of errors to see whether the prescription could be considered an ‘opportunity for error’. Where this was the case, additional questions were presented to the auditor to assess whether an error had occurred (figures 1 and 2). To give an example, one of the indicators was ‘Paracetamol prescribed at a dose of 4 g over a 24 hours period to a patient less than 50 kg’. For this, any orders where paracetamol was prescribed would be classified as an opportunity for error. Of these, any cases where the patient’s weight was less than 50 kg, and the dose prescribed to be administered was equal to or greater than 4 g in 24 hours, would be classified as an error having occurred. Where drugs were included in multiple defined errors, for example there were three that mentioned gentamicin (see online supplementary appendix 1), a single order would be classed as contributing multiple opportunities for error.

iMPACT data capture tool: prescribing errors. Drugs and drug classes are categorised for ease of use. iMPACT, Investigating Medication Prescribing Accuracy for Critical error Types.

Capturing drug and drug class data on iMPACT. Clicking on a drug or drug class reveals a series of questions about the prescription and/or the patient’s medical and drug history. iMPACT, Investigating Medication Prescribing Accuracy for Critical error Types.

In a pilot study of 100 patients at a paper-based hospital, it was estimated that, across all medicines prescribed, 5.5% would contain one of the 80 high-risk errors. The study was powered to detect a reduction in the overall error rate from 5.5% to 4.0% (approximately a quarter). In order to achieve power of 80% (with a two-sided P value of 5%) in each hospital, it was calculated that 3156 opportunities for error were required preimplementation and postimplementation. Assuming an average number of opportunities for error of 9.1 per patient (from pilot data), sites were informed that this was expected to translate to around 346 patients (ie, 3156/9.1). Based on an assumption of patients having an average of around 10 prescriptions each, the target sample size at each site was 4000 prescriptions in both the preimplementation and postimplementation periods.

The CPOE pharmacist at each hospital was trained on the use of iMPACT, so that they could train the pharmacist or technician allocated to perform the data collection for the study. The pharmacist/technician at each hospital site was asked to visit inpatient medical wards and review patients’ medication charts using iMPACT for a minimum of 4000 prescription orders on paper and for a further 4000 orders 6 months postimplementation of CPOE. Sites were asked not to review patients in the surgical, paediatric or critical care settings, since the 80 errors were not identified with these specialties in mind. Sites were asked to keep a track of which patients had been reviewed to avoid duplication. Case selection used a convenience sample approach, with sites given flexibility regarding the time that reviews were carried out and the patients chosen for review. This approach was used to minimise the impact of data collection on the organisation (ie, with minimal impact in workflow). The only exception to this was a request not to conduct the review when prescribers were likely to be unfamiliar with the prescribing processes at the site—specifically in August/September when doctors commence training or rotate to new posts in the UK. In cases where the pharmacist or technician perceived an error to have occurred during the audit process, they were asked to approach the reporting of the error according to local protocol at their hospital. On completion, each site exported their data via secure NHS email to the University Hospital Birmingham NHS Foundation Trust for analysis. Finally, sites were asked to provide feedback regarding their use of iMPACT, so that any problems with the use of the software could be resolved or highlighted to the researchers, or could be used to inform interpretation of the data during analysis (eg, any difficulties collecting information requested by the tool).

Decision support

Following data capture post-CPOE, the sites were provided with the list of 80 prescribing errors (online supplementary appendix 1) and asked to simulate these in their CPOE training systems (ie, not for live patient admissions). Sites were asked to document any CDS that appeared to the prescriber during the simulated prescribing process. This was necessary to determine the level of CDS existing for each of the errors, so that associated CDS and error rates could be assessed. After all the data had been received, a researcher (SS) analysed the details of the decision support for each error at each site and categorised these into one of five stages of alerting, according to set definitions (table 1). The classifications of each instance of CDS were discussed with a second researcher (SKP) to validate the coding.

Table 1

Levels of CDS relating to error prevention and the associated impact on prescriber workflow

Level of CDS		Description	Impact on workflow
5	Restricted	The prescriber is prevented from proceeding with the prescription with a hard-stop alert.	Interruptive
4	Permitted	An alert appears where a reason needs to be given by the prescriber to over-ride it in order to progress.	Interruptive
3	Guided	Elements of the prescription order are auto-populated for the prescriber (eg, dose, frequency, route).	Interruptive/ passive*
2	Alert	An alert appears, but the prescriber does not need to over-ride this with adding a reason in order to progress.	Passive
1	None	No CDS appears at the point of prescribing.	None

*Passive if the guided decision support is accepted, or interruptive if it is over-ridden.

CDS, clinical decision support.

Data analysis

For each of the sites, the total numbers of opportunities for error and the numbers of errors that occurred relating to each of the high-risk errors were calculated. The error rates were then defined as the number of errors that occurred divided by the numbers of opportunities for error. These rates were compared between the pre-CPOE and post-CPOE periods using Fisher’s exact tests. Errors were then classified according to one of the five levels of CDS that was implemented (table 1), which were then further divided for the analysis into none, passive or interruptive, and the comparisons repeated within each subgroup. Guided CDS could be either passive or interruptive, depending on the information entered by the prescriber. Since it was not possible to record sufficient data to identify whether passive or informative guidance was provided in each individual case, it was assumed that all guided CDS was at the passive level, as this is what would be expected to occur in the majority of cases (ie, where the prescription was within the guidelines enforced by the CDS system).

The approaches to implementation across the three sites were then compared. Each error was classified by its type and risk rating score. The latter was defined as per the original eDelphi study, using the National Patient Safety Agency risk matrix19 (online supplementary appendix 2). The proportion of errors within each category of these factors where CDS was applied was then compared, using Fisher’s exact test for the error type and Kendall’s tau for the risk rating, to account for the ordinal nature of this factor. All analyses were performed using IBM SPSS V.22, with P<0.05 deemed to be indicative of statistical significance throughout.

Results

Three out of the four hospitals eligible for inclusion in the study completed the data collection pre-CPOE and post-CPOE implementation. Two of the hospitals (sites 1 and 2) implemented the same CPOE system (table 2).

Table 2

Description of the hospitals that participated in the study

Hospital	CPOE system	Source of system	Ability to locally configure CDS	Interoperability with other electronic records
Site 1	System A	Commercial	Yes	Yes
Site 2	System A	Commercial	Yes	Yes
Site 3	System B	Commercial	Yes	No

CDS, clinical decision support; CPOE, computerised physician order entry.

Disparities with the data

Two of the 80 errors were identified by the sites as problematic during data collection: (1) low molecular weight heparin (LMWH) prescribed without the patient’s weight being used to calculate the treatment dose (risk of subtherapeutic or supratherapeutic dosing); and (2) LMWH omitted to be prescribed for prophylaxis (increased risk of thrombosis). For the first of these, feedback from one site identified that the CPOE system was designed to minimise the risk of the error occurring by mandating that the weight was entered as a separate field within the prescription and not just in the patient demographics section. If the weight had been omitted in the patient’s demographics section, it could appear to the data collector that the weight had not been used to inform the dosing regimen for the LMWH, giving a false rate of error. For the second error it was identified that, post-CPOE implementation, some sites struggled to access assessments for venous thromboembolism in order to answer the questions presented by the iMPACT tool. In view of these findings, these two errors were removed from the analysis, leaving 78 errors for consideration.

Opportunities for error

The drug charts of 2422 patients were audited across the three hospitals—1244 preimplementation and 1178 postimplementation of CPOE. A total of 28 526 individual medicine orders were prescribed across both periods, with 21 138 opportunities for error identified. A summary of the opportunities for error and high-risk errors occurring across all sites pre-CPOE and post-CPOE is provided in online supplementary appendix 3. Preimplementation, 75/78 (96%) of the defined prescribed errors across the three sites had at least one opportunity to occur (NOpp), and 70/78 (90%) postimplementation. Over the study period, 44 had >100 opportunities for error and 28 had >200. Of the top 10 found to have the highest opportunity for error, 7 involved just three drugs: paracetamol (n=2), LMWH (n=2) and opioid analgesic (n=3). There were three errors where no opportunity for the error occurred during the study (namely those relating to amphotericin B, orlistat and nefopam) (online supplementary appendix 1).

Across the three sites as a whole, the proportion of opportunities for error that resulted in an error occurring was found to reduce significantly after the introduction of CPOE, from 5.0% to 4.0% (P<0.001) (table 3). When analysed on a site level, similar reductions in error rates were observed after the introduction of CPOE at both site 1 (4.3% vs 2.8%, P=0.002) and site 3 (7.4% vs 4.4%, P<0.001). However, no significant change between periods was detected for site 2, with error rates of 4.0% vs 4.4% in the pre-CPOE versus the post-CPOE periods (P=0.294).

Table 3

Summary of the opportunities for an error occurring and high-risk errors that occurred in the three hospitals

Site	Pre-CPOE				Post-CPOE					P value
	Patient records	NOpp	NErr	Error rate (%)	Patient records	NOpp	NErr	Error rate (%)	Relative risk (95% CI)
1	365	3867	167	4.3	297	2634	74	2.8	0.65 (0.49 to 0.86)	0.002
2	615	4513	179	4.0	581	4527	200	4.4	1.11 (0.91 to 1.36)	0.294
3	264	2932	216	7.4	300	2665	116	4.4	0.59 (0.47 to 0.74)	<0.001
Total	1244	11 312	562	5.0	1178	9826	390	4.0	0.80 (0.70 to 0.91)	<0.001

NOpp represents the number of error opportunities generated, while NErr is the number of errors that were found to have occurred. P values are from Fisher’s exact tests, with bold values significant at P<0.05.

CPOE, computerised physician order entry.

Table 4 summarises the top 10 errors that occurred preimplementation and postimplementation of CPOE. Seven errors consistently occurred in both processes, although with a slightly different rank in overall frequency.

Table 4

Top 10 errors occurring pre-CPOE and post-CPOE implementation across all sites

Prescribing error	Pre-CPOE				Post-CPOE
	Rank	NOpp	NErr	Error rate (%)	Rank	NOpp	NErr	Error rate (%)
Regular opiates prescribed without concurrent use of laxatives (risk of severe constipation)	1	486	55	11.3	3	438	37	8.4
Benzodiazepines prescribed long term (ie, more than 2–4 weeks) (risk of dependence and withdrawal reactions)	2	129	40	31.0	2	122	38	31.1
Digoxin prescribed concomitantly with a diuretic (risk of hypokalaemia and subsequent digoxin toxicity)	3	76	39	51.3	5	59	25	42.4
Insulin prescribed to a patient at an inappropriate time, allowing for an administration without food (except once-daily long-acting insulins) (increased risk of hypoglycaemia)	4	104	28	26.9	1	113	50	44.2
Citalopram prescribed concomitantly with other QT-prolonging drugs (increased risk of arrhythmias)	5	161	28	17.4	8	141	19	13.5
Clopidogrel prescribed to a patient concomitantly with omeprazole or esomeprazole (antiplatelet effect of clopidogrel potentially reduced)	6	409	27	6.6	11*	366	14	3.8
Paracetamol prescribed at a dose of 4 g over 24 hours to a patient under 50 kg (risk of hepatocellular toxicity)	7	841	25	3.0	4	863	29	3.4
Selective serotonin reuptake inhibitor prescribed concomitantly with tramadol (increased risk of serotonin syndrome)	8	161	24	14.9	17*	141	5	3.5
Potassium-sparing diuretic (excluding aldosterone antagonists) prescribed to a patient also receiving an ACE inhibitor or angiotensin-II receptor antagonist (increased risk of severe hyperkalaemia)	9	334	23	6.9	6	272	22	8.1
Statin prescribed concomitantly with a macrolide antibiotic (increased risk of myopathy)	10	434	20	4.6	15*	406	6	1.5
More than one paracetamol-containing product prescribed to a patient at a time (maximum dose exceeded)	15*	841	14	1.7	7	863	20	2.3
ACE inhibitor or angiotensin-II receptor antagonist prescribed to a patient with a potassium level >5.0 mmol/L (can cause or exacerbate hyperkalaemia)	11*	334	19	5.7	9	272	19	7.0
Soluble insulin prescribed to a patient on a when-required basis (increased risk of serious episodes of hypoglycaemia and nocturnal hypoglycaemia post dose)	17*	104	13	12.5	10	113	16	14.2

*Errors ranked outside the top 10 for the stated period, but are included as they were inside the top 10 for the other period.

CPOE, computerised physician order entry; NErr, number of errors; NOpp, number of opportunities for error.

In an attempt to explain the lack of improvement in error rates observed at site 2, separate exploratory subgroup analyses were performed for each of the 78 errors being considered. This identified two errors relating to insulin prescribing that showed considerable increases in frequency after CPOE was introduced at this site. The first—‘Insulin prescribed to a patient at an inappropriate time, allowing for an administration without food (except once daily long-acting insulins) (Increased risk of hypoglycaemia)’—had an error rate that increased from 21.3% (10/47) to 54.2% (32/59) after CPOE implementation, while the second—‘Soluble insulin prescribed to a patient on a when required basis (Increased risk of serious episodes of hypoglycaemia and nocturnal hypoglycaemia post dose)’—saw an increase from 2.1% (1/47) to 22.0% (13/59). After excluding these, the error rate for the remaining 76 errors at site 2 was found to show a modest reduction post-CPOE implementation from 3.8% (168/4419) to 3.5% (155/4409).

Clinical decision support

A summary of the CDS implemented at each site for each of the 78 errors is provided in online supplementary appendix 4. A total of 43/78 (55%) errors had some degree of decision support implemented (ie, alert to restricted; table 1) in at least one of the sites—leaving 45% (35/78) of the errors with no CDS at all. The rate of CDS implementation did not differ significantly across the three sites (P=0.293), with 27/78 (35%) of errors triggering some form of CDS at sites 2 and 3, and 19/78 (24%) doing so at site 1 (figure 3). Only three errors had CDS implemented at all three sites, namely prescribing amphotericin B without stating the brand name and dose, generic prescribing of tacrolimus without stating the brand and prescribing penicillin-containing compounds to patients with penicillin allergy. Despite the overall rate of CDS being similar at the three sites, there were differences in the manner of implementation (online supplementary appendix 5, figure 4). Sites 2 and 3 used interruptive CDS almost exclusively, with these making up 100% (27/27) and 96% (26/27) of CDS, respectively. Site 1, on the other hand, had a significantly greater usage of passive CDS, with 26% (5/19) being at the alert or guidance level.

Figure 3

Implementation of any level of CDS by hospital site. P values are from Fisher’s exact test, and bold P values are significant at P<0.05. CDS, clinical decision support.

Figure 4

Implementation of passive and interruptive CDS at the hospital sites. * Passive CDS at Site 3 = 4%. CDS, clinical decision support.

For those errors with no CDS assigned, a significant reduction in the error rate was observed at site 1 (4.3% vs 3.0%, P=0.023) and site 3 (7.2% vs 3.9%, P<0.001) after CPOE implementation, while a significant increase in error rate was observed at site 2 (3.3% vs 5.0%, P=0.001) (online supplementary appendix 5). This is likely a result of the insulin errors previously described, since site 1 adopted a restricted CDS for this, while site 2 using the same system had no CDS implemented at all.

Passive CDS was used infrequently, not being used for any of the 78 errors at site 2. For site 1, no significant change in error rates was detected for those with passive CDS (P=0.673), while the single error with passive CDS at site 3 saw a reduction in the error rate from 6.0% to 0.0% (P=0.007). Errors with interruptive CDS saw significant reductions in error rate after CPOE implementation at site 1 (2.9% vs 0.3%, P<0.001) and site 2 (5.5% vs 3.0%, P=0.002), with a near-significant reduction at site 3 (7.8% vs 5.7%, P=0.056).

In all three sites, the rates of CDS differed significantly by the type of error, with implementation rates ranging from 0% to 88% across clinical contraindication, dose/frequency, drug interaction and other error types (all P<0.001; table 5). While sites 2 and 3 generally targeted similar types of error with CDS, the priorities for site 1 were significantly different (P<0.001). Of the 25 errors relating to drug interactions, sites 2 and 3 introduced CDS in 21 (84%) and 22 (88%), respectively, while site 1 only had CDS for 1 (4%) of these errors. On the other hand, site 1 introduced CDS for 6/29 (21%) of the clinical contraindication errors, while sites 2 and 3 only had CDS for 1 (3%) of these. The rate of CDS implementation was not found to differ significantly by the risk rating of the error at any of the sites (table 5).

Table 5

Summary of error types with any level of clinical decision support across the three sites

	Errors (n)	Errors with CDS, n (%)			P value (between-site)
		Site 1	Site 2	Site 3
Error type					<0.001
Clinical contraindication	29	6 (21)	1 (3)	1 (3)
Dose/Frequency	13	8 (62)	4 (23)	0 (0)
Drug interaction	25	1 (4)	21 (84)	22 (88)
Other†	11	4 (36)	2 (18)	4 (36)
P value (within-site)		<0.001	<0.001	<0.001
Risk rating‡					0.903
High risk§ (8–10)	40	7 (18)	13 (33)	12 (30)
High risk (12)	34	10 (29)	11 (32)	13 (38)
Extreme risk (15–25)	4	2 (50)	3 (75)	2 (50)
P value (within-site)*		0.113*	0.372*	0.463*

Within-site P values compare the CDS implementation rates across the error types/risk ratings. Between-site P values compare the distribution of CDS across the error/risk ratings between the three sites. P values are from Fisher’s exact tests.

*Kendall’s tau and bold P values are significant at P<0.05.

†Other: drug name (n=2); indication (n=1); omission (n=2); route (n=1); and timing/duration (n=6).

‡Risk rating scores are as detailed in online supplementary appendix 2 from the eDelphi study.13

§High-risk rating ranges from 8 to 12, with 12 being the highest risk.

CDS, clinical decision support.

Discussion

This study is the first of its kind to investigate the impact of commercial CPOE systems with integrated CDS on the rate of high-risk prescribing errors in the UK hospital setting. We analysed over 28 000 prescription orders from over 2400 patients for the presence of 78 defined high-risk prescribing indicators of error, preimplementation and postimplementation of commercial CPOE systems in three hospital sites. Our novel approach to data capture allowed for a direct comparison of error rates between sites, and for the influence of factors such as the level of CDS to be considered in the analysis.

The majority of the 78 errors identified in a previous eDelphi study13 had at least one opportunity to occur across the three sites, and almost half provided at least 100 opportunities each. This provides evidence on the effectiveness of the method at identifying opportunities for error, and subsequent errors for medicines that are likely to be prescribed and are high risk in the inpatient general medical setting. The drugs and drug classes with the most opportunities for error were medicines that are frequently prescribed in UK hospitals. Paracetamol is continuously listed as one of the top 10 medicines prescribed in the UK community setting,20 which increases the likelihood of a prescription also being generated for a patient on admission to hospital. LMWHs have the potential to be prescribed for every patient admitted to the hospital, owing to a national guideline recommending that all patients are assessed for risk of venous thromboembolism.21 In addition to the frequency of prescribing, LMWHs and opioid analgesics are listed as classes of medicines most associated with death or severe harm outcomes.22

The implementation of CPOE with integrated CDS was associated with a significant reduction in the rate of high-risk errors amendable to the software. However, when analysed on a site level, this was only found to be significant in two of the three hospitals (sites 1 and 3). Despite using the same system as site 1, a significant (and comparable) change in error rate was not observed at site 2. This was largely a result of an increase in the frequency of errors for which no CDS had been put in place by the hospital. The increased error rate may be a result of suboptimal or insufficient training provided to staff postimplementation of CPOE and for newer staff inducted at the hospital, the provision of which is essential for minimising unintended effects of the technology.23–25 Errors relating to the prescribing of insulin in particular were identified as problematic post-CPOE. This may reflect an overdependence on the capability of the system, with prescribers having a false expectation that the system will alert them to all types of error,26 and as such inadvertently generate prescriptions that are inaccurate or suboptimal. This overdependence is more likely to occur if prescribers assume that the alerts presented in a system in one hospital are the same as those in the same system in another. This reinforces the need for users of systems to be made aware during their training that systems may be configured very differently. It may also emphasise the importance of standardisation of configuration between the same systems for errors that are considered high risk.

The rate of CDS implementation was similar at each of the three sites, ranging from approximately a quarter to a third of the errors having some CDS configured. However, there were differences in the manner of implementation, even between those sites using the same system (sites 1 and 2 using system A). First, the types of errors targeted with CDS differed, with two of the hospitals (sites 2 and 3) choosing to focus most of their CDS on ‘Drug interaction’ errors, while the third hospital (site 1) focused more on ‘Clinical contraindication’ and ‘Dose/Frequency’ errors. Second, when hospitals did choose to implement CDS, the level of alerting implemented was significantly different, with two sites (sites 2 and 3) almost exclusively applying interruptive CDS and one opting to apply more passive alerts (site 1). These findings suggest that the decision-making process for local configuration is not entirely influenced by the technological capability of the system in use, and that other factors are likely to be at play. Site 1 had CDS implemented for the smallest number of errors and also had the lowest rate of interruptive CDS. This may have been due to a decision to take a more cautious approach to the implementation—mindful of the potential for ‘alert fatigue’, which may negate the beneficial effects of the software.27–30 In contrast, the two sites with the most CDS took a very different approach, and almost exclusively implemented alerts that were interruptive for the prescriber. This may highlight an organisational concern regarding liability should an error occur that the system had the functionality to reduce the risk of, or prevent entirely.27 Alternatively there may be a lack of knowledge relating to the unintended consequences of such configuration among the implementation team. Hospitals may also have been targeting errors that they felt were most likely to occur in their setting or have previously been known to cause harm (eg, based on local incident reports). Finally, with the rapid increase in the use of CPOE worldwide,31 32 there are a multitude of quantitative and qualitative studies regarding implementation. Since implementation in the UK is relatively new, it may also be the case that some hospitals are reviewing the literature to inform their strategy for configuration. For example, a US study by Phansalkar et al 33 lists 15 high-priority drug–drug interactions that should have CDS implemented, which may explain why some hospitals chose to implement more drug interaction alerts over any other error type. Six of the 15 alerts in this study were considered high risk in the UK, as identified through our eDelphi study.13 Two sites in this study had interruptive CDS implemented for all six of these drug interactions, and one site had none implemented at all. Further research would be beneficial to determine the sociotechnical factors that may influence the strategies adopted by hospitals to inform the process of CDS implementation.

The ability to locally configure CPOE systems is an important factor for the successful implementation of the technology in an organisation.24 34 However, local configuration demands both time and expertise. The variation in practice observed may highlight a lack of national guidance in relation to implementation strategies, previously highlighted as a problem in a UK-based study of hospitals in England.30 34 It may also highlight a lack of knowledge as a result of little or no informatics training and limited experience with the technology.35 In the UK, it costs between £0.5 and £3 million to implement CPOE, depending on the digital maturity of the hospital, existing information technology infrastructure and staff resource.17 Given the financial investments and the potential for unintended consequences when new technologies are introduced, it is important to assess whether organisations are using systems optimally to improve the quality of prescribing to benefit patient care.36 A more consistent approach for errors perceived as high risk (critical) could prove beneficial for healthcare organisations such as the NHS. Designing CDS to interrupt the prescriber for critical errors has certainly been recommended as a strategy for improving the utilisation of the software.37 This is supported by the findings from this study that showed the presence of interruptive CDS that had the biggest impact on the rate of error.

Since the design of inpatient drug charts has been suggested as a primary cause of medication errors in UK hospitals,9 adopting a standard for high-risk errors has the potential to improve clinical outcomes by reducing variations in prescribing practice,38 which may be particularly beneficial in healthcare systems where staff move between hospitals as part of their training. The findings from this study demonstrate a significant opportunity for system optimisation and the adoption of a more consistent approach to this. Given the risk rating of the errors, this could result in real improvements in prescribing safety.

Strengths and limitations

The eDelphi that formed the basis of the data capture was conducted in 2012 and was based on the UK hospital setting. As new therapeutic agents are introduced and older ones go out of favour, the likelihood scores for their occurrence in clinical practice may well adjust, possibly to the point that they would no longer qualify for inclusion in the list of high-risk errors, according to our methodology. As evidence of this, when the eDelphi was repeated in 2015, 126 indicators of error were identified by 35 experts, 80 of which were from the original eDelphi and 46 were new.39 Errors involving direct oral anticoagulants were included for the first time, reflective of the change in practice for prescribing these drugs. This demonstrates the importance of ongoing review of CDS and optimisation to ensure systems reflect current practice. iMPACT has now been designed as a web-based tool for NHS hospitals to capture data on prescribing error rates specific to these revised 126 indicators.39 Since the eDelphi and this study were conducted in the UK, the findings may not be transferable to hospitals in other countries.

This multisite study was conducted in three hospitals in England. A strength of this study is the number of prescriptions reviewed preimplementation and postimplementation of CPOE. Data collection was entirely dependent on when hospitals were due to go live, and as such the study was staggered over a period of 18 months. However, this was not believed to impact on the data captured, since sites were not aware of the other hospitals participating in the study and so could not discuss the process or findings. The research team did not influence the decision-making at each hospital regarding the levels of CDS, which was important for gaining a true representation of a ‘real-world’ CPOE implementation. To our knowledge, there were no other major transformations in the delivery of the service to facilitate the medication process at any of the hospitals during the study period. As such, the introduction of CPOE is most likely responsible for the outcomes observed.

The iMPACT tool facilitated a standardised approach to data collection, allowing for a direct comparison of error rates. The tool was tested for usability and interpretation by a pharmacist who was independent from the research group. However, the tool was not assessed for its sensitivity or specificity in identifying opportunities for error and errors that had occurred.

A convenience sampling approach was used to select patients for inclusion in the study. As a result, the selection of patients may not be fully representative of the overall general medicine inpatient population in a setting. Ideally, it would have been preferable to standardise patient selection across the sites based on the day of the week and time of day, and by randomising the patients selected for inclusion. However, this was not logistically possible in this study, owing to the variation in the time of CPOE implementation across sites. Nevertheless, this approach could be used in the future when sites have all achieved maturity. In order to obtain a snapshot of prescribing that was reflective of the majority of the year, sites were asked to avoid conducting the review when prescribers were more likely to be unfamiliar with the paper or electronic prescribing processes. Information relating to the ward and prescriber was not recorded on iMPACT. This ensured that any patient data collected (ie, age) would remain non-identifiable and adhered to the information governance processes at each hospital site. In view of this, it was not possible to account for the impact of clustering by these factors in the analysis.

CPOE and CDS have been associated with new error types,40 and this has been shown to be the case at two of the study sites that participated in this research.29 Identifying these errors is important in the ongoing monitoring and optimisation of systems to minimise the risks of the technology to patient safety. However, in view of our approach to capture data on the occurrence of defined high-risk prescribing errors, investigating the specifics of any new errors was outside the scope of the study. This would require analysis of all prescription data generated, along with case note review, as well as the monitoring of local patient safety incident reports. This approach could be adopted alongside the use of iMPACT in future preimplementation/postimplementation studies to provide a more comprehensive overview of the impact of the technology on error.

Conclusions

Our novel approach to standardise the capture of defined prescribing errors has enabled error rates to be directly compared between study sites. Findings should however be interpreted with caution given the pre-post design. Implementation of CPOE with integrated CDS was associated with significant reductions in the rate of high-risk prescribing errors overall, with these overall findings having been driven by two of the sites. The implementation of CDS was found to vary significantly between the hospitals investigated, both in relation to the error type targeted and the level of alert presented to the prescriber. Sites with the same system in use generated very different results, indicating that outcomes are influenced by system capability and crucially by the manner of implementation, including the decision-making process during local configuration. This emphasises the importance of considering sociotechnical factors when implementing CPOE/CDS technology. Our methodological approach should help organisations to identify significant opportunities for system optimisation, and our findings can be used to guide optimal configuration of CDS for high-risk errors amendable to such software, in order to promote consistent practice and to benefit patient care.