When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record.

BACKGROUND: Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs.
METHODS: We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set.
RESULTS: There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%-18%) and median medication retraction rate was 4% (IQR 2%-10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte.
CONCLUSION: Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Billions of dollars have been invested into electronic health records (EHRs) with the promise of improving quality by increasing workflow efficiency and reducing errors and unwanted care variability, but the effects have been mixed and not well understood.1 2 Clinical decision support (CDS) refers to tools such as computerised alerts and condition-specific order sets that are intended to intelligently filter information to the clinician to enhance workflow and decision-making. The ‘CDS Five Rights’ is a framework to guide the appropriate design of CDS to communicate the right information to the right person in the right CDS intervention format through the right channel at the right time in workflow.3 Nevertheless, CDS often falls short of these goals in practice, in part due to the difficulty in anticipating how the complex environment of clinical medicine affects clinician workflow. Poorly designed CDS may in fact hinder care delivery by creating new unintended downstream hazards due to factors such as poor user interface design and introduction of new complexities in workflow.4–6 For example, many CDS tools face issues such as alert fatigue with high override rates7 8 or automation bias where clinicians over-accept computer-generated output in EHRs “as a heuristic replacement of vigilant information seeking and processing” even if the recommendations are inappropriate.9 Purposeful and safer CDS design requires that we understand the gaps between the expectations and realities of how CDS impacts clinician workflow in the real-world setting.

Order sets are collections of clinically related items grouped together for a wide array of specified clinical scenarios and constitute a significant component of EHR order entry.10 11 Order sets intend to help clinicians more effectively access appropriate items compared with individual ‘a la carte’ order entry, thereby reducing undesirable care variability12 and improving adherence to evidence-based practices.13–16 The extent to which order sets actually support clinician workflow is not well understood. The large number of order sets that healthcare institutions have to build and manage make it difficult to systematically evaluate whether the content and design for each order set optimally aligns with clinician needs.17 Poorly built order sets with outdated content can clutter the ordering interface, create distractions or even prompt clinicians to order unnecessary items due to automation bias.18 Although federal requirements exist for regularly scheduled order set reviews,19 institutions typically rely on a ‘top down’ approach of employing small groups of clinicians to curate order sets based on clinical guidelines, institutional policies and expert opinion with limited quantitative insight into how the order sets are actually being used by clinicians.20 Consequences of suboptimally designed order sets may remain unnoticed, perhaps not until an adverse event serious enough to warrant institutional attention.21 Continuous improvement in the quality and safety of health information technology design requires an understanding of how tools like order sets are used in real-world settings.22

We can gain unique insights into end-user experience through implicitly crowdsourced experience derived from data in the EHR audit trail. EHR systems log the timestamps of all transactions that can reflect in situ clinician behaviour without the biases associated with purposefully collected user feedback.23 This type of data has been used to analyse clinician practice patterns in order to derive clinical pathways,24 create recommender systems that suggest commonly entered orders for certain clinical scenarios25–27 and detect incorrect orders in the EHR.28–30 In our study, we seek to analyse order set usage at an academic medical centre to assess alignment with clinician needs. We focus on examples of poor alignment such as bloated order sets, whose items are rarely used or of limited use, as metrics for order set optimisation.31 We assess potential shortcomings that interfere with workflow, such as poorly accessible order set items that prompt users to instead order them a la carte, or the risk of automation bias that prompts clinicians to order items in error. We specifically look for four indicators in the EHR: (1) infrequent ordering of order set items, (2) rapid retraction of order set items, (3) additional a la carte ordering of items not listed in an order set and (4) a la carte ordering of items despite being listed in an order set.

Methods

The following terminology will be used throughout this paper: an ‘item’ is something that can be ordered in the EHR (eg, a medication or laboratory test), an ‘order set item’ refers to an item listed in a specific order set (eg, the same medication in two different order sets would be considered two different order set items), an ‘order’ is a single transaction of ordering an item, an ‘order set order’ is an order originating from an order set, an ‘a la carte order’ is an order that did not come from an order set and an ‘order set use’ refers to an instance of an order set being used to order at least one item. We mined data from all inpatient orders of medication, laboratory, imaging and blood product items along with all order set uses in the EHR from 1 January 2014 to 15 July 2017 at a 613-bed tertiary care teaching hospital. Laboratory tests ordered together in a panel (eg, complete blood count) were counted as one individual order. Medications were normalised by their formulation and route of administration. During this time, order sets were created and maintained by a standing order set review committee that met every 3 months to review one to two order sets at a time using standards derived from the Institute for Safe Medication Practices32 as well as input from clinicians and clinical informaticians at our institution. An order set was typically reviewed once every 3 years. All clinicians practising in the inpatient setting had access to the entire list of order sets through the EHR, although training and awareness of each order set varied by provider specialty. All data were de-identified and extracted via our institution’s clinical data warehouse.33 We then assessed the following indicators of poor alignment:

Indicator 1

Infrequently ordered items were identified by ranking the ordering rate of order set items, which equals the number of orders for a given item divided by the respective number of order set uses.

Indicator 2

Rapidly retracted orders were defined as orders that were discontinued within 30 min, based on prior studies establishing these as surrogates for erroneous orders.28 30 Only medication orders were included in this analysis due to availability of data at the time of this study. For a given item, the order set retraction rate equals the number of retracted orders from a given order set divided by the total number of orders of the item from the order set, while the a la carte retraction rate equals the number of retracted a la carte orders divided by the total number of a la carte orders of the item. The relative risk of retraction of an order set item was calculated by dividing the order set retraction rate by the a la carte retraction rate.

Indicator 3

We employed an itemset mining approach25 26 34 to look for a la carte orders co-occurring within 10 min of a given order set use. These ‘additional a la carte orders’ represent orders of items that were not part of the order set, but were deemed necessary by the clinician at the point of care when using the order set. We chose our window to be 10 min a priori based on our experience using order sets in the clinical setting. For each order set, we identified such items with orders that co-occurred at a rate greater than the median ordering rate of the items in the order set. The co-occurring rate of an item–order set pair was calculated by dividing the number of times a given item co-occurred at least once by the total number of order set uses for that given order set. We also filtered for orders of items that were significantly more likely to co-occur with a given order set than with all order sets. We excluded the item ‘point of care glucose’ from the filtered results because it is a non-specific item commonly ordered a la carte by nurses.

Indicator 4

We identified a la carte orders co-occurring within 10 min of a given order set use that were of items included in that order set. These ‘a la carte over order set’ orders were of items ordered a la carte despite being listed in a given order set. We filtered for items that were ordered a la carte more frequently than the ordering rate of that item from the order set and calculated the a la carte to order set ordering rate ratios for each item–order set pair.

Tests for significance for all analyses were performed with Fisher’s exact tests at Bonferroni-corrected p-value thresholds.35

Case study

We conducted focus group interviews with five members of our institution’s inpatient order set review committee that explored how the indicators derived from this study could be incorporated into the order set review process. We then presented an example of our analysis for the inpatient congestive heart failure (CHF) order set to the committee while it was undergoing review as part of an effort to improve adherence to an evidence-based inpatient CHF clinical pathway.36

Results

Between 1 January 2014 and 15 July 2017, a total of 17 072 459 medication, laboratory, imaging and blood product orders were generated for 77 421 inpatient encounters. There were 11 108 distinct items, ranging from four blood product items to 8368 medication items (table 1). Moreover, 1785 of these items were available in the 545 order sets used during this period in the form of 11 762 distinct order set items, which produced 2 449 208 medication, laboratory, imaging and blood product orders. Among these order set orders, medications were the most commonly ordered items (59%) while imaging orders were the rarest (2%).

Table 1

Breakdown of the number of items and orders by item type

	All orders		Orders from order sets
	Number of items	Number of orders	Number of items	Number of orders
Medication	8368	4 715 232 (28%)	1151	1 441 338 (59%)
Blood product	4	413 415 (2%)	4	112 578 (5%)
Laboratory	1599	11 008 302 (64%)	529	849 886 (34%)
Imaging	1137	935 510 (5%)	101	45 406 (2%)
Total	11 108	17 072 459	1785	2 449 208

Laboratory items were the most commonly ordered among all orders (64%) while medication items were the most commonly ordered from order sets (59%).

Indicator 1: infrequently ordered order set items

We first examined the distribution of ordering rates of all 11 762 order set items. The ordering rates varied greatly, ranging from 0.001% to 100% with a median of 4.1% (IQR 0.6%–18%). Laboratory items tended to have higher ordering rates (median 11.5%, IQR 2.5%–33.3%) than medication (median 2.3%, IQR 0.4%–1.1%), imaging (median 4.7%, IQR 0.8%–14.6%) and blood product items (median 2.6%, IQR 0.3%–24.5%) (p<0.001; online supplementary material 1). The order set items with the highest and lowest ordering rates are shown in table 2.

Table 2

Order set items with the highest and lowest ordering rates, excluding uncommon order sets used fewer than 100 times

(a) Order set items with highest ordering rates
Order set	Item	Order count	Order set count	Ordering rate (%)
Blood culture	Blood culture (aerobic and anaerobic bottles)	3848	3848	100
Blood culture	Blood culture (2 aerobic bottles)	3848	3848	100
Oral steroid tapers	Dexamethasone PO	1405	1405	100
Lung transplant post op	Glucose by metre	693	693	100
Thoracentesis post procedure	Fluid culture and Gram stain	647	647	100
Lumbar puncture	Cell count and differential, CSF	367	367	100
IV steroid tapers	Dexamethasone IV	236	236	100
Nursing triage, altered mental status	Comprehensive metabolic panel	149	149	100
Arthrocentesis post procedure	Fluid culture and Gram stain	132	132	100
Arthrocentesis post procedure	Crystal analysis, synovial fluid	132	132	100

CSF, cerebrospinal fluid; IV, Intravenous; PACU, Post Anaesthesia Care Unit;PO, By mouth.

Indicator 2: rapidly retracted orders from order sets

We next focused on the retraction of medication items, as defined by medications ordered and discontinued within 30 min. At least one medication order was retracted in 13 769 (18%) patient encounters. Among all 1 441 338 medication orders from order sets, 32 072 (2.2%) were retracted, which was lower than the retraction rate for the 3 273 894 a la carte medication orders (3.9%) (p<0.001). Among the 2606 order set medication items that produced at least one retracted order, however, the order retraction rates varied greatly (range of 0.1% to 100% with a median of 4% (IQR 2%–10%)). Specifically, we evaluated whether certain order set medication items were more likely to be retracted than if they were ordered a la carte. One hundred forty-three (5%) order set medication items were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte (at a p<2×10−5 Bonferroni-corrected threshold). The order set medication items with the highest and lowest relative risks of retraction are shown in table 3.

Table 3

Order set items with the highest and lowest risks of order retraction relative to when the same items were ordered a la carte, excluding uncommon order sets used fewer than 100 times

(a) Order set items with the highest risk of retraction relative to a la carte
Order set	Item	Retraction count (order set)	Retraction rate (order set) (%)	Retraction count (a la carte)	Retraction rate (a la carte) (%)	Retraction relative risk
Vascular ICU	Esmolol IV	3	38	1	0.9	42.8
Neurology general ward admit	Atorvastatin PO	2	40	60	1.6	25.0
ED abdominal pain	Pantoprazole IV	2	100	688	5.3	18.8
Nursing triage pain management protocol	Ondansetron PO	2	67	153	3.6	18.6
NSTEMI admit	Atorvastatin PO	3	33	96	1.9	17.8
Pneumonia	Ceftriaxone IV	3	75	13	4.8	15.7
Medicine general admit	Warfarin PO	5	38	96	1.9	15.3
ICU surgery/trauma admit	Pantoprazole PO	5	23	624	2	11.6
Lung, heart–lung transplant post op	Piperacillin–tazobactam IV	3	60	466	5.6	10.6
Cardiac surgery admission	Polyethylene glycol	7	41	89	4	10.4

The retraction relative risk=(retraction rate from order set)/(retraction rate a la carte). Items in 3a represent those that are particularly poorly aligned with clinician workflow given how much more likely they are ordered in error from the corresponding order set, while items in 3b appear to be particularly well aligned.

ED, emergency department; ICU, intensive care unit; IV, intravenous; NSTEMI, non-ST elevation myocardial infarction; PO, by mouth.

Order set items with the lowest relative risks, such as lidocaine from the Bronchoscopy order set and heparin from the Haemodialysis order set, tended to be medications part of well-defined protocols (eg, lidocaine and heparin are routinely given at the onset of bronchoscopies and haemodialysis sessions for local anaesthesia and prevention of clotting in the dialysis circuit, respectively) in order sets that were specifically built for those protocols. Conversely, items with the highest relative risks, such as intravenous esmolol from the Vascular ICU order set and atorvastatin PO from the Neurology General Ward Admit order set tended to be part of larger order sets with broader clinical use cases. These medications may be relevant for some, but not all clinical situations that their respective order sets are used for. The distribution of relative risks of all order set medication items are shown in online supplementary material 1. There was a negative association between ordering rate and order retraction rate. Order set medication items with ordering rates in the lowest quartile were approximately twice as likely to be retracted than those in the second to fourth quartiles (p<0.001) (figure 1).

Figure 1

Relative risk of order retraction of order set items by ordering rate. Items with higher ordering rates were less likely to be retracted than those in the lowest ordering rate quartile (p<0.001).

Indicator 3: additional a la carte orders of items not listed in order set

When using an order set, clinicians may order additional items a la carte if they are also needed at the time but not available in the order set. We looked for such items that were additionally ordered a la carte within 10 min of an order set use more often than the median ordering rate of items in the corresponding order set. Of the total 545 order sets, 214 (39%) had such an item commonly added a la carte, with a median of 4 (IQR 1–12) distinct additional a la carte items per order set. The median ordering rate of these additional items was 2.3% (IQR 1%–6%). The items with the highest additional a la carte ordering rates are listed with their corresponding order sets in table 4a.

Table 4

(a) Order set items with the highest additional a la carte ordering rates, excluding uncommon order sets used fewer than 100 times (these are items that clinicians frequently needed at the time of order set use, but were not included in the order set build), and (b) order set items with the highest a la carte to order set ordering ratios, excluding order sets used fewer than 100 times (these are items included in order sets, but were frequently skipped over by clinicians and instead more likely to instead be ordered a la carte right after order set use)

(a) Additional a la carte items
Order set	Additional a la carte item	A la carte order count	A la carte ordering rate (%)
Kidney/pancreas transplant rejection	Diphenhydramine PO	35	28
Pneumonia	CBC without differential	49	26
Pneumonia	Basic metabolic panel	44	23
Hospice admission	Metoclopramide IV	24	22
ICU VAD post op	Lactate (whole blood)	350	22
Bronchoscopy	Cytology	493	22
Lumbar puncture	Cytology	41	21
Pneumonia	Ondansetron IV	38	20
COPD	Ondansetron IV	43	20
Nephrectomy post op	Ionised calcium	57	17

Only items with significant ordering ratios at a Bonferroni-corrected threshold of p<2.5×10−5 are included.

CBC, complete blood count; COPD, chronic obstructive pulmonary disease; ICU, intensive care unit; IV, intravenous; PACU, post anaesthesia care unit; PO, by mouth; VAD, ventricular assist device.

Indicator 4: a la carte ordering of items despite being listed in order set

When using order sets, clinicians may instead order a listed order set item a la carte because they do not see the item in the order set or purposely choose a la carte ordering because it is more convenient. We quantified how often this phenomenon occurs by looking for items that were ordered a la carte within 10 min of an order set use more often than directly from the order set. Of the 545 order sets, 243 (45%) contained at least one such ‘a la carte over order set’ item, with a median of 4 (IQR 2–9) distinct items per order set. The median ordering rate of these items was 1% (IQR 0.2%–3%). We further identified a subset of order set items for which the a la carte ordering rate was significantly higher than the order set ordering rate using a Bonferroni-corrected threshold of p value <2.5×10−5. In this subset, there were 73 order sets, a median of two distinct items per order set (IQR 1–6), with a median a la carte ordering rate of 2% among all the items. The median ratio of the a la carte ordering rate over order set ordering rate of these items was 8.9 (IQR 3.9–21.5). The order set items with the highest a la carte to order set ordering rate ratios are listed in table 4b.

Case study: assessing the inpatient CHF order set

We present several examples of how data-driven insights derived from the above ordering patterns were used to guide the review of the CHF order set by our institution’s order set committee. This order set was built to support both the floor admission process as well as medical decision-making around CHF management. Sections and subsections of the order set are listed in online supplementary material 1.

The committee was unanimous in agreeing that all four indicators of workflow alignment provided by our study added value beyond expert opinion for order set review. The following findings were unexpected and of particular interest to the order set committee: (1) the items with the highest ordering rates were general hospital admission laboratories (eg, basic metabolic panel (56%), complete blood count (52%)) while most CHF-specific items were relatively infrequently ordered (furosemide (10%), echocardiogram (10%) and NT-proBNP (9%) were the only CHF-specific items with ordering rates above 5%); (2) standard CHF medications such as carvedilol and lisinopril had very low ordering rates below 1% but relatively high retraction rates above 5% from the order set; (3) oral aspirin, which was not included in the order set, was added as an a la carte item 8% of the time the order set was used; and (4) most of the CHF medications listed in the order set were more likely to be ordered a la carte despite being available as order set items. The order set committee subsequently added oral aspirin to the order set and are now discussing ways to further separate CHF-specific medications from general inpatient admission items in the order set. The full list of indicators for the CHF order set items are in online supplementary material 1.

Discussion

We present quantitative evidence from EHR usage data that order sets often do not align with what clinicians need at the point of care. Misalignment was reflected in the infrequent ordering of order set items, retraction of medication orders from order sets and frequent a la carte ordering of additional items regardless of whether they were present in the order set. This gap between the assumptions and realities around order set use reflects an opportunity to use real-world usage patterns for optimisation.

We found significant variability in alignment across order sets. Certain order sets appeared to meet clinician needs well. These order sets contained items that were frequently ordered with low retraction rates, with some items having even lower retraction rates when ordered from the order set than if they were ordered a la carte, suggesting the order set content is well suited to the corresponding workflow. Yet, half of all order set items were rarely ordered (less than 4% of the time). This suggests that many order sets may be bloated with low yield items. These rarely used items were also more likely to be rapidly retracted, indicating they may predispose users to order items by mistake. We identified multiple medications that were more likely to be retracted from certain order sets than when ordered a la carte, suggesting automation bias when clinicians are exposed to these medication–order set combinations. Moving beyond anecdotes of order sets promoting wrong medication orders,21 our analysis provides more explicit evidence of such phenomenon.

Many order sets may also be missing items that users need. Of the order sets at our institution, 45% were associated with at least one additional a la carte item that was ordered more frequently than half of the items actually included in the order set. Table 3a illustrates many clinically reasonable additions to their corresponding order sets that were not included in the initial build. This is likely because their relevance was not immediately obvious to the order set authoring committee. For example, ondansetron is an antiemetic medication that does not directly treat pneumonia or chronic obstructive pulmonary disease (COPD) and thus was not included in those order sets. However, we see that clinicians were often adding ondansetron as a la carte orders, most likely done to treat nausea, a common symptom among patients presenting with acute pneumonia or COPD. Similarly, we identified aspirin as a common additional a la carte item for the CHF order set. While aspirin is not specifically a CHF medication, it is commonly appropriate for patients with CHF due to comorbid coronary artery disease. This was compelling enough for our local order set committee to add aspirin as an option to the CHF order set. Other examples from table 3a, such as the basic metabolic panel and complete blood count without differential, were most likely added because they are typically preferred by clinicians over the comprehensive metabolic panel and complete blood count with differential, two items that are in the pneumonia order set, because the latter panels contain additional tests that are usually unnecessary for pneumonia management.

These indicators need to be considered in the context of the clinical use cases specific to each order set item. For example, while a low ordering rate may indicate certain order set items are no longer relevant and should be removed, this may not be true for other infrequent items that are purposely included for rare but serious clinical situations that necessitate prompt action. Similarly, items added a la carte at a high rate should not necessarily be included in the preceding order set if they are not topically related. These trade-offs should be carefully evaluated with the appropriate stakeholders familiar with each unique clinical workflow.

Our findings also suggest that order sets may sometimes hinder order entry. We found clinicians skipping over certain order set items, only to order them a la carte immediately after. This phenomenon may reflect shortcomings in the order set build itself rather than the clinical appropriateness of the item. For example, either clinicians did not find the item in the order set or they purposely preferred to order the item a la carte because it was more convenient. The degree of preference for a la carte ordering is striking among the order set items in table 3b. For example, of the 17 250 times the orthopaedics inpatient joint replacement order set was used, intravenous hydromorphone was ordered directly off of the order set only once (0.005%), but ordered a la carte within 10 min of order set use 1043 times (6%). A closer inspection revealed that accessing intravenous hydromorphone from the order set required multiple clicks by first checking the ‘severe pain’ box in the ‘opioid analgesics’ section, which only then displayed the option to order the medication. Our results suggest that while this build feature does deter ordering of intravenous hydromorphone from the order set, clinicians develop a workaround and order it a la carte anyway.

This study raises the question of whether order sets are the right channel to deliver otherwise clinically appropriate content at the right time in workflow. For example, we found that the medications specific to an intended CHF clinical pathway were rarely ordered from the CHF order set. One explanation is that this order set also includes standard admission orders that are pre-selected when the order set is launched. Grouping these items with CHF medications may not be effective because clinical decision-making around administering disease-specific medications often occurs at a later time in workflow after processing standard admission protocols. Further, our EHR design precludes the user from viewing data such as vitals, laboratory results and home medications simultaneously while selecting items from order sets. This design limitation may explain why certain order set items, such as those featured in table 4b, tend to instead be ordered as a la carte items. Reconciled home medications are ordered as a la carte orders in our EHR, which may also explain why common outpatient CHF medications were rarely ordered from the CHF order set. We observe that the most commonly ordered items from the CHF order set were non–CHF-specific standard admission orders, suggesting that the CHF order set was in fact mostly used to access generic admission orders rather than items specific to a CHF clinical pathway.

While several recent studies have proposed different methods for data-driven order set construction using EHR usage data,25 27 31 our study applies additional methodologies to reveal historical population-level order set usage trends that inform several design considerations. Frequently ordered, rarely retracted items tend to be part of order sets for single, well-defined clinical processes with protocolised workflows. Order sets constructed for individual clinical protocols may be particularly well suited for clinician workflow and reduce practice variability. Conversely, it is less clear whether order sets that are intended to support multiple clinical processes involving more complex medical decision-making successfully achieve that purpose, especially if these clinical processes are meant to take place over different points in time. Items from disease-specific addendums to admission order sets, such as antibiotics for pneumonia, were rarely ordered and in fact were often quickly retracted if ordered. Our study suggests that while inpatient admission order sets may work well for standard general admission protocols, embedding additional condition-specific items may not be effective in supporting management of those conditions, especially if multiple conditions associated with different types of patients and workflows are represented in the order set. Alternate tools that enable more dynamic support of clinical workflow and decision-making needs that evolve over time may be required.

There are several limitations of this study. Our analysis was confined to inpatient orders from a single academic medical centre, so the order set usage patterns we observed may not generalise to other practice settings, although the four indicators we propose could be reproduced to evaluate any local order set usage. Changes in order set design and policies may confound metrics such as ordering rates of items, which were calculated with the assumption that all items were available in the order set through the entire study period. Given the order set review schedule at our institution, however, we found minimal changes made to most order sets during our study period. We were not able to distinguish between different ordering providers in our analysis, although it is common practice at our institution for only a single primary inpatient provider to enter orders for a given patient. Our relative risk analysis of order retraction between order set and non-order set items was unadjusted for potential confounders that could independently contribute to medication retraction. Further, we were unable to capture which order set items were pre-checked, which could provide further insight in assessing order set design. Our metrics derived from the EHR audit trail reflect observed clinician behaviour but are only indirect indicators of what clinicians were thinking. For example, a retracted order within 30 min may not necessarily mean that the item was initially ordered in error; there could be other reasons, such as a change in clinical condition, that warranted the order retraction. While these types of limitations are inherent in computational ethnographic studies, the broad findings from these metrics remain compelling with a large number of low bias observations captured passively from the EHR.

Conclusion

Real-world practice patterns extracted from the EHR audit trail demonstrate that order sets often do not align with what clinicians need at the point of care. Quantitative insights from electronic records of clinical orders can inform how clinical decision support like order sets can be optimised or replaced with other potentially more appropriate technology to facilitate clinician workflow and improve care delivery.