Structured override reasons for drug-drug interaction alerts in electronic health records.

OBJECTIVE: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records.
MATERIALS AND METHODS: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices.
RESULTS: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: "will monitor or take precautions," "not clinically significant," and "benefit outweighs risk." DISCUSSION: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved.
CONCLUSIONS: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

INTRODUCTION

Adverse drug events (ADEs) are a type of patient harm that commonly occur in both inpatient and outpatient settings. ADEs resulting from medication errors during prescribing, dispensing, or administration are considered preventable ADEs. Common types of medication errors include incorrect dosing or frequency, prescribing medications to which a patient is allergic, prescribing nephrotoxic drugs in patients with decreased renal function, and prescribing multiple drugs that have undesirable interactions with each other.

One strategy for preventing medication errors is clinical decision support (CDS) alerts, embedded within a computerized provider order entry (CPOE) system. Systematic reviews by Kaushal et al and Kuperman et al found that medication-related CDS alerts have potential to reduce the rates of ADEs. However, medication-related CDS alerts are frequently overridden,, sometimes with good cause, as adverse events rarely occur when prescribers override even the highest-level alerts., Historically, the rationale for high override rates was alert fatigue and poor usability, but recent research suggests a more complex picture, including inaccurate warnings and incorrect judgments by the prescriber.

One of the most common types of medication alerting is drug-drug interaction (DDI) checking, which the United States meaningful use incentive program required for adoption of electronic health records (EHRs)., The evidence base for including DDIs as a required EHR functionality was not as strong as that for other types of CDS, such as renal dose adjustment. However, DDI checking is simple to implement, as a variety of medication knowledge bases are available with data on potential drug interactions, typically stratified by severity. In a prior study, we found that the clinical content of DDI alerts varies considerably across institutions and EHRs, even for a set of alerts identified as very high priority by a national consensus group.

One controversy regarding CDS alerts is whether to employ hard stops—alerts a user cannot bypass. Hard stops are rare in practice, even for high-severity alerts. Instead, users can typically either accept an alert or override it. When users override an alert, there is often opportunity to provide a reason, typically drawn from a coded list. Users may also be able to supply a free-text reason instead of, or in addition to, a coded reason.

Literature on the rationale for collecting override reasons is scant; however, some common reasons include:

Documenting the ordering provider’s awareness of the potential for harm and rationale for overriding the alert; this contemporaneous documentation may be of use in the case that harm actually occurs or there is an allegation of malpractice

Communicating the provider’s awareness of the potential interaction downstream, such as to the pharmacy. Particularly in the inpatient setting, this may reduce the need for the pharmacist to call the ordering provider to verify the order. Override reasons rarely transmit to outpatient pharmacies, so this is less likely to be helpful in outpatient settings

Trying to influence the behavior of the provider by forcing them to justify their decision to override an alert

Collecting data about what alerts prescribers commonly override, and why, to support improvements in knowledge bases and alerting systems

In this study, we collected and analyzed data on DDI override reasons from organizations across the United States, with a goal of understanding available override reasons and how prescribers use them.

MATERIALS AND METHODS

Starting with the sample of organizations from our prior paper on DDIs, we identified sites across the country that used an EHR for the entirety of 2016, had active DDI alerts and allowed users to provide a coded reason for overriding alerts (whether optional or mandatory). We invited those organizations to participate in this new study by providing data on their DDI alert overrides.

Each participating site queried its data warehouse to find all DDI alerts shown during 2016 and determined alert acceptance and override rates. For overridden alerts, each site also determined whether an override reason was given and, if so, what it was. Sites aggregated the data to provide counts on the total number of DDI alerts shown to users, the number of alerts that were overridden, and the number of times prescribers chose each coded DDI override reason. Some sites also allowed free-text override reasons, but we did not collect these, as our prior work suggested that free-text entries frequently contained protected health information and sometimes even passwords. Further, several study sites were willing to participate only if the data sharing was limited to deidentified aggregate data.

Researchers at Brigham and Women’s Hospital collected, combined, and analyzed the data from each site. After collecting all of the override reasons from each site (shown in Supplementary Appendix) we conducted a multistage iterative card sort method to categorize the override reasons. We have previously used this method of empirical categorization in other studies of CDS. In some cases, reasons were not clear, and we sought clarification from the contributing site as needed. We also assessed whether each override reason was relevant to DDIs, as some sites offered override reasons that were clearly not relevant to DDIs, typically because they used a single list of override reasons for all alert types.

The Partners HealthCare Human Subjects Committee reviewed and approved this study.

RESULTS

Ten sites participated in the study. They are shown in Table 1.

Table 1.

Participating sites, including location and electronic health record used

Organization	Location	Electronic Health Record
Children's Hospital of Philadelphia	Philadelphia, Pennsylvania	Epic
Holy Spirit Hospital	Camp Hill, Pennsylvania	Allscripts (Sunrise)
Kaiser Permanente Northwest	Portland, Oregin	Epic
Memorial Hermann	Houston, Texas	Cerner
Partners HealthCare	Boston, Massachusetts	Epic
University of Alabama at Birmingham	Birmingham, Alabama	Cerner
UC San Diego Health	San Diego, California	Epic
University of Illinois at Chicago	Chicago, Illinois	Cerner
University of Texas Faculty Physicians	Houston, Texas	Allscripts (Enterprise)
Weill Cornell Medicine	New York, New York	Epic

Table 2 shows overall override statistics by site. Table 2 and subsequent tables present each site’s data by a number instead of name, and the order of the sites is shuffled to preserve anonymity. Consistent with past literature, the overall override rate is very high, at 91%. Two sites required override reasons for all DDI alerts. The other sites made override reasons optional, and users provided them between 8% and 82% of the time. Most sites offered fewer than 10 unique, coded override reason options, but one site offered 100 choices.

Table 2.

Override statistics by site, including anonymized site number and details of alert firing and override rates

Site	Alerts	Alerts Overridden	Overrides With Reasons	Unique Override Reasons
1	521 197	495 350 (95)	220 399 (44)	7
2	133 649	122 455 (92)	14 718 (129)	100
3	37 333	24 089 (65)	3979 (17)	6
4	59 295	56 532 (95)	10 329 (18)	6
5	122 659	96 775 (79)	25 692 (27)	6
6	1 434 937	1 335 764 (93)	1 335 764 (100)a	16
7	311 657	273 203 (88)	21 792 (8)	3
8	164 720	141 827 (86)	114 740 (81)	9
9	206 765	186 180 (90)	186 180 (100)a	21
10	104 136	93 610 (90)	77 213 (82)	3
Total	3 096 348	2 825 785 (91)	2 010 806 (71)	177

Values are n (%) unless otherwise indicated.

aOverride reasons were required at these 2 sites.

In total, there were 177 override reasons offered across the 10 sites. Table 3 shows the results of the categorization exercise, with 12 categories identified. Three categories accounted for 78% of all overrides: “will monitor or take precautions,” “not clinically significant,” and “benefit outweighs risk.” The table includes representative verbatim examples from the sites’ CPOE systems for each category, as well as the number of overrides in our sample in each category, the reasons unique to each category, and the number of sites offering at least 1 coded override reason in each category.

Table 3.

Categories of DDI reasons

Category	Examples (Verbatim)	Overrides	Unique Reasons	Sites With Reason in Category
Will monitor or take precautions	Will monitor patient for interaction Interaction noted Will take precautions Aware of interaction Will follow/monitor	795 990	7	5
Not clinically significant	This is not clinically significant This alert is NOT useful Disagree with stated interactions(s) Inaccurate warning	411 782	22	9
Benefit outweighs risk	Potential benefit outweighs risk No good alternative	365 734	14	9
Patient tolerated previously	Patient tolerated before Medication tolerated by patient Current therapy	117 923	5	5
Dose adjusted	Dosage appropriately adjusted Dosing interval appropriately adjusted Have adjusted or will adjust dose	101 593	4	3
Not related to DDIs	Patient does not have stated allergy New active cases in same unit Barrett's esophagus Expect improved renal function Clearance wrong due to wrong ht/length Patient is NOT pregnant	84 911	89	7
Not ordering a medication	Chart review: no action Entering an historical medication Patient expired	51 570	6	2
Alert is not the recipient’s responsibility	Treatment plan requirement Defer to primary physician Defer to pharmacist Spoke with transplant service	52 886	9	4
Agreement, though alert was overridden	Order this agent, will stop other drug RPh reviewed-acted	11 969	4	3
See comments	See comments Other (free text)	8387	4	3
Order is urgent	Deferring to other priorities Emergency The drug is needed urgently	5220	11	4
Error in data	Error in data	2841	2	2

DDI: drug-drug interaction; RPh: registered pharmacist.

DISCUSSION

Considerable variation in reasons offered to providers is apparent from our data. Most sites offer relatively few override reasons, though some offer dozens, and 1 site offers 100. The most widely used category of override reason, “will monitor or take precautions,” is only available at half of the participating sites, and no category of reasons is offered at every site. This variation likely stems from a lack of national consensus or standards for override reasons, leaving each site and vendor to determine their own list.

Further, 7 of 10 sites offered users at least 1 coded override reason for DDI alerts that were clearly irrelevant to DDI alerting. For example, many sites allowed users to choose reasons like “patient does not have stated allergy,” “clearance wrong due to wrong ht/length,” or “patient is NOT pregnant,” which apply to other types of medication alerting, but not to DDIs. In discussions with sites, we found that many used the same list of override reasons for all categories of medication alerting (eg, allergy alerting, drug-disease interaction, pregnancy alerting, dose range checking, renal and hepatic dose adjustments). Some sites reported their EHR software does not allow different lists to be used for different alert types, while others simply had not configured different lists.

When the list is too long, instead of picking an accurate reason, users may simply pick a reason at random, and, if a reason is not mandated, users may not provide a reason at all. Prior studies have shown that when mandatory free-text reasons are required, users often enter a space or random characters to move past the screen., It may similarly be true that with a prespecified list, many users select the top item from the list or a random item so they can move on with the medication order. Thus, one caveat of our results is that users may not always have picked a reason they felt was accurate; indeed, this is certainly true, as seen by the override reasons that were not related to DDIs. Improving the list of override reasons to be shorter and more relevant to alerts may be useful for improving the accuracy of override reasons.

Several categories of override reasons merit special discussion:

Will monitor or take precautions: Some override reasons in this category were generic, such as “will monitor,” but others mentioned specific precautions. An example of this is cardiac monitoring when ordering multiple drugs with the potential to prolong the QT interval, or additional monitoring of the international normalized ratio (INR) when prescribing drugs interacting with warfarin. However, these precautions were typically not actionable (only a single site had created actionable precautions, and only for a small number of interactions), which means users must separately order the precaution, increasing the time required and the likelihood of omitting the precaution.

“Not clinically significant” and “benefit outweighs risk”: Based on the literature in the reported previously, users are frequently correct when they indicate that a potential interaction is not clinically significant or the benefit of the drug outweighs the risks. These overrides represent a data source for identifying alerts to potentially disable or improve by tailoring. They may also represent an important source of documentation that the ordering provider was aware of the interaction but concluded the benefits outweighed the risks.

Patient tolerated previously: Although prior tolerance is not a guarantee of ongoing safety, it is an important indicator. Finding prior tolerance can be difficult, as not all of a patient’s past medications may be documented in the EHR, but it may be possible in certain cases.

Dose adjusted: Changing the dose of 1 or both drugs can sometimes mitigate a DDI. Classically, warfarin interacts with a variety of antibiotics, including sulfamethoxazole, leading to an increase in the INR, indicating diminished coagulability and a higher risk of bleeding. One option is to avoid these antibiotics in patients taking warfarin, but it may also be appropriate to reduce empirically the dose of warfarin and monitor the INR more frequently.

Alert is not the recipient’s responsibility: This was a diverse and potentially troublesome category, which was most commonly used in situations where multiple people were partially responsible for an order. For example, users might report that the order was required by a pre-existing, institutionally approved treatment plan, or specified by the transplant team but entered by someone else. In other cases, users chose to “defer to primary physician” or “defer to pharmacist” even though they placed (and often, were ultimately responsible for) the order. In these cases, the CPOE system does not necessarily direct those deferrals to the indicated party to acknowledge acceptance of that responsibility, creating further risk. In a CPOE system, certainty about the party responsible for the order is needed.

Agreement, though alert was overridden: In this category, users override an alert, but indicate they nonetheless intend to follow it by, for example, discontinuing another drug. There is an inherent risk with this category, as users may fail to take the action they indicate.

Across all the categories described in Table 3, one theme recurred: there is great variation in DDI alert reasons. We found most were not actionable, some included reasons irrelevant to DDI alerting at all, some may delay entry of orders required for safe prescribing of the interacting drugs, and others lack clear communication. This variability impedes accurate data collection and reporting. From these data, it seems clear that, in addition to improving the quality of override reasons, alerts need to be more specific to the drug pair and to the patient. We believe healthcare organizations, in partnership with EHR vendors and medication knowledge–based vendors,, should create more specific, actionable alerts. An ideal alert would:

Be tiered appropriately by severity. For example, severe interactions may be interruptive (eg, clarithromycin and simvastatin, which can cause rhabdomyolysis and acute kidney disease) while less severe interactions could be passive (eg, levothyroxine and calcium, which may reduce levothyroxine absorption if the doses are not separated by time),, or not shown at all. Many overrides are given because the user considers the interaction to not be clinically significant—when appropriate, eliminating these alerts or reducing their severity should reduce the number of overrides.

Display, and take into account, relevant data. As a representative example, DDI alerts related to QT prolongation should display and consider the most recent corrected QT interval, while alerts related to warfarin should show the most recent INR. This could reduce “not clinically significant” and “benefit outweighs risk” overrides by suppressing alerts that, while potentially valuable when only looking at the pair of drugs involved, are no longer relevant once additional data is taken into account.

Allow the user to take specific actions to monitor and reduce the risk of harm directly from the alert notification window. In general, we believe alerting systems should be actionable., Consider the representative example of DDI alerts related to QT prolongation. These alerts should allow the user to order an electrocardiogram or cardiac telemetry, while alerts related to warfarin should allow the user to order more frequent monitoring of the INR or make appropriate empiric adjustments to the warfarin dose directly from the alert, when such adjustments are supported by evidence. If appropriate precautions are already in place, it would be ideal to suppress the alert; if not, the applicable precautions should display, and be orderable, on the alert screen. Users are already indicating that they will monitor or take follow-up actions by choosing these as override reasons; however, in the systems under study, none of these override reasons actually accomplish an action—they simply log the user’s intent.

Allow the user to cancel any new medications being ordered and also permit discontinuation of existing medications that are part of the interaction. Some systems under study did not allow users to discontinue existing medications in an interaction, so the user is forced to override the alert, indicate that they plan to discontinue the existing drug, and then subsequently remember to actually take that action.

Provide a small number of tailored, accurate override reasons. These reasons should be customized to the exact drug interaction in the alert and should cover all the categories identified in Table 3 that are relevant to DDIs.

Allow the user to document his or her reasoning, as desired, such as with a coded override reason, a free-text comment, or both. This documentation should be stored and readily visible in the medical record and made available to other providers and to CDS developers.

Communicate documented reasoning about DDIs to other members of the care team, such as the nurse or pharmacist. In cases where a user’s override indicates deferral to another party (eg, “defer to pharmacist” or “defer to primary physician”), the CPOE system should require the ordering provider to select the user responsible for dealing with the alert, and then require the responsible party to review and approve the order. Depending on clinical urgency and institutional policies and procedures, this approval might be done before the drug is dispensed.

Allow the user to provide feedback directly to the team responsible for maintaining the local EHR alerting system and the drug interaction knowledge base, so the knowledge base can be continuously improved. The team receiving this feedback should regularly monitor it and respond directly to users about the feedback they provide in a timely fashion.

Satisfying these recommendations may be resource-intensive. Organizations may want to prioritize the most commonly firing, most frequently overridden, and most risky interactions for this treatment, while using basic functionality for others. We believe this would be feasible if starting with a short list of highly actionable DDI pairs, such as the 7 previously described for adults and 19 for pediatric patients., Further, drug knowledge base vendors should enhance data in their knowledge bases to include alternatives, specific monitoring recommendations and relevant information. This information should be structured and coded (not just present in written monographs), and EHR vendors should be able to use this information to tailor alerts shown to users. Although there would be some cost to add this information to drug knowledge bases and maintain it, doing so at the knowledge base level would be more efficient (and likely more effective) than leaving the work to implementing clinical sites. Additionally, CDS implementers and knowledge base vendors should also focus resources on improving the specificity of DDI alerting more broadly—after all, producing fewer inaccurate alerts reduces the need to override alerts and provide override reasons. Groups such as Leapfrog will need to validate these shorter lists of actionable pairs so health systems are not incentivized to overalert by publicly reported test results.

Our study has several strengths. First, it has a broad sample of U.S.-based hospitals using CPOE with DDI alerting, using a range of the most common CPOE systems. Second, the sample of over 2 million recorded DDI reasons is substantial, allowing for robust analysis of frequency.

Our study also has some limitations. First, we did not analyze free-text override comments. Free-text reasons may contain other categories of overrides not seen in the coded data. We recently conducted a separate study of free-text override reasons for nonmedication alerts at a single site and found them to be highly informative, so this may be a productive line of research in the future. Second, we did not receive data from the sites about the drugs that caused the overrides, so we were unable to stratify our analysis by drug type. Third, we did not collect data on the user experience related to override reasons, such as how many reasons were shown on the screen or whether scrolling was required, so we were not able to assess how these user interface issues might have affected override reasons. These data are difficult to collect retrospectively because changes to the user interface over time and differences in the interface across platforms and devices are not consistently recorded. Finally, we did not measure usefulness of the override reasons provided. As described in previously, override reasons may be used to encourage careful consideration by ordering providers, communicate to pharmacists, or document reasoning for future medicolegal need. We do not have outcome data to suggest which reasons are useful in supporting these goals or whether DDI alerts are the best mechanisms to serve these purposes.

CONCLUSION

Organizations, EHR vendors and third-party medication knowledge vendors should improve the options available to their providers who choose to override DDI alerts. These options should be specific to the drugs involved and, where possible, include concrete clinical actions prescribers can take directly from the alert. Furthermore, both EHR vendors and medication knowledge base vendors should enhance the functionality and capabilities of their respective product offerings to facilitate more targeted alerting based on other data in the EHR. Taken together, these improvements should lead to better CDS, safer patient care, and reductions in alert burden for clinicians.

FUNDING

This work was supported by National Library of Medicine of the National Institutes of Health grant number R01LM011966. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

AUTHOR CONTRIBUTIONS

AW wrote the first draft of the manuscript. DSM, SA, and AA performed the analysis. All authors participated in data acquisition and reporting. All authors provided data, contributed to the analysis and made critical revisions for important intellectual content.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Click here for additional data file.