Assessing the potential adoption and usefulness of concurrent, action-oriented, electronic adverse drug event triggers designed for the outpatient setting.

BACKGROUND: Adverse drug event (ADE) detection is an important priority for patient safety research. Trigger tools have been developed to help identify ADEs. In previous work we developed seven concurrent, action-oriented, electronic trigger algorithms designed to prompt clinicians to address ADEs in outpatient care.
OBJECTIVES: We assessed the potential adoption and usefulness of the seven triggers by testing the positive predictive validity and obtaining stakeholder input.
METHODS: We adapted ADE triggers, "bone marrow toxin-white blood cell count (BMT-WBC)," "bone marrow toxin - platelet (BMT-platelet)," "potassium raisers," "potassium reducers," "creatinine," "warfarin," and "sedative hypnotics," with logic to suppress flagging events with evidence of clinical intervention and applied the triggers to 50,145 patients from three large health care systems. Four pharmacists assessed trigger positive predictive value (PPV) with respect to ADE detection (conservatively excluding ADEs occurring during clinically appropriate care) and clinical usefulness (i.e., whether the trigger alert could change care to prevent harm). We measured agreement between raters using the free kappa and assessed positive PPV for the trigger's detection of harm, clinical usefulness, and both. Stakeholders from the participating health care systems rated the likelihood of trigger adoption and the perceived ease of implementation.
FINDINGS: Agreement between pharmacist raters was moderately high for each ADE trigger (kappa free > 0.60). Trigger PPVs for harm ranged from 0 (Creatinine, BMT-WBC) to 17 percent (potassium raisers), while PPV for care change ranged from 0 (WBC) to 60 percent (Creatinine). Fifteen stakeholders rated the triggers. Our assessment identified five of the seven triggers as good candidates for implementation: Creatinine, BMT-Platelet, Potassium Raisers, Potassium Reducers, and Warfarin.
CONCLUSIONS: At least five outpatient ADE triggers performed well and merit further evaluation in outpatient clinical care. When used in real time, these triggers may promote care changes to ameliorate patient harm.

Introduction

Adverse drug events (ADEs) are a significant patient safety problem in both hospitals and outpatient care.1 According to a recent study of United States ambulatory care data, approximately 4.5 million outpatient visits each year are related to ADEs.2 Outpatient ADE surveillance has primarily relied on manual medical record review,3,4 patient surveys,5 or retrospective analyses of administrative and clinical data sets.6 With the more widespread use of electronic health records (EHRs),7–10 interest has grown in electronic ADE trigger tools—a set of algorithms that searches electronic patient data and flags patients with findings suggestive of an ADE.11,12 The field of computerized ADE surveillance research is evolving toward concurrent, action-oriented trigger tools to detect ADEs in near real time and facilitate mitigating or preventing patient harm.13–15

The implementation of concurrent, action-oriented, electronic ADE triggers is challenging. To be effective when used concurrently with clinical care, the trigger algorithm—or rules engine—must have access to a wide range of current clinical data including provider notes, laboratory results, medications, problem lists, and orders. Similarly, for the trigger to be action-oriented, the algorithms must detect impending or ongoing harm that providers have not yet addressed. Lastly, for the triggers to be adopted, they must detect ADEs with reasonable accuracy, and they must target the types of ADEs that clinicians think are important.

Given these challenges, concurrent, action-oriented ADE triggers are rare.16 Published examples are in inpatient settings and often are for the purposes of general surveillance, as opposed to real-time clinical care.11,13,17 A previous study with national experts in patient safety and ADE surveillance developed a set of concurrent, action-oriented triggers to detect high-value outpatient ADEs.18,19 This research benefited from the participation of three large health care systems with both inpatient and outpatient care settings and comprehensive EHRs. In this paper, we first assess the positive predictive validity (PPV) of the previously developed triggers for detecting clinically actionable outpatient ADEs; and second, we assess stakeholder perceptions of the likelihood of trigger adoption and implementation. This information can be used by health system leadership to determine which triggers merit implementation and further evaluation in outpatient clinical care.

Methods

Setting

This research is part of a larger study to develop and test outpatient adverse event triggers conducted in three large health care systems: Boston Medical Center (BMC), Intermountain Healthcare, and the Veterans Health Administration’s Rocky Mountain Integrated Service Network. All relevant Institutional Review Boards approved this study. Each of these health care systems is unique, but all have high functioning EHRs that bridge the inpatient and outpatient care settings. They also all have Warfarin clinics.

BMC plays a key role in stabilizing and strengthening the community health safety net, providing the most free care of any New England hospital ($294 million in fiscal year 2006). Its 581-licensed-bed urban academic medical center and its presence in Boston’s underserved and working-class neighborhoods result in a racially and socioeconomically diverse mix of patients. In 2006, BMC provided over 430,000 outpatient visits, 128,000 emergency room (ER) visits, and 28,000 admissions.

Intermountain Healthcare (Intermountain) is a not-for-profit integrated health care delivery system located in Utah and Idaho that includes 23 hospitals (with a combined 2,200 beds), ranging from very small rural hospitals to a tertiary and quaternary care facility, and over 90 outpatient clinics. With more than 120,000 inpatient admissions, 98,000 outpatient surgeries, 430,000 ER visits and 5.8 million outpatient encounters in 2006, Intermountain provides more than 50 percent of all care delivered in the region.

Within the Veterans Health Administration (VHA), the Veterans Integrated Service Network (VISN) 19 geographic network serves over 700,000 veterans in an area covering the states of Utah, Montana, Wyoming, Colorado and portions of Idaho, Kansas, Nebraska, Nevada and North Dakota. It has 78 sites of patient care including 2 full-service teaching hospitals and 4 other hospitals.

Positive Predictive Validity (PPV) Assessment

We evaluated the performance of seven outpatient ADE triggers described in a previous paper (Table 1).19 These triggers are tied to common blood tests used to evaluate the safety of commonly prescribed drug classes such as warfarin or psychotropic drugs. These triggers target evolving ADEs by detecting decreased renal function (Creatinine), myelosupression (BMT-WBC and BMT-Platelet), overanticoagulation (Warfarin), hyperkalemia (Potassium Raiser), hypokalemia (Potassium Reducer), or impairment in consciousness or cognition (Sedative Hypnotic). The PPV of a trigger is a measure of the proportion of time that a case flagged by the algorithm actually represents an actual ADE. PPV is also known as “hit rate.” When deciding the usefulness of a trigger, clinicians weigh the PPV of the trigger and potential harm that the trigger might address. Clinicians will tolerate lower PPV for triggers that target severe ADEs. In general, clinicians prefer triggers with high PPV, as they have higher greater value for the time they spend on the trigger “hit.”

Table 1.

Evolution of Outpatient ADE Trigger Logic Prior to Implementation

Trigger Name	Original Logic		Revised Logic
Creatinine	[New order or increase in (direct GFR reducer OR volume reducer OR nephrotoxin) within (1– 5 days OR 1 day to last creatinine measure)AND (No new trimethoprim within 5 days)]AND (No decrease in any meds above since the last creatinine measure OR no repeat order for creatinine)AND (>25% reduction in creatinine clearance since initiation or increase of above med AND resulting creatinine clearance < 50)		[Dose change in GFR reducer or renal toxin and subsequent creatinine > 50% from average baseline creatinine value or > 33% from last post dose change creatinine value]AND (No new trimethoprim within 1 day prior to 7 days after creatinine lab value firing)AND (GFR Reducer or renal toxin used within 90 days prior to lab value)
Bone Marrow Toxin (BMT)	White Blood Cell Count (WBC)	(On bone-marrow-toxic drug with a course more than 2 weeks AND No chemotherapy within 2 weeks)AND [(WBCs<2,500 AND decrease from before course by more than 2,000) OR (WBCs<2,000 AND decrease from before course by more than 1,000) AND (no repeat CBC OR no decrease in drug) ≤ 5 days of triggering result	[Within 3 to 30 days of chemotherapy start, WBCs<2,500 AND decrease from before chemotherapy course by >=.5, course before baseline, baseline result includes composite of all WBC lab values 2 years before the ADE index date) AND (on WBC reducer within 90 days of firing lab value)
	Platelet	(On bone-marrow-toxic drug with a course more than 2 weeks AND No chemotherapy within 2 weeks)AND (Platelets<50k AND decrease by 75k within 1 week)]	[Within 3 to 30 days of chemotherapy start, ((Platelet<50,000 AND 50% decrease or more from baseline) OR (platelet value drops by 75,000 or more from baseline), baseline result includes composite of all platelet lab values 2 years before the ADE index date) AND (on platelet reducer within 90 days of firing lab value)
Warfarin	[Started on warfarin within 14 days AND (INR>3.0 AND INR increased by 1 within 2 days) AND no repeat INR within 2 days]OR (Started in warfarin longer than 14 days prior AND INR>4 AND no repeat INR within 2 weeks)OR (INR>6 AND no repeat INR within 2 days)		[(No INR within 7 days of starting warfarin) OR (Within 8 to 13 days after starting warfarin, INR >3 and INR increase from two days prior by >1) OR (After 14 days or more of starting warfarin INR>4)]AND (on warfarins within 90 days of firing lab value)AND (CANNOT have a warfarin prescribed 13 months prior to the initial warfarin start date (ADE index date))AND (CANNOT have INR lab result >2 within the 3 previous months of the initial warfarin start date)
Potassium Reducer	Use of potassium reducer AND [K <3.0 OR (K < 3.5 AND K decreased by >15%) versus previous measurement] AND (No new potassium raiser OR decreased potassium reducer) within 5 days of triggering potassium result		[(Increase in K reducer or decrease in K raiser) AND (K <3.0 OR (K < 3.5 AND K decreased by >15%) versus previous measurement] AND (on same K reducer or increaser drug class in which dose change occurred within 90 days of firing trigger)
Potassium Raiser	[(K+>5.5 and up by >10% since last measurement)OR (K+>6.0)]AND (Potassium raiser active OR Potassium reducer discontinued 1 day to 4 weeks prior) AND (No new potassium reducer OR decrease in potassium raiser) within 5 days of triggering result		[(Decrease in K reducer or increase in K raiser)AND(K+>5.5 and up by >10% since last measurement)OR (K+>6.0)]AND (on same K reducer or increaser drug class in which dose change occurred within 90 days of firing trigger)
Sedative Hypnotic	Active prescription of sedative hypnotic including anticholinergic AND Subsequent diagnosis of (dementia, fall, delirium)		Active prescription of sedative hypnotic including anticholinergic AND Subsequent diagnosis of (dementia, fall, delirium) AND (on sedative hypnotic within 90 days of dementia, fall, or delirium diagnosis)

Notes: GFR = glomerular filtration rate, INR = international normalized ratio, K+ = potassium

Sample

We defined a study cohort of trigger-eligible cases from each of the participating health care systems (Figure 1).20 We included data for patients if they were at least 18 years of age and used drugs targeted by triggers between January 1, 2007 and December 31, 2007 (Table 2). We obtained four years of data from each of the systems (January 1, 2004–December 31, 2008) to incorporate patient history prior to and following any potential ADE. For each patient, we collected demographics; inpatient and outpatient utilization (e.g., ICD-9-CM procedure codes, dates of admission and discharge, dates of service); coded observations (e.g., vital signs, height, and weight); inpatient and outpatient laboratory tests and results; pharmacy orders; and patients’ clinical notes (e.g., progress notes, operations and procedures, and discharge summaries). Before we merged all information into a common database, data managers at each health care system de-identified the data according to a uniform procedure.21 We anticipated a sample of 6,000 patients for each trigger based on prevalence of targeted drug users at each facility.

Figure 1.

Attrition Figure of Study Cohort

Table 2.

Outpatient Adverse Drug Event (ADE) Trigger Logic and Sample Criteria

Trigger Name	Clinical Rationale	ADE Trigger Logic*	Cohort Selection Criteria and Sample Size
Creatinine	Detect decreased renal function to prevent reactions from other medications that are cleared by the kidney by looking for a decrease in creatinine clearance to a concerning level that occurs soon after starting a drug that might decrease creatinine, after confirming that the new drug has not been decreased.	[Dose change in GFR reducer or renal toxin and subsequent creatinine > 50% from average baseline creatinine value or > 33% from last post dose change creatinine value]AND (No new trimethoprim within 1 day prior to 7 days after creatinine lab value firing)AND (GFR Reducer or renal toxin used within 90 days prior to lab value)Suppress If: Renal toxin or GFR reducer dose reduced 0–7 days after lab)OR (new creatinine lab result within 1–6 days of firing lab value)	New prescription for (direct GFR reducer OR volume reducer OR nephrotoxin)]-random sample of 1,000 patients/site; current prescription for (direct GFR reducer OR volume reducer OR nephrotoxin)]-random sample of 1,000 patients/site. N = 6,000
Bone Marrow Toxicity (BMT) White-Blood-Cell Count (WBC)	Detect early signs of myelo-suppression (white blood cell count/platelet) to prevent more severe cases by looking for a decrease in cells after noncancer drug without evidence that the drug has been decreased in response.	[Within 3 to 30 days of chemotherapy start, WBCs<2,500 AND decrease from before chemotherapy course by ≥0.5, course before baseline, baseline result includes composite of all WBC lab values 2 years before the ADE index date)]AND (on WBC reducer within 90 days of firing lab value)Suppress If: (WBC reducer discontinued 0–5 days after firing lab value)OR (WBC resulted 1–6 days after firing lab value)	New prescription for bone marrow toxic drug-random sample of 1,000 patients/site; current prescription for bone-marrow toxic drug-random sample of 1,000 patients/site. N = 6,000
Bone Marrow Toxicity (BMT) Platelet	Detect early signs of myelo-suppression (white blood cell count/platelet) to prevent more severe cases by looking for a decrease in cells after noncancer drug without evidence that the drug has been decreased in response.	[Within 3 to 30 days of chemotherapy start, ((Platelet<50,000 AND 50% decrease or more from baseline) OR (platelet value drops by 75,000 or more from baseline), baseline result includes composite of all platelet lab values 2 years before the ADE index date)]AND (on platelet reducer within 90 days of firing lab value)Suppress If: (Platelet reducer discontinued 0–5 days after firing lab value)OR (platelet resulted 1–6 days after firing lab value)	New prescription for bone marrow toxic drug-random sample of 1,000 patients/site; current prescription for bone marrow toxic drug-random sample of 1,000 patients/site. N = 6,000
Warfarin	Detect rapid or excessive anti-coagulation to prevent bleed by looking for overanticoagulation and no evidence of rechecking within reasonable window.	[(No INR within 7 days of starting warfarin) OR (Within 8 to 13 days after starting warfarin, INR >3 and INR increase from two days prior by >1) OR (After 14 days or more of starting warfarin INR>4)]AND (on warfarins within 90 days of firing lab value)AND (cannot have a warfarin prescribed < 13 months prior to the initial warfarin start date (ADE index date))AND (cannot have INR lab result >2 within the 3 previous months of the initial warfarin start date)Suppress If: (New INR within 2 days after triggering INR)OR (If triggering INR 4<=INR<6 and new INR within 1 to 6 days of triggering INR)	New prescription for warfarin-random sample of 1,000 patients/site; current prescription for warfarin-random sample of 1,000 patients/site. N = 6,000
Potassium Reducer	Detect hypokalemia to prevent further decline and arrhythmia by looking for dropping potassium without evidence of adjustments to medication.	[(Increase in K+ reducer or decrease in K+ raiser) AND (K+ < 3.0 OR (K < 3.5 AND K+ decreased by > 15%) versus previous measurement]AND (on same K+ reducer or increaser drug class in which dose change occurred within 90 days of firing trigger)Suppress If: (K+ reducer dose increased or K+ raiser dose reduced within 7 days after firing criteria) OR (K+ resulted 1–6 days after firing criteria)	New prescription for potassium raiser-random sample of 1,000 patients/site; current prescription for potassium raiser-random sample of 1,000 patients/site. N = 6,000
Potassium Raiser	Detect hyperkalemia to prevent further increase and arrhythmia by looking for rising potassium without evidence of adjustments to medication.	[(Decrease in K+ reducer or increase in K+ raiser) AND (K+ > 5.5 and up by > 10% since last measurement) OR (K+ > 6.0)]AND (on same K+ reducer or increaser drug class in which dose change occurred within 90 days of firing trigger)Suppress If: (K+ reducer dose reduced or K+ raiser dose increased within 7 days after firing criteria) OR (K+ resulted 1–6 days after firing criteria)	New prescription for potassium reducer-random sample of 1,000 patients/site; current prescription for potassium reducer-random sample of 1,000 patients/site. N = 6,000
Sedative Hypnotic	Detect impairment in consciousness and cognition to improve quality of life by looking for patients on psychotropic drugs with a subsequent decline in consciousness or cognition.	Active prescription of sedative hypnotic including anticholinergicAND Subsequent diagnosis of (dementia, fall, delirium)AND (on sedative hypnotic within 90 days of dementia, fall, or delirium diagnosis)	New prescription for sedative hypnotic or anticholinergic drug-random sample of 1,000 patients/site; current prescription for sedative hypnotic or anticholinergic drug-random sample of 1,000 patients/site. N = 6,000

Notes: GFR = glomerular filtration rate, INR = international normalized ratio, K+ = potassium

ADE Trigger Logic: We revised the logic from our previous work to improve trigger performance, including sample restrictions and the addition of trigger suppression logic.

We carried the logic of the trigger, developed in the Delphi consensus process, a bit further in an effort to improve the performance of the trigger (a comparison of the original and revised trigger logic is in Table 1).19 For example, we changed the Sedative Hypnotic trigger to define a time limit of 90 days during which a delirium episode could occur and be attributable to medications. We also excluded eligible cases with fewer than two clinical visits or with incomplete clinical note data in the EHR in the 12-month index period. To maximize clinical utility, we programmed a core trigger designed to detect evidence of harm and added suppression criteria that could be used to prevent the trigger from firing while the provider titrated a new medication and dose.

Chart Review

We applied the ADE trigger logic to each study cohort in order to identify trigger-flagged cases. To facilitate reviewing the merged data, a chart browser was created with similar look and functionality as the VHA’s EHR interface. Through block randomization, we assigned two out of four pharmacists to review each trigger-flagged event. To ensure that each trigger was programmed to fire as intended, the four trained pharmacist reviewers examined approximately five cases. We repeated these reviews and adjusted the trigger algorithms until the triggers fired without observed error.

We developed an ADE classification tool and trigger-specific clinical guidelines to evaluate whether a true ADE occurred, the level of harm associated with the ADE, and the clinical usefulness of the ADE trigger, i.e., whether a patient’s care plan could have been altered to prevent or mitigate ADE harm (see Appendix A). We defined an “ADE” to be “an event caused by a drug that reached the minimum level of harm specified by version 1.1 of the Agency for Healthcare Research and Quality (AHRQ) Harm Scale” (e.g., inconvenience for a patient, such as from a blood draw).22 The Harm Scale is part of the National Quality Forum (NQF)-endorsed Patient Safety Common Formats, a widely accepted tool for documenting patient safety events to ensure greater standardization in the field.23 We used structured World Health Organization (WHO) and Uppsala Monitoring Centre (UMC) criteria to rate causality and accepted events that were either probably or certainly related to a drug.24

For commonly titrated and monitored drugs, we developed conservative criteria for attribution of harm. For example, it is best practice to monitor a patient’s international normalized ratio (INR) monthly after a warfarin dose change. Likewise, it is good practice to check a patient’s serum potassium level after the patient is started on a drug that affects serum potassium. We did not count abnormalities found during good-practice monitoring for titration as ADEs, even though they met the harm criteria. We required that the patient had at least one monitoring event within a short time beyond that which would be expected in normal drug titration. Criteria for attribution of harm and causality are included in Appendix A.

Clinical usefulness was generally defined as present if the trigger alert could have prompted a change in the treatment plan to prevent or address harm. Thus, a trigger alert could change clinical care without flagging patient harm. For each event, we developed explicit criteria for what was considered clinically useful (see Appendix A). If the alert corresponded to an event for which no further patient treatment was necessary or beneficial, the alert was not deemed useful.

To assess interrater reliability (IRR), we used a marginal free kappa statistic. This statistic is appropriate when the distribution of positive and negative cases is highly skewed, as is the case with this study.25 We met with pharmacist reviewers and revised the chart review tool accordingly, until we reached a kappa of at least 0.6, which represents good agreement.26 The agreement target was for the main outcome of ADE defined by harm and causality thresholds.

Up to 51 (17 per site) charts for each trigger alert were reviewed to estimate PPV. Agresti-Coull 95 percent confidence intervals were calculated.27 The resulting confidence intervals’ widths narrow as the PPV deviates from 50 percent. We expected a confidence interval of 18 percent at a point estimate of 10 percent, and 27 percent at a point estimate of 50 percent PPV. Fewer cases would result in wider intervals.

Analyses

We measured the PPV of each trigger in detecting ADEs, having clinical usefulness, and a combination of both. These outcomes are relevant to concurrent, action-oriented triggers: for purposes of event detection, tables that include all alerts and ADEs are relevant, while for purposes of clinical interventions, tables that exclude suppressed alerts are relevant. We calculated 95 percent confidence intervals around the PPV estimates. All analyses were completed using SAS version 9.1.28

Stakeholder Usefulness Assessment

To achieve the second objective of our study, we also assessed the usefulness of the outpatient triggers among facility-based stakeholders who might make adoption decisions or use the trigger tools in administering patient care. Following the criterion-validity assessment, we recruited at least 10 outpatient adverse-event trigger stakeholders at each of the health care systems in our study to participate in focus groups or interviews (stakeholders who could not attend focus groups were interviewed by phone). During a 90-minute session, the focus groups covered all of the triggers in our study, including the ADE triggers presented in this paper. Stakeholders were selected by our facility-specific investigators based on their judgment that the participants would be able to provide meaningful information on the following: (1) the adoption of the baseline assessment triggers at their site, or (2) the hypothetical implementation of the triggers at their site, or (3) both. Stakeholders included frontline staff; middle managers; and executives (e.g., physicians, nurses, informatics and information system specialists, pharmacists, and quality and patient safety managers). We obtained informed consent forms for all participants.

Before focus groups or interviews, we provided an overview of trigger tools and tables summarizing trigger performance in their health care system. In the focus groups, we asked each participant to rate the triggers based on the likelihood for trigger adoption within their health care system, and whether trigger implementation was straightforward. Specifically, focus group participants were asked, “Imagine that your organization is considering whether to adopt trigger tools. How likely do you think it is that your organization will adopt each of these triggers?” The ratings for adoption were “likely,” “unclear,” and “unlikely.” They were next asked, “Now imagine that your organization has decided to adopt all of the triggers. How easy or difficult will it be to implement each of them?” The ratings for implementation were “straightforward,” “unclear,” and “difficult.” In both the focus groups and the interviews, we asked open-ended questions about the trigger logic and the various strengths or weaknesses of the triggers. These questions considered factors such as the perceived burden on staff and resources to implement the trigger, as well as the perceived benefit of the trigger in terms of clinical usefulness and integration with existing workflows.

Findings

We did not achieve the goal of 2,000 eligible patients per site (Table 3); one site found fewer than 2,000 patients met the trigger eligibility criteria for two triggers (Creatinine and Warfarin) and the other two sites gave us all their trigger-eligible patients because they were limited in their ability to randomly sample within their patient data. The number of trigger-eligible patients dropped for each trigger sample when we applied modified trigger logic (“Revised trigger-eligible cases” in Table 3). The triggers flagged less than 1 percent of trigger-eligible patients with the exception of Sedative Hypnotic and Warfarin (flag rates of 4.4 percent and 18.2 percent, respectively). Of the trigger-eligible patients, we planned to select up to 17 trigger-flagged cases from each of the facilities for chart review; however, we had fewer than 17 trigger-flagged cases from each site for a few of the triggers (“Cases reviewed” in Table 4 versus “Trigger-flagged” in Table 3).

Table 3.

ADE Trigger Assessment Samples

Trigger Name		Initial Trigger-eligible Cases				Revised Trigger-eligible Cases*	Trigger-flagged Cases (%)
		Site 1	Site 2	Site 3	Total
Creatinine		1,000	8,444	2,199	11,643	9,379	57 (0.6%)
BMT	WBC	2,000	1,342	2,165	5,507	3,377	5 (0.1%)
	Platelet						17 (0.5%)
Warfarin		1,876	1,059	2,199	5,134	3,710	677 (18.2%)
Potassium	K+-raiser	2,000	9,339	4,399	15,738	12,320	64 (0.7%)
	K+-reducer						90 (0.5%)
Sedative Hypnotic		2,000	7,925	2,198	12,123	8,186	359 (4.4%)

Notes:

ADE = adverse drug event

BMT = bone marrow toxin

WBC = white blood cell count

K+ = potassium

Revised trigger-eligible cases: We changed the trigger logic to improve sensitivity of the triggers, and therefore narrowed our trigger-eligible samples.

Table 4.

Comparison of Outpatient ADE Trigger Positive Predictive Value (PPV) Assessment Results With and Without Trigger Suppression Logic

	Positive Predictive Value (PPV) Assessment (includes suppressed and nonsuppressed cases)				Positive Predictive Value (PPV) Assessment (excludes suppressed cases)
Name	Cases Reviewed	Harm PPV (95% CI)	Clinical Usefulness PPV (95% CI)	Both Harm + Usefulness PPV (95% CI)	Non-Suppressed Cases (% Δ)**	Harm PPV (95% CI)	Clinical Usefulness PPV (95% CI)	Both Harm + Usefulness PPV (95% CI)
Creatinine	49	–	60% (43–75%)	60% (43–75%)	33 (32%)	–	58% (39–75%)	58% (39–75%)
BMT - WBC	5	–	–	–	4 (20%)	–	–	–
BMT - Platelet	17	6% (0–29%)	53% (28–77%)	59% (33–82%)	15 (12%)	7% (0–30%)	47% (21–73%)	53% (26–79%)
Warfarin	96	13% (7–21%)	14% (7–22%)	19% (12–28%)	84 (12%)	5% (1–12%)	8% (3–16%)	11% (5–19%)
Potassium Raiser	50	8% (2–19%)	28% (16–43%)	36% (23–51%)	28 (44%)	–	32% (16–52%)	32% (16–52%)
Potassium Reducer	85	17% (9–26%)	42% (31–54%)	58% (46–68%)	68 (20%)	16% (8–27%)	47% (35–60%)	62% (49–73%)
Sedative Hypnotic	68	10% (4–20%)	10% (4–20%)	19% (11–31%)	68 (0%)	10% (4–20%)	10% (4–20%)	19% (11–31%)

Notes

BMT = bone marrow toxin

WBC = white blood cell count

PPV = positive predictive value

95% CI = 95% confidence interval

Suppression logic prevents trigger firing when abnormal lab values occur during good-practice monitoring

Percent change in the number of cases flagged using the trigger’s suppression logic out of the number of cases reviewed

Our evaluation of IRR showed moderate agreement among pharmacist reviewers in their assessments of the presence of an ADE, ADE harm, and trigger usefulness (all marginal free kappa scores ≥ 0.60), with three exceptions limited to the BMT-Platelet and Creatinine triggers. Agreement was highest between the two pharmacists with advanced degrees.

Overall trigger PPVs for harm detection and clinical usefulness ranged from 19 percent for Warfarin and Sedative Hypnotic to 60 percent for Creatinine (Table 4). None of the Creatinine and BMT-WBC trigger-flagged cases had a harmful ADE; however 24 of the Creatinine-flagged cases (60 percent) identified potential ADEs that could have been avoided with clinical interventions.

Table 4 also presents results with the trigger suppression logic applied to remove trigger-flagged cases where there was sufficient evidence that the patient’s clinical care team was taking appropriate action. We found that for all but the Sedative Hypnotic trigger the number of trigger-flagged cases was lower, and the PPV for harm detected and clinical usefulness improved but without reaching statistical significance. We reviewed 370 trigger-flagged cases and found that 70 (19 percent) would not have been flagged using the suppression logic. The Potassium Raiser trigger had the largest drop in the number of patients flagged by the trigger when the suppression logic was imposed (22 out of 50 cases, or 44 percent, would not have been flagged using trigger suppression logic).

For the stakeholder usefulness assessment, 15 individuals participated in the focus groups on ADE triggers, and 17 individuals were interviewed. Table 5 presents the focus group results for the questions regarding the likelihood of trigger adoption and the perceived ease of trigger implementation presented to the focus group participants. Warfarin was by far the most popular trigger based on stakeholder feedback (93 percent of focus group participants said they would adopt the trigger and that it was straightforward to implement). The most unpopular trigger was BMT-WBC (only 7 percent of participants reported likelihood of adoption or straightforward implementation). Stakeholder’s unstructured feedback on individual triggers from focus groups and interviews is described in the following section with a summary of each trigger’s performance (i.e., stakeholder ratings combined with the criterion validity results).

Table 5.

Stakeholder Focus Group Results for Outpatient ADE Trigger (n = 15)

Name	Trigger Adoption (% of Participants)			Trigger Implementation (% of Participants)
	Likely	Unclear	Unlikely	Straightforward	Unclear	Difficult
Creatinine	40%	53%	7%	33%	47%	20%
BMT - WBC	20%	67%	13%	33%	53%	13%
BMT - Platelet	20%	67%	13%	40%	47%	13%
Warfarin	93%	7%	0%	93%	7%	0%
Potassium Raiser	27%	60%	13%	33%	33%	33%
Potassium Reducer	27%	60%	13%	27%	40%	33%
Sedative Hypnotic	7%	40%	53%	7%	40%	53%

Notes:

BMT = bone marrow toxin

WBC = white blood cell count

Trigger-Specific Results

Creatinine

This trigger targets the situation of a patient taking drugs cleared by the kidneys and having a subsequent decrease in renal function and unintended elevated drug levels. Focus-group participants indicated that adoption would be likely and implementation would be feasible; however, some interview participants expressed concerns about whether outpatient drug order information would be available in a timely manner in the EHR. There was no harm identified in trigger-positive cases, but in 60 percent of the cases, the patient care team could have adjusted the dose to both achieve the original effective dose and reduce the risk of overdose. This trigger appears to be suited to an action-oriented concurrent trigger system.

BMT – WBC

This trigger targets drug-related reduction of white blood cells. Stakeholders had moderate enthusiasm for the trigger and thought it would be feasible to implement; however one site indicated the trigger could be redundant with an existing laboratory-based alert. Despite its promise, the trigger did not detect any events despite a high number of trigger-eligible patients (PPV = 0). Without modification, this trigger is not suitable for any purpose.

BMT – Platelet

This trigger targets drug-related reduction of platelets. The focus groups’ results were neutral with respect to adoption and implementation. The trigger had poor performance detecting harm (PPV = 6 percent), but good performance as a trigger to change the care plan to prevent further reduction in platelets (PPV = 53 percent). Because the consequences of low platelets are high, some focus group and interview participants reported that they were willing to tolerate a relatively low PPV for harm. This trigger appears to be well-suited to an action-oriented concurrent trigger system.

Warfarin

This trigger targets high INRs that have not been rechecked within a reasonable time. Focus-group and interview participants had a great deal of enthusiasm for this trigger’s adoption and thought it was highly feasible to implement. Some stakeholders expressed concern that the trigger would be redundant with warfarin clinics; however, even with existing surveillance efforts in two of the health care systems, the trigger detected harm in 13 percent of cases and triggered a possible care change in 14 percent of cases. This trigger appears to be well-suited, as long as there is sufficient clinician education, for either a retrospective system to track harm or a concurrent system to help respond to and manage elevated INRs.

Potassium Raiser

This trigger targets patients with high serum potassium (hyperkalemia) without evidence of interventions to prevent further rise. Focus groups had moderate enthusiasm for adoption and thought the trigger would be moderately feasible to implement; however interview participants expressed concern about the timeliness of the drug order data in the EHR (this was also a concern for the Potassium Reducer trigger). The trigger had poor performance for detecting harm (PPV = 8 percent) but moderate performance for possibly being able to alter clinical care (PPV = 28 percent). This trigger appears to be suited to a concurrent system designed to inform clinical care.

Potassium Reducer

This trigger targets patients with low serum potassium (hypokalemia) without evidence of interventions to prevent further decline. Stakeholders had a moderate level of enthusiasm for adopting the trigger and thought it would be moderately feasible to implement. The trigger also had moderate performance for detecting harm (PPV = 17 percent) and good performance for possibly being able to alter clinical care (PPV = 42 percent). This trigger appears to be moderately suited to a trigger system designed to track ADEs and could have high impact in a concurrent system designed to inform clinical care.

Sedative Hypnotic

This trigger targets patients who may have had drug-related delirium. Stakeholders had generally low enthusiasm for this trigger primarily because they believed that this trigger did not present adequate opportunity to change clinical care. This trigger had the same low performance for both detecting harm and possibly being able to alter care (PPV = 10 percent). This trigger requires more study or adjustment before implementation.

Discussion

This study reports our assessment of the combined predictive validity of the ADE triggers with stakeholder assessments on the clinical usefulness of concurrent, action-oriented ADE triggers designed for use with outpatient EHR data. We found that five triggers were suitable for real-world implementation and further evaluation: Creatinine, BMT-Platelet, Warfarin, Potassium Raiser, and Potassium Reducer triggers. In interpreting these results, several factors should be considered.

Trigger PPV in our study is high despite a more conservative approach to flagging events and assessing harm than in other studies. A high PPV indicates that many of the cases flagged by the triggers are true ADEs. Earlier studies of outpatient ADE triggers found individual trigger PPVs ranged from 0–50 percent in detecting preventable ADEs.3,6,29 The majority of our triggers had PPVs for harm or clinical utility ranging from 11 percent to 62 percent when trigger-suppression logic was imposed. Three of the six triggers had measurable PPVs over 50 percent. ADE trigger-suppression logic is essential to an action-oriented trigger, as these triggers should not flag cases in which the health care team has already intervened. Our suppression logic effectively excluded flags during best-practice titration, when clinicians are explicitly monitoring for ADEs and additional flags would be a nuisance. However, if integrated into the EHR’s alerting system, generated flags would not be redundant and could help providers to focus their attention on drug causes of harm.

On the other hand, the prevalence of triggers in our population seems relatively low. Less than 1 percent of the approximately 50,000 patients at risk for an ADE targeted by the triggers were found to have a trigger-flagged ADE. To our knowledge, as there are no other studies that have reported trigger prevalence in an at-risk population, we cannot offer a good comparison for these findings. However, a systematic review of preventable outpatient ADEs conducted in 2007 found prevalence to range from 11 percent to 38 percent.30 Our trigger prevalence and proportion of ADEs found might be lower than expected for two reasons. First, we did adjust criteria thresholds of triggers to target more serious events. Second, we differ from most reports in that we use explicit and relatively conservative case definitions of ADEs that require evidence of patient harm that, in most cases, must be present, in addition to laboratory derangements. However, the low trigger prevalence raises questions of the sensitivity of our triggers. Because we did not have the resources available to conduct sensitivity analyses, we suggest that further studies that target class-specific ADE sensitivity should be undertaken,.

Facility-based stakeholders expressed moderate support for most of the triggers. We deliberately selected the seven triggers for further testing based on endorsement from national experts in our previous work,19 yet only the Warfarin trigger was rated highly for adoption and implementation by local stakeholders. We partially attribute the discrepancy between stakeholders and national experts to the difference in perspectives on population health and systemwide cost and quality control, as compared to the direct-patient-care orientation of stakeholders. Another explanation for these discrepancies was that some focus group participants may not have understood the application of trigger tools despite the educational material and discussion we provided. This misunderstanding was manifest in concerns that some triggers, such as Warfarin, might be redundant with existing surveillance programs, despite our intent that ADE triggers would these programs—in such cases, the staffing and workflow to manage the detection and response to ADEs is already in place, increasing the ease of implementation. Similarly, experts aware of the high burden of ED admissions related to sedatives valued that trigger higher.19 Local stakeholder’s perspectives clearly differ from those of national experts, and must be taken into consideration when working with individual health care systems to implement ADE triggers.

We would like to highlight several limitations that are both particular to this study and also generally applicable to studies of concurrent, action-oriented triggers. First, we found that pharmacists without clinical training underestimated clinical utility compared to clinically trained pharmacists and physicians. Consequently, IRR for clinical utility of three triggers was low. This result underlines the importance of clinically trained raters for trigger use. Second, we were not able to assess sensitivity of the triggers due to a very high expected uncertainty of the result, given the resources available to conduct chart review. Sensitivity is an important but rarely measured performance validity criterion that denotes the proportion of harm captured by a trigger. In our study the prevalence of trigger-flagged cases was mostly below 1 percent. Even if trigger sensitivity were less than 50 percent, the estimated confidence interval widths in our cohort for sensitivity would be substantially higher than 25 percent.31

Our study also has several strengths. We are the first to specifically test the performance of concurrent, action-oriented electronic ADE triggers for outpatient care. There have been evaluations of action-oriented outpatient triggers for other sources of harm such as missed diagnoses;32 however, all previous reports of ADE triggers for concurrent use with clinical care, regardless of care setting, have had the primary aim of detecting harm for general surveillance and secondarily for clinical intervention.9,33,34 Whereas previously described trigger systems target ADEs without regard to whether further action is warranted, this study featured trigger logic designed to exclusively flag events for which clinical intervention could be useful. Additionally, our study included patients and stakeholders from three varied health care systems, such that we expect our findings can be generalized to other integrated health care systems with electronic health data.

Conclusion

Outpatient ADEs as a whole are a significant source of patient harm and may be prevented or mitigated through the use of trigger tools that rely on EHR data. We tested seven outpatient ADE triggers and concluded that five triggers—Creatinine, BMT-Platelet, Warfarin, Potassium Raiser, and Potassium Reducer—merit further evaluation. They may also merit cautious implementation in a system that does not cause clinicians to become overly reliant on them, as their sensitivity is unknown. Trigger-system implementation requires significant investment both in resources and provider buy-in. Our results suggest that concurrent, action-oriented electronic triggers can meet the approval of facility-based stakeholders and may improve patient safety.