Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial.

OBJECTIVE: To determine whether electronic health record alerts for acute kidney injury would improve patient outcomes of mortality, dialysis, and progression of acute kidney injury.
DESIGN: Double blinded, multicenter, parallel, randomized controlled trial.
SETTING: Six hospitals (four teaching and two non-teaching) in the Yale New Haven Health System in Connecticut and Rhode Island, US, ranging from small community hospitals to large tertiary care centers. PARTICIPANTS: 6030 adult inpatients with acute kidney injury, as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) creatinine criteria.
INTERVENTIONS: An electronic health record based "pop-up" alert for acute kidney injury with an associated acute kidney injury order set upon provider opening of the patient's medical record. MAIN OUTCOME MEASURES: A composite of progression of acute kidney injury, receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included outcomes at each hospital and frequency of various care practices for acute kidney injury.
RESULTS: 6030 patients were randomized over 22 months. The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67). Analysis by each hospital showed worse outcomes in the two non-teaching hospitals (n=765, 13%), where alerts were associated with a higher risk of the primary outcome (relative risk 1.49, 95% confidence interval 1.12 to 1.98, P=0.006). More deaths occurred at these centers (15.6% in the alert group v 8.6% in the usual care group, P=0.003). Certain acute kidney injury care practices were increased in the alert group but did not appear to mediate these outcomes.
CONCLUSIONS: Alerts did not reduce the risk of our primary outcome among patients in hospital with acute kidney injury. The heterogeneity of effect across clinical centers should lead to a re-evaluation of existing alerting systems for acute kidney injury. TRIAL REGISTRATION: ClinicalTrials.gov NCT02753751. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

Introduction

Acute kidney injury—an abrupt decline in the kidney’s filtration function—is a common occurrence in patients in hospital and is associated with nearly a 10-fold increase in inpatient mortality.1 2 3 In survivors, acute kidney injury is associated with increased risks of chronic kidney disease, end stage kidney disease, and mortality.4 5 Despite the strong association with important outcomes, the syndrome (which is asymptomatic) is often missed in the course of clinical care; it occurs in roughly 15% of patients in hospital and yet is recorded in the electronic health record in fewer than half of patients with acute kidney injury.6 7 8 9 10 Best practices for acute kidney injury, such as routine urinalysis and measurement of urine output, occur infrequently, and even repeat measurement of creatinine is not universal. These deficiencies provide a reason for alerting providers about the presence of acute kidney injury.11

With the assumption that increased recognition of acute kidney injury will improve care of these patients and thus improve clinical outcomes, many health systems in the United States, and the National Health Service in the United Kingdom, have introduced an electronic alert for acute kidney injury as part of routine clinical care.12 13 14 Data on the efficacy of these alerts are limited, however, and definitive evidence of their benefit for patient outcomes remains absent. Additionally, as acute kidney injury is a heterogeneous condition, lacking proven strategies for its mitigation, an alert might prompt actions that are unnecessary or even harmful.

The effectiveness of alerting probably depends on a combination of patient, provider, and disease factors, and might differ in different care environments, thus requiring rigorous evaluation. We conducted a multicenter randomized clinical trial of usual care versus usual care plus pop-up acute kidney injury alerts delivered through an electronic health record. The alert was designed to reach multiple providers and was implemented using an identical protocol across a diverse set of hospitals. We hypothesized that alerts would influence provider behavior and improve clinical outcomes.

Methods

Extended details of the methods are shown in a supplementary web appendix.

Trial design and oversight

A detailed description of the trial design and rationale has been previously published.15 Briefly, we conducted a patient level, parallel group, randomized controlled trial of health record electronic alerts versus usual care in adults with acute kidney injury at six hospitals in the Yale New Haven Health System in Connecticut and Rhode Island, US. The trial was conducted in accordance with the principles of the Declaration of Helsinki, and two institutional review boards associated with the six study hospitals approved the study, which was deemed minimal risk. Patients were enrolled, and a requirement for informed consent was waived, as the alert was deemed unlikely to affect patient welfare and informing patients of their diagnosis of acute kidney injury would contaminate the usual care group. An external data and safety monitoring board performed four independent assessments during the trial, including one formal interim analysis at 50% recruitment. These analyses were not stratified by hospital. Hospitals were given guidelines to terminate the study if the P value associated with interim analysis was less than or equal to 0.001 for efficacy or less than or equal to 0.005 for harm. The hospitals also had the authority to terminate the study of their own accord regardless of those recommendations. This trial was registered with clinicaltrials.gov under registration number NCT02753751 on 25 April 2016, before study initiation and patient enrolment.

Patients

Inpatient adults aged 18 or older with acute kidney injury, defined by the Kidney Disease: Improving Global Outcomes (KDIGO) criteria as an increase in creatinine 0.3 mg/dL (26.5 μmol/L) within 48 hours or 1.5 times the lowest measured creatinine within the previous seven days of admission to hospital, were eligible.16 Owing to missing and inaccurate urine output data, urine output was not used to classify acute kidney injury. Patients with a history of end stage kidney disease (based on International Classification of Disease, ninth and 10th revision (ICD-9 and ICD-10) codes, a dialysis order in the past year, or an initial hospital creatinine greater than 354 μmol/L were excluded automatically by the detection algorithm. We subsequently excluded patients whose admission date was before the inception of alerts at a study hospital, whose first alert occurred after hospital discharge, who had been enrolled in a previous study, and who were enrolled during a two week period in which alerting ceased owing to an upgrade of the electronic health record system (fig S1).

Randomization and masking

Eligible patients were identified using an acute kidney injury detection algorithm built into our best practice alert within the Epic electronic health record system. When the chart is opened, the algorithm automatically assesses the patient record for inclusion and exclusion criteria. If all criteria are met, the patient is automatically enrolled into the trial and randomized. Simple randomization was performed within Epic using an internal random number rule with concealed allocation, as our electronic health record system cannot perform permutated block randomization or import external randomization lists. Randomization was performed at the patient rather than the provider level, as inpatients at participating hospitals are often cared for by many providers who change throughout their hospital stay. All study investigators were blinded to patient randomization status until enrollment was complete.

Intervention

The intervention was an automated, electronic, pop-up alert which fired whenever the patient’s electronic chart (Epic Systems, Verona, WI) was opened (fig 1). Alerts were displayed only to individuals who had authority to change or enter new orders on behalf of the patient—hereafter referred to as “providers”—which included interns, residents, fellows, attending physicians, nurse practitioners, and physician’s assistants. Alerts were displayed each time the chart was opened, provided the patient continued to meet criteria for acute kidney injury. If the provider agreed or disagreed with the presence of acute kidney injury, the alert was suppressed for 48 hours for that provider. If multiple providers used the electronic health record to care for the same patient, each of them separately received the alert whenever they opened the patient chart. Patients randomized to the usual care group generated “silent” alerts, which did not display to providers but were tracked, allowing the study team to observe which type of providers opened the chart and how often.

Fig 1

Screenshot of the acute kidney injury (AKI) alert. Creatinine 1 mg/dL=88.42 μmol/L.

The alert contained an option to add acute kidney injury to the patient’s problem list, and also a link to an acute kidney injury order set (supplemental fig S2); the set included options for blood and urine testing and kidney imaging but were limited to tests and procedures considered minimal risk (that is, intravenous fluid administration was not included). Providers at each of the study hospitals received education about acute kidney injury and the alert system before study roll out during departmental conferences. The educational program explained the definitions of acute kidney injury, the functioning of the alert, and the ethics of the study but did not provide suggested practices of care.

Outcomes

The primary outcome was a composite of inpatient acute kidney injury progression (defined as an increase in acute kidney injury stage), receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included the components of the primary outcomes and frequency of various practices of care for acute kidney injury. These practices included administration of contrast, fluids, or a nephrotoxic agent (a non-steroidal anti-inflammatory drug, angiotensin converting enzyme inhibitor/angiotensin receptor blocker, or aminoglycoside), ordering a urinalysis, documentation of acute kidney injury, monitoring of creatinine and urine output, and ordering a kidney consultation. An assessment of each hospital’s alert effects was also prespecified.

Statistical analysis

We present descriptive statistics as median (interquartile range) for continuous variables and proportions for categorical variables. For the primary analysis, and all comparisons of categorical variables between the intervention and control group, we used the Mantel-Haenszel test, accounting for each hospital site as a stratum. We used the Mantel-Haenszel approach to obtain the pooled relative risks across hospital strata without adjusting for other baseline factors.

As a sensitivity analysis, we used modified Poisson generalized estimating equations with a robust variance estimator to present the relative risk estimates, adjusting for the following characteristics assessed at the time of randomization: age, sex, race, creatinine, blood urea nitrogen, white blood cell count, heart rate, respiratory rate, systolic and diastolic blood pressure, chronic heart failure, hypertension, diabetes, malignancy, intensive care unit status, modified sequential organ failure assessment (mSOFA) score, Elixhauser comorbidity score, and hospital.17 We used the Van Elteren test to compare continuous variables across the intervention groups, accounting for hospital strata.

To compare time to event between study groups, we used Cox proportional hazard regression with intervention as the independent variable, stratified by hospital, with censoring at 14 days after randomization. For individual hospital analyses, Kaplan-Meier curves were generated and log rank tests were used. Patients discharged before 14 days without an outcome of interest were assumed to be free of that outcome at 14 days. Death was treated as a censoring event in analyses where death was not the outcome. We used Schoenfeld residuals to examine the proportional hazards assumption in Cox models; there were no violations. We specifically amended the protocol to include a test for site heterogeneity on 4 May 2018—that is, two months after the initial patient enrollment and before we had started alerting in any of the hospitals in which adverse outcomes were ultimately detected. This new protocol was based on ongoing executive committee calls and the hypothesis that the alert effect might differ by hospital. No data were used to drive this decision.

Our preliminary data suggested that 24.5% of patients would experience the composite outcome. We considered a relative 20% reduction in this outcome (to 19.6) was clinically significant and would represent an acceptable threshold to justify adding an interruptive alert to the clinician workflow. A sample size of 2512 in each arm of the study achieves 90% power to detect this degree of change, but given the potential for contamination across study arms (whereby clinicians “learn” to identify and care for acute kidney injury over the course of the trial), which would tend to bias the results toward the null, we inflated the sample size by 20% to a total of 6030. Although this sample increase does not deal with the bias toward the null, it does allow for increased detection of a smaller effect size.

Statistical analyses were conducted in Stata version 15.1 (College Station TX), SAS version 9.4 (SAS Institute, Cary, NC), and R (RStudio version 1.2.5033 (R version 3.5.3), Boston, MA), and P≤0.04 was considered statistically significant for the primary outcome (to account for the interim analysis). We do not report P values for secondary and exploratory analyses, except for safety outcomes, but report 95% confidence intervals in all cases. The widths of the 95% confidence intervals have not been adjusted for multiplicity, and inferences drawn from these might not be reproducible.

Patient and public involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for recruitment, design, or implementation of the study. No patients were asked to advise on interpretation or writing up of results. Patient involvement was not sought owing to the limited public understanding of acute kidney injury and a lack of definitive treatment.

Results

From 29 March 2018 to 14 December 2019, 6030 individuals met enrollment criteria and were randomized (supplemental fig S1). Those patients generated a total of 226 316 alerts (including silent alerts in the usual care arm), which reached 3632 unique providers, a median (interquartile range) of 7 (7-13) for each patient. Characteristics of the study population at the time of randomization are shown in table 1 (stratified analyses, supplemental tables 1-6). The median (interquartile range) age of patients was 71 (59-82) years, 2882 (48%) were female, and 946 (16%) self-identified as black. The majority (n=4564 (76%)) were admitted to a medical service, and 1923 (32%) were in an intensive care unit at the time of randomization.

Table 1

Patient characteristics at the time of randomization. Data are presented as median (interquartile range) or count (percent)

Variables	Usual care (n=2971)	Alert (n=3059)
Demographics:
Median (IQR) age (years)	71.3 (59.4-81.6)	71 (59-81.7)
Female sex	1440 (48.5)	1442 (47.1)
Black	462 (15.6)	484 (15.8)
Hispanic	289 (9.7)	331 (10.8)
Hospital admission characteristics:
Medical admission	2280 (76.7)	2284 (74.7)
Patient in the ICU	961 (32.3)	962 (31.4)
Patient in the emergency department	108 (3.6)	93 (3.0)
Patient in the ward	1902 (64.0)	2004 (65.5)
Hospital 1 (urban, teaching)	1238 (41.7)	1277 (41.7)
Hospital 2 (urban, teaching)	599 (20.2)	649 (21.2)
Hospital 3 (urban, teaching)	456 (15.3)	479 (15.7)
Hospital 4 (suburban, teaching)	292 (9.8)	275 (9.0)
Hospital 5 (suburban, non-teaching)	278 (9.4)	276 (9.0)
Hospital 6 (suburban, non-teaching)	108 (3.6)	103 (3.4)
Comorbidities:
Chronic kidney disease	1127 (37.9)	1163 (38.0)
Congestive heart failure	1307 (44.0)	1351 (44.2)
COPD	1015 (34.2)	1049 (34.3)
Diabetes mellitus	1197 (40.3)	1287 (42.1)
Hypertension	2434 (81.9)	2510 (82.1)
Malignancy	4712 (15.9)	460 (15.0)
Depression	687 (23.1)	655 (21.4)
Liver disease	397 (13.4)	458 (15.0)
Laboratory values (median (IQR))*:
eGFR at admission (mL/min/1.73 m2)	55.1 (34.2-84.4)	55.8 (36.6-83.5)
Creatinine (mg/dL)†	1.5 (1.1-2)	1.5 (1.2-2)
Nadir creatinine (mg/dL)†‡:	1.1 (0.8-1.5)	1.1 (0.8-1.5)
Sodium (mmol/L)	138 (135-141)	138 (135-141)
Potassium (mmol/L)	4.2 (3.8-4.6)	4.2 (3.8-4.6)
Chloride (mmol/L)	102 (98-106)	102 (98-106)
Bicarbonate (mmol/L)	24 (21-27)	24 (21-27)
Anion gap (mmol/L)	12 (10-14)	12 (10-15)
Blood urea nitrogen (mg/dL)†	28 (18-42)	28 (19-40)
White blood cell count (×1000/μL)	9.8 (7.2-13.5)	9.8 (7.2-13.8)
Hemoglobin (g/dL)	10.7 (9-12.3)	10.5 (8.9-12.3)
Platelet count (×1000/μL)	202.5 (149-266)	200 (144-266)
Exposures before acute kidney injury:
Contrast in prior 72 hours	586 (19.7)	623 (20.4)
Cardiothoracic surgery in prior 72 hours	110 (3.7)	135 (4.4)
ACE inhibitor/ARB in prior 72 hours	626 (21.1)	670 (21.9)
NSAID in prior 72 hours	403 (13.6)	388 (12.7)
PPI in prior 72 hours	684 (23.0)	677 (22.1)
Study timing (median (IQR)):
Time from admission to randomization (hours)	50.7 (28.8-104.3)	50.1 (29.8-101.0)
Time from AKI to randomization, hours	0.5 (0.2-1.3)	0.5 (0.2-1.2)
Alert distribution (median (IQR)):
Unique providers reached	7 (4-13)	7 (4-13)

ACE=angiotensin converting enzyme; AKI=acute kidney injury; ARB=angiotensin receptor blocker; COPD=chronic obstructive pulmonary disease; eGFR=estimated glomerular filtration rate; ICU=intensive care unit; IQR=interquartile range; NSAID=non-steroidal anti-inflammatory drug; PPI=proton pump inhibitor.

Values at randomization unless otherwise specified.

Blood urea nitrogen 1 mg/dL=0.357 mmol/L; creatinine 1 mg/dL=88.42 μmol/L.

Values in the 48 hours before randomization.

Patients randomized to acute kidney injury alerts were more likely to receive an order for intravenous fluids (38.4% alert arm v 34.8% usual care arm, absolute difference 3.8% (95% confidence interval 1.4% to 6.2%)), urinalysis (17.0% v 14.9%, 1.9% (0.1% to 3.7%)), and to have serum creatinine measured (87.2% v 85.2%, 1.8% (0.1% to 3.6%)) within 24 hours of randomization (table 2; stratified analyses, supplemental tables 7-12). Randomized patients were also more likely to have documentation of acute kidney injury in the problem list (70.0% alert arm v 63.0% usual care arm, absolute difference 7.0% (95% confidence interval 4.6% to 9.3%)) during the hospital admission. No significant differences were found in the frequency of intravenous contrast administration or recording of urine output between the arms at 24, 48, or 72 hours after randomization.

Table 2

Process outcomes stratified by study arm. Unless otherwise specified, outcomes reflect proportion completed within 24 hours of randomization. Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute differences are shown as percent (95% confidence interval) unless otherwise specified

Variables	Usual care (n=2971)	Alert (n=3059)	Absolute difference
Diagnostic and therapeutic investigation:
Intravenous fluid drip order	1034 (34.8)	1174 (38.4)	3.8 (1.4 to 6.2)
Intravenous fluid bolus order	339 (11.4)	397 (13.0)	1 (−0.4 to 2.3)
Urinalysis	444 (14.9)	519 (17.0)	1.9 (0.1 to 3.7)
Urine output measurement	2130 (71.7)	2242 (73.3)	1.7 (−0.6 to 4)
Subsequent creatinine measurement (28 hours)	2532 (85.2)	2666 (87.2)	1.8 (0.1 to 3.6)
Kidney consultation (within 14 days)	710 (23.9)	727 (23.8)	0.1 (−2.1 to 2.2)
Nephrotoxin use:
Contrast exposure	102 (3.4)	115 (3.8)	0.3 (−0.6 to 1.2)
Aminoglycoside exposure	19 (0.6)	14 (0.5)	*
Aminoglycoside exposure among those alreadyreceiving aminoglycoside (n=40)	8/20 (40.0)	5/20 (25.0)	*
NSAID exposure	166 (5.6)	144 (4.7)	−1 (−2 to 0.1)
NSAID exposure among those already receivingNSAIDs (n=600)	129/312 (41.3)	103/288 (35.8)	−6.7 (−14.4 to 1.0)
ACEi/ARB exposure	425 (14.3)	424 (13.9)	−0.1 (−1.8 to 1.6)
ACEi/ARB exposure among those alreadyreceiving ACEi/ARB (n=1093)	326/532 (61.3)	331/561 (59.0)	−2 (−7.8 to 3.8)
Administrative:
AKI documentation (at end of encounter)	1871 (63.0)	2141 (70.0)	7.0 (4.6 to 9.3)
Median (IQR) hospital direct costs ($)	10 300 (5400-21 900)	10 600 (5400-22 800)	400 (229.7 to 1029.7)
Median (IQR) hospital total costs ($)	19 100 (9900-42 500)	20 100 (10 200-43 600)	740.5 (287.6 to 1768.7)

ACEi=angiotensin converting enzyme inhibitor; ARB=angiotensin receptor blocker; AKI=acute kidney injury; NSAID=non-steroidal anti-inflammatory drug; PPI=proton pump inhibitor.

Too few events for reliable estimates.

Primary outcome: progression of acute kidney injury, dialysis, or death

The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67; table 3 and stratified analyses supplemental tables 13-18). No evidence of a change was seen in the relative effect across intervention groups over time (P=0.90 for interaction). Time to event analyses for the composite outcome and for each component and each hospital are shown in supplemental figures S3-S6 and were of similar magnitude.

Table 3

Primary and secondary outcomes of interest.* Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute difference is shown as percent (95% confidence interval) except for duration of AKI and maximum AKI stage achieved, in which odds ratios (95% confidence interval) are reported

Variables	Usual care (n=2971)	Alert (n=3059)	Absolute percentage difference
Primary outcome:
AKI progression,† dialysis or death at 14 days	622 (20.9)	653 (21.3)	0.6 (−1.4 to 2.7)
Secondary outcomes:
AKI progression† at 14 days	461 (15.5)	487 (15.9)	0.5 (−1.3 to 2.3)
Dialysis at 14 days	93 (3.1)	106 (3.5)	0.3 (−0.6 to 1.1)
Death at 14 days	265 (8.9)	272 (8.9)	−0.3 (−1.7 to 1.1)
Discharge to home	1494 (50.3)	1503 (49.1)	−1.4 (−4 to 1.1)
Discharge to inpatient or home hospice	165 (5.6)	173 (5.7)	0.06 (−1.1 to 1.2)
Median length of stay (IQR; days after randomization)	4.2 (2.1-9.3)	4.3 (2.2-9.2)	0.1 (−0.2 to 0.3)
Duration of AKI (days):			0.9 (0.8 to 1)
<2	2108 (71.0)	2239 (73.2)
2-<7	722 (24.3)	700 (22.9)
≥7	141 (4.7)	120 (3.9)
Max AKI stage achieved:			1 (0.9 to 1.1)
0	5 (0.2)	6 (0.2)
1	2302 (77.5)	2376 (77.7)
2	403 (13.6)	412 (13.5)
3	261 (8.8)	265 (8.7)

Eleven individuals who were randomized never developed AKI, owing to laboratory errors in creatinine measurement, allowing patients to meet AKI criteria, which were subsequently revised lower after enrollment.

Progression of acute kidney injury (AKI) was defined as achieving a higher Kidney Disease: Improving Global Outcomes (KDIGO) AKI stage than the one present at the time of randomization, dialysis, or both.

Assessment of heterogeneity of treatment effect

Evidence of heterogeneity of treatment effect was found across the six hospitals (P=0.05; fig 2). Some sites suggested a trend towards a protective effect of alerts, such as hospital 3 (relative risk 0.80, 0.62 to 1.04, P=0.09), but others showed potential harm from alerts. In a post hoc analysis, this heterogeneity was not accounted for by patient characteristics, but by whether the hospitals were stratified as “teaching” or “non-teaching.” In particular, there appeared to be harm from alerts at the two non-teaching hospitals in the study (n=765), where 24.3% of patients in the alert arm compared with 16.3% in the usual care arm met the primary outcome (relative risk 1.49, 1.12 to 1.98, P=0.006). No such effect was found at the four teaching hospitals (0.97, 0.87 to 1.07, P=0.54), with P=0.006 for interaction comparing teaching with non-teaching hospitals.

Fig 2

Primary and secondary outcome events, stratified by hospital type. Error bars are 95% confidence intervals of the observed proportion of events. AKI=acute kidney injury

The differences in the primary outcomes observed in the non-teaching versus teaching hospitals seemed to be driven by death as opposed to dialysis or progression of the acute kidney injury. We noted evidence of heterogeneity of the association between alerts and death across the enrolled hospitals (P=0.05). Again, alerts were associated with a significantly higher risk of death at 14 days in the non-teaching hospitals, where 59 (15.6%) patients in the alert group versus 33 (8.6%) patients receiving usual care met this outcome. The relative risk of death was 1.82 (1.22 to 2.72, P=0.003) in the non-teaching hospitals compared with a relative risk of 0.89 (0.74 to 1.06, P=0.18) at the teaching hospitals (P=0.001 for interaction).

Secondary outcomes and process measures

The frequency of kidney consultations was similar between the two arms. Within 14 days of randomization, 23.8% of the alert group and 23.9%of the usual care group had received a consultation (absolute difference 0.1% (95% confidence interval −2.1% to 2.2%)). No evidence of heterogeneity of this effect was seen across the study hospitals (P>0.99). Similarly, in time to event analysis, no increased rate of kidney consultations was found across the two arms (hazard ratio 1.00 (95% confidence interval 0.90 to 1.11). No significant difference was seen in the rate of discharge to hospice overall or in the non-teaching hospitals.

Subgroup analyses

Prespecified subgroup analyses are shown in figure 3. The effect of alerting was largely similar between medical and surgical patients, ICU and non-ICU patients, by age, race, and sex, and across baseline creatinine values.

Fig 3

Prespecified subgroup analyses show similar alert effect across a diverse array of patient characteristics. Diamonds reflect relative risk, with bars showing 95% confidence interval. Creatinine 1 mg/dL=88.42 μmol/L. ICU=intensive care unit

Sensitivity and post hoc analyses

We examined the adjusted relative risk of the primary outcome accounting for a variety of baseline factors individually and stratified into teaching and non-teaching hospitals (supplemental table 19). The adjusted relative risks were broadly similar, although the adjusted risk of death in the non-teaching hospitals attenuated to 1.58 (95% confidence interval 1.08 to 2.31) and in the teaching hospitals to 0.85 (0.72 to 1.01).

We also performed a formal mediation analysis to determine if certain actions taken after alerting might have mediated the adverse outcomes in the non-teaching hospitals (supplemental table 20).18 The use of intravenous fluid (in drip or bolus form, in high chloride or more physiologic form) did not appear to mediate the adverse effect of the alert. Other proxies of volume overload (respiratory rate, oxygen saturation, and the change in these parameters) similarly did not show significant effect mediation. Exposures to medication, kidney consultations, and the percent of alerts that went to attending versus other providers also showed no significant effect mediation. Finally, the burden of other electronic health record alerts (such as those for sepsis or interactions between drugs) did not appear to mediate the deleterious effects of alerts for acute kidney injury.

We also examined the risks of two components of our primary outcome, death and progression of acute kidney injury, in the period two years before the start of this trial in each hospital (fig S7). Broadly, these risks were similar to those of the control group in the randomized portion of the study. For example, two years before our study started, the risk of death at 14 days after acute kidney injury in the non-teaching hospitals was 8.6%, similar to the risk of death for the control group of 8.6%, and significantly lower than the risk of death of 15.6% for the intervention group.

Discussion

Principal findings

In this multicenter randomized trial of patients with acute kidney injury, the provision of an electronic health record alert to providers had no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effect was seen across study hospitals, however, with signals for harm emerging in the two non-teaching hospitals. We cannot be sure whether the observed differences in effect were due to the presence of teaching services or other practices that might differ between the hospitals in our study. The fact that some process measures (such as orders for intravenous fluids and urinalysis) occurred more often in the alert arm suggests that the alerts were being noticed by providers and that they could change provider behavior, but the magnitude of these changes was relatively small.

Reconciliation of those findings with the fact that the risk of death was higher in the alert arm in the non-teaching institutions is difficult. This increased risk might have been due to random chance, although the P value for the alert:death association of 0.003 suggests that chance is unlikely. Furthermore, adjustment for a variety of baseline factors did not ablate the observed association, suggesting that imbalance in baseline covariates is insufficient to explain the association. Additionally, in our prior, single center, acute kidney injury alert trial (conducted at a medical center not involved in this study), we noted a numerically higher, although statistically insignificant, risk of death in the alert versus the usual care arm. The risk of death at 14 days in that trial was 7.1% in the usual care group and 7.7% in the alert group (P=0.58).19 To date, that study is the only previous trial of acute kidney injury alerts that employed randomization at the patient level. Finally, we examined historical death rates of acute kidney injury in the study hospitals and found them to be quite similar to the rates seen in the usual care arm of our trial. Taken together, these data suggest that there might be true harm associated with the provision of alerts for acute kidney injury in certain situations. In view of the secondary nature of these analyses, the results should be considered hypothesis generating, and any attribution to the “non-teaching” nature of the hospital is premature.

As alerts of this type are in widespread clinical use, we sought to determine if any factor downstream of randomization drove these results. We speculated that an increase in fluid administration in the alert group might lead to harm through volume overload, but accounting for fluid administration did not attenuate the observed effect. We further considered that certain types of providers (eg, medical doctors v physician’s assistants) might respond to acute kidney injury differently, but there was no indication that the effect changed according to the type of provider receiving the alert. Furthermore, we assessed whether alert fatigue might be to blame for poorer outcomes in the alert group, but accounting for the frequency of other electronic health record alerts did not change the observed association with the alert for acute kidney injury. Possibly, while no single factor mediated the adverse effect in every patient, a constellation of factors resulted in the observed effect. We are left without a satisfying unifying explanation for the potential harm. Alerts might distract providers from other, more clinically important, patient care. Alternatively, in the larger teaching institutions, systems exist to protect patients from certain harms (like inappropriate drug dosing), which are not present in the smaller hospitals (regardless of academic status). Finally, we should consider that alerts could compel providers to do something, even if it is not clinically indicated, perhaps owing to fears of medicolegal consequences if they do not act.

Comparisons with other studies

Alerts of all kinds, from electronic to auditory to visual, have proliferated in medical care with the best of intentions under assumptions that risk is minimal.20 Notably, alerting systems for acute kidney injury have already been widely adopted across various health systems. The published literature suggests that alerts are often effective in changing processes of care, but their effect on clinical efficacy has been mixed.21 For example, a stepped wedge randomized trial of 24 059 episodes of acute kidney injury in the UK found a decreased length of stay, but no change in overall mortality risk between the control and intervention periods.22 In contrast, a before/after study of a clinical decision support system for acute kidney injury among 64 512 patients found a significant reduction in mortality risk from 10.2% to 9.4% from before to after the intervention.23 This design could not account for secular trends in the care of patients with acute kidney injury, although the mortality risk in patients with non-acute kidney injury did not change over the study period.

This trial cannot show a clear efficacy of alerts, but the phenomenon of “alert fatigue”—a tendency to disregard even important alerts owing to their abundance—is increasingly recognized.24 25 26 27 The economic costs of alerts might be minimal after the initial alert build, but the costs of provider attention, burnout, and potential overtreatment might be substantial and clearly deserve additional study.28 This trial was conducted under a waiver of informed consent, based on the prevailing assumption that informational alerts for acute kidney injury pose minimal risk to the patient and the recognition that obtaining the consent of those randomized to the control group would invalidate the study results. Unfortunately, for the alert deployed in this study, the assumption of minimal risk proved incorrect. These results do illustrate, however, that clinical trials of seemingly benign interventions, such as alerts, require rigorous evaluation and should not be implemented without robust evidence of safety and efficacy.

Strengths and limitations of this study

Limitations of the study include randomization at the patient level, allowing for providers potentially to learn how to better identify acute kidney injury throughout the course of the trial. This contamination would tend to bias the results toward the null hypothesis that the alerts did not affect the primary outcome, but an analysis accounting for the duration that the alerts had been in effect at each hospital found no evidence of this. Furthermore, our previous research has shown that patient level factors, such as baseline creatinine value and sex, contribute to alert benefit.29 Additionally, the alert was largely informational. Although the alert provided a link to an acute kidney injury order set, it provided no patient-specific recommendations (such as identifying current nephrotoxic drug treatment)—a key feature of the successful alert for pediatric acute kidney injury piloted by the nephrotoxic injury negated by just-in-time action (NINJA) program, and by certain “care bundles” for acute kidney injury that have been used in the UK.22 30 The heterogeneity of acute kidney injury might require more personalized interventions. Furthermore, the alert was sent only to certain care providers, notably excluding nurses and pharmacists, who might specifically benefit from receipt of the alert. The alert did not use urine output criteria, as many ward patients do not have reliable estimates of urine output. This might have meant that the study population was not fully representative of all patients with acute kidney injury. Finally, the alert was conducted across six hospitals within a single large health system in northeastern US, limiting generalizability. However, the hospitals represent a diverse mix of patients and care models, suggesting that these results might apply more broadly.

The strengths of the study include its large size with an ability to detect clinically meaningful changes in hard endpoints, its enrollment across a diverse group of hospitals, and the use of a fully automated enrollment and randomization system that could pave the way for future trials of alerting and clinical decision support systems.

Conclusions

In this large, randomized, multicenter study, we found that, overall, the provision of an automated, electronic alert for acute kidney injury had a modest effect on process measures and no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effects across hospitals was found, with the concerning finding of a significantly increased risk of death in the alert group in non-teaching hospitals. This study argues against the implementation of informational alerts for acute kidney injury and for a reconsideration of the alerts currently used.

Acute kidney injury occurs in 15% of patients in hospital and is associated with poor outcomes, yet fewer than 50% of these results are documented in the electronic health record, and best practices for acute kidney injury are performed inconsistently

Limited data from randomized trials exist to illustrate the clinical benefit of alerts for acute kidney injury or demonstrate their efficacy on patient outcomes

This randomized controlled trial found only a modest effect of electronic alerts on process methods for acute kidney injury and no overall effect on the risks of death, dialysis, or disease progression in patients in hospital with acute kidney injury

The heterogeneity of effect of the alert across hospitals argues for a more rigorous evaluation of safety and efficacy of both current and future alerting systems for acute kidney injury