Implementation of automated early warning decision support to detect acute decompensation in the emergency department improves hospital mortality.

What this paper adds…

Early warning alert scores based on clinical and physiological data have been shown to be predictive of risk of decompensation in patients in the emergency department (ED). However, the effect of using these systems to influence patient outcomes in the ED has been difficult to ascertain consistently. Part of the difficulty in comparing previous studies are the various systems chosen and faulty implementation models.

The unique coupling of our automated system with the vital sign-based National Early Warning Score score helped to reduce delay and error in the attempt to identify and stabilise decompensating patients. Our study demonstrates that using an automated decision support surveillance and alert system to trigger alerts for ED patients reduced both adjusted hospital mortality and hospital length of stay.

Introduction

Early detection of clinical decompensation among patients in the hospital is important to minimise mortality rates.1 2 Often, deviation in vital signs precede clinical decompensation and correlate with poor outcomes.3 4 Vital sign-based early warning scoring systems, such as the National Early Warning Score (NEWS),5 demonstrate improved early detection of all-cause clinical decompensation. These systems are common and disseminated at national scales, such as the National Health Service in the UK and are recommended by US federal agencies.5 6 However, there is scant comparative literature on the effect of early warning systems on patient outcomes in complex settings like EDs.7 Barriers to successful implementation have included poor compliance with recording vitals or incorrect calculation of warning scores, leaving an opening for implementation of automated, decision support systems.7

We have previously described an improvement in activation of rapid response teams and lactate measurements with ward-based implementation of this cloud-based system.8 The goal of this investigation was to study the effectiveness of this decision support system to detect and intervene on clinical decompensation in the ED by evaluating reductions in hospital mortality and length of stay (LOS).

Methods

In the ED at Baylor St. Luke’s Medical Center (Houston, Texas, USA), an automated, real-time decision support software system (Decisio Health; Houston, Texas, USA) was installed to help calculate the National Early Warning Score (NEWS) for each patient. This vital sign-based score was automatically calculated by the Decisio system and displayed on a video screen above each patient’s bed, along with other clinical data such as laboratory values. If and when a vital sign(s) deviation occurred to the point that an overall NEWS score of ‘5’ was reached, an electronic text alert was sent the pagers of both the charge nurse and ED physician, prompting the performance of a rapid clinical assessment. The text of the alert was simply ‘NEWS Alert Score of X, Bed XX.’ Prior to initiation of this system, both nursing and physician staff were trained on the notification process, the clinical significance of the NEWS scores, and expected action after alerts. If there was a confirmed concern for clinical worsening, the team could then respond with additional testing, interventions, escalation of care and specialty consultations. The provisional goal of these earlier interventions to was to ‘re-route’ the outcome trajectory of each patient, such as earlier identification of need for or prevention of critical care admission.

Outcomes data of patients admitted to the ED and subsequently to the hospital between 1 June 2017 and 31 May 2019 were examined. Based on a pre-intervention and post-intervention comparison methodology, the control months were 1 June 2017 to 31 May 2018, and the intervention months were 1 June 2017 to 31 May 2018. We compared monthly aggregate risk-adjusted mortality rates, arithmetic hospital LOS and geometric hospital LOS between intervention and control months (geometric values can represent a central value of a group of data points, as it accounts for outliers). Risk adjusted values were calculated as observed versus expected values (O/E) (Premier Healthcare Database (PHD), Charlotte, North Carolina, USA). O/E values are calculated by dividing the observed value for an outcome (such as mortality or LOS) by its expected value. O/E values greater than 1.0 indicate greater than expected mortality and LOS.

The PHD is a national database of more than 2400 hospitals and is used as a quality benchmarking tool. Data from the PHD is used to calculate O/E values using a validated, proprietary formula which takes into account factors such as demographics, current illnesses, chronic comorbidities and laboratory values. Our O/E values were based on a comparison of the top 16th percentile of national hospital performers.

Data analyses

A series of Wilcoxon Mann-Whitney U tests were used to examine differences in O/E mortality, arithmetic LOS and geometric LOS between control months and intervention months. Cohen’s d was subsequently calculated for each outcome. Local Institutional Review Board approval (IRB) was obtained for this analysis; a patient consent waiver was provided by the IRB. Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research. Funding was provided by an internal Baylor College of Medicine Precision Medicine/Population Health Initiative Grant (no grant number).

Results

The control group consisted of 11 150 admissions (across a period of 12 months) and the intervention group consisted of 8363 admissions (across a period of 9 months).

Both the O/E arithmetic and geometric hospital LOS were significantly lower for admissions during the intervention period (mean=1.47, SD=0.05 and mean=1.17, SD=0.05, respectively) compared with the control period (mean=1.54, SD=0.05 and mean=1.25, SD=0.04, respectively), both p’s <0.01 and Cohen’s ds=1.40 and 2.00, respectively (table 1). With regards to the effect on O/E mortality, although there was not a statistically significant difference between intervention and control periods (mean=1.00, SD=0.18 and mean=1.14, SD=0.14, respectively) (z=1.67, p=0.08), O/E mortality was 12% lower during intervention months (as compared with control months) and the effect size was large (d=0.87).

Table 1

Observed vs expected mortality and length of stay during the control and intervention periods and comparisons between the two using the Wilcoxon Mann-Whitney U test

	Control (n=12 months)	Intervention (n=9 months)	% Improvement	P value	Cohen’s d
	Mean (SD)	Mean (SD)
Mortality O/E**	1.14 (0.14)	1.00 (0.18)	12	0.09	0.87
Arithmetic LOS O/E††	1.54 (0.05)	1.47 (0.05)	4.5	0.004	1.40
Geometric LOS O/E††	1.25 (0.04)	1.17 (0.04)	6.5	0.001	2.00

*Based on data from 11 132 patient admissions for control months and 8346 for intervention months.

†Based on data from 11 101 patient admissions for control months and 8313 for intervention months.

LOS, length of stay; O/E, observed versus expected values.

Discussion

The reduction in O/E LOS was significant, and although the reduction in adjusted O/E mortality did not quite reach a p value of 0.05 (p=0.09) the effect size was large (d=0.87) indicating a substantial difference. Acknowledging likely confounders, we feel the clinical correlation between NEWS score and LOS is relevant in that earlier interventions help to reduce decompensations and improved outcomes such as LOS—interventions such as earlier identification of sepsis or need for intensive care unit admission.

The ED is a highly complex, and unpredictable clinical environment, making an automated early warning system a critical adjunct and safety net for patients in these dynamic and busy departments.9 10 This study demonstrates that a real-time decision support software system that automates calculation of and notification for abnormal early warning scores was correlated with an improvement of hospital mortality and LOS for ED patients facing hospitalisation.

During the time of this investigation there were no other concurrent quality improvement efforts being conducted in the ED that would confound the results of this study. To further understand the impact of such systems, future directions of research could include a more in-depth analysis of alerts such as correlation of time between alerts and interventions, and specific diagnoses and interventions that resulted in the largest impacts on improved mortality and LOS. Also, next steps could include adding patient-matched control data and/or a control/secular ED department.

Limitations

Due to logistical constraints and operational mandates, there was not an opportunity to create a simultaneous control and intervention arm of the study. However, we were able to compare similar annual timeframes in an attempt to adjust for seasonal variations in ED admission patterns and diagnoses. Also, during the intervention period, there were 3 months when the system was not fully operational due to technical issues. This included issues such as pager malfunction, and trial-and-error selection of a NEWS score threshold that was operationally feasible to execute (given that lower NEWS score alert thresholds translated to an overwhelming number of false positive cases of decompensation and untenable number of additional clinical assessment). We therefore excluded data from this time period. Our ED does not perform services for trauma, children or pregnant women so these results would not be generalisable to such settings.

The study demonstrates, that in a complex, dynamic clinical environment such as an ED, an automated decision support software system is an effective tool to implement a vital sign-based early warning score to change the outcomes of hospitalised patients. Next steps should include direct, prospective comparison of this system. Additionally, future efforts could aim to create a customised, non-proprietary warning score that takes into account additional clinical elements such as laboratory values, vital sign trajectory and comorbidities.