Data-driven approach to Early Warning Score-based alert management.

BACKGROUND: Increasing adoption of electronic health records (EHRs) with integrated alerting systems is a key initiative for improving patient safety. Considering the variety of dynamically changing clinical information, it remains a challenge to design EHR-driven alerting systems that notify the right providers for the right patient at the right time while managing alert burden. The objective of this study is to proactively develop and evaluate a systematic alert-generating approach as part of the implementation of an Early Warning Score (EWS) at the study hospitals.
METHODS: We quantified the impact of an EWS-based clinical alert system on quantity and frequency of alerts using three different alert algorithms consisting of a set of criteria for triggering and muting alerts when certain criteria are satisfied. We used retrospectively collected EHRs data from December 2015 to July 2016 in three units at the study hospitals including general medical, acute care for the elderly and patients with heart failure.
RESULTS: We compared the alert-generating algorithms by opportunity of early recognition of clinical deterioration while proactively estimating alert burden at a unit and patient level. Results highlighted the dependency of the number and frequency of alerts generated on the care location severity and patient characteristics.
CONCLUSION: EWS-based alert algorithms have the potential to facilitate appropriate alert management prior to integration into clinical practice. By comparing different algorithms with regard to the alert frequency and potential early detection of physiological deterioration as key patient safety opportunities, findings from this study highlight the need for alert systems tailored to patient and care location needs, and inform alternative EWS-based alert deployment strategies to enhance patient safety.

Introduction

Increasing adoption of electronic health records (EHRs) with integrated alerting systems is a key initiative for improving patient safety. Alerts are implemented in nearly every hospital in the USA and are included in all major commercial EHRs in order to call attention to a patient and notify providers of relevant clinical information, for example, abnormal vital signs and changing laboratory values.1–4 Considering the variety of clinical information, it remains a challenge to develop EHR-driven alerting systems that notify the right providers for the right patient at the right time. Without carefully designed filtering criteria for alert systems, the volume of alerts can be overwhelming to healthcare providers.5 In this context, filtering criteria refer to rules to prioritise alerts in order to present them adequately in a clinical setting.5 Filtering criteria can be applied to alert-generating algorithms as well as validation of the generated alerts.6 Alert-generating algorithms provide a data-driven approach that quantifies the count and frequency of alerts in a care delivery setting identifying opportunities for early detection of patient care needs.

Clinically meaningful and data-driven filtering criteria are especially critical for alerts triggered by Early Warning Scores (EWSs). EWSs provide a standardised method for quantifying a patient’s physiological condition and suggest interventions at certain critical thresholds.7 EWSs commonly use one or multiple physiological measures, calculate a dynamic score representing a patients’ health condition at the given time and convey risk using numerical values that indicate the need for assessment and interventions at the bedside.8–10 Since the elements that typically contribute to the EWSs (eg, heart rate, blood pressure) change over time based on the frequency of measurement, suboptimal filtering criteria can result in significant disturbance of clinical workflow due to the large number of EWS-triggered alerts.

The operationalisation of EWS-based alerts requires analysis of numerous factors that affect the accuracy, clarity and clinical relevance of alerts. Previous studies have examined the usability and patient outcomes associated with alert management. The proactive evaluation of alert management strategies prior to operationalisation of alerts in healthcare systems is unfortunately less common.11–17 The objective of this study is to proactively develop and evaluate a systematic approach for alert management as part of the implementation of an EWS in our health system: the Christiana Care Early Warning Score (CEWS).18 We use a proactive and data-driven approach to alert management rather than retroactively assessing the appropriateness of alerts and provider’s response.

In this paper, we will first provide an overview of the CEWS methodology and iterative implementation approach as a key patient safety initiative in our health system. Next, we compare various alert-generating algorithms regulating the CEWS-driven alerts and quantify their impact on patient-level and unit-level alert burden as well as the opportunity of earlier detection of physiological deterioration. We conclude with discussion of the key findings and their clinical implications.

Methods

Setting

Christiana Care Health System (Christiana Care), headquartered in Wilmington, Delaware is one of the largest healthcare providers in the USA with a total of >1100 patient beds across two hospitals (Christiana Hospital and Wilmington Hospital). It ranks 22nd for hospital admissions with over 53 000 admissions annually. The study population included adult inpatients (aged 18 years or older) with a record of being hospitalised at Christiana Hospital or Wilmington Hospital from December 2015 to July 2016.

Dataset

Data were derived retrospectively from the EHRs at Christiana Care. Emergency department, women and children units and intensive care units (ICUs) were excluded. Variables include patient and visit identifiers, clinical events, CEWS scores and all sets of vital signs. The dataset includes 4825 visits, 4328 patients, 476 clinical events and 285 910 records (vital signs and nursing assessments). Rows with likely erroneous values that fell outside reasonable clinical range (<0.1% of records) were eliminated.

Early warning system

Patients often exhibit signs of physiologic deterioration hours before harm events occur.19 Early detection of these warning signs may facilitate timely response to all-cause physiological deterioration and specific disease conditions such as sepsis with rising incidence and mortality rate.20 21 Christiana Care has developed and validated CEWS to predict and reduce all-cause physiological deterioration, including sepsis-induced deterioration. CEWS is a predictive algorithm that integrates seven physiological observations based on National Early Warning Score8 and eight clinical assessments derived from nursing assessments. CEWS takes values between 0 and 27.18 CEWS value of 0 is the best possible health condition, and higher CEWS values indicate higher risk for clinical events. Discrimination performance of CEWS was assessed by area under the receiver operating characteristic curve (AUROC). CEWS achieved an AUROC of 0.81 in the prediction of: (i) rapid response team (RRT) activations within 24 hours, and (ii) a composite event defined as RRT, Code Blue activation, transfer to higher level care (eg, transfer from medical unit to a stepdown unit22) or death within 24 hours.18 Based on analysis results with regard to risk of RRT, Code Blue activation, transfer to higher level care, and death within 24 hours, as well as clinical experts’ input, CEWS scores were categorised into four distinct ranges: 0–5, 6–9, 10–13 and 14 and above. The algorithm calculates an updated CEWS value every time a component of the score is entered into the EHR system. A CEWS score and corresponding individual CEWS components are carried forward up to 24 hours unless a new observation becomes available. If a clinical assessment component of CEWS is not measured within that timeframe, the system defaults to a passing value; however, that occurs very rarely due to regular nursing assessments at the study hospital that occur multiple times per day. In the case that a component is missing, CEWS is not able to be calculated and the value is considered missing and excluded from this analysis. Table 1 summarises the CEWS algorithm.

Table 1

Summary of the CEWS algorithm

Physiological observations	Weights associated with physiological observations
	3	2	1	0	1	2	3
Respiration rate (rpm)	≤8		9–11	12–20		21–24	≥25
Temperature (°C)	≤35.0		35.1–36.0	36.1–38	38.1–39.0	≥39.1
Systolic blood pressure (mm Hg)	≤90	91–100	101–110	111–219			≥220
Heart rate (bpm)	≤40		41–50	51–90	91–110	111–130	≥131
Blood oxygen saturation (%)	≤91	92–93	94–95	≥96
Level of consciousness				Alert			Verbal, pain, none
Supplemental oxygen		Yes		No

Frameworks

Based on clinical expert opinion supported by data analysis, a sudden increase of CEWS value ≥4 between two subsequent CEWS measurements, referred to as delta, was associated with higher risk of experiencing an RRT, Code Blue or in-hospital mortality within 24 hours of CEWS calculation. Alerts were defined as pop-ups that immediately interrupt the nurses’ clinical workflow independent of which patients’ chart the nurse is currently reviewing. With the goal of proactively investigating the CEWS-based alert management which uses an aggregate EWS (table 1), we quantify the number and frequency of CEWS-based alerts using three different data-driven alert-generating algorithms, referred as frameworks, applied on retrospective data from December 2015 to July 2016. Each framework has two components: (i) filtering criteria for triggering an alert and (ii) blocking criteria to mute alerts. Furthermore, based on clinical expert opinion, each CEWS and delta category combination is assigned to a specific risk level considering the risk of events within 24 hours, where the risk levels are defined as follows:

Low risk level includes CEWS 0–5 (with any delta value) and CEWS 6–9 with delta <4.

Medium risk level includes CEWS 6–9 with delta ≥4 and CEWS 10–13 with delta <4.

High risk level includes CEWS 10–13 with delta ≥4 and CEWS 14 and above with delta <4.

Critical risk includes CEWS 14 and above with delta ≥4.

In addition, each risk level is associated with a set of clinical recommendations ranging from reviewing the CEWS elements and trend to increasing the vital sign monitoring frequency or activating the RRT. Using the risk levels and clinical recommendations, we compared three frameworks representing different computerised alert-generating algorithms. The ability to implement in a standard EHR system was a design goal for the development of the frameworks.

Framework 1: conservative (minimum number of alerts)

In framework 1, nurses are alerted if a patient moves from a low risk or medium risk into a high or critical risk. There are no alerts for a movement into a lower risk level. There are two blocking criteria in place, assuming no increases to a higher risk level: (i) no alerts fire after the initial alert for high risk as long as the patients remain in the same risk level to avoid unnecessary alerting while the care team is providing increased frequency of monitoring and care interventions and (ii) due to criticality, there is no blocking in critical risk level. However, there are alerts and triggered tasks for every increase within high and critical levels. For low and medium risk levels, there are no alerts, but passive information display including CEWS value, trend and additional relevant clinical information as appropriate.

Framework 2: intermediate (intermediate number of alerts)

In framework 2, nurses are alerted if a patient moves from a low risk to a medium risk as well as if a patient moves from a low or medium risk into high or critical risk levels. There are no alerts for a movement into a lower risk level. Blocking criteria include: (i) no alerts fire after the initial alert for high risk level as long as the patients remain in the same risk level; (ii) no alerts fire after the initial alert for medium risk level as long as the patient remains in medium risk with delta <4 or moves between low risk with CEWS 6–9 and delta <4 and medium risk levels and (iii) there is no blocking in critical risk level.

Framework 3: liberal (maximum number of alerts)

In framework 3, nurses are alerted for every increase in risk level. There are no alerts for a movement into a lower risk level. Blocking criteria include: (i) no alerts fire after the initial alert for high risk levels as long as the patients remain in the same risk level; (ii) no alerts fire after the initial alert for medium risk level as long as the patient remains in medium risk with delta <4 or moves between low risk with CEWS 6–9 and delta <4 and medium risk levels, (iii) no alerts fire after the initial alert for low risk as long as the patient remains in low risk level and (iv) there is no blocking in critical risk level. All three frameworks are summarised in table 2.

Table 2

Summary of the alert-generating frameworks, associated labels and rules

Framework	Label of the framework	Rules of the framework
1	Conservative (minimum number of alerts)	No alerts for a movement into a lower risk level. Blocking: assuming no increases to a higher risk level, no alerts after the initial alert for high risk as long as the patients remain in same risk level. No alerts for low and medium risk levels. No blocking in critical risk level.
2	Intermediate (intermediate number of alerts)	No alerts for a movement into a lower risk level. Blocking: (i) no alerts after the initial alert for high risk level as long as the patients remain in the same risk level; (ii) no alerts after the initial alert for medium risk level as long as the patient remains in medium risk with delta <4 or moves between low risk with CEWS 6–9 and delta<4 and medium risk levels. No blocking in critical risk level.
3	Liberal (maximum number of alerts)	No alerts for a movement into a lower risk level. Blocking: (i) no alerts after the initial alert for high risk levels as long as the patients remain in the same risk level; (ii) no alerts after the initial alert for medium risk level as long as the patient remains in medium risk with delta <4 or moves between low risk with CEWS 6–9 and delta <4 and medium risk levels, (iii) no alerts after the initial alert for low risk as long as the patient remains in low risk level. No blocking in critical risk level.

Statistical analysis

The frameworks were evaluated using descriptive statistics in three units at the study hospital, including two stepdown and one medical unit, referred as stepdown unit 1, stepdown unit 2 and medical unit, respectively. The three units included a total of 4596 visits, 3949 patients and 476 adverse events during the study period. At the unit level, stepdown unit 1, stepdown unit 2 and medical unit saw 1485/1959/1381 visits, 1324/1836/1168 unique patients and experience an event rate of 0.98/0.67/0.36 events per day, respectively. Stepdown unit 1 is a care location primarily focusing on acutely ill patients and average CEWS in this unit is higher than the other two units. Between January 2015 and July 2016, stepdown unit 1 had an average of 34.16 patients per day (SD=2.28), and average CEWS value 6.84 (SD=3.11). Stepdown unit 2 is a care location with a slightly lower severity level (average CEWS of 5.36, SD=3.19) with an average of 23.17 patients per day (SD=2.11). Medical unit has the lowest severity care (average CEWS of 3.62, SD=2.76) and, on average the most patients per day (40.58, SD=4.26) of all three units. In this study, frameworks are compared by the median and IQR of alerts per day and number of patients alerted on per day. The statistical analysis uses only descriptive analysis, and does not include advanced analytics methods to draw conclusions on which framework is superior in different care settings.

Results

We compared the alert burden and the opportunity of early recognition of all-cause as well as sepsis-induced physiological deterioration informed by eight unique categories created using CEWS value and delta (table 3).

Table 3

Eight unique categories using Christiana Early Warning Score (CEWS) ranges and delta values, median number of patients per day and percentage of patients who experienced a rapid response team (RRT), Code Blue or a composite event within 24 hours of CEWS observation using retrospective data from December 2015 to July 2016

CEWS	Delta	Median patients/day	IQR	Composite event (%)	RRT (%)	Code Blue (%)
0–5	<4	621	61	1.23	1.15	0.05
	≥4	29	8	2.45	2.30	0.11
6–9	<4	253	28	2.99	2.75	0.12
	≥4	49	11	4.12	3.73	0.20
10–13	<4	51	8	6.37	5.38	0.29
	≥4	25	6	7.79	6.79	0.36
≥14	<4	4	3	14.78	11.79	0.37
	≥4	4	3	14.67	11.64	0.71

Table 3 presents the median number of patients per day that fall into each CEWS and delta category, as well as the percentages of patients who experienced an RRT, Code Blue or a composite of the three measured adverse events within 24 hours after the CEWS observation. For patients with multiple subsequent observations in the same risk level, only the first CEWS observation is counted to determine the median number of patients per day and percentage of observations followed by an event. Table 3 highlights that higher CEWS values and delta ≥ 4 are associated with fewer patients, but a higher risk of events within the next 24 hours. Furthermore, we evaluated the conservative, intermediate and liberal frameworks using retrospective data derived from the three considered units. At the unit level, each framework was evaluated based on median and IQR of alerts per day, number of unique patients with alerts per day, sensitivity and positive predictive value of the frameworks (table 4).

Table 4

Median and IQR of alerts per day, unique patients with alerts, sensitivity and positive predictive value associated with the three frameworks in three units at the study hospital: stepdown unit 1, stepdown unit 2 and medical unit

	Framework 1 (conservative)	Framework 2 (intermediate)	Framework 3 (liberal)
Number of alerts per day
Stepdown unit 1	16 (11–21)	37 (31–44)	61 (54–68)
Stepdown unit 2	4 (2–6.25)	12 (8–15)	24 (20–27)
Medical unit	2 (0–3)	6 (4–9)	19 (16–23)
Total	23 (19–29)	55 (48.75–63)	104 (97.75–113)
Number of patients per day
Stepdown unit 1	8 (6–10)	18 (16–20)	26 (25–28)
Stepdown unit 2	3 (2–4)	7 (6–9)	13 (11–15)
Medical unit	1 (0–2)	4 (3–6)	13 (11–15)
Total	12 (10–14)	29 (26–32)	52 (49–55)
Sensitivity
Stepdown unit 1	0.074	0.133	0.178
Stepdown unit 2	0.102	0.148	0.191
Medical unit	0.040	0.074	0.151
Positive predictive value
Stepdown unit 1	0.065	0.050	0.040
Stepdown unit 2	0.094	0.052	0.032
Medical unit	0.106	0.064	0.041

As shown in table 4, unit-level differences are observed when comparing the application of the same framework with different units. Considering the same framework, care location severity determines the number of alerts generated, that is, care location with higher severity experience higher number of alerts per day. For example, using framework 1 stepdown unit 1 would be exposed to a median of 16 alerts per day, stepdown unit 2 would be exposed to four alerts per day and the medical unit would have a median of 2 alerts per day. Furthermore, when applying different frameworks to the same unit there is a significant increase in the number of alerts as we move from conservative to liberal frameworks. For example, in the stepdown unit 1, framework 3 (liberal) results in 281% increase in the median number alerts per day compared with framework 1 (conservative), and a 65% increase compared with framework 2 (intermediate). Other considered units show similar trends when the liberal framework is compared with conservative and intermediate framework (table 4). The median number of alerts and median number of patients with alerts per day can be used as surrogate measures to quantify workload burden associated with adopting a given alert framework.

Furthermore, a trade-off analysis can identify the impact of moving between frameworks at the unit level. In this context, we quantified the results of switching from one framework to another for each considered unit by comparing impact on sensitivity and positive predicted value of a composite event within 24 hours. Table 4 shows the sensitivity and positive predictive value associated with each framework in the considered units. Results in table 4 show that switching from conservative to liberal framework increases sensitivity in all three units, but at the cost of decreased accuracy and increased false alarm rate, indicated by the positive predictive value. It should be noted that CEWS is an aggregated acuity metric to early identify physiological deterioration and as such sensitivity and positive predictive value need be considered in this context. As such, a patient who does not have a clinical event may not necessarily be interpreted as a false alarm since a care provider team can intervene at any point in the deterioration process, and potentially prevent events from occurring.

At the patient-level, comparing different frameworks can inform how many alerts the provider team would be exposed to prior to patient experiencing a deterioration event, as such providing a surrogate for quantifying the potential time to earlier detection as a key patient safety opportunity. Figure 1 presents visualisations of each of the three frameworks at the patient level for an individual patient from stepdown unit 1. In this example, the patient experienced RRT activation within 1 hour following the last recorded measurement. Each CEWS measurement and alert that would have been generated using each of the frameworks are plotted.

Figure 1

Three frameworks and associated alerts for a patient hospitalised in stepdown unit 1. Blue line represents the Christiana Early Warning Score (CEWS) observations from admission until 1 hour before the rapid response team activation. Vertical lines represent alerts where black represents critical, red represents high risk, yellow represent medium risk and green represent low risk alerts. Using framework 3 (liberal) would result all black, red, yellow and green alerts to be triggered. Using framework 2 (intermediate), only black, red and yellow alerts would have been triggered. Using framework 1 (conservative), only red and black alerts would have been triggered.

Using framework 1 (conservative), the earliest alert would provide approximately 12 hours of warning prior to the RRT activation, which is the time difference between the first high risk alert (red line) and the RRT activation. Using framework 3 (liberal) would have resulted in an alert approximately 10 hours earlier than framework 2 and 12 hours earlier than framework 1. In this particular example, framework 2 (intermediate) would result in one additional alert compared with framework 1 and framework 3 would result in one additional alert compared with framework 2. While a single patient may appear to benefit from earlier alerts, total alert burden should be evaluated with a larger sample of patient population.

Discussion

Continuously monitoring the functionality and use of alerts is common in health information technology, but a proactive approach to design and assessment of alert-generating algorithms is rare. Lack of knowledge regarding when and how to present alerts to providers has impeded alert operationalisation in clinical practice.23 It remains a challenge to implement EWS-triggered alerts effectively without inducing ‘alert fatigue’24 defined as a process ‘…wherein clinicians may inadvertently ignore clinically useful alerts, thus diminishing the systems’ effectiveness and possibly leading to serious adverse consequences for patients’.25 There are significant data indicating the burden of alerts and unintended consequences associated with alert burden.11 26 27 Research exists regarding the number of alerts providers receive through the EHR system but, to date, no best practices are identified.

In this article, we focused on providing a data-driven and proactive approach to evaluate EWS-based alert-generating algorithms using an aggregate score-based EWS in a healthcare delivery setting. We presented three frameworks for operationalising alert systems using an EWS that is implemented in the study hospitals. While EWSs promise a data-driven patient condition surveillance tool,28 systematic proactive approaches are needed to assess thoughtful EWS-based alert management strategies. The ultimate purpose of this work is to inform EWS-based alert systems to support the efficient and timely delivery of care. When EWSs are not used to their full advantage, errors in timely diagnosis and care may occur for a complex, fast-moving disease like sepsis. Specific to sepsis, such systems have demonstrated increased adherence with sepsis resuscitation and management bundle elements.29 Well-designed and appropriately used alert-generating algorithms coupled with strong clinical processes can provide clinical decision support tools and facilitate communication among providers.

Despite increasing implementation of EWS-triggered alert systems, the appropriateness of alert frequency or subsequent end-user decision making is not well understood. Detailed evaluations rarely occur due to the time and resources associated with chart review to determine alert response appropriateness.30 The proposed methodical approach provided unique insight into the tools, resources and processes needed to design and proactively assess alert-generating algorithms from both a unit and individual patient perspective. The presented frameworks demonstrated the inherent risks in health information technology as studies have documented mixed results in the EHR’s ability to detect and prevent errors.31 32

Our study has some limitations. Foremost, we relied on data collected retrospectively from the EHR system to identify the count and frequency of alerts that would have been generated using different frameworks. In clinical practice, the patients may have received appropriate care, which may have avoided harm events and limited the value of the alerting system. In addition, alerts can be overridden by clinical providers when appropriate. Prospective assessment of alert-generating algorithms can capture the impact of alerts on the workflow better compared with retrospective evaluation. Another limitation is that we quantified the count and frequency of alerts comparing three different frameworks applied in three care units which does not represent the entire health system. Future studies can focus on expanding the sample of units, patients and considered frameworks to obtain a better understanding of the alert-generating algorithms on workflow and patient safety. Finally, although the alert framework may be designed to improve patient safety, it cannot be effective if providers do not change their behaviour in response to relevant alerts. Ensuring alert system sustainability and maintaining quality and patient safety depends on our ability to evaluate both alerts and subsequent clinical actions in the context of an actual patient care episode.33

Conclusion

Early recognition of all-cause physiological deterioration in the healthcare delivery system, including sepsis-induced deterioration, requires data-driven approaches to re-engineer the way frontline providers detect and communicate early signs of deterioration. While EWS-based alerts can notify the right providers for the right patient at the right time, proactively evaluating alert-generating algorithms is essential for quantifying the impact of EWS implementation on workflow and recognition of at-risk patients. This study presents a data-driven approach to evaluate trade-offs between EWS-triggered alert-based interruptions where clinicians’ time and attention are diverted, and reliance on providers identifying patient trajectory. It allows system developers to carefully determine how to best present a given alert in terms of its disruptiveness.23 As the adoption and meaningful use of EHRs become a critical component of organisational infrastructure, the potential for health information technology-related harm will likely increase unless proactive evaluations allow for the integration of risk-reducing measures into EWS-based alert design. By evaluating alert count and frequency, and information technology capabilities, systems developers can improve alert-generating algorithms to account for factors that increase the specificity of the alerts. Further analyses of alert-generating algorithms embedded into EHRs will help investigators and providers to better understand the multifaceted roles that EWS-based alerts play throughout the spectrum of patient care.