Modified early warning score-based clinical decision support: cost impact and clinical outcomes in sepsis.

OBJECTIVE: The objective of this study was to assess the clinical and financial impact of a quality improvement project that utilized a modified Early Warning Score (mEWS)-based clinical decision support intervention targeting early recognition of sepsis decompensation.
MATERIALS AND METHODS: We conducted a retrospective, interrupted time series study on all adult patients who received a diagnosis of sepsis and were exposed to an acute care floor with the intervention. Primary outcomes (total direct cost, length of stay [LOS], and mortality) were aggregated for each study month for the post-intervention period (March 1, 2016-February 28, 2017, n = 2118 visits) and compared to the pre-intervention period (November 1, 2014-October 31, 2015, n = 1546 visits).
RESULTS: The intervention was associated with a decrease in median total direct cost and hospital LOS by 23% (P = .047) and .63 days (P = .059), respectively. There was no significant change in mortality. DISCUSSION: The implementation of an mEWS-based clinical decision support system in eight acute care floors at an academic medical center was associated with reduced total direct cost and LOS for patients hospitalized with sepsis. This was seen without an associated increase in intensive care unit utilization or broad-spectrum antibiotic use.
CONCLUSION: An automated sepsis decompensation detection system has the potential to improve clinical and financial outcomes such as LOS and total direct cost. Further evaluation is needed to validate generalizability and to understand the relative importance of individual elements of the intervention.

LAY SUMMARY

Sepsis is a leading cause of death and the most expensive cause of hospitalization in the United States. Late diagnosis or clinical decompensation after diagnosis leads to worse patient outcomes. To address this problem, we created a clinical alert system to detect when patients with sepsis were becoming sicker and notify providers to their worsening status. Our clinical alert system was created with a multidisciplinary team and integrated into usual clinical workflow in the electronic health record. In our study, we discovered that implementation of our clinical alert system led to improved patient outcomes. Specifically, we decreased cost and length of stay without increased use of board-spectrum antibiotics or intensive care unit resources. We believe that with the support of the clinical alert system, providers were able to detect worsening status of patients with sepsis earlier than using clinical judgment alone.

BACKGROUND AND SIGNIFICANCE

Sepsis, a life-threatening illness due to dysregulation of the host response to infection, is a leading cause of mortality and the most expensive cause of hospitalization in the United States. Sepsis was associated with approximately 1.6 million inpatient hospitalizations in the United States in 2009 and accounted for $20 billion in payer costs in 2011., Hospital length of stay (LOS) for patients with sepsis can average up to 9 days with an estimated $20 000 cost per case., Studies suggest sepsis survivors have a 3-fold increase in subsequent healthcare utilization with nearly half of patients requiring readmission within 1 year.

While there are clear guidelines for initial treatment and resuscitation of patients who are diagnosed with sepsis, these are most often applied to patients presenting with sepsis in the emergency department (ED)., From the ED, patients are admitted to the medical intensive care unit (MICU) if severely ill or to an acute care floor (hereafter referred to as “floor”) if they are not. Some septic patients will be admitted to the floor after diagnosis and resuscitation in the ED, but will then go on to develop clinical decompensation on the floor. Conversely, some patients will not have evidence of sepsis in the ED, but will then go on to develop sepsis after admission to the floor. Unfortunately, septic patients who are admitted to the floor and then decompensate, requiring transfer to the MICU, have nearly twice the mortality as patients who are directly admitted to the MICU. Although the reason(s) for the increase in mortality are complex, prolonged undetected decompensation may play an important role and at this time optimal surveillance methods for such decompensation outside intensive care settings are a matter of debate. One proposed solution to improve recognition and timely resuscitation of new sepsis or decompensating septic patients on the floor is for hospitals to implement an electronic health record (EHR)-based monitoring system., It is thought that an EHR algorithm may recognize decompensating septic patients on the floor earlier than regular clinical judgment., The use of early warning scores has been shown to predict in-hospital mortality and, when paired with a trigger to a provider, can lead to improved sepsis care., Given the significant costs associated with sepsis patients, we assessed the clinical and financial impact of implementing an EHR-based, modified Early Warning Score (mEWS) alerting system for decompensating septic patients in an academic medical center (Appendix Figure 1 and Section 2). We hypothesized that earlier recognition of new sepsis or decompensating septic patients on the floor may improve clinical and financial outcomes.

OBJECTIVE

The objective of this study was to assess the clinical and financial impact of a multifaceted, real-time sepsis detection system utilizing a mEWS-based clinical decision support intervention targeting early recognition of sepsis and sepsis-related decompensation.

METHODS

Intervention

The intervention consisted of two main elements: (1) displaying real-time mEWS scores in the EHR patient list dashboard (Appendix Figure 8) and (2) sending alerts when a patient’s mEWS reached a threshold of 5 (Appendix Figures 4–7). EHR generated alerts with links to nursing and provider order sets and associated automatic pages to trained staff were built into our medical center’s EHR, Epic Systems®. Alerts were triggered on the acute care floors when a patient’s vital signs suggested sepsis clinical decompensation based on mEWS. Individual components and corresponding vital sign ranges for mEWS are shown in the Appendix Figure 1. The Appendix provides details on the intervention development, rationale behind the scoring system, and changes made from previous published systems.

Study design, setting, and dates

A retrospective interrupted time series study was used to compare outcomes pre- and post-intervention. The study was conducted in a single 528-bed academic medical center. The pre-intervention cohort included all inpatient admissions to any of the floors that were later exposed to the intervention, with a diagnosis of sepsis between November 1, 2014, and October 31, 2015. The intervention was consecutively implemented across eight acute care floors including internal medicine, oncology, neurology, psychiatry, and acute inpatient rehabilitation services. The “wash-in” period from November 1, 2015, to February 28, 2016, when the intervention was being implemented, was not included in the analysis. The post-intervention cohort included all inpatient admissions for sepsis from March 1, 2016, to February 28, 2017, to any of the same floors.

Study population

Inclusion criteria for sepsis inpatient hospitalizations consisted of the following:

Inpatient hospitalizations exposed to one of the study floors in the pre- or post-intervention study periods.

Age 18 years and older.

International Classification of Diseases, Clinical Modification, Ninth or Tenth Revisions (ICD-9-CM or ICD-10-CM) facility diagnosis code for sepsis associated with the hospitalization on admission or discharge (Appendix Table 1).

To limit analysis to the intended patient population, all patients with hospital LOS greater than 90 days were excluded. Eight patients with missing cost or location data were also excluded.

Data sources

Visit data were documented in the EHR and later transferred to the academic center Enterprise Data Warehouse, an Oracle® database. Study data were collected from the data warehouse on January 10, 2020.

Visit characteristics

Visit characteristics included patient age, gender, Charlson Comorbidity Index (CCI), transfer from other facility, admission through the ED, sepsis diagnosis and severe sepsis diagnosis present on admission or acquired in the hospital, Systemic Inflammatory Response Syndrome (SIRS) onset, and mEWS ≥5.

Patient age was expressed as age on admission in years. CCI was calculated using a 17-category classification model using a superset of relevant ICD-9-CM and ICD-10-CM codes for chronic medical conditions, including discharge diagnoses., Transfers from other facilities was defined as a transfer from a different healthcare facility. Sepsis diagnoses were identified through identification of associated facility billing ICD code(s) consistent with sepsis (Appendix Table 1). SIRS was defined as fulfillment of at least two of four SIRS criteria. SIRS onset was identified as the earliest time when SIRS was present. mEWS ≥5 was calculated for two circumstances: (1) at any time during hospitalization and (2) while the patient was on one of the study floors. Specificity and sensitivity of SIRS and mEWS to detect sepsis diagnosis was calculated for inpatient hospitalizations and is shown in the Appendix Table 2.

Outcomes

This study evaluated process, clinical, and cost outcomes. LOS, mortality, and total direct cost were primary outcomes. All other outcomes were secondary.

Process outcomes included: (1) antibiotic administration within 24 h of SIRS onset and (2) time to opening the chart after mEWS ≥5. Broad-spectrum antibiotics were defined as intravenous (IV) antibiotics with activity toward resistant organisms (eg, methicillin-resistant Staphylococcus aureus, vancomycin-resistant Enterococci, Pseudomonas aeruginosa, or extended-spectrum beta-lactamases). Community antibiotics were those that did not meet this definition. Time to opening the chart after mEWS ≥5 was calculated as the time from a patient first having a threshold mEWS score of ≥5 to nurses and providers (ie, physician, nurse practitioner, physician assistant) opening the chart. Time to opening the chart after mEWS ≥5 was calculated from EHR logs.

Clinical outcomes included: (1) LOS, (2) mortality, (3) floor to intensive care unit (ICU) transfer, and (4) 30-day hospital readmission. LOS was calculated as the difference between admission and discharge time as recorded in the EHR. Mortality was defined as death while hospitalized and identified from the discharge disposition. Thirty-day readmissions were calculated as all-cause readmissions within 30 days after hospital discharge for patients who were alive at discharge.

Cost outcomes included: total direct cost and six sub-costs: facility utilization, pharmacy, laboratory tests, supplies, imaging, and laboratory management costs. Due to the sensitivity of cost data, costs are presented in normalized form. Prior to normalizing all costs were converted to 2015 dollars using Consumer Price Index for All Urban Consumers.

Statistical methods

R version 3.5.2 software was used for statistical analysis. P-values below .05 were considered significant. P-values less than .06 were expressed with three decimal points.

Visit characteristics pre- and post-intervention were summarized as number (frequency) or the mean (standard deviation) and compared using chi-squared tests and t-tests where appropriate.

To account for potential secular trends, we conducted an interrupted time series analysis with the “level change” impact model. We chose the “level change” impact model because alert-based interventions focused on patient safety often take effect immediately and there were no significant trend changes found in a sensitivity analysis., Due to high degrees of skewing, hospital LOS and costs were summarized using monthly medians. Other outcomes were summarized using monthly percentages. The following segmented linear regression model was used: where β0 represents the measure level in the beginning of the study, β1 represents the pre-intervention trend, and β2 represents the level change following the intervention. To adjust for seasonality, we used harmonic terms with one sine and cosine pair. We did not find significant autocorrelation after adjusting for seasonality. In order to normalize costs, total direct cost, predicted for the first month of the pre-intervention period (ie, November 2014), is shown as having a cost of 100 units, and all other costs are shown proportionally.

The University of Utah Institutional Review Board approved the study (protocol #96120).

RESULTS

There were 23 078 inpatient visits included in this study. A total of 3664 (15.88%) patient visits met the inclusion criteria for sepsis diagnosis. A total of 1546 visits by 1360 unique patients had a sepsis diagnosis in the pre-intervention period and 2118 visits by 1874 unique patients had a sepsis diagnosis in the post-intervention period. The baseline visit characteristics for the pre- and post-intervention cohorts are presented in Table 1. The post-intervention cohort was characterized by a higher incidence of sepsis present on admission (P < .001), lower incidence of hospital-acquired sepsis (P = .027), and a lower incidence of hospital-acquired severe sepsis (P = .008). The post-intervention sepsis cohort was characterized by a lower CCI (P < .001), a higher frequency of ED admissions (P < .001), and transfers from other facilities (P = .047). Other visit characteristics remained unchanged pre- and post-intervention.

Table 1.

Visit characteristics pre- and post-intervention

Visit characteristics	Pre-intervention	Post-intervention	P-value
Inpatient visits (n = 23 078)
All inpatient visits on the study floors, n	10 397	12 681
Unique patients with inpatient visits, n	7596	9311
Visits with sepsis diagnosis, n (%)	1546 (14.87)	2118 (16.70)	<.001
Visits with sepsis present on admission, n (%)	1343 (12.92)	1919 (15.13)	<.001
Visits with hospital-acquired sepsis, n (%)	203 (1.95)	199 (1.57)	.027
Visits with severe sepsis diagnosis, n (%)	827 (7.95)	1048 (8.26)	.39
Visits with severe sepsis present on admission, n (%)	708 (6.81)	946 (7.46)	.06
Visits with hospital-acquired severe sepsis, n (%)	119 (1.14)	102 (0.8)	.008
Visits without sepsis and with mEWS < 5: true negatives, n (%)	8548 (82.22)	10 185 (80.32)
Visits without sepsis and with mEWS ≥ 5: false positives, n (%)	303 (2.91)	378 (2.98)
Visits with sepsis and with mEWS < 5: false negatives, n (%)	1269 (12.21)	1679 (13.24)
Visits with sepsis and with mEWS ≥ 5: true positives, n (%)	277 (2.66)	439 (3.46)
Visits with sepsis (n = 3664)
Visits with sepsis diagnosis, n	1546	2118
Unique patients with sepsis diagnosis, n	1360	1874
Female gender, n (%)	722 (46.7)	991 (46.79)	.95
Age (years), mean (SD)	56.58 (17.95)	55.44 (17.49)	.053
CCI, mean (SD)	5.05 (3.58)	4.66 (3.45)	<.001
Emergency department admission, n (%)	1334 (86.29)	1978 (93.39)	<.001
Transfer from other facility, n (%)	393 (25.42)	601 (28.38)	.047
Comfort care only order placed, n (%)	112 (7.24)	137 (6.47)	.36
SIRS criteria met, n (%)	1441 (93.21)	1998 (94.33)	.16
mEWS ≥ 5 at any time during hospitalization, n (%)	675 (43.66)	941 (44.43)	.64
mEWS ≥ 5 on the study floor, n (%)	277 (17.92)	439 (20.73)	.034

Note: P-values are based on chi-square and t-tests.

Abbreviations: CCI: Charlson Comorbidity Index; mEWS: modified Early Warning Score; SD: standard deviation; SIRS: Systemic Inflammatory Response Syndrome.

The full results of the interrupted time series analysis were too lengthy to include in the manuscript, so we report most of them in the Appendix Tables 3 and 4, but parameter estimates from the interrupted time series models for the primary outcomes (ie, LOS, mortality, and total direct cost) are summarized in Table 2. Changes in primary outcomes over time are shown in Figure 1. There was a trend toward decreased median LOS by 0.63 days (95% confidence interval [CI], −1.28 to 0.03, P = .059) (Table 2). There was no significant change in mortality rate (Table 2). Normalized total direct cost decreased by 23.36% (95% CI, −46.32% to −0.39%, P = .047) following the intervention (Table 2).

Table 2.

Parameter estimates from interrupted time series models for primary outcomes

				95% CI		Hypothesis test
Outcome	Parameter	Beta	Standard error	Lower	Upper	t-Value	Pr > |t|
Length of stay (d)	Baseline level	5.45	0.14	5.15	5.75
	Baseline trend	0.01	0.02	−0.03	0.04	0.33	.75
	Level change	−0.63	0.31	−1.28	0.03	−2.01	.059
Mortality (%)	Baseline level	5.38	1.23	2.8	7.96
	Baseline trend	0.2	0.16	−0.12	0.53	1.3	.21
	Level change	−3.14	2.69	−8.76	2.48	−1.17	.26
Total visit direct cost	Baseline level	100	5.03	89.46	110.54
	Baseline trend	0.82	0.64	−0.52	2.15	1.28	.21
	Level change	−23.36	10.97	−46.32	−0.39	−2.13	.047

Note: Parameter estimates are based on interrupted time series linear regression models. Parameters correspond to measure levels and trends over time. For example, median length of stay was 5.45 days in November 2014, was increasing by 0.01 days each month in the pre-intervention period and dropped by 0.63 days following the intervention. Costs are expressed as normalized medians using November 2014 total direct median cost as the reference value.

Abbreviation: CI: confidence interval.

Figure 1.

Changes in primary outcomes over time: (A) median length of stay, (B) mortality, and (C) median total direct cost. NOTE: Visit outcomes are aggregated by arrival month. Monthly sample size ranged from 105 to 195 visits. P values and regression lines are based on interrupted time series linear regression models with harmonic terms to adjust for seasonality effects.

There were no changes in 30-day readmission, ICU transfers, or proportion of patients who received antibiotics within 24 h of SIRS onset (Appendix Table 3). The time to opening the chart after mEWS ≥5 was significantly reduced from 12 to 7 min for nurses (P = .006) and 52 to 25 min (P < .001) for providers (physicians, nurse practitioners, and physician assistants), respectively (Table 3).

Table 3.

Time to opening the chart after mEWS ≥5

Time to opening the chart after mEWS ≥5	Pre-mEWS	Post-mEWS	P-value
Nurse	11.89 (33.36)	6.9 (14.13)	.006
Provider	52.49 (90.25)	25.12 (56.14)	<.001

Note: Data reported as mean (SD). P-values are based on t-tests.

Abbreviations: mEWS: modified Early Warning Score; SD: standard deviation.

Composition of total direct cost in November 2014 is visualized in Figure 2. Pharmacy, supplies, and imaging sub-costs were significantly decreased post-intervention (Appendix Table 4). Cost for non-septic patient visits with elevated mEWS scores, that is, false positives, did not increase (Appendix Table 5).

Figure 2.

Median costs estimated for November 2014.

DISCUSSION

The implementation of an mEWS-based clinical decision support system in eight acute care floors at an academic medical center was associated with reduced LOS and total direct cost of hospitalization for patients with sepsis without a concomitant increase in ICU utilization or broad-spectrum antibiotic use. Stable rates of ICU utilization and broad-spectrum antibiotic use were especially important to us because clinical stakeholders and antibiotic stewardship had concerns that such a program would inappropriately drive them up. All of these findings were seen without an associated increase in cost for non-septic patient visits with elevated mEWS scores, that is, in false positives cases (Appendix Table 5).

The exact mechanism by which the mEWS-based alert system and dashboard may have decreased total direct cost is subject to interpretation. However, we hypothesize that with the support of an mEWS dashboard and alert system, providers were able to detect decompensating septic floor patients earlier than “ordinary clinical judgment,” as discussed by Escobar and Dellinger. Earlier sepsis detection and evaluation by a provider is a difficult outcome to measure, although our results demonstrate a significant decrease in the time from detecting a mEWS ≥5 to a provider or nurse opening a patient’s electronic chart, supporting our hypothesis (Table 3). This earlier detection and evaluation may have led to closer monitoring and earlier therapeutic interventions among acutely ill patients, thereby preventing further clinical deterioration, complications, and prolonged LOS. It is plausible that reducing the time a patient is severely ill or recovering from associated complications reduces the need for more expensive treatments such as IV antibiotics, enteral nutrition, and IV fluids, and supplies such as IV tubing, central venous catheters, and urinary and/or fecal catheters. In addition, a decreased LOS translates into reduced need for inpatient medication doses, supplies, and therapy services. Finally, we observed a higher incidence of sepsis present on admission and a lower incidence of hospital-acquired sepsis in the post-intervention cohort, reflecting an overall earlier recognition and treatment of sepsis.

The intervention did not significantly change clinical outcomes for sepsis patients as anticipated. Mortality, for example, did not change. There are several possible explanations for this outcome. First, the intervention parameters may not have been sufficiently sensitive to affect the trajectory of patients with a very high risk of mortality. mEWS thresholds were chosen that knowingly sacrificed some sensitivity in order to improve clinician compliance. Balancing the tradeoff between maximizing intervention sensitivity and minimizing provider alert fatigue is challenging and has been described in previous sepsis screening studies., Second, while there was clinical decision support, no therapies or transfers to a higher level of care were enforced as it is still unclear in the literature what the best practice is after being alerted to a decompensating patient. Evaluating the patient after an alert was highly encouraged, but treatment was ultimately up to the providers. Third, since a large proportion of inpatients diagnosed with sepsis do not require transfer to an ICU and do not die, it is foreseeable that our study may have been underpowered to detect a change in mortality. Furthermore, it is possible that a ceiling effect exists for mortality in some institutions that have established quality and safety programs. Finally, while it intuitively makes sense that recognizing septic or decompensating septic patients earlier would lead to decreased mortality, this may not universally be the case; perhaps this trajectory cannot be changed or there are only certain sub-populations that would experience mortality benefit from such an intervention. For example, perhaps patients with very low Sequential Organ Failure Assessment (SOFA) scores have too low of a baseline mortality risk to receive mortality benefit, patients with very high SOFA scores have unchangeable trajectories, and only patients with mid-range SOFA scores obtain a mortality benefit.

One strength of this study is that we measured direct costs attributable to individual patient encounters, accurately estimating actual cost savings as a result of the study intervention. Although the actual cost figures are not disclosed here due to the sensitivity of such data, reporting proportional cost savings may be more generalizable due to geographic variability in healthcare cost. Other studies attempting to evaluate the relationship between sepsis and cost typically rely on hospital charges or cost-to-charge ratios.,, While charges are frequently used as a surrogate measure for costs in healthcare financial analyses, they perform poorly in this role due to their limited correlation with actual costs., The value-driven outcomes (VDO) tool used in the study was developed at our site and designed to measure the actual cost attributable to patient care at the level of individual encounters and patient care activities. For example, medication and supply costs are based on actual hospital acquisition costs, whereas facility utilization costs are based on the cumulative time each patient spends on an inpatient unit in conjunction with the hospital’s actual general ledger expenses for operating that unit. We believe the VDO tool allowed detailed measurement of cost savings attributable to the mEWS-based intervention.

Another strength of this study is the potential generalizability of the intervention based on the ubiquity of the EHR system in which the intervention-related software was built. Specifically, the EpicCare Inpatient Clinical System® is one of the most widely implemented inpatient EHRs in acute care hospitals across the country. Additionally, vendor-neutral technical frameworks for this type of intervention are emerging with implementation potential across various EHR platforms. A further strength is that we utilized an interrupted time series analysis which accounts for baseline trends in outcomes, even if the underlying covariates causing that change were not recorded.

Limitations

This study has several limitations. First, causality cannot be concretely demonstrated given the retrospective nature of the study. Second, the study was conducted in a single center, limiting generalizability. Third, an additional acute care floor was opened concurrent with the intervention. This floor was included in the study but there were no pre-intervention data for this floor. However, the floor was similar to the other acute medical floors and should have had a similar mix of patients. Fourth, the multifaceted nature of the intervention makes it difficult to rigorously determine the impact of the individual components. Fifth, at the time of our study we did not have the capability to calculate scores such as APACHE or SOFA on each patient to adjust for risk and to measure organ failure/severity of illness. However, any secular trend in patient risk profiles should have been accounted for by the interrupted time series analysis. Sixth, other quality improvement projects were concurrently being implemented at our institution which made it challenging to solely attribute the results to our study intervention. Seventh, we developed our study cohorts using ICD coding data. The validity of such administrative data is known to vary across institutions. Further studies are needed to assess the generalizability of the findings across multiple centers and heterogeneous patient populations, as well as to evaluate the relative contribution of individual intervention components.

CONCLUSION

Implementation of an mEWS-based clinical decision support system has the potential to improve clinical and financial outcomes such as duration of hospitalization and total care costs. Further evaluation is needed to validate generalizability, and efforts need to be made to understand the relative importance of individual elements of our multifaceted intervention.

FUNDING

This work was primarily supported by the University of Utah Health Sciences Center. Statistical analyses were partly supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health grant number 5UL1TR001067-05 (formerly 8UL1TR000105 and UL1RR025764).

AUTHOR CONTRIBUTIONS

Study concept and design: DJH, KKG, PVK, KK, MS, SAJ, WAD, JJA, and DR. Acquisition, analysis, or interpretation of data: all authors. Drafting of the manuscript: DJH, KKG, PVK, MC, KK, and SAJ. Critical revision of the manuscript for important intellectual content: all authors. Statistical analysis: PK. Obtained funding: DJH and KK. Administrative, technical, or material support: MS, WAD, MW, DR, and JJA. Study supervision: DJH and KK. Final approval: all authors. Accountability for all aspects of the work: all authors.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

KK reports honoraria, consulting, or sponsored research in the past year outside the submitted work with McKesson InterQual, Pfizer, Hitachi, Klesis Healthcare, RTI International, the Mayo Clinic, the University of California at San Francisco, and the U.S. Office of the National Coordinator for Health IT (via Security Risk Solutions) in the area of health information technology. KK was also a co-developer of the University of Utah value-driven outcomes tool used for cost analyses in the submitted work. DJH reports honoraria from the CHART Institute for an online continuing medical education webinar given to nurses about sepsis quality improvement. DJH also reports travel support from SPOK, Inc. for speaking at its annual conference in Arizona and for a presentation at the 2019 Healthcare Information and Management Systems Society (HIMSS) national conference in Orlando. MS reports travel support from SPOK, Inc. for a presentation given at the 2019 HIMSS national conference in Orlando. All other authors declared no conflict of interest.

Click here for additional data file.