High-performance detection and early prediction of septic shock for alcohol-use disorder patients.

BACKGROUND: The presence of Alcohol Use Disorder (AUD) complicates the medical conditions of patients and increases the difficulty of detecting and predicting the onset of septic shock for patients in the ICU.
METHODS: We have developed a high-performance sepsis prediction algorithm, InSight, which outperforms existing methods for AUD patient populations. InSight analyses a combination of singlets, doublets, and triplets of clinical measurements over time to generate a septic shock risk score. AUD patients obtained from the MIMIC III database were used in this retrospective study to train InSight and compare performance with the Modified Early Warning Score (MEWS), the Simplified Acute Physiology Score (SAPS II), and the Systemic Inflammatory Response Syndrome (SIRS) for septic shock prediction and detection.
RESULTS: From 4-fold cross validation, InSight performs particularly well on diagnostic odds ratio and demonstrates a relatively high Area Under the Receiver Operating Characteristic (AUROC) metric. Four hours prior to onset, InSight had an average AUROC of 0.815, and at the time of onset, InSight had an average AUROC value of 0.965. When applied to patient populations where AUD may complicate prediction methods of sepsis, InSight outperforms existing diagnostic tools.
CONCLUSIONS: Analysis of the higher order correlations and trends between relevant clinical measurements using the InSight algorithm leads to more accurate detection and prediction of septic shock, even in cases where diagnosis may be confounded by AUD.

Introduction

Alcohol Use Disorder (AUD) encompasses alcohol dependency, abuse, and addiction [1]. In the United States, AUD affects over 18 million people, and can lead to increased severity of illness for a variety of conditions [2], [3]. AUD is estimated to be present in between 10% and 33% of patients in the Intensive Care Unit (ICU) [2]. AUD patients have increased hospital stays by 2.4 days on average, and are up to 8% more likely to experience unplanned rehospitalization within 30-days of discharge [4], [5]. According to the World Health Organization, “In 2012, about 3.3 million net deaths, or 5.9% of all global deaths, were attributable to alcohol consumption. 139 million net DALYs (disability-adjusted life years), or 5.1% of the global burden of disease and injury, were attributable to alcohol consumption.” [6] Through increased complications [7] and longer length of stays, AUD increases costs and burdens on the health care system [8], [9].

Sepsis has been one of the leading causes of death in the United States for over a decade [10], [11]. It is a major public health concern, costing over $20 billion per year in the U.S. alone [12]. New definitions for sepsis and septic shock have recently been introduced, in an effort to simplify and streamline the clinical diagnoses of sepsis [13]. While these new definitions may prove useful and eventually find wide adoption, they are currently still under debate. Therefore for the purposes of this manuscript we have utilized the standard definitions of sepsis, severe sepsis, and septic shock, which are summarized in Table 1.

Table 1

Classification of sepsis, severe sepsis, and septic shock.

Classification	Clinical indication
Sepsis [14], [15]	Documented or suspected infectionDysregulated host responseSIRS criteria are common indicator
Severe sepsis [16]	Sepsis-induced organ dysfunction or tissue hypoperfusion
Septic shock [17]	Severe sepsis with hypotension despite adequate fluid resuscitation

Patients with AUD are 1.7 times more likely to develop any healthcare associated infection, including sepsis, than patients who do not have AUD [18]. In particular, AUD is known to complicate and exacerbate infections and sepsis in hospitalized patients [19], [20]. Although the relationships between AUD, septic shock, and infections are still being explored, increased sepsis mortality in patients with AUD may be impacted by the effects of AUD on cortisol and cytokine production [21]. The exact pathophysiologic mechanisms for the increased risk of sepsis and adverse outcomes in the AUD patient have not clearly been elucidated but a number of potential mechanisms (specific and general) have been suggested. These include the compromise of cellular immune function [22] and the alteration in the ratio of T1 helper cells to T2 helper cells [23]. Abuse of alcohol also directly affects the functioning of macrophages [24]. Complicating the recognition of emerging septic shock, AUD patients often suffer from chronic hypertension [25]. Therefore hypotension, which correlates with septic shock [17], may be difficult to identify among patients in the AUD subpopulation. Additionally, lactate, a common biomarker test which is used in the recently proposed updated septic shock definition [13], [26], may be inaccurate for AUD patients [27] because patients with AUD often suffer from chronic lactic acidosis [28], [29]. While this study was not meant to explore or explain the pathophysiologic mechanisms of sepsis and failure in AUD patients, it does recognize and attempt to correlate the clinical presentation of these patients and propose methods to identify those at risk for sepsis and septic shock before they have fully manifested themselves.

The higher costs and increased risks from sepsis and septic shock in the AUD population, in conjunction with suboptimal existing septic shock diagnostic screening performance, demonstrate the need for improved risk scoring systems for septic shock in AUD patients. Here, we analyze the performance of a risk scoring system, InSight, when detecting and predicting septic shock onset for AUD patients. We have determined to use septic shock as the gold standard for the InSight program because accurate identification and prediction of septic shock is crucial for the timely administration of antibiotics and supportive treatments to reduce mortality [30]. Additionally, the onset time of septic shock is well defined, and thus provides a clear time point for predictive assessment. We will demonstrate that InSight outperforms existing methods in discriminating between septic shock and non-septic shock patients, as well as providing early warning of impending septic shock onset.

Methods

Data set

The Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) III database [31] was queried to obtain the 29,083 patients used in InSight training and testing. MIMIC III contains de-identified patient records collected from Beth Israel Deaconess Medical Center during the years 2001–2012. We filtered a total of 61,532 MIMIC III ICU stays to obtain patients aged 15 or more years with admission to any of the intensive care units (the age filter primarily excludes neonatal ICU patients and a handful of pediatric cases), and with at least one observation of the following measurements: blood oxygen saturation, heart rate, pH, pulse pressure, respiration rate, systolic blood pressure, temperature, and white blood cell count. We also recorded the presence of ICD-9 codes for septic shock (785.52) and of alcohol abuse and related conditions (291.X, 291.XX, 303.XX, 305.XX, 357.5, 425.5, 535.3X, 571.2, and 571.3, where X denotes a wildcard).

Gold standard

Patients were assigned outcomes of septic shock upon meeting the following, hierarchical definition. Septic shock was identified using the following criteria: (1) SIRS criteria score ≥ 2, [16] (2) presence of an infection-related ICD-9 code, (3) organ dysfunction, (4) systolic blood pressure below 90 mmHg for at least 1 hour, and (5) total fluid replacement ≥1200 mL or ≥20 mL/kg for 24 hours. Combined with the requirement that patients have an AUD-related ICD-9 code, a total of 270 ICU stays were associated with AUD patients who also contracted septic shock, giving a prevalence of 0.9%. This prevalence is reasonable since septic shock and AUD prevalences are roughly 10% each and, assuming independence, a 1% net prevalence would be expected.

InSight training, score assignment, and comparison with MEWS and SIRS

InSight performs multidimensional analysis on streams of patient measurements. When working with these patient time-series data, we used a standard time resolution of one hour. We used the most recent value of measurements that were not updated by the end of each hour period. For singlet measurements, we fit a continuous function approximating the measurement value-conditioned probability distribution of the gold standard outcome. Doublets and triplets of measurement trends were binned according to heuristic tables. These tables associate a bin's empirical septic shock risk with ranges of measurement values, similar to the calculation of Modified Early Warning Score (MEWS) [32]. In the next step, we estimated the correlation between each feature and septic shock for AUD patients. The features were weighted by these correlations, then all of the measurements, singlet, doublet, and triplet trends were summed. Finally, these aggregates were combined through logistic regression, in order to assign risk scores which best reflect training data. This process and an associated equation were detailed in a previous publication, though a different clinical use-case was modeled [33], [40]. The testing results that follow were generated from assigning risk scores to patient data not included in the training.

The MEWS scores for each patient were calculated by applying the corresponding heuristic table [32]. The Simplified Acute Physiology Score II (SAPS II) [34] was calculated for each patient using the open source SQL queries for MIMIC III [35]. The SIRS criteria were also applied at each time, with the number of criteria met equaling the score assigned at that time. These scores provide comparison in the following results.

Results

Septic shock onset detection and prediction performance of InSight, MEWS, SAPS II, and SIRS is summarized by the Receiver Operating Characteristic (ROC) curves in Fig. 1. Each curve represents the average performance under 4-fold cross validation. At the time of onset, InSight had an average Area Under the ROC (AUROC) value of 0.965, compared to 0.7397 for MEWS, 0.6220 for SAPS II, and 0.5998 for SIRS. Four hours prior to onset, InSight, MEWS, SAPS II, and SIRS had average AUROCs of 0.8149, 0.6355, 0.6220, and 0.6128, respectively. This analysis was repeated across the one, two, and three hours prior to septic shock to demonstrate the improvement in each tool's prediction quality nearer the time of onset (Fig. 2). For any of these prediction times, InSight outperformed the comparison methods.

Fig. 1

Receiver Operating Characteristic (ROC) curves for InSight, Modified Early Warning Score (MEWS), Simplified Acute Physiology Score (SAPS II), and the Systemic Inflammatory Response Syndrome (SIRS) criteria at septic shock onset and four hours prior. Each curve was generated by averaging the results from 4-fold cross validation.

Fig. 2

Area Under the ROC curve (AUROC) metric for InSight, MEWS, SAPS II, and SIRS, at septic shock onset and the four hours prior. Each curve is the average result under 4-fold cross validation.

The score thresholds which determine the patient classifications of septic shock and non-septic shock can be tuned to optimize performance metrics of interest. This is illustrated for septic shock detection in Table 2. InSight, MEWS, SAPS II, and SIRS score thresholds were chosen to yield sensitivities similar to 0.90, which highlights InSight's improvement across a variety of metrics, including a 6-fold improvement in positive predictive value (PPV), 33-fold improvement in diagnostic odds ratio (DOR), and 5-fold improvement in F1 score, which is particularly relevant given the low prevalence of AUD patient septic shock. InSight manages to catch nearly all septic shock cases while maintaining a specificity more than double than that of SIRS. Over the comparison method with highest specificity, InSight reduces false alarms by 82%.

Table 2

Performance metric comparison of InSight with other scoring systems at the time of septic shock onset (with score cutoffs chosen to produce sensitivities near 0.90). Diagnostic odds ratio (DOR) and the likelihood ratios (LR) are abbreviated.

	InSight	MEWS (≥4)	SAPS II (≥16)	SIRS (≥2)
Sensitivity	0.930	0.811	0.893	0.774
Specificity	0.909	0.486	0.192	0.350
Accuracy	0.909	0.489	0.198	0.354
DOR	132.711	4.058	1.973	1.845
LR+	10.220	1.578	1.105	1.191
LR−	0.0770	0.389	0.560	0.645
F1 score	0.161	0.029	0.020	0.0218

A second comparison is displayed in Table 3 for the time four hours prior to septic shock onset, with sensitivities fixed near 0.75. Similar to the time-of-onset results, InSight predictions improve upon those made by MEWS, SAPS II, and SIRS, across all metrics. Compared with MEWS, the most specific of the other methods, InSight gives a 48% reduction in false alarms.

Table 3

Performance metric comparison of InSight with other scoring systems four hours prior to the time of septic shock onset (with score cutoffs chosen to produce sensitivities near 0.75). Diagnostic odds ratio (DOR) and the likelihood ratios (LR) are abbreviated.

	InSight	MEWS (≥4)	SAPS II (≥19)	SIRS (≥2)
Sensitivity	0.742	0.711	0.715	0.778
Specificity	0.731	0.486	0.417	0.350
Accuracy	0.731	0.488	0.419	0.354
DOR	7.795	2.326	1.790	1.845
LR+	2.757	1.383	1.226	1.191
LR−	0.354	0.595	0.684	0.645
F1 score	0.0491	0.025	0.022	0.0218

In Fig. 3, we illustrate the evolution of the septic shock risk score distribution, at the time of onset and four hours prior. At the time of onset, the distribution of scores assigned to septic shock positive and negative patients are highly separated, leading to reliable classifications. Four hours before septic shock onset, it becomes more difficult to discriminate between the populations. Note that, because the prevalence of AUD patients with septic shock in this dataset is small, shifting the score cutoff to include as positive even small portions of the negative septic shock population can lead to a large false positive rate.

Fig. 3

Scaled InSight score distributions for the time of septic shock onset and four hours prior.

Discussion

We have described a risk scoring system, InSight, for early septic shock detection in the AUD population. Vital signs, lab tests, patient demographics, and their changes over time were processed into dimensionless features. These features captured the complicated, higher-order correlations between distinct measurements, enabling their aggregation into septic shock risk scores. Our results demonstrated that this approach is capable of high quality septic shock detection and prediction several hours prior to onset, with sensitivity and specificity that exceed those of existing diagnostic tools.

This was demonstrated in Fig. 1, Fig. 2, where InSight maintains a relatively high AUROC for the time of onset and the four hours prior, across the trials of 4-fold cross validation. However, there are significant decreases in performance for each hour earlier the prediction is made. An explanation for this was shown in Fig. 3, where the score distributions of the two populations become more distinct closer to the onset time. Considering the low prevalence, the consequence of this overlap is a large increase in the absolute number of false alarms that are set off in order to obtain more true alarms. Despite this, InSight performance was quantifiably strong, as highlighted by Table 2, Table 3. Note the bolded rows of the tables, which emphasize diagnostic odds ratio and F1 score, in addition to sensitivity and specificity. The diagnostic odds ratio and F1 score are particularly important metrics for predictions made on small subpopulations and which, unlike accuracy, PPV, and NPV, are independent of prevalence. InSight performs particularly well on these metrics in comparison with SIRS, SAPS II, and MEWS. Because these results were generated through cross validation, they demonstrate that InSight is a robust alternative to the SIRS criteria, SAPS II, and MEWS in predicting septic shock onset.

Homeostasis is the process of maintaining balanced biological health through positive and negative feedback loops. In cases of septic shock, the host response to infectious insult is dysregulated. This host dysregulation involves the loss of coupling of biological processes involved in the homeostatic mechanism. By examining the degree of interrelations of vital sign measurements over time, our goal is to detect the dysregulation of biological processes before individual vital sign analysis would indicate a deterioration of homeostasis.

A simple example of this is the shock index (the ratio of heart rate to systolic blood pressure). This ratio is used because independently heart rate and blood pressure are often poor predictors of perfusion status. However, their ratio provides a much better estimate of patient condition. We extend this concept to the correlations of doublets and triplets between eight common clinical measurements (blood oxygen saturation, heart rate, pH, pulse pressure, respiration rate, systolic blood pressure, temperature, and white blood cell count) over time. Through analysis of patient trends the algorithm is able to predict which correlations correspond most closely with the risk of impending septic shock. While these correlations and trends are meaningful, it does not provide relevant clinical information to deconstruct these measurements to single values of the component vital signs.

The InSight algorithm is designed to integrate with a hospital's existing EHR system, and to be trained on the data set and patient population available at the site of implementation. InSight's function is to continuously monitor patient vital signs and autonomously provide status indication of patient risk for septic shock. Alert levels can be set at the sensitivity and specificity desired by the supervising clinician to provide the desired sensitivity while minimizing alarm fatigue.

A challenge in developing predictive algorithms for clinical settings is the identification of patients who are at risk for adverse outcomes either before the outcome occurs or the underlying diagnosis has been made. In particular, while the AUD inpatient subpopulation is particularly vulnerable to infection and sepsis, the identification of AUD patients has been problematic, especially in trauma centers where patients may present with acute intoxication and acute medical/surgical conditions [36]. There have been several strategies suggested for detecting AUD patients that rely on medical record review or interviews with the patient [37], [38], [39] – strategies that in many cases may not be practical in a busy critical care setting. However, the robust results of InSight demonstrate accurate predictions of developing septic shock for AUD patients, even if their AUD status is unrecognized. Application of the InSight algorithm will identify those AUD patients most at risk of sepsis prior to onset, despite confounding diagnostic elements, and may provide the opportunity for meaningful clinical intervention to prevent adverse outcomes.

Conclusion

We have developed a risk scoring system that predicts the onset of septic shock. Vital signs, lab tests, patient demographics, and time series of these measurements were converted to dimensionless indicators; a combination of singlets, doublets, and triplets of these measurements were used by InSight to develop a risk score. The results indicate that InSight can predict the onset of septic shock better than MEWS, SAPS II, and SIRS, and continues to outperform these systems even four hours prior to the onset of septic shock. The strong performance of InSight in this study suggests that it will be efficacious in predicting septic shock in patients in spite of an unrecognized AUD status.

Ethical approval

Not applicable.

Sources of funding

This material is based upon work supported by the National Science Foundation under Grant No. 1549867. The funder had no role in the conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication.

Author contribution

Jake Calvert, Thomas Desauutels: Study design, data collection, data analysis, writing.

Uli Chettipally, Christopher Barton: Study design, review.

Jana Hoffman: Study design, writing.

Melissa Jay, Qingqing Mao, Hamid Mohamadlou: Data collection, data analysis.

Ritankar Das: Study design, writing.

Conflicts of interest

We declare no conflicts of interest.

Guarantor

Ritankar Das.