A Data Mining Approach to Determine Sepsis Guideline Impact on Inpatient Mortality and Complications.

Sepsis incidents have doubled from 2000 through 2008, and hospitalizations for these diagnoses have increased by 70%. The use of the Surviving Sepsis Campaign (SSC) guidelines can lead to earlier diagnosis and treatment; however, the effectiveness of the SSC guidelines in preventing complications for this population is unclear. The overall purpose of this study was to apply SSC guideline recommendations to EHR data for patients with severe sepsis or septic shock and determine guideline compliance as well as its impact on inpatient mortality and sepsis complications. Propensity Score Matching in conjuction with Bootstrap Simulation were used to match patients with and without exposure to the SSC recommendations. Findings showed that EHR data could be used to estimate compliance with SSC recommendations as well as the effect of compliance on outcomes. Compliance with guideline recommendations ranged from 9% to 100%. For individual recommendations with sufficient data, association with outcomes varied. Checking lactate influenced four outcomes; however, two were negative and two positive. Use of a ventilator for patients with respiratory distress had a positive association with three outcomes.

Introduction

According to the Center for Disease Control and Prevention, the incidence of sepsis or septicemia has doubled from 2000 through 2008, and hospitalizations have increased by 70% for these diagnoses[1]. In addition, severe sepsis and shock have higher mortality rates than other sepsis diagnoses, accounting for an estimated mortality between 18% and 40%[2,3]. During the first 30 days of hospitalization, mortality can range from 10% to 50%[4] depending on the patient’s risk factors. Patients with severe sepsis or septic shock are sicker, have longer hospital stays, are more frequently discharged to other short-term hospital or long-term care institutions, and represent the most expensive hospital condition treated in 2011[2].

The use of evidence-based practice (EBP) guidelines, such as the Surviving Sepsis Campaign (SSC)[5], could lead to an earlier diagnosis, and consequently, earlier treatment. However, these guidelines have not been widely incorporated into clinical practice[6]. The SSC is a compilation of international recommendations for the management of severe sepsis and shock[5]. Many of these recommendations are interventions to prevent further system deterioration during and after diagnosis. Even when the presence of sepsis or progression to sepsis is suspected early in the course of treatment, timely implementation of adequate treatment management and guideline compliance are still a challenge[3,7]. Therefore, the effectiveness of the guideline in preventing clinical complications for this population is still unclear to clinicians and researchers alike.

The majority of studies have focused on early detection and prevention of sepsis and little is known about the compliance rate to SSC and the impact of compliance on the prevention of sepsis-related complications. Further, the measurement of adherence to individual SSC recommendations rather than the entire SSC is, to our knowledge, limited[8]. The majority of studies have used traditional randomized control trials with analytic techniques such as regression modeling to adjust for risk factors known from previous research[4,9]. Data-driven methodologies, such as data mining techniques and machine learning, have the potential to identify new insights from electronic health records (EHRs) that can strengthen existing EBP guidelines.

The national mandate for all health professionals to implement interoperable EHRs by 2015 provides an opportunity for the reuse of potentially large amounts of EHR data to address new research questions that explore patterns of patient characteristics, evidence-based guideline interventions, and improvement in health[10,11,12]. Furthermore, expanding the range of variables documented in EHRs to include team-based assessment and intervention data can increase our understanding of the compliance with EBP guidelines and the influence of these guidelines on patient outcomes. In the absence of such data elements, adherence to guidelines can only be inferred; it cannot be directly observed.

In this manuscript, we present a methodology for using EHR data to estimate the compliance with the SSC guideline recommendations and also estimate the effect of the individual recommendations in the guideline on the prevention of in-hospital mortality and sepsis-related complications in patients with severe sepsis and septic shock.

Methods

Data from the EHR of a health system in the Midwest was transferred to a clinical data repository (CDR) at the University of Minnesota which is funded through a Clinical Translational Science Award[13]. After IRB approval, de-identified data for all adult patients hospitalized between 1/1/09 to 12/31/11 with a severe sepsis or shock diagnosis was obtained for this study.

Data and cohort selection

The sample included 186 adult patients age 18 years or older with an ICD-9 diagnosis code of severe sepsis or shock (995.92 and 785.5*) identified from billing data. Since 785.* codes corresponding to shock can capture patients without sepsis, patients without severe sepsis or septic shock, and patients who did not receive antibiotics were excluded. These exclusions aimed to capture only those patients who had severe sepsis and septic shock, and were treated for that clinical condition. The final sample consisted of 177 patients.

Variables of interest

Fifteen predictor variables (baseline characteristics) were collected. These include sociodemographics and health disparities data: age, gender, race, ethnicity, and payer (Medicaid represents low income); laboratory results: lactate and white blood cells count (WBC); vital signs: heart rate (HR), respiratory rate (RR), temperature (Temp), mean arterial blood pressure (MAP); and diagnoses for respiratory, cardiovascular, cerebrovascular, and kidney-related comorbid conditions. ICD-9 codes for comorbid conditions were selected according to evidence in the literature (Appendix A is available at https://www.dropbox.com/s/lxvwl3enj9coxy3/Sepsis_codes.docx?dl=0). Comorbidities were aggregated from the patient’s prior problem list to detect preexisting (upon admission) respiratory, cardiovascular, cerebrovascular, and kidney problems. Each category was treated as yes/no if any of the ICD-9 codes in that category were present.

The outcomes of interest were inhospital mortality and development of new complications (respiratory, cardiovascular, cerebrovascular, and kidney) during the hospital encounter. New complications were determined as the presence of ICD-9 codes on the patient’s billing data that did not exist at the time of the admission.

Study design

This study aimed to analyze compliance with the SSC guideline recommendations in patients with severe sepsis or septic shock. Therefore, the baseline (“TimeZero”) was defined as the onset of sepsis and the patients were under observation until discharged. Unfortunately, the timestamp for the diagnoses is dated back to the time of admission; hence the onset of sepsis needs to be estimated. The onset time for sepsis was defined as the earliest time during a hospital encounter when the patient meets at least two of the following six criteria: MAP < 65, HR >100, RR >20, temperature < 95 or >100.94, WBC < 4 or > 12, and lactate > 2.0. The onset time was established based on current clinical practice and literature on sepsis[5]. The earliest time when two or more of these aforementioned conditions were met, a TimeZero flag was added to the time of first occurrence of that abnormality, and the timing of the SSC compliance commenced.

Guideline compliance

SSC guideline recommendations were translated into a readily computable set of rules. These rules have conditions related to an observation (e.g. MAP < 65 Hgmm) and an intervention to administer (e.g. give vasopressors) if the patient meets the condition of the rule. The SSC guideline was transformed into 15 rules in a computational format, one for each recommendation in the SSC guideline recommendations, and each rule was evaluated for each patient (see Figure 1). After each rule is an abbreviated name subsequently used in this paper.

Figure 1:

SSC rules for measuring guideline compliance.

We call the treatment of a patient compliant (exposed) for a specific recommendation, if the patient meets the condition of the corresponding rule any time after TimeZero and the required intervention was administered; the treatment is non-compliant (unexposed) if the patient meets the condition of the corresponding rule after TimeZero, but the intervention was not administered (any time after TimeZero); and the recommendtion is not applicable to a treatment if the patient does not meet the condition of the corresponding rule. In estimating compliance (as a metric) with a specific recommendation, we simply measure the number of compliant encounters to which the recommendation is applicable. In this phase of the study, the time when a recommendation was administered was not incorporated in the analysis.

We also estimate the effect of the recommendation on the outcomes. We call a patient exposed to a recommendation, if the recommendation is applicable to the patient and the corresponding intervention was administered to the patient. We call a patient unexposed to a recommendation if the recommendation is applicable but was not applied (the treatment was non-compliant). The incidence fraction in exposed patients with respect to an outcome is the fraction of patients with the outcome among the exposed patients. The incidence fraction of the unexposed patients can be defined analogously. We define the effect of the recommendation on an outcome as the difference in the incidence fractions between the unexposed and exposed patients. The recommendation is beneficial (protective against an outcome) if the effect is positive, namely, the incidence faction in the unexposed is higher than the incidence fraction in the unexposed patients.

Data quality

Included variables were assessed for data quality regarding accuracy and completeness based on the literature and domain knowledge. Constraints were determined for plausible values, e.g., a CVP reading could not be greater than 50. Values outside of constraints were recoded as missing values. Any observation that took place before the estimated onset of sepsis (TimeZero) was considered a baseline observation. Simple mean imputation was the method of choice for imputing missing values. Imputation was necessary for lactate (7.7%), temperature (3%), and WBC (3%). There was no missing data for the other variables and for the outcomes of interest. Central venous pressure was not included as a baseline characteristic due to the high number of missing values (54%).

Propensity score matching

Patients who received SSC recommendations may be in worse health than patient who did not receive SSC recommendations. For example, patients whose lactate was measured may have more apparent (and possibly advanced) sepsis than patients whose lactate was not measured. To compensate for such disparities, propensity score matching (PSM) was employed. The goal of PSM is to balance the data set in terms of the covariates between patients exposed and unexposed to the SSC guideline recommendations. This is achieved by matching exposed patients with unexposed patients on their propensity (probability) of receiving the recommendations. This ensures that at TimeZero, pairs of patients, one exposed and one unexposed, are at the same state of health and they only differs in their exposure to the recommendation. PSM is a popular technique for estimating treatment effects[14,15].

To compute the propensity of patients to receive treatment, a logistic regression model was used, where the dependent variable is exposure to the recommendation and the independent variables are the covariates. The linear prediction (propensity score) of this model was computed for every patient. A new (matched) population was created from pairs of exposed and unexposed patients with matching propensity scores. Two scores match if they differ by no more than a certain caliper (.1 in our study)[16]. The effect of the recommendation was estimated by comparing the incident fraction among the exposed and unexposed patients in the matched population.

PSM nested inside bootstrapping simulation.

In order to incorporate the effect of additional sources of variability arising due to estimation in the propensity score model and variability in the propensity score matched sample, 500 bootstrap samples were drawn from the original sample[14,17]. In each of these bootstrap iterations, the propensity score model was estimated using the above caliper matching techniques and the effect of the recommendation was computed with respect to all outcomes. In recent years, bootstrap simulation has been widely employed in conjunction with PSM to better handle bias and confounding variables[17]. For each recommendation and outcome, the 500 bootstrap iterations result in 500 estimates of the effect (of the recommendation on the outcome), approximating the sampling distribution of the effect.

Results

Table 1 shows the baseline characteristics of the study population. Results are reported as total count for categorical variables, and mean with inter-quartile (25%–75%) range for continuous variables. As shown in Table 1, the majority of patients were male, Caucasian, and had Medicaid as the payer. Before the onset of sepsis, Cardiovascular comorbidities (56.4%) were common, the mean HR (101.3) was slightly above the normal, as well as lactate (2.8), and WBC (15.8). The mean length of stay for the sample was 15 days, ranging from less than 24 hours to 6 months. TimeZero was within the first 24 hours of admission, and patients at that time were primarily (86.4%) in the emergency department.

Table 1:

Baseline characteristics for septic patients between 2011 and 2013 (n=177) in a Midwest health system.

Characteristics	Patient Count n=177		Characteristics	Patient Count n=177
	Mean (IQR)			Mean (IQR)
Age (years)	61 (51–71)		Temperature	98.4 (97.3–99.5)
Gender (Male)	102*		Heart rate	101.3 (87.4–200.4)
Race (Caucasian)	97*		Respiratory rate	20.6 (17.1–22.8)
Ethnicity (Latino)	11*		Cardiovascular	100*
Payer (Medicaid)	102*		Cerebrovascular	66*
White blood cell	15.8 (9.1–18.6))		Respiratory	69*
Lactate	2.8 (1.6–2.8)		Kidney	62*
Mean blood pressure	73.9 (40.7)

Note:

total count is reported for categorical data.

Fifteen rules from the SSC recommendations were identified. Table 2 presents a description for these rules along with the number of patients whose treatment was compliant with the recommendations. The ‘Y’ column indicates the number of patients exposed (compliant) with a rule, the ‘N’ column includes the number of patients not exposed (non-compliant) with a rule, and ‘N/A’ includes the number of patients the rule was not applicable or could not be calculated. The ‘% Compl’ column includes the number of patients exposed to the rule divided by the total number of patients for which the rule was applicable. Using this information, rules LactateFluid, GlucoseInsulin, MAP, MAPFluids, CVPFluids, Albumin, and Diuretic were removed as patient coverage was insufficient or there was a high intercorrelation with another rule (CVP and MAP). This means that these excluded rules were not included in subsequent analyses in this study. Rules BCulture, Antibiotic, Lactate, BGlucose, Vasopressor, CVP, RespDistress, and Ventilator were included. The included rules are highlighted in bold in Table 2.

Table 2:

Rules description and results from the guideline application.

Rules Description	Patient Count / %
	Y	N	% Compl	N/A
1. Was Blood Culture done? (BCulture)  2. Was Antibiotic given after Blood Culture? (Antibiotic)	126 99	51 27	71 79	0 51
3. Was Lactate checked? (Lactate)  4. Was Fluid Resuscitation done if Lactate > 4? (LactateFluid)	127 36	50 0	72 100	0 141
5. Was Blood Glucose checked? (BGlucose)  6. Was Insulin given if two Blood Glucose measures were > 180? (GlucoseInsulin)	132 38	45 8	75 83	0 131
7. Was MAP checked? (MAP)  8. Was Fluid Resuscitation give if MAP > 65? (MAPFluids)   9. Was Vasopressor given if MAP < 65 after Fluid Resuscitation? (Vasopressor)	177 160 26	0 6 140	100 96 16	0 11 11
10. Was CVP checked? (CVP)  11. Was Fluid Resuscitation done if CVP < 2? (CVPFluids)   12. Was Albumin given if CVP < 2 after Fluid Resuscitation? (Albumin)  13. Was a Diuretic given if CVP above 12? (Diuretic)	121 15 4 10	56 162 11 71	68 9 27 12	0 0 162 96
14. Was there Respiratory Distress*? (RespDistress)  15. Was a ventilator given if there was Respiratory Distress? (Ventilator)	167 92	10 75	94 55	0 10

Note:

respiratory distress was defined as RR lower than 12 or greater than 20, and/or Oxygen saturation lower than 92% according to the SSC.

In Figure 2, the effects of various rule-combination pairs are depicted. An effect is defined as the difference in the mean rate of progression to complications between the exposed and unexposed groups. Since we used bootstrap simulation, for each rule-complication pair, 500 replications were performed resulting in a sampling distribution for the effect. Sampling distribution for each rule-association pair is presented as boxplots. The boxplots represent the statistic measured, i.e. in this study, the differential impact of a recommendation on mortality between the exposed and unexposed population. When this statistic is 0, the recommendation has no effect. If the recommendation is greater than 0, it means that the recommendation is protective for that specific condition; and if the recommendation is below 0, the recommendation may even increase the risk for the outcome for that specific condition.

Figure 2:

Box-plots of the mean difference between groups (unexposed - exposed) to the guideline recommendations and each of the outcomes of interest.

The panes (groups of boxplots) correspond to the complications and the boxes within each pane correspond to the recommendation (rule). For example, the effect of the Ventilator rule (Recommendation #15: patients in respiratory distress should be put on ventilator) on mortality (Death) is shown in the rightmost box (Ventilator) in the bottom-most pane (Death). Since all effects in the boxplot are above 0, namely the number of observed complications in the unexposed group is higher than in the exposed, compliance with the Ventilator rule reduces the number of deaths. Therefore, the corresponding recommendation is beneficial to protect patients from Death (mortality). In Table 3, we present the 95% Confidence Intervals for various rule-outcome pairs.

Table 3:

95% Confidence intervals for various rule-outcome pairs.

	Cardiovascular	Respiratory	Kidney	Cerebrovascular	Death
BCulture	(−0.11, 0.15)	(−0.16, 0.12)	(−0.15, 0.11)	(−0.09, 0.20)	(−0.14, 0.09)
Antibiotic	(−0.16, 0.10)	(−0.23, 0.13)	(−0.08, 0.26)	(−0.09, 0.28)	(−0.21, 0.10)
Lactose	(−0.05, 0.19)	(−0.20, 0.07)	(−0.08, 0.18)	(−0.04, 0.21)	(−0.12, 0.10)
BGlucose	(−0.02, 0.25)	(−0.02, 0.28)	(−0.16, 0.14)	(−0.06, 0.18)	(−0.19, 0.09)
Vasopressor	(−0.11, 0.27)	(0.04, 0.35)	(−0.20, 0.17)	(−0.32, −0.07)	(−0.10, 0.21)
CVP	(−0.03, 0.16)	(−0.06, 0.17)	(−0.10, 0.14)	(−0.08, 0.16)	(−0.08, 0.13)
RespDistress	(−0.25, 0.36)	(−0.36, 0.37)	(−0.14, 0.40)	(−0.30, 0.37)	(−0.25, 0.14)
Ventilator	(0.04, 0.19)	(0.08, 0.32)	(−0.11, 0.09)	(−0.08, 0.11)	(0.03, 0.20)

To further ensure the validity of the results, we examine the propensity score distribution in the exposed and unexposed group. As an example, Figure 3 illustrates the propensity score distribution for a randomly selected bootstrap iteration to measure the effect of Ventilator on Death. The horizontal axis represents the propensity score, which is the probability of receiving the interventions, and the vertical axis represents the density distribution, namely the proportion of patients in each group with a particular propensity for being put on Ventilator. Figure 3 shows substantial overlap between the propensity scores in the exposed and unexposed group. The propensity score overlap represents the distribution; the predictor “Ventilator” across the exposed and unexposed populations regarding the outcome “Death;” the balance was successful when the propensity score was applied for this population. Other rule-complication pairs exhibit similar propensity score distribution.

Figure 3:

Distribution of the propensity scores between exposed and unexposed groups for the outcome Death when patients and the SSC recommendation was Ventilator.

Discussion

The overall purpose of this study was to use EHR data to determine compliance with the Surviving Sepsis Campaign (SSC) guideline and measure its impact on inpatient mortality and sepsis complications in patients with severe sepsis and septic shock. Results showed that compliance with many of the recommendations was > 95% for MAP and CVP with fluid resuscitation given for low readings. Other high compliance (≥80%) recommendations were: insulin given for high blood glucose and evaluating respiratory distress. The recommendations with the lowest compliance (< 30%) were: vasopressor or albumin for continuing low MAP or CVP readings. This may be due to a study design artifact, where the rule only considered interventions initiated after TimeZero (estimated onset of sepsis) while the fluid resuscitation may have taken place earlier. Alternatively, the apparently poor compliance could also be explained with issues related to the coding of fluids: during data validation, we found that it was difficult to track fluids.

Our study also demonstrates that retrospective EHR data can be used to evaluate the effect of compliance with guideline recommendations on outcomes. We found a number of SSC recommendations that were significantly protective against more than one complication: Ventilator was protective against Cardiovascular and Respiratory complications as well as Death; use of Vasopressors was protective for Respiratory complications.

Other recommendations, BCulture, Antibiotic, Vasopressor, Lactate, CVP, and RespDistress, showed results less consistent with our expectation. For instance, Vasopressor used to treat low MAP, appears to increase cerebrovascular complications. While this finding is not statistically significant, it may be congruent with the fact that small brain vessels are very sensitive to changes in blood pressure. Low MAP can cause oxygen deprivation, and consequently brain damage.

Ventilator, Vasopressor, and BGlucose showed protective effects against Respiratory complications. The SSC guideline recommends the implementation of ventilator therapy as soon as any change in respiratory status is noticed. This intervention aims to protect the patient against further system stress, restore hypoxia, help with perfusion across the main respiratory-cardio vessels, and decrease release of toxins due to respiratory efforts[3,5,9].

Our study is a proof-of-concept study demonstrating that EHR data can be used to estimate the effect of guideline recommendations. However, for several combinations of recommendations and outcomes, the effect was not significant. We believe that the reason is that guidelines represent workflows and the effect of the workflow goes beyond the effects of the individual guideline recommendations. For example, by considering the recommendations outside the context of the workflow, we may ignore whether the intervention addressed the condition that triggered its administration. If low MAP triggered the administration of vasopressors, without considering the workflow, we do not know whether MAP returned to the normal levels thereafter. Thus we cannot equate an adverse outcome with the failure of the guideline, it may be the result of the insufficiency of the intervention. Moving forward, we are going to model the workflows behind the guidelines and apply the same principles that we developed in this work to estimate the effect of the entire workflow.

This phase of our study did not address the timing of recommendations nor the time prior to TimeZero. For this analysis, guideline compliance was considered only after TimeZero (the estimated onset), since compliance with SSC is only necessary in the presence of suspected or confirmed sepsis. There is no reason to suspect sepsis before TimeZero. However, some interventions may have started earlier, without respect to sepsis. For example, 100% of the patients in this sample had antibiotics (potentially preventive antibiotics), but only 99 (55%) patients received it after TimeZero.

The EHR does not provide date and time for certain ICD-9 diagnoses. During a hospital stay, all new diagnoses are recorded with the admission date. We know whether a diagnosis was present on admission or not, thus we know whether it is a preexisting or new condition, but do not know precisely when the patient developed this condition during the hospitalization. For this reason, we are unable to detect whether the SSC guideline was applied before or after a complication occurred, thus we may underestimate the beneficial effect of some of the recommendations. For example, high levels of lactate is highly related to hypoxia and pulmonary damage[9]. If these patients were checked for lactate after pulmonary distress, we would consider the treatment compliant with the Lactate recommendation, but we would not know that the respiratory distress was already present at the time of the lactate measurement and we would incorrectly count it as a complication that the guideline failed to prevent.

Conclusion and Future Research

This study demonstrated that retrospective EHR data could be used to estimate compliance with individual guideline recommendations in the SSC guideline. Further, EHR data can be used to estimate the effect of guideline adherence on sepsis-related complications in patients with severe sepsis and septic shock. We found that most treatment courses we observed were compliant with many guideline recommendations and were able to demonstrate these recommendations have significant beneficial (protective) effect on some outcomes. Since guidelines encapsulate a workflow, which goes beyond a mere collection of recommendations, further study is needed to prove the beneficial effect of the entire SSC workflow. The next step in this study is to address timing of guideline recommendations and the impact on multiple outcomes.