Risk Adjustment for Sepsis Mortality to Facilitate Hospital Comparisons Using Centers for Disease Control and Prevention's Adult Sepsis Event Criteria and Routine Electronic Clinical Data.

Variability in hospital-level sepsis mortality rates may be due to differences in case mix, quality of care, or diagnosis and coding practices. Centers for Disease Control and Prevention's Adult Sepsis Event definition could facilitate objective comparisons of sepsis mortality rates between hospitals but requires rigorous risk-adjustment tools. We developed risk-adjustment models for Adult Sepsis Events using administrative and electronic health record data.
DESIGN: Retrospective cohort study.
SETTING: One hundred thirty-six U.S. hospitals in Cerner HealthFacts (derivation dataset) and 137 HCA Healthcare hospitals (validation dataset). PATIENTS: A total of 95,154 hospitalized adult patients (derivation) and 201,997 patients (validation) meeting Centers for Disease Control and Prevention Adult Sepsis Event criteria.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: We created logistic regression models of increasing complexity using administrative and electronic health record data to predict in-hospital mortality. An administrative model using demographics, comorbidities, and coded markers of severity of illness at admission achieved an area under the receiver operating curve of 0.776 (95% CI, 0.770-0.783) in the Cerner cohort, with diminishing calibration at higher baseline risk deciles. An electronic health record-based model that integrated administrative data with laboratory results, vasopressors, and mechanical ventilation achieved an area under the receiver operating curve of 0.826 (95% CI, 0.820-0.831) in the derivation cohort and 0.827 (95% CI, 0.824-0.829) in the validation cohort, with better calibration than the administrative model. Adding vital signs and Glasgow Coma Score minimally improved performance.
CONCLUSIONS: Models incorporating electronic health record data accurately predict hospital mortality for patients with Adult Sepsis Events and outperform models using administrative data alone. Utilizing laboratory test results, vasopressors, and mechanical ventilation without vital signs may achieve a good balance between data collection needs and model performance, but electronic health record-based models must be attentive to potential variability in data quality and availability. With ongoing testing and refinement of these risk-adjustment models, Adult Sepsis Event surveillance may enable more meaningful comparisons of hospital sepsis outcomes and provide an important window into quality of care.

There is intense interest in improving and measuring the quality of sepsis care provided by U.S. hospitals (1). Current benchmarking efforts, such as the Centers for Medicare and Medicaid Services Severe Sepsis and Septic Shock Early Management Bundle ("SEP-1"), primarily focus on compliance with time-sensitive process measures rather than sepsis outcomes (2). The development of a reliable risk-adjusted sepsis outcome measure could complement process metrics and provide a broader window into variations in quality of care (3).

Previous studies have documented wide variability in hospital risk-adjusted sepsis mortality rates using administrative data (4–6), but these are potentially confounded by large variations in diagnosis, documentation, and coding practices for sepsis and organ dysfunction that markedly affect perceived levels of severity of illness (7). Surveillance using electronic health record (EHR) data allows for more consistent case finding while still allowing for the possibility of automated national surveillance.

In March 2018, the U.S. Centers for Disease Control and Prevention (CDC) released the “Adult Sepsis Event” surveillance definition that uses objective clinical data routinely available in EHRs to identify sepsis (8). Compared with the Sequential Organ Failure Assessment score used in Sepsis-3 criteria, the Adult Sepsis Event definition identifies similar types of patients, has comparable or better predictive validity for mortality, and may facilitate more widespread automated surveillance due to its simpler organ dysfunction criteria (9). Although primarily meant to help hospitals track internal sepsis rates and outcomes, a standardized surveillance definition could serve as the foundation for credible hospital comparisons if coupled with rigorous risk adjustment (10).

Sepsis risk-adjustment models using administrative data already exist (4, 11, 12), sometimes supplemented with manually collated clinical data (e.g., the Surviving Sepsis Campaign database, mandatory reporting requirements in New York state) (13, 14). There are also validated mortality prediction scores for critically ill patients, including the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS) scores (15–17). These models, however, may not be ideal for routine operational benchmarking of sepsis outcomes because they either have limited granularity and variable accuracy (e.g., administrative data), require clinical parameters infrequently available in structured electronic format (e.g., urine output, mental status, chills with rigors), or are derived from one geographic location (e.g., New York state). Scores specific to the ICU also may not suffice because more than 50% of sepsis patients never require critical care services (18, 19).

Our objective was to develop and validate risk-adjustment models for Adult Sepsis Events in two large cohorts of U.S hospitals using EHR and administrative data. In doing so, we examined the incremental benefit of successive sets of covariates to understand the balance between data collection burden and model performance.

METHODS

Data Sources

This study was approved by the Institutional Review Board at Harvard Pilgrim Health Care Institute. Our dataset for model development was Cerner HealthFacts, a deidentified database that contains detailed clinical data from U.S. academic and community hospitals that use the Cerner EHR system (Cerner Corporation, Kansas City, MO) (9, 20–24). We included all adults (≥ 20 yr old) admitted to 136 hospitals from January 2009 to September 2015 (20). We externally validated all models using 2013–2014 data from 137 hospitals in the HCA Healthcare network (20). HCA Healthcare includes urban, suburban, and rural community medical centers across 20 states and primarily use the Meditech EHR system (Medical Information Technology Incorporated, Westwood, MA) (25). Comprehensive clinical data are stored centrally and undergo line-item validation until more than 99% accuracy is achieved (20). Study hospital characteristics are summarized in Supplemental Table 1 (Supplemental Digital Content 1, http://links.lww.com/CCX/A104). Encounters with missing discharge dispositions and with International Classification of Diseases, 10th Revision, Clinical Modification codes instead of International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes were excluded.

Outcomes and Predictors

The primary outcome for our models was in-hospital mortality among patients meeting Adult Sepsis Event criteria during hospitalization. This definition flags “presumed serious infection” (blood culture order and new antibiotics continued for ≥ 4 d or until ≤1 d prior to death, discharge to hospice, or transfer to another hospital) and “concurrent organ dysfunction” (initiation of vasopressors or mechanical ventilation, elevated lactate, increase in baseline creatinine or total bilirubin, or decrease in baseline platelets) (for complete CDC criteria, see Supplemental Table 2, Supplemental Digital Content 1, http://links.lww.com/CCX/A104) (8).

Potential predictors from administrative and EHR data included demographics, admission source, comorbidities, infectious syndromes, laboratory tests, medications, mechanical ventilation initiation, blood culture results, vital signs, and Glasgow Coma Scale (GCS) measurements (Table ). For physiologic variables (laboratory tests, vital signs, GCS), we used the most abnormal values within ±1 calendar day from the “day of sepsis onset,” defined as the earliest day the blood culture or first qualifying antibiotic day occurred (concurrent with organ dysfunction). Elixhauser comorbidities (26) were supplemented with other comorbidities (based on ICD-9-CM codes) that might influence sepsis mortality: leukemia (204–208), stem cell transplant (V42.81 and V42.82), and solid organ transplant (V42.0–42.7, V42.83, V42.84, V42.89, V42.9). ICD-9-CM codes for infectious syndromes were adapted from prior work (27, 28).

We rated the qualities of each potential predictor by “expected availability” (i.e., ubiquity in different EHR systems) and “objectivity” (i.e., consistent ascertainment between clinicians and hospitals) (Table 1). Administrative data, for example, should be ubiquitous, whereas mental status is not routinely recorded in all EHR systems. Basic vital signs and laboratory data are generally objective, but diagnosis and coding practices for infections and organ dysfunction differ between hospitals (7).

TABLE 1.

Overview of Predictors and Relevant Surveillance Characteristics

Statistical Methods

We performed exploratory univariate analyses of the associations between each potential predictor and in-hospital mortality. For continuous physiologic variables that did not appear to have a linear association with the log odds of mortality, we included quadratic terms in the model. We then constructed regression models of incremental complexity. We first used administrative candidate predictors (“basic administrative model”): demographics, comorbidities, admission source, infectious diagnoses, ICU admission, and days from hospital admission to sepsis onset. We then implemented an “advanced administrative model” developed by Ford et al (11) that incorporates demographics, comorbidities, and proxies for severity of illness at admission (codes for mechanical ventilation, shock, hemodialysis, and ICU admission) (11).

We then added clinical data in a sequential fashion based on their expected availability. For the simplest model (clinical model 1), we only incorporated the data elements required to identify Adult Sepsis Events (vasopressors, mechanical ventilation, lactate, creatinine, bilirubin, and platelet counts on the day of sepsis onset). Subsequent models included additional laboratory data from chemistries, complete blood cell counts, and liver function tests (clinical model 2), and vital signs and GCS measurements (clinical model 3). Quadratic terms for continuous clinical variables were added to the model, and backward elimination was performed at a significance level of 0.05. To keep the model hierarchically well formulated, the linear term was retained in the model if the corresponding quadratic term was significant. To properly quantify the uncertainty in parameter estimates accounting for the clustering effect within hospitals, we used a sandwich variance estimator robust to misspecification of the working correlation structure (29).

All models were developed using two thirds of the Cerner cohort and internally validated on the remaining third. The models were then applied to the HCA Healthcare dataset for external validation using the coefficients developed with Cerner, with the exception of clinical model 3 as vital signs and GCS measurements were unavailable from HCA.

We handled missing severity-of-illness covariates in two ways in Cerner: multiple imputation (supplemental methods, Supplemental Digital Content 1, http://links.lww.com/CCX/A104) and imputation of normal values (for normal values, see Supplemental Table 3, Supplemental Digital Content 1, http://links.lww.com/CCX/A104). We compared results from multiple imputation versus normal imputation, as is commonly done with clinical severity-of-illness scores (30–32). We reasoned that if successful, normal imputation could facilitate real-world application of our risk-adjustment models in facilities where more complicated strategies for handling missing data might be impractical. Performance of clinical models 1 and 2 was similar using both imputation methods (described below); thus, we report the model results using normal value imputation. Due to a high quantity of missing vital sign data in Cerner (Supplemental Fig. 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A104), for clinical model 3, we limited the cohort to patients with nonmissing vital signs (since missing vital sign data are likely related to the ability of hospitals to electronically capture and report vitals, rather than a decision by clinicians not to measure vitals). Of note, the number of hospitals reporting vital signs increased in later years in the study period, and the characteristics of Adult Sepsis Event patients with and without missing vital signs were generally similar (Supplemental Table 4, Supplemental Digital Content 1, http://links.lww.com/CCX/A104). When applying models to the HCA Healthcare dataset, normal values were imputed for missing covariates.

We assessed model discrimination using the area under the receiver operating characteristic curve (AUROC) and calibration using the adjusted Brier score (33) and calibration plots. The Hosmer-Lemeshow goodness-of-fit tests were performed on 1,000 random samples of size 1,000 with group size set to 100. Standardized mortality ratios were calculated. Analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC) and R version 3.3.1 (R Core Team, Vienna, Austria; http://r-project.org).

RESULTS

Study Cohort and Patient Characteristics

Derivation of the study cohort is shown in Figure . The primary Cerner dataset included 136 hospitals; representation was highest from the South (35%), 65% of hospitals had less than 200 beds, and 60% were nonteaching hospitals. This cohort included 2,221,032 patients, of whom 95,154 met Adult Sepsis Event criteria and 17,876 died (18.8% mortality). Median age was 68 years (interquartile range, 55–79), and comorbidities were common including diabetes (29%), chronic lung disease (23%), and congestive heart failure (22%) (Table 2). The most common infection sites were urinary tract (27%), lungs (25%), and intra-abdominal space (16%). The most common organ dysfunctions were elevated lactate (40%), acute kidney injury (39%), and hypotension requiring vasopressors (32%). The HCA validation cohort included 201,997 hospitalizations with Adult Sepsis Events.

TABLE 2.

Characteristics of Sepsis Patients in Primary Cerner HealthFacts Dataset

Flowchart for study cohort derivation in primary dataset (Cerner HealthFacts) (A) and external validation dataset (HCA Healthcare) (B). ICD-9 = International Classification of Diseases, 9th Edition, ICD-10 = International Classification of Diseases, 10th Edition.

Model Performance

Model performance for discriminating in-hospital mortality is summarized in Figure . The basic administrative model achieved an AUROC of 0.726 (95% CI, 0.719–0.733) in the Cerner validation cohort. The advanced administrative model achieved an AUROC of 0.776 (95% CI, 0.770–0.783). Adding the clinical data elements required to identify Adult Sepsis Event organ dysfunction further improved performance (clinical model 1; AUROC, 0.817; 95% CI, 0.811–0.822), but there was only modest incremental improvements thereafter with adding more laboratory data (clinical model 2; AUROC, 0.826; 95% CI, 0.820–0.831), and vital signs and GCS (clinical model 3; AUROC, 0.840; 95% CI, 0.833–0.848). Performance of the clinical models was similar when using multiple imputation or normal value imputation for missing laboratory covariates (Supplemental Table 5, Supplemental Digital Content 1, http://links.lww.com/CCX/A104). AUROC values in the external HCA Healthcare validation dataset were nearly identical. For example, the AUROC in HCA for clinical model 1 was 0.819 (95% CI, 0.816–0.821) and 0.827 (95% CI, 0.824–0.829 CI) for clinical model 2.

Performance of each risk-adjustment model in discriminating for in-hospital mortality among patients with Adult Sepsis Events (ASE). The internal validation cohort was a subset of the Cerner HealthFacts dataset (n = 31,718). The external validation cohort was the HCA Healthcare dataset (n = 201,997). The results for clinical model 3 shown above are from development and internal validation in Cerner in patients with nonmissing vital signs (n = 16,714 in validation cohort). Vital signs and Glasgow Coma Scale (GCS) were unavailable in the HCA Healthcare validation dataset. AST = aspartate aminotransferase, AUROC = area under the receiver operating curve, Bili = total bilirubin, Cr = creatinine, Hct = hematocrit, INR = international normalized ratio, Mech. Vent = mechanical ventilation, Na = sodium, Plts = platelet count, RR = respiratory rate, SBP = systolic blood pressure, Temp = temperature.

Clinical models 1 and 2 are shown in Supplemental Tables 6 and 7 (Supplemental Digital Content 1, http://links.lww.com/CCX/A104), whereas the distributions of the clinical variables are shown in Supplemental Table 8 (Supplemental Digital Content 1, http://links.lww.com/CCX/A104). In general, basic chemistries and blood cell counts were near complete in both datasets (< 5–10% missing), but lactate was missing in 46% of patients in Cerner and 32% in HCA (Supplemental Fig. 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A104). A variable importance plot illustrating the relative contribution of each set of covariates is shown in Supplemental Figure 2 (Supplemental Digital Content 1, http://links.lww.com/CCX/A104).

Model Calibration

Calibration of clinical models 1 and 2 was excellent across all deciles of risk in the Cerner internal validation cohort but less successful in the HCA Healthcare validation cohort, with lower observed mortality rates than predicted (Fig. ). This was in the context of lower overall sepsis mortality in the HCA (15.4%) versus Cerner cohort (18.8%). By comparison, the advanced administrative model performed well at lower risk deciles, but calibration was poor in higher deciles (particularly in the HCA dataset) and worse than the clinical models. Using multiple imputation for missing covariates yielded similar calibration compared with normal value imputation for clinical model 1 in the HCA cohort, but slightly better calibration for clinical model 2 (Supplemental Fig. 3, Supplemental Digital Content 1, http://links.lww.com/CCX/A104).

Calibration plots showing observed versus predicted sepsis in-hospital mortality by decile of risk for various models in internal validation cohort (Cerner HealthFacts) (A) and external validation cohort (HCA Healthcare) (B). Clinical model 1 includes administrative data and basic clinical variables needed to compute Adult Sepsis Event criteria. Clinical model 2 includes the same but adds extended laboratory data. The advanced administrative model includes administrative data and coded severity-of-illness markers.

DISCUSSION

As patients, clinicians, administrators, and regulators increasingly focus on the quality of sepsis care provided by hospitals and their sepsis outcomes, hospitals need reliable surveillance methods and robust, generalizable risk-adjustment methods to compare outcomes across time and between institutions. We developed several risk-adjustment models for sepsis mortality based on CDC’s Adult Sepsis Event criteria and an array of clinical and administrative data in a large cohort of U.S. hospitals. A model based on the minimum clinical data elements necessary to identify Adult Sepsis Events substantially improved calibration and mortality prediction over administrative models and yielded AUROC values of 0.82, with the successive addition of more complex clinical covariates resulting in diminishing gains. Clinical models performed similarly when validated in an independent hospital system, suggesting good generalizability.

Our study builds on the recent development of CDC’s Adult Sepsis Event surveillance definition, which is more sensitive than explicit diagnosis codes for sepsis and has superior positive predictive value compared with “implicit” combinations of infection and organ dysfunction codes (20, 34, 35). Hospital sepsis mortality rates measured using explicit versus implicit administrative definitions have only moderate agreement and yield very different hospital rankings (6). The Adult Sepsis Event definition by contrast minimizes bias from variability in diagnosis, documentation, and coding practices (36). Its simpler criteria also make it easier to automate across diverse EHRs compared with the Sequential Organ Failure Assessment Score and Sepsis-3 criteria, although still retaining good concordance with Sepsis-3 and comparable if not better predictive validity for mortality (9).

Other investigators have developed sepsis risk-adjustment models, but each has important limitations for wide-scale implementation for national benchmarking. Although administrative data are readily available, models using clinical data had superior discriminatory performance and calibration at higher deciles of baseline risk compared with the advanced administrative model developed by Ford et al (11). This likely reflects the value of more granular measures of physiology and timing of sepsis onset. Our model may be more generalizable than the New York state sepsis risk-adjustment model because the New York model relies extensively on manually collected data specific to the state’s regulatory requirements (14). The Surviving Sepsis Campaign’s model was derived in a large, international prospective cohort (13); however, it includes criteria such as “chills with rigor,” “bilateral pulmonary infiltrates,” and “history suggestive of new infection” that are subjective and inconsistently captured in EHRs.

Our findings provide important insight into the tradeoffs between feasibility of data collection and model performance. Of the various models we created, clinical model 1 is appealing for operational use because it relies only on the clinical data that hospitals must already gather to identify Adult Sepsis Events. Adding additional laboratory data (clinical model 2) slightly improved performance with a modest increase in complexity. However, calibration across deciles of risk for both clinical models 1 and 2 diminished in the HCA validation dataset versus Cerner cohort, such that observed mortality was generally lower than predicted. This may reflect differences in care patterns and lower sepsis mortality overall in the HCA Healthcare system, which has focused on sepsis quality improvement for the better part of this decade. Adding vital signs and GCS (clinical model 3) provided only small incremental benefit, suggesting that the marginal value of these data may not be commensurate with the effort required to gather and store them.

Indeed, vital signs were frequently missing in Cerner and unavailable in the HCA Healthcare dataset. Conversely, laboratory data such as lactates were more often missing in Cerner. Missing data are clearly an important problem when developing and applying severity-of-illness scores; they can reflect data quality issues, clinical decisions not to test, and/or variability in practice patterns across hospitals. Lactate testing, for example, is likely influenced by provider specialty, unit location at the time of sepsis onset, and quality improvement initiatives (37, 38). Importantly, we found that missing laboratory values could be handled equally well using multiple imputation versus normal value imputation. This is reassuring since normal value imputation is more feasible for many hospitals. Calibration, however, was slightly worse with clinical model 2 in the HCA dataset using normal value imputation versus multiple imputation. This underscores the challenges inherent to applying risk-adjustment models derived in one dataset to a separate cohort with variable data availability.

Our study has important limitations. We could not perform head-to-head comparisons with the other sepsis severity scores mentioned above or existing ICU severity-of-illness models such as APACHE or SAPS. However, this reflects the practical reality that some of the data needed to calculate these scores are not available as structured data in many hospitals' EHR datasets. More importantly, many septic patients never require ICU care and focusing only on ICU-specific populations risks additional confounding from variability in hospitals' ICU capacities and admission thresholds (18, 19). We did not have data on mortality occurring after hospital discharge. Our datasets did not include insurance status, which may be an important predictor of sepsis mortality (11, 12, 14). Our models relied on comorbidities and infectious diagnoses, which may be variably diagnosed and coded across institutions. We only compared model performance using EHR versus administrative data and did not assess the potential impact of different risk-adjustment methods on hospitals' sepsis mortality rankings. This is an important area for future research. Missing data were common in both datasets, with variable quantities of missing laboratory data and unavailable vital signs in the HCA Healthcare dataset. We used ICD-9-CM codes in our models; further research is needed to refine and update our models to use ICD-10 data. Real-world implementation of our risk-adjustment models for hospital benchmarking would require periodic updating and recalibration over time, ideally using nationally representative datasets with minimal missing data. This would be analogous to the periodic updates applied to risk-adjustment models and baseline standardized infection ratio levels for hospitals' healthcare-associated infection rates reported to CDC’s National Healthcare Safety Network (39).

Finally, Adult Sepsis Event criteria do not perfectly match Sepsis-3 criteria; one prior study demonstrated 70% sensitivity based on expert clinician-adjudicated medical record reviews (20). Of note, however, the Sepsis-3 cases missed by Adult Sepsis Events tend to be mild infections with low risk of mortality (9, 20). Furthermore, there is no true “gold standard” for sepsis and there is increasing recognition that clinical care and surveillance merit different definitions (40). Adult Sepsis Events were created and optimized for widespread surveillance rather than clinical care and as such are easier to apply to routine EHR data than Sepsis-3 and the Sequential Organ Failure Assessment score (9).

In conclusion, models based on administrative and EHR data accurately predict hospital mortality for patients with Adult Sepsis Events and outperform models based on administrative data alone. A risk-adjustment model incorporating laboratory test results, vasopressors, and mechanical ventilation without vital signs may achieve a good balance between data collection needs and performance, but EHR-based models must be attentive to potential variability in data quality and availability. With ongoing testing and refinement of these risk-adjustment models, Adult Sepsis Event surveillance may enable more meaningful comparisons of hospital sepsis outcomes and provide an important new window into quality of care.