Prospective Observational Study Comparing Sepsis-2 and Sepsis-3 Definitions in Predicting Mortality in Critically Ill Patients.

BACKGROUND: Sepsis definitions have evolved, but there is a lack of consensus over adoption of the most recent definition, Sepsis-3. We sought to compare Sepsis-2 and Sepsis-3 in the classification of patients with sepsis and mortality risk at 30 days.
METHODS: We used the following definitions: Sepsis-2 (≥2 systemic inflammatory response syndrome criteria + infection), Sepsis-3 (prescreening by quick Sequential Organ Failure Assessment [qSOFA] of ≥2 of 3 criteria followed by the complete score change ≥2 + infection), and an amended Sepsis-3 definition, iqSOFA (qSOFA ≥2 + infection). We used χ 2 or Wilcoxon rank-sum tests, receiver-operator characteristic curves, and survival analysis.
RESULTS: We enrolled 176 patients (95% in an intensive care unit, 38.6% female, median age 61.4 years). Of 105 patients classified by Sepsis-2 as having sepsis, 80 had sepsis per Sepsis-3 or iqSOFA (kappa = 0.72; 95% confidence interval [CI], 0.62-0.82). Twenty-five (14.8%) died (20 of 100 with sepsis per Sepsis-2 [20%], and 20 of 77 [26.0%] with sepsis per Sepsis-3 or iqSOFA). Results for Sepsis-3 and iqSOFA were identical. The area under the curve of receiver-operator characteristic (ROC) curves for identifying those who died were 0.54 (95% CI, 0.41-0.68) for Sepsis-2, 0.84 (95% CI, 0.74-0.93) for Sepsis-3, and 0.69 (95% CI, 0.60-0.79) for iqSOFA (P < .01). Hazard ratios for death associated with sepsis were greatest for sepsis or septic shock per Sepsis-3.
CONCLUSIONS: Sepsis-3 and iqSOFA were better at predicting death than Sepsis-2. Using the SOFA score might add little advantage compared with the simpler iqSOFA score.

Sepsis is a common condition in the United States associated with adverse outcomes. Early recognition and prompt institution of therapy for sepsis might improve outcomes including short- and long-term mortality, readmission, persistent organ dysfunction, and poor quality of life [1-6]. Since 1992, the definition of sepsis, severe sepsis, and septic shock (Sepsis-2) has relied on the presence of infection and features of the systemic inflammatory response syndrome (SIRS) [7]. Updated definitions of sepsis and septic shock (Sepsis-3) are predicated on the presence of organ dysfunction accompanying infection [3, 8, 9]. We compared how Sepsis-2 and Sepsis-3 classified patients as having sepsis or septic shock, and we predicted the subset of patients who would not survive. Our null hypothesis is that Sepsis-2 and Sepsis-3 do not differ in these regards.

METHODS

Study Design and Population

We studied a cohort assembled initially for a prospective observational study of a novel biomarker of sepsis (ClinicalTrials.gov NCT02052895, first posted February 3, 2014). Patients were enrolled from January 2014 through June 2015. We screened patients who were admitted to 2 medical centers (Tufts Medical Center, Boston, MA and Lahey Hospital and Medical Center, Burlington, MA), through the emergency department, to an intensive care unit (ICU) from the general wards, transferred from other institutions or who had blood cultures ordered. Our inclusion criteria were as follows: age ≥21 years, presence of ≥2 SIRS criteria [7], enrollment within 18 hours of the development of SIRS, and ability of the patient or representative to provide informed consent. Two infectious disease physicians independently determined and confirmed the presence or absence of infection at presentation or shock within the first 24 hours of enrollment using clinical data from the first 7 days of study entry and specific criteria according to the International Sepsis Forum Definition of Sepsis and the Centers for Disease Control National Healthcare Safety Network definitions of infection [10, 11]. Disagreements were decided by a third physician. The study was approved by the Institutional Review Boards of the involved centers. Informed consent was obtained for all patients.

Definitions

Definitions [3, 7, 9] are summarized in Supplementary Table 1.

Data Collection

We collected data for the first 7 days after enrollment including demographic data, history of present illness, physical and physiologic findings, laboratory, microbiological and radiographic data, medication history, and assessments of the primary team. Follow-up was for 30 days after enrollment, at which time we contacted subjects or their representatives to determine vital status.

We calculated the modified Sequential Organ Failure Assessment (SOFA) and quick SOFA (qSOFA) scores from data obtained within the first 24 hours of enrollment. Data for the Glasgow Coma Score and measures of blood oxygen were available only at the time of enrollment. We assumed that patients had a premorbid SOFA score of zero as has been done by others [12]. Due to the number of patients without direct arterial PaO2 measurement, we imputed the PaO2/FiO2 ratio for the SOFA score from oxygen saturation values [13]. Some patients were placed on vasopressors not specified in the Sepsis-3 definition [9]. In that event, if the patient was placed on vasopressin, we assigned a cardiovascular component score of 3 if the dose used was 0.04 units/minute and 4 if greater than that dose. If the patient was placed on phenylephrine, we assigned a cardiovascular component score of 2 if the dose used was less than 50 mg/minute, 3 if the dose was 51–200 mg/minute, and 4 if greater than 200 mg/minute (written personal communication, M. Singer, August 10, 2016).

Outcomes

The primary study outcome was mortality within 30 days of enrollment.

Statistical Analysis

We compared patient characteristics using χ 2 or Wilcoxon rank-sum test. We used the simple kappa statistic to compare classifications of patients by sepsis definitions. We used receiver-operator characteristic (ROC) curve analysis to test the ability of definitions to discriminate between those who lived and died within 30 days of enrollment. We estimated the univariate risk of death with survival analysis and adjusted the hazard ratio (HR) with single covariates. We performed all analyses using SAS version 9.4 (SAS Institute, Cary, NC) with a 2-sided alpha level of 0.05.

RESULTS

Patient Characteristics

We enrolled 196 patients between January 2014 and June 2015, 183 (95.3%) cared for in an ICU. Four were excluded because entry criteria were not met or consent was revoked. Of the remaining subjects, 176 (92%) had complete data for Sepsis-2 determinations and calculation of SOFA and qSOFA scores for Sepsis-3 determinations, and, of these, 167 (94.9%) were cared for in an ICU. Two physician adjudicators agreed on the presence or absence of infection in 166 (94%). A third physician adjudicator was used to complete the assessment in the remaining 10 (6%). Patients with sepsis according to the Sepsis-2 definition did not differ in baseline characteristics except need for ICU care compared with patients with at least 2 SIRS criteria but no infection and who therefore did not meet the criteria for Sepsis-2 (Table 1). One hundred five patients (59.6%) met Sepsis-2 criteria. There were 123 sites of infection and 18 patients with more than 1 site of infection (Supplementary Table 2). Microbiological diagnoses were found for 66 patients, 21 of which were polymicrobial.

Table 1.

Baseline (Within 24 Hours of Enrollment) Demographic and Clinical Characteristics of Study Patients With and Without Sepsis According to the Sepsis-2 Definition

	Total Cohort	SIRS ≥2 + Infection	SIRS ≥2 + No Infection
	N = 176	N = 105	N = 71
Characteristic	(%)	(%)	(%)	P a
Female sex	68 (38.6)	45 (42.9)	23 (32.3)	.16
Age (median, IQR)	61.4 (47.7–70.9)	62.2 (48.0–72.3)	60.0 (46.7–68.2)	.26
ICU care	167 (94.9)	96 (91.4)	71 (100)	.01
Race
White	86 (80.7)	86 (81.9)	56 (78.9)	.29
Black	12 (6.8)	9 (8.6)	3 (4.2)
Asian	9 (5.1)	5 (4.8)	4 (5.6)
Other or unreported	13 (7.4)	5 (4.8)	8 (11.3)
Study Site 1	124 (70.5)	74 (70.5)	50 (70.4)	.99
Documented statin therapy on enrollment	43 (24.4)	27 (25.7)	16 (22.5)	.63
Immunosuppression	44 (25.0)	26 (24.8)	18 (25.4)	.93
APACHE III (median, IQR)	47.0 (34.5–62.5)	48.0 (37.0–61.0)	46.0 (31.0–63.0)	.38
SOFA ≥2	175 (99.4)	104 (99.1)	71 (100)	.41
qSOFA ≥2	126 (71.6)	80 (76.2)	46 (65.8)	.10
Received vasoactive drug(s)	44 (25.0)	30 (28.6)	14 (19.7)	.18
Lactate >2 mmol/L	84 (47.7)	48 (45.7)	36 (50.7)	.52

Abbreviations: APACHE, Acute Physiologic Assessment and Chronic Health Evaluation; ICU, intensive care unit; IQR, interquartile range; qSOFA, quick Sequential Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.

aχ 2 test, Fisher’s exact test, or Wilcoxon rank-sum test, as appropriate.

Comparison of Sepsis-2 and Sepsis-3 Definitions of Sepsis

When not using qSOFA ≥2 as a prescreening step for defining sepsis per Sepsis-3, Sepsis-2 and unscreened Sepsis-3 were almost identical in classifying patients as having sepsis (Table 2). Twenty-four (23.1%) patients who met the unscreened Sepsis-3 definition and the Sepsis-2 definition had a qSOFA score <2. One additional patient who met the Sepsis-2 definition and had a qSOFA score <2 did not meet unscreened Sepsis-3 criteria due to a low SOFA score. When we incorporated a prescreening step of using qSOFA ≥2 in the Sepsis-3 definition, all 71 patients classified by Sepsis-2 as not having sepsis were similarly classified by qSOFA-prescreened Sepsis-3. However, 25 of 105 patients classified as having sepsis by Sepsis-2 did not meet sepsis criteria by qSOFA-prescreened Sepsis-3 (kappa = 0.72; 95% confidence interval [CI], 0.62–0.82). We performed all subsequent analyses using a Sepsis-3 definition with prescreening by qSOFA unless otherwise mentioned. The iqSOFA definition of sepsis (Supplementary Table 1) modification of the Sepsis-3 definition classified patients identically to the qSOFA-prescreened Sepsis-3 definition.

Table 2.

Sepsis Diagnoses and Death According to Definition of Sepsis

Diagnosis	Sepsis-2	Sepsis-3 No Prescreening	Sepsis-3 Prescreening for qSOFA ≥2
No sepsis	71	72	96
Sepsis	105	104	80
Severe sepsis	94	--	--
Septic shock	40	26	26
Deatha
Without sepsis (%)	5 (7.25)	5 (7.25)	5 (5.4)
With sepsis (%)	20 (20)	20 (20)	20 (26.0)
With severe sepsis (%)	19 (20.9)	--	--
With septic shock (%)	15 (39.5)	13 (52.0)	13 (52.0)

Abbreviations: qSOFA, quick Sequential Organ Failure Assessment.

aVital status at 30 days was available for 169 patients.

All 94 patients classified as having severe sepsis by Sepsis-2 met unscreened Sepsis-3 criteria, whereas 10 patients meeting unscreened Sepsis-3 criteria did not meet criteria for severe sepsis by Sepsis 2, 1 of whom died (data missing for 1 patient). Twenty of 94 patients with severe sepsis did not meet qSOFA-prescreened Sepsis-3 criteria, and all survived to 30 days.

Comparison of Sepsis-2 and Sepsis-3 Definitions of Septic Shock

Of the 40 patients classified as having septic shock per Sepsis-2, 14 did not meet criteria for septic shock per Sepsis-3 (kappa = 0.74; 95% CI, 0.62–0.87) (Table 2), all with lactate <2 mmol/L. All 136 patients without septic shock per Sepsis-2 were identically classified as not having septic shock per Sepsis-3. Results did not change when applying a Sepsis-3 definition that omitted the screening step of qSOFA ≥2.

Mortality

Mortality at 30 days was 14.8% in the 169 of 176 patients for whom mortality data were available (Table 2). Of note, all 23 of 24 patients who had qSOFA <2 but met unscreened Sepsis-3 criteria and for whom mortality data were available survived. Mortality results for the iqSOFA definition of sepsis were identical to the prescreened Sepsis-3 definition.

We examined the ability of the definitions to classify patients with sepsis into those who did and did not survive (Figure 1). The Sepsis-3 definition used the greatest number of points (9) and had an area under the curve (AUC) significantly greater than for the Sepsis-2 definition (P = .002) or the iqSOFA definition (P = .007), each of which used 3 points.

Figure 1.

Comparison of receiver-operator characteristic curves for ability to predict mortality. Sepsis-2: dot-dash line, area under the curve (AUC) = 0.54, 95% confidence interval (CI) = 0.41–0.68. Sepsis-3: solid line, AUC = 0.84, 95% CI = 0.74–0.93. iqSOFA: dashed line, AUC = 0.69, 95% CI = 0.60–0.79.

We found that patients meeting the Sepsis-3 definitions of sepsis and septic shock had a higher risk of death (HR) compared with those meeting Sepsis-2 definitions (Table 3 and Figure 2). The HR for death when we compared sepsis or severe sepsis defined by Sepsis-2 was not different. The HR for death associated with a Sepsis-2 or Sepsis-3 diagnosis of sepsis or septic shock increased somewhat when we adjusted for male sex (Table 3). There were small changes when we adjusted for age. Also of note, the HR for death associated with sepsis per Sepsis-2 adjusted for age, qSOFA score greater than or equal to 2, or the need for pressors trended toward but did not attain statistical significance. The HR for death associated with septic shock per Sepsis-2 remained statistically significant (Table 3) after adjustment for qSOFA ≥2. The HR for death associated with sepsis or septic shock per Sepsis-3 were not adjusted for qSOFA score greater than or equal to 2 or the need for pressors because these covariates were part of the definitions for sepsis and septic shock per Sepsis-3. Likewise, the risk estimate of death associated with septic shock per Sepsis-2 was not adjusted for the need for pressors. Results for iqSOFA and Sepsis-3 definitions of sepsis were identical.

Table 3.

Adjusted Survival Analysis for Risk of Deatha

	Adjustment
	Male Sex		Age (Per Year Increase)		qSOFA ≥2		Pressors
Definition	HR (95% CI)	P Value	HR (95% CI)	P Value	HR (95% CI)	P Value	HR (95% CI)	P Value
Sepsis-2
Sepsis	3.10 (1.15–8.36)	.03	2.60 (0.96–7.02)	.06	2.47 (0.92–6.68)	.07	2.55 (0.95–6.89)	.06
Severe Sepsis	3.02 (1.19–7.67)	.02	2.53 (1.00–6.44)	.05	--	--	--	--
Septic Shock	6.89 (2.99–15.9)	<.0001	5.86 (2.55–13.4)	<.0001	4.32 (1.79–10.4)	.001	--	--
Sepsis-3
Sepsis	5.38 (1.99–14.5)	.0009	4.49 (1.66–12.1)	.003	--	--	--	--
Septic Shock	9.06 (3.97–20.7)	<.0001	8.59 (3.77–19.6)	<.0001	--	--	--	--

Abbreviations: CI, confidence interval; HR, hazard ratio; qSOFA, quick Sequential Organ Failure Assessment.

aSee Supplementary Table 3 for univariable analyses.

Figure 2.

Kaplan-Meier Analysis for death for those with and without sepsis. x-axis: days followed until death, loss to follow-up, or alive at 30 days; each axis tick represents 5 days. y-axis: survival distribution function, 0–1.00; each axis tick represents 0.2. Lines represent subjects who did (dashed lines) or did not (solid lines) have sepsis or septic shock according to the indicated definitions. Survival curves were compared by the log-rank test. (A) Sepsis-2 (P = .04); (B) Sepsis-3 (P = .0006), (C) septic shock according to Sepsis-2 (P < .0001); (D) septic shock according to Sepsis-3 (P < .0001).

DISCUSSION

Publication of Sepsis-3 came with the recommendation that new studies would be needed to validate the importance and utility of these new definitions. Our prospective study revealed that qSOFA-prescreened Sepsis-3 and iqSOFA are more stringent definitions than Sepsis-2 in detecting sepsis and septic shock. We found that some patients meeting Sepsis-2 criteria did not meet qSOFA-prescreened Sepsis-3 criteria. However, omitting the screening qSOFA step in the Sepsis-3 definition resulted in almost no difference in classification of sepsis when using Sepsis-2 or Sepsis-3. Sepsis-3 and iqSOFA were also better than Sepsis-2 at identifying patients classified as having sepsis or septic shock at higher risk of death. We also showed that iqSOFA classified patients as having sepsis in an identical manner to the prescreened Sepsis-3 definition despite its reduced complexity and increased ease of administration. However, AUC of ROC for identifying patients who would die was significantly greater if diagnosed with sepsis by Sepsis-3 than those for either iqSOFA or Sepsis-2. This is possibly due to the ability to create far more data points on the ROC for Sepsis-3 compared with the few points of data available for plotting ROC for iqSOFA or Sepsis-2.

Results of ROC analyses for the prediction of in-hospital mortality in a retrospective study of ICU patients for Sepsis-2, Sepsis-3, and a definition identical to iqSOFA were very similar to ours [12]. However, the crude mortality rates of the 3 definitions were very similar, differing from our findings. A possible explanation is that patients who meet Sepsis-3 but not Sepsis-2 criteria have a lower mortality than patients with Sepsis-2 [14]. We were unable to perform this comparison because all patients with infection in our study met Sepsis-2 criteria.

Studies of the prediction of mortality by Sepsis-3 definitions in emergency department patients found that a definition of sepsis equivalent to iqSOFA performed better than Sepsis-2 for predicting death in patients with sepsis or severe sepsis diagnoses when comparing area under the ROC curves (AUROCs) or mortality rates [15, 16]. Furthermore, one of these studies found that the iqSOFA equivalent was similar to if not slightly better than mortality prediction using Sepsis-3 [15] when comparing AUROCs. In survival analysis, the iqSOFA equivalent and Sepsis-3 definitions had very similar HRs for mortality. An accompanying editorial observed that qSOFA in the presence of infection might be helpful in identifying those patients at risk for sepsis [17].

These 2 studies in conjunction with our own support supplanting Sepsis-2 with Sepsis-3 or iqSOFA [12, 15]. This change in the approach to the diagnosis of sepsis has been rejected by some, with the concern that the higher specificity of Sepsis-3 would result in reduced screening for sepsis and a delay in recognition and treatment. A recent study of emergency department patients with prospective and retrospective components led to the proposal to retain the use of the Sepsis-2 definition [14]. The authors found that the presence of SIRS was equal to meeting at least 2 qSOFA criteria in identifying patients with organ dysfunction as defined by Sepsis-3. Patients with severe sepsis as defined by Sepsis-2 had mortality rates almost identical to those who met Sepsis-3 criteria but greater than patients meeting Sepsis-3 criteria but not Sepsis-2 criteria. Because the Sepsis-2 definition was associated with organ dysfunction and mortality, the authors argued that abandoning the incorporation of SIRS into a definition of sepsis should be reconsidered. Others have found that qSOFA performed less well than other measures but better than meeting at least 2 SIRS criteria as a screen for death or ICU transfer of emergency department or ward patients with suspected infection [18], cautioning against supplanting qSOFA as a screen for sepsis. For all of these considerations, care must be taken in extrapolating findings from particular studies to patient populations because the performance of both SIRS and qSOFA in predicting poor outcomes in the presence of infection varies depending upon the population studied.

Another study suggested caution in abandoning the Sepsis-2 definition of septic shock [19]. In this secondary analysis of 2 combined studies of sepsis, 57% of patients meeting the Sepsis-2 definition did not meet the Sepsis-3 definition for septic shock. However, those 57% patients still had a substantial mortality rate of 14%, even though this was far less than the mortality rate for patients meeting the Sepsis-3 definition of septic shock (29%). Furthermore, the authors found that patients who met only the Sepsis-2 definition of septic shock had lower mortality if treated (compared to not treated) with a sepsis treatment protocol under study. They suggest that abandoning the Sepsis-2 definition of septic shock could result in delayed recognition and treatment of sepsis in the population that could most benefit.

The differences between studies might be due to the different patient populations studied, different study designs, different methods of diagnosing infection, and potentially inaccurate SOFA scoring due to missing SOFA score components in 30% of patients [15]. In fact, others have proposed using sepsis definitions for screening but not clinical management due to such heterogeneity [20]. It should be underscored that sepsis screening is not the equivalent of estimating the prognosis of sepsis. Rather, sepsis screening is a tool to be used to identify patients who would benefit from treatments designed to enhance outcomes after a sepsis diagnosis.

Our study’s strengths were its prospective study design, adjudication of infection by a panel of experts, and more complete data collection for SOFA and iqSOFA scoring compared with other studies that excluded 14%–30% of potentially eligible patients due to missing data elements compared with our study, in which 8% of potentially eligible patients had missing data for determining SIRS, qSOFA, or SOFA scores [8, 12, 15]. Limitations included its small size, which might have masked differences between SOFA and iqSOFA by type II error and/or limited the ability to comprehensively adjust the survival analysis models. Additional limitations were the inability to study patients who met criteria for Sepsis-3 but not Sepsis-2 due to the entry criterion of meeting at least 2 SIRS criteria and the possibility of bias introduced by loss to follow-up.

CONCLUSIONS

Our patients were primarily those cared for in medical or surgical ICUs enrolled from the emergency department, upon transfer from another facility, or after admission to the general wards. It is not clear how our results will pertain to a less ill cohort of patients or patients who are more homogeneous such as obstetrical or postsurgical patients who develop sepsis. Future studies should (1) be multicenter, with a broad array of patients, (2) use a prospective design with acquisition of sufficient data elements for calculation of SIRS, qSOFA, and iqSOFA scores, and (3) use experts to adjudicate the presence or absence of infection. Patients who are misclassified by one definition or another should be studied in terms of characteristics and outcomes because this information will be useful for carefully honing a sepsis definition that functions as a sensitive screen and identifies patients at risk (or not at risk) for adverse outcomes. Finally, our studies and others have not presented a consensus of the utility of qSOFA compared with the full SOFA score in identifying patients at risk for increased mortality in the setting of infection. This is an important topic for future study especially because the qSOFA is much simpler to calculate and might present a predictive advantage over the full SOFA score.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.