Performance of Sepsis-3 Definitions in a Middle Income Country Intensive Care Unit.

BACKGROUND: Since they were first published in 2016, Sepsis-3 definitions have not been universally accepted. Rather, they have become a source of controversy because the clinical and laboratory parameters used had been derived mainly from patients hospitalized in Intensive Care Units (ICU) in the United States.
PURPOSE: The aim of this study was to evaluate the performance of the Sepsis-3 definitions for the prediction of ICU-mortality in a Tunisian ICU population as compared to the 2003 Consensus Definitions (Sepsis-2 definitions).
METHOD: The study, conducted in an 18-bed medical-surgical ICU at the Military Hospital of Tunis (Tunisia), was retrospective in nature. From January 2012 to January 2016, all patients admitted to the ICU for sepsis, severe sepsis, or septic shock as defined according to the 2003 Consensus Definitions (Sepsis-2 consensus) were eligible for this study. The new Sepsis-3 definition was then used to classify the included patients. The primary area of interest was ICU mortality, defined as death before ICU discharge.
RESULTS: A total of 1080 patients were included during the recruitment period. When Sepsis-2 definitions were used, there was a difference in mortality only between septic shock and sepsis patients. Sepsis-3 definitions show that mortality increased from 16 % among no-dysfunction-infected patients to 30 % among patients with qSOFA ≥ 2 and 44% and 46% for sepsis or septic shock patients, respectively.
CONCLUSIONS: Sepsis-3 definitions were better than sepsis-2 definitions at stratifying mortality among septic patients admitted to an ICU of a middle-income country (Tunisia).

Introduction

Sepsis is the major health threat among all the infectious pathologies, having the biggest impact in terms of mortality before or after the era of antibiotics, especially among critically ill patients.1 The first definition of sepsis, provided in 1991 and published in 1992, revolved around the systemic inflammatory response syndrome (SIRS) criteria.2 However, many have noted the limitations of the SIRS criteria and the need for improvement, which led to a second definition in 2003.3 However, this latter definition has not changed the first classification. In 2016, an international consensus task force published the Sepsis-3 definition,4 recognizing sepsis as more complex than infection and inflammation and defining it as a “life-threatening organ dysfunction due to a dysregulated host response to infection”. In this new definition (Sepsis 3), the host response resulting in organ failure from an infection is stressed.

In contrast, the inflammation stage known as SIRS in the previous definitions « Sepsis 2 » is removed. Accordingly, the Sequential Organ Failure Assessment (SOFA) score is now crucial for classifying sepsis. So, in providing early bedside evaluation of a patient for the possibility of sepsis, the quick SOFA (qSOFA) score was used.

It should be noted that sepsis-3 definitions are not universally accepted. Rather, they have become a source of controversy. Because clinical and laboratory parameters used for the development of these definitions were derived mainly from patients hospitalized in Intensive Care Units (ICU) in the United States, the primary study’s endpoint analysis was mortality. However, the presence of organ dysfunction should also be part of this analysis. The Sepsis-3 definitions Task Force clearly points out the need to validate the newly proposed definition using databases of non-US patients. The authors highlight in particular the need to validate the utility of the qSOFA score.

Tunisia is a developing country with limited healthcare resources. A more clearly defined strategy is therefore needed for admitting very severely ill patients to Intensive care units. Currently, in our intensive care unit, patients admitted for suspected infection are classified according to sepsis-2 definitions, mainly using SIRS criteria.

The main aim of this study was to evaluate the performance of the Sepsis-3 definitions for the prediction of ICU mortality in a Tunisian ICU population compared to 2003 Consensus Definitions (Sepsis-2 definitions). The study’s secondary objective was to compare the performance of qSOFA and the SIRS criteria for the early prediction of ICU mortality.

Materials and Methods

Patients and Study design

The study was a retrospective descriptive study conducted in an 18-bed medical-surgical ICU at the Military Hospital of Tunis (Tunisia).

From January 2012 to January 2016, all patients admitted to the ICU were eligible for this study. Inclusion criteria were as follows: age >18 years, and an admission diagnosis of sepsis, severe sepsis or septic shock as defined by the 2003 Consensus Definitions (Sepsis-2 consensus). In addition, patients with incomplete data in their records or those hospitalized for less than 48 h were excluded (patients who passed away within 48 h were not excluded).

The study had been reviewed and approved by the Institutional Ethics Authorities. However, informed consent was not deemed necessary because of the retrospective and observational design of the study.

Data collection

Data were collected using standardized forms. The following information was retrieved: gender, age, Acute Physiology and Chronic Health Evaluation (APACHE) II score, Sequential Organ Failure Assessment (SOFA) score, reasons for hospitalization, etiological diagnosis, worst and best vital signs during the first ICU day, comorbidities, ICU length of stay (LOS), highest lactate level of the first day, source of infection, causal organisms, use of antimicrobials and clinical ICU outcomes. The SOFA score was determined at the time of ICU admission. In addition, laboratory variables were retrieved from the database specific to laboratory data.

Sepsis and Reclassification definitions

Patients in our database were primarily classified using Sepsis-2 definitions. According to previously published consensus, sepsis, severe sepsis, and septic shock categories were used.3 The new Sepsis-3 definition was secondarily used.4

The points used for definitions were those collected at admission to the ICU.

We calculated the quick Sequential Organ Failure Assessment (qSOFA) score by assigning patients 1 point for each of the following criteria: systolic blood pressure less than or equal to 100mmHg, a respiratory rate greater than or equal to 22 breaths/min, or altered mental status documented by the physician, using the most abnormal value during the first 24 hours of admission.

Regarding reclassification, patients defined as either sepsis according to Sepsis-3 and/or severe sepsis using Sepsis-2 definitions were considered together as severe cases. Among these severe cases, those defined as sepsis by the Sepsis-3 definitions and as uncomplicated sepsis by the Sepsis-2 definitions were considered reclassified by the Sepsis-2 definitions. The clinical cases defined as infection by the Sepsis-3 definitions and as severe sepsis by the Sepsis-2 definitions were considered reclassified by the Sepsis-3.

Study endpoint

The primary study endpoint was the ICU mortality, defined as death before ICU discharge.

Data statistical Analysis

For each patient, we primarily calculated the qSOFA score. We subsequently calculated the sensitivity and specificity of the qSOFA score greater than or equal to 2 and SIRS of the previous definitions of sepsis and severe sepsis for ICU mortality. Statistical analysis was carried out using SPSS v.20.0 (SPSS, Inc., Chicago, IL, USA.). Continuous variables are expressed as mean ± standard deviation, while categorical variables are expressed with absolute and relative frequencies. The normality assumption of continuous variables was evaluated using the Kolmogorov Smirnov criterion.

Parametric continuous variables were compared among groups using the t-test; nonparametric variables were compared using the Mann-Whitney test. The Chisquared test was used to compare categorical variables among the groups.

To assess the performances of the qSOFA to predict ICU mortality, we calculated diagnostic performances (sensitivity, specificity) for a qSOFA score of 2 or higher. We constructed a receiver operating characteristic (ROC) curve and calculated the corresponding area under the ROC curve (AUROC). The performance of qSOFA was compared to that of SIRS, mainly at least for 2 elements. The DeLong test was used to compare the AUCs of the two criteria.

All statistical analyses were 2-tailed, and a P value less than 0.05 was required for statistical significance.

Results

Study population

Out of the 3246 participants who enrolled between 2012 and 2016, we included 1080 individuals whose follow-up information was available (Figure 1).

Figure 1

Study flow chart.

Patients’ characteristics

The general characteristics of patients and clinical outcomes, source of infection, and comorbidities are given in Table 1.

Table 1

General characteristics of patients

	Sepsis-2 consensus			Sepsis-3 consensus
	Sepsis (n=322)	Severe sepsis (n=382)	Septic shock (n=376)	No-dysfonction (n=74)	qSOFA ≥ 2 (n= 379)	Sepsis (n= 360)	Septic shock (n= 267)
Age (mean ±SD, years)	52 ± 21	52 ± 21	55 ± 20	54 ± 19	56 ± 20	54 ± 18	55 ± 21
Males (n, %)	196 (60.8)	219 (57.5)	226 (60)	44 (59.5)	217 (57.2)	210 (58)	170 (63.5)
SAPS II (mean ±SD)	50 ± 14	52 ± 22	54 ± 18	36 ± 16	49 ± 20	51 ± 23	55 ± 17
Comorbidities (n, %)
Diabetes mellitus	100 (31)	105 (27.5)	121 (32)	21 (28.3)	120 (31.6)	102 (28.3)	83 (31)
Dyslipidemia	45 (14)	42 (11)	44 (11.8)	11 (14.8)	46 (12)	44 (12.2)	30 (11.2)
Hypertension	108 (33.5)	133 (34.8)	142 (37.5)	24 (32.4)	136 (35.8)	126 (35)	97 (36.3)
COPD	24 (7.4)	25 (6.5)	18 (4.8)	5 (6.7)	26 (6.8)	25 (7)	11 (4.2)
Malignancy	22 (6.8)	27 (7)	24 (6.4)	5 (6.7)	24 (6.4)	24 (6.6)	20 (7.4)
Liver disease	13 (4)	10 (3)	21 (5.5)	3 (4)	15 (3.9)	12 (3.3)	14 (5.2)
Sepsis sites (n, %)
Pulmonary	102 (31.7)	131(34.3)	129 (34.3)	22 (30)	132 (34.8)	110 (30.5)	98 (36.7)
Abdominal	52 (16.1)	69 (18)	63 (16.7)	12 (16.2)	60 (15.8)	63 (17.5)	49 (18.3)
Catheter related	45 (14)	56 (14.6)	53 (14)	8 (10.8)	56 (14.7)	53 (14.7)	37 (13.8)
Endocarditis	14 (4.3)	2 (0.6)	2 (0.5)	0 (0)	4(1)	1 2 (3.3)	2 (0.7)
Urinary	53 (16.5)	68 (17.8)	64 (17)	15 (20.3)	64(16.8)	65 (18)	41 (15.3)
Unidentifiable	18 (5.6)	14 (3.6)	16 (4.2)	6 (8.1)	19 (5)	15 (4.1)	8 (3)
More than tow sites	38 (11.8)	42 (11)	49 (13)	11 (14.8)	44 (11)	42 (11.6)	32 (12)
ICU mortality (n, %)	90 (28)	145 (38)	176 (46)	12 (16)	116 (30)	158 (40)	125 (46)

SAPS: simplified acute physiology score, COPD: Chronic obstructive pulmonary disease, ICU: Intensive care unit.

The most common comorbid conditions were diabetes and hypertension. The most frequent infectious site was the respiratory site, followed by the urinary and abdominal tract sites. ICU mortality was 38%.

Performance of Sepsis-3 definitions

When the Sepsis-2 definitions were used, there was a difference in mortality only among septic shock and sepsis patients (46 and 28%, respectively). However, there is no difference in mortality among sepsis and septic shock categories. Sepsis-3 criteria clearly categorized septic patients according to a spectrum of severity, since mortality increased from 16% in no-dysfunction-infected patients to 30, and 44% for qSOFA ≥ 2 and sepsis patients categories, respectively (Figure 2).

Figure 2

Mortality according to sepsis definition.

Reclassification of the patients

When the new Sepsis-3 definitions were used, 71 % of the patients defined by the Sepsis-2 consensus as patients with septic shock were also classified by the Sepsis-3 definitions as septic shock patients (Table 2). The reclassification of cases by the Sepsis-2 definitions occurred among 390 out of 1080 cases (36%). Using the Sepsis-3 definitions, this occurred among 74 cases (7%) (p =0.001)

Table 2

Reclassification of patients using the Sepsis-3 definitions.

		Sepsis-2 consensus
		Sepsis	Severe sepsis	Septic shock	Total
Sepsis-3 consensus	No-dysfonction	72	2	0	74
	qSOFA ≥ 2	206	123	50	379
	Sepsis	44	257	59	360
	Septic shock	0	0	267	267
	Total	322	382	376	1080

Performance of the qSOFA

The sensitivity and specificity of qSOFA ≥ 2 to predict ICU mortality were 53.5% and 76.2%, respectively. The qSOFA performed better than SIRS in predicting in-hospital mortality, with an area under the receiver operating curve (AUROC) of 0.65 (95% CI, 0.60–0.68) vs 0.48 (95% CI, 0.45–0.52) for SIRS (P < 0.001; Improvement AUROC, 0.17; 95% CI, 0.08–0.24) (Figure 3).

Figure 3

Receiving Operator Characteristic (ROC) of qSOFA ≥ 2 and SIRS for ICU mortality.

Discussion

When the sepsis 2 definitions were used in our study, there was a difference in mortality only among septic shock and sepsis patients. On the other hand, sepsis 3 definitions show that mortality increased from 16% among no-dysfunction infected patients to 30% among patients with qSOFA ≥ 2 and 44% or 46% for sepsis or septic shock patients, respectively. Therefore, sepsis-3 was better than sepsis 2 at stratifying mortality among septic patients admitted to an ICU in a developing country. The validity of sepsis definitions based on SIRS criteria has recently been extensively questioned. This is because 90% of the patients admitted to an ICU meet the SIRS criteria.5,6

On the other hand, some patients who suffer from an infectious disease and new organs’ failure do not meet 2 SIRS criteria and therefore do not achieve previous sepsis definitions.7 More importantly, the categories of sepsis, severe sepsis, and septic shock indicate an actual spectrum of severity why representing different outcome categories.8 Thus, the present study emphasizes a lack of accuracy of Sepsis-2 definitions of sepsis, severe sepsis, and septic shock in stratifying patients’ mortality risk. Additionally, our data demonstrate that sepsis-3 definitions of infection without organ dysfunction, qSOFA ≥ 2, sepsis, and septic shock clearly represent a progressive stratification of mortality risk.

The two studies carried out in the emergency department and intensive care units will be reported here. An International prospective cohort study was conducted in France, Spain, Belgium, and Switzerland between May and June 2016.9 For 4 weeks in the 30 participating emergency departments, consecutive patients who visited the emergency departments with suspected infection were included. The aim of the study was to prospectively validate qSOFA as a mortality predictor and compare the performances of the new sepsis criteria to the previous ones. Out of the 1088 patients screened, 879 were included in the analysis. Overall in-hospital mortality was 8%: 3% for patients with a qSOFA score lower than 2 vs. 24% for those with qSOFA score of 2 or higher (absolute difference, 21%; 95%CI, 15%–26%). The qSOFA performed better than both SIRS and severe sepsis in predicting in-hospital mortality, with an area under the receiver operating curve (AUROC) of 0.80 (95%CI, 0.74–0.85) vs 0.65 (95%CI, 0.59–0.70) for both SIRS and severe sepsis (P < .001; incremental AUROC, 0.15; 95%CI, 0.09–0.22). The hazard ratio of qSOFA score for death was 6.2 (95%CI, 3.8–10.3) vs. 3.5 (95%CI, 2.2–5.5) for severe sepsis. It was concluded that among the patients admitted to the emergency department with suspected infection, the use of qSOFA resulted in greater prognostic accuracy for in-hospital mortality than did either SIRS or severe sepsis. These findings support the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria in the emergency department setting.

Comparatively, the validity of these new definitions has also been tested among patients admitted to intensive care. A retrospective cohort analysis of 184 875 patients with an infection-related primary admission diagnosis in 182 Australian and New Zealand intensive care units (ICUs) from 2000 through 2015 was carried out. The main aim was to assess the discriminatory capacities of an increase in SOFA score by 2 or more points, 2 or more SIRS criteria, or a qSOFA score of 2 or more points for outcomes among patients with suspected infection. The primary outcome was in-hospital mortality. SOFA score ≥2 was much more discriminating (AUROC 0,753) than the SIRS criteria (AUROC 0.589) and the qSOFA score (AUROC 0.607) for the criterion of the primary outcome (p <0.001). The conclusion was that among adults with suspected infection admitted to an ICU, an increase in SOFA score of 2 or more had greater prognostic accuracy for in-hospital mortality than SIRS criteria or the qSOFA score. These findings suggest that SIRS criteria and qSOFA may have limited utility for predicting mortality in an ICU setting.10

The data from these studies, including ours, can encourage the adoption of these new prognostic tools. Nevertheless, several questions remain unanswered regarding the news definitions:11–14 (a) The inclusion of patients from the United States and Europe and not from economically developing countries; (b) The focus on adult patients without including pediatric patients; (c) The changes in stages of sepsis, making comparisons with prior research difficult; (d) The term “organ dysfunction” used in the new definition is still unclear, since organs may have more than one function. Organ dysfunction may emerge for multiple reasons other than sepsis, making it difficult to distinguish between sepsis-related organ dysfunction. In the same way, when the infection is not certain, and organ dysfunction is there, it is difficult to exclude a sepsis diagnosis; and (e)There was no clear emphasis on the added benefit of lactate for patients with SOFA scores <2, possibly because of lactate values lack in the derivation (about 60%) and validation cohorts (about 90%). Interestingly, lactate was not retained in the novel qSOFA during model construction. However, the authors state that for a qSOFA score of 1, high lactate values characterized a population with a similar risk to patients with a qSOFA score of 2.

Based on our study, misclassification of cases by the Sepsis-2 definitions occurred among 390 out of 1080 cases (36%). Using the Sepsis-3 definitions, this occurred among 74 cases (7%) (p =0.001). In the study published in 2017 by Giamarellos-Bourboulis E et al, misclassification of severe cases by the 1991 definition occurred in 734 out of 2172 severe cases (33.8%). Using the Sepsis-3 definitions, this occurred among 128 out of 2172 severe cases (5.9%) (p <0.0001 between the 1991 and Sepsis-3 definitions).15 The introduction of Sepsis-3 definitions limited the misclassification of severe cases.

Our study has not been designed to evaluate Sepsis-2 or Sepsis-3 as a screening tool, as only patients admitted to ICU were included. Nevertheless, the role of sepsis-2 criteria in the identification of septic patients is undeniable. However, its role in the stratification of severity can be questioned, as demonstrated by our data. As a result, our study clearly shows a better discriminant performance of Sepsis-3 in predicting mortality in ICU.

The main strengths of our study were its large sample size and the detailed health information about the enrolled participants. As far as we know, this is the first study dealing with the validation of sepsis 3 definitions among patients from a middle-income country. Despite these strengths, our findings must be interpreted in light of several limitations. First, the retrospective character of this study makes it difficult to elucidate known confounders that could have biased the outcome measures. Second, this was a single-center study, and our results may not be generalizable to other centers since there is a wide variation in outcomes when comparing different settings. Third, the validity of Sepsis-3 criteria in this study was assessed based on ICU mortality. Although ICU mortality is not the most appropriate endpoint to be evaluated, it reflects the care provided to septic patients before and after their admission to ICU. Fourth, we did not follow up discharged patients and only focused on in-hospital mortality.

Conclusions

Two important findings from our study should be emphasized: (a) Sepsis-3 definitions were better than sepsis-2 definitions at stratifying mortality, and (b) The rate of misclassification of severe patients is lower using the Sepsis-3 definitions compared with sepsis-2 definitions.