Quick Sepsis-related Organ Failure Assessment score is not sensitive enough to predict 28-day mortality in emergency department patients with sepsis: a retrospective review.

OBJECTIVE: To test the hypothesis that the quick Sepsis-related Organ Failure Assessment (qSOFA) score, derived from vital signs taken during triage and recommended by current sepsis guidelines for screening patients with infections for organ dysfunction, is not sensitive enough to predict the risk of mortality in emergency department (ED) sepsis patients.
METHODS: Patients diagnosed with severe sepsis and septic shock using the old definition between May 2014 and April 2015 were retrospectively reviewed in three urban tertiary hospital EDs. The sensitivities of systemic inflammatory response syndrome (SIRS) criteria, qSOFA, and Sequential Organ Failure Assessment (SOFA) scores ≥2 were compared using McNemar's test. Diagnostic performances were evaluated using specificity, positive predictive value, and negative predictive value.
RESULTS: Among the 928 patients diagnosed with severe sepsis or septic shock using the old definition, 231 (24.9%) died within 28 days. More than half of the sepsis patients (493/928, 53.1%) and more than one-third of the mortality cases (88/231, 38.1%) had a qSOFA score <2. The sensitivity of a qSOFA score ≥2 was 61.9%, which was significantly lower than the sensitivity of SIRS ≥2 (82.7%, P<0.001) and SOFA ≥2 (99.1%, P<0.001). The specificity, positive predictive value, and negative predictive value of a qSOFA score ≥2 for 28-day mortality were 58.1%, 32.9%, and 82.2%, respectively.
CONCLUSION: The current clinical criteria of the qSOFA are less sensitive than the SIRS assessment and SOFA to predict 28-day mortality in ED patients with sepsis.

INTRODUCTION

Sepsis is a clinical syndrome associated with significant mortality [1,2]. Previously, sepsis was diagnosed when two or more systemic inflammatory response syndrome (SIRS) criteria (caused by infection) were observed [3,4]. However, the SIRS criteria were not specific to sepsis, and sepsis patients with fewer than two SIRS criteria had a substantial mortality rate (16.1%) [5,6]. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) was updated in 2016. The new definition emphasized organ dysfunction to define sepsis (previously called severe sepsis) [7]. The clinical criteria for sepsis in intensive care unit (ICU) patients were defined as suspected infections with two or more increments in the Sequential Organ Failure Assessment (SOFA) score. For non-ICU patients, a suspected infection with a quick Sepsis-related Organ Failure Assessment (qSOFA) score of two or higher was defined as an early warning tool for sepsis. The qSOFA consists of a systolic blood pressure (BP) measurement ≤100 mmHg, a respiration rate of 22 bpm or higher, and an altered mentation. This simple model has been developed for quick use at the bedside while providing a valid diagnostic performance outside the ICU.

The emergency department (ED) is one of the main sources of sepsis admissions, and early identification of sepsis is crucial to provide the appropriate management [8]. The advantage of the qSOFA is its simplicity, which allows the qSOFA to be easily calculated during the ED triage. Recently, a prospective validation study of qSOFA performed in multicenter EDs has been published [9]. According to that study, the qSOFA had a greater prognostic accuracy for in-hospital mortality than either SIRS or severe sepsis among patients presenting to the ED with suspected infection. However, the qSOFA in that study was derived from the worst qSOFA criteria value during the ED stay, which could have biased the results to a higher qSOFA score. Therefore, the value of the qSOFA as an ED triage tool was not fully determined.

We hypothesized that a qSOFA derived from vital signs taken during triage would not be sensitive enough to predict mortality in sepsis patients. This study aimed to test this hypothesis by evaluating the predictive value of qSOFA scores derived from vital signs taken during triage for 28-day mortality in ED patients with sepsis.

METHODS

Study setting

This retrospective study was performed in three hospital EDs from May 2014 to April 2015. The three institutions were urban teaching hospitals, with annual ED censuses of over 50,000, 70,000, and 80,000 patients. Patients with a previous definition of severe sepsis or septic shock were retrospectively identified from May 2014 to July 2014 and prospectively identified from August 2014 to April 2015. Prospective data were derived from the Severe Sepsis and Septic Shock Registry (H-1408-003-599, BRMH-14-03-01, and B-1409/266-401). The institutional review board also approved this retrospective analysis (H-1610-110-801). Written informed consent was obtained from patients in the prospective sepsis registry cohort; otherwise, it was waived.

Participants and data collection

Patients who fulfilled the following criteria during their ED stays were eligible for enrollment in the retrospective and prospective registry cohorts: suspected or proven infection and at least one organ dysfunction. Organ dysfunction was defined as follows: cardiovascular, a systolic BP <90 mmHg or a mean BP <70 mmHg; respiratory, a PaO2/FiO2 ratio <250 in the absence of pneumonia or PaO2/FiO2 ratio <200 in the presence of pneumonia; renal, a urine output of <0.5 mL/kg for at least 2 hours or a serum creatinine level >2.0 mg/dL; hepatic, total bilirubin >2 mg/dL; hematologic, a platelet count <100,000/μL or a prothrombin time international normalized ratio >1.5; and hypoperfusion, a lactate level above the upper limit of laboratory normal.

SIRS criteria (fever >38.3°C or hypothermia <36°C, tachycardia over 90 beats per minute, tachypnea over 20 breaths per minute, and leukocytosis [white blood cell count over 12,000/μL] or leukopenia [white blood cell less than 4,000/μL]) were evaluated in all participants. However, SIRS criteria ≥2 were not necessary for study inclusion. The following exclusion criteria were applied: an age younger than 18 years, cardiac arrest at presentation, and inadequate data to calculate the SIRS criteria, qSOFA, and SOFA scores (all the following data were required: mental status, BP, heart rate, respiration rate, body temperature, white blood cell count, serum creatinine, total bilirubin, platelet count, and prothrombin time international normalized ratio).

The following data were abstracted from the electronic medical records system for the retrospective cohort and obtained from the Severe Sepsis and Septic Shock Registry for the prospective cohort: age, gender, underlying diseases (hypertension, diabetes, and chronic liver disease), infection site (respiratory, hepatobiliary, genitourinary, gastrointestinal, and others), initial vital signs, including the Glasgow Coma Scale (GCS) during the ED triage, and old sepsis category (severe sepsis and septic shock). Initial vital signs and white blood cell counts were used to calculate the qSOFA score and SIRS criteria. For the qSOFA calculation, a GCS score of less than 14 was considered to be positive for altered mentation. The SOFA and Acute Physiology and Chronic Health Evaluation (APACHE) II scores were calculated using the worst physiologic parameters obtained during the first 24 hours of the ED visit. The initial lactate level was collected if available. Organ supportive management (mechanical ventilation and renal replacement therapy) during the hospital stay was also recorded.

The primary outcome was defined as the 28-day mortality of any cause. All patients discharged or transferred before 28 days were contacted by telephone for a follow-up to determine the primary outcomes.

Statistical analysis

Categorical variables were described as the percent frequency of occurrence, and continuous variables were expressed as the mean (with 95% confidence intervals [CIs]). The chi-squared test was used to compare binomial variables, and the Wilcoxon rank-sum test was used to compared continuous variables, as appropriate. The SIRS criteria, qSOFA and SOFA scores of 2 or higher, and lactate level of 2 mmol/L or higher were evaluated using sensitivity, specificity, positive predictive value, negative predictive value, and area under the receiver operating characteristics (AUROC) curve. McNemar’s chi-squared test was used to compare the sensitivity between two scores. The prognostic accuracy of SIRS criteria, qSOFA scores, full SOFA scores, and initial lactate levels for 28-day mortality were evaluated using the AUROC curve. All statistical analyses were performed using Stata ver. 13.1 (StataCorp., College Station, TX, USA), and a two-tailed P-value <0.05 was considered to be statistically significant.

RESULTS

A total of 942 patients with the old definition of severe sepsis or septic shock were identified during the study period. After excluding 14 patients with inadequate data, 928 were finally analyzed.

Table 1 compares the characteristics of surviving and non-surviving patients. Patients who died within 28 days (n=231, 24.9%) had lower systolic BP, body temperature, and GCS scores and higher heart rates and respiration rates than surviving patients. Respiratory infections and septic shock were more frequent in patients who died within 28 days. Both SIRS and qSOFA scores ≥2 were associated with increased mortality. However, 493 (53.1%) severe sepsis or septic shock patients and 88 (38.1%) patients who died within 28 days had qSOFA scores <2. In contrast, only 40 (17.3%) and 2 (0.9%) among 231 patients who died within 28 days had SIRS and SOFA scores <2, respectively.

Table 1.

Patients’ characteristics

Characteristics	All patients (n = 928)	Alive at 28-day (n = 697)	Died at 28-day (n = 231)	P-value
Age (yr)	70.1 (69.3–71.0)	69.2 (68.2–70.2)	72.9 (71.3–74.5)	0.001
Sex (male)	552 (59.5)	402 (57.7)	150 (64.9)	0.051
Underlying diseases
Hypertension	389 (41.9)	291 (41.8)	98 (42.4)	0.857
Diabetes	289 (31.1)	220 (31.6)	69 (29.9)	0.630
Chronic liver disease	97 (10.5)	70 (10.0)	27 (11.7)	0.479
Initial vital signs
Systolic BP (mmHg)	100.3 (98.5–102.1)	101.1 (99.1–103.2)	97.9 (94.3–101.6)	0.045
Diastolic BP (mmHg)	58.9 (57.9–60.0)	59.4 (58.2–60.5)	57.6 (55.5–59.7)	0.085
Heart rate (min)	108.1 (106.5–109.7)	107.1 (105.3–109.0)	111.1 (107.8–114.3)	0.032
Respiration rate (min)	22.5 (22.1–23.0)	22.0 (21.5–22.5)	24.1 (23.2–25.0)	< 0.001
Body temperature (°C)	37.4 (37.3–37.5)	37.5 (37.4–37.7)	37.0 (36.9–37.2)	< 0.001
GCS	12.2 (11.9–12.4)	12.6 (12.3–12.9)	11.0 (10.5–11.5)	< 0.001
Site of infection
Respiratory	404 (43.5)	271 (38.9)	133 (57.6)	< 0.001
Hepatobiliary	186 (20.0)	152 (21.8)	34 (14.7)
Genitourinary	173 (18.6)	148 (21.2)	25 (10.8)
Gastrointestinal	67 (7.2)	52 (7.5)	15 (6.5)
Others	98 (10.6)	74 (10.6)	24 (10.4)
Old sepsis category
Severe sepsis	386 (41.6)	321 (46.1)	65 (28.1)	< 0.001
Septic shock	542 (58.4)	376 (54.0)	166 (71.9)
SIRS ≥ 2
No	225 (24.3)	185 (26.5)	40 (17.3)	0.005
Yes	703 (75.8)	512 (73.5)	191 (82.7)
qSOFA ≥ 2
No	493 (53.1)	405 (58.1)	88 (38.1)	< 0.001
Yes	435 (46.9)	292 (41.9)	143 (61.9)
SOFA ≥ 2
No	31 (3.3)	29 (4.2)	2 (0.9)	0.016
Yes	897 (96.7)	668 (95.8)	229 (99.1)
SOFA score	7.5 (7.3–7.7)	6.9 (6.6–7.1)	9.4 (8.9–9.9)	< 0.001
Cardiovascular	2.2 (2.1–2.3)	2.1 (2.0–2.2)	2.4 (2.2–2.7)	0.010
Respiratory	1.6 (1.5–1.6)	1.4 (1.3–1.5)	2.0 (1.9–2.2)	< 0.001
Kidney	0.9 (0.9–1.0)	0.9 (0.7–0.9)	1.2 (1.1–1.4)	< 0.001
Liver	0.6 (0.5–0.8)	0.6 (0.5–0.7)	0.6 (0.5–0.8)	0.447
Nervous	1.4 (1.3–1.5)	1.2 (1.1–1.3)	2.0 (1.8–2.2)	< 0.001
Coagulation	0.8 (0.7–0.9)	0.7 (0.6–0.8)	1.1 (0.9–1.2)	< 0.001
Lactate (mmol/L)[a)]	3.7 (3.4–3.9)	3.0 (2.8–3.2)	5.4 (4.8–6.0)	< 0.001
Lactate ≥ 2 mmol/L[a)]
No	239 (36.4)	195 (41.2)	44 (23.9)	< 0.001
Yes	418 (63.6)	278 (58.8)	140 (76.1)
APACHE II score	24.8 (24.2–25.5)	22.7 (22.0–23.3)	31.4 (30.0–32.8)	< 0.001
Organ supportive management
MV	252 (27.2)	153 (22.0)	99 (42.9)	< 0.001
RRT	82 (8.8)	32 (4.6)	50 (21.7)	< 0.001

Values are presented as mean (95% confidence intervals) or frequency (%). Categorical variables were compared using the chi-square test and continuous variables were compared using the Wilcoxon rank-sum test.

BP, blood pressure; GCS, Glasgow Coma Scale; SIRS, systemic inflammatory response syndrome; qSOFA, quick Sepsis-related Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment; APACHE II, Acute Physiology and Chronic Health Evaluation II; MV, mechanical ventilation; RRT, renal replacement therapy.

Only 657 cases with available lactate results were analyzed.

Fig. 1 illustrates the patient results and 28-day mortality rates for each SIRS, qSOFA, and SOFA score.

Fig. 1.

Stacked bar graph demonstrating the prevalence of 28-day mortality. (A) Stratified by the systemic inflammatory response syndrome (SIRS) criteria. (B) Stratified by the quick Sepsis-related Organ Failure Assessment (qSOFA) score. (C) Stratified by the Sequential Organ Failure Assessment (SOFA) score. The numbers on the top of the bar represent the patient numbers.

The SIRS results, qSOFA and SOFA scores ≥2, and lactate level ≥2 mmol/L used to predict the 28-day mortality are described in Table 2. The sensitivity of a qSOFA score ≥2 (61.9%) was significantly lower than the sensitivity of SIRS ≥2 (82.7%, P<0.001) and SOFA ≥2 (99.1%, P<0.001). The sensitivity of a SOFA score ≥2 was higher than that of a SIRS score ≥2 (P<0.001). However, the negative predictive value of a qSOFA score ≥2 (82.2%) was similar to that of a SIRS score ≥2 (82.2%) but lower than that of a SOFA score ≥2 (93.5%). Among patients with available initial lactate levels (n=657), the sensitivity of lactate ≥2 mmol/L (76.1%) was higher than that of a qSOFA score ≥2 (64.7%; 95% CI, 57.3% to 71.6%; P=0.006) and lower than that of a SIRS score ≥2 (85.3%; 95% CI, 79.4% to 90.1%; P<0.001). The sensitivity of SIRS ≥2 or lactate ≥2 mmol/L (95.7%) was higher than that of qSOFA ≥2 or lactate ≥2 mmol/L (90.9%, P=0.008).

Table 2.

Diagnostic performance of SIRS, qSOFA score, SOFA score, and lactate for 28-day mortality (n=928)

Scores	Sensitivity	Specificity	PPV	NPV	AUROC
SIRS ≥ 2	82.7 (77.2–87.3)	26.5 (23.3–30.0)	27.2 (23.9–30.6)	82.2 (76.6–87.0)	0.55 (0.52–0.58)
qSOFA ≥ 2	61.9 (55.3–68.2)	58.1 (54.3–61.8)	32.9 (28.5–37.5)	82.2 (78.5–85.4)	0.60 (0.56–0.64)
SOFA ≥ 2	99.1 (96.9–99.9)	4.2 (2.8–5.9)	25.5 (22.7–28.5)	93.5 (78.6–99.2)	0.52 (0.51–0.53)
Lactate ≥ 2 mmol/L[a)]	76.1 (69.3–82.1)	41.2 (36.8–45.8)	33.5 (29.0–38.2)	81.6 (76.1–86.3)	0.59 (0.55–0.63)
SIRS ≥ 2 or lactate ≥ 2 mmol/L[a)]	95.7 (91.6–98.1)	12.1 (9.3–15.3)	29.7 (26.1–33.6)	87.7 (77.2–94.5)	0.54 (0.52–0.56)
qSOFA ≥ 2 or lactate ≥ 2 mmol/L[a)]	90.9 (86.1–94.4)	20.2 (16.9–23.9)	30.7 (27.1–34.5)	85.0 (77.6–90.7)	0.56 (0.53–0.58)

Values are presented as percentages with 95% confidence intervals.

SIRS, systemic inflammatory response syndrome; qSOFA, quick Sepsis-related Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment; PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the receiver operating characteristics.

Only 657 cases with available lactate results were analyzed.

Fig. 2 demonstrates the receiver operating characteristics curves of the SIRS, qSOFA, and SOFA scores as well as the initial lactate level for predicting the 28-day mortality. The AUROC of the qSOFA (0.627) was significantly greater than that of the SIRS (0.540) (P<0.001); however, it was smaller than that of the SOFA score (0.687) (P=0.009). The same result was observed in a subgroup of the available initial lactate results (Fig. 2B). The AUROC of the qSOFA score (0.613) was significantly smaller than that of the initial lactate level (0.692) (P=0.012).

Fig. 2.

(A) The receiver operating characteristics curve used to predict 28-day mortality in all patients. (B) The receiver operating characteristics curve used to predict 28-day mortality in patients with initial lactate results. AUROC, area under the receiver operating characteristics; CI, confidence interval; SIRS, systemic inflammatory response syndrome; qSOFA, quick Sepsis-related Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment.

DISCUSSION

The most interesting finding of this study was that more than half of sepsis patients with organ dysfunction (493/928, 53.1%) and more than one-third of sepsis patients with mortality (88/231, 38.1%) had qSOFA scores <2 at ED triage. As a result, the sensitivity of a qSOFA score ≥2 for predicting the 28-day mortality was significantly lower than those of both a SIRS score ≥2 and a SOFA score ≥2. Among the patients with available initial lactate levels, the sensitivity of a qSOFA score ≥2 for predicting the 28-day mortality was significantly lower than that of an initial lactate level ≥2 mmol/L.

The qSOFA was suggested as a simple tool to identify patients at risk of sepsis outside the ICU, and the predictive validity for in-hospital mortality of qSOFA was better than SOFA and SIRS in the original study [10]. Three validation studies published in this year had variable results. One study with an 8% in-hospital mortality rate showed that the prognostic accuracy of the qSOFA was higher than that of SIRS and comparable to that of the SOFA [9]. In that study, the sensitivity of qSOFA ≥2 for predicting in-hospital mortality was 73% (95% CI, 61.4% to 82.6%), and 22 (29.7%) of 74 mortality cases had qSOFA scores <2. Another study with an 18.7% in-hospital mortality rate showed that the qSOFA score had lower prognostic accuracy than the SOFA score [11]. The sensitivity of a qSOFA score ≥2 for predicting in-hospital mortality was 65.8% (95% CI, 65.3% to 66.3%), and 11,820 (34.2%) of 34,578 mortality cases had qSOFA scores <2. The last study that reported a 5.4% in-hospital mortality rate showed that the prognostic accuracy of qSOFA was higher than that of SIRS but lower than those of commonly used early warning scores [12]. The sensitivity of a qSOFA score ≥2 for predicting in-hospital mortality was 68.7%, and fewer than one in five patients who eventually died or were transferred to the ICU had fulfilled the qSOFA ≥2 criteria by the time infection was suspected.

The qSOFA is a simple method that includes only 3 parameters that do not require laboratory tests. It is reasonable that complex scoring systems, such as the SOFA and APACHE II, outperform the qSOFA. However, calculating the SOFA and APACHE II scores would require many laboratory results and significant time, making those scoring systems not feasible in the initial ED triage. The dilemma is that qSOFA is not sensitive enough to screen for sepsis patients who will benefit from early interventions. The sensitivity of a qSOFA score ≥2 for patient mortality was only 61.9%, which was lower than the SIRS ≥2 criteria in this study. Furthermore, one-third of mortality cases did not meet the qSOFA ≥2 criteria at ED triage. As described above, these results were similar to those reported in previous studies [9,11,12]. One study used the worst vital signs during an ED stay to calculate the qSOFA score, which may have resulted in a higher sensitivity [11]. However, this process is not practical because the qSOFA score should be used as a screening tool during the ED triage.

The predictive value of qSOFA might be lower than SOFA in this study because of the relatively high mortality rate (24.9%). An original validation study has shown that qSOFA was useful in non-ICU encounters for cases in which the in-hospital mortality rate was only 3% [11]. In contrast, the predictive validity for in-hospital mortality of qSOFA was inferior to that of SOFA for ICU encounters, in which the in-hospital mortality rate was 16% [11]. Different outcomes can impact the performance of a prediction method [13]. The qSOFA is a screening tool with high specificity for organ dysfunction, and therefore, it would be more useful in a population with a low risk of death.

Although the Sepsis-3 study recommended that organ dysfunction should be re-assessed using SOFA scores if sepsis is still suspected in patients with a qSOFA score <2, physicians should be aware that qSOFA scores derived from initial vital signs are not sensitive enough to identify patients at a high risk for in-hospital mortality [10].

The sensitivity and AUROC of initial lactate levels were significantly higher than that of the qSOFA in this study. Seymour et al. have described that the addition of serum lactate level significantly improved the predictive validity of the qSOFA and had a discriminative ability for in-hospital mortality in patients with a qSOFA score of 1 [10]. In addition, combining the qSOFA score with plasma lactate concentration was suggested as a useful early warning tool in a previous study [14]. Also, the combination of SIRS and lactate or qSOFA and lactate have increased the sensitivity in this study. We think that repeated measurements of the qSOFA and serum lactate level or a full SOFA score calculation would be necessary to detect sepsis in ED patients with infection.

This study has several limitations. First, all patients had severe sepsis and septic shock. Analyzing all ED patients with suspected infections would be ideal to evaluate the usefulness of the qSOFA. Second, this study partly includes patients who were identified retrospectively. However, at least the initial vital signs and mental status used to determine the qSOFA scores were accurate because it was obligatory to input those variables electronically in the NEDIS (National Emergency Department Information System) [15]. Therefore, only 14 patients with inadequate data were excluded.

In summary, the current clinical criteria of the qSOFA are less sensitive than SIRS and SOFA for predicting 28-day mortality in ED patients with sepsis. The qSOFA ≥2 criteria missed half of the sepsis cases and one-third of the mortality cases; therefore, further assessment of organ failure using serial qSOFA, serum lactate levels, or SOFA scores would be necessary for ED patients with confirmed or suspected infection.