Predictive Accuracy of the Quick Sepsis-related Organ Failure Assessment Score in Brazil. A Prospective Multicenter Study.

Rationale: Although proposed as a clinical prompt to sepsis based on predictive validity for mortality, the Quick Sepsis-related Organ Failure Assessment (qSOFA) score is often used as a screening tool, which requires high sensitivity.
Objectives: To assess the predictive accuracy of qSOFA for mortality in Brazil, focusing on sensitivity.
Methods: We prospectively collected data from two cohorts of emergency department and ward patients. Cohort 1 included patients with suspected infection but without organ dysfunction or sepsis (22 hospitals: 3 public and 19 private). Cohort 2 included patients with sepsis (54 hospitals: 24 public and 28 private). The primary outcome was in-hospital mortality. The predictive accuracy of qSOFA was examined considering only the worst values before the suspicion of infection or sepsis.Measurements and Main
Results: Cohort 1 contained 5,460 patients (mortality rate, 14.0%; 95% confidence interval [CI], 13.1-15.0), among whom 78.3% had a qSOFA score less than or equal to 1 (mortality rate, 8.3%; 95% CI, 7.5-9.1). The sensitivity of a qSOFA score greater than or equal to 2 for predicting mortality was 53.9% and the 95% CI was 50.3 to 57.5. The sensitivity was higher for a qSOFA greater than or equal to 1 (84.9%; 95% CI, 82.1-87.3), a qSOFA score greater than or equal to 1 or lactate greater than 2 mmol/L (91.3%; 95% CI, 89.0-93.2), and systemic inflammatory response syndrome plus organ dysfunction (68.7%; 95% CI, 65.2-71.9). Cohort 2 contained 4,711 patients, among whom 62.3% had a qSOFA score less than or equal to 1 (mortality rate, 17.3%; 95% CI, 15.9-18.7), whereas in public hospitals the mortality rate was 39.3% (95% CI, 35.5-43.3).Conclusions: A qSOFA score greater than or equal to 2 has low sensitivity for predicting death in patients with suspected infection in a developing country. Using a qSOFA score greater than or equal to 2 as a screening tool for sepsis may miss patients who ultimately die. Using a qSOFA score greater than or equal to 1 or adding lactate to a qSOFA score greater than or equal to 1 may improve sensitivity.Clinical trial registered with www.clinicaltrials.gov (NCT03158493).

At a Glance Commentary

Scientific Knowledge on the Subject

Although proposed as a clinical prompt to sepsis based on predictive validity for mortality, the Quick Sepsis-related Organ Failure Assessment (qSOFA) score is used as a screening tool for sepsis, which requires high sensitivity. Previous studies on qSOFA validation were mostly single-center, retrospective, from high-income countries, and had disparate results. Thus, the assessment of the role of qSOFA on low- and middle-income countries is lacking and the authors of the original validation specifically requested these studies.

What This Study Adds to the Field

We carried out a robust multicenter prospective study on 74 Brazilian hospitals, assessing the qSOFA score predictive performance for mortality, focusing on sensitivity, as well as other scores, in a Brazilian cohort of non-ICU patients with suspected sepsis. We showed that qSOFA has a low sensitivity to predict mortality, and it may fail to identify a significant number of high-risk patients. We also propose alternative tools to improve its sensitivity.

Sepsis is an important cause of death worldwide, including low- and middle-income countries (LMICs). Although mortality rates are decreasing in high-income countries (1–4), the burden is still high in LMICs, with mortality rates of 30% to 70% (5–8). Recently, the Sepsis-3 task force proposed a new score based on clinical parameters and designated as the Quick Sepsis-related Organ Failure Assessment (qSOFA) as a strategy to identify among patients with suspected infection those with higher risk of poor outcomes (9). In large databases from high-income countries, qSOFA had adequate prediction for mortality and for longer ICU stays among non-ICU patients (9).

The Sepsis-3 task force suggested qSOFA as a simple tool to prompt clinicians “to further investigate for organ dysfunction, to initiate or escalate therapy as appropriate, and to consider referral to critical care or increase the frequency of monitoring” (9). Although the authors clearly stated that failure to meet two or more qSOFA criteria should not delay in investigation or treatment of infection deemed necessary by the practitioners, qSOFA is often considered as a screening tool to “rule out” sepsis in many emergency departments (EDs) and wards (10–14). In LMICs, where rapid laboratory testing is often not readily available, healthcare personnel may be particularly tempted to use qSOFA as rule-out screening tool (15). This misuse of qSOFA use is questionable because recent publications have shown it has low sensitivity for predicting mortality (16–19). In LMICs, where sepsis awareness is low and mortality is high, the use of a low-sensitivity screening tool may delay diagnosis and endanger missed cases.

We hypothesized that qSOFA has a low sensitivity to predict mortality in sepsis when assessed at the moment the clinical suspicion of sepsis is made by the healthcare team. Under this hypothesis, mortality rates of patients with a qSOFA score less than or equal to 1 at the time suspicion of infection or sepsis would be high. We also hypothesized that alternative tools would have higher sensitivity. Thus, we designed this prospective study to assess the sensitivity of qSOFA assessed at the clinical suspicion of sepsis to predict mortality in ED and ward patients in Brazilian hospitals affiliated with the Instituto Latino-Americano de Sepsis (ILAS) network, compared with alternative tools.

Methods

Study Design and Setting

This is an observational, prospective study conducted in two cohorts of ED and ward patients derived from a quality improvement network of Brazilian hospitals from May 2016 to March 2017. In this initiative, the hospitals collected data to allow audit and feedback mechanisms for performance improvement. This study was specifically designed to assess the role of qSOFA. During the study period, we asked all participant institutions to collect qSOFA variables, in addition to regularly collected data. We based our report on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.

Hospitals

The ILAS network is a multihospital quality improvement initiative with the mission of improving bundle compliance and outcomes in patients with sepsis. All hospitals receive training on implementation strategies and data collection for 6-hour sepsis bundle and hospital outcomes for all admitted patients with sepsis (20). Participation is voluntary and open to all Brazilian hospitals. We invited all hospitals that were currently active in the network to participate in the present study.

Cohorts

We included patients screened for infection and sepsis outside of the ICU (i.e., in the ED or on the ward). Among these patients we studied two different subsets. In the first subset, labeled cohort 1, we included patients at institutions that collect data from all patients with suspected sepsis at initial assessment. This cohort comprised all patients with suspected sepsis in the ED or ward, including those with infection without organ dysfunction and those with diagnosed sepsis. These institutions were mainly from the private sector because the workload to include patients with infection but without organ dysfunction in the database is high and thus largely unaffordable for public hospitals. In this cohort we were able to fully calculate the predictive accuracy of qSOFA and alternative tools.

In the second subset, labeled cohort 2, we included only patients in hospitals that opted for a similar data collection strategy but which involved only patients diagnosed with sepsis. These institutions were both public and private institutions. We included this cohort because public hospitals in Brazil generally have a higher sepsis mortality rate than public hospitals and, therefore, are essential for understanding the role of sepsis screening tools (20), However, because all patients in this cohort had sepsis, we could not calculate the predictive accuracy of qSOFA. Instead, in this cohort, we performed a descriptive analysis of the percentage of patients who would have a qSOFA score less than or equal to 1 at the moment the sepsis is suspected and their respective mortality rates.

Patients and Variables

Consecutive ED and ward patients over 16 years old were included. Because previous data suggested that the performance of qSOFA is better in non-ICU patients (9), we only included patients who had infection or sepsis initially suspected outside the ICU; that is, in the ED or hospital wards, regardless if they were later transferred to the ICU. Patients were identified based on the presence of either two of the systemic inflammatory response syndrome (SIRS) criteria or any single clinical organ dysfunction, according to local discretion. All institutions used similar screening strategies.

In cohort 1, we defined patients with infection without organ dysfunction as those who received antibiotics after blood cultures collection based on the presence of a suspected source of infection, according to the assessment of the attending physician, in the absence of organ dysfunction. In both cohorts we defined sepsis as any life-threatening organ dysfunction (see the online supplement for criteria) secondary to a suspected source of infection. Because this study is part of an ongoing quality improvement initiative run by ILAS since 2004, we used a pragmatic definition of sepsis derived from the Surviving Sepsis Campaign criteria, similar to the definition used by the quality improvement program from Center of Medicare and Medicaid in United States and aligned with the broad definition of the Sepsis-3 task force (20). The presence of SIRS criteria was not a requirement of this definition. In both cohorts, we excluded patients under end-of-life care and those previously included in the database during the same hospital admission.

The Research and Ethics Committee of the Universidade Federal de São Paulo approved the study on behalf of the entire network, under number 00.691.812.3.0000.5505. Informed consent was waived because of the observational nature of the study and absence of direct patient contact.

Data Collection

The case manager of each institution prospectively entered all data into the study database. The managers were instructed to register the worst Glasgow coma score, the highest respiratory rate, and the lowest systolic blood pressure for each patient at the moment of the sepsis suspicion. The details for data collection are available in the online supplement. All patients were followed until hospital discharge. Hospitals were characterized by their economic profile (i.e., public or private), teaching status, and geographic region.

Prognostic Tools and Outcomes

Our primary prognostic tool was qSOFA, which was classified as greater than or equal to 2 or less than or equal to 1 point. We also considered the following alternative prognostic tools: modified qSOFA, in which a positive score was 1 or more (i.e., qSOFA ≥ 1), a qSOFA score greater than or equal to 1 or lactate greater than 2 mmol/L, number of organ dysfunctions, SIRS criteria greater than or equal to 2, SIRS greater than or equal to 2 plus 1 or more organ dysfunctions, and Sequential Organ Failure Assessment (SOFA) score.

The primary outcome was hospital mortality. Secondary outcomes were admission to the ICU within 24 hours after sepsis diagnosis and a composite outcome of ICU admission within 24 hours plus ICU length of stay greater than 48 hours.

Statistical Analysis

We used percentage to describe categorical variables, and median and interquartile range to describe continuous variables. For comparisons of survivors and nonsurvivors, we compared continuous variables with a normal distribution via Student’s t test and those with a nonnormal distribution via the Mann-Whitney test. Categorical variables were compared with Pearson’s chi-square test.

First, we analyzed cohort 1, comprising ED and ward patients with suspected infection largely in private hospitals. We described the percentage of patients with a qSOFA score score greater than or equal to 2 and the mortality rates, both in patients with a qSOFA score greater than or equal to 2 and less than or equal to 1. We constructed receiver operating characteristic curves and calculated the corresponding area under the receiver operating characteristic curve (AUROC) with the 95% confidence interval (CI) to assess the performance of qSOFA and the alternative prognostic tools (qSOFA ≥ 1 or lactate > 2 mmol/L, number of organ dysfunctions, SIRS criteria, SIRS ≥ 2 plus one or more organ dysfunctions, and SOFA score) to predict hospital mortality and the other secondary outcomes. We compared the AUROC values of the different tools using the DeLong method (21). We also calculated sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and corresponding 95% CI for qSOFA and the alternative prognostic tools described above.

Second, we analyzed cohort 2, comprising patients with sepsis in both public and private hospitals. This cohort represents the distribution of qSOFA in a sample of patients more likely representing the universe of patients with sepsis cared for in Brazil. Here we performed only a descriptive analysis of the percentage of patients with each of the scores and their respective mortality rates. We also performed a subgroup analysis only on patients admitted to public hospitals because these hospitals represent the greatest burden of sepsis in Brazil.

A two-tailed P value less than 0.05 was considered statistically significant. Analyses were performed using R software (R Core Team, 2017). Missing prognostic variables were not imputed and cases with missing data were not considered in the analyses for these variables. As a sensitivity analysis, we estimated the sensitivity and specificity of qSOFA in cohort 1 assuming the extreme scenarios in which all missing values of qSOFA were imputed as either greater than or equal to 2 or less than or equal to 1.

Results

A total of 74 of 84 hospitals in the ILAS network participated in the study. The general characteristics of these institutions are available at Table E1. In cohort 1, 22 institutions (3 public and 19 private) enrolled 5,583 ED and ward patients with suspected infection (Figure 1). Data for qSOFA was missing for 123 patients (2.2%); therefore, 5,460 were included in the analyses. In cohort 2, 52 institutions (24 public and 28 private) collected data on 5,284 ED and ward patients with sepsis (see Figure 1). Data for qSOFA was missing in 573 patients (10.8%); therefore, 4,711 patients were analyzed. In both cohorts, compared with patients not missing qSOFA, patients missing qSOFA used mechanical ventilation more frequently and had higher mortality rates (see Table E2).

Figure 1.

Study flow chart. In cohort 1, we included patients presenting outside of the ICU with infection but without organ dysfunction or sepsis. In cohort 2, we included only patients presenting outside the ICU with sepsis. ED = emergency department; ILAS = Instituto Latino-Americano de Sepsis; qSOFA = Quick Sepsis-related Organ Failure Assessment.

Cohort 1: Patients Presenting outside of the ICU with Suspected Infection but without Organ Dysfunction or Sepsis on Enrollment

Among the 5,460 patients in cohort 1, 4,355 (79.8%) were from the ED. Almost all the patients (5,225; 95.7%) were from private institutions. The overall hospital mortality rate was 14.0% (766/5,460; 95% CI, 13.1–15.0). The overall mortality rate for patients with sepsis and septic shock was 23.1% (689/2,983; 95% CI, 21.6–24.7). The main characteristics of the cohort are available in Table 1.

Table 1.

Main Characteristics of Patients in Cohort 1 (Patients Presenting outside of the ICU with Suspected Infection but without Organ Dysfunction or Sepsis), Both for All Patients and for Survivors Compared with Nonsurvivors

Variable	All Patients (n = 5,460)	Survivors (n = 4,694)	Nonsurvivors (n = 766)	P Value
Type of institution				<0.0001
Public	235 (4.3)	154 (3.3)	81 (10.6)
Private	5,225 (95.7)	4,540 (96.7)	685 (89.4)
Age, yr	64 (41–80)	61 (38–78)	77 (64–85)	<0.0001
Sex, M	2,519 (46.1)	2,137 (45.5)	382 (49.9)	0.03
Comorbidities
Cancer	711 (13.0)	543 (11.6)	168 (21.9)	<0.0001
Diabetes	1,368 (25.1)	1,129 (24.1)	239 (31.2)	<0.0001
Chronic heart failure	557 (10.2)	435 (9.3)	122 (15.9)	<0.0001
COPD	456 (8.4)	375 (8.0)	81 (10.6)	0.02
Chronic renal failure	464 (8.5)	342 (7.3)	122 (15.9)	<0.0001
Arterial hypertension	2,522 (46.2)	2,079 (44.3)	443 (46.2)	<0.0001
Immunosuppression	860 (15.8)	730 (15.6)	130 (17.0)	0.32
SAPS 3 score, points	54 (43–64)	51 (40–60)	67 (56–80)	<0.0001
Source of infection				<0.0001
Lung	2,218 (40.6)	1,775 (37.8)	443 (57.8)
UTI	1,234 (22.6)	1,128 (24.0)	106 (13.8)
Abdominal	750 (13.7)	662 (14.1)	88 (11.5)
Others	1,258 (23.0)	1,129 (24.1)	129 (16.8)
Type of infection				<0.0001
Community acquired	4,181 (76.6)	3,638 (77.5)	465 (60.7)
Health care–associated*	1,279 (23.4)	1,056 (22.5)	301 (39.3)
Severity of illness				<0.0001
Infection without organ dysfunction	2,477 (45.4)	2,400 (51.1)	77 (12.1)
Sepsis	2,427 (44.5)	2,023 (43.1)	404 (52.7)
Septic shock	556 (10.2)	271 (5.8)	285 (37.2)
Location at sepsis presentation				<0.0001
ED	4,355 (79.8)	3,820 (81.4)	535 (69.8)
Wards	1,105 (20.2)	874 (18.6)	231 (30.2)
ICU admission in 24 h†
From ED	1,789 (41.1)	1,407 (36.8)	382 (71.4)	<0.0001
From wards	362 (32.8)	215 (24.6)	147 (63.6)	<0.0001
Time to sepsis diagnosis, h	0.1 (0.0–0.4)	0.4 (0.1–0.8)	0.5 (0.2–1.4)	0.004
Mechanical ventilation	589 (10.8)	227 (4.8)	362 (47.3)	<0.0001
ICU length of stay, d	4.4 (2.1–9.3)	3.9 (2.0–7.3)	8.1 (2.7–19.4)	<0.0001
Hospital length of stay‡, d	5.9 (2.0–12.0)	5.5 (1.8–10.8)	9.7 (3.1–22.3)	<0.0001

Definition of abbreviations: COPD = chronic obstructive pulmonary disease; ED = emergency department; SAPS = Simplified Acute Physiologic Score; UTI = urinary tract infection.

Data are expressed as n (%) or median (interquartile range). Percentages are column percentages when given for the entire population and row percentages when given for the population categorized by hospital outcomes.

Healthcare-associated infections include those infections acquired by out-clinic, hospice, and home care patients, as well as those not present at hospital admission and started after 48 hours of hospital stay.

Percentages for survivors and nonsurvivors calculated for the total number of ED patients (n = 4,355) or ward patients (n = 1,105).

Hospital length of stay calculated from the diagnosis of sepsis until hospital discharge.

Table 2 shows the results of the prognostic tools for all ED and ward patients according to hospital outcomes. The majority of the patients (78.3%) had a qSOFA less than or equal to 1 (see Figure E1) and had different mortality rates according to the number of qSOFA components (Figure 2A and see Table 2). The mortality rate of patients with a qSOFA score less than or equal to 1 was 8.3% (353/4,276; 95% CI, 7.5–9.1).

Table 2.

Screening Tools among Patients with Suspected Infection in Cohort 1, Both for All Patients and for Survivors Compared with Nonsurvivors

Variable	All Patients (n = 5,460)	Survivors (n = 4,694)	Nonsurvivors (n = 766)	P Value
qSOFA ≥ 2				<0.0001
No	4,276 (78.3)	3,923 (91.7)	353 (8.3)
Yes	1,184 (21.7)	771 (65.1)	413 (34.9)
qSOFA criteria
Respiratory rate ≥ 22				<0.0001
No	3,693 (67.6)	3,347 (90.6)	346 (9.4)
Yes	1,767 (32.4)	1,347 (76.2)	420 (23.8)
Glasgow				<0.0001
No	4,398 (80.5)	4,042 (91.9)	356 (8.1)
Yes	1,062 (19.5)	652 (61.4)	410 (38.6)
SBP ≤ 100				<0.0001
No	3,785 (69.3)	3,516 (90.6)	412 (9.4)
Yes	1,675 (30.7)	1,340 (75.5)	493 (24.5)
qSOFA				<0.0001
0	2,452 (44.9)	2,336 (95.3)	116 (4.7)
1	1,824 (33.4)	1,587 (87.0)	237 (13.0)
2	872 (16.0)	636 (72.9)	236 (27.1)
3	312 (5.7)	135 (43.3)	177 (56.7)
Organ dysfunctions	1 (0–2)	0 (0–1)	2 (1–3)	<0.0001
Number of organ dysfunctions				<0.0001
0	2,477 (45.4)	2,400 (96.9)	77 (3.1)
1	1,617 (29.6)	1,413 (87.4)	204 (12.6)
2	836 (15.3)	617 (73.8)	219 (26.2)
3	392 (7.2)	213 (54.3)	179 (45.7)
4 or more	138 (2.5)	51 (37.0)	87 (63.0)
Lactate > 2 mmol/L	416 (8.1)	251 (5.7)	165 (23.0)	<0.0001
qSOFA ≥ 1 or lactate > 2 mmol/L*				<0.0001
No	1,707 (32.1)	1,641 (95.1)	66 (3.8)
Yes	3,612 (67.9)	2,918 (80.8)	694 (19.2)
SIRS ≥ 2 + organ dysfunction ≥ 1				<0.0001
No	3,244 (59.4)	3,004 (92.6)	240 (7.4)
Yes	2,216 (40.6)	1,690 (76.2)	526 (23.7)
SOFA score, points	1 (0–4)	1 (0–3)	6 (3–10)	<0.0001
SOFA*				<0.0001
0	1,689 (32.5)	1,653 (97.9)	36 (2.1)
1	911 (17.6)	872 (95.7)	39 (4.3)
2	749 (14.4)	672 (89.7)	77 (10.3)
3	510 (9.8)	431 (84.5)	79 (15.5)
4	374 (7.2)	298 (79.7)	76 (20.3)
5 or more	956 (18.4)	508 (53.1)	448 (46.9)
SOFA ≥ 2* points				<0.0001
No	2,600 (50.1)	2,525 (97.1)	75 (2.9)
Yes	2,589 (49.9)	1,909 (73.7)	680 (26.3)
SIRS criteria				0.001
0	94 (1.7)	77 (81.9)	17 (18.1)
1	1,248 (22.9)	1,078 (86.4)	170 (13.6)
2	2,457 (45.0)	2,160 (87.9)	297 (12.1)
3	1,393 (25.5)	1,181 (84.8)	212 (15.2)
4	268 (4.9)	198 (73.9)	70 (26.1)
SIRS ≥ 2				0.366
No	1,342 (24.6)	1,155 (86.1)	187 (13.9)
Yes	4,118 (75.4)	3,539 (85.9)	579 (14.0)

Definition of abbreviations: qSOFA = Quick Sepsis-related Organ Failure Assessment; SBP = systolic blood pressure; SIRS = systemic inflammatory response syndrome; SOFA = Sequential Organ Failure Assessment.

Data are expressed as n (%) or median (interquartile range). Percentages are column percentages when given for the entire population and row percentages when given for the population categorized by hospital outcomes.

Lactate is available for 5,319 patients and SOFA score for 5,189. Data are expressed as n (%).

Figure 2.

In-hospital mortality rates according to the qSOFA score. (A) Cohort 1: patients presenting outside of the ICU with infection but without organ dysfunction or sepsis. The number of patients according to the type of hospital is not reported because there were only 241 patients from public hospitals. (B) Cohort 2: patients presenting outside the ICU with sepsis, both in all hospitals and categorized by public and private hospitals. qSOFA = Quick Sepsis-related Organ Failure Assessment.

The predictive accuracy of the various tools is shown in Table 3. A qSOFA score greater than or equal to 2 had the lowest sensitivity among all analyzed tools (53.9%; 95% CI, 50.3–57.5), although with a reasonable specificity (83.6%; 95% CI, 82.5–84.6). The sensitivity and specificity calculated with missing qSOFA values imputed as either all greater than or equal to 2 or all less than or equal to 1 were similar to those calculated considering cases with available qSOFA (see Table E3). The sensitivity improved to 84.9% (95% CI, 82.1–87.3) with the modified qSOFA score (qSOFA ≥ 1). The use of a qSOFA score greater than or equal to 1 or lactate greater than 2 mmol/L also improved sensitivity (91.3%; 95% CI, 89.0–93.2). When the presence of any organ dysfunction was used as the cutoff, the sensitivity was 89.9% (95% CI, 87.5–91.9). The sensitivity for SIRS plus at least one organ dysfunction was 68.7% (95% CI, 65.2–71.9) and for a SOFA score of 2 or more points it was 88.3% (95% CI, 85.8–90.5).

Table 3.

Performance for the Prediction of Hospital Mortality among Patients with Suspected Infection in Cohort 1

Variable	qSOFA ≥ 2	qSOFA ≥ 1	qSOFA ≥ 1 or Lactate > 2 mmol/L	SIRS Criteria ≥ 2	Organ Dysfunction ≥ 1	SIRS ≥ 2 + Organ Dysfunction ≥ 1	SOFA ≥ 2
N	5,460	5,460	5,319	5,460	5,460	5,460	5,189
Sensitivity, %	53.9 (50.3–57.5)	84.9 (82.1–87.3)	91.3 (89.0–93.2)	75.6 (72.4–78.6)	89.9 (87.5–91.9)	68.7 (65.2–71.9)	88.3 (85.8–90.5)
Specificity, %	83.6 (82.5–84.6)	49.8 (48.3–51.2)	36.0 (34.6–37.4)	24.6 (23.4–25.9)	51.1 (49.7–52.6)	64.0 (62.6–65.3)	59.5 (58.0–60.9)
PPV, %	34.9 (32.2–37.7)	21.6 (20.2–23.1)	19.2 (17.9–20.5)	14.1 (13.0–15.2)	23.1 (21.6–24.7)	23.7 (22.0–25.6)	27.1 (25.3–28.9)
NPV, %	91.7 (90.9–92.5)	95.3 (94.3–96.1)	96.1 (95.1–97.0)	86.1 (84.1–87.9)	96.9 (96.1–97.5)	92.6 (91.6–93.5)	96.8 (96.0–97.4)
Positive LR	3.28 (2.99–3.60)	1.69 (1.62–1.76)	1.43 (1.38–1.47)	1.00 (0.96–1.05)	1.84 (1.77–1.91)	1.91 (1.79–2.03)	2.18 (2.09–2.28)
Negative LR	0.55 (0.51–0.60)	0.30 (0.26–0.36)	0.24 (0.19–0.31)	0.99 (0.87–1.13)	0.20 (0.16–0.24)	0.49 (0.44–0.55)	0.20 (0.16–0.24)
AUROC, %	75.0 (76.9–73.2)	75.0 (76.9–73.2)	82.4 (83.9–80.8)	53.4 (55.6–51.2)	79.2 (80.8–77.5)	79.3 (81.0–77.5)	83.3 (84.9–81.7)

Definition of abbreviations: AUROC = area under the receiver operating characteristic curve; LR = likelihood ratio; NPV = negative predictive value; PPV = positive predictive value; qSOFA = Quick Sepsis-related Organ Failure Assessment; SIRS = systemic inflammatory response syndrome; SOFA = Sequential Organ Failure Assessment.

Data are expressed as estimate (95% confidence interval).

As a severity score, qSOFA performed well, with adequate mortality prediction (AUROC, 75.0; 95% CI, 73.2–76.9). However, all other scores had a better performance than qSOFA to predict mortality: AUROC for number of organ dysfunctions, 79.2 (95% CI, 77.5–80.8); SIRS plus at least one organ dysfunction, 79.3 (95% CI, 77.5–81.0); and SOFA score, 83.3 (95% CI, 81.7–84.9; P < 0.001 between qSOFA and each of the other tools) (Figure 3).

Figure 3.

Discrimination of the different tools for prediction of in-hospital mortality for cohort 1 (patients presenting outside of the ICU with infection but without organ dysfunction or sepsis). P < 0.001 between qSOFA and the other tools (DeLong method) (21). qSOFA = Quick Sepsis-related Organ Failure Assessment; ROC = receiver operating characteristic; SIRS = systemic inflammatory response syndrome; SOFA = Sequential Organ Failure Assessment.

The qSOFA score predicted ICU admission within 24 hours well (AUROC, 73.3; 95% CI, 72.0–74.5; P < 0.001; see Table E4); however, the sensitivity was low at 39.7% (95% CI, 37.7–41.9). The use of a single positive component of qSOFA improved the sensitivity to 78.3% (95% CI, 76.5–80.0). We obtained similar results for prediction of the composite endpoint of early ICU admission and an ICU length of stay longer than 48 hours (see Table E4).

Cohort 2: Patients Presenting outside the ICU with Sepsis

Among the 4,711 patients in cohort 2, 3,509 (74.5%) were from the ED. Patients from public institutions constituted 30.8% (1,451/4,711 patients) of the sample. The overall hospital mortality rate was 28.4% (1,338/4,711; 95% CI, 27.1–29.7). The overall mortality rate for patients from public institutions was 50.3% (730/1,451; 95% CI, 47.7–52.9). The main characteristics of the patients are available in Table E5.

The majority of the patients had a qSOFA score less than or equal to 1 (2,934/4,711; 62.3%; see Figure E2). The mortality rates according to qSOFA score in public and private institutions are available in Figure 2B. The mortality rate for all patients with a qSOFA score less than or equal to 1 was 17.3% (507/2,934; 95% CI, 15.9–18.7). When only public institutions were considered, the mortality rate for those with a qSOFA score less than or equal to 1 was 39.3% (243/618; 95% CI, 35.5–43.3). The full descriptive results of the prognostic tools for all patients in cohort 2 according to hospital outcomes are available in Table E6.

Discussion

This large prospective observational study demonstrated that qSOFA assessed at the moment of sepsis suspicion has a low sensitivity to predict mortality among ED and ward patients with suspected infection or sepsis in a middle-income country. Because this score lacks adequate sensitivity, its use as the only instrument in screening strategies might result in a significant number of missed patients who are severely ill and have a high mortality rate. Alternative strategies for improving sensitivity in our scenario include the use of any organ dysfunction as a screening tool, the use of only one of the components of qSOFA, the use of only one component of qSOFA or elevated lactate, or using SIRS associated with the presence of any organ dysfunction.

In the original Sepsis-3 paper, the authors aimed to evaluate the validity of clinical criteria to identify patients with suspected infection who were at risk of sepsis (9). They concluded that a qSOFA score greater than or equal to 2 could be used as a prompt to consider possible sepsis, suggesting that such a patient should be “ruled in” in a process to further investigate for organ dysfunction. Their original intention was not to rule out sepsis. However, one of the consequences of using predictive validity for hospital mortality is an excessive weight on specificity at the expense of sensitivity. Because the bedside physician may have difficulties understanding the meaning of mortality prediction, the intended use of qSOFA might be misinterpreted.

The low sensitivity of qSOFA to predict mortality observed in our study is consistent with previous results, most of them retrospective and from developed countries (16–18). Our findings of a low sensitivity of qSOFA to predict mortality in Cohort 1 suggest that even in institutions involved in quality improvement initiatives in which the mortality rates are lower, be they public or private, qSOFA has a limited role as a screening tool and caution is needed. The overall mortality rate for patients with sepsis and septic shock in this cohort (23.1%) was similar to those reported in quality improvement initiatives in the United States and other high-income countries (9, 11). Accordingly, other authors have shown similar low sensitivity for mortality in high-income countries (16) and also for the receipt of critical care interventions (22), as well as low specificity (23).

However, the institutions in cohort 1 are not representative of Brazil or other middle-income countries. In our quality improvement program, patients with sepsis in public institutions have a higher mortality rate than in private ones (20). Because public institutions are better represented in cohort 2, we believe this cohort is more representative of the landscape of Brazil hospitals. Among patients with organ dysfunction in cohort 2, most (62%) had a qSOFA score less than or equal to 1. These patients had high in-hospital mortality, especially in public hospitals, where it was unacceptably high (39%). This should alert the healthcare team that many patients are at risk of dying even having a qSOFA score less than or equal to 1. This is in consonance with the concept that qSOFA is a rule-in and not a rule-out tool, and that might be harmful to misuse it as a rule-out tool in our settings. A small number of retrospective reports from LMICs confirm the low sensitivity of qSOFA, with higher mortality rates than those reported for developed countries (24, 25); however, the largest study on qSOFA in resource-constrained settings did not report sensitivity (15).

Our results demonstrate that there are alternatives to increase sensitivity. The use of a single organ dysfunction, as suggested by the Surviving Sepsis Campaign since 2005, resulted in a higher sensitivity. In fact, the components of qSOFA are similar to the clinical dysfunction used by the Surviving Sepsis Campaign. The potential role of using a single component of qSOFA was already highlighted by Rudd and colleagues in LMICs (15). The authors demonstrated that a qSOFA score of 1 was associated with an increased risk of death. In our study, the use of only one component of qSOFA increased sensitivity, although at the cost of losing specificity. Another alternative for increasing sensitivity is to measure lactate levels. Although the laboratory tests to detect organ dysfunctions are not always available in LMICs, using lactate levels, if available, improves sensitivity (26, 27). In settings with high mortality rates, the use of sensitive tools may be a key step for enhancing early detection. On the contrary, the lack of specificity might compromise the prompt assessment of more severely ill patients in very busy ED and increase the burden for the healthcare team in the wards. The specificity of qSOFA and the simplicity of this tool make it a good option to rapidly identify those patients at higher risk of death and thus those who should be prioritized. The ideal balance between sensitivity and specificity in screening strategies for sepsis is currently unknown and probably should be adjusted for each setting. Using combining tools is also a possible option.

Strengths and Limitations

A major strength of this study is the multicenter prospective design in a large cohort from 74 hospitals in a developing country assessing patients both from wards and ED in contrast to validation studies that assessed patients only from the ICU (28) or ED (29). We only considered the variables before the diagnosis of sepsis, which enabled us to assess the sensitivity to predict mortality at this specific time point. Additionally, the suspicion of infection in our study was pragmatically based on clinical evaluation and chart documentation of infection, followed by antibiotic administration.

On the other hand, our study has several limitations. First, the ILAS network is a quality improvement initiative, and the hospitals involved may not generalize to the rest of Brazil, as indicated by the overall mortality rate of 28.4% in cohort 2. Although our population included patients with low disease severity not admitted to an ICU, a previous Brazilian study conducted on a random representative sample of ICUs showed a mortality rate of 55% (5). This finding suggests that the harmful impact of the misuse of qSOFA as a screening tool might be even greater in other hospitals. Second, although we used the broad Sepsis-3 definition, we did not use the current clinical criteria to define sepsis (i.e., variation in the SOFA score) because it is not suitable for quality improvement initiatives (30, 31). Of note, our definition of sepsis was similar to the one used by the Surviving Sepsis Campaign and by SEP-1, the quality improvement program led by the Centers for Medicare and Medicaid Services in the United States (32). Third, these data were not collected primarily for research purposes but, rather, for quality improvement initiatives. Therefore, some data for qSOFA was missing in both cohorts and we decided a priori to exclude all patients with missing data in any of the components of qSOFA. Although this might have compromised our capacity to assess the sensitivity of this tool, our estimates of sensitivity did not change substantially even after imputing qSOFA values according to extreme scenarios (all cases ≥2 or all cases ≤1). Fourth, because we did not adjudicate the diagnosis of sepsis, we cannot assure its adequacy.

Conclusions

In this large prospective multicenter study in a middle-income country, among ED and ward patients with suspected infection, qSOFA had low sensitivity for the detection of patients who would die. Thus, its misuse as screening tool for sepsis would have resulted in a high percentage of missed cases with high mortality rates. The use of alternative approaches to prompt sepsis alerts, such as modifying qSOFA, adding lactate to a qSOFA score greater than or equal to 1, or using a single organ dysfunction, may minimize this issue.