Comparative study on quick sequential organ failure assessment, systemic inflammatory response syndrome and the shock index in prehospital emergency patients: single-site retrospective study.

AIM: The quick sequential organ failure assessment (qSOFA) score, shock index (SI), and systemic inflammatory response syndrome (SIRS) criteria are simple indicators for the mortality of patients in the emergency department (ED). These simple indicators using only vital signs might be more useful in prehospital care than in the ED due to their quick calculation. However, these indicators have not been compared in prehospital settings. The aim of the present study is to compare these indicators measured in prehospital care and verify whether the qSOFA score is useful for prehospital triage.
METHODS: We undertook a single-site retrospective study on patients transferred by ambulance to the Kumamoto Medical Center ED (Kumamoto, Japan) between January 2015 and December 2016. We compared areas under the receiver operating characteristic (AUROC) curves of the qSOFA score, SI, and SIRS criteria measured in prehospital care. We also carried out sensitivity and specificity analyses using the Youden index.
RESULTS: A total of 4,827 patients were included in the present study. The AUROC (95% confidence interval) of the qSOFA score for in-hospital mortality was 0.64 (0.61-0.67), which was significantly higher than those of the SIRS criteria (0.59 [0.56-0.62]) and SI (0.58 [0.54-0.62]). According to the optimal cut-off values (qSOFA ≥ 2) decided on as the Youden index, the sensitivity of the qSOFA score was 52.3% and its specificity was 69.9%.
CONCLUSIONS: The qSOFA score had the highest AUROC among three indicators. However, it might not be practical in actual prehospital triage due to its low sensitivity.

Background

The annual report on the status of emergency and rescue in 2016 from the Fire and Disaster Management Agency of Japan1 showed that more than 5 million patients were transferred annually to hospitals by ambulance in Japan, with approximately 9% being severe patients, 41% moderate, and 50% mild. As numerous patients receive triage, initial treatment, or require transfers, a simple scoring tool, such as the quick sequential organ failure assessment (qSOFA) score,2 systemic inflammatory response syndrome (SIRS)3 criteria, or shock index (SI),4 are needed for triage during the limited time in prehospital emergency care. Vital signs were one of the objective indicators available in prehospital triage. The qSOFA score was proposed in the Third International Consensus Definitions for Sepsis and Septic Shock (SEPSIS‐3),2 and the qSOFA score is recommended for use in daily routine sepsis screening.5 This score includes systolic blood pressure ≤100 mmHg, a respiratory rate ≥22 breaths/min, and an altered mental status (one point each; 0–3). The SIRS criteria were proposed in the Definitions for Sepsis and Organ Failure guidelines in 1992.3 This score includes a body temperature <36°C or >38°C, heart rate >90 b.p.m., respiratory rate >20 breaths/min or PaCO2 <32 Torr, and white blood cell count (WBC) <4,000/mm3 or >12,000/mm3 (one point each; 0–4). The SI is commonly used for patients with trauma and hemorrhagic shock and is expressed as heart rate/systolic blood pressure.4

The qSOFA score is superior to the SIRS criteria3 and conventional SOFA score6 for predicting the mortality of patients with suspected infection outside of intensive care unit settings.2 Singer et al.7 suggested that the qSOFA score is associated with mortality in adult patients with and without suspected infection in the emergency department (ED). In recent prehospital studies, a strong relationship was observed between the qSOFA score and mortality in infected patients8; however, its sensitivity for detecting sepsis was low.9 Kitahara et al.10 suggested that the prehospital qSOFA score was more strongly associated with in‐hospital mortality in non‐infected patients than the modified early warning score. However, these simple indicators, such as the qSOFA score, SIRS criteria, and SI, have not been compared for in‐hospital mortality in prehospital patients.

Laboratory tests, such as those for WBC and PaCO2, can be carried out rapidly in the ED. Indicators using only vital signs might be more useful in prehospital care than in the ED. Although the calculation of the SIRS criteria requires WBC, it is an indicator that is more frequently compared to the qSOFA score. The SI is one of the most classical indicators that uses only vital signs. Although the modified early warning score is a relatively new indicator using only vital signs, its calculation is slightly complex in prehospital care. We undertook a single‐site retrospective study, the aim of which was to compare the qSOFA score, SIRS criteria without WBC and PaCO2 (because of the impossibility of a blood examination in prehospital care), and SI for predicting the in‐hospital mortality of patients transferred to the ED by ambulance and also to verify whether the qSOFA score is useful for prehospital triage. The qSOFA score and SIRS criteria have mainly been used for infectious diseases, whereas SI is mainly used for trauma;11 therefore, we undertook a subgroup analysis of infectious diseases and trauma.

Methods

This study was approved by the Institutional Review Board of the National Hospital Organization Kumamoto Medical Center (KMC) (Kumamoto, Japan). A retrospective study was carried out among adult (age ≥ 18 years) patients transferred by ambulance to the KMC ED between January 2015 and December 2016. The KMC is a tertiary emergency hospital in Japan. It has 550 beds, including 6 in the intensive care unit, 4 in the coronary care unit, 44 emergency dedicated beds and 50 in the psychiatric ward. We searched prehospital medical records for vital signs (consciousness, body temperature, systolic blood pressure, heart rate, and respiratory rate) during transfers to the ED. We excluded the following: out‐of‐hospital cardiac arrest patients, patients transferred from other hospitals, patients for whom one or more vital signs were not available, patients not requiring admission, and patients whose outcomes were unknown due to their admission to other hospitals. Data on vital signs were the first vital signs measured on patient contact with emergency medical technicians; data on systolic blood pressure and heart rate were simultaneously measured vital signs. The outcome was in‐hospital mortality. Statistical analyses were undertaken with Stata (version 15.0; Stata Corp., College Station, TX, USA). Univariate analyses of baseline characteristics and their indicators were carried out by the Wilcoxon rank‐sum test or Fisher's exact test. A receiver operating characteristic (ROC) analysis and Delong's method12 were used to compare each indicator. The optimal cut‐off values of each indicator were decided upon as the point of the highest sum of sensitivity and specificity (the Youden index).13

Results

A total of 16,764 patients were transferred to the KMC by ambulance between January 2015 and December 2016 (Fig. 1). Excluded patients included 788 (4.7%) <18 years, 2,467 (14.7%) who were transferred from other hospitals, 372 (2.2%) with cardiac arrest, 6,954 (41.5%) not requiring admission, 182 (1.1%) of unknown outcomes due to their admission to other hospitals, and 1,174 (7.0%) for whom one or more vital signs were unavailable, yielding 4,827 patients (28.8%) who met the inclusion criteria. There were 4,542 survivors (94.1%) and 285 non‐survivors (5.9%). A comparison of non‐survivors and survivors with a univariate analysis revealed that survivors were significantly younger than non‐survivors. Heart rate and respiratory rate were significantly higher in non‐survivors than in survivors, systolic pressure was significantly lower in non‐survivors than in survivors, and the number of patients with an altered mental status was significantly higher in non‐survivors than in survivors (Table 1). The number that conformed to the SIRS criteria and qSOFA score was significantly higher in non‐survivors than in survivors. The SI was also significantly higher in non‐survivors than in survivors. Regarding the area under the ROC curve (AUROC) of each score; the 95% confidence interval was 0.64 (0.61–0.67) for the qSOFA score, 0.59 (0.56–0.62) for the SIRS criteria, and 0.58 (0.54–0.62) for the SI (Fig. 2, Table 2). The AUROC of the qSOFA score was significantly higher than those of the SIRS criteria and SI (Fig. 2, Table 2). In the subgroup analysis of infectious diseases, trauma, and others, the AUROC of the qSOFA score was significantly higher than those of the SIRS criteria in infectious diseases and others (Fig. 2, Table 2). The optimal cut‐off values of each indicator were two points of the qSOFA score, two points of the SIRS criteria, and 0.9 of the SI by the Youden index.13 The sensitivity of the qSOFA score (52.3%) was lower than that of the SIRS criteria (54.7%) and higher than that of the SI (32.3%). The specificity of the qSOFA score (69.9%) was higher than that of the SIRS criteria (59.0%) and lower than that of the SI (82.8%) (Table 3).

Figure 1

Flowchart of the present study including patients transferred by ambulance to the Kumamoto Medical Center ED (Kumamoto, Japan) between January 2015 and December 2016.

Table 1

Comparison of characteristics, vital signs, systemic inflammatory response syndrome (SIRS) criteria, shock index, and quick sequential organ failure assessment (qSOFA) score between survivors and non‐survivors

		Survivors	Non‐survivors	P‐value
Number of all patients	n (%)	4,542 (94.1)	285 (5.9)
Infection	n (%)	770 (93.9)	50 (6.1)
Trauma	n (%)	620 (97.0)	19 (3.0)
Others	n (%)	3,152 (93.6)	216 (6.4)
Age	Median (IQR)	76 (60–85)	81 (70–87)	<0.001
Sex	Male : female	2,272:2,270	150:135	0.427
Body temperature	Median (IQR)	36.6 (36.0–37.3)	36.6 (36.0–37.6)	0.761
Systolic blood pressure	Median (IQR)	139 (117–161)	130 (106–165)	<0.001
Heart rate	Median (IQR)	90 (76–105)	94 (80–112)	<0.001
Respiratory rate	Median (IQR)	20 (20–24)	24 (20–30)	<0.001
Altered mental status	n (%)	2,244 (49.4)	192 (67.4)	<0.001
SIRS criteria w/o WBC	Median (IQR)	1 (1–2)	2 (1–2)	<0.001
Shock index	Median (IQR)	0.64 (0.52–0.82)	0.72 (0.53–1.00)	<0.001
qSOFA score	Median (IQR)	1 (0–2)	2 (1–2)	<0.001

IQR, interquartile range; WBC, white blood cell count.

Figure 2

Comparison of receiver operating characteristic curves of the quick sequential organ failure assessment (qSOFA) score, systemic inflammatory response syndrome (SIRS) criteria, and shock index in all patients (A), patients with infectious diseases (B), patients with trauma (C), and patients without infectious diseases or trauma (D).

Table 2

Comparison of the area under the receiver operating characteristic curve (AUROC) (95% confidence interval [CI]) of the quick sequential organ failure assessment (qSOFA) score, systemic inflammatory response syndrome (SIRS) criteria, and shock index

	n	AUROC (95% CI)			P‐value
		(A) qSOFA	(B) SIRS	(C) Shock index	A versus B	A versus C	B versus C
All patients	4,827	0.64 (0.61–0.67)	0.59 (0.56–0.62)	0.58 (0.54–0.62)	<0.001	0.001	0.528
Infection	820	0.66 (0.59–0.73)	0.56 (0.48–0.63)	0.62 (0.53–0.70)	0.008	0.377	0.204
Trauma	639	0.60 (0.48–0.71)	0.51 (0.38–0.63)	0.51 (0.36–0.65)	0.100	0.978	0.295
Others	3,368	0.64 (0.60–0.67)	0.60 (0.56–0.64)	0.57 (0.53–0.61)	0.033	0.004	0.207

Table 3

Comparison of diagnostic accuracy of the quick sequential organ failure assessment (qSOFA) score, systemic inflammatory response syndrome (SIRS) criteria, and shock index (SI) for in‐hospital mortality

Patients	n	Cut‐off value	Sensitivity, %	Specificity, %
All	4,827	qSOFA ≥2	52.3	69.9
		SIRS (w/o WBC) ≥2	54.7	59.0
		SI ≥0.9	32.3	82.8
Infection	820	qSOFA ≥2	66.0	62.7
		SIRS (w/o WBC) ≥2	70.0	40.5
		SI ≥0.8	58.0	60.5
Trauma	639	qSOFA ≥2	31.6	80.2
		SIRS (w/o WBC) ≥2	31.6	72.1
		SI ≥0.8	21.1	84.4
Others	3,368	qSOFA ≥2	50.9	69.6
		SIRS (w/o WBC) ≥2	53.2	61.0
		SI ≥0.9	31.9	83.2

Cut‐off values were selected as the point of the highest sum of sensitivity and specificity (the Youden index).

WBC, white blood cell count.

Discussion

We compared three simple and commonly used indicators for in‐hospital mortality. In the present study, qSOFA more strongly predicted in‐hospital mortality as per the AUROC than the SIRS criteria and SI in patients transferred to the ED by ambulance. The ROC curves showed that the qSOFA score had a higher AUROC than the SIRS criteria and SI for infectious diseases, trauma, and others as well as in all patients. Furthermore, when the optimal cut‐off values of each indicator were assessed by the Youden index,13 the sensitivity of the qSOFA score was higher than that of the SI and slightly lower than that of the SIRS criteria; the specificity of the qSOFA score was higher than that of the SIRS and lower than the SI. Although the qSOFA score had the highest ability to predict in‐hospital mortality in ROC analyses, the sensitivity of the qSOFA score for in‐hospital mortality was low.

These three indicators, especially SI and the qSOFA score without the need for laboratory testing, are easy to calculate and convenient to use for triaging patients during the limited time in prehospital care by emergency medical technicians. The SI is a simple score defined as heart rate/systolic blood pressure.4 Although it is primarily used for patients with hemorrhagic shock or trauma,4 it is also used for critically ill patients or those with sepsis.14, 15 Elevations in the SI indicate an increased heart rate or decreased systolic blood pressure. As distributive shock is the most common form of shock among patients, followed by cardiogenic and hypovolemic shock, and obstructive shock is relatively rare,16 we considered the SI to generally be applicable to most patients with shock. In the present study, the SI had the highest specificity and lowest sensitivity for mortality among the three indicators tested. It is useful for circulatory disturbances, including hemorrhagic shock or septic shock, but does not detect any other abnormalities, such as respiratory failure or disturbance of consciousness. Therefore, the SI might be less suitable than the qSOFA score for the triage of prehospital emergency patients. The SIRS criteria (w/o WBC) had the highest sensitivity and lowest specificity for mortality among the three indicators tested. We used the SIRS criteria without WBC because the present study was undertaken in prehospital settings. The ability of the SIRS criteria (w/o WBC) to predict in‐hospital mortality was different from the original SIRS criteria. Although this difference could have influenced the results obtained in the present study, the AUROC was similar for in‐hospital mortality among patients with suspected infections in a previous study.17 Because the SIRS criteria has high sensitivity for admission,18 it might have low specificity for in‐hospital mortality. Therefore, the SIRS criteria could be more useful for predicting admission than in‐hospital mortality. After the publication of SEPSIS‐3,2 some investigators reported that the qSOFA score predicted the mortality of patients in EDs with infections, such as pneumonia,19, 20, 21 or regardless of suspected infections;7, 22 furthermore, the qSOFA score was superior to the SIRS criteria for predicting mortality.23 In this cohort, the AUROC of the qSOFA score to predict in‐hospital mortality was superior to that of the SIRS criteria and SI in prehospital emergency patients and it had similar AUROC for mortality in a previous study.7 A previous meta‐analysis showed that the sensitivity (95%CI) of qSOFA for in‐hospital mortality in hospitalized patients with suspected infection was 0.56 (0.47–0.65) and its specificity (95% CI) was 0.78 (0.71–0.83), indicating that its ability as a predictive marker is poor.24 Sensitivity and specificity for in‐hospital mortality in prehospital patients in the present study were similar to these values. Although the qSOFA score had higher AUROC than the SIRS criteria and SI, its sensitivity for in‐hospital mortality was low. Therefore, the qSOFA score might not contribute to triage for in‐hospital mortality in prehospital emergency patients.

Our study had several limitations. It was a single‐site retrospective study. Furthermore, one or more vital signs were missing in 1,174 patients including 111 (9.5%) non‐survivors. These limitations could have contributed to a selection bias.

Conclusions

The qSOFA score is easy to calculate using only vital signs without the need for laboratory testing and had the highest AUROC among the three simple indicators examined; however, it might be insufficient for triage patients in prehospital settings due to its low sensitivity for in‐hospital mortality.

Disclosure

Approval of the research protocol: This study was approved by the Institutional Review Board of the National Hospital Organization Kumamoto Medical Center in Japan (reference nos. 753 and 768).

Informed consent: N/A.

Registry and the registration no. of the study/trial: UMIN000032365.

Animal studies: N/A.

Conflict of interest: None.