A comparison of scoring systems for predicting mortality and sepsis in the emergency department patients with a suspected infection.

OBJECTIVE: We aimed to compare the modified National Early Warning Score (mNEWS), quick Sequential Organ Failure Assessment (qSOFA) score, modified Systemic Inflammatory Response Syndrome (mSIRS) score, and modified Search Out Severity (mSOS) score in predicting mortality and sepsis among patients suspected of first observed infections in the emergency department. The modified scores were created by removing variables for simplicity.
METHODS: This was a prospective cohort study that enrolled adult patients presenting at the emergency department with signs and symptoms suggesting infection. The mNEWS, qSOFA score, mSIRS score, and mSOS score were calculated using triage data. The SOFA score was a reference standard for sepsis diagnosis. All patients were monitored for up to 30 days after the initial visit to measure each scoring system's ability to predict 30-day mortality and sepsis.
RESULTS: There were 260 patients included in the study. The 30-day mortality prediction with mNEWS ≥5 had the highest sensitivity (91.18%). The highest area under the receiver operating characteristic curve (AUC) for the 30-day mortality prediction was mNEWS (0.607), followed by qSOFA (0.605), mSOS (0.550), and mSIRS (0.423). The sepsis prediction with mNEWS ≥5 had the highest sensitivity (96.48%). The highest AUC for the sepsis prediction was also mNEWS (0.685), followed by qSOFA (0.605), mSOS (0.480), and mSIRS (0.477).
CONCLUSION: mNEWS was an acceptable scoring system screening tool for predicting mortality and sepsis in patients with a suspected infection.

INTRODUCTION

Sepsis is one of the most challenging global health issues. In 2017, the total sepsis-related mortality worldwide was 19.7% and was higher in males, especially those in low, low-middle, or middle sociodemographic index areas [1]. Rapidly diagnosing sepsis results in early treatment, which decreases mortality. Currently, the Surviving Sepsis Campaign provides guidelines for early detection and treatment [2].

There are several screening tools for predicting sepsis and mortality. The quick Sequential Organ Failure Assessment (qSOFA) is used as a screening tool among patients with suspected infection in the emergency department (ED) or general hospital ward settings [3,4]. Previous studies showed that qSOFA was better than the Systemic Inflammatory Response Syndrome (SIRS) for predicting mortality [5,6]. qSOFA also had higher specificity but lower sensitivity compared to SIRS [5,7]. Further, the National Early Warning Score (NEWS) correlated with increased mortality [8-10], and a score that increased by three or more indicated a risk of severe sepsis and septic shock [11]. NEWS was hypothesized to be comparable with qSOFA and SIRS as a sepsis screening tool, but a later study showed that NEWS was the most accurate screening tool for sepsis [12]. Another study reported that the Search Out Severity (SOS) score could also identify patients with an infection, but the study was performed only on admitted hospital patients [13,14].

We found these scoring systems challenging to use at triage owing to immediate data unavailability, including laboratory results, supplemental oxygen decisions, and urine output records. Therefore, we removed some variables to simplify these scores, then compared each modified scoring system’s ability to predict the 30-day mortality and sepsis in ED patients with a suspected infection. This study aimed to identify the quickest screening tool to use in triage.

METHODS

Study design

This was a prospective cohort study. Data were collected from adult patients who met the inclusion criteria. The patient or a legally authorized representative gave informed consent according to ethical conduct. The institutional review board of the Faculty of Medicine, Chiang Mai University approved the study (470/2017), which is registered with the Thai Clinical Trials Registry (TCTR20200526012, https://www.thaiclinicaltrials.org). We followed the Strengthening the Reporting of Observational Studies in Epidemiology statement recommendations.

Study setting and population

The study was conducted in a university-based, tertiary care hospital ED, with more than 33,000 visits annually. Adult patients with signs and symptoms of suspected infection who visited the ED between December 15, 2017 and March 31, 2018, were enrolled. The signs and symptoms of suspected infection included (but were not limited to): fever, productive cough, turbid urine, and swelling/redness of the skin. The exclusion criteria were patients admitted due to reinfection, pregnancy, patients transferred from other clinics or hospitals, patients lost to follow-up, or incomplete medical records.

Study protocol

Registered nurses trained in triage informed eligible patients, obtained informed consent, and recorded data to calculate the scores. As this study aimed to identify the quickest tool to use in triage, we removed variables from the screening assessments that were not readily available at triage. For NEWS, the oxygen supplement parameter was removed owing to the subjectivity of decision-making by health care personnel. Moreover, patients with sepsis could present with increased respiratory rate, decreased oxygen saturation, and declined consciousness level, all listed as NEWS parameters. Additionally, we removed the white blood cell count and the urine output record from the SIRS and SOS scores, respectively, owing to their unavailability at triage. In this study, the scoring systems were termed modified SIRS (mSIRS), modified NEWS (mNEWS), and modified SOS (mSOS) (Supplementary Tables 1-5) [4,9,14]. The qSOFA variables were already practical, so variables were not removed. The scoring variables were recorded in conventional and electronic medical records.

Outcome measures

The patient characteristics and scoring variables for each patient were obtained from the medical records by a researcher. Each patient’s Sequential Organ Failure Assessment (SOFA) score was calculated as a reference standard for sepsis diagnosis as there is no gold standard. A SOFA score of 2 points or more was considered sepsis according to the standard definition in Sepsis-3 [3]. The patient’s baseline SOFA score was zero unless the patient had a documented preexisting organ dysfunction. Cutoff points of mNEWS ≥5, qSOFA ≥2, mSIRS ≥2, and mSOS ≥4 were used to calculate each scoring system’s ability to identify patients with sepsis, which are the same cutoff points as in the original scoring system [4,10,14]. All patients were monitored for 30 days to evaluate each scoring system’s ability to predict 30-day mortality. All incomplete data affecting mNEWS, qSOFA, mSIRS, mSOS, and SOFA score calculations were excluded. However, other missing data were handled with multiple imputation methods.

Data analyses

The sample size was calculated by the infinite population proportion to ensure an adequate number of patients in the study. A 172 patient sample size was adequate with an alpha error of 0.05, a maximum tolerated error of 0.075, and a dropout rate of 20%, based on a previously reported sensitivity of 72.3% for SIRS ≥2 [7].

Descriptive data were reported as counts and percentages. The mean, standard deviation, median, and interquartile ranges were used for continuous variables. Categorical variables were analyzed using the chi-square or Fisher exact tests. Continuous variables were compared using an independent samples t-test for normally distributed data or a Mann-Whitney U-test for non-normally distributed data. The Shapiro-Wilk test was used to determine the data distribution. Diagnostic accuracy was reported as the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio, negative likelihood ratio, accuracy, and the area under the receiver operating characteristic curve (AUROC). IBM SPSS Statistics ver. 22 (IBM Corp., Armonk, NY, USA) was used for general statistical analyses, and Stata ver. 16 (StataCorp., College Station, TX, USA) for AUROC comparisons. Statistical significance was designated as P<0.05.

RESULTS

In total, 260 patients were included in the study (Fig. 1). Of these, 142 (54.6%) were diagnosed with sepsis and had no missing data. Age, oxygen saturation, consciousness, the partial pressure of arterial oxygen, the platelet count, the bilirubin and creatinine levels, the mNEWS and qSOFA scores, and intensive care unit admission significantly differed between patients with and without sepsis (all P<0.05) (Table 1). There were more patients with alteration of consciousness among the sepsis patients (P<0.001). All patients were hospitalized. Seventeen patients (11.97%) with sepsis were admitted to the intensive care unit. Death within 30 days occurred in 23 patients with sepsis (16.20%) and 11 patients without (9.32%).

Fig. 1.

The study flowchart. SOFA, Sequential Organ Failure Assessment score.

Table 1.

The characteristics of patients

Variable		Patients with sepsis (SOFA ≥ 2) (n = 142)	Patients without sepsis (SOFA < 2) (n = 118)	P-value
Age (yr)		69.5 (59.0–82.0)	64.0 (51.0–77.0)	0.009[a)]
Sex, male		72 (50.70)	56 (47.46)	0.602[b)]
Oxygen saturation (%)		93 (90–95)	96 (94–98)	< 0.001[a)]
Temperature (°C)		38.6 (38.0–39.3)	38.5 (38.1–39.4)	0.598[a)]
Systolic blood pressure (mmHg)		126 (104–148)	128 (112–148)	0.462[a)]
Diastolic blood pressure (mmHg)		70 (57–83)	73 (63–85)	0.106[a)]
Pulse rate (beats/min)		111 ± 26	114 ± 19	0.166[c)]
Respiratory rate (breaths/min)		28 (24–32)	24 (22–32)	0.386[a)]
Consciousness				0.004[b)]
	Awake	75 (52.82)	83 (70.34)
	Responds to verbal stimulation	52 (36.62)	33 (27.97)
	Responds to painful stimulation	11 (7.75)	2 (1.69)
	Unresponsive	4 (2.82)	0 (0)
Partial pressure of arterial Oxygen (mmHg)		67 (57–80)	84 (71–91)	< 0.001[a)]
Platelet count (× 103/mm3)		194 (112–268)	226 (169–285)	0.006[a)]
Bilirubin (mg/dL)		0.87 (0.50–1.71)	0.60 (0.42–1.09)	< 0.001[a)]
Creatinine (mg/dL)		1.18 (0.85–1.84)	0.94 (0.75–1.29)	0.002[a)]
Patients with altered mental status		40 (28.17)	10 (8.47)	< 0.001[b)]
Scores[a)]
	mNEWS	9 (7–10)	7 (5–9)	< 0.001
	qSOFA	1 (1–2)	1 (1–2)	0.002
	mSIRS	3 (2–3)	3 (2–3)	0.459
	mSOS	5 (4–6)	5 (4–6)	0.569
Patients admitted to the intensive care unit		17 (11.97)	2 (1.69)	0.002[b)]
Death within 30 days		23 (16.20)	11 (9.32)	0.102[b)]

Values are presented as median (interquartile range), number (%), or mean±standard deviation.

SOFA, Sequential Organ Failure Assessment; mNEWS, modified National Early Warning Score; qSOFA, quick SOFA score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Analyzed by Mann-Whitney U-test.

Analyzed by chi-square test or the Fisher exact test.

Analyzed by an independent samples t-test.

Table 2 compares the scoring systems’ ability to predict 30-day mortality. mNEWS ≥5 had the highest sensitivity (91.18%), while qSOFA ≥2 had the highest specificity (63.27%), PPV (19.42%), and NPV (91.08%). The highest AUROC for the 30-day mortality prediction was mNEWS (0.607), followed by qSOFA (0.605), mSOS (0.550), and mSIRS (0.423) (Fig. 2). Pairwise comparison of AUROC showed differences between mNEWS versus mSIRS and qSOFA versus mSIRS (Table 3).

Table 2.

Comparing the scoring systems’ ability to predict 30-day mortality and sepsis

Category	mNEWS ≥5	qSOFA ≥2	mSIRS ≥2	mSOS ≥4
30-Day mortality prediction
Sensitivity	91.18	55.82	82.35	76.47
Specificity	9.73	63.27	4.87	20.80
Positive predictive value	13.19	19.42	11.52	12.68
Negative predictive value	88.00	91.08	64.71	85.45
Positive likelihood ratio	1.01	1.60	0.87	0.97
Negative likelihood ratio	0.91	0.65	3.63	1.13
Accuracy	20.38	62.69	15.00	28.08
Sepsis prediction
Sensitivity	96.48	47.18	90.85	78.17
Specificity	16.95	69.49	3.39	20.34
Positive predictive value	58.30	65.05	53.09	54.15
Negative predictive value	80.00	52.23	23.53	43.64
Positive likelihood ratio	1.16	1.55	0.94	0.98
Negative likelihood ratio	0.21	0.76	2.70	1.07
Accuracy	60.38	57.31	51.15	51.92

mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Fig. 2.

Receiver operating characteristic curves of the scoring systems’ ability to predict 30-day mortality. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Table 3.

Area under the receiver operating characteristic curve comparisons of 30-day mortality and sepsis

Scoring systems[a)]	mNEWS	qSOFA	mSIRS
30-Day mortality prediction
qSOFA	0.961	-	-
mSIRS	< 0.001	0.001	-
mSOS	0.508	0.504	0.108
Sepsis prediction
qSOFA	0.001	-	-
mSIRS	< 0.001	0.003	-
mSOS	< 0.001	0.012	0.954

mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Area under the receiver operating characteristic curve comparisons as P-value.

Table 2 also shows a comparison of the scoring systems’ ability to predict sepsis. mNEWS ≥5 had the highest sensitivity (96.48%) and NPV (80%). qSOFA ≥2 had the highest specificity (69.49%) and PPV (65.05%). The highest AUROC for predicting sepsis was observed in mNEWS (0.685), followed by qSOFA (0.605) (Fig. 3). Pairwise comparison of AUROC showed differences between mNEWS versus qSOFA, mNEWS versus mSIRS, mNEWS versus mSOS, qSOFA versus mSIRS, and qSOFA versus mSOS (Table 3). The subsequent calculation results showed that the cutoff point for mNEWS with the best accuracy was ≥7, which had higher specificity (45.76%) and accuracy (66.54%) in sepsis predictions. However, the ≥7 cutoff point had a lower sensitivity (83.80%) than ≥5.

Fig. 3.

Receiver operating characteristic curves of each scoring systems’ ability to predict sepsis. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

DISCUSSION

Many studies have shown that NEWS is the most accurate scoring system [12]. Our NEWS modification, which removed the need for supplemental oxygen at triage, was also the most accurate for predicting 30-day mortality and sepsis. The mNEWS cutoff point ≥5 had high sensitivity but low specificity, implying that mNEWS may be used as a triage screening tool. Although mNEWS had the highest AUROC for the 30-day mortality and sepsis predictions, the value was less than 0.7, suggesting poor discrimination ability. Redfern et al. [15] reported that NEWS ≥5 had a higher AUROC than qSOFA ≥2 in terms of in-hospital mortality predictions, whereas Goulden et al. [10] found that the NEWS ≥5 AUROC was similar to qSOFA ≥2, but had a higher value than SIRS in in-hospital mortality predictions. However, the mortality endpoints differed from those in our study.

We found that mNEWS was better than qSOFA in predicting sepsis. Similarly, Usman et al. [12] found NEWS was the most accurate in detecting severe sepsis or septic shock compared to SIRS ≥2 and qSOFA ≥2. They reported a higher AUROC than ours, which might be due to population differences and the use of unmodified NEWS and SIRS.

In this study, qSOFA had the highest specificity and was the second-best way for predicting 30-day mortality and sepsis. However, its low sensitivity makes it unfit for its use as a screening tool. Similar studies also yielded the same result. Brink et al. [16] found that qSOFA ≥2 had a higher AUROC and specificity than SIRS ≥2 in 30-day mortality predictions. Usman et al. [12] also reported that qSOFA ≥2 had the highest specificity in severe sepsis or septic shock detection. Additionally, Redfern et al. [15] and Churpek et al. [17] showed that qSOFA ≥2 had a higher specificity than SIRS ≥2 in in-hospital mortality predictions. However, the latter studies had different mortality endpoints. One study proposed using the qSOFA score as a sepsis screening tool [3]. However, our study, similar to Usman et al. [12] and Churpek et al. [18] found that qSOFA had low sensitivity and might not be suitable as a sepsis screening tool.

SIRS was used for formally diagnosing sepsis, and a subsequent study found it to have high false positives [19]. Our study showed that modified SIRS had very low specificity and was not suitable as a sepsis screening tool. The mSOS scoring system was also not fit to use at triage owing to its low sensitivity and accuracy.

Our study has some limitations. First, patients’ known medical conditions, which might affect disease progression, were not included in the scoring system calculations. Second, 30 patients (11.54%) were referred out to other hospitals for continuing treatment and were considered alive at 30 days, and 23 patients (8.85%) who refused treatment and returned home were considered dead at 30 days. These patients might have different outcomes, which could alter the study results. Third, the number of patients included in our study was relatively small. Extending the enrollment period to include additional patients should be considered for future studies. Fourth, the infection sources were not documented, although several baseline characteristics were explored in this study.

In summary, mNEWS is an acceptable scoring system screening tool for predicting 30-day mortality and sepsis and could provide quicker evaluation at triage.