Quick Sequential Organ Failure Assessment Score and the Modified Early Warning Score for Predicting Clinical Deterioration in General Ward Patients Regardless of Suspected Infection.

BACKGROUND: The quick sequential organ failure assessment (qSOFA) score is suggested to use for screening patients with a high risk of clinical deterioration in the general wards, which could simply be regarded as a general early warning score. However, comparison of unselected admissions to highlight the benefits of introducing qSOFA in hospitals already using Modified Early Warning Score (MEWS) remains unclear. We sought to compare qSOFA with MEWS for predicting clinical deterioration in general ward patients regardless of suspected infection.
METHODS: The predictive performance of qSOFA and MEWS for in-hospital cardiac arrest (IHCA) or unexpected intensive care unit (ICU) transfer was compared with the areas under the receiver operating characteristic curve (AUC) analysis using the databases of vital signs collected from consecutive hospitalized adult patients over 12 months in five participating hospitals in Korea.
RESULTS: Of 173,057 hospitalized patients included for analysis, 668 (0.39%) experienced the composite outcome. The discrimination for the composite outcome for MEWS (AUC, 0.777; 95% confidence interval [CI], 0.770-0.781) was higher than that for qSOFA (AUC, 0.684; 95% CI, 0.676-0.686; P < 0.001). In addition, MEWS was better for prediction of IHCA (AUC, 0.792; 95% CI, 0.781-0.795 vs. AUC, 0.640; 95% CI, 0.625-0.645; P < 0.001) and unexpected ICU transfer (AUC, 0.767; 95% CI, 0.760-0.773 vs. AUC, 0.716; 95% CI, 0.707-0.718; P < 0.001) than qSOFA. Using the MEWS at a cutoff of ≥ 5 would correctly reclassify 3.7% of patients from qSOFA score ≥ 2. Most patients met MEWS ≥ 5 criteria 13 hours before the composite outcome compared with 11 hours for qSOFA score ≥ 2.
CONCLUSION: MEWS is more accurate that qSOFA score for predicting IHCA or unexpected ICU transfer in patients outside the ICU. Our study suggests that qSOFA should not replace MEWS for identifying patients in the general wards at risk of poor outcome.

INTRODUCTION

Some hospitalized patients experience clinical deterioration associated with hospital mortality or unexpected intensive care unit (ICU) transfers, which can result in increased morbidity and mortality.12 Previous studies postulate that patients show signs of increased risk hours before clinical deterioration; consequently, these early signals can be captured by regularly measured vital signs.123 Therefore, the early warning score (EWS) based on the vital signs was developed and adopted for screening patients with a high risk of clinical deterioration.12345 The Modified Early Warning Score (MEWS) based on five physiological parameters is an evaluation tool for the early identification of medical patients who needed intensive care.6

Sepsis, an inflammatory response to infection, contributed to ≥ 50% hospital mortality.7 The Sepsis-3 task force recommends the quick sequential (sepsis-related) organ failure assessment (qSOFA) score to rapidly identify patients with suspected infection at high risk of poor outcomes.8910 The qSOFA score, which is similar to the EWSs, was used in the general wards for screening and alerting when a patient is at high risk of a serious adverse outcome, irrespective of the underlying diagnosis.61112 However, the recent Surviving Sepsis Campaign recently recommended that qSOFA not be used as a single screening tool for sepsis or septic shock compared to the Systemic Inflammatory Response Syndrome criteria or MEWS.13

Introducing qSOFA in hospitals already using MEWS might pose a challenge because of the similar physiological variables and different weighting thresholds. Consequently, using both screening tools can lead to unnecessary duplication of staff effort, needless protocol complexity, and increased educational requirements. Recent publications demonstrated that MEWS discriminates in-hospital mortality, ICU admission, and their combined outcomes was better than qSOFA in patients with suspected infection.1415 However, data on the comparison in unselected hospital admissions to highlight the benefit of introducing qSOFA in hospitals already using MEWS are limited. Therefore, the aim of this study was to compare the performance of qSOFA as an early warning score with MEWS in hospitalized non-ICU patients regardless of diagnosis of infection by using a large database of routinely collected vital signs from five different characteristic hospitals in Korea.

METHODS

Study design and population

Over a 12-month period, a retrospective cohort study was conducted on all consecutive adult patients admitted to the general ward of five hospitals—Mediplex Sejong Hospital (a 323-bed secondary cardiovascular-specific hospital in Bucheon, South Korea, from January 2017 to December 2017), Sejong General Hospital (a 301-bed secondary cardiovascular-specific hospital in Inchon, South Korea, from March 2018 to March 2019), Inha University Hospital (a 925-bed university-affiliated, tertiary referral hospital in Inchon, South Korea, from January 2018 to December 2018), Seoul National University Bundang Hospital (a 1324-bed university-affiliated, tertiary referral hospital in Seoul, South Korea, from January 2017 to December 2017), and Samsung Medical Center (a 1989-bed university-affiliated, tertiary referral hospital in Seoul, South Korea, from July 2017 to June 2018). Each admission was collected separately for analysis in patients with multiple hospital admissions during the study period. Patients who met the following criteria were excluded: 1) admission directly to the ICU at hospital admission, 2) no vital sign data recorded in the 24 hours before in-hospital cardiac arrest (IHCA) or unexpected ICU transfer, or 3) only vital sign data of < 30 minutes recorded.

Data collection

Databases of vital signs collected from consecutive hospitalized patients older than 18 years over 12 months were created from five participating hospitals. The data was extracted using an each institution’s clinical data warehouse, which automatically retrieved data from electronic medical records. All of time-stamped basic vital signs including systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate (HR), respiratory rate (RR), body temperature (BT), and level of consciousness were collected during the hospitalization of the patients. The exact time and location of event occurrences were also extracted from the electronic medical records. Levels of consciousness were collected using the AVPU scale (alert [A]; responds to voice [V]; responds to pain [P]; or unresponsive [U]).1617 In addition, we categorized patients scoring V, P, or U as having “altered mental status” when calculating qSOFA scores.1819 From the initially collected data, erroneous values which are extremely outside of the acceptable range of each vital sign (30 to 300 mm Hg of SBP, 40 to 120 mm Hg of DBP, 10 to 300 beats/min of HR, 3 to 60 breaths/min of RR, and 30 to 45°C of BT) or non-numeric values were excluded and assumed to be missing values. Missing values are processed by using the most recent values measured before the time the missing value existed. Subsequently, the MEWS and qSOFA score were calculated. The performance of MEWS and qSOFA were compared to predict the outcomes within 24 hours of vital sign observation.

The primary outcome of the study was clinical deterioration defined as the composite of IHCA and unexpected ICU transfer. IHCA was defined as whether cardiac pulmonary resuscitation was performed during general ward admission in the electronic medical record. Unexpected ICU transfer was defined as an event wherein patients were transferred to the medical ICU that did not originate from either the emergency department or an operating room.2021 The secondary outcome of the study was IHCA and unexpected ICU transfer, respectively.

The qSOFA criteria were defined as SBP ≤ 100 mmHg, RR ≥ 22 breaths per minute, and altered mental status (defined as AVPU scale other than “Alert”).8 The MEWS was calculated based on previously published table (Supplementary Table 1).6

Statistical analysis

Data were presented as either median with interquartile range (IQR) or mean with standard deviation for continuous variables and number (%) for categorical variables. The discriminatory power of each score was assessed by calculating the area under each receiver operating characteristic curve (AUC). Accuracy comparisons were performed using sensitivity, specificity, positive and negative predictive values. The Youden index, defined as (sensitivity + specificity) − 1, was calculated to verify the current thresholds of qSOFA and MEWS. To calculate the AUC, true negative and false positive were calculated using each patient’s serial vital signs during their entire stay outside the ICU stay in patients who were not deteriorating clinically. In addition, each patient’s highest scores during their hospitalization were used in clinically not deteriorated patients, and each patient’s highest scores from 24 hours to 30 minutes before IHCA or unexpected ICU transfer were used in patients who deteriorated clinically. False positive and true negative were calculated using each patient’s serial vital signs from 24 hours to 30 minutes before IHCA or unexpected ICU transfer in patients who deteriorated clinically. Subsequently, the AUC for primary and secondary outcomes was calculated on the receiver operating characteristic curve, and values for the estimated AUC were compared using the McNeil test.22 To report predictive performance of qSOFA and MEWS at the critical threshold, the cutoff values of qSOFA ≥ 2 and MEWS ≥ 5 were used.68 Reclassification was evaluated using the net reclassification index (NRI),23 which quantifies the relative improvement of clinical deterioration prediction between MEWS ≥ 5 and qSOFA ≥ 2. The NRI is defined as (proportion of all events reclassified to clinical deterioration - proportion of all events reclassified to without clinical deterioration) - (proportion of all non-events reclassified to clinical deterioration - proportion of all events reclassified to without clinical deterioration). All analyses were performed using R Statistical Software (Version 3.5.1; R Foundation for Statistical Computing, Vienna, Austria). All tests of significance were two-tailed, and P < 0.05 was considered statistically significant.

Ethics statement

The Institutional Review Boards of all participating hospitals approved this study (Samsung Medical Center, SMC-2019-09-129; Seoul National University Bundang Hospital, B-2003-598-004; Inha University Hospital, 2020-02-013-000; Mediplex Sejong Hospital, 2018-054; Sejong General Hospital, 2018-0689) and waived the requirement for informed consent because of the observational nature of the research. In addition, this study adhered to relevant ethical guideline and the statement for reports of diagnostic accuracy studies with Standards for Reporting of Diagnostic Accuracy Studies (STARD).

RESULTS

Patient characteristics

During the study period, 173,057 patients hospitalized in the general wards, with available medical records and documented vital signs, were included. The baseline characteristics of the patients are summarized in Table 1. The mean age of the patients was 57.4 (± 15.8) years, and 86,549 (50.0%) patients were men. The median length of hospital stay was 5.9 days. At admission, the initial mean SBP, DBP, HR, RR, and BT was 126.5 (± 19.8) mmHg, 74.5 (± 12.3) mmHg, 77.8 (± 14.4) beats per minute, 18.1 (± 2.0) rates per minute, and 36.5°C (± 0.5°C). Of all patients, 755,724 (91.1%) were alert at admission. The prevalence of IHCA and unexpected ICU transfer was 1.29 per 1,000 admissions and 2.57 per 1,000 admissions, respectively. The baseline characteristics according to each of five hospitals are presented in Supplementary Table 2.

Table 1

Characteristics and clinical outcomes of hospitalized non-ICU patients

Variables			Total (N = 173,057)
Age, yr			57.4 ± 15.8
Sex, male			86,549 (50.0)
Length of hospital stay, day			5.9 (1.6–6.7)
Vital signs at admission
	Systolic blood pressure, mmHg		126.5 ± 19.8
	Diastolic blood pressure, mmHg		74.5 ± 12.3
	Heart rate, beat per minute		77.8 ± 14.4
	Respiratory rate		18.1 ± 2.0
	Body temperature, °C		36.5 ± 0.5
Mental status
	Alert		755,724 (91.1)
	Reacting to voice		29,123 (3.5)
	Reacting to pain		9,676 (1.1)
	Unresponsive		5,204 (0.6)
Clinical outcomes
	In-hospital cardiac arrest		224
		per 1,000 admission	1.29
	Unexpected ICU transfer		444
		per 1,000 admission	2.57

ICU = intensive care unit.

Comparison of qSOFA with MEWS

Algorithm discrimination for the clinical deterioration for MEWS (AUC, 0.777; 95% confidence interval [CI], 0.770–0.781) was higher than that for qSOFA (AUC, 0.684; 95% CI, 0.676–0.686; P < 0.001) (Fig. 1). The receiver operating characteristic curves of outcomes at each participating hospital are presented in Supplementary Figures. Besides, MEWS was better in predicting IHCA (AUC, 0.792; 95% CI, 0.781–0.795 vs. AUC, 0.640; 95% CI, 0.625–0.645; P < 0.001) and unexpected ICU transfer (AUC, 0.767; 95% CI, 0.760–0.773 vs. AUC, 0.716; 95% CI, 0.707–0.718; P < 0.001) than qSOFA. Most patients met the MEWS ≥ 5 criteria 13 hours before clinical deterioration compared with 11 hours for qSOFA ≥ 2 criteria (Fig. 2).

Fig. 1

The receiver operating characteristic curves of outcomes. (A) In-hospital cardiac arrest, (B) unexpected intensive care unit transfer, (C) composite outcome.

MEWS = Modified Early Warning Score, qSOFA = quick sequential organ failure assessment, AUC = areas under the receiver operating characteristic curve.

Fig. 2

Cumulative percentage of patients meeting ≥ 2 qSOFA score or ≥ 5 MEWS in the 24 hours before the outcomes. (A) In-hospital cardiac arrest, (B)unexpected intensive care unit transfer, (C) composite outcome.

MEWS = Modified Early Warning Score, qSOFA = quick sequential organ failure assessment.

Using the patient’s highest scores, the sensitivity and specificity of qSOFA ≥ 2 were 13.9% (95% CI, 13.3–14.5) and 98.6% (95% CI, 98.6–98.7) compared with those of MEWS ≥ 5 at 17.0% (95% CI, 16.1–17.7) and 99.4% (95% CI, 99.4–99.4) for clinical deterioration, respectively. The sensitivity, specificity, and positive and negative predictive values for the clinical deterioration at different cutoff points for each scoring system are presented in Table 2. The MEWS ≥ 5 resulted in reclassification of 3.7% of the patients from qSOFA ≥ 2. The NRI for clinical deterioration was 0.0033 (95% CI, 0.0002–0.0062) and the NRI for without clinical deterioration was 0.0073 (95% CI, 0.0072–0.0074; P = 0.043). The Supplementary Tables 3 and 4 show the predictive accuracy for IHCA and unexpected ICU transfer separately.

Table 2

Prediction accuracy for primary outcome according to different score thresholds

Score/threshold		Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Youden index
qSOFA
	≥ 1	53.9	80.3	0.3	99.9	0.3429
	≥ 2a	13.9	98.6	1.3	99.8	0.1259
	≥ 3	1.2	99.9	2.5	99.8	0.0116
MEWS
	≥ 0	100	0	0.1	-	0
	≥ 1	68.9	77.2	0.4	99.9	0.4600
	≥ 2	48.3	93.6	0.9	99.9	0.4200
	≥ 3	48.3	93.6	0.9	99.9	0.4201
	≥ 4	28.9	98.0	1.8	99.9	0.2704
	≥ 5a	17.0	99.4	3.5	99.8	0.1641
	≥ 6	8.7	99.8	5.5	99.8	0.0860
	≥ 7	3.7	99.9	6.4	99.8	0.0363
	≥ 8	1.0	99.9	4.9	99.8	0.0098
	≥ 9	0.4	99.9	5.6	99.8	0.0041
	≥ 10	0.2	100.0	5.0	99.8	0.0015
	≥ 11	0.0	100.0	0	99.8	0
	≥ 12	0.0	100.0	0	99.8	0
	≥ 13	0.0	100.0	0	99.8	0
	≥ 14	0.0	100.0	0	99.8	0
	≥ 15	0.0	100.0	-	99.8	0

PPV = positive predictive value, NPV = negative predictive value, qSOFA = quick sequential organ failure assessment, MEWS = Modified Early Warning Score.

aCommonly used cutoff thresholds.

DISCUSSION

The present study investigated the performance of qSOFA compared with MEWS for clinical deterioration in hospitalized non-ICU patients regardless of diagnosis of infection by using a large database of routinely collected vital signs from five different characteristic hospitals in South Korea. The qSOFA showed worse discrimination for clinical deterioration, IHCA, and unexpected ICU transfer than MEWS. In addition, qSOFA ≥ 2 showed lower sensitivity and specificity compared with MEWS ≥ 5.

The EWSs have been developed based on the findings that abnormal changes in physiologic parameters, such as vital signs or mental status, often precede overt clinical deterioration by several hours.324 The MEWS, and its derivatives, such as the national early warning score (NEWS), which was endorsed by the Royal College of Physicians for standard use across the United Kingdom, have been already used in many hospitals for the rapid response system.2526 MEWS includes SBP, HR, RR, BT, and level of consciousness, and values of 0 to 3 are given to each parameter based on the degree of abnormality.6 Several studies reported that recording of all vital signs in MEWS might be complex, and these complexities may be associated with adverse events from an incomplete set of observations.2728 Furthermore, there is debate regarding the performance of EWSs. Shappell et al.29 reported that NEWS has poor discriminatory power (AUC, 0.66; 95% CI, 0.66–0.67) for 28-day mortality. In addition, Ahn et al. also reported poor discriminatory power of MEWS (AUC, 0.58; 95% CI, 0.56–0.59) and NEWS (AUC, 0.60; 95% CI, 0.59–0.62), respectively.30

The Sepsis-3 task force proposed the qSOFA (also known as quick SOFA), an empirically derived score using simple clinical criteria, that may identify patients with suspected infection who are at greater risk for a poor outcome outside the ICU.8 It uses three criteria, assigning one point for low blood pressure (SBP ≤ 100 mmHg), high RR (≥ 22 breaths per minutes), or altered mentation (Glasgow Coma Scale < 15),8 which are not specific variables for infection. Compared with MEWS, qSOFA has the advantage of simplicity because it includes only three binary elements and does not require a reference table or calculator. However, the qSOFA was identified as a predictor of poor outcome in patients with known or suspected infection.8 Although numerous studies have investigated the potential use of the qSOFA as a screening tool for sepsis, the results have been contradictory to its usefulness.31 In addition, a retrospective register-based validation study demonstrated that qSOFA performs similarly in patients not yet diagnosed or suspected with infection, implying that qSOFA is not infection specific and could simply be regarded as a general EWS.32 These results support recent Survival Sepsis Campaign, which recommends against using qSOFA compared to other EWSs as a single screening tool for sepsis or septic shock.13

However, previous studies have shown that general EWSs are more accurate than qSOFA when predicting adverse outcomes in the general wards (Table 3).14151833 Churpek et al.14 compared the performance of qSOFA with MEWS and NEWS for predicting death and ICU transfer in patients with suspected infection at the emergency room or general ward in a single center during 8 years. They revealed that MEWS (AUC, 0.73; 95% CI, 0.71–0.74) and NEWS (AUC, 0.77; 95% CI, 0.76–0.79) were more accurate than qSOFA (AUC, 0.69; 95% CI, 0.67–0.70). Furthermore, Redfern et al.18 compared the performance of qSOFA with NEWS for predicting in-hospital mortality and unexpected ICU transfer in general ward patients regardless of infection in a single center during 6 years. They showed that NEWS (AUC, 0.813, 95% CI, 0.810–0.817) were more accurate than qSOFA (AUC, 0.671; 95% CI, 0.666–0.675). In the present study, MEWS was more accurate than qSOFA for predicting clinical deterioration in general ward patients regardless of suspected infection in five centers during 12 months. Taken together, these studies provide evidence that hospitals already using EWSs should not change to or introduce qSOFA for screening patients at risk of a poor outcome outside the ICU.

Table 3

Characteristics of studies comparing qSOFA and MEWS

Study	Participating center	Setting	Duration of data collection	Number of patients	Admission	Suspected infection	MEWS (95% CI)	qSOFA (95% CI)	Primary outcome
Churpek et al.[14]	Single	Urban tertiary academic center	111 months	30,677 (12,154 in ward and 18,523 in ED)	Ward or ED	Yes	0.73 (0.71–0.74)	0.69 (0.67–0.70)	In-hospital mortality
Churpek et al.[15]	Single	Urban tertiary academic center	111 months	53,849	Ward or ED	Yes	0.67 (0.66–0.68)	0.65 (0.62–0.66)	In-hospital mortality
Redfern et al.[18]	Single	Tertiary general hospital	61 months	241,996	Ward	Mixed	0.82 (0.82–0.82)a	0.68 (0.67–0.68)	In-hospital mortality
Liu et al.[33]	Multi (32 hospitals in the USA)	Mixed	156 months	1,487,263	Ward	Mixed	0.83 (0.83–0.84) in California and 0.84 (0.84–0.85) in Illinois	0.78 (0.78–0.79) in California and 0.78 (0.77–0.78) in Illinois	In-hospital mortality
Present study	Multi (5 hospitals in the Korea)	Mixed	12 months	173,057	Ward	Mixed	0.77 (0.77–0.78)	0.68 (0.67–0.68)	Clinical deterioration (in-hospital cardiac arrest and unexpected ICU transfer)

qSOFA = quick sequential organ failure assessment, MEWS = Modified Early Warning Score, ED = emergency department, ICU = intensive care unit, CI = confidence interval.

aNational Early Warning Score (NEWS).

Although this study provided additional information on the performance of qSOFA and MEWS for predicting clinical deterioration in hospitalized, non-ICU patients regardless of suspected infection, there are several limitations that should be acknowledged. First, the study was conducted in a single country, which may lessen the generalizability of our findings to other countries. However, we included various featured hospitals. Therefore, our results represent the actual performance of MEWS in non-ICU patients. Nevertheless, further comparisons in well-designed prospective studies are needed. Second, the use of electronic medical records data may also be limited by missing data resulting from incomplete documentation of certain clinical observations. Third, because only the vital signs of non-ICU patients were collected, comorbidity or reason for admission, including whether it was related to infection, were not investigated. In addition, detailed clinical information of patients such as comorbidities were not collected. Fourth, the single-payer national health system, which provides mostly private provision of healthcare in South Korea, has resulted in different hospitalization or ICU admission standards compared with other countries. Finally, our outcome included IHCA and/or unexpected ICU transfers, which had the potential of including expected IHCA and unexpected ICU transfers that did not reflect actual clinical deterioration. Although this composite outcome might reduce the clinical utility of risk scores in practice, it would unlikely influence the overall ordering of model discrimination.34

In this multicenter study, MEWS was more accurate than the qSOFA for predicting IHCA and unexpected ICU transfer in patients outside the ICU, suggesting that the qSOFA should not replace MEWS for identifying patients at risk of poor outcome in hospitalized patients in the general wards.