Comparing Rapid Scoring Systems in Mortality Prediction of Critically Ill Patients With Novel Coronavirus Disease.

OBJECTIVES: Rapid and early severity-of-illness assessment appears to be important for critically ill patients with novel coronavirus disease (COVID-19). This study aimed to evaluate the performance of the rapid scoring system on admission of these patients.
METHODS: A total of 138 medical records of critically ill patients with COVID-19 were included in the study. Demographic and clinical characteristics on admission used for calculating Modified Early Warning Score (MEWS) and Rapid Emergency Medicine Score (REMS) and outcomes (survival or death) were collected for each case and extracted for analysis. All patients were divided into two age subgroups (<65 years and ≥65 years). The receiver operating characteristic (ROC) curve analyses were performed for overall patients and both subgroups.
RESULTS: The median [25th quartile, 75th quartile] of MEWS of survivors versus nonsurvivors were 1 [1, 2] and 2 [1, 3] and those of REMS were 5 [2, 6] and 7 [6, 10], respectively. In overall analysis, the area under the ROC curve for the REMS in predicting mortality was 0.833 (95% confidence interval [CI] = 0.737 to 0.928), higher than that of MEWS (0.677, 95% CI = 0.541 to 0.813). An optimal cutoff of REMS (≥6) had a sensitivity of 89.5%, a specificity of 69.8%, a positive predictive value of 39.5%, and a negative predictive value of 96.8%. In the analysis of subgroup of patients aged <65 years, the area under the ROC curve for the REMS in predicting mortality was 0.863 (95% CI = 0.743 to 0.941), higher than that of MEWS (0.603, 95% CI = 0.462 to 0.732).
CONCLUSION: To our knowledge, this study was the first exploration on rapid scoring systems for critically ill patients with COVID-19. The REMS could provide emergency clinicians with an effective adjunct risk stratification tool for critically ill patients with COVID-19, especially for the patients aged <65 years. The effectiveness of REMS for screening these patients is attributed to its high negative predictive value.

The novel coronavirus disease (COVID‐19), which is an infectious disease caused by the newly discovered virus, has quickly become a global threat to health, travel, and commerce. , , Most COVID‐19 patients will experience mild to moderate respiratory or digestive symptom and recover without special therapy. A few COVID‐19 patients can have severe disease that is life‐threatening. The mortality of critically ill patients with COVID‐19 is reported to vary from 11% to 61% and dramatically increases among 65 years old patients and above. , , ,

Reducing mortality of critically ill patients with COVID‐19 needs early medical intervention, which requires emergency physicians to quickly select severe patients from a large number of patients. , , Thus, rapid and effective assessment of these patients is a crucial task in the emergency department (ED). However, this is made particularly difficult because of the lack of medical personnel and patient overcrowding.

In the past decades, many researchers have developed physiologic scoring systems for early detection of high‐risk patients to regulate management of patients in EDs. One of these physiologic scoring systems was the Modified Early Warning Score (MEWS), which includes variables such as heart rate, systolic blood pressure, respiratory rate, body temperature, and consciousness. Although researchers are still trying to improve its accuracy, the prognostic value of this model has been found to be acceptable for application in the ED.

Another recently presented model is the Rapid Emergency Medicine Score (REMS). This model incorporates six variables including the heart rate, blood pressure, respiratory rate, Glasgow Coma Scale (GCS), level of oxygen saturation, and chronological age of patients. The REMS model was initially proposed for predicting mortality in nonsurgical patients admitted to the ED. Since a limited number of simple variables have been included in MEWS and REMS models, assessing and calculating the score based on them is feasible, and they can be easily applied in EDs.

Thus far, both rapid scoring systems have been widely used in the ED. , To our knowledge, no studies have compared the effectiveness of different scoring systems for critically ill patients with COVID‐19. Accordingly, this study aimed to assess and compare the prognostic value of MEWS and REMS models for in‐hospital mortality of critically ill patients with COVID‐19 presenting to the ED.

Methods

Study Design

This study was a retrospective review of electronic patient care reports from an emergency medical team of a Chinese hospital. The team, which was composed of 131 medical members from China International Emergency Medical Team (Sichuan), was deployed to Wuhan from February 7 to independently manage the COVID‐19 ward. The COVID‐19 ward, which contained 80 beds, was built temporarily for COVID‐19 patients.

Study Subjects and Setting

We collected study subjects through a computer‐aided systemic search of the registration system of the COVID‐19 ward from February 7, 2020, to March 7, 2020. We then included 138 patients aged >18 years with the diagnosis of COVID‐19 (critical). The criteria for COVID‐19 diagnosis, classification, and treatment used at the COVID‐19 ward followed the recommendations of the “Diagnosis and Treatment Plan of Novel Coronavirus” (Version 6) issued by the National Health Commission of China. We then divided all the patients into two age subgroups (<65 and ≥65 years) for analysis.

The local institutional review committee approved the study and waived the requirement for informed consent from the study subjects due to the study design. The study complied with the Declaration of Helsinki, and the accessed data were anonymized.

Measurement

Demographic data, clinical manifestations at admission, treatments, and outcomes in the medical reports of COVID‐19 patients included in the study were collected and analyzed. These data included all the factors needed for calculating MEWS and REMS models. Individual scores of MEWS were calculated based on heart rate (beats/min), systolic blood pressure (mm Hg), respiratory rate (breaths/min), body temperature(°C),and consciousness. Likewise, individual scores of REMS were calculated based on mean arterial pressure (mm Hg), pulse rate (beats/min), respiratory rate (breaths/min), oxygen saturation (%), GCS, and patient age (year). Case fatality was defined as death during hospitalization.

Data Analysis

Descriptive data are reported as mean ± standard deviation (SD) for continuous variable of normal distribution, median [25th quartile, 75th quartile] for continuous variable of nonnormal distribution, and percentages for categorical data. Continuous data of normal distribution were tested using the t‐test, continuous data of nonnormal distribution were tested using the Mann‐Whitney U‐test for nonnormal distribution, and categorical variables were compared by either the chi‐square test or Fisher's exact test (when the expected value < 5 in one cell), as appropriate.

We compared variables including demographics, clinical features, laboratory results, and rapid physiologic scoring systems (MEWS and REMS) between survivors and nonsurvivors (MEWS and REMS score systems are shown in Data Supplement S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1111/acem.13992/full). The receiver operating characteristic (ROC) curve analysis was used for the assessment of the discriminatory power of both MEWS and REMS.

The ability of both rapid scoring systems to discriminate between survivors and nonsurvivors was evaluated by calculating the area under the curve (AUC) of the ROC and its 95% confidence interval (95% CI). Delong's test was applied for comparison of AUCs of MEWS and REMS.

The optimal cutoff value for predicting COVID‐19 mortality equals the score with the largest Youden's index for each rapid scoring system. Based on the optimal cutoff value, we calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for assessment of both rapid scoring systems. A critical α value of 0.05 was accepted as statistically significant. Data were entered into IBM Statistical Product and Service Solutions (SPSS) Statistics for Windows, version 20 (IBM Corp., Armonk, NY) and were analyzed by MedCalc version 12.7 (MedCalc Software bvba).

Results

Baseline Characteristics

Excluding 33 missing data, 105 cases were analyzed (Figure 1). In total, the mean age was 60.82 ± 16.32 years, and 54 patients (50.94%) were male. Baseline characteristics are listed in Table 3. In terms of consciousness, all survivors were alert. Only two nonsurvivors had altered consciousness: one was reacting to verbal stimuli, and the other was reacting to pain.

Figure 1

Flow chart of cases exclusion. From all 138 cases, we excluded 15 cases that were missing information on motor response used to calculate GCS, 11 cases were missing information on verbal response used to calculate GCS, and seven cases were missing information on oxygen saturation. Finally, we analyzed 105 cases. GCS = Glasgow Coma Scale.

The characteristics with significant difference, expressed in terms of survivors versus nonsurvivors, were as follows: ages of 57.71 ± 15.34 years versus 75.05 ± 12.94 years, respiratory rate of 20.27 ± 3.27 breaths/min versus 23.21 ± 5.38 breaths/min, systolic blood pressure of 130.79 ± 18.33 mm Hg versus 143.26 ± 24.85 mm Hg, peripheral oxygen saturation of 96.17% ± 3.1% versus 86.53% ± 12.46%, and GCS of 15 [15, 15] versus 15 [15, 15]. The median [25th quartile, 75th quartile] of MEWS of survivors versus nonsurvivors were 1 [1, 2] and 2 [1, 3] and that of REMS were 5 [2, 6] and 7 [6, 10], respectively. In Table 1, we also reported existing underlying diseases between the survivors and nonsurvivors, including diabetes, hypertension, cardiovascular disease, chronic pulmonary disease, cerebral vascular disease, and malignant tumor.

Table 1

Comparison of the Baseline Characteristics of Survivors and Nonsurvivors

Variable	Survivors (n = 86)	Nonsurvivors (n = 19)	p‐value
Male, n (%)	48 (55.81%)	14 (73.68%)	0.152
Female, n (%)	38 (44.19%)	5 (26.32%)
Age (years), mean ± SD	57.71 ± 15.34	75.05 ± 12.94	<0.001*
Prehospital day, mean ± SD	10.24 ± 7.51	8.37 ± 5.78	0.309
PR (beats/min)	84.23 ± 17.48	87.47 ± 13.56	0.450
HR (beats/min)	84.23 ± 17.48	87.47 ± 13.56	0.450
RR (breaths/min)	20.27 ± 3.27	23.21 ± 5.38	0.032*
Temperature (°C)	36.62 ± 0.48	36.71 ± 0.72	0.826
SBP (mm Hg)	130.79 ± 18.33	143.26 ± 24.85	0.014*
DBP (mm Hg)	80.76 ± 11.01	83.95 ± 14.12	0.281
MAP (mm Hg)	97.07 ± 12.35	103.42 ± 15.91	0.057
SpO2 (%)	96.17 ± 3.1	86.53 ± 12.46	0.003*
GCS score, median [25th quantile, 75th quantile]	15[15, 15]	15[15, 15]	0.003*
Underlying disease, n (%)
Diabetes	4 (4.65%)	0 (0.00%)	0.444
Hypertension	22(25.58%)	6(31.58%)	0.593
Cardiovascular disease	5(5.81%)	1(5.26%)	0.925
Chronic pulmonary disease	7(8.14%)	5(26.32%)	0.040*
Cerebral vascular disease	1(1.16%)	3(15.79%)	0.018*
Malignant tumor	5(5.81%)	1(5.26%)	0.703
MEWS, median [25th quantile, 75th quantile]	1 [1, 2]	2 [1, 3]	0.006*
REMS, median [25th quantile, 75th quantile]	5 [2, 6]	7 [6, 10]	<0.001*

PR = pulse rate; HR = heart rate; RR = respiratory rate; SBP = systolic blood pressure; DBP = diastolic blood pressure; MAP = mean arterial pressure; SpO2 = peripheral oxygen saturation; GCS = Glasgow Coma Scale; MEWS = Modified Early Warning Score; REMS = Rapid Emergency Medicine Score.

p < 0.05.

Overall Analysis

Figure 2A depicts the ROC of MEWS and REMS models in predicting in‐hospital mortality. The AUC of MEWS and REMS models in predicting in‐hospital mortality were 0.677 (95% CI = 0.579 to 0.765) and 0.841 (95% CI = 0.757 to 0.905), respectively, and the difference between the two was found to be statistically significant (p = 0.028 < 0.05; Table 2).

Figure 2

ROC of MEWS and REMS models in predicting in‐hospital mortality. (A) The ROC of MEWS and REMS models of overall critically ill patients; (B) the ROC of MEWS and REMS models in the subgroup of persons aged <65 years; (C) the ROC of MEWS and REMS models in the subgroup of persons aged ≥65 years. The AUC of the solid line, which represents REMS, is greater than the AUC of the dotted line, which indicates MEWS. AUC = area under the curve; MEWS = Modified Early Warning Score; REMS = Rapid Emergency Medicine Score; ROC = receiver operating characteristic curve.

Table 2

AUC of MEWS and REMS Models in Predicting In‐hospital Mortality

Models	AUC	95% CI of AUC	△AUC	z statistic	p‐value
Overall (n = 105)
MEWS	0.677	0.579 to 0.765	0.164	2.992	0.028*
REMS	0.841	0.757 to 0.905
Subgroup
Age <65 years (n = 55)
MEWS	0.603	0.462 to 0.732	0.260	2.223	0.026*
REMS	0.863	0.743 to 0.941
Age ≥65 years (n = 50)
MEWS	0.708	0.562 to 0.828	0.0581	0.813	0.416
REMS	0.766	0.625 to 0.874

AUC = area under the curve of the receiver operating characteristic; △AUC = difference between AUCs, MEWS = Modified Early Warning Score; REM = Rapid Emergency Medicine Score.

p < 0.05.

The various cutoff values and corresponding diagnostic performance for differentiating the nonsurvivors from survivors were list in Table 3. According to the best Youden's index, an optimum cutoff value of 2 was used to predict in‐hospital mortality using the MEWS, with sensitivity, specificity, PPV, and NPV of 68.42, 65.12, 30.23, and 90.32%, respectively. Likewise, an optimum cutoff value of 6 was used to predict in‐hospital mortality using the REMS, with sensitivity, specificity, PPV, and NPV of 89.47, 69.77, 39.53, and 96.77%, respectively.

Table 3

Sensitivities and Specificities of MEWS and REMS for Predicting In‐hospital Mortality

Models	cut‐off value	Youden's index	Sen. (%)	Spe. (%)	PPV (%)	NPV (%)
Overall
MEWS	1	0.00	100.00	0.00	18.10	N/A
	2	0.34*	68.42	65.12	30.23	90.32
	3	0.22	36.84	84.88	35.00	85.88
	4	0.02	10.53	91.86	22.22	82.29
	5	0.09	10.53	98.84	66.67	83.33
	6	0.05	5.26	100.00	100.00	82.69
REMS	1	0.19	100.00	18.60	21.35	100.00
	2	0.19	100.00	18.60	21.35	100.00
	3	0.25	94.74	30.23	23.08	96.30
	4	0.37	94.74	41.86	26.47	97.30
	5	0.41	94.74	46.51	28.13	97.56
	6	0.59*	89.47	69.77	39.53	96.77
	7	0.38	57.89	80.23	39.29	89.61
	8	0.39	47.37	91.86	56.25	88.76
	9	0.31	36.84	94.19	58.33	87.10
	10	0.26	31.58	94.19	54.55	86.17
Subgroup
Age <65 years (n = 55)
MEWS	1	0.50*	100.00	49.50	7.27	100.00
	2	0.33	50.00	83.17	10.53	97.67
	3	0.16	25.00	91.09	10.00	96.84
	4	0.20	25.00	95.05	16.67	96.97
	5	0.24	25.00	99.01	50.00	97.09
	6	0.25	25.00	100.00	100.00	97.12
REMS	3	0.51	75.00	76.24	11.11	98.72
	4	0.61	75.00	86.14	17.65	98.86
	5	0.65*	75.00	90.10	23.08	98.91
	6	0.47	50.00	97.03	40.00	98.00
	7	0.24	25.00	99.01	50.00	97.09
Age ≥65 years (n = 50)
MEWS	1	0.61*	100.00	61.11	30.00	100.00
	2	0.59	73.33	85.56	45.83	95.06
	3	0.36	40.00	95.56	60.00	90.53
	4	0.04	6.67	97.78	33.33	86.27
	5	0.07	6.67	100.00	100.00	86.54
REMS	4	0.61	100.00	61.11	30.00	100.00
	5	0.61	100.00	61.11	30.00	100.00
	6	0.76*	100.00	75.56	40.54	100.00
	7	0.50	66.67	83.33	40.00	93.75
	8	0.48	53.33	94.44	61.54	92.39
	9	0.42	46.67	95.56	63.64	91.49

Sen. = sensitivity; Spe. = specificity; PPV = positive predictive value; NPV = negative predictive value; N/A = unable to calculate because the denominator is zero; MEWS = Modified Early Warning Score; REMS = Rapid Emergency Medicine Score.

The largest Youden's index.

The sensitivity and the specificity of REMS model were considerably higher than those of MEWS. Therefore, the value of the REMS model in predicting mortality in COVID‐19 patients is better than that of the MEWS model.

Subgroup Analysis

Figures 2B and 2C depict the ROC of MEWS and REMS models of subgroups in predicting in‐hospital mortality. The AUC of MEWS and REMS models in subgroup of persons aged <65 years were 0.603 (95% CI = 0.462 to 0.732) and 0.863 (95% CI = 0.743 to 0.941), respectively, and the difference between the two was found to be statistically significant (p = 0.026 < 0.05). And the AUC of MEWS and REMS models in the subgroup of persons aged ≥65 years were 0.708 (95% CI = 0.562 to 0.828) and 0.766 (95% CI = 0.625 to 0.874), respectively, and the difference between the two was not statistically significant (p = 0.416> 0.05) (Table 2).

In the subgroup of persons aged <65 years, according to the best Youden's index, an optimum cutoff value of 1 was used to predict in‐hospital mortality using the MEWS, with sensitivity, specificity, PPV, and NPV of 100.00, 49.50, 7.27, and 100.00%, respectively. Likewise, an optimum cutoff value of 5 was used to predict in‐hospital mortality using the REMS, with sensitivity, specificity, PPV, and NPV of 75.00, 90.10, 23.08, and 98.91%, respectively (Table 3).

In the subgroup of persons aged ≥65 years, according to the best Youden's index, an optimum cutoff value of 1 was used to predict in‐hospital mortality using the MEWS, with sensitivity, specificity, PPV, and NPV of 100.00, 61.11, 30.00, and 100.00%, respectively. Likewise, an optimum cutoff value of 6 was used to predict in‐hospital mortality using the REMS, with sensitivity, specificity, PPV, and NPV of 100.00, 75.56, 40.54, and 100.00%, respectively (Table 3).

Discussion

COVID‐19 is a new infectious disease discovered in December 2019. The use of screening methods for critically ill patients could help emergency physicians to respond the COVID‐19 epidemic. In this study, the univariate analysis found a significant difference between survivors and nonsurvivors in terms of age, respiratory rate, systolic blood pressure, SpO2, and GCS. We also reported that the proportion of critically ill patients with COVID‐19 with existing underlying diseases of chronic pulmonary disease and cerebral vascular disease was higher in nonsurvivors than in survivors. Some other studies had explored prognosis factors: Chen et al. analyzed 64 COVID‐19 patients and reported that being elderly and having underlying diseases were related to disease severity. Zhu et al. reported that the COVID‐19 patients with diabetes, chronic obstructive pulmonary disease, tumors, or autoimmune diseases should be considered as potentially severe patients. To date, most studies on prognosis predictors focused on the indices of laboratory examination such as D‐dimer, myoglobin, troponin, lymphocytes, and lactate dehydrogenase. , Further, some studies explored rapid simple assessment for COVID‐19 patients, such as tests for vital signs or consciousness, available in the ED. Nevertheless, the sample sizes of these studies, including ours, were small. Thus, further studies are needed to confirm our findings.

Given various predictors for COVID‐19 patients, a comprehensive scoring system involving multiple factors may enable the more accurate screening of COVID‐19 patients. To our knowledge, no study has assessed the predictive value of a rapid scoring system for in‐hospital mortality of COVID‐19 in the ED. In this study, we analyzed two rapid scoring systems (MEWS and REMS) used to assess in‐hospital mortality in patients with COVID‐19.

The MEWS was a modified version of the Early Warning Score developed in 2000 by Subbe et al. As a bedside tool that can be computed easily in a busy clinical area, MEWS might help identify the need for active intervention in the setting of emergency patients. , The REMS was developed in 2003 by Olsson et al. and was based on a retrospective observational study from 12,006 nonsurgical patients. Olsson et al. , , were able to accurately predict mortality based on six variables acquired easily in the ED. They suggested that the REMS is a simple and powerful tool that adequately predict in‐hospital mortality, in contrast to other existing scores that are not suitable for early use in patients admitted to the ED for a wide range of common nonsurgical disorders.

In our analysis, both MEWS and REMS illustrated acceptable predictive values for in‐hospital mortality as reflected in the ROC curves. In addition, the AUC of REMS was greater than that of MEWS, so we considered the accuracy of REMS higher. The advantages of REMS are its more suitable physiological parameters and the inclusion of four important variables (age, respiratory rate, SpO2, and GCS), allowing stratification of patients with higher accuracy.

In overall analysis, the AUC value of REMS is 0.841, which also indicated that it was ideal for predicting mortality in COVID‐19 patients. A high NPV of 96.77% for REMS enabled emergency physicians to decisively exclude COVID‐19 patients with an REMS score below 6 from the high‐risk group. In the subgroup of persons aged <65 years, the REMS score for predicting in‐hospital mortality is superior to MEWS, while in the subgroup of persons aged ≥65 years, both scores are at the same performance. The results of the subgroup analysis suggest that the advantage of REMS lies in its predictive effect on patients aged <65 years. The critically ill patients aged <65 years with a high REMS score have a poor prognosis.

Although REMS contains more parameters than MEWS, additional information required can be easily determined through medical assessment and point‐of‐care testing. We nevertheless recognize that when REMS parameters cannot be completed in the ED, MEWS, which also has a high NPV (90.32%) for screening, is also a second option for COVID‐19 patients. Although not as good as REMS, the prediction accuracy of MEWS was acceptable.

Limitations

To our knowledge, this is the first study to explore the effectiveness of rapid scoring systems for critically ill patients with COVID‐19 and possessed practical value in the rapid screening for them. However, its limitations should be considered. First, this study is limited by a small number of samples. More studies that include a large number of COVID‐19 patients who fulfill these criteria would aid in providing a more effective assessment of REMS and MEWS. Second, in our study, the prognostic values of the scoring systems among the patient subgroups with different ages were not analyzed because of the limited sample size. Third, many rapid scoring tools were widely used in the ED. However, in this retrospective study from medical records, we could only collect limited information to calculate REMS and MEWS. The effectiveness of other rapid scoring systems should be explored in the future. Fourth, out‐of‐hospital treatment and care were neglected because of unavailable information. Finally, this single‐center study was also affected by selective bias in COVID‐19 patients of the special COVID‐19 ward. Further multicenter studies to prospectively validate the use of REMS and MEWS in critically ill patients with COVID‐19 are also required.

Conclusion

Both the Modified Early Warning Score and the Rapid Emergency Medicine Score demonstrated acceptable predictive values for the in‐hospital mortality of critically ill patients with COVID‐19. If each parameter could be collected in a busy ED, the Rapid Emergency Medicine Score might be more accurate than the Modified Early Warning Score for critically ill patients with COVID‐19, especially for the patients aged <65 years. The Rapid Emergency Medicine Score could provide emergency clinicians with an effective adjunct risk stratification tool for such patients, because of its high negative predictive value.

The authors thank all the physicians and nurses who worked in the COVID‐19 ward for helping us collect the data, and we thank Editage (www.editage.cn) for English language editing.

Data Supplement S1. Supplemental material.

Click here for additional data file.