A New Rapid Approach for Predicting Death in Coronavirus Patients: The Development and Validation of the COVID-19 Risk-Score in Fars Province (CRSF).

BACKGROUND: Patients who are identified to be at a higher risk of mortality from COVID-19 should receive better treatment and monitoring. This study aimed to propose a simple yet accurate risk assessment tool to help decision-making in the management of the COVID-19 pandemic.
METHODS: From Jul to Nov 2020, 5454 patients from Fars Province, Iran, diagnosed with COVID-19 were enrolled. A multiple logistic regression model was trained on one dataset (training set: n=4183) and its prediction performance was assessed on another dataset (testing set: n=1271). This model was utilized to develop the COVID-19 risk-score in Fars (CRSF).
RESULTS: Five final independent risk factors including gender (male: OR=1.37), age (60-80: OR=2.67 and >80: OR=3.91), SpO 2 (≤85%: OR=7.02), underlying diseases (yes: OR=1.25), and pulse rate (<60: OR=2.01 and >120: OR=1.60) were significantly associated with in-hospital mortality. The CRSF formula was obtained using the estimated regression coefficient values of the aforementioned factors. The point values for the risk factors varied from 2 to 19 and the total CRSF varied from 0 to 45. The ROC analysis showed that the CRSF values of ≥15 (high-risk patients) had a specificity of 73.5%, sensitivity of 76.5%, positive predictive value of 23.2%, and negative predictive value (NPV) of 96.8% for the prediction of death (AUC=0.824, P<0.0001).
CONCLUSION: This simple CRSF system, which has a high NPV, can be useful for predicting the risk of mortality in COVID-19 patients. It can also be used as a disease severity indicator to determine triage level for hospitalization.

Introduction

An acute respiratory infection, the coronavirus disease 2019 (COVID-19) was first identified in China in Dec 2019, rapidly spread to many countries, and became a global pandemic (1, 2). As of Dec 30, 2020, more than 80 million individuals have been confirmed to be COVID–19 positive worldwide (3). The overall mortality rate of COVID-19 disease is variable ranging from 0.7% to 10.8% (4, 5). Up to this date, the Iranian Ministry of Health has confirmed 1,218,752 positive COVID-19 cases and 55,095 deaths (death rate=5.33%) (6).

Although the majority of infected individuals have mild respiratory symptoms, the clinical deterioration rate is very fast and the death rate increases rapidly in severe cases requiring ventilation or intensive care unit (ICU) admission. Such problems as limited ICU beds, ventilators, and medical resources as well as medical staff shortages will be intensified over time. To decrease the mortality rate, more efficient programs and optimal allocation of finite medical resources are essential. In particular, the outcomes of COVID-19 may be affected if treatment is delayed. It can also affect the performance of time-sensitive operations. Therefore, to decrease the mortality rate, suitable hospitalization and risk-recognition strategies are required.

Important laboratory abnormalities (such as leukopenia and lymphopenia), individuals over the age of 60, male gender, and the existence of comorbidities are the characteristics of the severity of the COVID-19 infection (7–10).

Since most COVID-19 patients experience mild to moderate respiratory illness and recover without a particular treatment, the early medical assessment and proper management of the severity of COVID-19 appears to be essential and significant for severe and critically ill patients (11). In addition, reducing the risk of death in these patients needs rapid medical attention and intervention. This in turn requires sufficient treatment staff (especially emergency physicians) to pick out these patients quickly from a large number of positive cases (12). Thus, the early and effective evaluation of severe COVID-19 patients is a crucial and important task in the Emergency Medicine Department (EMD).

To the best of our knowledge, for the early detection of high-risk COVID-19 patients and managing them in EMDs, two previous scoring systems (i.e. MEWS: Modified Early Warning Score and REMS: Rapid Emergency Medicine Score) have been used until now (13, 14). The training dataset for feeding these physiological scoring systems are based on the general population, not those suffering from COVID-19 (15). Therefore, it seems vital to create a new scoring system based on the data of COVID-19 patients. This study aimed to propose a new scoring tool (named COVID-19 Risk-Score in Fars (CRSF)) for the admission of COVID-19 patients and classify them into different triage levels.

Methods

Study Design and Setting

This multicenter retrospective cross-sectional study (from Jul 22, 2020, to Nov 5, 2020) was conducted on all inpatients who were referred to one of the thirty-four health medical centers (affiliated with Shiraz University of Medical Sciences (SUMS)) in Fars Province, southern Iran. The inclusion criteria were all inpatients with known COVID-19 symptoms (i.e. cough, dyspnea, and fever), confirmation of the disease by the real-time polymerase chain reaction (RT-PCR) test, and high-resolution computed tomography (CT) scan of the lungs. The patients with missing data and unknown last status (death or discharge from the hospital) were excluded from the study.

The current study was performed following the Declaration of Helsinki and was approved by the Vice-Chancellor of Research and Technology as well as the Ethics Committee of SUMS (IR.SUMS.MED.REC.1399.516).

Data Collection and Processing

The patients’ data such as age, gender, underlying diseases, peripheral oxygen saturation (SpO 2 ), pulse rate (PR), respiratory rate (RR), temperature, diastolic blood pressure (DBP), and systolic blood pressure (SBP) were extracted from the electronic registry of SUMS.

Statistical Analysis

In order to develop and validate the final model, the patients’ data were divided into the training set (n=4183 (75%)) and the testing set (n=1271 (25%)), respectively, using a simple randomized sampling method. A multiple logistic regression model was fed to one dataset (the training set) and its prediction performance was tested on another (the testing set). This model was used to develop the CRSF. A continuous factor was categorized using the LOESS smoothing procedure (16).

First, a scoring system was created for the above-mentioned simple formula using the training data. Then, some points were assigned to each factor according to the magnitude of its estimated regression coefficient (17). A total CRSF for each patient was computed as the sum of the points for each factor.

A receiver-operating characteristic (ROC) curve and the area under the ROC curve (AUC) were produced for the assessment of the specificity and sensitivity of the scores for all the surviving and non-surviving COVID-19 patients. The cut-off score was determined by the ROC curve analysis in a way as to maximize the separation of the high-risk and low-risk groups. In other words, the cut-off score maximized the distance between the diagonal line and the ROC curve. The statistical analyses were performed in R software (ver. 4.0.2).

Results

Out of 5,606 inpatients from thirty-four health centers affiliated with SUMS, 152 cases were excluded because of their missing data and unknown last status. Therefore, the final sample size of the current study was 5454 patients with confirmed COVID-19. Out of them, 2,888 (53%) were men with the mean (±SD) age of 54.5 (19.3) yr and 2,566 (47%) were women with the mean (±SD) age of 55.1 (18.5) yr. The overall in-hospital mortality rate was 526 out of 5,454 (9.65%).

Figure 1 depicts the LOESS smoothing curves for the cut-off points of the different features of the study. For instance, the LOESS analysis showed a cut-off value of 85% for categorizing SpO 2 (i.e. ≤85% and >85%). The details of the optimal cut-off points for all the factors and results of the univariate logistic regression analyses on the training set are illustrated in Table 1. In the current study, no significant relation was observed between the temperature levels and the mortality outcomes (P=0.480). Therefore, temperature was not used in making the final risk score. However, there were statistically significant differences between the survivors and non-survivors regarding gender, age, underlying diseases, SpO 2 , PR, RR, SBP, and DBP (all P<0.05).

Fig. 1:

LOESS smoothing curves plotting the probability of death against SpO 2 , pulse rate, respiratory rate, and temperature

Table 1:

Comparison of the baseline features of non-survivors and survivors with univariate logistic analysis (n=4183)

Features		Survivors	Non-survivors	Univariate logistic regression
		No. (%)	No. (%)	OR	(95% CI)	P-value
Gender	Women	1813 (43.3)	166 (4.0)	1 (Reference)	-	-
	Men	1975 (47.2)	229 (5.5)	1.30	1.03-1.56	0.027
Age (year)	<60	2271 (54.3)	98 (2.3)	1 (Reference)	-	-
	60–80	1203 (28.8)	203 (4.9)	3.90	3.04–5.03	<0.001
	>80	314 (7.5)	94 (2.2)	6.90	5.11–9.43	<0.001
Underlying diseases	No	2029 (48.5)	159 (3.8)	1 (Reference)	-	-
	Yes	1759 (42.1)	236 (5.6)	1.70	1.39–2.11	<0.001
SpO2 (%)	≤85	786 (18.8)	282 (6.7)	9.50	7.56–12.02	<0.001
	>85	3002 (71.8)	113 (2.7)	1 (Reference)	-	-
PR (beats/min)	<61	56 (1.3)	17 (0.4)	3.20	1.85–5.60	<0.001
	61–120	3356 (80.2)	317 (7.6)	1 (Reference)	-	-
	>120	376 (9.0)	61 (1.5)	1.72	1.28–2.30	<0.001
RR (breaths/min)	<20	1749 (41.8)	146 (3.5)	1 (Reference)	-	-
	≥20	2039 (48.7)	249 (6.0)	1.45	1.18–1.81	<0.001
Temperature (°C)	<37.4	2833 (67.7)	289 (6.9)	1 (Reference)	-	-
	≥37.4	955 (22.8)	106 (2.5)	1.09	0.86–1.38	0.480
Features		Mean (±SD)	Mean (±SD)	OR	(95% CI)	P-value
DBP (mm Hg)		78.8 (13.8)	74.1 (15.8)	0.976	0.968–0.983	<0.001
SBP (mm Hg)		126.7 (21.2)	123.3 (24.2)	0.992	0.987–0.997	0.002

Note: Bold numbers indicate statistically significant with P-value <0.05. CI: confidence interval; DBP: diastolic blood pressure; No.: number; OR: odds ratio; PR: pulse rate; RR: respiratory rate; SpO 2 : peripheral oxygen saturation; SD: standard deviation; SBP: systolic blood pressure.

The multiple logistic regression analysis indicated that five final independent risk factors including gender, age, underlying diseases, SpO 2 , and PR were significantly associated with in-hospital mortality. According to the results of the multiple logistic regression analysis as well as the cut-off points of the LOESS smoothing curve analyses, the male gender, age of ≥60 yr, having underlying diseases, SpO 2 ≤85%, and PR<61 beats/min or PR>120 beats/min posed the highest risk of death. The adequacy of the fitted model was properly evaluated and confirmed by narrow confidence intervals for its estimated parameters (Table 2).

Table 2:

The COVID-19 risk-score in Fars based on the multiple logistic regression model (n=4183)

Factors		Coefficient	OR	(95% CI)	P-value	Point value
Gender	Women	-	1 (Reference)	-	-	0
	Men	0.312	1.37	1.08-1.72	0.008	3
Age (yr)	<60	-	1 (Reference)	-	-	0
	60–80	0.984	2.67	2.04–3.51	<0.001	10
	>80	1.364	3.91	2.80–5.45	<0.001	14
Underlying diseases	No	-	1 (Reference)	-	-	0
	Yes	0.219	1.25	0.99–1.57	0.064	2
SpO2 (%)	≤85	1.949	7.02	5.52–8.93	<0.001	19
	>85	-	1.00	-	-	0
PR (beats/min)	<61	0.699	2.01	1.08–3.74	0.027	7
	61–120	-	1 (Reference)	-	-	0
	>120	0.472	1.60	1.16–2.22	0.005	5

Note: P-values of ≤0.05 were considered significant and P-values of less than 0.07 were regarded as marginally significant. Range of total score, 0–45. CI= confidence interval; OR= odds ratio; PR= pulse rate; SpO 2 = peripheral oxygen saturation.

For ease of interpretation and constructing the CRSF formula, the estimated regression coefficients of gender, age (60≤age≤80 or age>80), underlying diseases, SpO 2 , and PR (<61 or >120) were multiplied by a factor of 10 and rounded (Table 2). The following formula can be written for the CRSF. Suppose that I(x) is an indicator function defined as follows: where A is an arbitrary set. The mortality risk for COVID-19 inpatients can be formulated via the following equation:

In the CRSF formula, for example, I (SpO 2 ≤85%) is equal to 1 if SpO 2 ≤85%. Otherwise, it is zero. In general, the male gender, 60≤age≤80, age>80, having underlying diseases, SpO 2 ≤85, PR<61, and PR>120 were given the scores of 3, 10, 14, 2, 19, 7, and 5, respectively.

According to the CRSF (formula II), the risk score value for a COVID-19 patient is in the range of 0–45. Moreover, the lowest risk of death or the minimum CRSF (i.e. zero) is attributed to a female patient without underlying diseases, age of <60 yr, SpO 2 of ≤85%, and PR of between 61 and 120 beats per minute.

After developing the risk score using the training set, the analysis results of the ROC curve demonstrated that patients with the score of ≥15 had a high risk of death (specificity=73.5%, sensitivity=76.5%, and AUC=0.824, P<0.0001) (Table 3 and Fig. 2).

Table 3:

AUC, sensitivities, specificities, PPV, NPV, PLR, and NLR of COVID-19 risk score in Fars (CRSF) for predicting in-hospital mortality for training and validation sets

Measure	Data set
	Training set	Testing set
Optimal cut-off point	15	15
AUC (95% CI)	0.824 (0.812–0.836)	0.812 (0.790–0.833)
Sensitivity (95% CI)	76.5% (72.0–80.6%)	80.2% (72.3–86.6%)
Specificity (95% CI)	73.5% (72.1–74.9%)	71.6% (68.9–74.2%)
PPV (95% CI)	23.2% (20.9–25.4%)	24.5% (20.4–28.5%)
NPV (95% CI)	96.8% (96.1–97.4%)	96.9% (95.7–98.1%)
PLR (95% CI)	2.89 (2.68–3.12)	2.82 (2.49–3.20)
NLR (95% CI)	0.32 (0.27–0.38)	0.28 (0.20–0.39)
Youden’s index	0.50	0.52
P-value (area=0.5)	<0.0001	<0.0001

Note: AUC: area under the ROC curve; CI: confidence interval; PLR: positive likelihood ratio; PPV: positive predictive value; NLR: negative likelihood ratio; NPV: negative predictive value

Fig. 2:

Receiver operating characteristic (ROC) curves for COVID-19 risk score’s prediction of in-hospital mortality for training (solid line: AUC=0.824 (95% CI: 0.812–0.836, P<0.0001)) and validation (dashed line: AUC=0.812 (95% CI: 0.790–0.833, P<0.0001)) sets

For the validation analysis, 1271 patients with COVID-19 were recruited. The predictive performance results in the validation analysis were broadly similar to those in the training analysis. The developed risk score (CRSF) predicted the high-risk COVID-19 patients in the testing set as well with an AUC of 0.812 (P<0.0001). The cut-off point of 15 was also obtained in the testing set. Moreover, at this cut-off point, specificity, sensitivity, positive predictive value (PPV), negative predictive value (NPV), PLR, and NLR were 71.6%, 80.2%, 24.5%, 96.9%, 2.82%, and 0.28%, respectively (Table 3 and Fig. 2).

Table 4 shows how the triage level is determined according to the CRSF category. Moreover, there is a good performance of the CRSF system in identifying the high-risk patients. These results can be used for determining triage levels and allocating the resources in the EMDs. As shown in this table, an increase in the CRSF value was associated with an increased risk of death. This means that the patient should be allocated to higher triage levels (the lower the number, the higher the triage level). For example, when a COVID-19 patient is admitted and his/her calculated CRSF is equal to 37, allocating him or her to level 2 triage may be a good option. The death rate increases from 1.45% in the group with the score of ≤9 to 53.57% in the group with the score of 40–45 (Table 4).

Table 4:

Determining the triage level according to the categories of the COVID-19 risk score in Fars (CRSF)

	COVID-19 risk score in Fars
	≤9	10–19	20–29	30–39	40–45
Training set	1.45% (28/1936)	7.81% (93/1192)	16.29% (72/441)	31.91% (187/586)	53.57% (15/28)
Testing set	1.90% (11/579)	9.04% (31/343)	14.07% (19/135)	33.00% (67/203)	27.30% (3/11)
Allocated triage level	5	4	3	2	1

: The lower the number, the higher the triage level

Discussion

Using appropriate screening procedures for patients with COVID-19 could assist emergency physicians in classifying high-risk patients. To the best of our knowledge, the present study was the first one to develop a new rapid scoring system for predicting high-risk COVID-19 patients at admission according to the Iranian population. This scoring system was trained on the COVID-19 dataset and was incorporated the simple and available characteristics of the patients at admission. Five inexpensive and readily available risk factors (gender, age, SpO 2 , underlying diseases, and PR) were finally included in the CRSF formula, which had a significant performance in the prediction of death.

The CRSF can help nurses and clinicians make suitable decisions about the triage of patients with COVID-19 through identifying low- or high-risk cases with poor prognosis. This risk score determines five triage levels with high accuracy. It also helps to recognize very low-risk patients (i.e. those with the CRSF of ≤9) who might be immediately discharged from the hospital. Moreover, the proposed scoring system suggests that while low-risk COVID-19 patients may be assigned safely to low-intensity care units (level 4 triage), higher intensity wards (triage levels of 1 and 2) must be alerted of the high-risk cases during triage.

A few researches have explored the simple evaluation of patients with COVID-19 in the EMDs based on testing the available vital signs. Unlike that of the present study, the sample sizes of these surveys were relatively very small (13, 14, 18, 19). Moreover, the scoring system (CRSF) of the present study has been presented for the first time and has been trained according to a large number of admitted COVID-19 positive patients during the virus outbreak from Jul to Nov 2020 (n=5454). Testing set comprising 1271 COVID-19 patients was used to calibrate the developed CRSF.

Recently, Hu et al. compared two scoring systems (REMS and MEWS) to evaluate the mortality rate of COVID-19 patients in the EMDs (18). Both the REMS and MEWS systems had an acceptable predictive performance for in-hospital mortality. However, the AUC of the REMS method was higher than that of the MEWS, demonstrating the higher accuracy of the REMS system for COVID-19 patients. An advantage of the REMS system is that it considers four important risk factors including the patient’s age, SpO 2 , RR, and Glasgow Coma Scale (GCS) in the classification step. The study of Hu et al. had two major disadvantages: a) only a small number of patients (n=105) were used to build the final risk score formula which calls into question the generalizability of the results, b) the REMS and the MEWS scoring systems were not originally trained for predicting death in COVID-19 patients (13, 14, 18). A clinical risk score was developed to predict in-hospital mortality in COVID-19 patients. Five hundred and sixteen patients (≥18 yr) were enrolled in their study and their data were analyzed using the Cox-adjusted regression analysis. They also adjusted six independent risk factors including age, RR, number of chronic diseases, PaO 2 /FiO 2 , serum creatinine, and platelet count to build their final risk score. Using the Kaplan-Meier analysis, their clinical risk score stratified the patients into the three categories of low, intermediate, and high risk (19).

The findings of the current study showed that the AUC value of CRSF is 0.824. This demonstrates that it is a suitable tool for predicting mortality in COVID-19 patients. In addition, the high NPV of 97% for the CRSF system enables the emergency physicians to separate the patients with a CRSF value of greater than or equal to 15 from the low-risk ones. Furthermore, the obtained NLR (0.32; 95% CI: 0.27–0.38) of less than 0.5 means the lower probability of a severe COVID-19 disease in the patients with a lower CRSF. The results of the present research were consistent with that of Hu et al. (18) in which the REMS had a high NPV value (96.8%).

In this study, one out of ten patients died and the patients’ SpO 2 and age were the potential predictors of an adverse outcome. The odds of death were about seven times higher in patients with the SpO 2 of ≤85% compared with those with the SpO 2 of >85%. Moreover, compared with COVID-19 patients younger than 60 yr of age, the odds of death were approximately three and four times higher in patients with 60–80 and >80 yr of age, respectively. Such a significant association between death and older age has been seen in previous surveys on COVID-19, although with a less rapid rise in age-specific mortality rates (20). This difference could be attributed to the lower median age of those studies and to the fact that the current study reported a wider age range (1–104 yr) with one-tenth of the study population above the age of 80 (18, 20, 21).

Although the strengths of the current study are substantial because it had a large sample size conducted in several centers, it had several limitations. Some vital parameters such as GCS, found to be of prognostic relevance in other studies (18, 22), were not gathered for most of the cases in our study possibly due to the different severity degrees. In accordance with our aim, we only considered the vital parameters available soon after admission.

Conclusion

The CRSF had an acceptable performance for predicting the in-hospital mortality of COVID-19 patients and could be a promising system for risk classification among these patients. This simple rapid scoring system might be an appropriate disease severity indicator to determine triage levels for hospitalization. Moreover, the high NPV of CRSF makes it a useful additional instrument for emergency clinicians to classify the mortality risk in COVID-19 patients. COVID-19 patients who achieve the highest CRSF values be diligently observed, treated, and followed up. It is also recommended that future studies consider such risk factors as GCS and the time interval between symptom onset and admission in making future scoring systems besides the risk factors mentioned in the current study.

Ethical considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.