A risk score based on baseline risk factors for predicting mortality in COVID-19 patients.

BACKGROUND: To develop a sensitive and clinically applicable risk assessment tool identifying coronavirus disease 2019 (COVID-19) patients with a high risk of mortality at hospital admission. This model would assist frontline clinicians in optimizing medical treatment with limited resources.
METHODS: 6415 patients from seven hospitals in Wuhan city were assigned to the training and testing cohorts. A total of 6351 patients from another three hospitals in Wuhan, 2169 patients from outside of Wuhan, and 553 patients from Milan, Italy were assigned to three independent validation cohorts. A total of 64 candidate clinical variables at hospital admission were analyzed by random forest and least absolute shrinkage and selection operator (LASSO) analyses.
RESULTS: Eight factors, namely, Oxygen saturation, blood Urea nitrogen, Respiratory rate, admission before the date the national Maximum number of daily new cases was reached, Age, Procalcitonin, C-reactive protein (CRP), and absolute Neutrophil counts, were identified as having significant associations with mortality in COVID-19 patients. A composite score based on these eight risk factors, termed the OURMAPCN-score, predicted the risk of mortality among the COVID-19 patients, with a C-statistic of 0.92 (95% confidence interval [CI] 0.90-0.93). The hazard ratio for all-cause mortality between patients with OURMAPCN-score >11 compared with those with scores ≤ 11 was 18.18 (95% CI 13.93-23.71; p < .0001). The predictive performance, specificity, and sensitivity of the score were validated in three independent cohorts.
CONCLUSIONS: The OURMAPCN score is a risk assessment tool to determine the mortality rate in COVID-19 patients based on a limited number of baseline parameters. This tool can assist physicians in optimizing the clinical management of COVID-19 patients with limited hospital resources.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has imposed a significant burden on healthcare systems worldwide, particularly in major hotspot cities,. While the spread of infections appears to be abating in some countries, a growing COVID-19 crisis is still emerging in the developing world, where the medical resources and infrastructure are far less advanced and can be easily overwhelmed. There is an urgent need to develop readily applicable triage criteria to identify COVID-19 patients with a poor prognosis at admission to enable the allocation of the limited medical resources to those who most need them,. Currently, the disease severity and the risk of death in patients with COVID-19 are often graded with the use of either a single clinical variable, such as lymphopenia or hypoxemia, or by doctors’ empirical evaluations. Identifying COVID-19 patients at high risk of mortality based on a sensitive and quantitative risk assessment tool would be valuable for optimizing care and reducing mortality. Recently, ISARIC4C investigators have developed a 4 C mortality score stratify patients admitted to hospital with COVID-19 into the different risk of mortality based on the European population. However, COVID-19 is still widely spreading in Asian countries.

Here, we report the development and validation of a risk assessment tool using readily accessible clinical variables at baseline based on an observational outcome study from a large cohort of COVID-19 in-hospital patients in Hubei Province, China. An integrated risk score (OURMAPCN score) was derived from eight risk factors, namely, Oxygen saturation, blood Urea nitrogen (BUN), Respiratory rate, admission before the date the first national Maximum number of daily new cases was reached, Age 60 and above, Procalcitonin, C-reactive protein (CRP) and absolute Neutrophil counts. We showed that a higher OURMAPCN score predicted a higher mortality rate in COVID-19 patients with adequate specificity and sensitivity across the training, testing, and three different validation cohorts, including two from China and one from Italy. Therefore, the OURMAPCN score has satisfactory performance with regard to predicting the risk of mortality across different clinical cohorts. A website dedicated to the OURMAPCN score (http://compute.covid-ourmap.cn:8888/comput-OURMAPCN) will support frontline physicians performing risk stratification among COVID-19 patients. We propose that the application of the OURMAPCN score could assist physicians in accurately assessing the mortality risk in COVID-19 patients at admission to optimize management options, particularly in areas with limited resources in Asian countries.

Methods

Inpatient cohorts and study procedure

We performed a retrospective observational study with a total of 15,488 inpatients with confirmed COVID-19 in Hubei Province, China and Milan, Italy. As COVID-19 cases in China were concentrated in January to March in 2020, and the number of daily new cases was very limited after March, we included COVID-19 cases mainly admitted to hospitals from January to March, 2020 in Hubei cohort.

For the training and testing cohorts, a total of 6415 patients with confirmed COVID-19 who were admitted between 1 January 2020, and 20 March 2020, were consecutively included in the study from seven COVID-19 designated hospitals in Wuhan city. The final date of follow-up for outcome determination was 8 April 2020. A total of 611 patients remaining in the hospital at the end of follow-up were treated as censor in Cox models. No exclusion criterion was applied in this study. A total of 462 patients died out of the 6415 patients during hospitalization. Among these patients, 70% (4492) were randomly assigned to the training cohort, and the remaining 30% (1923) patients were assigned to the testing cohort (Figure 1 and Supplemental Table 1).

Figure 1.

Flowchart for patient selection and distribution of the training, the test and the validation cohorts. FUD indicates the last of follow-up date.

There were 6351 patients with COVID-19 who were admitted between 1 January 2020 and 17 April 2020, in another three designated hospitals in Wuhan city; these patients were consecutively enrolled in the study and were assigned to validation cohort 1 (Figure 1 and Supplemental Table 1). The final date of the follow-up was 26 April 2020. A total of 257 patients remaining in the hospital at the end of follow-up were treated as censor in Cox models. No exclusion criterion was applied in this study. A total of 587 patients died out of the 6351 patients during hospitalization.

A total of 2169 patients with COVID-19 from eleven hospitals in Hubei Province outside of Wuhan city were designated as validation cohort 2 (Figure 1 and Supplemental Table 1). The inclusion of this subset of patients started from 1 January 2020, and ended on 9 March 2020; patients were enrolled consecutively. The last follow-up date was 27 March 2020. 18 patients remaining in the hospital at the end of follow-up were treated as censor in Cox models. No exclusion criterion was applied to the study. A total of 132 patients died during hospitalization out of the 2169 patients.

The Italian cohort (validation cohort 3) initially enrolled 553 patients with confirmed COVID-19 admitted between 12 February 2020, and 12 April 2020, in Humanitas Research Hospital in Milan, Italy (Figure 1). The patients were enrolled consecutively. The last follow-up date was 30 April. At the end of the study, 24 patients remained in the hospital and were treated as censor in Cox models. A total of 124 patients died during hospitalization out of the 553 patients.

COVID-19 was diagnosed by clinical manifestations, chest CT, or real-time RT-PCR according to the World Health Organization (WHO) interim guidance and/or the New Coronavirus Pneumonia Prevention and Control Program (5th edition) published by the National Health Commission of China,. The demographics, clinical characteristics, medical history, laboratory tests, radiological reports, therapeutic intervention, and outcome data were obtained from patients’ electronic medical records. All variables used for establishing the predictive models were based on the measurements at admission and not at another time point during hospitalization in the training and test cohorts. Measurements at admission were also used in all validation cohorts. Patients transferred between hospitals were not included in our study. The ethics committee of each hospital approved the study protocols. The need to obtain informed consent from the patients was waived by each ethics committee.

Preparation of candidate variables

We followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement for reporting multivariable prediction model development and validation. We identified candidate variables for potential inclusion in our models from a review of the literature and identified clinical variables available at hospital admission in COVID-19 patients, including patients’ demographics, clinical presentation, medical history, laboratory tests and radiological examinations, outcomes, admission date and hospital sites. All major software and code used to analyze these datasets are referenced in Supplemental Table 2.

Initially, a total of 162 variables on admission were collected from 6415 patients hospitalized with COVID-19 in Wuhan. The variables with a large proportion of missing (>50%) or with a correlation of 0.9 or higher were not included for variable selection process. Categorical variables with small categories (less than 5%) were either combined categories or removed. Continuous variables from different institutions were scaled by site and gender and categorical variables were categorized with/without their intrinsic ordering. Therefore, the variables for the risk prediction model were selected from a total of 64 candidate clinical parameters through a two-round process (Figure 2). Before the variable selection process, this Wuhan cohort was randomly split at a ratio of 70%/30% into the training (4492) and testing (1923) cohorts, respectively, and multiple imputations were used to account for missing data in these two separate data sets.

Figure 2.

Flowchart for variable selection. The random forest algorithm was applied to identify variables that may influence prediction among 64 clinical variables collected on admission. 21 variables were derived from the random forest analysis. For better clinical applicability, the 21 variables were dichotomized into another 34 risk factors for a Least Absolute Shrinkage and Selection Operator (LASSO) regression analysis. A total of eight risk factors at admission were extracted.

Variable selection using random forest

In the first-round of variable selection, a random forest model was applied in the training dataset. This machine learning method works on both categorical and numerical variables on any scale, obviating the need for the conversion of features or the normalization,. The random forest model is also robust with regard to controlling potential overfitting and is adequately parsimonious in a large dataset. To avoid overfitting, a pre-pruning algorithm was set for each node to stop the process if all variables belonged to the same class or if all the variable values were the same,. Additionally, the number of tree was set at 500, and the number of variables randomly sampled at each split was optimized with respect to Out-of-Bag error estimates (mtry = 6). Then, variables were dropped one by one with the least relevant variable until the mean decreased accuracy larger than 0.001. The selected variables were ranked by variable importance measures (VIMs) to identify those that would most significantly influence the prediction,.

Variables dichotomization and candidate risk factors selection using LASSO model

To improve the clinical utility, all variables selected in the random forest model were dichotomized. The abnormal cutoff for dichotomization was applied according to the reference range in each hospital (Supplemental Table 3). The date of the first peak in the relevant country’s daily number of new cases was obtained from the WHO and worldometers.info (Supplemental Table 4). We referred to the dichotomized variables as risk factors.

To further select candidate risk factors in the second step, a Least Absolute Shrinkage and Selection Operator (LASSO) regression was applied to minimize the potential collinearity of variables and overfitting (Figure 2). The glmnet package in R was applied to efficiently search for sparse solutions and to optimize the solutions using a 10-fold cross-validation schema over the training dataset and in the study, λ (=0.013) is chosen as the largest lambda at which the mean squared error (MSE) is within one standard error of the minimal MSE by 10 times cross validation. A final LASSO model was selected from the most regularized model which had a within one standard error of the minimum (Figure 2).

The OURMAPCN score derivation

In the score building stage, a multivariate Cox hazard regression analysis was employed for the final list of risk factors from the second round of selection. The assignment of points to risk factors was based on a rounded number that was approximately four times of the coefficient in the Cox model with the maximal value set at five.

Model validation and calibration

The bootstrapped C-statistic was applied to validate the performance of the Cox hazard model as well as calibration plot. Separate C-statistics were analyzed for OURMAPCN risk score in the training, test, and validation groups. Additionally, datasets from all groups were split equally into several groups. For a dataset with fewer events than the controls, we created balanced splits of the data in all groups by maintaining the outcome structure for the training and testing datasets with a 70%/30% ratio in total of cases.

Missing data imputation and cross validation

Because multivariate analysis requires a complete set of variables for each patient, missing data from one or more noninvasive tests were imputed in the study. Multiple imputations were implemented to handle missing data by the random forest package “missForest” algorithm, which is a highly accurate method for lab data,,. Mixed types of data (continuous and categorical) were imputed in a parallel process. All variables entering the random forest model were included in the imputation models. Missing patterns were compared across age and sex to ensure that missing data were not correlated with basic personal characteristics. The variables with a large proportion of missing (>50%) were excluded from the random forest model variable selection process. The MissForest algorithm was applied to the missing parameters to estimate the imputation error as follows: 1.30 for the normalized root mean squared error and 5.00% for false classification. A bootstrapped cross validation process was applied to the training data by artificially and randomly introducing 10% more missing data, and after 10 iterations, MissForest yielded 4.21% (IQR 3.60%-4.90%) differences in the continuous parameters and 0.17% (IQR 0.15–0.18%) differences in the categorical parameters when the two datasets with or without artificially introduced missing parameters were compared. In brief, the levels of missing data for the selected parameters in final score building were as follows: 0% for age, 0% for admission date, 5.48% for respiratory rate, 15.05% for absolute neutrophil count, 15.76% for BUN, 18.14% for oxygen saturation, 31.97% for CRP, and 30.00% for procalcitonin.

Sensitivity analyses

A sensitivity analysis was performed for all the variable/factor selection processes on the complete cases without imputation. We also conducted a sensitivity analysis using univariate general linear model (GLM) followed by pair-matching comparison and random forest model in step 1, and GLM stepwise backward model in step 2, to select candidate variables. A multivariate Cox proportional hazard model was used to calculate the risk score in this sensitivity analysis.

Results

Patient characteristics in the training and testing cohorts

To determine the risk factors significantly associated with in-hospital mortality, a total of 6415 confirmed COVID-19 patients from seven hospitals in the city of Wuhan, Hubei Province, were included and assigned to the training and test cohorts at a 70%/30% ratio (see details in Section “Methods” and Figure 1). Their baseline characteristics are listed in Table 1. The median age of the participants was 59 years (interquartile range [IQR], 46–68), the median respiratory rate was 20 (IQR, 18–21), the median oxygen saturation (SpO2) was 98% (IQR, 96–98) and the median follow-up duration was 15 days (IQR, 10–23). A total of 1208 (25.5%) patients had elevated procalcitonin levels, and 2329 (49.5%) patients had increased CRP levels. A total of 535 (9.6%) patients showed a higher level of BUN, and 819 (14.5%) patients had an elevated neutrophil count. A total of 3836 (59.8%) patients were admitted to the hospital before 12 February (the date of the peak number of daily new cases in China). A total of 462 patients died in the study period. The differences in baseline parameters between the patients who survived and the patients who died in the training and testing cohort are shown in Table 1.

Table 1.

Patients’ characteristics of the training and test cohorts.

Variables	All (6415)	Survived (5342)	Died (462)	p Valued
Clinical characteristics on admission
Median age (IQR)—yr	59 (46–68)	57 (45–66)	71 (63–80)	<.001
Male sex—no./total no. (%)	3028/6415 (47.2)	2442/5342 (45.7)	291/462 (63.0)	<.001
Median heart rate (IQR)—bmpa	84 (78–96)	84 (78–96)	89 (79–104)	<.001
Median respiratory rate (IQR)—bmpa	20 (18–21)	20 (18–21)	21 (20–26)	<.001
Median SBP (IQR)—mmHg	129 (120–140)	128 (120–140)	128 (118–142)	.91
Median DBP (IQR)—mmHg	79 (72–86)	79 (72–86)	76 (69–85)	<.001
Fever—no./total no. (%)	4466/6139 (72.7)	3728/5160 (72.2)	343/421 (81.5)	<.001
Median follow-up time (IQR)—days	15 (10–23)	16 (10–23)	8 (4–15)	<.001
Comorbidities on admission
Chronic obstructive pulmonary disease—no./total no. (%)	75/6414 (1.2)	60/5342 (1.1)	9/461 (2.0)	.12
Diabetes—no./total no. (%)	806/6414 (12.6)	613/5342 (11.5)	97/461 (21.0)	<.001
Coronary heart disease—no./total no. (%)	460/6414 (7.2)	328/5342 (6.1)	85/461 (18.4)	<.001
Cerebrovascular diseases—no./total no. (%)	204/6414 (3.2)	138/5342 (2.6)	40/461 (8.7)	<.001
Hypertension—no./total no. (%)	1935/6414 (30.2)	1496/5342 (28.0)	234/461 (50.8)	<.001
Laboratory examination on admission
Neutrophil count > 6.3 × 10^9/L—no./total no. (%)	819/5630 (14.5)	493/4656 (10.6)	227/423 (53.7)	<.001
Lymphocyte count < 1.1 × 10^9/L—no./total no (%)	2193/5631 (38.9)	1579/4656 (33.9)	351/423 (83.0)	<.001
Platelet count < 125 × 10^9/L—no./total no. (%)	539/5637 (9.6)	347/4662 (7.4)	131/423 (31.0)	<.001
C-reactive protein > ULN—no./total no. (%)b	2329/4706 (49.5)	1773/3978 (44.6)	334/341 (97.9)	<.001
Procalcitonin > ULN—no./total no. (%)b	1208/4740 (25.5)	783/3875 (20.2)	268/378 (70.9)	<.001
BUN > ULN—no./total no. (%)b	535/5593 (9.6)	280/4630 (6.0)	186/421 (44.2)	<.001
Total cholesterol > 5.17 mmol/L—no./total no. (%)	609/4899 (12.5)	537/4022 (13.4)	13/374 (3.5)	<.001
D-dimer > ULN—no./total no. (%)b	2106/4760 (44.2)	1516/3897 (38.9)	316/380 (83.2)	<.001
Low density lipoprotein > 3.37 mmol/L—no./total no. (%)	608/4899 (12.4)	509/4022 (12.7)	23/374 (6.1)	<.001
Median SpO2 (IQR)—%	98 (96–98)	98 (96–98)	91 (81–96)	<.001
Other risk factors
Teaching hospital—no./total no. (%)	3359/6415 (52.4)	2532/5342 (47.4)	309/462 (66.9)	<.001
Admission before Feb 12th—no./total no. (%)c	3836/6415 (59.8)	2004/5342 (37.5)	356/462 (77.1)	<.001

Abbreviations. SBP, systolic blood pressure; DBP, diastolic blood pressure; BUN, blood urea nitrogen; SpO2, oxygen saturation; ULN, upper limit of normal.

The bpm in heart rate and respiratory rate denotes beat per minute and breath per minute, respectively.

ULN indicates the upper limit of the normal range of each biochemical test. The reference ranges of the test in each hospital were provided in Supplemental Table 3.

The patients who admitted to hospital before 12 February 2020 when the daily newly diagnosed cases start to decline in China.

p Values for the comparison of survivors with patients who died and were calculated by the Mann-Whitney U test for non-normally distributed continuous variables and by the chi-square test or Fisher’s exact test for categorical variables.

OURMAPCN score and Cox analysis

After the random forest model selection, 21 clinical variables were selected among 64 clinical parameters recorded at admission based on the patient’s data from training cohort (Supplemental Table 5). To increase clinical applicability, the 21 variables were dichotomized into another 34 risk factors and further evaluated by a LASSO algorithm, leading to the extraction of eight risk factors at admission that were closely associated with mortality in COVID-19 patients (Supplemental Table 6). The final eight risk factors, namely, Oxygen saturation < 90%, blood Urea nitrogen > upper limit of normal (ULN), Respiratory rate> 30, admission before the date of the first national Maximum number of daily new cases was reached, Age ≥ 60, Procalcitonin > ULN, C-reactive protein (CRP) >ULN and absolute Neutrophil counts > ULN, were incorporated into the risk score model for the prediction of mortality in COVID-19 patients. Sensitivity analysis with the complete cases without imputation obtained seven of the eight variables except procalcitonin > ULN.

To calculate the risk score, we assigned each of the eight prognostic factors a numeric point value that was proportional to its specific regression coefficient in a multivariate Cox proportional hazard model. Calibration curves for predicting probabilities of survival at 14 days, 21 days, and 28 days were computed from the Cox models on which the risk score based. We observed a high degree of similarity between the observed and the estimated rate at 14 days, 21 days, and 28 days (Supplemental Figure 1). Patients who remained in the hospital at the end of follow-up time were treated as right censor. Scores were calculated by summing the points for each of the eight factors; the points for each factor were obtained by multiplying the coefficient in the Cox model by four and then rounding, with a maximum value set at five (Table 2).

Table 2.

Cox model for the scores of selected risk factors.

Risk factor	Incidence of death %c	Cox model
		HR (95% CI)d	Coefficient	Pointse
C-reactive protein > ULNa	11.02	6.50(3.68–11.48)	1.87(1.30–2.44)	5
SpO2 < 90%	54.98	3.17 (2.37–4.23)	1.15(0.86–1.44)	4
Admission before Feb 12thb	13.38	1.85 (1.41–2.43)	0.61(0.34–0.89)	3
Age ≥ 60	20.41	1.99 (1.42–2.78)	0.69(0.35–1.02)	3
BUN > ULNa	35.62	2.29 (1.79–2.94)	0.83(0.58–1.08)	2
Respiratory rate ≥ 30	43.00	2.57(1.64–4.04)	0.94(0.49–1.40)	2
Procalcitonin level > ULNa	14.62	2.10 (1.61–2.72)	0.74(0.48–1.00)	2
Neutrophil count > 6.3 × 10^9/L	29.47	2.11 (1.65–2.70)	0.75(0.50–0.99)	2

Abbreviations. BUN, blood urea nitrogen; SpO2, oxygen saturation; ULN, upper limit of normal; HR, hazard ratios.

The ULN denotes the upper limit of the normal range of each biochemical test. The reference ranges of tests in each hospital were provided in Supplemental Table 3.

The proportion of patients who admitted to hospital before 12 February 2020 when the daily newly diagnosed cases start to decline in China.

The incidences of death in each risk factor were calculated by the number of deaths to the total number of patients with each increased risk.

Hazard ratios with corresponding 95% confidence intervals (CIs) were calculated by multivariate Cox regression analysis and were for the comparison of survivors with patients who died.

The assignment of points to risk factors was based on a multivariable Cox regression coefficient.

We also conducted a sensitivity analysis using univariate GLM followed by pair-matching comparison and random forest model in step 1, and GLM stepwise backward model in step 2, to select candidate variables. A multivariate Cox proportional hazard model was used to calculate the risk score. As shown in Supplemental Table 7, the variables in the final risk score in sensitivity analysis were the same as the OURMAPCN score.

Prediction of mortality in the training and test cohorts

The OURMAPCN score had a high prognostic significance in the training cohort, with an AUROC of 0.92 (95% CI, 0.90–0.93) (Table 3 and Supplemental Figure 2). A density plot was generated to determine a cutoff value for the OURMAPCN score to discriminate the patients with low and high risks of mortality: the low-risk group had an OURMAPCN score from 0 to 11 points, and a high-risk group had an OURMAPCN score from 12 to 23 points (Supplemental Figure 3). The Kaplan–Meier analysis showed that the patients with scores above 11 points had a significantly decreased survival rate in the training cohort, with an HR of 18.18 (95% CI, 13.93–23.71; p < .0001) (Supplemental Figure 4(A)). In the test cohort, the OURMAPCN score also had high C statistic indexes, with AUROCs of 0.90 (95% CI, 0.87–0.92) (Table 3 and Supplemental Figure 2). The HR for mortality was 12.36 (95% CI, 8.79–17.38; p < .0001) in patients with scores of more than 11 points in the test cohort (Supplemental Figure 4(A)). A webpage dedicated to OURMAPCN score calculation (http://compute.covid-ourmap.cn:8888/comput-OURMAPCN) has been developed to support doctors at the point-of-care.

Table 3.

Performance of OURMAPCN score in training and test dataset.

	OURMAPCN score in training dataset	OURMAPCN score in test dataset
N	4063	1741
AUROC (95% CI)	0.92 (0.90–0.93)	0.90 (0.87–0.92)
Cutoff value	≤11	≤11
Total accuracy, % (95% CI)	77.48 (76.00–89.54)	80.59 (78.58–82.83)
Sensitivity, % (95% CI)	92.36 (77.39–95.54)	87.16(81.08–91.89)
Specificity, % (95% CI)	76.18 (74.58–90.34)	79.97 (77.84–82.49)
PPV, % (95% CI)	24.66 (23.31–41.03)	28.81 (26.36–31.77)
NPV, % (95% CI)	99.16 (97.96–99.51)	98.54 (97.85–99.08)
PLR, (95% CI)	3.86 (3.61–4.12)	4.47 (3.97–5.03)
NLR, (95% CI)	0.10 (0.07–0.14)	0.16 (0.10–0.23)

Abbreviations. AUROC, area under the receiver operating characteristics; PPV, positive predictive value; NPV, negative predictive value; PLR, positive likelihood ratio; NLR, negative likelihood ratio; 95% CI, 95% confidence interval.

OURMAPCN score performance in the external validation cohorts

The patients’ characteristics in validation cohort 1, which consisted of 6351 patients from three independent hospitals in Wuhan city, are detailed in Supplemental Table 8. The OURMAPCN score had an AUROC of 0.89 (95%CI, 0.88–0.91) in the validation cohort 1 (Supplemental Figure 2) (Table 4). Based on the OURMAPCN score, patients with scores greater than 11 points had a higher risk of mortality than those with lower scores, with an HR of 14.35 (95% CI, 11.00–18.73; p < .0001) (Supplemental Figure 4(B)).

Table 4.

OURMAPCN score performance in validation cohorts in Wuhan, other cities in Hubei, China, and Milan, Italy.

	Wuhan validation cohort 1	Other cities in Hubei validation cohort 2	Milan, Italy validation cohort 3
AUROC (95% CI)	0.89 (0.88–0.91)a	0.90 (0.88–0.93)a	0.81 (0.76–0.86)a
Total accuracy, % (95% CI)	89.07 (72.74–89.92)	83.45 (80.80–90.10)	77.10 (62.81–84.35)
Sensitivity, % (95% CI)	72.57 (68.82–91.31)	86.36 (76.52–92.42)	67.82 (51.72–88.51)
Specificity, % (95% CI)	90.92 (70.93–91.81)	83.06 (80.39–90.74)	79.38 (57.63–90.11)
PPV, % (95% CI)	45.70 (24.68–48.50)	25.65 (22.29–36.31)	44.70 (32.58–60.00)
NPV, % (95% CI)	96.89 (96.49–98.73)	98.92 (98.32–99.41)	91.12 (88.14–95.58)
PLR, (95% CI)	7.99 (7.25–8.82)	5.14 (4.56–5.79)	3.29 (2.57–4.27)
NLR, (95% CI)	0.30 (0.27–0.35)	0.16 (0.10–0.24)	0.40 (0.28–0.54)

Abbreviations. AUROC, area under the receiver operating characteristics; PPV, positive predictive value; NPV, negative predictive value; PLR, positive likelihood ratio; NLR, negative likelihood ratio; 95% CI, 95% confidence interval.

The C statistic metrics for OURMAPCN score were reported in validation cohorts in Wuhan, other cities in Hubei, China and Milan, Italy.

Another cohort of 2169 patients with COVID-19 from eleven hospitals outside of Wuhan city was designated validation cohort 2. The baseline characteristics of the patients in the validation cohort 2 are described in Supplemental Table 9. The OURMAPCN score had an AUROC of 0.90 (95%CI, 0.88–0.93) (Supplemental Figure 2) (Table 4) in the validation cohort 2. Again, patients with an OURMAPCN score greater than 11 points had a higher risk of mortality than those with lower scores, with an HR of 32.14 (95% CI, 7.95–129.9; p < .0001) (Supplemental Figure 4(C)).

The OURMAPCN score (AUROC 0.89 [95%CI, 0.88–0.91] in validation cohort 1 and AUROC 0.90 [95%CI, 0.88–0.93] in validation cohort 2) significantly outperformed the existing scoring systems, including the MuLBSTA (AUROC 0.58 [95%CI, 0.56–0.60] in validation cohort 1 and AUROC 0.79 [95%CI, 0.75–0.83] in validation cohort 2) and CURB-65 (AUROC 0.81 [95% CI, 0.79–0.83] for validation cohort 1 and AUROC 0.80 [95% CI, 0.76–0.84] in validation cohort 2) scores for the prediction of mortality in COVID-19 patients (Table 5). When compared with a recently published COVID-19 risk score COVID-GRAM, the OURMAPCN score has a similar performance in the Wuhan population and a superior performance in patients outside of Wuhan city (Table 5).

Table 5.

Comparison of OURMAPCN score with the MuLBSTA, CURB-65, and COVID-GRAM scores for the identification of high risk of mortality.

	Validation 1: Wuhan		Validation 2: Other cities in Hubei
Scores	N	AUROC (95% CI)	N	AUROC (95% CI)
OURMAPCN score	6094	0.89 (0.88–0.91)	2151	0.90 (0.88–0.93)
MuLBSTA scorea	6094	0.58 (0.56–0.60)	2151	0.79 (0.75–0.83)
CURB-65 scoreb	6094	0.81 (0.79–0.83)	2151	0.80 (0.76–0.84)
COVID-GRAMc	6094	0.87 (0.86–0.89)	2151	0.85 (0.82–0.89)

Abbreviations. AUROC, area under the receiver operating characteristics;.

MuLBSTA, a score used in patients with viral pneumonia, to predict clinical characteristics that affect mortality. It comprises factors: multilobe infiltrate (5), absolute lymphocyte counts ≤0.8 × 10^9/L (4), bacterial infection (4), acute smoker (3), quit smoking (2), hypertension (2), and age ≥60 years (2).

CURB-65, a score used in the emergency department setting to risk stratify a patient’s community-acquired pneumonia. It comprises factors: confusion (1), BUN above 7 mmol/L (1), respiratory rate ≥ 30 (1), systolic BP < 90 mmHg or diastolic BP ≤ 60 mmHg (1), and Age ≥ 65 (1).

COVID-GRAM, a score used in patients with COVID-19, to predict the risk of developing critical illness. It comprises factors: X-ray abnormality, age, hemoptysis, dyspnea, unconsciousness, no. of comorbidities, cancer history, neutrophil to lymphocyte ratio, lactate dehydrogenase, and direct bilirubin.

Importantly, in addition to the evaluation of the scoring system in the training and the test cohorts, the predictive performance of the OURMAPCN score was evaluated in the subgroup of patients with different age ranges, sexes, and medical histories of chronic obstructive pulmonary disease (COPD), coronary artery disease (CAD), type 2 diabetes mellitus (T2DM) and hypertension (Table 6). The performance of the OURMAPCN score was robust and stable across different subsets of patients, with AUROCs ranging from 0.79 to 0.92.

Table 6.

Performance of OURMAPCN score for subgroups in validation cohorts from Wuhan and other cities in Hubei.

Subgroupsa	Mortality rate %	AUROC (95% CI)	Sensitivity% (95% CI)	Specificity% (95% CI)	PPV% (95% CI)	NPV% (95% CI)
Age
<65	4.28	0.90 (0.88–0.92)	83.26 (77.68–87.98)	85.32 (84.33–88.38)	20.30 (18.87–23.84)	99.13 (98.84–99.38)
≥65	17.36	0.84 (0.82–0.86)	72.63 (68.72–76.54)	83.10 (81.59–84.66)	47.42 (44.83–50.13)	93.52 (92.69–94.42)
Gender
Female	5.60	0.90 (0.89–0.92)	88.09 (83.40–94.04)	78.03 (71.78–79.60)	18.96 (16.01–20.39)	99.11 (98.76–99.52)
Male	11.97	0.89 (0.88–0.91)	87.60 (69.21–91.53)	73.41 (71.22–91.35)	31.48 (29.50–53.26)	97.73 (95.60–98.45)
COPD
No	7.83	0.92 (0.90–0.93)	87.98 (84.14–91.56)	82.14 (78.18–83.44)	29.40 (25.87–31.21)	98.77 (98.40–99.11)
Yes	35.16	0.87 (0.79–0.94)	84.38 (56.25–96.88)	77.97 (59.32–96.61)	67.44 (54.54–91.30)	89.36 (79.71–97.78)
CAD
No	7.74	0.91 (0.90–0.92)	89.39 (72.84–92.02)	76.91 (75.77–93.13)	24.55 (23.51–47.56)	98.85 (97.60–99.13)
Yes	20.41	0.79 (0.74–0.83)	80.00 (50.77–88.46)	62.92 (55.23–88.17)	36.03 (32.04–55.78)	92.09 (87.35–95.23)
Diabetes
No	7.64	0.91 (0.90–0.92)	89.31 (86.49–91.74)	77.67 (76.66–78.66)	24.87 (23.87–25.87)	98.88 (98.59–99.13)
Yes	14.67	0.85 (0.81–0.88)	69.35 (62.90–84.95)	85.49 (67.74–87.71)	45.08 (30.49–49.62)	94.19 (93.02–96.31)
Hypertension
No	5.97	0.91 (0.90–0.93)	87.31 (83.38–91.84)	81.81 (77.59–83.06)	23.12 (20.08–24.77)	99.02 (98.72–99.35)
Yes	14.36	0.86 (0.84–0.88)	71.91 (67.27–76.29)	86.17 (84.70–87.55)	46.52 (43.53–49.57)	94.82 (93.98–95.59)

Abbreviations. AUROC, area under the receiver operating characteristics; PPV, positive predictive value; NPV, negative predictive value; COPD, chronic obstructive pulmonary disease; CAD, coronary artery disease.

aThe C statistic for OURMAPCN score is reported in age, gender, COPD, CAD, diabetes, and hypertension subgroup population among all study population.

OURMAPCN score performance in the Italian cohort

To further explore the performance of the OURMAPCN score in patients outside of China, we calculated the C-statistic in a cohort of COVID-19 patients from Milan, Italy. The baseline characteristics of this cohort are described in Supplemental Table 10. In this Italian cohort, serum urea levels were not commonly measured in patients with COVID-19. We used serum creatinine as a substitute risk factor for BUN. The modified OURMAPCN score had an AUROC of 0.81 (95%CI, 0.76–0.86), and the other C-statistic metrics are listed in Supplemental Figure 2 and Table 4. Therefore, the OURMAPCN score yielded a satisfactory accuracy in the Italian cohort.

Discussion

We have developed the OURMAPCN score using eight baseline risk factors at admission, namely, age, respiratory rate, oxygen saturation, absolute neutrophil counts, CRP, BUN, procalcitonin, and admission before the date of the first maximum number of daily new cases in the country. The OURMAPCN score is a robust and reliable tool for predicting mortality in patients with COVID-19. These eight factors are based on common clinical parameters routinely obtained in the emergency room or at hospital admission. Therefore, the OURMAPCN score can be readily determined and implemented as a very practical tool for patient risk stratification, not only by intensive care specialists but also by physicians with diverse backgrounds and specialties. This robust yet easy-to-implement clinical risk score will be valuable in the dedicated COVID-19 clinics, particularly when medical resources are limited in Asian population during the COVID-19 pandemic.

At the functional level, the OURMAPCN score integrates multidimensional information. One domain represented by the model is the degree of pulmonary impairment (respiratory rate and oxygen saturation); one may reflect the status of the patient’s systemic response to infection and the ensuing inflammation (absolute neutrophil counts, CRP and procalcitonin), and another domain (BUN,) may indicate the level of tissue damage and breakdown and renal function. Age is recognized as a major risk factor that is positively associated with the risk of mortality due to COVID-19. Finally, this cohort comprises a large proportion of patients at the start of the pandemic when the experience of treating patients with SARS-CoV-2 infection and medical resources were limited (before the date of the first maximum number of daily new cases). It is intuitive that limited medical resources and lack of experiences can lead to a lack of life-saving intervention for critically ill COVID-19 patients. Fortunately, the majority of countries have reached the first peak and with accumulated knowledge in dealing with COVID-19 patients (Supplemental Table 4). Thus, these parameters, which were selected through an unbiased approach in a large COVID-19 cohort may offer some valuable insights into the potential pathogenic mechanisms underlying the poor outcome in the subset of COVID-19 patients.

There are several other baseline clinical variables that have been reported to be associated with COVID-19 severity and mortality, including BMI, COVID-19 symptoms (e.g. fever, short of breath), D-Dimer, and troponin. Indeed, we have included BMI, fever, and D-Dimer in our original set of variables. However, these variables were ruled out in the first-round of variable selection using a random forest model. The potential explanation for BMI has not been selected by the model might be associated with the lower BMI and narrow BMI distribution in the Chinese population. The narrow distribution of a predictor limited its ability to capture its impact on an outcome. Because short of breath had a high correlation with respiratory rate, to avoid overfitting, we only applied respiratory rate in our variable selection process. As for troponin, this test was chosen only when the patients were suspected of having a severe cardiac injury or cardiac ischemia. The proportion of available values only counted up 5% for cardiac troponin I and 10% for cardiac troponin T in our cohort, thus, troponin was not included in our variable selection process.

The OURMAPCN score was specifically developed and optimized with the goal of identifying COVID-19 patients at a high risk of mortality at hospital admission. The score was sufficiently validated in multiple independent cohorts from China and Italy. The score shows robust performance with regard to predicting mortality across different populations, ethnicities and clinical practices. However, further studies with larger cohorts in different populations will be needed to establish whether the performance of the score is influenced by sample sizes or population heterogeneity.

Our data show that this score significantly outperformed two existing pneumonia scores, the MuLBSTA and CURB-65 scores, which were initially developed to estimate the mortality due to pneumonia. Indeed, neither of these scores showed adequate predictive accuracy when applied to the same COVID-19 validation cohorts in this study. In addition, certain parameters required for the MuLBSTA score and CURB-65 score, such as bacterial infection, were not commonly available at hospital admission in COVID-19 patients, which hinders their application as a stratification and triage tool at admission. When compared to the COVID-GRAM score by Liang et al., the two models had similar predictive ability in the population in Wuhan. However, the OURMAPCN score has better performance in patient populations outside of Wuhan city. Since five out of the ten risk factors in the COVID-GRAM score were patient-reported variables, inadequate collection of these variables may lead to the underestimation of disease severity. This probably explained the decrease in the AUROC in the population outside of Wuhan city. Recently, a 4 C Mortality Score was established for estimating the in-hospital mortality in patients from European countries. Due to the Glasgow coma scale score was not routinely carried out for patients who admitted to hospitals in China, 4 C Mortality Score was not able to be estimated in the population involved in this study.

A critical aspect of the OURMAPCN score is that its capability for predicting the risk of mortality is not significantly impacted by age, sex, or several commonly observed comorbidities that have been associated with COVID-19 severity,. We recommended an OURMAPCN score of 11 as a cut-off threshold to maximize sensitivity while allowing some false positives.

The overall validity of the OURMAPCN score for the prediction of mortality was strengthened by the large cohort size, including in the training and testing cohorts; validation in multiple independent validation cohorts in China and Europe, and the well-balanced distribution in terms of age, sex and prevalence of comorbidities. Retrospective analyses indicated that only approximately 10–20% of patients with COVID-19 developed severe complications and need hospitalization or admission to the ICU, while the remaining patients are at lower risk of severe disease and mortality. If most patients at lower risk can recover from COVID-19 in community hospitals or isolation facilities, more medical resources can be allocated to patients at higher risk who need advanced therapy. Therefore, our scoring system provides a method of identifying patients with COVID-19 who are at high risk for severe complications and mortality. When medical resources are limited, prioritizing patients who are at high risk by allocating them more advanced medical resources is an effective means of reducing the mortality rate.

Limitation

Our study also has some limitations. First, our study was only based on baseline parameters at admission and focused on prediction. Hence no causal conclusions can be drawn from our algorithm,. Second, the OURMAPCN score may simplify model utility and implementation with the resultant potential loss of information; however, the score performed well compared to the logistic model with eight-variables. Third, during the COVID-19 pandemic, several possible variables, including prehospital medication, personal habits, and socioeconomic factors, were not collected or were insufficiently documented. Even though rigorous validation of the data imputation process was conducted, the impacts of these unmeasured parameters and missing values on patient outcomes may reduce the power and accuracy of our model, especially when it is applied in more heterogeneous populations. Fourth, the variable selection process started form 64 candidate variables; however, the random forest model and LASSO model were set up to robustly limit the overfitting. Importantly, the developed score demonstrated good discrimination and calibration in all three validation cohorts. Fifth, fewer patients in the study cohort were recruited in the very early period of the pandemic outbreak, which may create unknown bias in the temporal pattern of mortality following infection, such as delayed therapy, or the lack of knowledge and therapeutic experiences with COVID-19. Sixth, the patient cohort included COVID-19 in-hospital patients mainly from China, only one city in Italy. The generalizability of this model to patients of different genetic backgrounds and from different geographic environments was not examined. Extrapolation of the model to general or community patients with COVID-19 requires further examination. Seventh, the cutoff value used for dichotomization was derived according to the reference ranges of individual hospitals. Due to the stable performance of the score across the 22 hospitals involved in our study, we propose that other hospitals may use their own reference threshold for the cutoff value. Accumulating data from different institutions will provide us with increasing confidence in this conclusion in the future.

Conclusion

We have developed and validated a risk assessment tool, the OURMAPCN score, to evaluate the mortality risk in patients with COVID-19 using limited clinical parameters obtained at hospital admission. Previous data revealed that only approximately 10–20% of COVID-19 patients develop severe complications and mortality and therefore need hospitalization or admission to the ICU,. Therefore, the OURMAPCN score may assist frontline clinicians in optimizing medical treatment in the context of limited resources by prioritizing patients who are at higher risk of mortality. While this risk score was derived from a patient population in China, it was validated in an independent European cohort. Satisfactory performance of the OURMAPCN score suggested that it could be a candidate tool to predict the mortality risk in COVID-19 patients elsewhere,. However, before its clinical application, additional modification and validation of the OURMAPCN score to fit each country's specific situation are still needed.

Click here for additional data file.