Development and validation of a prediction model for 30-day mortality in hospitalised patients with COVID-19: the COVID-19 SEIMC score.

OBJECTIVE: To develop and validate a prediction model of mortality in patients with COVID-19 attending hospital emergency rooms.
DESIGN: Multivariable prognostic prediction model.
SETTING: 127 Spanish hospitals. PARTICIPANTS: Derivation (DC) and external validation (VC) cohorts were obtained from multicentre and single-centre databases, including 4035 and 2126 patients with confirmed COVID-19, respectively.
INTERVENTIONS: Prognostic variables were identified using multivariable logistic regression. MAIN OUTCOME MEASURES: 30-day mortality.
RESULTS: Patients' characteristics in the DC and VC were median age 70 and 61 years, male sex 61.0% and 47.9%, median time from onset of symptoms to admission 5 and 8 days, and 30-day mortality 26.6% and 15.5%, respectively. Age, low age-adjusted saturation of oxygen, neutrophil-to-lymphocyte ratio, estimated glomerular filtration rate by the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation, dyspnoea and sex were the strongest predictors of mortality. Calibration and discrimination were satisfactory with an area under the receiver operating characteristic curve with a 95% CI for prediction of 30-day mortality of 0.822 (0.806-0.837) in the DC and 0.845 (0.819-0.870) in the VC. A simplified score system ranging from 0 to 30 to predict 30-day mortality was also developed. The risk was considered to be low with 0-2 points (0%-2.1%), moderate with 3-5 (4.7%-6.3%), high with 6-8 (10.6%-19.5%) and very high with 9-30 (27.7%-100%).
CONCLUSIONS: A simple prediction score, based on readily available clinical and laboratory data, provides a useful tool to predict 30-day mortality probability with a high degree of accuracy among hospitalised patients with COVID-19. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The development of a predictive prognostic model is essential for improving the management of patients with severe COVID-19.

In a recent systematic review and critical appraisal of prediction models for COVID-19, 50 prognostic models were identified. All models were considered to have a high risk of bias, and none were recommended for clinical use.

The COVID-19 SEIMC score was developed and externally validated with two large datasets from patients hospitalised with laboratory-confirmed COVID-19. The score based on age, low age-adjusted saturation of oxygen, neutrophil-to-lymphocyte ratio, estimated glomerular filtration rate by the CKD-EPI equation, dyspnoea and sex could identify the probability of 30-day mortality with a high degree of accuracy among patients with COVID-19.

Introduction

The clinical spectrum of the novel SARS-CoV-2 associated COVID-19 varies broadly, from asymptomatic disease to pneumonia and life-threatening complications, including acute respiratory distress syndrome, multisystem organ failure and death.1–4

The main poor prognostic factor identified in different series of COVID-19 is advanced age.3 5 6 Other factors that have been associated with poor outcomes include male gender, several comorbidities, lymphocyte counts, high concentrations of different inflammatory or coagulation markers, serum levels of different cytokines and features derived from imaging studies.5 7–10

Prediction prognostic models are developed to aid healthcare providers in estimating the probability or risk that a specific event will occur, to inform their decision-making.11 Prediction models can be based on regression or machine learning.12 In a recent systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19, 50 prognostic models were identified; 23 estimated mortality risk, 8 aimed to predict severe disease or critical illness and the remaining 19 assessed other outcomes.13 The majority of the models included in the review used clinical and laboratory data from Chinese patients. All models were considered to have a high risk of bias due to a combination of poor reporting and poor methodological conduct for participant selection, predictor description and statistical methods, and none were recommended for clinical use.13 14 Eight additional studies of prognostic prediction models for COVID-19, including predominantly participants from China, have been published.15–22 Outcomes included mortality in five studies16 17 19–21 and severe disease or critical illness in three.15 18 22 The model performance was good across all studies, although the same methodological limitations found in the meta-analysis also applied.

The development of a high-quality clinical predictive model of death to stratify patients into risk groups is essential for improving the management of patients with severe COVID-19 and evaluating therapeutic interventions' efficacy. Our study’s objective was to develop and validate a prediction score to estimate the probability of 30-day mortality in patients with severe COVID-19.

Methods

The predictive model’s development followed the recommendations stated in the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Initiative11 23 (see online supplemental appendix table 1).

Source of data

The data source was the databases of two large retrospective cohorts of hospitalised patients with COVID-19 in Spain in 2020. The derivation cohort (DC) was the COVID-19@Spain, a multicentre cohort of patients hospitalised from 2 February to 17 March, with 17 April as the follow-up censoring date, sponsored by the Spanish Society of Infectious Diseases and Clinical Microbiology (SEIMC), and registered in ClinicalTrials.gov (NCT04355871).24 The external validation was COVID-19@HULP, a large single-centre cohort of patients admitted to La Paz University Hospital in Madrid (Spain) from 25 February (the first case admitted) to 19 April; and registered in the European Union Electronic Register of Post-Authorisation Studies (EUPAS34331).25

Participants

The DC included the first consecutive 4035 patients with COVID-19 admitted to 127 hospitals distributed across all regions in Spain. The external validation cohort (VC) included 2126 of the 2226 patients from COVID-19@HULP after the exclusion of the 100 patients contributing to COVID-19@Spain. The eligibility criteria in the DC and external VC were hospital admission due to COVID-19 confirmed with real-time PCR for SARS-CoV-2. No age limit was required in the DC, whereas an age of 18 years or older was an eligibility criterion in the external VC. The DC and VC were identical in terms of setting and definitions for outcomes and predictors. Besides, data in both cohorts were collected using the same modified version of the case report form (CRF) of the WHO–International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) Core CRF.26

Outcome

The outcome was 30-day all-cause mortality, measured from the day of hospital admission. Patients that were discharged alive before 30 days after admission were assumed to have survived for at least 30 days.

Predictors

Predictors were preselected among the 17 baseline variables, recorded at hospital admission, independently associated with death in the COVID-19@Spain cohort by multivariable Cox regression analyses.24 These variables were distributed in the following five clusters: (1) demographics, age in years and sex at birth; (2) comorbidities defined as diagnoses included in the medical record such as hypertension, obesity (body mass index >30), liver cirrhosis, chronic neurological disorder, active neoplasia (solid or haematologic) and dementia; (3) signs or symptoms, including dyspnoea and confusion; (4) low age-adjusted capillary oxygen saturation (SaO2) on room air, defined as ≤90% for patients aged >50 years and ≤93% for patients aged ≤50 years27; (5) tests results, including white cell count, neutrophil-to-lymphocyte ratio, platelet count, international normalised ratio (INR), estimated glomerular filtration rate (eGFR) measured by the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation28 and serum concentrations of C reactive protein.

Statistical analysis methods

We followed recent recommendations to calculate the minimum sample size required for prediction model development.29 We carried out a complete-case analysis (primary analysis) and two sensitivity analyses. In the first sensitivity analysis, we included all patients and missing values for predictors were considered as a separate category (missing indicator method). In the second sensitivity analysis, we also included all patients and missing values for predictors were left blank (equivalent to the lowest risk situation). No missing values for outcomes occurred in the DC or the external VC.

Continuous variables were categorised for the analysis. As mortality from COVID-19 among hospitalised patients is highly correlated with age, this variable was divided into 11 levels: <40 years that was the reference category and after that into 11 5-year to 10-year intervals up to ≥90 years that was the last category. The neutrophil-to-lymphocyte ratio was categorised into tertiles: <3.22, which was the reference category, 3.22 to 6.33, and >6.33. The eGFR in mL/min/1.73 m2 was grouped before the analysis into three categories: >60 (normal to mildly decreased eGFR), 30–59 (moderately to severely decreased eGFR) and <30 (severely decreased eGFR).

We used univariable and multivariable logistic regression in the derivation dataset to estimate the coefficients of each potential predictor of 30-day overall mortality. We fitted the final model by choosing predictors based on the strength of their unadjusted association with death. The model started with the predictor with the highest area under the receiver operating characteristics (AUROC) to predict 30-day mortality. Subsequently, the rest of the variables were introduced one by one, creating all the possible models of two independent variables, and the combination of higher AUROC was chosen. This process was repeated to form models of 3, 4 and more variables, always choosing the combination with the highest AUROC. The process stopped when the inclusion of a new variable in the model meant an increase lower than 0.005 unit in the AUROC.

We assessed the predictive performance of the model by examining measures of calibration and discrimination. We developed a calibration plot with estimates of the calibration slope and intercept. Calibration was also assessed using the Hosmer-Lemeshow test. Discrimination was examined by calculating its AUROC with the 95% CI. We carried out internal validation through a bootstrap with 1000 random samples with replacement to estimate the model optimism and shrinkage factor.

The logistic regression model’s coefficients were converted to a simplified score to facilitate its application in clinical practice. The score was developed, dividing each coefficient by the coefficient with the lowest value and rounding to an integer. Risk groups were created using the 30-day probability of death according to the simplified score. The sensitivity, specificity, positive and negative predictive values, and likelihood ratios were calculated for different scores.

The statistical analyses were performed using Stata software (V.15.0; Stata Corporation, College Station, Texas, USA).

Results

The developing cohort included 4035 patients, of which 1074 (26.6%) died and 2961 were alive within 30 days of hospital admission. The cohort size was more than twice the required for developing a clinical prognostic model (online supplemental appendix figure 1). The external VC included 2202 patients, 341 (15.5%) died and 1861 were alive within 30 days of hospital admission. The median time to death since hospital admission was 10 (IQR 6–16) days in the -DC and 5 (IQR 3–10) days in the VC.

The characteristics of the participants, including demographics, presenting signs and symptoms, presence of lung infiltrates on chest radiograph, oxygenation and laboratory parameters, are shown in table 1. Patients in the DC were, on average, 9 years older, and more frequently, males than patients in the external VC. Statistically significant differences between the cohorts were found in all the analysed variables.

Table 1

Comparison of participant characteristics in the derivation and external validation cohorts

Characteristic	Derivation cohort(N=4035)			External validation cohort(N=2202)			P value
	Missing values	Valid cases	Value	Missing values	Valid cases	Value
Demographics
Median age (IQR)—years	4	4031	70 (56–80)	0	2202	61 (46–78)	<0.001
Male sex—N (%)	48	3987	2433 (61.0)	1	2201	1054 (47.9))	<0.001
Comorbidity
Current smoker—N (%)	1.118	2917	197 (6.8)	97	2105	156 (7.4)	<0.001
Hypertension—N (%)	25	4010	2052 (51.2)	17	2185	907 (41.5)	<0.001
Diabetes—N (%)	33	4002	871 (21.8)	16	2186	378 (17.3)	<0.001
Chronic kidney disease—N (%)	35	4000	199 (5.0)	2039	163	76 (46.6)	<0.001
Obesity (BMI>30)—N (%)	429	3606	497 (13.8)	61	2141	233 (10.9)	0.001
Chronic inflammatory disease—N (%)	38	3997	231 (5.8)	0	2202	255 (11.6)	<0.001
HIV/AIDS—N (%)	73	3962	26 (0.7)	20	2182	13 (0.6)	<0.001
Disease chronology
Δt onset of symptoms to admission, days—median (IQR)	462	3573	5 (2–7)	939	1263	8 (5–11)	<0.001
Symptoms and signs							<0.001
History of fever—N (%)	35	4000	3240 (81.0)	35	2167	1568 (72.4)	<0.001
Cough—N (%)	51	3984	2862 (71.8)	36	2166	1098 (50.7)	<0.001
Malaise—N (%)	121	3914	2505 (64.0)	38	2164	907 (41.9)	<0.001
Dyspnoea—N (%)	55	3980	1953 (49.1)	37	2165	1098 (50.7)	<0.001
Myalgia/Arthralgia—N (%)	226	3809	947 (24,9)		2160	588 (27.2)	0.045
Sputum production—N (%)	72	3963	956 (24.1)	61	2141	311 (14.5)	<0.001
Vomiting/Nausea—N (%)	111	3924	488 (12.4)	0	2202	295 (13.4)	<0.001
Diarrhoea—N (%)	123	3912	471 (12.0)	37	2165	482 (22.3)	<0.001
Radiology
Lung infiltrates on admission—N (%)	165	3870	3002 (77.6)	8	2194	1559 (71,1)	<0.001
Oxygenation
Age adjusted low SaO2—N (%)	490	3545	942 (26.6)	423	1779	344 (19.3)	<0.001
Laboratory parameter
Neutrophil-to-lymphocyte ratio—Median (IQR)	90	3945	4.5 (2.7–7.7)	636	1566	4.7 (2.9–8.0)	0.013
Platelets—number×1012 L—Median (IQR)	75	3960	178 (139–226)	636	1566	218 (169–285)	<0.001
D-dimer—ng/mL—Median (IQR)	2472	1563	580 (339–1040)	1325	877	736 (418–1374)	<0.001
eGFR—mL/min/1.73 m2 (CKD-EPI)—Median (IQR)	140	3895	78.4 (56.5–93.6)	645	1557	88.9 (71.5–103.1)	<0.001
ALT—U/L—Median (IQR)	796	3239	26 (18–41)	719	1483	31 (20–48)	<0.001
Serum albumin—g/dL—Median (IQR)	2624	1411	3.5 (3.2–3.9)	1071	1131	4.3 (3.9–4.5)	<0.001
Lactate dehydrogenase—U/L—Median (IQR)	1457	2578	290 (224–403)	967	1235	320 (254–404)	<0.001
C reactive protein—mg/L—Median (IQR)	358	3677	54 (20–116)	782	1420	75 (25–151)	<0.001

ALT, alanine aminotransferase; BMI, body mass index; eGFR, estimated glomerular filtration rate; SaO2, saturation of oxygen; Δt, time interval.

In the DC, targeted viral agents were administered to 82.0% of patients, including lopinavir/ritonavir (LPV/r) (70.4%), hydroxychloroquine (65.5%) and subcutaneous interferon-beta (29.2%), usually in combination with LPV/r. In the external VC, targeted viral agents were administered to 65.3% of patients. The most frequent combination was hydroxychloroquine plus azithromycin (31.7%), followed by hydroxychloroquine alone. Host-targeted agents in the DC included systemic corticosteroids in 28.0% patients and tocilizumab in 9.4% patients. In the VC, corticosteroids and tocilizumab were administered to 13.3% and 2.3% patients, respectively.

Model development and performance

The number of participants in the DC without missing values for each predictor, the number of outcomes per predictor and the unadjusted associations between predictors and outcomes are shown in table 2.

Table 2

Unadjusted association between candidate predictor variables and outcome in the derivation cohort (N=4035)

Characteristic	Number/with data (%)	Death by day 30		OR(95% CI)	P value
		Yes	No
Sex					<0.001
Female	1554/3987	341	1213	1
Male	2433/3987	721	1712	1.5 (1.29 to 1.74)
Age (years)					<0.001
<=40	302/4031 (7.5)	9	293	1
40–49	374/4031 (9.3)	16	358	1.45 (0.63 to 3.34)
50–54	266/4031 (6.6)	19	247	2.50 (1.11 to 5.63)
55–59	279/4031 (6.9)	38	241	5.13 (2.43 to 10.8)
60–64	356/4031 (8.8)	53	303	5.69 (2.76 to 11.7)
65–69	401/4031 (9.9)	78	323	7.86 (3.87 to 15.0)
70–74	522/4031 (12.9)	123	399	10.0 (5.02 to 20.1)
75–79	521/4031 (12.9)	201	320	20.4 (10.3 to 40.6)
80–84	410/4031 (10.2)	196	214	29.8 (14.9 to 59.5)
85–89	379/4031 (9.4)	200	179	36.4 (18.3 to 72.8)
>=90	221/4031 (5.5)	140	81	56.3 (27.5 to 115)
Hypertension	2052/4010 (51.2)	764	1288	3.22 (2.76 to 3.74)	<0.001
Obesity	497/3606 (13.8)	169	328	1.57 (1.29 to 1.93)	<0.001
Liver cirrhosis	54/3998 (1.4)	23	31	2.08 (1.21 to 3.58)	0.008
Chronic neurological disorder	373/4002 (9.3)	161	212	2.31 (1.85 to 2.87)	<0.001
Neoplasm (active)	352/4035 (8.7)	152	200	2.28 (1.82 to 2.85)	<0.001
Dementia	315/3979 (7.9)	184	131	4.52 (3.57 to 5.73)	<0.001
Myalgia/Arthralgia	947/3809 (24.9)	155	792	0.49 (0.40 to 0.59)	<0.001
Cough	2862/3984 (71.8)	688	2174	0.68 (0.59 to 0.79)	<0.001
Dyspnoea	1953/3980 (49.1)	668	1285	2.19 (1.89 to 2.53)	<0.001
Altered consciousness	450/3931 (11.4)	220	230	3.15 (2.58 to 3.86)	<0.001
White cell count—cells/×109/L					<0.001
<=4000	666/3971	132	534	1
4000–12 000	2993/3971	778	2215	1.42 (1.15 to 1.75)
>12 000	312/3971	151	161	3.79 (2.83 to 5.08)
Neutrophil-to-lymphocyte ratio					<0.001
<3.22	1316/3945	207	1109	1
3.22–6.33	1314/3945	298	1016	1.57 (1.29 to 1.91)
>6.33	1315/3945	547	768	3.82 (3.17 to 4.59)
eGFR—mL/min/1.73 m2 (CKD-EPI)					<0.001
>=60	2786/3895 (71.5)	512	2274	1
30–59	844/3895 (21.7)	379	465	3.62 (3.07 to 4.27)
<30	265/3895 (6.8)	153	112	6.07 (4.67 to 7.88)
Low SaO2 (age-adjusted)*	942/3545 (26.6)	413	529	3.44 (2.93 to 4.05)	<0.001
INR>1.1	1503/3301 (45.5)	524	979	2.20 (1.88 to 2.57)	<0.001
CRP>5 μg/L	3378/3677	939	2439	3.21 (2.21 to 4.67)	<0.001

*≤90% for patients aged >50 years and ≤93% for patients aged ≤50 years.

CKD-EPI, Chronic Kidney Disease Epidemiology Collaboration equation; CRP, C reactive protein; INR, international normalised ratio; SaO2, saturation of oxygen.

The final prediction model generated without recoding missing values (3358 participants) is shown in table 3. The variables used in the model to generate the score were those in table 2. The model started with the variable age since it was the one with the highest predictive capacity for death at 30 days (AUROC (95% CI) 0.768 (0.753 to 0.784)). The final input sequence of the variables to the model, following the procedure described in the Methods section, was age, low age-adjusted SaO2, neutrophil-to-lymphocyte ratio, eGFR by the CKD-EPI equation, dyspnoea and sex.

Table 3

Predictive model for 30-day mortality at presentation in hospitalised patients with COVID-19

Predictor variable	Coefficient	SE	OR (95% CI)	p>z
Age				<0.001
40–49 years	0.082	0.446	1.09 (0.45 to 2.6)
50–54 years	0.471	0.448	1.60 (0.67 to 3.86)
55–59 years	1.058	0.412	2.88 (1.28 to 6.46)
60–64 years	1.228	0.394	3.42 (1.58 to 7.4)
65–69 years	1.655	0.381	5.23 (2.48 to 11.04)
70–74 years	1.772	0.372	5.88 (2.84 to 12.21)
75–79 years	2.268	0.373	9.66 (4.65 to 20.07)
80–84 years	2.695	0.377	14.8 (7.08 to 30.96)
85–89 years	2.803	0.379	16.49 (7.84 to 34.67)
>90 years	3.103	0.397	22.26 (10.22 to 48.48)
Low age adjusted SaO2	0.875	0.102	2.40 (1.97 to 2.93)	<0.001
Neutrophil-to-lymphocyte ratio				<0.001
3.22–6.33	0.173	0.123	1.19 (0.93 to 1.51)
>6.33	0.657	0.119	1.93 (1.53 to 2.44)
eGFR (CKD-EPI)				<0.001
30–59 mL/min/1.73 m2	0.498	0.109	1.65 (1.33 to 2.04)
<30 mL/min/1.73 m2	1.093	0.176	2.98 (2.11 to 4.21)
Dyspnoea	0.414	0.097	1.51 (1.25 to 1.83)	<0.001
Male sex	0.466	0.098	1.59 (1.31 to 1.93)	<0.001
Intercept	−4.266	0.360

CKD-PI, Chronic Kidney Disease Epidemiology Collaboration equation; eGFR, estimated glomerular filtration rate calculated by the CKD-EPI; SaO2, oxygen saturation.

The predicted probability of 30-day mortality was determined by the following equation:

P death at day 30 = 1 / (1+exp (-b)),

where b=0 (if age <40)+0.082 (if age 40–49)+0.471 (if age 50–54)+1.058 (if age 55–59)+1.228 (if age 60–64)+1.655 (if age 65–69)+1.771 (if age 70–74)+2.268 (if age 75–79)+2.695 (if age 80–84)+2.803 (if age 85–89)+3.103 (if age>=90)+0.875 (if low age-adjusted SaO2)+0.173 (if neutrophil-to-lymphocyte ratio 3.22–6.33)+0.657 (if neutrophil-to-lymphocyte ratio >6.33)+0.498 (if eGFR 30–59)+1.093 (eGFR <30)+0.414 (if dyspnoea)+0.466 (if male sex)−4.266.

The final model showed good calibration across the range of risk (figure 1), and the goodness-of-fit Hosmer-Lemeshow test was 11.21, p=0.1902 vs p<0.05, confirming the calibration of the model. Using bootstrapping techniques, an optimism of 0.006 and a shrinkage factor of 0.968 were estimated. In 600 of the samples (60%), the Hosmer-Lemeshow test was significant.

Figure 1

Calibration of the final prognostic model in the derivation cohort. Observed versus predicted risk of 30-day mortality, with estimates of the calibration slope and intercept (Hosmer-Lemeshow test=11.21, p=0.1902 vs p<0.05).

The AUROC (95% CI) of the model for prediction of 30-day mortality was 0.822 (0.806 to 0.837) in the DC and 0.845 (0.819 to 0.870) in the external VC (online supplemental appendix table 2).

Simplified score development and performance

The simplified point score (from 0 to 30) resulting from the division of the regression coefficients of predictors in the final model by the coefficient of age 40–49, which was the lowest value among all coefficients, is shown in figure 2A. The prediction of 30-day mortality on presentation in hospitalised patients with COVID-19 according to the point score in the DC and in the external VC is shown in table 4.

Figure 2

(A) Simple scoring system to predict 30-day mortality on presentation in hospitalised patients with COVID-19. (B) 30-day mortality probability according to the total risk score in the derivation cohort and the external validation cohort. CKD–EPI, Chronic Kidney Disease Epidemiology Collaboration equation; eGFR, estimated glomerular filtration rate; SaO2, oxygen saturation.

Table 4

Prediction of 30-day mortality on presentation in hospitalised patients with COVID-19 according to the point score in the derivation cohort and in the external validation cohort

Risk score	Derivation cohort					External validation cohort
	Total	30-day mortality				Total	30-day mortality
		Yes		No			Yes		No
		N	%	N	%		N	%	N	%
0	48	1	2.1	47	97.9	20	0	0.0	20	100
1	139	0	0.0	139	100	68	0	0.0	68	100
2	193	3	1.6	190	98.4	104	0	0.0	104	100
3	215	10	4.7	205	95.3	103	0	0.0	103	100
4	230	11	4.8	219	95.2	109	1	0.9	108	99.1
5	254	16	6.3	238	93.7	107	4	3.7	103	96.3
6	235	25	10.6	210	89.4	112	5	4.5	107	95.5
7	237	32	13.5	205	86.5	80	8	10.0	72	90.0
8	200	39	19.5	161	80.5	63	8	12.7	55	87.3
9	191	53	27.7	138	72.3	42	8	19.0	34	81.0
10	136	39	28.7	97	71.3	45	12	26.7	33	73.3
11	133	45	33.8	88	66.2	45	11	24.4	34	75.6
12	94	36	38.3	58	61.7	26	5	19.2	21	80.8
13	91	40	44.0	51	56.0	18	7	38.9	11	61.1
14	75	32	42.7	43	57.3	19	5	26.3	14	73.7
15	80	32	40.0	48	60.0	27	9	33.3	18	66.7
16	83	36	43.4	47	56.6	32	10	31.3	22	68.8
17	123	48	39.0	75	61.0	40	14	35.0	26	65.0
18	97	51	52.6	46	47.4	49	16	32.7	33	67.3
19	104	55	52.9	49	47.1	41	13	31.7	28	68.3
20	96	50	52.1	46	47.9	23	9	39.1	14	60.9
21	74	51	68.9	23	31.1	17	6	35.3	11	64.7
22	44	24	54.5	20	45.5	17	7	41.2	10	58.8
23	37	23	62.2	14	37.8	12	4	33.3	8	66.7
24	33	20	60.6	13	39.4	15	8	53.3	7	46.7
25	23	14	60.9	9	39.1	13	5	38.5	8	61.5
26	33	17	51.5	16	48.5	9	4	44.4	5	55.6
27	25	14	56.0	11	44.0	8	6	75.0	2	25.0
28	20	19	95.0	1	5.0	3	1	33.3	2	66.7
29	9	7	77.8	2	22.2	2	2	100	0	0.0
30	6	6	100	0	0.0	0	0	0.0	0	0.0
Total	3358	849	25.3	2509	74.7	1269	188	14.8	1081	85.2

The AUROC (95% CI) of the simplified score for prediction of 30-day mortality was 0.806 (0.790 to 0.821) in the DC and 0.831 (0.806–0.856) in the external VC (online supplemental appendix table 2). The sensitivity, specificity, positive and negative predictive values, and likelihood ratios for the different scores in the DC and external VC are shown in table 5 and online supplemental appendix table 3, respectively.

Table 5

Simplified score to predict 30-day mortality in hospitalised patients with COVID-19 in the derivation cohort: sensitivity, specificity, likelihood ratios and predictive values for the different scores (0–30) in the derivation cohort

Score	Participants			Sen (%)	Spe (%)	+LR	1/-LR	PPV (%)	NPV (%)
	Total	Dying within 30 days
		N	%
0	48	1	2.1	100	0	1	–	25.3	–
1	139	0	0.0	99.9	1.9	1.018	15.900	25.6	97.9
2	193	3	1.6	99.9	7.4	1.079	62.940	26.7	99.5
3	215	10	4.7	99.5	15.0	1.171	31.810	28.4	98.9
4	230	11	4.8	98.4	23.2	1.280	14.040	30.2	97.6
5	254	16	6.3	97.1	31.9	1.425	10.830	32.5	97.0
6	235	25	10.6	95.2	41.4	1.623	8.567	33.5	96.2
7	237	32	13.5	92.2	49.7	1.835	6.398	38.3	95.
8	200	39	19.5	88.5	57.9	2.102	5.017	41.6	93.7
9	191	53	27.7	83.9	64.3	2.351	3.986	44.3	92.2
10	136	39	28.7	77.6	69.8	2.573	3.120	46.5	90.2
11	133	45	33.8	73.0	73.0	2.776	2.732	48.4	89.0
12	94	36	38.3	67.7	77.2	2.971	2.392	50.1	87.6
13	91	40	44.0	63.5	79.5	3.099	2.178	51.2	86.6
14	75	32	42.7	58.8	81.5	3.185	1.978	51.9	85.4
15	80	32	40.0	55.0	83.3	3.286	1.850	52.6	84.5
16	83	36	43.4	51.2	85.2	3.456	1.747	53.9	83.8
17	123	48	39.0	47.0	87.0	3.628	1.642	55.1	82.9
18	97	51	52.6	41.3	90.0	4.149	1.535	58.4	81.9
19	104	55	52.9	35.3	91.9	4.346	1.421	59.5	80.8
20	96	50	52.1	28.9	93.8	4.671	1.319	61.3	79.6
21	74	51	68.9	23.0	95.7	5.287	1.242	64.1	78.6
22	44	24	54.5	17.0	96.6	4.948	1.163	62.6	77.5
23	37	23	62.2	14.1	97.4	5.373	1.134	64.5	77.0
24	33	20	60.6	11.4	97.9	5.513	1.106	65.1	76.6
25	23	14	60.9	9.1	98.4	5.835	1.083	66.4	76.2
26	33	17	51.5	7.4	98.8	6.206	1.067	67.7	75.9
27	25	14	56.0	5.4	99.4	9.710	1.051	76.7	75.7
28	20	19	95.0	3.8	99.9	31.520	1.038	91.4	75.4
29	9	7	77.8	1.5	99.9	19.210	1.015	86.7	75.0
30	6	6	100	0.7	100	–	1.007	100	74.9

The number of individuals in different risk categories was low (0–2 points; 380 (11.3%)), medium (3–5 points; 699 (20.8%)), high (6–8 points; 672 (20.0%)) and very high (9–30 points; 1607 (47.9%)).

-LR, negative likelihood ratio; +LR, positive likelihood ratio; NPV, negative predictive value; PPV, positive predictive value; Sen, sensitivity; Spe, specificity.

We considered the risk of 30-day mortality as low with 0–2 points (0%–2.1%), moderate with 3–5 (4.7%–6.3%), high with 6–8 (10.6%–19.5%) and very high with 9–30 (27.7%–100.0%) (figure 2B). Kaplan-Meier survival plots for the different 30-day mortality risk categories according to the simplified score in the DC and VC are shown in online supplemental appendix figure 2.

Sensitivity analyses

Sensitivity analysis 1

When we generated the final prediction model recoding missing values for predictors as a separate category, the AUROC (95% CI) was 0.822 (0.809 to 0.836) in the DC and 0.850 (0.831 to 0.867) in the external VC. Likewise, when we applied the same approach to the simplified point score, the AUROC (95% CI) was 0.805 (0.791 to 0.820) in the DC and 0.848 (0.830 to 0.866) in the external VC (online supplemental appendix table 2).

Sensitivity analysis 2

When we applied the final prediction model to all patients, and missing values for predictors were left blank (equivalent to the lowest risk situation), the AUROC (95% CI) was 0.818 (0.805 to 0.832) in the DC and 0.859 (0.842 to 0.876) in the external VC. Likewise, when we applied the same approach to the simplified point score, the AUROC (95% CI) was 0.806 (0.791 to 0.820) in the DC and 0.849 (0.831 to 0.866) in the external VC (online supplemental appendix table 2).

Discussion

The COVID-19 SEIMC score for predicting 30-day mortality of patients attending hospital emergency rooms was developed and externally validated with two large datasets from patients hospitalised with laboratory-confirmed COVID-19 in Spain. The predictors were age, low age-adjusted SaO2, neutrophil-to-lymphocyte ratio, eGFR by the CKD-EPI equation, dyspnoea and sex. The model showed good performance in both the DC and the external VC and permitted an easy stratification of patients into four risk categories.

Our prediction model uses widely accessible clinical and laboratory data, and its simplicity would allow clinicians to perform rapid risk stratification of patients with COVID-19. Of note, our model does not take into account comorbidities, which have been associated with worse COVID-19 prognosis in descriptive studies and included in most prognostic prediction models reported to date.13 15–22 In our study, underlying diseases such as hypertension, obesity, liver cirrhosis, chronic neurological disorder, active neoplasia and dementia were independently associated with an increased risk of 30-day mortality. However, none of these conditions improved the model’s discrimination capacity and, following the principle of parsimony, were discarded.

Once again, our study highlights the extraordinary impact of age on COVID-19 mortality, which is, to the best of our knowledge, unparalleled in infectious diseases. For example, our score would classify a 65-year-old male patient attending the emergency room— regardless of the results of the other variables—as a high-risk category with a 30-day mortality probability that could reach up to 19.5%. For younger patients, our score also shows the importance of basic laboratory parameters. A 55-year-old man without dyspnoea, normal SaO2 and normal renal function but with a neutrophil-to-lymphocyte ratio higher than 6.33 would also be classified as high risk.

At the time of writing, an eight variable mortality score developed and validated in a UK prospective cohort of 57 824 patients admitted to hospital with COVID-19, the 4C Mortality Score, has been published.30 Some of the variables included in this score, such as respiratory rate, Glasgow Coma Scale score and urea, are not available in the COVID-19@Database precluding the cross-validation the 4C Mortality Score in our population.

Our study is limited, as is the case with other reported studies, by the retrospective capture of data. Another potential limitation is that it was based exclusively on predictors from patients attending hospital emergency rooms. However, we believe that our score could be applied in primary care settings if capillary SaO2 and routine laboratory tests such as blood counts and serum creatinine could be determined. Finally, our score was derived from hospitalised patients in a single country, raising the question about their transportability to other countries, a common limitation to all currently described prognostic models of COVID-19. We believe that it would be of interest to carry out cross-validation between the SEIMC COVID-19 score and other scores in a large multinational dataset.

Our study has several strengths. In contrast with the majority of prior published prognostic models, ours adhere to the TRIPOD statement’s recommendations. Besides, the large sample size and the high number of events in the DC minimise the risk of model overfitting, a general limitation of previous studies. Our model’s strengths also include the calibration, the internal validation by bootstrapping rather than by random split of the DC and the validation in a large external cohort. Finally, the sensitivity analyses exploring different approaches for missing values for predictors did not modify the model’s performance, suggesting that missing values in both cohorts occurred at random.

The SEIMC COVID-19 score could be a useful triage tool enabling quick decision-making for patients with COVID-19. For example, patients in the low-risk category are likely suitable for outpatient care, whereas hospital admission or intensive or high dependency care should be considered for patients in high and very high-risk categories. Besides, management in emergency department observation units or makeshift medicalised facilities could be considered for patients in the moderate risk category. Another potential application of the SEIMC COVID-19 score is the risk stratification of patients with COVID-19 in observational studies or clinical trials.

Our study showed that the COVID-19 SEIMC score, a simple prediction tool using readily available clinical and laboratory data results, could identify the probability of 30-day mortality with a high degree of accuracy among patients with COVID-19.