The CCEDRRN COVID-19 Mortality Score to predict death among nonpalliative patients with COVID-19 presenting to emergency departments: a derivation and validation study.

BACKGROUND: Predicting mortality from COVID-19 using information available when patients present to the emergency department can inform goals-of-care decisions and assist with ethical allocation of critical care resources. The study objective was to develop and validate a clinical score to predict emergency department and in-hospital mortality among consecutive nonpalliative patients with COVID-19; in this study, we define palliative patients as those who do not want resuscitative measures, such as intubation, intensive care unit care or cardiopulmonary resuscitation.
METHODS: This derivation and validation study used observational cohort data recruited from 46 hospitals in 8 Canadian provinces participating in the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN). We included adult (age ≥ 18 yr) nonpalliative patients with confirmed COVID-19 who presented to the emergency department of a participating site between Mar. 1, 2020, and Jan. 31, 2021. We randomly assigned hospitals to derivation or validation, and prespecified clinical variables as candidate predictors. We used logistic regression to develop the score in a derivation cohort and examined its performance in predicting emergency department and in-hospital mortality in a validation cohort.
RESULTS: Of 8761 eligible patients, 618 (7.0%) died. The CCEDRRN COVID-19 Mortality Score included age, sex, type of residence, arrival mode, chest pain, severe liver disease, respiratory rate and level of respiratory support. The area under the curve was 0.92 (95% confidence interval [CI] 0.90-0.93) in derivation and 0.92 (95% CI 0.90-0.93) in validation. The score had excellent calibration. These results suggest that scores of 6 or less would categorize patients as being at low risk for in-hospital death, with a negative predictive value of 99.9%. Patients in the low-risk group had an in-hospital mortality rate of 0.1%. Patients with a score of 15 or higher had an observed mortality rate of 81.0%.
INTERPRETATION: The CCEDRRN COVID-19 Mortality Score is a simple score that can be used for level-of-care discussions with patients and in situations of critical care resource constraints to accurately predict death using variables available on emergency department arrival. The score was derived and validated mostly in unvaccinated patients, and before variants of concern were circulating widely and newer treatment regimens implemented in Canada. STUDY REGISTRATION: ClinicalTrials.gov, no. NCT04702945.

Throughout the COVID-19 pandemic, health care systems around the world have been confronted with, and at times overwhelmed by, high numbers of critically ill patients.1,2 For every critically ill patient in the emergency department, several less severely ill patients present for care; in some cases, their condition deteriorates later, which places additional pressure on resources. Accurate, disease-specific mortality prediction is needed to inform shared decision-making with patients and their families around the patients’ goals of care. Disease-specific mortality prediction can also allow health care systems to allocate resources in the most transparent, objective and fair manner possible to save as many lives as possible and facilitate timely access to palliative care, if needed.3–5

Numerous models have been developed to predict mortality from COVID-19, but most are at high risk of bias.6–8 Many were developed in small or nonrepresentative patient samples; enrolled patients from early in the pandemic, before evidence-based treatments had been identified; included palliative patients (i.e., those who do not want resuscitative measures, such as intubation, intensive care unit care or cardiopulmonary resuscitation); censored outcomes; and had moderate predictive performance.6 The ISARIC 4C Mortality Score is the strongest model developed.9 However, it includes palliative patients, which limits its utility in stratifying risk among nonpalliative patients with COVID-19. In addition, the ISARIC 4C Mortality Score was developed based on data from early in the pandemic. Most other published rules involve the use of imaging or laboratory tests,9–14 which precludes their use as a first-line triage tool in the emergency department, where decisions on the appropriateness of intubation and mechanical ventilation may have to be made on arrival.

Our objective in the present study was to develop and validate a clinical score that accurately predicts mortality among nonpalliative patients with COVID-19, using clinical variables that are readily available on emergency department arrival.

Methods

Study design and setting

This study is a derivation and validation of a clinical decision rule based on observational data. Model development and reporting followed TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) standards.15 Our protocol is registered with clinicaltrials.gov (NCT04702945).

Study setting

The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) is an ongoing multicentre pan-Canadian registry that enrols consecutive eligible patients with COVID-19 presenting to emergency departments in hospitals located in 8 of the 10 Canadian provinces, including the 4 most populous (only Prince Edward Island and Newfoundland and Labrador are not represented).16 The network conducts observational studies and develops clinical decision rules to inform emergency department care.17–19 The CCEDRRN patient engagement committee reviewed and provided input into the development of the research question, the choice of outcomes and the study protocol, and reviewed the submitted manuscript. Patient partners were involved in developing CCEDRRN’s website and knowledge translation tools to disseminate study results (https://www.ccedrrn.com/).

Participants

Participating sites needed to show at least 99% compliance in enrolling consecutive eligible patients for their data to be included in this study. At each site, we used trackers employing medical microbiology lists of all tested patients to identify all eligible patients. A national coordinator reviewed the lists of all sites on a weekly basis to ensure compliance. We included data from 46 of 50 CCEDRRN sites that met this criterion by the time of the data cut (Appendix 1, Table S1, available at www.cmajopen.ca/content/10/1/E90/suppl/DC1). The characteristics, resource use and outcomes of patients with COVID-19 included in the registry are posted as a preprint.20

We included patients with confirmed COVID-19 who presented to the emergency department of a participating site between Mar. 1, 2020, and Jan. 31, 2021. We defined confirmed COVID-19 as disease in patients who presented with ongoing COVID-19 symptoms and a positive result of a nucleic acid amplification test (NAAT) for SARS-CoV-2 obtained within 14 days before their arrival in the emergency department or after their arrival, which reflects the maximal incubation period.21 This allowed us to capture patients who were diagnosed in the community and subsequently presented to the emergency department, and those with early false-negative test results that became positive. We also included patients who presented with COVID-19 symptoms and were diagnosed with “confirmed COVID-19” to capture patients who were transferred to a CCEDRRN hospital and whose NAAT at the sending site could not be confirmed, and patients who were presumed by treating clinicians to have COVID-19 despite persistently negative NAAT results.

We excluded patients less than 18 years of age and those whose goals of care precluded invasive mechanical ventilation in order to not bias the study with self-fulfilling prophecy bias,22 whereby the prognostic model predicts the outcome that occurred as a result of a decision to withhold life-sustaining measures.22–24 We also excluded patients transferred to a hospital outside of CCEDRRN, as we would have been unable to ascertain their outcomes. We followed patients for 30 days if they were discharged from the emergency department or until hospital discharge if their stay lasted longer than 30 days.

Data sources

Trained research assistants abstracted data from electronic and paper-based medical records into a central, Web-based REDCap database. They captured patient demographic characteristics, vital signs, symptoms, comorbidities, COVID-19 exposure risk, diagnostic test results and outcomes. The research assistants recorded the first set of vital signs and symptoms documented in the emergency department record. The lowest oxygen saturation level was the lowest level recorded in the emergency department or by paramedics (some sites transcribed the vital signs obtained in the ambulance into their triage forms) with the patient receiving oxygen. We abstracted code status from the patients’ medical records. If a patient’s code status was not documented, we assumed that the status was full code.

We evaluated the interrater agreement of key predictor variables by comparing data collected retrospectively with prospective data.16 We developed the clinical prediction score after all chart abstraction was complete; research assistants were thus unaware of which clinical variables would be candidate predictor variables.

Outcome

The primary outcome was all-cause emergency department and in-hospital mortality. All patients had complete follow-up data at the time of the data cut. We categorized patients who were discharged from hospital as alive according to their latest hospital admission.

Predictor variables

All candidate predictor variables were recorded in the emergency department record. We selected candidate predictor variables based on a literature review of risk factors for COVID-19 mortality, review of other COVID-19 risk-prediction models, the availability of candidate predictor variables on patient arrival and the clinical knowledge of the investigator team.9,10,12,13,25–29 The candidate predictor variables included age, sex, pregnancy, type of residence, mode of arrival at the emergency department, comorbidities, symptoms, heart rate on arrival, systolic blood pressure, oxygen saturation level, respiratory rate, Glasgow Coma Scale score, oxygen delivery in the emergency department, lowest oxygen saturation level, physician or nurse impression of respiratory distress, and use of alcohol, tobacco, vaping or illicit substances (Appendix 1, Table S2).

Statistical analysis

Sample size

Assuming an event rate of less than 10%, shrinkage of 0.9 and a conservative Cox–Snell r2 of 0.1, we estimated that 8.5 events per degree of freedom were required for reliable prediction modelling in the derivation cohort.30 The 42 candidate predictor variables had 49 degrees of freedom, which indicated that 417 events were required. In the derivation cohort, there were 6758 patients who made 7420 emergency department visits. The derivation cohort had 471 events, which exceeded the number of events required.

Model development and validation

We randomly assigned participating sites to derivation or validation, with the goal of assigning 75% of eligible patients and outcome events to derivation, and the remaining to validation. We examined candidate predictors for collinearity and missing and extreme values in the derivation cohort. A few variables had missing values (systolic blood pressure had the most missing, at 4.7%) (Appendix 1, Table S2). We used 5 multiple imputations for predictors if missing categoric data could not reasonably be assumed to be absent (e.g., missing documentation of illicit substance use was classified as no substance use) and used these imputed data sets to develop the final model. The initial logistic regression model considered all candidate predictors, with continuous predictors fit with restricted cubic splines with 3 knots.

We assessed the strengths of associations between predictor variables and the outcome using an analysis of variance plot to inform the degrees of freedom to allocate to each predictor. No additional knots were allocated to continuous predictors. We used a fast step-down procedure to reduce the model to key predictors (fastbw function in the R rms package [R Foundation for Statistical Computing]). We conducted an internal bootstrap validation with 1000 bootstrap samples to provide an optimism-corrected area under the receiver-operating characteristic (AUC). We conducted this internal validation separately for each imputed data set and combined the imputation-specific AUCs using Rubin’s rules on the logit scale.31,32

We produced a nomogram from the reduced model to assign points. To enable easy clinical use, we categorized age into decades (as the nomogram points were sensitive to age), and cut points of 20 and 30 for respiratory rate on arrival based on the relation between its spline function and outcome. The resulting score ranged from −1 to 17. We calculated sensitivity and specificity at various point thresholds, along with the score’s discrimination and calibration, using one of the imputed data sets.

We validated the model in a cohort of geographically distinct sites that were not part of derivation and used a single imputation for the few missing respiratory rates (4%). We assessed outcomes independently for emergency department visits, irrespective of potential subsequent visits leading to death. We performed analyses in R using the rms and pROC packages, and used the pmsampsize package for sample size determination. To ensure patient privacy, a cell size restriction policy prohibited reporting counts of less than 5.

Validation of previously published models

We used our study cohort to externally validate other risk-prediction tools: the SEIMC score,11 the 4C Mortality Score10 and the VACO Index.33 We chose these 3 because they performed well in validation yet had critical methodologic differences that limit their translation to emergency department patients with suspected COVID-19: their cohorts included palliative patients, which may have introduced self-fulfilling prophecy bias,22 and they were from early in the pandemic, when COVID-19 testing was restricted to those with severe disease, which may have introduced spectrum bias. To facilitate comparison with our rule, we calculated their AUCs in our cohort using patients with complete data on as many predictors as possible. We compared AUCs using paired DeLong tests if the complete data for each score had the same patients and unpaired tests if patients with complete data differed by score.34

Ethics approval

This study was approved by the research ethics boards of all participating institutions with a waiver of informed consent for enrolment.

Results

We assessed 9704 consecutive patients with COVID-19 who made 10 670 emergency department visits between Mar. 1, 2020, and Jan. 31, 2021 (Figure 1). We excluded 943 patients who met 1 or more exclusion criteria, and included 8761 patients who made 9605 visits in our analyses. The follow-up time was 30 days for discharged patients and 30–229 days for admitted patients. Of the 8761 patients, 618 (7.0%) died in the emergency department or hospital. The characteristics and outcomes of patients in the derivation (n = 6758) and validation (n = 2054) cohorts are shown in Table 1.

Figure 1:

Flow diagram showing included and excluded emergency department (ED) visits. *Numbers total more than 8761 because 51 patients made visits to multiple EDs, some of which were derivation EDs and some of which were validation EDs.

Table 1:

Characteristics and outcomes of nonpalliative patients with COVID-19 presenting to emergency departments in the derivation and validation cohorts

Characteristic/outcome	Cohort; no. (%) of visits*
	Derivationn = 7420	Validationn = 2185
Age, mean ± SD, yr	54.7 ± 19.8	53.6 ± 19.9
Female sex	3544 (47.8)	1149 (52.6)
Province†
Quebec	2992 (40.3)	369 (16.9)
British Columbia	1839 (24.8)	291 (13.3)
Alberta	1718 (23.2)	775 (35.5)
Ontario	732 (9.9)	379 (17.4)
Saskatchewan	75 (1.0)	329 (15.1)
Nova Scotia	64 (0.7)	33 (1.51)
New Brunswick	< 5	9 (0.41)
Arrival from
Home	6639 (89.5)	2005 (91.8)
Institution	631 (8.5)	120 (5.5)
No fixed address	108 (1.5)	60 (2.8)
Missing	42 (0.6)	0 (0.0)
Arrival mode
Self	4432 (59.7)	1236 (56.6)
Ambulance or police	2987 (40.3)	948 (43.4)
Missing	1 (0.0)	1 (0.0)
Infection risk
Household or caregiver contact	1064 (14.3)	253 (11.6)
Institutional exposure (e.g., long-term care, prison)	828 (11.2)	152 (7.0)
Health care worker	385 (5.2)	107 (4.9)
Travel from country with known cases within 14 d	296 (4.0)	78 (3.6)
Unknown/not documented	4847 (65.3)	1595 (73.0)
Arrival heart rate, mean ± SD, beats/min	93.4 ± 18.6	92.6 ± 18.1
Arrival systolic blood pressure, mean ± SD, mm Hg	131.0 ± 21.1	130.9 ± 20.4
Arrival diastolic blood pressure, mean ± SD, mm Hg	78.2 ± 13.0	78.9 ± 13.0
Arrival respiratory rate, mean ± SD, breaths/min	21.0 ± 6.2	20.6 ± 5.5
Arrival temperature, mean ± SD, °C	37.2 ± 0.9	37.0 ± 0.9
Presence of respiratory distress	1567 (21.1)	446 (20.4)
10 most common COVID-19 symptoms
Cough	3938 (53.1)	1247 (57.1)
Shortness of breath (dyspnea)	3616 (48.7)	1096 (50.2)
Fever	3216 (43.3)	832 (38.1)
Fatigue/malaise	1992 (26.9)	697 (31.9)
Chest pain (includes discomfort or tightness)	1606 (21.6)	497 (22.8)
Headache	1263 (17.0)	415 (19.0)
Nausea/vomiting	1201 (16.2)	468 (21.4)
Chills	1194 (16.1)	339 (15.5)
Muscle aches (myalgia)	1136 (15.3)	384 (17.6)
Diarrhea	1016 (13.7)	373 (17.1)
10 most common comorbidities
Hypertension	2179 (29.4)	625 (28.6)
Diabetes	1247 (16.8)	319 (14.6)
Dyslipidemia	1161 (15.7)	252 (11.5)
Mental health diagnosis	670 (9.0)	270 (12.4)
Hypothyroidism	547 (7.4)	141 (6.5)
Asthma	535 (7.2)	186 (8.5)
Coronary artery disease	487 (6.6)	114 (5.2)
Chronic neurologic disorder (not dementia)	423 (5.7)	101 (4.6)
Chronic lung disease (not asthma or pulmonary fibrosis)	379 (5.11)	132 (6.0)
Dementia	362 (4.9)	86 (3.9)
Smoking or vaping
Current	573 (7.7)	190 (8.7)
Past or never	6847 (92.3)	1995 (91.3)
Illicit substance use
Current	130 (1.8)	83 (3.8)
Past or never	7290 (98.3)	2102 (96.2)
Oxygen required in emergency department	1341 (18.1)	302 (13.8)
Emergency department disposition
Discharged	4488 (60.5)	1320 (60.4)
Admitted	2858 (38.5)	848 (38.8)
Left against medical advice	18 (0.2)	6 (0.3)
Died	22 (0.3)	6 (0.2)
Missing	34 (0.5)	5 (0.2)
Died in emergency department or in hospital	471 (6.4)	147 (6.7)

Note: SD = standard deviation.

Except where noted otherwise.

No participating site in Manitoba showed at least 99% compliance in enrolling consecutive eligible patients by the time of the data cut.

The patients in the derivation cohort made 7420 emergency department visits to 32 different sites (Appendix 1, Table S1), and those in the validation cohort made 2185 emergency department visits to 14 different sites. In the derivation cohort, 2705 visits (36.5%) occurred early during the pandemic (between Mar. 1 and June 30, 2020), and 4715 (63.5%) occurred between July 1, 2020, and Jan. 31, 2021. The corresponding values for the validation cohort were 618 (28.3%) and 1567 (71.7%).

In derivation, the step-down procedure produced a final model with 8 variables (Table 2). The derived model had an optimism-corrected AUC of 0.92. The resulting risk score ranged from −1 to 17. When the CCEDRRN COVID-19 Mortality Score was used as the only predictor in the logistic regression model, the intercept was −9.049, and the slope for the risk score was 0.652 (Table 3). The derivation cohort was well distributed across the score range and had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination, with an AUC of 0.92 (95% confidence interval [CI] 0.90–0.93) (Appendix 1, Figure S1).

Table 2:

Adjusted associations between predictor variables and death, and points for the CCEDRRN COVID-19 Mortality Score

Variable	Estimate (SE)	OR (95% CI)	Points
Age, yr*	2.85 (0.00–41.00)	17.30 (7.75–38.60)
< 40			0
40–49			4
50–59			5
60–69			6
70–79			7
≥ 80			8
Sex
Male	Ref	Ref	1
Female	−0.61 (0.12)	0.54 (0.43–0.69)	0
Arrival from
Home or community	Ref	Ref	0
No fixed address	0.17 (0.63)	1.19 (0.35–4.09)	0
Institution	0.59 (0.14)	1.80 (1.38–2.35)	1
Arrival mode
Self	Ref	Ref	0
Ambulance or police	0.63 (0.15)	1.89 (1.41–2.52)	1
Chest pain	−0.80 (0.24)	0.45 (0.28–0.72)	−1
Moderate/severe liver disease	1.94 (0.50)	6.95 (2.61–18.50)	2
Arrival respiratory rate, breaths/min†	0.29 (0.10)	1.34 (1.09–1.63)
< 20			0
20–29			2
≥ 30			3
Mode and level of oxygen in emergency department
No oxygen	Ref	Ref	0
Nasal prongs, < 6 L/min	0.70 (0.14)	2.00 (1.53–2.62)	1
Face mask, simple rebreather or nasal prongs, ≥ 6 L/min	1.94 (0.19)	6.98 (4.79–10.16)	2
BiPAP/CPAP/HFNO	2.56 (0.27)	12.98 (7.62–22.08)	3
Intubation	2.53 (0.29)	12.50 (7.11–21.98)	3

Note: BiPAP = bilevel positive airway pressure, CCEDRRN = Canadian COVID-19 Emergency Department Rapid Response Network, CI = confidence interval, CPAP = continuous positive airway pressure, HFNO = high-flow nasal oxygen, OR = odds ratio, Ref = reference category, SE = standard error.

Odds ratio was calculated for the upper quartile (70 yr) versus the lower quartile (39 yr).

Odds ratio was calculated for the upper quartile (22 breaths/min) versus the lower quartile (18 breaths/min).

Table 3:

Performance of the CCEDRRN COVID-19 Mortality Score to rule out and rule in in-hospital mortality at various cut-off values in the combined derivation and validation cohorts, and the predicted mortality risk for each value based on the CCEDRRN COVID-19 Mortality Score*

Score	No. (%) of visitsn = 9605	Sensitivity (95% CI), %	Specificity (95% CI), %	Negative LR	Positive LR	PPV, %	NPV, %	Mortality rate, %	Score	Predicted risk with score
Rule out
≤ −1	178 (1.8)	100.0 (99.4–100.0)	2.0 (1.7–2.3)	0.0	1.0	6.6	100	0.0	−1	0.0001
≤ 0	825 (8.6)	100.0 (99.4–100.0)	9.2 (8.6–9.8)	0.0	1.1	7.0	100	0.0	0	0.0001
≤ 1	1499 (15.6)	100.0 (99.4–100.0)	16.7 (15.9–17.5)	0.0	1.2	7.6	100	0.0	1	0.0002
≤ 2	1926 (20.0)	100.0 (99.4–100.0)	21.4 (20.6–22.3)	0.0	1.3	8.0	100	0.0	2	0.0004
≤ 3	2420 (25.2)	100.0 (99.4–100.0)	26.9 (26.0–27.9)	0.0	1.4	8.6	100	0.0	3	0.0008
≤ 4	2910 (30.3)	100.0 (99.4–100.0)	32.4 (31.4–33.4)	0.0	1.5	9.2	100	0.0	4	0.0016
≤ 5	3685 (38.4)	99.8 (99.1–100.0)	41.0 (40.0–42.0)	0.0	1.7	10.4	100	0.0	5	0.0031
≤ 6	4563 (47.5)	99.0 (97.9–99.6)	50.7 (49.7–51.7)	0.0	2.0	12.1	99.9	0.1	6	0.0058
≤ 7	5497 (57.2)	97.4 (95.8–98.5)	61.0 (60.0–62.0)	0.0	2.5	14.7	99.7	0.3	7	0.0112
≤ 8	6410 (66.7)	94.0 (91.8–95.7)	70.9 (70.0–71.9)	0.1	3.2	18.2	99.4	0.6	8	0.0212
≤ 9	7276 (75.8)	88.5 (85.7–90.9)	80.2 (79.3–81.0)	0.1	4.5	23.5	99.0	1.0	9	0.0399
≤ 10	7974 (83.0)	77.7 (74.2–80.9)	87.2 (86.5–87.9)	0.3	6.1	29.4	98.3	1.7	10	0.0739
Rule in
≥ 10	2329 (24.2)	88.5 (85.7–90.9)	80.2 (79.3–81.0)	0.1	4.5	23.5	99.0	23.5	–	–
≥ 11	1631 (17.0)	77.7 (74.2–80.9)	87.2 (86.5–87.5)	0.3	6.1	29.4	98.3	29.4	11	0.1327
≥ 12	1083 (11.3)	65.0 (61.1–65.0)	92.4 (91.9–93.0)	0.4	8.6	37.1	97.5	37.1	12	0.2271
≥ 13	604 (6.3)	46.3 (42.3–50.3)	96.5 (96.1–96.8)	0.6	13.1	47.4	96.3	47.4	13	0.3605
≥ 14	276 (2.9)	28.2 (24.6–31.9)	98.9 (98.6–99.1)	0.7	24.8	63.0	95.2	63.0	14	0.5197
≥ 15	100 (1.0)	13.1 (10.5–16.0)	99.8 (99.7–99.9)	0.9	62.0	81.0	94.4	81.0	15	0.6750
≥ 16	29 (0.3)	4.4 (2.9–6.3)	100.0 (99.9–100.0)	1.0	196.3	93.1	93.8	93.1	16	0.7995
≥ 17	5 (0.0)	0.8 (0.3–1.9)	100.0 (100.0)	1.0	–	100.0	93.6	100.0	17	0.8844

Note: CCEDRRN = Canadian COVID-19 Emergency Department Rapid Response Network, CI = confidence interval, LR = likelihood ratio, NPV = negative predictive value, PPV = positive predictive value.

Predicted risk = exp (−9.048 + 0.652 * CCMS)/(1 + exp [−9.049 + 0.652 * CCMS]), where CCMS = CCEDRRN COVID-19 Mortality Score.

The CCEDRRN COVID-19 Mortality Score had similar performance in validation. The validation cohort was also distributed across the score ranges, had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination (AUC 0.92, 95% CI 0.90–0.93) (Figure 2).

Figure 2:

Distribution and performance of the CCEDRRN COVID-19 Mortality Score in the validation cohort (left panel) and combined derivation and validation cohorts (right panel): (A) distribution of the score, (B) observed in-hospital mortality rates across the range of the score, (C) predicted versus observed risk of in-hospital death (dashed line represents line of no difference between predicted and observed risk) and (D) receiver-operating characteristic curve with area under the curve and associated 95% confidence interval.

The score had excellent performance across a range of thresholds to rule in and rule out in-hospital mortality (Table 3; Appendix 1, Table S4). These results suggest that a score of 6 or less would categorize patients as being at low risk for in-hospital death, with a negative predictive value of 99.9%. Patients in the low-risk group had an in-hospital mortality rate of 0.1%. For scores of 15 or higher, the observed in-hospital mortality rate was 81.0%, and the CCEDRRN COVID-19 Mortality Score would categorize patients as being at high risk for in-hospital death, with a specificity of 99.8%.

Validation of previously published models

The 3 risk scores performed well for the patients with data available, yielding AUCs of nearly 0.88, less than that of the CCEDRRN COVID-19 Mortality Score. Unadjusted for multiple testing and with the use of the validation cohort, the AUC for the CCEDRRN COVID-19 Mortality Score (n = 2185) was higher than that for the SEIMC score (p = 0.04, n = 1620) and the VACO Index (p = 0.002, n = 2185); there was no evidence of a difference for the 4C Mortality Score (p = 0.07, n = 610) (Appendix 1, Figure S2, Table S5).

Interpretation

We derived and validated a parsimonious, simple score to predict in-hospital mortality among nonpalliative patients with COVID-19 presenting to emergency departments: the CCEDRRN COVID-19 Mortality Score. We found that 8 readily available clinical variables that can be ascertained at the bedside without any diagnostic testing predicted mortality accurately. The score had excellent calibration and discrimination in a geographically distinct cohort of patients who presented to other sites. It can be used as a highly sensitive score to rule out in-hospital mortality in patients at low risk with a score of 6 or less. In patients with a score of 15 or higher, the risk of in-hospital death is very high despite maximum medical therapy.

The CCEDRRN COVID-19 Mortality Score has strengths compared to prior models.6 We developed the model using simple variables that are readily available at the bedside. In contrast to prior models, we excluded patients with palliative goals of care, for whom invasive mechanical ventilation was not offered, to ensure our model did not predict risk of death among patients who were expected to die or were ineligible for the highest level of critical care.9–11,13,14,27,28,33,35,36 This avoids the potential for self-fulfilling prophecy bias.22 Prior models were derived or validated early in the pandemic, when COVID-19 testing was restricted to those with severe disease, and they did not include consecutive eligible patients; both these factors may have resulted in selection bias. The mortality rate in prior studies ranged from 13% to 30%,9,11,12,14,27,28,36 in contrast to 7% in our study. Our study was also able to capture patients who were readmitted to CCEDRRN sites and subsequently died in hospital.

Critically ill patients with COVID-19 typically require aggressive medical management in the emergency department shortly after their arrival. Being able to accurately predict mortality risk before endotracheal intubation can offer the opportunity to have discussions about patients’ goals of care and facilitate early high-quality end-of-life care for those most likely to die. Accurate mortality prediction may be essential when surging cases threaten to overwhelm critical care resources. In those rare situations, the CCEDRRN COVID-19 Mortality Score can guide allocation of scarce resources.

Our rule’s predictive ability depends mostly on age and respiratory status. The only comorbidity retained in the final model was moderate to severe liver disease. Two other rules also identified liver disease as a risk factor for death, perhaps owing to the potential for virus-induced liver inflammation. 14,33,37 Other prognostic decision rules have used similar analytic approaches27,28,33 but had lower predictive performance, with C-statistics ranging from 0.80 to 0.82, and were based on patients from early in the pandemic. Other rules incorporate measures of hypoxemia or respiratory support, which is indicative of their strong predictive power.9–13,27,28,33,35,36 Goodacre and colleagues28 identified physical performance status as a risk factor for increased mortality. In our data set, this was most closely reflected by arrival from long-term care, which indicated patients with lower performance status.

Limitations

Canada is a culturally diverse country that offers its citizens publicly funded health coverage for necessary physician and hospital care. The CCEDRRN COVID-19 Mortality Score needs to be validated externally in other health care systems. Our model could be validated externally for use in remote areas without access to laboratory testing or imaging, and in low-income countries. It predicts in-hospital mortality and may have missed deaths that occurred at home or deaths among discharged patients who returned to non-CCEDRRN hospitals. We captured readmissions to CCEDRRN sites, although it is unlikely that they had an influence on our model given their small numbers.20

Our model was based on patients with confirmed COVID-19. Although many presented to the emergency department with NAAT-confirmed COVID-19, validation in a cohort of patients with suspected COVID-19 is needed. We excluded patients if they were transferred to a non-CCEDRRN institution, as we would have been unable to ascertain their outcomes. However, the numbers of such patients were small, and their exclusion likely did not have a significant effect on the model. Our results apply to unvaccinated patients. The performance of risk tools also needs to be evaluated in patients who are fully or partially vaccinated and may have a different risk of dying from COVID-19 than unvaccinated patients.

We were unable to link our data set with vaccine registry data. However, by Jan. 31, 2021, only 0.3% of Canadians, most of whom were health care workers or residents of long-term care, had been fully vaccinated.38 Most residents of long-term care were excluded from our study owing to palliative code status. Thus, our results are applicable to unvaccinated patients. The performance of the CCEDRRN COVID-19 Mortality Score needs to be evaluated in a cohort of vaccinated patients, who may have a different risk of dying from COVID-19 than unvaccinated patients. The score also needs to be validated in cohorts infected with variants of concern and in a more recent data set, given newer treatment regimens that may affect mortality risk.

Conclusion

The CCEDRRN COVID-19 Mortality Score is a simple clinical risk score that can be applied at the bedside in the emergency department to predict unvaccinated patients’ mortality risk. There have been changes in treatment and emergence of new variants that may limit its utility. This tool can be used to inform goals-of-care decisions.