External validation of the COVID-19 4C Mortality Score in an urban United States cohort.

BACKGROUND: Identifying patients at risk for mortality from COVID-19 is crucial to triage, clinical decision-making, and the allocation of scarce hospital resources. The 4C Mortality Score effectively predicts COVID-19 mortality, but it has not been validated in a United States (U.S.) population. The purpose of this study is to determine whether the 4C Mortality Score accurately predicts COVID-19 mortality in an urban U.S. adult inpatient population.
METHODS: This retrospective cohort study included adult patients admitted to a single-center, tertiary care hospital (Philadelphia, PA) with a positive SARS-CoV-2 PCR from 3/17/2020 to 6/06/2020. Variables were extracted through a combination of automated export and manual chart review. The outcome of interest was mortality during hospital admission or within mortality 30 days of discharge.
RESULTS: This study included 426 patients; mean age was 64.4 years, 43.4% were female, and 54.5% self-identified as Black or African American. All-cause mortality was observed in 71 patients (16.7%). The area under the receiver operator characteristic curve of the 4C Mortality Score was 0.85 (95% confidence interval, 0.79-0.89).
CONCLUSIONS: Clinicians may use the 4C Mortality Score in an urban, majority Black, U.S. inpatient population. The derivation and validation cohorts were treated in the pre-vaccine era so the 4C Score may over-predict mortality in current patient populations. With stubbornly high inpatient mortality rates, however, the 4C Score remains one of the best tools available to date to inform thoughtful triage and treatment allocation.

Introduction

The management of patients with COVID-19 is challenging for front-line healthcare providers given limited validated, evidence-based clinical decision support. Determining patient mortality risk is critical for effective triage, management, and discharge decision making. Numerous COVID-19 risk prediction models of varying country of origin, population demographic, and robustness exist.1, 2, 3 The 4C Mortality Score, created by the International Severe Acute Respiratory and emerging Infections Consortium (ISARIC), is one of the largest-scale, high-performing predictive models published to date.

The initial 4C Mortality Score study included adult patients hospitalized with COVID-19 across 260 hospitals in England, Wales, and Scotland, utilizing eight weighted variables – age, sex, select comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale (GCS), blood urea nitrogen (BUN), and C-reactive protein (CRP) – to predict 30-day inpatient mortality. The area under receiver operator characteristic curve (AUC) of the derivation (n=35,463) and validation (n=22,361) cohorts was 0.79 (95% confidence interval [CI], 0.78-0.79) and 0.77 (95% CI, 0.76-0.77), respectively. This model outperformed 16 risk stratification scores, including SOFA, NEWS, CURB-65 and four novel COVID-19 risk scores.

The 4C Mortality Score has since been validated in a London cohort (n=8239; AUC=0.68 [95% CI, 0.67-0.69]); a Dutch emergency department cohort (n=403; AUC=0.84 [95% CI, 0.79-0.88]); a Brazilian and Spanish cohort (n=1363; AUC = 0.78 [95% CI, 0.75-0.81]); and an Italian cohort of greater than 60 years old (n=210, AUC=0.80 [95% CI, 0.74-0.85]). For United States clinicians to confidently utilize the 4C Mortality Score, however, the model must be validated in a population similar to its intended implementation, particularly in areas with minorities under-represented in prior research. The objective of this study is to determine whether the 4C Mortality Score is a clinically appropriate model to predict hospital mortality in United States patients with COVID-19.

Methods

Study population

Patients included in this study had a positive SARS-CoV-2 PCR test result and were admitted through the emergency department to Thomas Jefferson University Hospital (TJUH), a tertiary care academic medical center in Philadelphia, PA, between March 1st and June 6th, 2020. Symptomatic and asymptomatic patients were included. Patients transferred from outside hospitals were excluded. Patients under the age of 18, pregnant, and/or incarcerated were excluded from this study. This study was approved by the TJUH Institutional Review Board (IRB#: 20E.737).

Variables and outcome

The following 4C Mortality Score comorbidities were extracted from electronic health records (EHR) through manual chart review: chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic kidney disease (estimated glomerular filtration rate ≤ 30 mL/min/1.73m2), liver disease, connective tissue disease, diabetes mellitus, HIV/AIDS, active malignancy, and obesity (body mass index [BMI] ≥ 30 kg/m2). These data were obtained from the last discharge summary, the emergency department note, the admission history and physical, the EHR “Past Medical History” and/or “Active Problems” sections. All comorbidities were retrieved by two separate researchers; a third independent reviewer adjudicated data discrepancies. The study database was created prior to the release of the 4C Mortality Score; therefore, two comorbidities – dementia and “chronic neurological disease ” – were not included in this study. Respiratory rate, peripheral oxygen saturation, GCS, BUN, and CRP, within 24 hours of initial hospital encounter were digitally exported. The initial hospital encounter was defined as the first time-value of vitals entered in the EHR for the admission of interest. The primary outcome was a composite outcome of mortality during admission or mortality within 30 days of discharge. Mortality status was determined by chart review which included deaths occurring at other hospitals using the same EHR (the three largest academic medical centers in Philadelphia are included). This study is reported consistently with the Transparent Reporting of a multivariate prediction model for Individual Prognosis or Diagnosis guidelines.

Statistical analysis

Data was missing for less than 1.4% of variables. Values were missing for four variables of the 4C Score: CRP, GCS, BUN, and obesity (Table 1 ). Missing data was assumed to be missing at random. Multiple imputation methods were applied to the 4C Score components using R (Version 4.0.4, R Project for Statistical Computing) and the Multivariate Imputation by Chained Equations package (MICE, v3.13.0). Twenty imputed datasets were created, using predictive mean matching for numerical variables (e.g., CRP) and logistic regression for binary variables (e.g., obesity).

TABLE 1

Demographics, outcomes, and clinical variables of the study cohort and 4C Mortality Score cohort.

Characteristic	External Validation Cohort		4C Score Derivation Cohort
	Number of patients (% of total)	% Missing	Number of patients (% of total)
Age	-	0	-
<50	91 (21.4)	-	4876 (13.8)
50-69	168 (39.4)	-	10183 (28.9)
70-79	68 (16.0)	-	8017 (22.7)
≥80	99 (23.2)	-	12 201 (34.6)
Sex at birth	-	0	-
Female	184 (43.4)	-	14741 (41.7)
Male	241 (56.6)	-	20615 (58.3)
Ethnicity	-	0	-
Black	232 (54.5)	-	1256 (3.9)
White	109 (25.6)	-	26300 (82.2)
Other ethnicity	85 (19.9)	-	2513 (7.9)
Outcomes
Mortality	71 (16.7)	0	11426 (32.2)
ICU Status	85 (19.9)	-	-
Discharge to Hospice	14 (3.3)	-	-
Comorbities	No. of pts. (%)	% Missing	No. of pts. (%)
Chronic cardiac disease	123 (28.8)	0	10513 (31.8)
Chronic respiratory disease	75 (17.6)	0	5830 (17.7)
Chronic kidney disease	41 (9.6)	0	5653 (17.2)
Active cancer	28 (6.6)	0	3312 (10.2)
Liver disease	17 (4.0)	0	604 (1.9)
Obesity	172 (40.4)	1.2	3414 (11.4)
Diabetes	162 (38.0)	0	8487 (26.0)
Number of Comorbities
0	94 (22.1)	-	8497 (24.0)
1	142 (33.3)	-	9941 (28.0)
≥2	190 (44.6)	-	17025 (48.0)
Continuous variables	Median (IQR)	% Missing	Median (IQR)
Respiratory rate (breaths/min)	20 (5)	0	22 (9)
Oxygen saturation (%)	96 (5)	0	94 (6)
Glasgow Coma Scale	15 (0)	7.7	15 (0)
Blood urea nitrogen (mg/dL)	18.0 (19.0)	1.4	19.6 (17.6)
C-reactive protein (mg/dL)	5.9 (8.2)	11.7	8.5 (12.2)

Adjusted 4C Mortality Scores were computed for each record using the data in each imputed dataset. Mortality probability for each 4C Score was sourced from the ISARIC online 4C calculator (https://isaric4c.net/risk/v2/). Receiver operating characteristic curves were calculated for each imputed dataset comparing the predicted 4C Score and actual mortality for each record. Area under the receiver operator curve and Brier score, another measure of goodness of fit (ranging from 0 to 1; smaller values indicate superior calibration), were calculated for each curve. AUC, corresponding 95% CI, and Brier score were pooled for all 20 imputation results using Rubin's rule.

Results

Population demographics and outcomes

There were 426 patients included in this study. Mean age was 64.4 years; 43.4% were female; 54.5% were Black or African American (hereafter “Black”). Cohort demographics, clinical variables, and outcomes are presented in Table 1. The primary outcome of mortality during admission or within 30 days from discharge occurred in 16.7% of patients (71/426 patients). Of the 71 deaths, 10 occurred after discharge. Of deaths occurring after the index admission, the average time from index admission to death was 13.9 days (minimum: 7 days; maximum 21 days). The incidence of other measures of clinical interest (i.e., ICU status, discharge to hospice) can also be seen in Table 1.

Performance of 4C mortality score

The AUC for the pooled dataset was 0.85 (95% CI, 0.79-0.89). The Brier score for the dataset was 0.246. The calibration curve for all imputed datasets is plotted in Fig. 1 ; calibration curves for each individual imputation are plotted in the Supplemental Materials.

FIGURE 1

Predicted (4C Mortality Score) vs. actual mortality for all imputed datasets.

Discussion

With an AUC of 0.85 (95% CI, 0.79-0.89) and Brier score of 0.246, the 4C Mortality Score performed well in a United States urban population with a majority Black patient cohort. These findings are consistent with the results of external validation studies in other countries.6, 7, 8 , , We note a non-statistically significant trend toward overprediction of mortality, particularly visually noticeable in 4C scores <15 as seen in Fig. 1. This may be non-significant data variation, but suggests the need for further research in larger United States populations to investigate. We suspect that the 4C Score will tend to over-predict mortality in current populations given the increased availability of vaccines and effective outpatient and inpatient treatments relative to the study population (prior to June 7, 2020).12, 13, 14, 15, 16, 17, 18, 19, 20 Given stubbornly high inpatient mortality rates to date, however, this assumption may not hold true.

Notably, this dataset utilized manual chart review, which was shown to outperform automated EHR data export for the sensitivity of capturing comorbidities within this study population. This may have led to higher scores in our validation cohort relative to the original derivation cohort or other validation cohorts. Further strengths of this study include the representation of the racial diversity of the center's patient population, with a majority of the study cohort identifying as Black (54.5%). Recent research into the association of race, socioeconomic status, and COVID-19 outcomes highlights the necessity for including under-represented minorities in the validation of risk models.22, 23, 24

There are notable limitations to this study, including the limited generalizability of a validation performed in a single, tertiary care center. The absence of dementia and “chronic neurological disease” may have yielded lower risk scores for patients with 0-1 other comorbid conditions. Additionally, the original 4C Mortality Score used clinician-defined obesity, whereas this study used a BMI of > 30 kg/m2. We anticipate that the effect of missing comorbidities is opposed by the increased sensitivity of manual chart review in identifying comorbidities, mitigating any significant effect on model calibration, especially considering that comorbidities account for a small portion of the 4C Mortality Score. The primary outcome or the original 4C Mortality Score was 30-day in-hospital mortality, as compared to mortality during admission or within 30 days of discharge. As the intent of our validation of the 4C Mortality Score was to evaluate its clinical utility, we chose to include patients who died within 30 days of discharge under the assumption that they were likely to have died of complicated related to their index admission; that the average time from index admission to death was 13.9 days appears to support this assumption. Given these limitations, we believe the overall conclusion is still appropriate and reinforces the importance of clinicians and public health leaders thoughtfully applying the 4C Score to current specific scenarios.

The 4C Mortality Score is a clinically useful tool to assess risk for COVID-19 mortality as validated in an early 2020 United States urban tertiary care center. The validation of the 4C Mortality Score in a majority Black population is important for providers in many United States urban and rural centers to confidently use this tool for clinical decision making. The score is particularly useful for emergency department triage, as well as patient-specific treatment decision making (e.g., monoclonals, oral antivirals, remdesivir dosing, etc.) for scenarios in which treatment regimens are not clearly defined in national guidelines. Given the vaccination efforts and improved treatment options since this study cohort, the 4C Mortality Score should be applied thoughtfully by clinicians, carefully accounting for differences between an individual patient's clinical scenario and the populations and timing of our validation cohort. Further external validation of COVID-19 risk prediction tools in cohorts including vaccinated and unvaccinated patients receiving more recent treatment regimens would be of great value to clinicians. Ultimately, the 4C Mortality Score provides one of the best evidence-based approaches to date for patient triage, treatment choice and duration, and discharge planning to best serve patients afflicted by COVID-19.

Funding

None.

Conflicts of Interest

The authors have no conflicts of interest to disclose.