Categorization of COVID-19 severity to determine mortality risk.

PURPOSE: Algorithms for classification of inpatient COVID-19 severity are necessary for confounding control in studies using real-world data.
METHODS: Using Healthverity chargemaster and claims data, we selected patients hospitalized with COVID-19 between April 2020 and February 2021, and classified them by severity at admission using an algorithm we developed based on respiratory support requirements (supplemental oxygen or non-invasive ventilation, O2/NIV, invasive mechanical ventilation, IMV, or NEITHER). To evaluate the utility of the algorithm, patients were followed from admission until death, discharge, or a 28-day maximum to report mortality risks and rates overall and by stratified by severity. Trends for heterogeneity in mortality risk and rate across severity classifications were evaluated using Cochran-Armitage and Logrank trend tests, respectively.
RESULTS: Among 118 117 patients, the algorithm categorized patients in increasing severity as NEITHER (36.7%), O2/NIV (54.3%), and IMV (9.0%). Associated mortality risk (and 95% CI) was 11.8% (11.6-12.0%) overall and increased with severity [3.4% (3.2-3.5%), 11.5% (11.3-11.8%), 47.3% (46.3-48.2%); p < 0.001]. Mortality rate per 1000 person-days (and 95% CI) was 15.1 (14.9-15.4) overall and increased with severity [5.7 (5.4-6.0), 14.5 (14.2-14.9), 32.7 (31.8-33.6); p < 0.001].
CONCLUSION: As expected, we observed a positive association between the algorithm-defined severity on admission and 28-day mortality risk and rate. Although performance remains to be validated, this provides some assurance that this algorithm may be used for confounding control or stratification in treatment effect studies.

Key Points

Developed and described an inpatient COVID‐19 severity score with utility for real‐world data study designs.

In this observational cohort study, mortality risk and rate significantly increased with increasing severity using the newly developed algorithm.

The positive association between the algorithm‐defined severity and mortality provides some assurance that this algorithm may be used for confounding control or stratification in treatment effect studies.

Plain Language Summary

Algorithms for classification of inpatient COVID‐19 severity are necessary to conduct studies using real‐world data. We developed an algorithm to classify disease severity in hospitalized COVID‐19 patients based on respiratory support requirements (supplemental oxygen or non‐invasive ventilation, O2/NIV, invasive mechanical ventilation, IMV, or NEITHER). Using a cohort study, we evaluated the utility of the algorithm by determining if patients classified as having greater disease severity at admission are at higher risk for inpatient mortality. Among the 118,117 patients hospitalized with COVID‐19 from HealthVerity claims and chargemaster data between April 2020 and February 2021, the algorithm categorized patients in increasing severity as NEITHER (36.7%), O2/NIV (54.3%), and IMV (9.0%). Associated mortality risk was 11.8% overall and increased with severity (3.4% NEITHER, 11.5% O2/NIV, and 47.3% IMV). Mortality rate per 1000 person‐days was 15.1 overall and increased with severity (5.7 NEITHER, 14.5 O2/NIV, and 32.7 IMV). This provides some assurance that this algorithm may be used for confounding control or stratification in real world data studies.

INTRODUCTION

Following the emergence of coronavirus disease 2019 (COVID‐19) in the United States (US), there were over 30 million cases and over 550 000 related deaths reported during the first year alone. Patients who are hospitalized for COVID‐19 may require respiratory support including supplemental oxygen (O2) with or without noninvasive ventilation (NIV) or invasive mechanical ventilation (IMV) with intubation. In more severe cases when organs start to fail, additional support may be added to IMV to help the heart and lungs pump oxygen into the blood (extracorporeal membrane oxygenation [ECMO]), help the kidneys with filtration (renal replacement therapy), or improve blood and oxygen delivery to vital organs (vasopressors).

As indicators of COVID‐19 severity, O2, NIV, and IMV respiratory support requirements at admission may be critical measures of risk, prognosis, and severe outcomes, such as death. The U.S. Food and Drug Administration (FDA) has also issued guidance recommending that patients be classified according to baseline disease severity in all clinical trials aimed to determine the effectiveness of new COVID‐19 treatments and prevention. Independent of treatment, we expected an increased COVID‐19 severity level to be associated with an increased mortality risk.

The World Health Organization (WHO) developed a Clinical Progression Scale to classify COVID‐19 severity. While this scale was developed for determining patient outcomes, it may potentially be used to determine severity at the time of admission. However, the scale relies heavily on the availability of clinical information that may not always be available in real‐world data (RWD) ‐‐ health care data routinely collected, such as claims and billing activities or electronic health records (EHR). The FDA Sentinel Initiative also developed a practical severity score using RWD to classify patient severity ranging from asymptomatic to critical. However, the categorization utilizes data from the entire hospitalization and relies on day‐level diagnoses that are often unavailable or under‐recorded within the coded variables of inpatient data sources. Therefore, we developed an inpatient RWD algorithm, called the mWHO score to classify COVID‐19 severity (respiratory support requiring O2/NIV, IMV, or neither) that is a modified version of the WHO Scale and influenced by the FDA Sentinel score.

This study (which is a result of a research collaboration agreement between Aetion and FDA to use RWD to advance the understanding of the natural history and treatment of COVID‐19) estimates mortality risk and incidence rate in a cohort of patients hospitalized for COVID‐19 stratified by mWHO subgroups at admission. The aim was to establish the utility of the mWHO algorithm for use in confounding control or subgroup characterization in treatment effectiveness studies using RWD by demonstrating that patients with greater algorithm‐defined COVID‐19 disease severity at admission are at higher risk for mortality.

METHODS

Data Source

We used RWD from HealthVerity between 01‐April‐2020 and 25‐February‐2021 to conduct this study. The data are comprised of medical and pharmacy open claims (sourced in near‐real‐time from practice management systems, billing systems and claims clearinghouses) and closed claims (sourced from insurance providers and payers), laboratory test history and results, and chargemaster administrative hospital billing data for inpatient and outpatient encounters from all of the US states (see additional detail in Appendix ). The data include all major payer types (commercial, Medicaid and Medicare) and have been previously used for scientific publications of COVID‐19 research. , , , , , ,

mWHO COVID‐19 severity algorithm

The WHO Clinical Progression Scale scores a range of COVID‐19 severity outcomes from uninfected (score of 0) to dead (score of 10; see Table 1). We developed a modified version, referred to here as the mWHO score, that restricts to severity levels applicable to hospitalized COVID‐19 patients (WHO original scores of 4–9), collapsed into three mutually exclusive categories for neither O2/NIV nor IMV, O2 or NIV without IMV, and any IMV with or without additional support (NEITHER, O2/NIV, and IMV, respectively; see supplemental Appendix B for additional algorithm detail). These categories correspond to the FDA Sentinel's Moderate, Severe, and Critical categories, respectively, leaving out the Asymptomatic and Mild categories that apply only to non‐hospitalized patients. The algorithm includes procedure codes, diagnosis codes, and free‐text search terms to query the chargemaster data indicating the occurrence of respiratory support procedures. Diagnoses indicating a clinical need for O2/NIV (hypoxia or hypoxemia) or IMV (acute respiratory distress syndrome) were also added to the algorithm to increase the sensitivity of reporting these procedures.

TABLE 1

Comparison of WHO clinical progression scale to modified WHO score

WHO Clinical Progression Scale (Outcomes)			Modified WHO Score (Inpatient Baseline COVID‐19 Severity)
Patient state	Description	Score	Severity Level
Uninfected	Uninfected; no viral RNA detected	0	Not applicable for inpatient
Ambulatory mild disease	Asymptomatic; viral RNA detected	1
	Symptomatic; independent	2
	Symptomatic; assistance needed	3
Hospitalized: moderate disease	Hospitalized; no oxygen therapy	4	NEITHER
	Hospitalized; oxygen by mask or nasal prongs	5	O2/NIV (supplemental oxygen or non‐invasive ventilation)
Hospitalized: severe disease	Hospitalized; oxygen by NIV or high flow	6
	Intubation and mechanical ventilation, pO2/FiO2 ≥ 150 or SpO2/FiO2 ≥ 200	7	IMV (invasive mechanical ventilation)
	Mechanical ventilation pO2/FIO2 < 150 (SpO2/FiO2 < 200) or vasopressors	8
	Mechanical ventilation pO2/FiO2 < 150 and vasopressors, dialysis, or ECMO	9
Dead	Dead	10	Not applicable for baseline severity

Study population

We identified a study cohort of patients hospitalized with an admission date between 01‐April‐2020 and 25‐February‐2021, available chargemaster data (necessary for capture of inpatient medication use), confirmed COVID‐19 (ICD‐10 diagnosis of U07.1 or a positive NAAT SARS‐CoV‐2 laboratory result) on the admission date or within 21 days prior, and at least one medical encounter during the 183‐day baseline period (see supplemental Appendix C for study diagram). We excluded patients with missing sex, age, or geographic region and patients with any record of a COVID‐19 vaccine on or prior to admission. We then stratified the cohort into subgroups according to their greatest mWHO COVID‐19 severity level at admission (NEITHER, O2/NIV, and IMV). The day prior to admission was included to minimize potential misclassification from situations in which patients received O2 or IMV in other medical settings (e.g., emergency room or ambulance), or cases where the billing date for a procedure was captured at a later calendar date than the procedure was performed due to a lag in reporting. The day after admission was included when categorizing COVID‐19 severity at admission to account for patients who may be admitted late in the evening who do not have any record of respiratory support until the next morning.

Statistical analysis

This study reports descriptive statistics for patient characteristics determined a priori, including demographic characteristics upon admission, and comorbidities (individual, the combined comorbidity score, and the frailty index ), and medication use during the 183‐day baseline period. Follow‐up to ascertain 28‐day mortality, defined as an inpatient chargemaster encounter with a discharge status of “expired”, began on the admission date and continued until occurrence of the outcome, hospital discharge, or 28 days following admission. Mortality risk was calculated as the total number of qualifying mortality endpoints divided by the total number of patients in the cohort at any point in time over the entire study period and was reported per 100 patients with corresponding 95% confidence intervals (CI). Mortality incidence rate was calculated as the total number of qualifying mortality endpoints divided by the total follow‐up of all patients in the cohort and was reported per 1000 person‐days with corresponding 95% CIs. We also plotted cumulative incidence curves to confirm expected divergence among the mWHO subgroups. Descriptive statistics and the mortality risk and incidence rates are reported among the overall cohort and for each mWHO subgroup.

We conducted three sensitivity analyses to evaluate the robustness of our findings. The first analysis evaluated mortality risk and incidence rate for inpatient severity at admission, based on the greatest level of the following groups that more closely resembled those of the WHO Progression Scale: NEITHER, O2, NIV, IMV without additional organ support (vasopressors, dialysis, or ECMO), or IMV with additional organ support (see additional detail in supplemental Appendix D, Table D.1). The second analysis aimed to understand mortality trends over time by plotting the mortality rate each month of the study period. The third analysis aimed to confirm the utility of the mWHO algorithm during the early months of the COVID‐19 pandemic, when the disease landscape was still rapidly evolving, by evaluating mortality risks and incidence rates among a subset of patients admitted between 01‐April‐2020 and 31‐August‐2020.

Analyses were conducted using the Aetion Evidence Platform® (2021), a software for RWD analysis, validated for a range of studies. Trends for heterogeneity in mortality risk and incidence rate across subgroups were evaluated using Cochran‐Armitage and log rank trend tests, respectively, via the DescTools and survminer packages in R (v4.0.3).

Ethics

The study was approved under exemption by the New England Institutional Review Board.

RESULTS

Among 1 109 733 patients hospitalized with admission dates between 01‐April‐2020 and 25‐February‐2021, 12.3% had confirmed COVID‐19 (Figure 1). After application of the remaining exclusions, the cohort included 118 117 patients overall, 43 330 (36.7%) were in NEITHER, 64107 (54.3%) in the O2/NIV, and 10 680 (9.0%) in the IMV mWHO subgroups. Compared to patients in the NEITHER subgroup, patients in the O2/NIV and IMV subgroups were more likely to be older (median of 67 and 66 vs. 62 years old), male (50.5% and 56.4% vs. 44.1%), have Medicare or Medicaid coverage (59.0% and 61.6% vs. 57.2%), and less likely to be from the Northeast region (18.8% and 19.3% vs. 26.2%) (Table 2). Use of systemic corticosteroids (dexamethasone, methylprednisolone, prednisone, and hydrocortisone) was equally common among patients in the O2/NIV and IMV subgroups (32.6% and 32.5%) and higher than patients in the NEITHER subgroup (24.1%). Compared to patients in the NEITHER subgroup, patients in the O2/NIV and IMV subgroups were more likely to have a record of diabetes, coronary artery disease, chronic pulmonary disease, and hypertension. Baseline comorbidity and frailty scores were slightly higher among patients in the IMV subgroup as compared to O2/NIV and NEITHER subgroups, with minimal difference in the mean scores between O2/NIV and NEITHER patients. Baseline use of other potential COVID‐19 therapies occurred in less than 6% of patients overall, with increased use in IMV and O2/NIV subgroups as compared to NEITHER. In contrast, patients in the O2/NIV subgroup were less likely than those in the NEITHER subgroup to have liver and end‐stage kidney disease, and baseline use of antiplatelets, immunosuppressives, or anticoagulants.

FIGURE 1

Study cohort attrition of patients hospitalized for COVID‐19. IMV, invasive mechanical ventilation; NEITHER, neither O2/NIV nor IMV; O2/NIV, supplemental oxygen or noninvasive ventilation

TABLE 2

Select patient characteristics of patients hospitalized for COVID‐19, overall and stratified by mWHO severity subgroups

	Overall	mWHO COVID‐19 Severity Subgroup
		Neither	O2/NIV	IMV
Number of patients (% of total)	118 117 (100%)	43 330 (36.7%)	64 107 (54.3%)	10 680 (9.0%)
Demographic characteristics; n (%) unless otherwise specified
Age, mean (SD)	63.24 (18.94)	58.90 (21.80)	65.91 (16.74)	64.80 (15.41)
Age, med [IQR]	65 [52–76]	62 [43–75]	67 [56–77]	66 [57–75]
Male sex	57 545 (48.7%)	19 117 (44.1%)	32 400 (50.5%)	6028 (56.4%)
Insurance with commercial coverage	29 999 (25.4%)	10 786 (24.9%)	16 622 (25.9%)	2591 (24.3%)
Insurance with Medicare/Medicaid only	69 176 (58.6%)	24 795 (57.2%)	37 800 (59.0%)	6581 (61.6%)
Insurance type missing	18 942 (16.0%)	7749 (17.9%)	9685 (15.1%)	1508 (14.1%)
Northeast region	25 457 (21.6%)	11 347 (26.2%)	12 052 (18.8%)	2058 (19.3%)
Midwest region	10 413 (8.8%)	2863 (6.6%)	6458 (10.1%)	1092 (10.2%)
South region	53 717 (45.5%)	18 612 (43.0%)	30 726 (47.9%)	4379 (41.0%)
West region	28 530 (24.2%)	10 508 (24.3%)	14 871 (23.2%)	3151 (29.5%)
Select comorbidities during 183‐day baseline; n (%) unless otherwise specified
Diabetes diagnosis or medication	46 355 (39.2%)	15 117 (34.9%)	26 103 (40.7%)	5135 (48.1%)
Coronary artery disease	23 101 (19.6%)	7897 (18.2%)	12 879 (20.1%)	2325 (21.8%)
Chronic pulmonary disease	25 778 (21.8%)	7833 (18.1%)	15 245 (23.8%)	2700 (25.3%)
Liver disease	7884 (6.7%)	3207 (7.4%)	3807 (5.9%)	870 (8.1%)
End‐stage kidney disease	5587 (4.7%)	2187 (5.0%)	2837 (4.4%)	563 (5.3%)
Hypertension diagnosis or medication	80 936 (68.5%)	27 372 (63.2%)	45 464 (70.9%)	8100 (75.8%)
Comorbidity score, mean (SD) a	2.17 (2.98)	2.19 (2.93)	2.10 (2.97)	2.55 (3.21)
Comorbidity score, med [IQR] a	1 [0–4]	1 [0–4]	1 [0–3]	1 [0–4]
Frailty score, mean (SD) b	0.16 (0.06)	0.16 (0.06)	0.16 (0.06)	0.17 (0.07)
Frailty score, med [IQR] b	0.14 [0.11–0.19]	0.14 [0.11–0.19]	0.14 [0.11–0.19]	0.15 [0.12–0.20]
Medication use during 183‐day baseline; n (%)
Statins	39 287 (33.3%)	12 822 (29.6%)	22 287 (34.8%)	4178 (39.1%)
Antiplatelet medication	22 459 (19.0%)	8365 (19.3%)	11 926 (18.6%)	2168 (20.3%)
Immunosuppressive medication	4747 (4.0%)	1804 (4.2%)	2404 (3.7%)	539 (5.0%)
Anticoagulant	30 655 (26.0%)	12 073 (27.9%)	15 871 (24.8%)	2711 (25.4%)
Systemic corticosteroids c	34 821 (29.5%)	10 447 (24.1%)	20 905 (32.6%)	3469 (32.5%)
Other COVID‐19 therapies d	6741 (5.7%)	1954 (4.5%)	4085 (6.4%)	702 (6.6%)

Abbreviations: IMV, invasive mechanical ventilation; IQR, interquartile range; med, median; NEITHER, neither O2/NIV nor IMV; O2/NIV, supplemental oxygen or noninvasive ventilation; SD, standard deviation.

Defined according to the Combined comorbidity score developed by Gagne et al. (2011).

Defined according to the Frailty Index developed by Kim et al. (2018).

Comprised dexamethasone, methylprednisolone, prednisone, and hydrocortisone.

Patients could have initiated a potential COVID‐19 therapy prior to the index admission due to use for an indication other than COVID‐19, transfer from another hospital outside of the chargemaster network, transfer from the emergency department, or an inpatient or outpatient visit prior to qualifying for index admission.This category comprised the following listed in the NIH guidelines (2021) other than corticosteroids: hydroxychloroquine/ chloroquine, remdesivir, lopinavir/ritonavir and other HIV protease inhibitors, ivermectin, IL‐6 inhibitors (sarilumab, tocilizumab, siltuximab), JAK inhibitors (baricitinib, ruxolitinib, tofacitinib, upadacitinib), BTK inhibitors (acalabrutinib, ibrutinib, zanubrutinib), interferons (alfa‐2b or beta‐1a), COVID‐19 monoclonal antibody treatments (casirivimab/imdevimab, bamlanivimab, etesevimab), convalescent plasma, ivermectin, mesenchymal stem cells, fluvoxamine, and colchicine.

Among the 13 906 mortality endpoints overall, 1462 were among patients in the NEITHER, 7395 in the O2/NIV, and 5049 in the IMV subgroups (Table 3). This yielded an overall mortality risk of 11.8% (95% CI: 11.6–12.0%) that increased with increasing respiratory support requirements (severity) for NEITHER, O2/NIV, and IMV subgroups, with a significant trend for heterogeneity [NEITHER 3.4% (95% CI: 3.2–3.5%), O2/NIV 11.5% (11.3–11.8%), IMV 47.3% (46.3–48.2%); p < 0.001]. Overall, mortality incidence rate per 1000 person‐days was 15.1 (95% CI: 14.87–15.37) over a median (IQR) follow‐up of 5 (3–10) days and similarly increased with severity with a significant trend for heterogeneity [NEITHER 5.7 (95% CI: 5.4–6.0), O2/NIV 14.5 (14.2–14.9), and IMV 32.7 (31.8–33.6); p < 0.001], along with an increase in the median [IQR] follow‐up (4 [2-7], 6 [4-10], and 13 [6-23] days, respectively). A clear divergence between mWHO subgroups was also observed via cumulative incidence plots that held over the entire follow‐up (Figure 2).

TABLE 3

Risks and rates of 28‐day mortality among patients hospitalized for COVID‐19, overall and stratified by mWHO severity subgroups

	Overall	mWHO COVID‐19 Severity Subgroup
		Neither	O2/NIV	IMV
No. of patients in each (% of total)	118 117 (100%)	43 330 (36.7%)	64 107 (54.3%)	10 680 (9.0%)
No. of patients with a mortality endpoint	13 906	1462	7395	5049
Follow‐up (days) a , Total	919 713	256 223	509 185	154 306
Follow‐up (days) a , Median [IQR]	5 [3–10]	4 [2–7]	6 [4–10]	13 [6–23]
Mortality risk per 100 patients (95% confidence interval)	11.8% (11.6–12.0%)	3.4% (3.2–3.5%)	11.5% (11.3–11.8%)	47.3% (46.3–48.2%)
Mortality risk heterogeneity p for trend b	NA	p < 0.001
Mortality incidence rate per 1000 person‐days (95% confidence interval)	15.12 (14.87–15.37)	5.71 (5.41–6.00)	14.52 (14.19–14.85)	32.72 (31.82–33.62)
Mortality incidence rate heterogeneity p for trend c	NA	p < 0.001

Abbreviations: IMV, invasive mechanical ventilation; NEITHER, neither O2/NIV nor IMV; O2/NIV, supplemental oxygen or noninvasive ventilation.

Follow‐up began on the admission date and continued until occurrence of the outcome or was censored upon hospital discharge or at a maximum of 28 days.

Mortality risk heterogeneity trend evaluated via Cochran‐Armitage trend test.

Mortality incidence rate heterogeneity trend evaluated via Log rank trend test.

FIGURE 2

Cumulative incidence of mortality among the cohort of patients hospitalized for COVID‐19, stratified by mWHO severity subgroups [Colour figure can be viewed at wileyonlinelibrary.com]

In the first sensitivity analysis, using the five more granular severity subgroups that were like the WHO Progression Scale (supplemental Appendix D, Table D.2), the 64 107 patients in the original O2/NIV subgroup were categorized as either O2 [52 363 (81.7%)] or NIV [11 744 (18.3%)]. Among the 10 680 in the IMV subgroup, 6979 (65.3%) did not have additional organ system support, while 3701 (34.7%) did. Mortality risk continued to be associated with increasing severity and a significant trend for heterogeneity was observed [NEITHER 3.4% (95% CI: 3.2–3.5%), O2 9.1% (8.9–9.4%), NIV 22.4% (21.6–23.1%), IMV without additional support (IMV‐), 42.1% (40.9–43.3%), IMV with additional support (IMV+) 57.0% (55.4–58.6%); p < 0.001]. Similarly, mortality incidence rate per 1000 person‐days continued to have an increasing trend [NEITHER 5.7 (5.4–6.0, 12.1 (95% CI: 11.7–12.4), O2/NIV 23.0 (22.1–23.8), IMV‐ 28.4 (27.4–29.4), IMV+ 41.6 (39.8–43.4); p < 0.001].

In the second sensitivity analysis to examine mortality trends over time, we observed monthly mortality incidence rates that were generally stable over the study period, with a slight decline among patients in the NEITHER and O2/NIV subgroups and a slight incline among patients in the IMV subgroup (supplemental Appendix D, Figure D.1). In the third sensitivity analysis, the increasing trend for mortality risk and incidence rate stratified by the algorithm‐defined mWHO subgroups remained among patients hospitalized in the early months of the COVID‐19 pandemic (supplemental Appendix D, Table D.3).

DISCUSSION

The mWHO algorithm was developed to categorize inpatient COVID‐19 severity based on respiratory support requirements (O2/NIV, IMV, or NEITHER). To operationalize this concept, we included diagnoses that indicate a clinical requirement for O2/NIV or IMV in addition to procedure‐based encounters. While the mWHO algorithm has not been validated against medical records, we note agreement with expectations among factors known to be plausibly associated with both COVID‐19 severity and mortality, such as age, coronary artery disease, and chronic pulmonary disease within the three mWHO subgroups. Further agreement with expectations was observed via a positive association between the mWHO algorithm‐defined severity level at admission and 28‐day mortality risk and rate. A lack of overlap in 95% CIs of the risks or rates across subgroups with increasing severity and the heterogeneity tests further substantiate the increasing trend observed. This finding was expected given that the algorithm was designed to differentiate the degree of severity and therefore the associated risk of adverse outcomes, such as death. Sensitivity analyses also confirmed similar increasing trends in mortality risks and rates, providing further assurance that our algorithm was operating as anticipated.

Strengths and limitations of the HealthVerity administrative data were taken into consideration. First, although the open claims data has the benefit of near‐real‐time capture, it may be less complete for the most recent calendar dates. However, we truncated our study period to end 60 days prior to the last date of data available to minimize this concern (protocol available on clinicaltrials.gov, NCT04926571). When determining the cohort selection criteria, we required at least one medical encounter during the baseline period to help ensure adequate patient history for describing baseline patient characteristics. Although this may have excluded healthier patients that avoided the healthcare system due to the pandemic conditions, we expect this to have minimal impact on our findings, since this criterion excluded only 11% of patients hospitalized with COVID‐19. We also excluded patients with a record of receiving a COVID‐19 vaccine as they may have a selectively different mortality risk. Vaccine status is likely under‐reported in our data due to vaccines received outside of the healthcare system (e.g., free vaccine clinics). However, since the first vaccine received emergency use authorization in December 2020, only two months before the end of our study period (February 2021), we assume the impact of this missing data to be negligible. Given its invasiveness and cost, we assumed IMV would be well recorded in administrative (billing based) RWD. It is likely, however, that O2 and NIV procedures are not as well captured. Further, the data also does not include nursing notes, which have been shown in a previous study conducted by the FDA Sentinel System to increase capture of O2 use. Despite the potential for under‐capture of standardized procedure codes for O2/NIV in this study, we supplemented our algorithm with free‐text chargemaster data and diagnoses indicative of clinical need for O2/NIV to improve overall capture of disease severity.

Given that the HealthVerity data source is relatively new, there are no validation studies yet for mortality or other endpoints. However, discharge status to determine mortality endpoints among hospitalized COVID‐19 patients has been used in both in a similar chargemaster data source and for national surveillance reporting. In addition, we explored weekly mortality events over time as they compared to two external national benchmarks, the Centers for Disease Control and data sourced from State and local health agencies, and the similar trends we observed minimized potential concern for misclassification of mortality data in our study. As with most RWD sources, data indicating cause of death or the occurrence of death outside of the hospital is not available in the data. Further, we were unable to assess the distribution of deaths with COVID‐19 as the immediate cause of death and deaths occurring after discharge were considered outside the scope of the research question. The performance of the mWHO algorithm to identify each severity level on hospital admission remains to be validated against a “gold standard” such as manual review of medical records to quantify specificity and sensitivity. Further revisions to the algorithm may be considered moving forward. Although this study evaluates its utility to define COVID‐19 severity at admission and in only one administrative data source, the algorithm has the potential for use more broadly in other RWD sources using the code lists provided, and may be expanded to utilize additional data available during the hospitalization (after admission) as appropriate (e.g., ascertaining study outcomes).

This algorithm is an important addition to the COVID‐19 and pulmonary RWD literature. The positive association observed between the algorithm‐defined severity level at hospital admission and 28‐day inpatient mortality risk and incidence rate provide some assurance that this algorithm may be useful for severity level confounding control or subgroup characterization in treatment effectiveness studies using RWD.

DISCLOSURES

This paper is part of an unfunded research collaboration agreement between the U.S. Food and Drug Administration (FDA) and Aetion, Inc. to use real‐world data to advance the understanding and the natural history of coronavirus disease (COVID‐19) in specific patient populations, as well as treatment and diagnostic patterns during the COVID‐19 pandemic. This paper reflects the views of the authors and should not be construed to represent FDA's views or policies. EMG, ARW, SEV, IJE, JAR and NMG are employees of Aetion, Inc., with stock options or existing equity. This research was previously shared as a poster presentation at the 2021 FDA Science Forum. Science as the Foundation for Protecting and Promoting Public Health and as a virtual podium presentation in August 2021 at the 37th International Conference for Pharmacoepidemiology.

Appendix S1 Supporting information

Click here for additional data file.