Machine Learning as a Precision-Medicine Approach to Prescribing COVID-19 Pharmacotherapy with Remdesivir or Corticosteroids.

PURPOSE: Coronavirus disease-2019 (COVID-19) continues to be a global threat and remains a significant cause of hospitalizations. Recent clinical guidelines have supported the use of corticosteroids or remdesivir in the treatment of COVID-19. However, uncertainty remains about which patients are most likely to benefit from treatment with either drug; such knowledge is crucial for avoiding preventable adverse effects, minimizing costs, and effectively allocating resources. This study presents a machine-learning system with the capacity to identify patients in whom treatment with a corticosteroid or remdesivir is associated with improved survival time.
METHODS: Gradient-boosted decision-tree models used for predicting treatment benefit were trained and tested on data from electronic health records dated between December 18, 2019, and October 18, 2020, from adult patients (age ≥18 years) with COVID-19 in 10 US hospitals. Models were evaluated for performance in identifying patients with longer survival times when treated with a corticosteroid versus remdesivir. Fine and Gray proportional-hazards models were used for identifying significant findings in treated and nontreated patients, in a subset of patients who received supplemental oxygen, and in patients identified by the algorithm. Inverse probability-of-treatment weights were used to adjust for confounding. Models were trained and tested separately for each treatment.
FINDINGS: Data from 2364 patients were included, with men comprising slightly more than 50% of the sample; 893 patients were treated with remdesivir, and 1471 were treated with a corticosteroid. After adjustment for confounding, neither corticosteroids nor remdesivir use was associated with increased survival time in the overall population or in the subpopulation that received supplemental oxygen. However, in the populations identified by the algorithms, both corticosteroids and remdesivir were significantly associated with an increase in survival time, with hazard ratios of 0.56 and 0.40, respectively (both, P = 0.04). IMPLICATIONS: Machine-learning methods have the capacity to identify hospitalized patients with COVID-19 in whom treatment with a corticosteroid or remdesivir is associated with an increase in survival time. These methods may help to improve patient outcomes and allocate resources during the COVID-19 crisis.

Introduction

Faced with the emergence of severe acute respiratory syndrome coronavirus (SARS-CoV)-2 and the novel disease it causes, coronavirus disease 2019 (COVID-19), the scientific and medical communities have raced to identify, test, and implement effective treatments. As infection and mortality rates within the United States continue to climb, these efforts remain critically important to national public health. The effects of >400 therapeutics are being studied, with >300 currently in clinical trials. Potential treatments have been identified based on historical effectiveness against related diseases, theoretical activity at key points for SARS-CoV-2 infection and COVID-19 disease progression, and in vitro antiviral activity.4, 5, 6 Findings from research to date support the use of corticosteroids, such as dexamethasone, or the antiviral remdesivir for the treatment of specific subpopulations of patients with COVID-19.

Corticosteroids and other immunosuppressant medications were initially identified as candidate treatments for patients with COVID-19 based on the finding of a dysregulated systemic inflammatory response in severe cases. , This phenomenon, sometimes referred to as a "cytokine storm," is thought to provoke the most severe clinical manifestations of COVID-19, including the need for mechanical ventilation or other oxygen support, thromboembolic events (as the pro-inflammatory and pro-coagulant stimulation pathways are interconnected), acute respiratory distress syndrome, acute cardiac injury, and multiorgan dysfunction. Clinical trials have since demonstrated that treatment with a corticosteroid may benefit patients with severe manifestations of COVID-19 that are suggestive of this inflammatory hyperactivation. The RECOVERY (Randomised Evaluation of COVID-19 Therapy) trial reported that, in a subgroup that received mechanical ventilation or other supplemental oxygen, the 28-day mortality rate was 3% lower with dexamethasone than with standard of care (P < 0.05); this benefit was not observed in the subgroup that did not require respiratory support. Based on findings from this study and similar clinical research, the current clinical guidelines from the National Institutes of Health and the Infectious Diseases Society of America recommend dexamethasone and related corticosteroids in the treatment of patients with severe COVID-19, defined as decreased oxygenation on room air or the requirement of supplemental oxygen or mechanical ventilation. , However, ambiguity remains about optimal dosing strategies, contraindications for treatment, and the effectiveness of corticosteroids in the treatment of different subpopulations of patients with COVID-19. , 13, 14, 15

Remdesivir, a nucleotide analogue that inhibits RNA-dependent RNA polymerase, was developed in 2009 for the treatment of patients with hepatitis C virus or respiratory syncytial virus, and was subsequently repurposed for the treatment of patients with Ebola viral infection. , Based on in vitro evidence of antiviral activity against SARS-CoV-2 and other single-stranded RNA viruses, patients with COVID-19 were treated with remdesivir under the US Food and Drug Administration's Compassionate Use authorization, and clinical trials were designed to test its efficacy.

In observational research and clinical trials, including the recently completed ACTT-1 (Adaptive Covid-19 Treatment Trial), treatment with remdesivir was associated with a reduced recovery time, but not with consistently reduced mortality across all time points after administration in any subgroup of patients with COVID-19.16, 17, 18 Improvements with remdesivir treatment have been most pronounced among patients with severe respiratory COVID-19, defined as a requirement of oxygen support. As a result, clinical guidelines in the United States recommend treatment with remdesivir only in patients within this category; remdesivir is not currently recommended for use in patients with COVID-19 who do not require supplemental oxygen. , However, even among patients requiring oxygen support, variability in the response to treatment with remdesivir has been noted.

For example, a recent meta-analysis revealed that remdesivir treatment was associated with a higher recovery rate and a lower mortality rate in patients who received noninvasive oxygen support relative to patients on mechanical ventilation; however, an interim analysis of data from the World Health Organization's (WHO) Solidarity trial did not show a mortality benefit with remdesivir among any patient group, including those who received respiratory support. While the findings from Solidarity must be interpreted in light of its limitations in methodology, the findings have fueled ongoing debate as to whether patients with COVID-19 would benefit from the use of remdesivir, and recently led WHO to amend its own clinical guidelines to recommend against the use of remdesivir in hospitalized patients with COVID-19, due to a perceived lack of consistent evidence of efficacy. As with corticosteroids, uncertainty thus remains about the effectiveness of remdesivir across different patient populations, and the patient populations most appropriate for treatment. ,

Precision medicine offers a potential avenue for addressing remaining questions about treatment efficacy of, as well as a means of identifying responsive populations with the use of, COVID-19 therapeutics. Precision-medicine approaches have successfully improved patient outcomes in other clinical areas.23, 24, 25, 26 Machine learning (ML) represents a means by which the potential effectiveness of specific treatments in a given individual may be predicted. ML has been applied to diverse tasks related to COVID-1928; however, within the context of therapeutics, this technology has overwhelmingly been used for identifying not which patients are most likely to experience a survival benefit, but rather which novel and repurposed drugs may be effective in treating patients with COVID-19.28, 29, 30, 31, 32, 33, 34 To fill this gap, we present a pair of ML algorithms (MLAs) to encourage precision-medicine treatment with remdesivir or dexamethasone and related corticosteroids in patients with COVID-19, using readily available data derived from electronic health records (EHRs).

Participants and Methods

Data Processing and Machine-Learning Models

Two MLAs were developed and trained to predict survival times with corticosteroids and remdesivir. Algorithms were trained on a dataset from patients with COVID-19 admitted to 9 US hospitals (Table I ). Use of these deidentified data was approved by an independent institutional review board (protocol 20-DASC-121; Pearl IRB, Indianapolis, Indiana), including a waiver for obtaining patient consent for the inclusion of data in the study.

Table I

Hospital characteristics for included data.

Characteristic	No. of Hospitals
Geographic region
Northeast	4*
South	2
Midwest	1
West	3
Hospital size
Small (<175 beds)	3
Medium (175–275 beds)	4
Large (>275 beds)	3

Two of the clinical sites in the Northeast were within the same health care system. All other clinical sites are from distinct, unrelated health care systems.

Eligible patients had a length of stay of >4 hours and, if treated, treatment within 2 days (corticosteroids) or 7 days (remdesivir) of admission.

The corticosteroid algorithm was trained on data from patients admitted between December 18, 2019, and March 1, 2020. Data from patients admitted between March 2, 2020, and October 18, 2020 (826 of 1471 patients [56%]), were set aside into a holdout test set.

Given the more recent approval and subsequent availability of remdesivir, the remdesivir algorithm was trained on data from patients admitted between March 1, 2020, and June 15, 2020. Data from patients admitted between June 16, 2020, and October 18, 2020 (185 of 893 patients [21%]), were set aside into a holdout test set.

Input Features

Data on the first 4 hours after hospital admission were extracted from the EHRs. Data used for generating predictions included age, sex, vital sign measurements (temperature, respiratory rate, peripheral oxygen saturation, heart rate, systolic and diastolic blood pressure), laboratory results (blood pH; concentrations of glucose, creatinine, blood urea nitrogen, bilirubin, and hemoglobin; hematocrit; red and white blood cell counts; percentages of lymphocytes and neutrophils; and platelet count), timing of COVID-19 diagnosis (early vs late in hospitalization or prior to hospitalization), need for oxygen support (via supplemental oxygen or mechanical ventilation), and medical history (myocardial infarction, congestive heart failure, peripheral vascular disease, cardiovascular disease, chronic obstructive pulmonary disease, pneumonia, rheumatologic disease, renal disease, diabetes mellitus with or without complications, and/or cancer). These predictive factors were chosen to make use of a wide variety of commonly collected data present in the EHR, including relevant comorbid medical conditions.

Machine Learning

The architecture of each MLA was a gradient-boosted decision tree, implemented using the XGBoost library (Apache Software Foundation, apache.org) in the Python programming language. The XGBoost method iteratively trains collections of gradient-boosted decision trees to classify training data. Each step incorporates a new decision tree, which preferentially weights the correct classification of previously misclassified training examples. XGBoost progressively builds on the loss generated by weak decision-tree base learners, learns quickly and effectively from large amounts of data, and learns even from missing features. The XGBoost method was chosen for this study due to its simplicity, high performance, and useful implementation features, which provide options for handling imbalanced classes and regularization. The XGBoost method combines results from various decision trees to generate prediction scores. Each tree has several branches. Each branch splits the patient population into successantly smaller groups based on their individual feature values. For example, a branch might send a patient along one of two directions depending on whether a patient's creatinine is >1.2 or <1.2 mg/dL. If the creatinine value is missing, the model chooses the branching direction that, on average, results in the better prediction. Additionally, a single decision tree might contain multiple creatinine branching points, such as one that comes after a male branching point and one that comes after the female branching point. This would allow for two different cutoff values for creatinine, conditioned on the sex of the patient. At the end of the decision tree, each patient encounter was represented in one "leaf" of the tree, with the patients in each leaf predicted to have the same risk for mortality with the given drug (remdesivir model vs corticosteroid model).

The task of predicting responsiveness to treatment was multifactorial, and clinical improvement was dependent on several important factors unrelated to treatment. However, it was still possible to design a target for the MLA for the purpose of training the MLA to extract any signal present in the clinical data that may have improved the ability of the model to predict the outcome of interest (ie, treatment responsiveness). In the present study, treatment responsiveness was predicted through a binomial logistic objective that was used for predicting positive class (improved disease if treated vs worsened disease if not treated) and negative class (worsened disease if treated vs improved disease if not treated). For our purposes, improved disease was defined as a last recorded oxygen saturation of ≥95%, or survival (defined as discharged alive), and worsened disease was defined as a last recorded oxygen saturation of <95%, or death.

Within the training dataset, 3-fold cross-validation was used for selecting model hyperparameters. In both MLAs, final hyperparameters were: a base score of 0.5, a learning rate of 0.1, a maximum depth of 3, and a regularization penalty of 1.0. When trained in this manner, the AUCs of the prediction of positive and negative class were 0.57 for remdesivir and 0.65 for corticosteroids. Unlike the standard use of AUCs in MLAs, which is to gauge the performance of MLAs in the diagnosis of disease and in which an AUC of >0.85 indicates reasonable decision making, in this case, the AUC was used simply for gauging whether any signal at all (AUC >0.5) could be extracted for assisting in the prediction of survival benefit (ie, increased survival time) with treatment. As a signal was found, we proceeded with model implementation and survival analysis.

Treatment Ascertainment

For the development of each algorithm, we classified patients as uniquely treated or nontreated with the relevant pharmaceutical (corticosteroids or remdesivir). Patients were classified as treated with a corticosteroid if they received IV or PO treatment with dexamethasone, prednisone, prednisolone, methylprednisolone or hydrocortisone in the first 2 days following hospital admission, or with remdesivir if they received treatment within the first 7 days following hospital admission. Data from patients who received these drugs beyond the initial specified treatment windows were excluded from analysis.

Outcome Ascertainment

The outcome of interest was survival time (measured in days). Algorithms were trained on the training set to identify patients for whom treatment was associated with an increase in survival time. For training purposes only, patients were classified as having survived if they were discharged alive to any setting, and as not having survived if their discharge disposition was dead.

To expand the number of patients included in the training set, those in whom survival status could not be ascertained were included. In these patients, oxygen saturation was used as a proxy for survival outcome. If the final recorded oxygen saturation prior to discharge was ≥95%, patients were classified as having survived, while patients with a final recorded oxygen saturation of <95% were classified as not having survived. This method was chosen because the proxy outcome is correlated with survival in the appropriate direction.

Patients were included in the test dataset only if their survival status could be ascertained. The discharge disposition (survived vs not survived) was ascertained in each patient, as was time to death in patients who experienced in-hospital mortality.

Covariates

To control for confounding by indication, information on several patient characteristics was extracted from the EHR. These characteristics included demographics (age, sex, race, institution at which the patient received care), vital sign measurements (temperature, respiratory rate, peripheral oxygen saturation, heart rate, systolic and diastolic blood pressure), laboratory results (white blood cell count, platelet count, glucose, blood pH, lactate, D-dimer), comorbid diagnoses (cardiovascular disease, hypertension, long QT interval, chronic pulmonary disease (asthma or pulmonary fibrosis), chronic obstructive pulmonary disease, pneumonia, acute respiratory distress syndrome, cancer including metastatic cancer, obesity, hypoglycemia, acute kidney injury, rheumatologic disease, diarrhea, and/or sepsis), medications (insulin, β-agonists, β-antagonists, angiotensin II receptor blockers, angiotensin-converting enzyme inhibitors, macrolide antibiotics, any antibiotics, statins, NSAIDs and hydroxychloroquine), location of COVID diagnosis (community or the hospital), and oxygen requirement status (supplemental oxygen or mechanical ventilation). More specific diagnostic groups were used for controlling for confounding, while more general diagnostic groups were used for model-training purposes. Given that some of these diagnoses were relatively rare in the datasets, reliance on them for model-training purposes may have biased the model toward better performance in those in whom more granular data were available. However, to accurately control for confounding, we prioritized the use of specific diagnoses in cases in which they were available.

Statistical Analysis

Each algorithm was applied to the holdout test set of COVID-19–positive patients 4 hours after inpatient admission. All performance metrics reported herein are from the test dataset, which was not seen by the model during the training process. No performance metrics on the training dataset have been included.

The performance of the 2 algorithms (corticosteroids and remdesivir) in identifying patients for whom treatment was associated with an increase in survival time was measured using a time-to-event analysis. Survival time was measured through adjusted hazard ratios (HRs). Adjustment for confounding was appropriate given that sicker patients were generally more likely to have received treatment with either of the drugs for which the algorithms were developed. Adjusted covariates varied by treatment, as described in detail subsequently.

Survival analysis was performed using a comparison of the survival times in the full population of treated and nontreated patients, and in the subpopulation of patients who received supplemental oxygen (a more critically ill population, and a population for whom corticosteroids and remdesivir are explicitly recommended per current clinical guidelines). The analyses were then repeated in the population of patients indicated by the algorithm.

To control for confounding, we constructed stabilized inverse probability-of-treatment weights (IPTWs) separately for each treatment. IPTWs were constructed using gradient-boosted decision trees, as this method implicitly handles missing data prevalent in EHR information. This method also allowed for the inclusion of a larger number of covariates than regression methods generally allow, enabling us to make use of all available patient data. All variables listed in the Covariates section were used for constructing the IPTWs for each treatment; each participant was weighted by the IPTWs in the time-to-event models. To mitigate the effects of any misspecification in a model in the IPTWs, all adjustment covariates were also included in the final time-to-event models. The event of interest was time to in-hospital mortality; hospital discharge was therefore treated as a competing event under a Fine-Gray framework for competing risks. Fine-Gray survival models for the subdistribution hazard allow for a direct estimate of the cumulative prevalence of in-hospital mortality despite the presence of a competing event; this in turn allows for the computation of HRs in the presence of competing events. Analyses were performed, and are presented, separately for the corticosteroids and remdesivir models. We examined the associations between each treatment and mortality in unadjusted models (eg, models containing neither adjustment covariates nor IPTWs) and adjusted time-to-event models. For all analyses, the level of significance was set at α = 0.05.

In addition to assessing survival time, we evaluated the model inputs using Shapley Additive Explanation values to determine which features were most strongly associated with model predictions. Shapley Additive Explanation is a method of quantifying the contribution of an individual feature when that feature interacts with several other features in determining the output. The method considers the model predictions with and without the individual feature, in the context of different combinations of other features and other branching orders of features.

Results

Data from 2364 patients were included, with men comprising slightly more than 50% of the sample; 893 patients were treated with remdesivir, and 1471 were treated with a corticosteroid.

Corticosteroids

In total, 826 patients were included in the corticosteroid algorithm test set, 525 of whom received supplemental oxygen and 616 of whom were indicated by the algorithm as suitable for treatment with a corticosteroid (Table II ). In the full population, 200 were treated with a corticosteroid, while 174 of those who received supplemental oxygen and 161 of those indicated by the algorithm received a corticosteroid. Patients were more likely to be Hispanic in the general population than in the population recommended for corticosteroid treatment (P = 0.03). No other differences between the general population and the population for which the MLA recommended corticosteroid treatment were statistically significant (P > 0.1 for all other comparisons).

Table II

Demographic characteristics of test set used for evaluating machine-learning algorithm recommendations on corticosteroid treatment of COVID-19. Data are given as the number (%) of patients.

Characteristic	All Patients (N = 826)	Patients with Indication for Corticosteroid Treatment (n = 616)
Age group
18–44 y	172 (20.8)	126 (20.5)
45–64 y	267 (32.3)	211 (34.3)
65–79 y	253 (30.6)	19 (31.5)
≥80 y	134 (16.2)	86 (13.7)
Sex
Male	445 (53.9)	331 (53.8)
Female	381 (46.1)	284 (46.2)
Race/ethnicity
Hispanic	271 (32.8)	165 (26.8)
White	79 (9.6)	54 (8.8)
Black	8 (1.0)	6 (1.0)
Asian	0	0
Other	4 (0.5)	4 (0.7)
Unknown	464 (56.2)	386 (62.8)
Medical history
Cardiovascular disease	208 (25.2)	167 (27.2)
Pneumonia	199 (24.1)	152 (24.7)
Cancer	143 (17.3)	118 (19.2)
Diabetes mellitus	134 (16.2)	110 (17.9)
COPD	49 (5.9)	39 (6.3)
Rheumatologic disease	18 (2.2)	14 (2.3)
Baseline clinical characteristics
Spo2, ≤94%	288 (34.9)	218 (35.4)
WBC ≤4 × 103 cells/μL	86 (11.0)	67 (11.7)
WBC >10 × 103 cells/μL	171 (21.9)	140 (24.3)
Temperature >38°C	59 (7.2)	42 (6.9)
Respiratory rate >20 breaths/min	261 (31.9)	197 (32.4)
HR >99 bpm	281 (34.0)	202 (32.8)
SBP ≤100 mm Hg	71 (8.6)	39 (6.4)
SBP ≥140 mm Hg	286 (34.7)	215 (35.0)
Outcomes
Supplemental oxygen	525 (63.6)	386 (62.8)
Mechanical ventilation	106 (12.8)	80 (13.0)
Death	98 (11.9)	63 (10.2)

COPD = chronic obstructive pulmonary disease; HR = heart rate; SBP = systolic blood pressure; Spo2 = peripheral oxygen saturation; WBC = white blood cell count.

In the unadjusted time-to-event analysis, corticosteroid use was not associated with a decrease in survival time in the general population (HR = 1.38; P = 0.13). After adjustment for confounding by indication, corticosteroid use was not associated with survival in the general COVID-19 population (Table III ). Among patients requiring supplemental oxygen, the relationship remained statistically nonsignificant, although the point estimate supported a survival benefit (HR = 0.731; P = 0.20). However, among the patients indicated by the MLA, corticosteroid use was significantly associated with an increase in survival time (HR = 0.56; P = 0.04). Adjusted survival curves for all 3 groups are presented in Figure 1 . These results support that the MLA can identify patients in whom corticosteroid use is associated with a survival benefit. Most important model features for generating predictions included timing of COVID-19 diagnosis, systolic blood pressure, and red blood cell count.

Table III

Adjusted in-hospital mortality with corticosteroid treatment of COVID-19.

Statistic	All Patients(N = 826)	Patients Requiring Oxygen Supplementation(n = 525)	Patients with Indicationfor CorticosteroidTreatment (n = 616)
Hazard ratio(95% CI)	0.872(0.549–1.386)	0.731(0.454–1.176)	0.561(0.320–0.983)
P	0.563	0.197	0.043

Figure 1

Adjusted survival curves comparing patients with COVID-19 with or without machine-learning algorithm indication for treatment with corticosteroids. A, All patients. B, Subset that required supplemental oxygen. C, Patients indicated for treatment with corticosteroids.

Remdesivir

In total, 185 patients were included in the remdesivir algorithm test set, 157 of whom received supplemental oxygen and 110 of whom were indicated by the algorithm as suitable for treatment with remdesivir (Table IV ). In the full population, 60 were treated with remdesivir, while 57 of those who received supplemental oxygen and 43 of those indicated by the algorithm received remdesivir. No differences between the general population and the population for which the MLA recommended treatment with remdesivir were statistically significant (P > 0.2 for all comparisons).

Table IV

Demographic characteristics of used for evaluating machine-learning algorithm recommendations on treatment with remdesivir of COVID-19. Data are given as the number (%) of patients.

Characteristic	All Patients(N = 185)	Patients with Indicationfor RemdesivirTreatment(n = 110)
Age group
18–44 y	39 (21.1)	20 (18.2)
45–64 y	60 (32.4)	33 (30.0)
65–79 y	66 (35.7)	45 (40.9)
≥80 y	20 (10.8)	12 (10.9)
Sex
Male	103 (55.7)	63 (57.3)
Female	82 (44.3)	47 (42.7)
Race/ethnicity
Hispanic	22 (11.9)	10 (9.1)
White	16 (8.6)	10 (9.1)
Black	2 (1.1)	1 (0.9)
Asian	0	0
Other	2 (1.1)	0
Unknown	143 (77.3)	89 (80.9)
Medical history
Cardiovascular disease	58 (31.4)	38 (34.5)
Cancer	51 (27.6)	31 (28.2)
Pneumonia	45 (24.3)	36 (32.7)
Diabetes mellitus	43 (23.2)	30 (27.3)
COPD	8 (4.3)	6 (5.5)
Rheumatologic disease	4 (2.2)	3 (2.7)
Initial clinical characteristics
Spo2 ≤94%	82 (44.4)	64 (58.2)
WBC ≤4 × 103 cells/μL	25 (13.5)	14 (12.7)
WBC >10 × 103 cells/μL	38 (20.5)	28 (25.5)
Temperature >38°C	13 (7.0)	9 (8.2)
Respiratory rate >20 breaths/min	52 (28.1)	37 (33.6)
HR >99 bpm	74 (40.0)	44 (40.0)
SBP ≤100 mm Hg	22 (11.9)	13 (11.8)
SBP ≥140 mm Hg	48 (25.9)	21 (19.1)
Outcomes
Supplemental oxygen	157 (84.9)	102 (92.7)
Mechanical ventilation	37 (20.0)	30 (27.3)
Death	19 (10.3)	16 (14.5)

COPD = chronic obstructive pulmonary disease; HR = heart rate; SBP = systolic blood pressure; Spo2 = peripheral oxygen saturation; WBC = white blood cell count.

The unadjusted time-to-event analysis found that remdesivir use was significantly associated with a decrease in survival time (HR = 2.52; P = 0.04). Adjustment for confounding attenuated the relationship, and remdesivir was not significantly associated with survival time in the general population after adjustment (Table V ). The adjusted association in the group that received supplemental oxygen was similarly nonsignificant. However, remdesivir use was statistically significantly associated with an increase in survival among those indicated by the algorithm as suitable for treatment with remdesivir (HR = 0.40; P = 0.04). Adjusted survival curves for all 3 groups are presented in Figure 2 . As with corticosteroids, these results support that the MLA can identify patients in whom remdesivir use is associated with improved survival outcomes. For remdesivir, the most important model features included the use of supplemental oxygen, peripheral oxygen saturation measure, and diastolic blood pressure.

Table V

Adjusted in-hospital mortality with remdesivir treatment of COVID-19.

Statistic	All Patients(N = 185)	Patients Requiring Oxygen Supplementation(n = 157)	Patients with Indicationfor RemdesivirTreatment(n = 110)
Hazard ratio(95% CI)	0.924(0.439–1.942)	0.827(0.384–1.780)	0.402(0.167–0.969)
P	0.835	0.626	0.042

Figure 2

Adjusted survival curves comparing patients with COVID-19 with or without machine-learning algorithm indication for treatment with remdesivir. A, All patients. B, Subset that required supplemental oxygen. C, Patients indicated for treated with remdesivir.

Discussion

In this study, MLAs had the capacity to identify a group of hospitalized patients with COVID-19 in whom treatment with either a corticosteroid or remdesivir was associated with a statistically significant survival benefit. These algorithms were able to do so while relying only on routinely collected EHR information, such as blood pressure, oxygen saturation, and common laboratory measurements.

These survival predictions were possible despite the relatively low AUC of the models for predicting mortality conditioned on treatment as a binomial outcome. The AUC was likely low because treatment with remdesivir or a corticosteroid was a less important contributor to the final patient outcome when compared to covariates such as age, severity of infection, and comorbidities. However, the AUC of >0.5 does indicate that mortality could be predicted with an effectiveness greater than random chance, and the results of the survival analysis support that these ML methods may help to improve patient survival and allocate drug resources. Neither treatment was significantly associated with survival time among the general COVID-19 inpatient population after adjustment for confounding.

Perhaps more importantly, we found no association between treatment and survival time among patients who received supplemental oxygen, despite recommendations for use in this subgroup. , This finding indicates that clinicians may be currently limited in their ability to identify patients with COVID-19 most likely to benefit from treatment with either a corticosteroid or remdesivir.

The results of the present study add to existing clinical evidence supporting the use of remdesivir and a corticosteroid for the treatment of patients with severe COVID-19 in certain circumstances and populations. The WHO Rapid Evidence Appraisal for COVID-19 Therapies (REACT) Working Group determined that overall mortality was lower in patients who received a corticosteroid than in a control group, although there was variation in this response in several studies in patients who received mechanical ventilation. A meta-analysis similarly found evidence that treatment with remdesivir for COVID-19 was associated with reduced mortality and faster recovery, although only in patients with specific clinical parameters. These variations in the observed efficacy of remdesivir may have been the result of patient heterogeneity, as well as of variation in the severity of infection. These findings were further complicated by the fact that studies of remdesivir have not consistently shown a mortality benefit in conventionally defined treatment groups (eg, by disease severity or other patient metrics).

While the emerging body of research supports a role for both corticosteroids and remdesivir in the treatment of patients with COVID-19, further research is needed to develop methods and tools for use in identifying which patient subpopulations are most likely to benefit from treatment.

Conversely, it is just as important to identify which patient populations are not likely to benefit from treatments, so as to prevent undue exposure to the risks associated with treatment. For example, corticosteroids can interfere with the regulation of blood sugar or blood pressure, compromise mental status, and render patients at risk for secondary infection via immunosuppression. Patients with COVID-19 treated with a corticosteroid may be at elevated risk for primary, secondary, or mixed adrenal insufficiency, particularly if also treated with an antiviral, which might increase the half-life and bioactivity of corticosteroids through cytochrome P-450 inhibition. It has also been hypothesized that corticosteroid-mediated immunosuppression, particularly in milder cases of COVID-19, may interfere with the host-adaptive immune response to the SARS-CoV-2 virus, including delaying viral clearance and increasing infectivity. , The possible risks of immunosuppression are illustrated by research showing that patients on long-term, high-dose corticosteroids for the treatment of autoimmune disease were more likely to require hospitalization for COVID-19, and that mortality was higher in patients with moderate to severe immunosuppression than in the general population.43, 44, 45, 46, 47 Patient selection is therefore key to balancing the risks associated with corticosteroid treatment with the potential benefits of modulating the hyperactive inflammatory response to SARS-CoV-2 infection that is present in some, but not all, patients with COVID-19.

While a range of adverse effects have been reported with remdesivir use, meta-analyses have depicted a generally favorable risk profile, with fewer serious adverse events such as acute respiratory failure or septic shock among patients who received remdesivir compared to patients who received placebo or the standard of care. , However, adverse-events reporting in the literature describing trials of remdesivir is largely considered of low quality by Consolidated Standards of Reporting Trials standards, and the scope of possible adverse effects is likely not yet understood. Adverse events are relatively common and can lead to treatment discontinuation. , An increased duration of treatment has been associated with a greater risk for discontinuation due to adverse effects. In addition, the data from multiple at-risk patient populations (including patients with preexisting severe renal or hepatic dysfunction and pregnant women) have been excluded from completed clinical trials of remdesivir, precluding assessments of tolerability in these patient segments. Indeed, as alteration in liver function is relatively common during treatment with remdesivir in all patients, , caution is particularly warranted when considering treatment with remdesivir in patients with impaired hepatic function. The recency of US Food and Drug Administration approval of remdesivir and multitude of ongoing clinical trials indicate that the clinical understanding of the safety profile of remdesivir is evolving, highlighting the need for judicious clinical use. In light of the limited availability and high cost of remdesivir, patient selection is also crucial from the perspective of resource allocation. Taken together, these concerns point to the need for more targeted methods of identifying patients with COVID-19 in whom a corticosteroid or remdesivir should be given, and those in whom other treatment avenues should be pursued.

Projections suggest that the spread of COVID-19 will continue in the coming months, even with the adoption of public health mandates designed to limit community transmission. , 54, 55, 56 While findings from recent analyses of data from trials of vaccines have raised hopes that effective vaccines are on the horizon, , widespread distribution of vaccines may take significant time and leave patients at risk during the interim. Furthermore, even with an available vaccine, herd immunity may not be achievable in the near future, due to the high rate of community vaccination required. , The use of effective therapeutics may therefore aid in reducing morbidity and mortality throughout the remainder of the COVID-19 crisis.

Limitations

This research had several limitations. First, because of the retrospective nature of this work, we could not determine how treatment recommendations may influence prescribing practices and patient outcomes in clinical settings. Additionally, because the present study utilized data from a cohort in which treatment was not randomized, it is possible that residual confounding may have influenced the results despite the efforts to adjust for confounding variables. Due to the limitations of the data available from the EHR, we were unable to determine whether supplemental oxygen was delivered at the time of final oxygen saturation measurements in all patients. Similarly, we could not determine supplemental oxygen status at the time that peripheral oxygen saturation measurements were delivered as model inputs in all patients. Supplemental oxygen status was therefore not used for normalizing either of these oxygen saturation measures; this information could have been meaningful during the model-training phase, had it been available. Additionally, as the primary end point of this study was survival time, we did not compare the frequencies of adverse events between the study groups, nor did we develop an algorithm for use in identifying patients at increased risk for adverse events explicitly. However, because adverse events were likely to have been associated with poorer patient outcomes, the algorithm may have inherently selected for patients with a reduced risk for adverse events. Future exploration of the risk for adverse events in the algorithm-indicated population is therefore appropriate.

Other limitations were related to the study sample itself. In particular, the data used for training the algorithms were collected early in the pandemic. Understanding of the progression and treatment of COVID-19 has significantly improved since then, and patient demographics and outcomes have shifted compared to those in early cases. , For these reasons, the training data may not have well reflected the data on which the algorithms were tested. Furthermore, the algorithms may perform less accurately on data collected in the future and on data that may be even more dissimilar from the training data.

The small size of the study sample used for testing the remdesivir algorithm was another limitation; the replication of these findings in a larger-scale cohort is warranted for confirming these results. Additionally, the small sample size precluded any analysis of combinatorial treatments. Given the potential for drug–drug interactions, future work exploring the performance of MLAs in identifying patients who may benefit from, or be harmed by, combinations of therapies would be of significant clinical interest.

Finally, although the focus was on patient survival times, there are other clinically relevant end points related to COVID-19. However, because MLA systems can be readily retrained, they likely have the potential to identify a population that would experience improved symptoms, such as oxygenation, as well as an improved likelihood of survival, to help with treatment selection for clinical trials and clinical care. Assessment of the performance of MLAs in identifying patients in whom treatment is associated with additional end points may be an important area of future work.

Conclusions

Due to the continued global threat posed by COVID-19, effective treatment for patients hospitalized with COVID-19 remains an important area of research and a key consideration for clinicians. Our study has shown that ML has the capacity to identify patients most likely to derive a survival benefit from treatment with either a corticosteroid or remdesivir, both of which are recommended for the treatment of patients with COVID-19. These MLAs have implications for improving patient outcomes and appropriately allocating resources. To the authors' knowledge, this report is the first description of the use of ML as a method of evaluating the effectiveness of treatments for individual patients with COVID-19. This finding supports that precision-medicine approaches are viable for treating patients during the COVID-19 pandemic.