Deploying Randomized Controlled Trials during the COVID-19 Pandemic: Reason and Bayesian Designs.

As of early May 2020, we have largely failed to provide new answers regarding the treatment of patients with coronavirus disease (COVID-19). The massive number of cases has overwhelmed healthcare systems, which have responded with varying degrees of effectiveness (1). Physicians, feeling powerless when confronted with a new disease, have responded in startling ways. Decades of centering medicine around administration of prescription drugs, combined with wishful thinking that arose from fear, made the search for a single pharmacological treatment look plausible. Like excited children who just found a box filled with old toys, physicians dug through the literature in search of anything that could be useful. Old whispers from the past came to the forefront of physicians’ minds. Drugs, the physician’s favorite toys, were being used, added, mixed, and promoted (2). “What harm can it be?”, many asked. And so harm may have been done. Medicine’s long history of bias favoring acts of commission instead of omission once again prevailed.

Fortunately, scientists who read and understood the whole history of drug development took a stand, and initiatives to provide reasonable scientific evidence during the pandemic began to appear. In this issue of AnnalsATS, Brown and colleagues (pp. 1008–1015) describe the protocol for the HAHPS (Hydroxychloroquine versus Azithromycin for Hospitalized Patients with Suspected or Confirmed COVID-19) trial (3). This trial will evaluate two drugs that frequently resurface during acute viral (respiratory) diseases, namely, hydroxychloroquine and azithromycin (4, 5). The authors discuss the background that justifies the choice for these drugs while clearly acknowledging that much of the motivation for the trial stems from an overwhelming availability cascade (6). Specifically, for hydroxychloroquine, a drug that has a history of failing to improve meaningful clinical outcomes in viral diseases, small reports collided with a need for action demanded by physicians, patients, and regulatory authorities alike (2). This situation became so intense that equipoise, in the absence of any compelling data favoring any drug, was questioned (2). The authors should be commended for initiating the trial under these circumstances. I also extend my admiration to the ethics committees that provided timely review and approval of the trial, and the statisticians involved.

This work has many strengths that need to be highlighted. The authors use transparency in discussing the trial design, and an interesting approach to causal inference from the study’s results, emphasizing that HAHPS will rely, as would any randomized clinical trial, on external evidence for proper interpretation. The a priori reliance on the need for network meta-analyses is both a weakness and a strength of the trial. No trial is “definitive.” Network meta-analyses, as discussed by the authors, are an interesting way to provide indirect evidence from comparisons that were not tested directly in different clinical trials. For example, suppose that another trial comparing hydroxychloroquine with placebo for COVID-19 occurs. A network meta-analysis can both enrich the comparison between hydroxychloroquine and placebo by using data from both HAHPS and the new trial, and provide indirect evidence of the effects of azithromycin versus placebo. As more trials are deployed, these networks can be enriched (7).

One outstanding aspect of HAHPS is the use of Bayesian methods for data analysis and interpretation, which addresses several shortcomings of conducting randomized controlled trials in this pressing situation. Specifically, Bayesian methods allow for sequential interpretation of evidence as new patients are enrolled without prespecifying rigid (and sometimes arbitrary) stopping rules (8). In addition, by adding priors to effect sizes, some regularization of the results can be done. The authors elected conservative neutral priors that concentrate most pretest probabilities to odds from 0.5 to 2.0, centered on the absence of effect; that is, it makes the model very skeptical of effect sizes that are extremely rare in modern clinical practice. Centering on the absence of effect makes complete sense in this scenario, where both benefit and harm are equally possible. This approach is in contrast to frequentist reasoning, where all ranges of effect sizes (from −Inf to +Inf) are considered plausible. Bayesian methods produce a distribution of posterior possible effect sizes given data and prior assumptions. This can provide answers to relevant questions, such as, what is the probability that one treatment is better than another, and what is the probability that the effect size is greater than a given value (say, an odds ratio lower than 0.8) or even the probability that the effect size is within a range (such as between 0.9 and 1.0)? Clinicians can easily interpret the results without having to rely on null hypothesis tests, which are sometimes misinterpreted. A hypothetical example of interpreting the results of HAHPS is shown in Figure 1.

Figure 1.

Distribution of log(odds ratio [OR]) for prior (dashed line) and posterior (thin line) effect sizes for a hypothetical simulation of the effects of hydroxychloroquine and azithromycin in 300 patients with COVID-19 in HAHPS (Hydroxychloroquine versus Azithromycin for Hospitalized Patients with Suspected or Confirmed COVID-19), showing changes in the ordinal scale. The dotted line marks absence of effect (log(OR = 0) = 1). In this hypothetical example, the mean (and 95% credible intervals) for the OR is 0.93 (0.63–1.37). The following probabilities are defined by the HAHPS statistical analysis plan: 1) P1 = Pr(OR < 1), indicating the evidence for any benefit = 64.1% (shaded in blue in the figure); 2) P2 = Pr(OR < 1/1.25), indicating the evidence for a moderate or greater benefit = 22%; 3) P3 = Pr(OR > 1), indicating the evidence for any harm (shaded in brown in the figure) = 35.9%; 4) P4 = Pr(OR > 1.25), indicating the evidence for a moderate or greater harm = 6.7%; P5 = Pr(1/1.2 < OR < 1.2), indicating the evidence for similarity between the two treatments = 61.4%. P5 can be defined as the region of practical equivalence as stipulated by the authors in their study design. This area is colored in darker colors in the figure. COVID-19 = coronavirus disease.

The weaknesses of the trial include the (justified) lack of placebo and a true control group, the small sample size (which translates to a maximum detectable risk ratio of 0.702 for an ordinal scale over three points for the maximum sample size), and the (initial) proposal of assessing the primary outcome at fixed time points. This last point deserves some attention. The ACTT (Adaptive COVID-19 Treatment Trial) trial of remdesivir for COVID-19 also initially planned to consider an ordinal scale at a given time for the primary endpoint, which was eventually converted to a continuous time to recovery endpoint (9). Fixed time points present an issue when dealing with a fluctuating disease such as COVID-19. As the authors acknowledge, there is a high chance that the odds for ordinal endpoints will not be proportional. Although estimates may still be reliable in this situation, this presents a potential limitation to data interpretation. Fortunately, the online statistical analysis plan presented by the authors considers a mixed model with time as planned, which can address some of these issues. The final statistical analysis plan is eagerly anticipated.

In conclusion, HAHPS will provide high-quality evidence for patients with COVID-19. The trial limitations are clearly discussed. It is not designed to be a “definitive” trial, or to provide standalone evidence for COVID-19 management, but it can serve as an invaluable brick in building evidence and fostering proper clinical research during these chaotic and politicized times.