COVID-19 pandemic and the tension between the need to act and the need to know.

Pandemics associated with a deadly and contagious virus, such as COVID‐19, generate a giant thirst for instant knowledge in how to treat the disease. The imperative becomes doing something, anything, that we think may help. This creates tension between exploiting whatever actionable clues we can gleam from observations and anecdotes versus conducting proper clinical trials – the quick and dirty versus the slow and rigorous. Can we increase the tempo of discovery while staying faithful to scientific method?

In identifying effective therapies, there are already hundreds of COVID‐19 trials registered worldwide, with numbers increasing daily. This is a positive – the rapid mobilisation of trialists around the world, many suspending their prior research programmes to focus on COVID‐19 and enlisting the support of multiple funders. But with so many different trials, there is also the potential for wasteful duplication, competition for limited resources, including recruitable patients, and reporting of poorly designed and underpowered studies that result in premature rejection of promising drugs or premature adoption of ineffective ones as standards of care.

The false promise of rushed science

A pandemic as serious as COVID‐19 will compel some clinicians and patients to try unproven therapies based on theory, in vitro data, animal models, clinical anecdotes, observational studies and uncontrolled trials that may later be shown to be misleading. This was the case during the novel influenza A (H1N1) viral pandemic in 2009, when low‐quality and incomplete data suggested oseltamivir was potentially beneficial in preventing complications and deaths among patients hospitalised with influenza. As a result, countries stockpiled and used the drug extensively and at great expense, but to this day, there has been no high‐quality randomised controlled trial (RCT) to confirm definitively the efficacy of oseltamivir.

Numerous studies have already been reported during the current pandemic, many as pre‐prints, in an effort to disseminate results quickly, but which are yet to undergo peer review. Others have undergone fast‐track review, which may be less stringent or probing than the usual process, which can take many weeks. Studies reporting promising results have featured cohort studies, case series and even single case reports that, at any other time, would have been regarded as no more than hypothesis‐generating on account of their high risk of bias due to their observational design. Instead, some now receive considerable publicity as ‘treatments’ from media outlets, helped along in some instances by endorsement as ‘game changers’ by high‐profile celebrities and politicians.

Medications lacking any approval for any indication are also being widely used outside of clinical trial protocols. Unproven therapies have become accepted as new standards of care, thus extinguishing the equipoise needed to undertake RCT. As an example, concerned by reports of increased incidence of thrombosis and disseminated intravascular coagulation in seriously ill COVID‐19 patients,9, 10 the results of a very low‐quality observational study of the use of anticoagulants (AC)11, 12 led one US hospital system to protocolise the use of high‐dose AC in patients with COVID‐19 admitted to intensive care (Box 1). While more rigorous propensity‐matched observational studies comparing one treatment group to another lend useful insights, the best adjustment methods can still miss major systematic biases, especially when responding to a rapidly spreading and deadly pandemic.

In extreme times, RCT too have been flawed by ill‐specified inclusion criteria, convenience sampling, small samples from single centres with inadequate power, use of surrogate outcome measures (such as reduction in viral shedding, radiological progression, numbers of ‘cough days’ ), ad hoc administration of a host of co‐treatments (such as steroids, various antiviral drugs and antibiotics), underpowered subgroup analyses, early termination and short in‐hospital duration. Unsurprisingly, such flaws cause different trials of the same therapy, such as hydroxychloroquine, to report conflicting findings.10, 15 Too much faith has been prematurely placed on what seemed to be useful therapies, as exemplified by the off‐label use of corticosteroids and hydroxychloroquine, despite evidence of no effect or even possible excess mortality. Such publicity also led to inappropriate overprescribing of hydroxychloroquine to treat mild cases and even for prophylaxis, leading to shortages of the drug for patients with proven indications for its use, such as rheumatoid arthritis and systemic lupus erythematosus. Hopefully, the same problems will not occur with remdesivir, which – despite limited and conflicting evidence of clinical improvement from only two placebo‐controlled RCT19, 20 and one non‐controlled cohort study – has now become a ‘standard of care’ in the United States for COVID‐19 patients with severe pneumonia.

The urgency to investigate and report therapeutic trials during a pandemic is not surprising, but the human desire of researchers and journal editors to be the first to report and publish potentially ground‐breaking research also needs consideration. Many study protocols are being designed in haste by inexperienced researchers, with inadequate attention being paid to both methodological rigour and appropriate informed patient consent. Unscrupulous investigators have fabricated or manipulated data22, 23 or submitted duplicate articles relating to the same patient populations, which imperils the accuracy of subsequent estimates of therapeutic outcomes and precludes valid meta‐analyses if individual patient data cannot be obtained to reveal such duplication. The websites of many mainstream journals now feature COVID‐19 resource centres containing pre‐proofs and early views of articles that help boost their citation and website hit rates, but without any ranking or labelling of articles according to quality. Regulators, such as the US Food and Drug Administration, are pressured to expedite approval of investigational therapies under emergency user access or compassionate use schemes.

Making RCT more doable and informative during a pandemic

There are many challenges to conducting RCT in a pandemic – randomisation to placebo seems unethical; conventional trials are slow and cumbersome; and conducting multi‐site trials that require participating researchers to agree on inclusion and diagnostic criteria, drug administration schedules and outcome measures is particularly challenging. This multiplicity of study designs also poses difficulties for meta‐analysts trying to render estimates of therapeutic effects more precise and certain. So how can RCT be rendered more doable and informative during a pandemic?

First, trialists should wait until sufficient data have been gathered to allow a better understanding of the disease process and its natural history and then use this to design more definitive trials. Based on the assumption that SARS‐CoV‐2 was a coronavirus similar to those that caused Severe Acute Respiratory Syndrome (SARS) and Middle East Respiratory Syndrome (MERS), drugs that had shown antiviral effects on the latter, at least in vitro, were quickly repurposed to treat the former, and their effects were assessed in observational studies. It has now become clear that, while sharing many traits, SARS‐CoV‐2 is genetically different to its predecessors, as are the varied clinical manifestations of infection, such as profound hypoxia, extensive inflammatory activation, endothelial damage and coagulopathies. These observations suggest multiple pathogenic pathways activated to varying degrees across different individuals at different times.

Second, this complex pathogenesis means that no single drug is likely to be effective, and even for multiple drugs, significant heterogeneity of treatment effects is likely among individuals and at various stages of the disease. Separating out these effects, and determining which drugs to use and when (early in the disease course or only at deterioration), will likely require large‐scale trials with multiple treatment arms that are sufficiently powered to enable analyses of primary and, where indicated, secondary outcomes across different patient subgroups. Conventional multi‐arm trials take time and expose patients to ineffective or even harmful treatments for the duration of the trial in the absence of frequent interim analyses. Weighing the pros and cons of multiple competing protocols also wastes time and resources. It would be more efficient to use designs that leverage a common structure for trial entry, data collection and testing of multiple therapies. Adaptive study designs and platform protocols allow rapid cycle testing of different therapies using response‐adaptive randomisation (Box 2).25, 26 Such trials take account of the heterogeneity of the patient population (e.g. based on age or other patient characteristics, illness severity, prognostic factors) and aim to find, in the shortest time possible, the best treatment, or combination of treatments, for patient subgroups defined by these variables. What may take 12 months for traditional two‐arm sequential trials to produce an answer could be reduced to 3 months using adaptive designs. Such designs underpin several ongoing COVID‐19 trials, such as AustralaSian COVID‐19 trial (ASCOT), Adaptive COVID‐19 treatment trial (ACTT), World Health Organization Solidarity, Randomised embedded multifactorial adaptive platform trial for community‐acquired pneumonia and Randomised evaluation of COVID‐19 therapy (RECOVERY).

Third, trials must strive to be multi‐site and for investigators to share individual patient data from similar trials within multi‐trialist collaborations. These groups need to share a common taxonomy of outcome measures for which the minimal clinically important difference between treatment groups, from a patient's perspective, has been determined. This would allow for additional analyses, even if the analyses of the combined data were not pre‐planned, and would be considered exploratory. The goal is to expand what is known about possible treatments so that future trials can be improved using the adaptive designs already mentioned.

Fourth, funding agencies responsible to taxpayers need the political cover and authority to support international studies; pharmaceutical companies need support and incentives from regulatory authorities to participate in collaborative trials; and academic investigators need a structure that provides academic credit and incentive to collaborate in efforts where they might otherwise perceive anonymity and loss of control and intellectual property. An example of such multi‐sectoral collaboration is the recently announced Accelerating COVID‐19 Therapeutic Interventions and Vaccines (ACTIV) partnership in the United States whose industry partners agree to support the prioritisation of therapeutic and vaccine candidates, no matter who has developed them, and to share their respective clinical trial capacities, irrespective of the agent to be studied. For their part, the public partners led by the National Institutes of Health resolve to streamline research and regulatory issues to drive expedited evaluation and rapid scale‐up and manufacturing of candidate therapies with predicted successful outcomes.

Finally, every clinician involved in managing COVID‐19 patients must commit to acknowledging equipoise and seeking definitive evidence of the benefits and harms of any proposed treatment by enrolling patients into multi‐site RCT that are being transparently conducted in communication with global partners. This strategy allows clinicians to be satisfied that they are doing everything possible for their severely ill patients while contributing to new knowledge – an approach with which patients also agree. We must avoid the ‘just do it’ option of administering therapies based on unreliable observational evidence and instead commit to reducing uncertainty by testing therapies in RCT as the ‘must learn’ alternative. It is not a matter of choosing between one or the other.

Researchers from Mount Sinai Hospital in New York City reported a retrospective observational study of 2773 patients with COVID‐19 hospitalised in March and April 2020.11 They found that, among patients prescribed therapeutic doses of anticoagulation (AC), the in‐hospital mortality was 22.5%, with a median survival of 21 days, compared with 22.8% and a lower median survival of 14 days among patients who did not receive this treatment. However, among patients who required mechanical ventilation (n = 395), in‐hospital mortality was 29%, with a median survival of 21 days, for AC patients but 63%, with median survival of 9 days, in non‐AC patients. The adjusted hazard ratio for mortality was 0.86 per day (95% confidence interval 0.82–0.89, P < 0.001), while bleeding events were similar in both groups: 3% versus 2%. The authors acknowledged their study was limited by: indication bias for AC, with no reporting of why AC was commenced in some patients but not others at varying times throughout the hospitalisation; non‐standardised use and dosing of oral, subcutaneous or intravenous AC; subgroup analysis with a lack of metrics to further classify illness severity in the mechanically ventilated patients; absence of data regarding the precise cause of death (coagulopathy‐related or otherwise); and other unobserved confounders. The median duration of AC was only 3 days, which makes such a large decrease in mortality from such a short exposure to the drug among ventilated patients implausible. The authors also omitted mentioning immortal time bias.12 Looking at the survival curves of both groups, at day 5, about 20% of the patients in the non‐AC group had died, but nearly all the AC patients were still alive. But to receive AC at day 5, the patient had to survive, or be ‘immortal’, to that point in time, with credit being given to AC for those 5 days of survival. In contrast, the non‐AC arm includes all the patients who did not live long enough to receive AC and who could have died any time during their hospitalisation, including on day 1, and are thus considered ‘mortal.’ In a similar fashion, any other intervention given to a patient who survived to day 5, such as a garlic necklace, could be given credit for preventing death up to that point in time. Despite these multiple flaws, the authors concluded that systemic AC may be associated with improved survival after adjusting for mechanical ventilation. The second author of the paper also happened to be the editor‐in‐chief of the journal in which the article was published, thus raising concerns about the rigour of the peer review process. This author also spoke to the media extolling the virtues of AC in all COVID‐19 patients admitted to intensive care and announced that the Mount Sinai hospital system had changed its protocols to begin giving such patients therapeutic doses of AC. Multiple commentators took to Twitter exposing the flaws of the study within hours of publication, emphasising that observational studies are prone to bias, often report over‐inflated effect sizes and – if adopted as new standards of care – impede the ability to mount robust RCT capable of providing more definitive results. Flawed data can be worse than no data, and observational studies should not be used to establish a new normal.

Adaptive randomisation Adaptive trials allow pre‐specified changes in key trial characteristics while it is being conducted in response to information accumulating during the trial.25 Adaptive randomisation (AR) allows changes to be made to the probabilities of participants being randomised to one treatment (or treatment combination) versus another during the trial with the aim of allocating a greater proportion of patients to treatments that are demonstrating evidence of better performance than others. This evidence of better performance can comprise information from the trial itself, evidence emanating from other trials and expert opinion from multiple groups or societies. Bayesian statistical methods are used to continually update trials with new information as it becomes available while at the same time maintaining trial integrity, thus allowing trials to ‘learn as they go’. This level of flexibility is difficult with classical, non‐Bayesian approaches that have a less informative focus on what ‘works’ or ‘doesn't work’ according to a statistical test. The Bayesian approach is to define and recursively update the probability that a treatment works based on combining information more naturally, better resembling how clinicians think in the real world. Non‐Bayesian trials struggle to confirm whether a treatment works under uncertainty because the sample size and design features of the trial rely on assumptions about how the treatment will work. The trial design cannot be modified easily, so if those assumptions, including sample size calculations, are ultimately incorrect, the trial may finish without providing any useful evidence about what treatments are effective. A Bayesian adaptive trial can swiftly and more efficiently learn about existing treatments, abandon any that prove futile and expand to include new and promising candidates.26 It has all the advantages of classic group sequential designs but can also alter maximum sample size, switch end‐point from non‐inferiority to superiority, alter number and spacing of interim analyses, investigate a larger dose range in order to select effective doses, incorporate biomarkers that may predict differential treatment response and proceed to completion with increased enrolment and resolution of responses in all enrolled patients, instead of being terminated early with risk of compromise from unknown or unadjudicated responses. Platform protocols Platform protocols facilitate the study of multiple targeted therapies in a perpetual manner, with therapies allowed to enter or leave the platform on the basis of a decision algorithm or stopping rule. The platform trial is ongoing over time, with no fixed finish date, and is governed by a master protocol that envisions adding and dropping strata. At trial start, entering patients are assigned to different strata (potentially on the basis of illness severity, such as severe (A), moderate (B) or mild (C) COVID‐19 disease). Strata A patients are then randomly assigned to one of three groups, testing two investigational drugs (drugs 1 and 2) against placebo. When investigational drug 1 meets the pre‐specified criteria for success (based on, in many instances, the Bayesian likelihood of a treatment benefit), drug 1 replaces the placebo group as the control. From this point, newly recruited patients are randomised to another investigational drug (drug X), and the new control group becomes drug 1, while recruitment of patients into the previous protocol comparing drugs 1 and 2 completes enrolment and is ceased. In a similar manner, strata B patients may also be randomised to drugs 1 and 2 or placebo or to different drugs (drugs 3 and 4) or placebo. In a similar manner to strata A, whichever drug shows superiority in strata B then becomes the control group for newly recruited patients into that strata once patient enrolment is completed for the first protocol. The same process applies to strata C. The design can also accommodate comparisons of drug combinations (e.g. drugs 1 + 2 vs placebo or drugs 1 and X vs drug 1). The statistical methods throughout involve randomised treatment assignment, sharing of common control patients and sequential interim analyses with the possibility of stopping early for futility.