Treating COVID-19: Evolving approaches to evidence in a pandemic.

The rapid pace of the COVID-19 pandemic precluded traditional approaches to evaluating clinical research and guidelines. We highlight notable successes and pitfalls of clinicians' new approaches to managing evidence amidst an unprecedented crisis. In "Era 1" (early 2020), clinicians relied on anecdote and social media, which democratized conversations on guidelines, but also led clinicians astray. "Era 2" (approximately late 2020 to early 2021) saw preprints that accelerated new interventions but suffered from a surfeit of poor-quality data. In the current era, clinicians consolidate the evidentiary gains of Era 2 with living, online clinical guidelines, but the public suffers from misinformation. The COVID-19 pandemic is a laboratory on how clinicians adapt to an absence of clinical guidance amidst an informational and healthcare crisis. Challenges remain as we integrate new approaches to innovations made in the traditional guideline process to confront both the long tail of COVID-19 and future pandemics.

Main text

Introduction

Clinical guidelines evolve incrementally, with major changes driven by systematic review of randomized controlled trials (RCTs) and other high-quality evidence, followed by consensus among international experts. However, the pace of the COVID-19 pandemic precluded a traditional approach to clinical guidelines. Here, we review the evolution of clinical standards of care during the pandemic, often exemplified in the pages of Cell Reports Medicine. We focus on inpatient care and are informed by our experiences at three academic medical centers across the USA. Here, we spotlight the most influential successes and pitfalls that reshaped traditional methods of evidence-based medicine and guideline creation (Figure 1). In the earliest days of the pandemic, “Era 1” catalyzed the adoption of pre-COVID-19 respiratory interventions but also prioritized anecdotal observations. “Era 2” moved further into 2020, when clinicians struggled to evaluate non-peer-reviewed research and rising misinformation. The current era consolidates our previous successes and failures and seeks a sustainable hybrid of pre- and post-COVID-19 approaches to clinical evidence.

Figure 1

The evolving approach to evidence across three eras of the COVID-19 pandemic

Successes (blue), pitfalls (red), and their relative impacts and overlap are noted.

Era 1: The rise of anecdotal medicine (January to May 2020∗)

∗These dates reflect approximate time periods at our centers. These time frames for “eras” likely varied across different centers and regions of the USA and the world.

The clinician’s dilemma

In March 2020, an elderly woman presents with fever, cough, and hypoxemia. SARS-CoV-2 PCR results will not return for days. Chest computed tomography (CT) imaging shows lung infiltrates. Antibiotics are prescribed, as there are no treatments specific for COVID-19. Without clear understanding of viral transmission, nurses perform time-consuming decontamination protocols, including wiping down bloodwork tubes. Families do video encounters instead of in-person visits. Worldwide, the trickle of patients becomes a surge. Non-COVID-19 patients avoid hospitals, and COVID-19 patients that do arrive at hospitals are aggressively triaged to return home due to patient capacity limits. “Elective” procedures are canceled. Medical centers expand intensive care units (ICUs) to several times normal capacity by increasing patient-staff ratios and redesigning bed assignments. Without global coordination, institutions bid against each other for personal protective equipment, ventilators, dialysis machines, and basic supplies.

Clinicians asked the question: without data, how do we treat patients in the absence of traditional clinical guidelines?

Successes of Era 1: Democratization of clinical guidelines and implementing pre-COVID-19 evidence

Early in the pandemic, physicians lacked evidence-based guidelines to treat hospitalized patients. They obtained epidemiologic data from private institutions (e.g., Johns Hopkins University or The New York Times) and colleagues in the earliest affected areas, namely China, Italy, and Seattle (USA). Within social media’s democratic ecosystem, frontline providers and community hospitals could lead discussions in clinical protocols. Elmhurst Hospital’s early adoption of “awake prone positioning” was widely shared by peer-to-peer communication and on platforms like YouTube, Twitter, and Facebook. Furthermore, institutional protocols based on rapid literature review and expert opinion now existed on one’s mobile phone, such as our institutions’ COVIDProtocols (https://covidprotocols.org/). Era 1 also delivered newfound success in the adoption of interventions for which strong evidence or rationale existed before COVID-19. Prone positioning of mechanically ventilated patients with severe acute respiratory distress syndrome (ARDS) is one example. In 2013, the seminal, multicenter PROSEVA RCT demonstrated a mortality benefit for proning in severe ARDS, but the maneuver was not routinely practiced. However, positive anecdotal reports in COVID-19 catalyzed widespread adoption of proning in both intubated and awake patients. Another significant pandemic-driven innovation has been the utilization of telemedicine and video conferencing. Albeit with its limitations, video conferencing allowed increased access to educational updates and nimble collaborations in the face of rapidly changing evidence. Telemedicine also expanded to help keep vulnerable patients at home, reduce opportunities for COVID-19 transmission, and address local shortages of expertise. During the pandemic, many centers expanded telemedicine for outpatient clinic visits and initiated new telemedicine inpatient services, such as palliative care, critical care, and psychological consultation. In the USA this was facilitated by the federal government’s massive reimbursement plan for telemedicine under the 1135 waiver authority and the Coronavirus Preparedness and Response Supplemental Appropriations Act.

Pitfalls of Era 1: Anecdotal observations and hypothetical pathophysiology

Social media and peer-to-peer conversation were well-suited to circulate early observations leading to an explosion of ideas on pathophysiology and treatments. The second issue of Cell Reports Medicine chronicled examples in May 2020 with Perspectives on COVID-19 immunotherapy and a hypothesized vulnerability for patients with Down syndrome. However, social media led to an availability bias that amplified anecdotes and untested hypotheses. For example, reports of severely hypoxemic, but clinically comfortable, patients spurred a hypothesis that COVID-19 patients maintained normal lung compliance and could tolerate higher lung tidal volumes after intubation, a pathophysiology distinct from typical, non-COVID-19 ARDS. In a second example, post-mortem examination of COVID-19 lungs showed microthromboses, and case series showed a higher incidence of venous thromboembolism (VTE). These limited data steered some centers to increase heparin dosages for VTE prophylaxis or administer therapeutic anticoagulation to COVID-19 ICU patients without evidence of thrombosis. In a third example, early concerns of cardiac injury drove practices like aggressive cardiac monitoring (by biomarkers and echocardiography) in COVID-19 inpatients. Later in the pandemic, consensus opinion rejected higher lung tidal volumes, and both VTE and cardiac injury assumptions in the critically ill were unsupported by RCTs. These assumptions also affected research. As USA research labs were shut down to all but the most essential work, some laboratories prioritized studies inspired by the limited case series, such as cardiac injury in COVID-19.

On the therapeutic side, the early pandemic was notable for the sheer number of untested treatments given outside of clinical trials. Examples included lopinavir/ritonavir, ribavirin, hydroxychloroquine, and azithromycin, all given based on hypotheses or (at best) in vitro assays. Convalescent plasma illustrates medical centers’ varied response to this “anecdotal” era. Clinicians hypothesized that administration of convalescent plasma from volunteers who had recovered from SARS-CoV-2 would aid viral clearance. Some centers gave convalescent plasma liberally, using local sources or via the Mayo Clinic’s Expanded Access Program funded by the Food and Drug Administration, which provided investigational plasma prior to its Emergency Use Authorization. Other centers limited convalescent plasma to multicenter RCTs, and still others focused on patient subcohorts, such as those with humoral immunodeficiency. This tendency toward unproven therapies stemmed from a desire to offer any intervention when faced with desperate patients. While understandable, it is a reminder that, in the absence of evidence, there is a significant bias favoring active intervention, a phenomenon seen in contexts as varied as cardiac catheterization and international aid.

Era 1 summary

This question captured Era 1: how does a physician act without evidence? Remarkable successes included new voices in clinical conversations, implementing protocols from prior RCTs, and the rapid development of expert guidelines. Significant pitfalls occurred as anecdotes, hypotheses, and limited evidence, such as in vitro studies, replaced medicine’s traditional evidence-based practice.

Era 2: The rise of preprints and the rapid iteration of clinical guidelines (approximately June 2020 to March 2021)

January 2021 brings a new surge and a middle-aged man to a Boston emergency department with fever and dyspnea. A chest CT is unnecessary, as the SARS-CoV-2 PCR test confirms COVID-19 within hours. Fully vaccinated staff activate COVID-19 order sets, and dexamethasone and remdesivir are initiated. The patient is started on high-flow nasal cannula, a practice avoided earlier in the pandemic out of concern for aerosolization. He deteriorates and is given tocilizumab (anti-IL-6 receptor monoclonal antibody [mAb]). His 4-week ICU course is informed by a wealth of COVID-19 clinical practices that are now routine, from careful monitoring for bacterial and fungal superinfection to the option of tracheostomy while still COVID-19 positive. During his daily in-person visits, the patient’s son, who is an ICU nurse elsewhere, remarks on the differences in standard care among hospitals, such as visitor policies for COVID-19 patients and VTE prophylaxis.

The rapid rise in hospital admissions led clinicians to ask the question: how do we practice evidence-based medicine during a pandemic when traditional processes are too slow?

Successes of Era 2: Rapid generation of stronger evidence and its implementation in frontline care

Era 1’s anecdotes and case series gave way to more reliable evidence in Era 2. Hospitals joined multicenter cohorts that provided robust, cross-sectional COVID-19 data correcting early misconceptions (e.g., rates of cardiac injury). The RECOVERY, REMAP, and Solidarity clinical trial groups represented massive successes in multicenter collaboration. For example, RECOVERY’s steroid RCT compared 2,100 patients receiving steroids with 4,300 patients in usual care across 176 centers in the UK. These trials’ adaptive designs allowed them to simultaneously test multiple therapeutics in a fraction of the usual time frame. Instead of three separate immunomodulator RCTs, REMAP-CAP’s adaptive platform compared tocilizumab, sarilumab (an anti-IL-6 receptor mAb), and anakinra (IL-1 antagonist) in over 2,200 COVID-19 patients in one RCT. These RCTs also improved on embedding trials into routine care, remote consent, integrating local and central ethical review, and virtual monitoring. The success of these RCTs highlights the benefits of investing in multicenter research networks and nationally coordinated electronic health records.

The advent of increasing evidence led to profound clinical practice changes at unheard of speed during Era 2. In February 2020, the DEXA-ARDS RCT demonstrated the utility of glucocorticoids in non-COVID-19 ARDS. Despite high-quality data, steroids were not widely adopted. However, in June 2020, a press release from the RECOVERY RCT reported a mortality benefit from steroids in COVID-19 patients requiring supplemental oxygen. Based on pre-COVID-19 data, cross-sectional data on inflammation in COVID-19, and the reputation of the RECOVERY consortium, many centers immediately adopted the use of steroids, some even prior to the preprint, let alone the peer-reviewed manuscript. This radical, new approach of changing clinical practice in the absence of multiple, peer-reviewed studies reflected several factors: pre-COVID-19 studies, the pandemic’s immediate needs, and the relative homogeneity of COVID-19 cohorts.

In another example of the evolution from Era 1 to 2, some centers empirically used tocilizumab in critically ill COVID-19 patients based on anecdotal evidence and hypothetical mechanisms. However, most centers did not use tocilizumab in COVID-19 given the lack of traditional RCTs or evidence in non-COVID-19 ARDS. The anecdotes of Era 1 inspired more rigorous translational studies quantifying the dysregulation of the COVID-19 inflammatory responses. These translational studies set the stage for new RCTs on immunomodulation, and in January and February 2021, the REMAP and RECOVERY trial groups uploaded preprints of their adaptive RCTs that supported a mortality benefit from tocilizumab in COVID-19 inpatients. In a major shift from pre-COVID-19 practices, many centers used the preprint data to justify new guidelines for tocilizumab in hospitalized COVID-19 patients. Previously, multiple peer-reviewed publications and national consensus guidelines would have been required before adopting an entirely new class of intervention. Tocilizumab’s increased use demonstrated clinicians’ growing confidence in their ability to parse which preprints were appropriate to use for immediate changes in practice.

A new commitment to open access research aided rapid evaluation of evidence. The January 2020 Wellcome COVID-19 statement made a commitment to data sharing, preprints, and open access publications; over 150 publishers, funders, and scientific institutions signed it. COVID-19-related research exploded into an estimated 150,000 manuscripts in Eras 1 and 2, of which 20%–30% were preprints., Although quality was difficult to assess, COVID-related preprints were shared and cited more widely and reached peer-reviewed publication more often than their non-COVID-19 counterparts.

Pitfalls of Era 2: Substandard clinical trials and delayed implementation of strong evidence

While many journals encouraged preprints, the low barriers to entry increased the number of ill-supported conclusions presented to frontline clinicians. In Era 2, a desire for therapeutics produced a glut of poorly designed, underpowered single-center trials. More problematic were the large numbers of preprinted poor-quality trials and meta-analyses that went viral on social media. Although social media helped share vital knowledge early in the pandemic, the notion that “all data are good data” was severely tested, as in the case of hydroxychloroquine. Hydroxychloroquine had demonstrated in vitro activity against SARS-CoV-2; what followed was a poorly designed, non-randomized study of 36 patients that claimed to prove its benefit. Politicians and social media algorithms inappropriately amplified this and similar studies to feed a movement based on substandard science. Within months, definitive RCTs proved hydroxychloroquine’s lack of efficacy, but not before thousands of patients were exposed. Despite the significant scientific advances during this time, entire domains of inpatient care remained under-investigated, such as methods to prevent in-hospital transmission (e.g., the efficacy of negative airflow rooms) or algorithms to ration scarce resources during crisis standards of care and their potential to exacerbate racial, ethnic, or socioeconomic inequities.

Era 2 summary

As cases surged in Era 2, an influx of good and poor-quality research flooded the medical community. Era 2 had notable successes in innovating approaches to evidence, such as adaptive design in RCTs and utilization of preprints. However, pitfalls in Era 2 included an inundation of low-quality studies, understudied topics in patient care, and inequities in COVID-19 outcomes across race, ethnicity, and socioeconomic status.

The current era: The inflection point (approximately April 2021 to present day)

In January 2022, an unvaccinated young man arrives in an emergency department with shortness of breath and hypoxemia due to COVID-19. He refuses dexamethasone and demands ivermectin. With vaccines widely available in the USA, hospital beds are filled with unvaccinated patients, alongside a fraction of those who are vaccinated but immunocompromised. COVID-19 variants and loosening of social restrictions brings a dramatic rise in cases, including in children who had previously been spared.

In the current era of new variants, a familiar sense of unease and uncertainty emerges. How will hospitals and patients fare with a recurrence of Era 2’s challenges?

Current successes and pitfalls

Conventional pre-pandemic wisdom stated that it took years for a research publication to change bedside practice. Now, it’s remarkable how quickly sound RCTs can be completed and used to change bedside practice. Utilization of “living” online guidelines and other sources of continually updated guidance, such as commercial websites (e.g., https://www.uptodate.com/), government agencies and professional societies (e.g., the U.S. National Institutes of Health, the Infectious Diseases Society of America, and the World Health Organization), and institutional platforms (e.g., the authors’ COVID-19 dashboard at https://covid19treatmentguidelines.org/), will continue to help physicians absorb the next wave of evidentiary challenges. Our current challenges include how to best communicate the ongoing scientific discussion to the public. As with hydroxychloroquine, social media helped fuel public demand for ivermectin, a drug with poor-quality meta-analyses based on low-quality or fraudulent clinical trials. Social media misinformation about vaccine science and therapeutics is misleading a significant proportion of the globe. Researchers must now anticipate how their preprint results could be misconstrued in a manner that counters public health messaging. The early commitment of journals to open access for COVID-19 articles was a successful innovation to increase global equity. Unfortunately, that same goal has not been realized with participation in RCTs or the allocation of vaccines and therapeutics across the globe. Furthermore, inequities continue to persist in outcomes along lines of race, ethnicity, and socioeconomic status.

In contrast to the early days of the pandemic, the impetus to put untested ideas into practice has been tempered by previous failures and established alternatives. Unlike as in Eras 1 and 2, it may take more than one or two positive studies to put a novel therapeutic to use. For example, both gold standard trial networks and smaller groups have published their RCT and cohort studies for therapeutic anticoagulation and deep vein thrombosis (DVT) prophylaxis in COVID-19 inpatients, but great variation remains in guidelines across different institutions. A further challenge for clinical researchers is how to operate in an environment with established therapies for COVID-19. With dexamethasone and tocilizumab as a standard of care, reaching clinical equipoise is challenging. Is it ethical to have a placebo arm without tocilizumab? Would providers hesitate to enroll patients in a trial adding another immunomodulator on top of tocilizumab?

Frontline clinicians continue to be flooded by new research, particularly as the rapid pace of the pandemic has enlarged the scope of research considered relevant to bedside care. For example, a bench research study on the recognition of SARS-CoV-2 viral variants by T cells from vaccinated individuals could soon inform practice. There is no chance of any single practitioner navigating the rising volume of research without ongoing guidance from trusted institutions and their living guidelines.

Lastly, changes in outpatient care may take center stage. The trajectory of a hospitalized patient’s illness is greatly determined by interventions that occur before the hospital, such as vaccination, anti-spike protein monoclonal antibodies, and oral antiviral treatments. The impact of new variants highlights the need for both further research and a global response to inequities in vaccination and testing.

Conclusion

After the HIV epidemic was first detected, it took several years for the sequencing of the retrovirus and nearly 15 years before the first protease inhibitor. We contrast that with today’s SARS-CoV-2 pandemic and its unimaginable pace in the development of novel therapeutics. The prior 2 years provide a case study in new approaches to the acquisition and application of evidence, with all their successes and pitfalls. However, the previous eras may not predict clinician behavior in the new evidentiary and societal environment of an endemic disease. Looking forward, as we anticipate new practice-changing data and clinical needs, how we innovate approaches to evidence-based medicine should be closely examined.