Balancing Scientific Rigor With Urgency in the Coronavirus Disease 2019 Pandemic.

Knowledge dissemination during COVID19 has been notable for the rise of pre-prints and publication by press release, favoring speed over accuracy. We stress the importance of journals in this landscape and highlight the value of editorial judgment and peer review.

In the midst of the worst pandemic in a century, the medical community must contend with an unprecedented deluge of scientific information. Coronavirus disease 2019 (COVID-19) has stretched the capacity of journals to ensure rapid dissemination of studies to inform the response to the pandemic while maintaining quality standards. At the same time, the ecosystem of knowledge dissemination is changing, with the rise of nonpeer-reviewed pathways, including the use of preprint servers and the apparent trend of publication by press release. We argue that peer-reviewed journals are more critical than ever, and that it is imperative that journals not abandon principles of scientific rigor in favor of urgency.

Peer review of scientific literature was introduced approximately 300 years ago, but it has only gained widespread traction in the last century [1]. Fundamentally, peer review functions as a check against poor methodology and claims unsupported by the data and is a profound catalyst for improving manuscripts. Conversely, COVID-19 has marked a dramatic rise in posting of clinical manuscripts to preprint servers [2]. These repositories for manuscripts that have not (yet) undergone peer review have been widely used in basic sciences and mathematics. In a pandemic, preprint servers allow for rapid dissemination of results that have the ability to affect clinical management and public health policies, without the need to wait for peer review and editorial decisions, frequently needing to be repeated with multiple submissions. On the other hand, without the quality control of editors and peer reviewers, there is risk that data can be misinterpreted or misrepresented, and readers may not have the expertise to identify methodological weaknesses. In some instances, the lay media has headlined stories based on preprint studies, resulting in widespread dissemination without any quality assurance regarding the methodology, the data, or the validity of the conclusions. Examples from basic science and clinical science preprints that were shared in print and online media include a study that suggested sequence homology with human immunodeficiency virus [3] (which fueled conspiracy theories of laboratory-generated bioweaponry [4]) and a cohort study examining use of hydroxychloroquine in Veterans Affairs hospitals [5] that was touted by media as evidence that the drug caused harm without mentioning the concern of confounding by indication. Although the former study was retracted, the latter was ultimately published in a peer-reviewed journal with some methodological flaws addressed and conclusions tempered [5, 6]. However, there were little to no corrections in the lay media, as the headlines have moved to other stories. Another concern that has been raised is that although preprint publications can spur practice changes, their retractions do not leave a record that the study findings are disputed. As an example, when a preprint study suggested that ivermectin was associated with dramatic survival benefits [7], the drug was rapidly incorporated into national protocols in some Latin American countries [8]. Although the study was retracted from the preprint server, this has been done quietly, and without a mark on the permanent record that would accompany a journal article retraction, making it more difficult to convince policymakers to reverse course [8].

Furthermore, we have witnessed an apparent increase in study results being disseminated first through press release, sometimes with publication delayed for weeks or months to follow. For example, a French group released a vague headline that a tocilizumab trial appears to improve mortality in COVID-19, but without releasing any data (other than the total number of participants in each arm) [9], in the same week that a study into sarilumab, an alternate agent for interleukin-6 blockade, was sharply curtailed by the sponsor for futility [10]. It is unfortunately, people may have been treated with harmful medication based on press releases as we await detailed publications.

While peer reviewing and making editorial decisions, scientists should be weary of uncontrolled studies of treatment. The reliance on study designs without controls is fraught with peril. Even within the European Union, the mortality rate has ranged from 15.4% (Belgium) to 3.8% (Germany) [11], making comparisons to a universal baseline impossible. This mortality also varies between stages of the pandemic curve; the mortality is much lower in the beginning of the local outbreak, while healthcare resources are available, and peaks when the disease peaks itself, as healthcare is stretched. Those 2 characteristics make uncontrolled trials largely uninterpretable [12-15]. Studies of treatments that suggest (1) “good” or “better than expected” outcomes or (2) use of study methodologies without a control group are thus unhelpful. These designs do not answer a question about efficacy of an intervention, because positive and negative outcomes can be coincidence, and this could misdirect research from promising treatment candidates.

Reviewers should also be cautious of choices of poor outcomes in the literature, most notorious of which is how patients who are still hospitalized are managed in the dataset. A now notorious study was widely discussed for suggesting 75% mortality in vented patients [16], because patients that were still hospitalized at the closure of the study were excluded. Such details can and should be addressed before publication.

In this landscape, journals are critically important because manuscripts are carefully screened for quality by editors and reviewers. However, peer review is not infallible, and we have also seen dramatic examples of journal editors and reviewers failing to detect problematic irregularities in consequential studies. Over the course of 3 weeks in May 2020, both the New England Journal of Medicine and The Lancet published studies from a previously unknown (but now infamous) registry called Surgisphere [17, 18]. Within days of the latter publication, readers detected inconsistencies that raised major concerns about the veracity of the study [19], which the authors claimed was the result of analyzing data from over 96 000 hospitalized patients from 671 hospitals on 6 continents. On June 4, 2020, when it became clear the data could not be verified, both articles were retracted [17, 18]. In another example, a letter published in Annals of Internal Medicine that claimed that masks were ineffective at blocking emission of viral particles during coughing was retracted because reader comments led the authors to realize that they did not appreciate the limit of detection of their assay [20].

The consequences of publishing poor science can be substantial. A French study published by the International Journal of Antimicrobial Agents that claimed that hydroxychloroquine and azithromycin cleared the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) from patients with 100% efficacy [15] ultimately led to the endorsement of the regimen by President Trump and an emergency use authorization for hydroxychloroquine from the US Food and Drug Administration, ultimately to be revoked months later. Publication of the Surgisphere study by The Lancet caused instant upheaval: the reported findings that chloroquine or hydroxychloroquine were associated with a 30% adjusted increase in mortality [17] led to temporary suspension of dozens of randomized control trials (RCTs) studying the drug, including the global World Health Organization-led SOLIDARITY trial [21]. Ultimately, this delay in achieving definitive answers from RCTs comes at the cost of faster results, and the ability to widely roll out or sharply curb hydroxychloroquine use. Rejected funding applications cited the study as a safety concern. Most importantly, these episodes have resulted in the erosion of trust in science. Ultimately, much like the parable of the turtle and the hare, when speed is prioritized in early publications, much time is wasted on studies that refute those early weak findings, instead of focusing on more promising avenues of research.

Journals have been inundated with submissions relating to COVID-19, and the burden to review these falls to editors and peer reviewers. Moreover, the massive scale of the global impact from the pandemic, in an age of digital interconnectedness, has resulted in a widespread demand for immediate results. The literature cycle in COVID-19 is churning at warp speed. Editors are human, and they make mistakes. Moreover, peer review can be expected to falter as a consequence of increased pressure for rapid turnaround of reviews, especially when reviewers are unpaid volunteers who may already be stretched thin by the extraordinary professional and personal obligations during COVID-19.

With clinical and research programs suspended by COVID-19, and as attention on COVID-19 among funders becomes hyperfocused, it is unsurprising that researchers from a range of academic backgrounds are gravitating to study this disease. Similarly, journals from a wide range of disciplines have begun publishing on the topic. One result is that editors and reviewers are evaluating studies without the expertise required. Editors of the New England Journal of Medicine have stated that the lack of expertise for reviewing studies with analyses of big datasets from electronic medical record data contributed to publishing the since-retracted Surgisphere manuscript [22]. Proceedings of the National Academy of Sciences editors also failed to recognize major flaws in articles on SARS-CoV-2 phylogenomics [23] and transmission [24]. In both cases, articles were contributed by National Academy of Sciences members, which allow authors to hand-select and approach reviewers before submission.

CONCLUSIONS

Gains in scientific publishing—driven by editorial decision-making and more recently peer review—have helped move medicine away from the days of leeches, bloodletting, and purgatives. Although there is a desire for immediacy that is inherent to the pandemic, it is imperative that we do not abandon principles of science that moved us away from those dark times for the sake of expediency. Every day that passes without an effective treatment, more patients will suffer, and more will die. However, seriously flawed studies that are lent credibility by publication in reputable journals and are amplified by journalists and political figures do not merely fail to move us forward, but they also set us backwards. They delay us in finding a treatment on which lives may depend by sending scientists on wild goose chases, and they erode public trust. Ultimately, it is incumbent on us as scientists, peer-reviewers, and editors to make sure that this literature does not develop at the expense of diligence and accuracy. Journals are and will remain the most important avenue for communicating scientific results, which is why editors need to balance the desire to publish helpful data with maintaining quality, peer-reviewers need to continue their skepticism, and scientists need to engage in methodology over publicity.