The COVID-NMA Project: Building an Evidence Ecosystem for the COVID-19 Pandemic.

Even before the coronavirus disease 2019 (COVID-19) pandemic, the ability of the evidence synthesis model to meet the needs of stakeholders was challenged (1, 2). There are too many low-quality systematic reviews that mainly address pairwise comparisons and are rarely updated, resulting in redundancies and gaps. Producing high-quality, up-to-date systematic reviews requires substantial time and resources. In addition, although evidence synthesis is directly affected by the quality of primary research, interaction is limited between the evidence generation and synthesis communities. These issues have been highlighted and exacerbated by the COVID-19 pandemic, where stakeholders urgently need relevant, accessible, up-to-date, and trustworthy syntheses of high-quality evidence to inform their decisions. Thousands of randomized controlled trials (RCTs) have been initiated during the pandemic, and their results are frequently rushed to publication or communicated through non–peer-reviewed preprints. The situation is further complicated by changes in the questions of interest and trial components (such as standard of care) as the pandemic develops (3).

To tackle COVID-19, we developed and implemented a previously proposed model (4, 5) to address the challenges and help to connect evidence generation, synthesis, and decision making. Rather than focusing on 1 specific treatment or comparison, the COVID-NMA project provides a living mapping of all trials and a comprehensive living synthesis of all available trial evidence evaluating the effect of interventions for the prevention or treatment of COVID-19 (Figure). We developed a master protocol (6) and subprotocols dedicated to specific questions, which are discussed and agreed on by a steering committee.

Figure.

Process of the COVID-NMA project.

The project aims to provide an up-to-date mapping of trials; a comprehensive, critical, up-to-date synthesis of all available trial-based evidence about the efficacy and safety of interventions for the prevention or treatment of coronavirus disease 2019; and a living monitoring on trial planning, conduct, and reporting. ICTRP = International Clinical Trials Registry Platform; RCT = randomized controlled trial.

Every week, we screen the COVID-19 database produced by the World Health Organization's International Clinical Trials Registry Platform to identify eligible RCTs. The living mapping produced provides a description of all registered RCTs. The data retrieved and extracted can be explored through interactive data visualizations to identify research gaps and help prioritize and improve future trials.

We are also conducting a living systematic review based on a living protocol (6) that is scalable to stakeholders' evolving needs. All changes in the protocol (for example, primary study design and outcomes) are discussed by a steering committee and reported transparently. As part of the living process, we do a systematic search daily, collect data as soon as we identify any trial that has published results or is available in preprint, and assess risk of bias fully using the Cochrane Risk of Bias Tool, version 2.0 (7). We provide the descriptive data online and produce forest plots of appropriately pooled data with GRADE (Grading of Recommendations Assessment, Development and Evaluation) summary-of-findings tables and evidence profiles. We have developed a tool to automatically identify new versions or publication of preprints. We contact trialists at the outset (that is, trial registration) to request information (protocol) and inform them of the outcomes (consistent with the core outcome sets developed by the COMET [Core Outcome Measures in Effectiveness Trials] initiative [8, 9]) that should be reported to enable their trial to be incorporated into the meta-analyses. When results are available, we systematically request from trial authors any missing data and update the reviews accordingly. We have established robust quality control processes in collaboration with the Cochrane Bias Methods Group. Collectively, COVID-NMA data are used to conduct systematic reviews on specific questions, meta-analyses of individual participant data (IPD), and network meta-analyses and to support the guideline development process and health decision making. Our databases can also be shared to allow guideline developers to do their own analyses.

To improve research planning, we monitor trials' quality related to outcomes, completeness of reporting (that is, adherence to some CONSORT [Consolidated Standards of Reporting Trials] items), risk of bias, and data sharing (intended and realized). As a feedback loop, we provide trialists and funders the results of this monitoring to increase the value of COVID-19 trials research. We also send automatic e-mails to investigators of completed trials to encourage them to post results on registries (10) and share IPD, and we have developed a secure process to enable them to do this at no cost.

Our collaborative project involves an international consortium of 85 persons, including methodologists, clinicians, and statisticians. On 31 August 2020, our research mapping identified 1686 registered RCTs, of which 944 are recruiting. Overall, 54% have fewer than 100 participants. We have screened more than 42 000 records and reported detailed data for 45 RCTs, with forest plots for all comparisons. We have contacted about 1000 investigators of ongoing trials and requested missing data from 45 authors.

This new approach is creating challenges and threats. First, sustainability is an issue as the crisis continues. We developed COVID-NMA with the support of many volunteers from various countries who were available during the containment period but must now return to normal activities. As the amount of data increases, we need to move to a long-term and sustainable structure with a website that is more accessible and useful to end users. The resources necessary to maintain this model are critical because the volume of evidence is increasing, the scope is expanding at end users' request (for example, new focus on vaccine trials), and new sources (clinical study reports) or new types of data (such as IPD) are becoming available. We need funders to provide long-term funding for this platform. This would be far more cost-effective than funding a disparate and uncoordinated series of systematic reviews on narrow research questions.

Second, some cultural issues exist. The success of this approach depends entirely on the acceptance of and engagement with this model by stakeholders, in particular funders and trialists. Some may be reluctant to add new outcomes, adhere to reporting guidelines, or share IPD because this involves change in culture, as well as time and effort. We hope that the urgency associated with the COVID-19 pandemic, combined with external pressure, may help to overcome these barriers.

Governance of the project is an important consideration. We must ensure that volunteers and researchers involved in the platform receive the appropriate reward and recognition for their contributions. We are developing transparent processes for both the researchers involved and the users of the data, and our work is overseen by an independent steering committee.

Overall, the present crisis unmasks the shortcomings of the current synthesis model and provides a strong impetus for change and improvement. We hope COVID-NMA plays a role in this work.