Clinical trials in COVID-19 management & prevention: A meta-epidemiological study examining methodological quality.

OBJECTIVE: To describe the characteristics of Covid-19 randomized clinical trials (RCTs) and examine the association between trial characteristics and the likelihood of finding a significant effect. STUDY
DESIGN: We conducted a systematic review to identify RCTs (up to October 21, 2020) evaluating drugs or blood products to treat or prevent Covid-19. We extracted trial characteristics (number of centers, funding sources, and sample size) and assessed risk of bias (RoB) using the Cochrane RoB 2.0 tool. We performed logistic regressions to evaluate the association between RoB due to randomization, single vs. multicentre, funding source, and sample size, and finding a statistically significant effect.
RESULTS: We included 91 RCTs (n = 46,802); 40 (44%) were single-center, 23 (25.3%) enrolled <50 patients, 28 (30.8%) received industry funding, and 75 (82.4%) had high or probably high RoB. Thirty-eight trials (41.8%) reported a statistically significant effect. RoB due to randomization and being a single-center trial were associated with increased odds of finding a statistically significant effect.
CONCLUSIONS: There is high variability in RoB among Covid-19 trials. Researchers, funders, and knowledge-users should be cognizant of the impact of RoB due to randomization and single-center trial status in designing, evaluating, and interpreting the results of RCTs. REGISTRATION: CRD42020192095.

There is important variability in risk of bias (RoB) amongst covid-19 trials.

Most trials of Covid-19 prophylaxis and therapy trials had a high or probably high RoB in at least one domain.

RoB due to the randomization process and single center trial status were associated with a three-fold increase in the odds of finding a statistically significant effect on a study's primary outcome.

Trial characteristics, including RoB, contribute to low quality evidence which may be misleading to knowledge-users, cause harm to patients, and absorb a disproportionate amount of attention and resources away from other potentially effective interventions.

Researchers and funders are encouraged to consider the potential impact of RoB in their design and prioritization of RCTs.

Knowledge users should consider trial design characteristics in their critical appraisal and application of trial findings.

Introduction

The rapid rise in the number of cases, hospitalizations, and deaths due to Coronavirus disease 2019 (Covid-19) has been paralleled by an exponential rise in scientific publications related to Covid-19. The number publications with the terms ‘COVID-19’ or ‘SARS-CoV-2’ in their title or abstract was over 17,000 as of May 31, and over 57,000 as of October 5, 2020.

The global search to identify effective interventions against Covid-19 has led to an unprecedented rise in clinical trial activity worldwide. As of October 5, 2020, the World Health Organization (WHO) Global Coronavirus COVID-19 Clinical Trial Tracker reports that there are currently over 2,300 clinical trials at various stages of completion. The rapidity with which clinical trials in Covid-19 are being planned, completed, and disseminated has triggered concerns about their methodological quality [1,2]. Flaws in study design may lead to biased estimates of intervention effects, leading to treatment decisions that are at best ineffectual, and at worst harmful to patients. The well-known waste in biomedical research may be enhanced by the COVID-19 pandemic [3].

Several recent reports have described the design characteristics of registered trials of Covid-19 therapies [3], [4], [5], [6], [7], [8]. These reports, however, are based on registered trials, many of which will not proceed to completion and will therefore not impact clinical knowledge or practice. In addition, the appraisal of trial quality from registries does not include assessment of trial conduct as well as analysis.

We conducted a meta-epidemiological study of published Covid-19 randomized controlled trials (RCTs) to (1) describe trial characteristics, including risk of bias (RoB), and (2) evaluate the association between trial characteristics and the likelihood of finding statistically significant results for the primary outcome.

Methods

Study design

We performed this meta-epidemiological study as part of a living systematic review and network meta-analysis of RCTs examining Covid-19 prevention and therapy [9]. We prepared this manuscript in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement [10].

Data sources and searches

The complete search strategy is shown in Appendix A. We used the ongoing literature search performed by Centre for Disease Control, which includes 25 databases of published studies and repositories of unpublished studies (medRxiv and bioRxiv), to find potentially relevant articles of therapies related to SARS-CoV-2 and COVID-19 from January 1 to October 21, 2020. To identify randomized trials, we filtered the results of the daily searches through a validated and highly sensitive machine learning model [11]. For pragmatic reasons, we excluded trials published in languages other than English.

Study selection

We included English language RCTs of any publication status (peer-reviewed publication or preprint) that enrolled patients with suspected, probable or confirmed COVID-19, or at risk for contracting COVID-19, and compared the effect of pharmacologic agents or blood products against standard care, a placebo, or an active comparator (i.e., another pharmacologic agent or blood product). We excluded trials of vaccines or traditional herbal medicines that included more than one molecule or did not have a specific molecular weight dosing.

Working in pairs, reviewers screened, independently and in duplicate, titles and abstracts and then full-texts for articles found potentially eligible at the title and abstract screening stage. We resolved discrepancies by discussion and where needed, by third party adjudication.

Data extraction and quality assessment

Using a pre-developed data extraction form, we extracted study characteristics including: registration status (registered vs. non-registered), publication status (preprint vs. peer reviewed publication), trial design (single-center or multicenter), funding source (industry vs. non-industry), study interventions (number of study arms, intervention details, type of comparator [active vs. not]). We also extracted details about the trial's reported primary outcome(s), including whether the outcome was binary vs. continuous vs. ordinal, patient-important or surrogate, event rates and summary statistics for binary and continuous outcomes, respectively, and whether there was a statistically significant difference detected in the primary outcome. For the purposes of this analysis, where a trial did not indicate a primary outcome and reported more than one primary outcome, we included the outcome which had the largest relative treatment effect between study arms.

Three reviewers evaluated RoB of included studies using the modified version of the Cochrane RoB 2.0 tool independently (Appendix B) based on information from the trial publication, pre-print, and protocol registration, as available. Discrepancies were resolved by consensus. The modified Cochrane RoB tool rates methodological quality of each included study as low, probably low, probably high, or high RoB across each of five domains, reflecting bias: (i) from the randomization process, (ii) due to deviations from the intended intervention (which included blinding procedures), (iii) due to missing data, (iv) due to measurement of the outcome, and (v) in selection of the reported results. We categorized overall study RoB as the highest rating in any of the five domains (i.e., if one domain is rated as ‘high’ RoB, then the overall study RoB was rated as ‘high’).

Data synthesis and analysis

We used descriptive statistics (means and standard deviations, medians and interquartile ranges, and proportions and confidence intervals, as appropriate) to summarize trial characteristics and RoB for the included trials.

We then conducted logistic regression analyses to assess the association between a trial finding a statistically significant effect (defined as a P-value equal to or less than 0.05) and pre-specified trial characteristics, including:

RoB due to randomization: dichotomized into low/ probably low RoB and high/ probably high RoB

Centre status: Multicenter vs. single center trial

Funding source: those with any industry funding vs. those without industry funding

Trial sample size (using the total number randomized as a continuous variable)

We selected these trial characteristics a priori based on the hypothesis that these specific trial characteristics were most important in influencing trial findings. We included RoB due to the randomization process, as opposed to other RoB domains, as we anticipated the randomization process, which incorporates the procedures used to randomize participants, allocation concealment, as well as baseline imbalanced in unblinded trials, to have the highest association with trial outcomes and due to the anticipated limited variability between trials in other RoB domains, which would not allow for meaningful interpretation or conclusions.

Among the four selected predictor variables, we used purposeful selection of predictor variables according to the approach described by Bursac and colleagues [12]. The process began with univariate analysis of each of the four pre-specified predictors. Then, variables that yield a P-value of less than 0.25 are selected as candidates for the multivariable analysis and entered into the model. Through an iterative process of variable selection, variables are retained in the model only if they (1) have an association with the outcome as defined by a P-value of <0.1 or (2) have a confounding effect, defined a change in the group coefficient by more than 15% when the variable is removed as compared to the full model. This approach allows for iterative selection of predictor variables and retains in the model those predictors that are not themselves significantly associated with the outcome but contribute to the effect of other predictors. We planned to perform subgroup analyses to evaluate the impact of trial characteristics on trial outcomes among trials that were preprints compared to those published in peer-reviewed journals but the relatively small number of trials prohibited this analysis. We used Statistical Package for the Social Sciences version 26.0 (IBM Corporation) for all descriptive and regression analyses and Stata/IC 16.1 (StataCorp LLC) to produce the forest plot of effect sizes.

Role of the funding source

The funding source had no role in the design of the study, the collection, analysis and interpretation of the data, and the decision to approve publication of the finished manuscript.

Protocol registration

We registered the protocol for this study in the Prospective Register of Systematic Reviews (PROSPERO 2020: CRD42020192095).

Results

The search identified 13,536 records which were reviewed in duplicate as part of a living network meta-analysis, and yielded 103 trials of therapeutic or prophylactic interventions for Covid-19 [9]. We excluded five RCTs published in languages other than English, two trials that reported on a cohort overlapping with another included trial, two that reported preliminary results but not findings related to their primary outcomes, and three unpublished studies that were included in a meta-analysis with insufficient information to include in our review. We included a total of 91 clinical trials (54 peer-reviewed publications, 37 preprints) in this analysis.

Trial characteristics

Overall trial characteristics

Table 1 and Appendix C present the aggregate characteristics of included studies. The 91 included trials enrolled a total of 46,802 patients between January 18 (first recruitment) and October 4 (last recruitment). Included trials evaluated one or more drugs (n = 76,[13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87], [88]) or blood products (n = 9,[89], [90], [91], [92], [93], [94], [95], [96], [97]) to treat patients with suspected or confirmed Covid-19 or drugs used as prophylaxis for patients at risk for Covid-19 (n = 6, [98], [99], [100], [101], [102], [103]). All but one of the trials were parallel group design (one trial was a cluster randomized design). Thirty of 91 trials were conducted by a country in the Western Pacific Region, primarily China (n = 27). Fig. 1 illustrates the proportion of trials that were led by countries in various regions, as defined by the WHO. All but three trials were pre-registered. Fifty-one trials were multicenter whereas 40 were single center. Trial sample size ranged from 10 to 14,247 (median: 84, interquartile range [IQR]: 151); 23 trials enrolled less than 50 patients, 51 enrolled 50 to 400 patients, and 17 enrolled over 400 patients. Only one trial was terminated early by the data and safety monitoring board due to slowed recruitment as a result of declining cases of Covid-19 [75].

Table 1

Study characteristics & risk of bias

Study characteristics		All studiesN = 91	Statistically significant effect reported?
			YesN = 38	NoN = 53
Design characteristics
Centre status	Single centerMulticenter	40 (44%)51 (56%)	24 (63.2%)14 (36.8%)	16 (30.2%)37 (69.8%)
Funding source	No industry funding/ supportIndustry funding/ support/ not reported	60 (65.9%)31 (34.1%)	25 (65.8%)13 (34.2%)	35 (66%)18 (34%)
Trial sample size	Median (IQR)	84 (48, 199)	77 (33, 100)	102 (54.5, 402.5)
Level of blinding	UnblindedOnly patients blindedPatients and treating clinicians blinded	68 (74.7%)3 (3.3%)20 (22.0%)	30 (79.0%)2 (5.3%)6 (15.8%)	38 (71.7%)1 (1.9%)14 (26.4%)
Type of intervention	Therapeutic interventionPharmacological agentBlood productProphylaxis intervention	76 (83.5%)9 (9.9%)6 (6.6%)	34 (89.5%)4 (10.5%)0 (0%)	42 (79.2%)5 (9.4%)6 (11.3%)
Risk of bias (RoB)
Overall RoB	Low/ Probably LowHigh/ Probably High	16 (17.6%)75 (82.4%)	6 (15.8%)32 (84.2%)	10 (18.9%)43 (81.1%)
Bias from randomization process	Low/ Probably LowHigh/ Probably High	43 (47.3%)48 (52.7%)	10 (26.3%)28 (73.7%)	33 (62.3%)20 (37.7%)
Bias due to deviation from intended intervention	Low/ Probably LowHigh/ Probably High	16 (17.6%)75 (82.4%)	5 (13.2%)33 (86.8%)	11 (20.8%)42 (79.2%)
Bias due to incomplete outcome data	Low/ Probably LowHigh/ Probably High	85 (93.4%)6 (6.6%)	35 (92.1%)3 (7.9%)	50 (94.3%)3 (5.7%)
Bias due to primary outcome measurement	Low/ Probably LowHigh/ Probably High	79 (86.8%)12 (13.2%)	31 (81.6%)7 (18.4%)	48 (90.6%)5 (9.4%)
Bias due to selective outcome reporting	Low/ Probably LowHigh/ Probably High	88 (96.7%)3 (3.3%)	37 (97.4%)1 (2.6%)	51 (96.2%)2 (3.8%)

Fig. 1

The geographical distribution of trials according to WHO region.

Among 88 studies that reported their funding source, 28 received at least some industry support including complete industry funding in 10 trials, partial industry funding for 7 trials, and provision of intervention/ medications by industry in 11. The 60 trials that reported no industry support were funded by governmental sources (n = 31), academic institutions (n = 9), multiple sources (government, academic institutional, and/ or not-for-profit organization; n = 13) or received no funding (n = 7).

Trial risk of bias

There was variability across various RoB domains. Seventy-five (82.4%) having overall high or probably high RoB (Table 1). Across individual RoB domains, there was high/ probably high RoB from the randomization process in 48 trials (52.7%), due to deviations from the intended protocol (which incorporates blinding procedures) in 75 (82.4%), due to incomplete primary outcome data in 6 (6.6%), due to incomplete primary outcome measurement in 12 (13.2%), and due to selective outcome reporting in 3 (3.3%; Table 1).

Trial primary outcomes

Appendix C presents the primary outcomes of included studies and their characteristics. The primary outcomes were binary in 39 trials, continuous in 37, ordinal in 5, and the remaining 10 trials reported more than one primary outcome. Among the 85 therapy trials, most trials (26% or 28.6%) reported a measure of clinical recovery or symptom resolution as the primary outcome. Thirty-eight studies reported a statistically significant effect (41.8%) and 53 reported no statistically significant difference (58.2%; Appendix C).

Association between trial characteristics and findings

We evaluated the association between each of the pre-specified trial characteristics on trial findings (whether or not a statistically significant effect was found). Bias due to the randomization process was high or probably high in 28 of 38 (73.7%) of trials that found a statistically significant effect on their primary outcome, compared with 20 of 53 (37.7%) of trials that found no statistically significant effect. Fig. 2 shows the RoB across the five domains on the modified Cochrane RoB tool across the two groups of trials (additional details provided in Appendix D).

Fig. 2

Risk of bias according to trial finding.

Single center studies accounted for 24 of 38 (63.2%) trials that reported a statistically significant effect compared with 16 of 53 (30.2%) trials that reported no statistically significant effect (OR 3.93, 95% CI, 1.38–11.19). Thirteen of 38 trials (34.2%) that found a statistically significant effect were industry funded compared with 18 of 53 (34.0%) trials that found no statistically significant effect (OR 1.82, 95% CI, 0.61–5.43). Median sample size was 77 (IQR: 67) among trials that found a statistically significant effect and 102 (IQR: 348) in trials that found no statistically significant effect (OR 1.00 per patient randomized, 95% CI: 1.00–1.00, P = 0.74). Bias due to the randomization process was associated with higher odds of finding a statistically significant effect (OR 3.89, 95% CI, 1.46–10.36).

In univariate analysis, only bias due to the randomization process was associated with trial outcome (whether or not a statistically significant intervention effect was found); there was no association between trial outcome and center status, funding source, and sample size (Table 2 ). In multivariable analysis, we found that higher bias due to the randomization process (OR 3.77, 95% CI, 1.47–9.72) and single center trial status (OR 3.15, 95% CI, 1.25–7.97) were predictors of a trial finding a statistically significant effect.

Table 2

Association between trial characteristics and statistically significant results in primary outcome of Covid-19 clinical trials

Predictor variables	Univariable analysis		Multivariable model
	OR (95% CI)	P	OR (95% CI)	P
RoB due to randomization processa	3.89 (1.46–10.36)	0.01	3.77 (1.47 to 9.72)	0.01
Single center vs. multicenter	3.93 (1.38–11.19)	0.01	3.15 (1.25 to 7.97)	0.02
Industry vs. non-industry support	1.82 (0.61–5.43)	0.28	-	-
Total sample size	1.00 (1.00–1.00)	0.76	-	-

Abbreviations: CI, confidence interval; OR, odds ratio; RoB, risk of bias

Dichotomized into low/ probably low vs. high/ probably high

Conclusions

In this meta-epidemiological study of clinical trials of Covid-19 prophylaxis and treatments, we found that 82.4% of trials had high or probably high RoB, 82.4% due to deviations from the intended intervention (including blinding) and 52.7% due to the randomization process (including allocation concealment and adequacy of the randomization procedure). Other trial characteristics were highly variable across studies: 44% were single center trials, slightly less than one-third received at least some support from an industry source and all but 3 trials were registered in advance. Sample sizes were highly variable across studies, ranging from 10 to over 14,247, with one-quarter enrolling less than 50 patients.

The Covid-19 pandemic has seen the global research community embark on a collective search to identify effective prophylactic and therapeutic interventions against the disease. This global response has substantially exceeded that of previous pandemics: in the first six months, thousands of clinical trials had already been registered and hundreds were underway, compared with 71 registered trials after the onset of the H1N1/09 virus pandemic in 2009 and no registered trials after the severe acute respiratory syndrome (SARS) and Middle East Respiratory Syndrome epidemics during the same time frame [104]. This pandemic has also seen an unprecedented level of public interest. Early research findings are now routinely disseminated by researchers in preprint form (bypassing the long-held tradition of peer-review process), and on social media by mainstream media and the healthcare community. In most cases, this is done with inadequate attention to issues related to study design and methodologic quality.

Trial characteristics, including RoB, lead to low quality evidence, which may be uninformative at best and may cause harm to patients. In addition, poor quality trials absorb a disproportionate amount of attention from the general public and divert attention and research resources (i.e., efforts, financial support) away from other interventions which may be beneficial but remain under-investigated. These concerns are undoubtedly compounded when we consider the research resources allocated observational studies and RCTs that remain unpublished. The ultimate effect may be diminished public confidence in the scientific process, especially as data from low quality trials may not be reproducible and likely to be contradicted in subsequent, well-designed trials. In this study, we found that bias due to randomization process and single center trial status were associated with increased odds of finding a statistically significant effect on the primary outcome, independent of the effect of sample size or industry funding source.

Bias due to the randomization process, including inadequate randomization procedures, failure to ensure allocation concealment, and lack of blinding, increases the risk of selection bias. We found that RoB due to the randomization process was associated with a three-fold increase in the odds of a trial finding a statistically significant effect. Previous studies have shown a similar association between selection bias and increased estimate of treatment benefit [105], [106], [107], [108], [109]. Similarly, a systematic review found that selection bias (due to inadequate randomization procedures and allocation concealment) was the most methodological bias across registered clinical trials of Covid-19 therapies [8].

We also found that single center trial status was associated with a three-fold increase in the odds of a trial reporting a statistically significant interventional effect relative to multicenter trials, independent of the effect of sample size. The lack of an association between industry funding and the likelihood of finding a statistically significant effect is consistent with the findings of some previous meta-epidemiological studies [110,111], but inconsistent with other studies that found that industry funded trials are more likely to report a statistically significant effect [112,113]. Lastly, we found no association between sample size and the likelihood of a statistically significant effect. While a previous meta-epidemiological study showed that small studies tend to overestimate effect sizes [114], that study also found that smaller trials had higher RoB across all domains, which may be the more likely explanatory variable.

This study has several strengths. We performed a comprehensive search as part of a living systematic review and NMA peer-reviewed and published in the BMJ, searched a large number of databases, included all Covid-19 RCTs examining drugs or blood products as therapeutics as well as drugs for prophylaxis. This living systematic review is currently informing the WHO living guidelines performed in collaboration with the MAGIC Evidence Ecosystem Foundation [115]. The linkage to these trustworthy guidelines adds further rigor to the assessments of RoB through involvement of methodologists and unconflicted clinical experts making use of GRADE evidence summaries from the systematic review. In addition, we conducted RoB evaluation in duplicate, carefully assessed other trial characteristics that could influence likelihood of findings a statistically significant result. This study has several limitations. First, we did not include non-English trials which may influence the association between trial characteristics and trial outcomes. In addition, it is likely that trials with non-significant findings are less likely to be published than those with that demonstrate a significant treatment effect and our analyses do not account for such potential publication bias. Furthermore, although the current study included only pharmacological agents with a known molecular weight, future analyses can include trials of other drugs, including other traditional medicines, being evaluated for their role in preventing or treating Covid-19. In addition, the relatively small sample of RCTs precluded our ability to conduct pre-planned subgroup analyses to evaluate the impact of trial characteristics on trial outcomes among trials that were preprints compared to those published in peer-reviewed journals. As such, updates on this report as more trials are published will allow for evaluation of a broader range of trial design characteristics and subgroup analyses to further understand the association between trial characteristics and trial outcomes. Finally, a comprehensive evaluation of outcomes being evaluated in clinical trials of Covid-19, including patient-important outcomes, is currently underway [116].

Moving towards producing trustworthy research during a pandemic

These findings offer several future considerations for researchers, funding agencies, and knowledge users. In their design and planning of Covid-19 (or other pandemic-based) trials, researchers are encouraged to consider the impact of trial characteristics and, as with non-pandemic research, strive to generate reliable, high quality evidence. The pandemic should not be an excuse for producing low quality research or cutting corners in trial design. The findings of this review highlight the need for researchers to minimize the risk of producing misleading trial results by focusing on rigor in trial design (often competing with expediency), and with a particular focus on the randomization process. In addition to centralized randomization procedures, based on our findings, researchers are encouraged to prioritize allocation concealment and blinding of healthcare providers, as baseline imbalances between treatments groups contribute to a reduction in the trustworthiness of trial findings. In addition, researcher groups are encouraged to capitalize on the vast collaborations that have evolved through the Covid-19 pandemic in planning future trials as here is enhanced generalizability in multicenter RCTs as compared to single center studies. Specifically, during COVID, we have seen numerous platform trials which have efficiently evaluated multiple interventions in large patient populations across multiple centers and countries [16,44] that serve as fantastic examples of when this works well. In addition, funders should be cognizant of the ongoing research waste, accentuated during a pandemic, limiting their grant support to well-designed trials that are likely to yield reliable, high quality evidence, even if producing this high-quality data takes a little more time. Finally, knowledge users, particularly clinicians, should be mindful of methodological characteristics of RCTs when critically appraising and applying their findings at the bedside.

Author contributions

Kimia Honarmand: Conceptualization, Methodology, Data curation, Formal analysis, Visualization, Writing - Original Draft, Writing- Review & Editing. Jeremy Penn: Investigation, Data curation, Writing-Review & Editing. Arnav Agarwal: Investigation, Data curation, Writing- Review & Editing. Reed Siemieniuk: Conceptualization, Methodology, Supervision, Writing-Review & Editing, Funding acquisition. Romina Brignardello-Petersen: Conceptualization, Methodology, Supervision, Writing-Review & Editing, Funding acquisition. Jessica J. Bartoszko: Methodology, Investigation, Supervision, Data curation, Software, Writing-Review & Editing. Dena Zeraatkar: Methodology, Investigation, Supervision, Data curation, Writing-Review & Editing. Thomas Agoritsas: Methodology, Writing-Review & Editing. Karen Burns: Methodology, Writing-Review & Editing. Shannon M. Fernando: Methodology, Data curation, Writing-Review & Editing. Farid Foroutan: Methodology, Formal analysis, Writing-Review & Editing. Long Ge: Methodology, Formal analysis, Writing-Review & Editing. Francois Lamontagne: Methodology, Writing-Review & Editing. Mario A. Jimenez-Mora: Methodology, Writing-Review & Editing. Srinivas Murthy: Methodology, Writing-Review & Editing. Juan Jose Yepes-Nuñez: Methodology, Writing-Review & Editing. Per O. Vandvik: Methodology, Writing- Review & Editing. Zhikang Ye: Methodology, Data curation, Writing-Review & Editing. Bram Rochwerg: Conceptualization, Methodology, Writing - Original Draft, Writing-Review & Editing. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Ethical approval

Not applicable. All the work was developed using published/pre-print data.

Funding

This is a substudy of the living network meta-analysis, supported by the Canadian Institutes of Health Research (grant CIHR-IRSC: 0579001321). Dr. Rochwerg is supported by a Hamilton Health Sciences Early Career Research Award. The funders had no role in the conduct of this study.

Transparency declaration

KH affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. KH affirms that all authors had access to all the study data, take responsibility for the accuracy of the analysis and had authority over manuscript preparation and the decision to submit the manuscript for publication. All authors approve the manuscript and agree to adhere to all terms outlined in the Journal of Clinical Epidemiology information for authors including terms for copyright.

Dissemination declaration

It is not applicable to disseminate the results to study participants and or patient organizations.

Data sharing

No additional data available.

Provenance and peer review

Not commissioned; externally peer reviewed.

Patient and public involvement statement

It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research.

Conflict of interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: The study received funding support from the Canadian Institutes of Health Research. BR is also supported by a Hamilton Health Sciences Early Career Research Award; LG reports grants from Ministry of Science and Technology of China, outside the submitted work; no other relationships or activities that could appear to have influenced the submitted