A systematic bias assessment of top-cited full-length original clinical investigations related to COVID-19.

With the emergence of coronavirus disease 2019 (COVID-19) pandemic, there has been an exponential rise in the number of scientific publications to convey relevant information about this novel disease [1]. Some data suggest that original investigations related to COVID-19 are published in a fairly shorter time frame by some journals compared with non-COVID articles, which might affect the quality and rigor of these original investigations [2]. Despite these concerns, there is a lack of an objective analysis to study the rigor of original clinical investigations related to COVID-19.

Using Dimensions, an online searchable platform that collects data on >100 million publications, [3] we identified the top 50 cited COVID-19 full-length original clinical investigations on June 24, 2020. The following types of studies were included: observational studies (i.e., diagnostic, prognostic, and non-randomized studies of intervention), randomized clinical trials (RCTs), and meta-analyses. Case reports, case series, and descriptive studies were excluded since these categories of studies are not typically published in high impact journals unless for emerging or exceptional conditions. Modelling studies were excluded since there are no formal quality assessment tool for these studies. Systematic reviews without quantitative data synthesis were excluded since there is a lack of consensus if these types of investigations are considered as original investigations [4]. Brief research reports and research letters were also excluded since we expected that various aspects of the methods might be not fully discussed given the brevity of these publications. Finally, animal and pre-clinical studies were excluded since the focus of this study was on clinical investigations. Studies in language other than English were excluded. In order to obtain a 1:1 matched historical control group, for each COVID-related study, we screened consecutive articles published in the same journal in 2019 until a full-length original investigation of the same aforementioned study design category was identified. If a matched control article could not be identified, then the corresponding COVID-19 related article was excluded. The quality assessment of the articles was performed by 2 independent investigators (NN and JI), and verified by a third investigator (IE). The following checklists were used to evaluate the studies: i) ROBINS-I for non-randomized studies of intervention; [5] ii) the Center for Evidence-Based Medicine tool for diagnostic studies; [6] iii) the Center for Evidence-Based Medicine tool for prognostic studies;[7] iv) RoB 2 for RCTs; [8] and v) AMSTAR 2 for meta-analyses [9]. The percent agreement between reviewers for each study type was calculated.

The top-cited 564 articles related to COVID-19 on June 24, 2020 were screened to identify the top 50 cited full-length original clinical investigations related to COVID-19 (i.e., full-length original clinical investigations compromised 8.8%). The included articles were published in 32 journals. The New England Journal of Medicine had the largest number of articles (n=6, 12%), followed by Radiology (n=5, 10%), followed by both Clinical Infectious Diseases and Journal of Medical Virology (n=3, 6% each). Nearly 40% of the included articles were published in journals with impact factor >10. Observational studies comprised the vast majority of the included studies (82%; prognostic studies [46%], diagnostic studies [22%], and non-randomized intervention studies [14%]), while only 6% were RCTs. Compared with non-COVID full-length original investigations, COVID articles likely originated from Asia (74% versus 18%), were published in open access format (100% versus 66%), and were non-industry funded (72% versus 60%). The median number of citations was higher for the COVID articles (207 versus 10), as well as the median Altmetric score (611 versus 20).

Fourteen non-randomized studies of intervention were scored based on 7 domains of bias. The reviewer agreement was 84%. There was a significant difference in the percentage of domains judged to be at low risk for bias (COVID-articles 35% [17/49] versus non-COVID articles 94% [46/49]; P<0.001). The main domains that were at increased susceptibility to bias in the COVID non-randomized studies of intervention included those related to: confounding, selection of participants, classification of interventions, measurement of outcomes, and selection of reported results (Figure A). The included 22 observational diagnostic studies were scored based on 4 domains of bias. The reviewer agreement was 91%. There was a significant difference in the percentage of domains judged to be at low risk for bias (COVID-articles 64% [28/44] versus non-COVID articles 100% [44/44]; P<0.001). The domains that were most often at increased susceptibility to bias in the COVID diagnostic studies included those related to: lack of representativeness of patient sample, non-uniform and unblinded application of index test and reference standard to all patients in the study. Forty-six observational prognostic studies were scored based on 4 domains of bias. The reviewer agreement was 91%. There was a significant difference in the percentage of domains judged to be at low risk for bias (COVID-articles 60% [55/92] versus non-COVID articles 96% [88/92]; P<0.001). The domains that were most often at increased susceptibility to bias in the COVID prognostic studies included those related to: measurement of outcomes and adjustment for covariates that could affect prognosis (Figure B). There were 6 RCTs that were scored on the basis of 5 domains of bias. The reviewer agreement was 80%. There was no significant difference in the percentage of domains judged to be at low risk for bias (COVID-articles: 80% [12/15] versus non-COVID articles: 87% [13/15]; P=0.99) (Figure C). A total of 12 meta-analyses were scored based on 16 domains of bias. The reviewer agreement was 96%. There was a significant difference in the percentage of domains judged to be at low risk for bias (COVID-articles 61% [59/96] versus non-COVID articles 88% [84/96]; P<0.001). The following domains were more frequently at increased susceptibility to bias in the COVID meta-analyses: assessment and handling of bias risk for the individually included studies (Figure D).

In this cross-sectional analysis of the top 50 cited full-length original clinical investigations related to COVID-19 published early in the pandemic in 32 journals, we found that the vast majority of top-cited full-length original clinical investigations were observational in nature. Compared with a matched historical cohort of non-COVID articles, COVID-related observational studies and meta-analyses were more likely to be at increased risk of bias on several domains. The quality of published RCTs related to COVID were not at increased risk of bias, albeit only a small number of RCTs were included in this analysis.

The observations from this analysis supports the notion that an accelerated process of handling COVID-related articles might have compromised the peer review process and facilitated the publication of some studies at much higher risk for bias than what is typically accepted by the same journals. While there is a timely need to deliver medical knowledge through scientific publications, the introduction of lower quality studies to the medical literature might lead to misdirected academic efforts, drawing inaccurate conclusions, and retractions [10].

This study has limitations that are worth mentioning. The findings of this analysis apply to the included journals, and might not be generalizable to other journals. Some of the included studies were conducted early during the pandemic. Many ongoing high quality RCTs and prospective studies are on the way. It is reassuring that the included RCTs in this study seemed to be at low risk of bias.

Disclosures

The authors declare they have no conflict of interest

Funding

None

Fig. 1

Fig. 1

Panel [A]: risk of bias across the 7 domains of the ROBINS-I tool for assessment of non-randomized intervention studies for (A) COVID-related versus (B) non-COVID related articles; Panel [B] risk of bias across the 4 questions of the Center for Evidence-Based Medicine tool for diagnostic studies for (A) COVID-related versus (B) non-COVID related articles. The risk of bias across the 4 questions of the Center for Evidence-Based Medicine tool for prognostic studies for (C) COVID-related versus (D) non-COVID related articles; Panel [C]: risk of bias across the 5 domains of the RoB 2 tool for assessment of randomized clinical trials for (A) COVID-related versus (B) non-COVID related articles; [D] The risk of bias across the 16 questions of the AMSTAR 2 tool for assessment of meta-analyses for (A) COVID-related versus (B) non-COVID related articles.

NRIS= non-randomized intervention studies; RCT= randomized clinical trial.