COVID-19 randomized controlled trials in medRxiv and PubMed.

Dear editor,

Fast-track preprints have garnered increased attention since the COVID-19 pandemic. However, little is known about the quality of clinical research published as preprints. The objective of this study is to explore the differences in COVID-19 randomized controlled trial (RCT) articles between medRxiv and PubMed.

This is a meta-epidemiological investigation. We published the protocol prior to conducting this study [1]. S1 Appendix is the reporting checklist [2]. We included RCT articles covering topics related to the COVID-19 practice indexed in PubMed or MedRxiv from 1st January 2020 to 15th June. We searched abstracts that used the word “random” from medRxiv COVID-19 SARS-CoV-2 preprints [3]. We searched abstracts from PubMed using Shokraneh's filter for COVID-19 [4] and Cochrane filter for identifying randomized trials [5]. Two of three review authors (YK, SO, TA) selected abstracts from search results independently. Disagreements were resolved through discussion. One of three review authors (YK, SO, TA) selected full text articles and extracted data after calibration exercises. Disagreements were resolved through discussion. We evaluated the characteristics of included articles and the risk of bias using risk of bias tool 2 for the registered first primary outcome, or the outcome presented first in reports without the registration information [6]. We judged the presence of SPIN when there were positive expressions in the title or abstract conclusion in studies whose pre-registered primary outcome was non-significant [7]. We estimated the risk differences and confidence intervals for binary variables. We used Wilcoxon rank sum test for continuous variables. A p-value < .05 was considered statistically significant.

Fig. 1 shows the study selection procedure. We included a total of 29 RCT articles (13 from medRxiv, and 16 from PubMed). Table 1 shows the summary characteristics of the included articles. The citations of included articles and details of evaluations are shown in S2 Appendix. Three articles from PubMed did not show the trial registration numbers. Overall, more than 70% (21/29) articles were at high or some concerns for risk of bias, especially in the domain of selection of the reported results. There were 11 articles whose pre-registered primary outcomes were negative. Among them, there were four articles with SPIN (80%) indexed in medRxiv and another (17%) indexed in PubMed. For example, one article in medRxiv stated “… effects in COVID-19 may be clinically important and warrant further consideration and studies” based on the result of the primary outcome that was not statistically significant (odds ratio 0.59, 95% CI 0.148 to 2.352 P = .454) [8]. There was no statistically significant difference between the number of mentions on social networking sites (median [interquartile range (IQR)] in medRvix 80 [5-245] vs. PubMed 342.5 [78.5-5197], p = .09) or citations (median [IQR] in medRvix 0 [0-39] vs. PubMed 10.5 [2.5-47], p = .17) between the two sources. Two pairs of articles were published both in medRxiv and peer-reviewed journal. One of them discussed limitations in the abstract in PubMed but not in medRxiv, apparently after peer-review.

Fig. 1

Study flowchart.

Table 1

Characteristics of included articles.

	medRxiv	PubMed	Total	Risk differences (%) and confidence intervalsg
	N = 13	N = 16	N = 29
Number of participants of full analysis set	81 (42–92)	115 (54.5–260.5)	86 (50–199)	.24h
Without the information of trial registration	0 (0%)	3 (19%)	3 (10%)	19 [−0.37 to 38]
Inconsistent with trial registrationa	8 (62%)	4 (30%)	12 (46%)	−31 [−67 to 5.7]
Referring limitations in abstracts
Present	1 (8%)	4 (25%)	5 (17%)	19 [−7.7 to 46]
Without an abstract	0 (0%)	1 (6%)	1 (3%)
High or some concerns of risk of biasb
Randomization process	3 (23%)	5 (31%)	8 (28%)	8.2 [−24 to 40]
Deviations from intended interventions	3 (23%)	4 (25%)	7 (24%)	1.9 [−29 to 33]
Missing outcome data	2 (15%)	0 (0%)	2 (7%)	−15 [−35 to 4.2]
Measurement of the outcome	6 (46%)	6 (38%)	13 (45%)	−8.7 [−45 to 27]
Selection of the reported result	9 (69%)	8 (50%)	17 (60%)	−19 [−54 to 16]
Overall Bias	10 (77%)	11 (70%)	21 (72%)	−8.2 [−40 to 24]
SPINc in titles (n = 11)d
Present	1 (20%)	0 (0%)	1 (3%)	−20 [−55 to 15]
SPINc in conclusions (n = 11) d
present	4 (80%)	1 (17%)	5 (45%)	−63 [−100 to −17]
Number of SNS sharee				.09h
0	3 (23%)	2 (13%)	5 (17%)
10> ≥1	1 (8%)	1 (6%)	2 (7%)
100> ≥10	3 (23%)	1 (6%)	2 (14%)
1000> ≥100	5 (38%)	5 (31%)	10 (34%)
10,000> ≥1000	1 (8%)	4 (25%)	5 (17%)
≥10,000	0 (0%)	3 (19%)	3 (10%)
Number of citationsf				.17h
0	7 (54%)	3 (19%)	10 (34%)
10> ≥1	1 (8%)	5 (31%)	6 (21%)
100> ≥10	4 (31%)	5 (31%)	9 (31%)
≥100	1 (8%)	3 (19%)	4 (14%)

Discussion

Our results suggest the existence of RCT articles that had problems with research methods and reporting in both medRxiv and PubMed journals related to the COVID-19 practice. In particular, the SPIN in the abstract conclusion was more frequently seen in reports in medRxiv.

This study has several imitations, including small sample size, and not including preprint servers other than medRxiv.

Further investigation on the impact of accelerated publication without peer review on the quality of reporting of clinical studies is warranted. Readers should pay attention to the overstatements in preprints of RCT. mmc1.docx mmc2.xlsx

Compliance with ethical stantards

Because all data were retrieved from public databases, institutional review board approval is not required.

Author contributions

Yuki Kataoka had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Yuki Kataoka.

Critical revision of the manuscript for important intellectual content: All authors.

Supervision: Toshi A. Furukawa.

Funding

Self funding.

Declaration of Competing Interest

Toshi A. Furukawa reports personal fees from Meiji, Mitsubishi-Tanabe, MSD and Pfizer and a grant from Mitsubishi-Tanabe, outside the submitted work; TAF has a patent 2018-177688 pending. Other authors declare that they have no conflict of interest.