In the emerging evidence base on coronavirus disease 2019 and loss of smell, how many preprint papers are subsequently published?

INTRODUCTION

Traditional medical publishing involves a process of peer review prior to acceptance and publication, with manuscripts often embargoed prior to publication. Preprint servers have emerged since the 1990s, but have played little part in medicine prior to the onset of the coronavirus disease 2019 (COVID‐19) pandemic. In 2017, the UK's Medical Research Council announced it was actively encouraging researchers to share research on preprint servers ahead of publication. The British Medical Journal, in collaboration with Yale University and Cold Spring Harbor Laboratory, launched the first preprint server for medical research, MedRxiv, in June of 2019, noting that “there is some evidence that pre‐prints can accelerate progress in handling outbreaks of infectious disease.” The need for rapid dissemination of COVID‐19–related research has led to a surge in the use of preprint servers. An online report “How COVID‐19 is changing research culture” has highlighted the emergence of the use of preprints in the response to the pandemic, with preprints accounting for one‐quarter of research output by the beginning of May 2020. The value of preprints is under debate, with concern that they are being used to circumvent formal peer review and little understanding of how widely their findings are disseminated.

The first reports of a link between COVID‐19 and anosmia emerged in late March 2020, followed by a rapid growth in associated publications. We set out to evaluate what proportion of papers posted on MedRxiv subsequently appear in peer‐reviewed journals and how frequently they were cited.

METHODS

MedRXiv was searched for the terms COVID and anosmia, restricting the search to the period April 1, 2020, to May 31, 2020. Abstracts were screened and included if the paper reported on the prevalence of olfactory dysfunction relating to COVID‐19, its role in predicting results of COVID‐19 testing or prognosis, or if the study evaluated underlying pathophysiology. Studies were rejected if they did not report on any aspect relating to olfactory dysfunction.

MedRXiv states whether a paper has been published and reports metrics such as tweets. The digital object identifier (DOI) was used to search Google scholar for the number of citations. The title was used to check if the publication had been missed. Abstracts of the preprint and peer‐reviewed version were compared for significant differences, which included a significant change to the results, subject numbers, authorship, or title. A Student t test was used to compare the mean number of citations between published and preprint papers.

RESULTS

Eighty five papers were identified, of which 39 were included after screening abstracts.

Ten papers had been published in peer‐reviewed journals as of October 16, 2020 (Table 1). An additional 4 papers had been subsequently published (shown with *) but with substantial changes to the number of included subjects, analysis, and authorship in 1 paper. The mean ± standard deviation (SD) number of citations was higher for published papers (34.9 ± 34.8; 95% confidence interval [CI], 13.3–56.5), compared with those that had not (9.2 ± 13.2; 95% CI, 4.4–14), p = 0.002.

TABLE 1

Papers published on MedRxiv preprint server between April 1, 2020 and May 31, 2020 addressing loss of sense of smell in association with COVID‐19, ordered by date of posting

				Metrics
First author	DOI	Published	Journal	Tweets	Citations
Menni	https://doi.org/10.1101/2020.04.05.20048421	Yes	Nature Medicine (https://doi.org/10.1038/s41591-020-0916-2)	388	54
Levinson	https://doi.org/10.1101/2020.04.11.20055483	No	–	12	20
Olibris	https://doi.org/10.1101/2020.04.14.20065631	No	–	15	1
Lechien	https://doi.org/10.1101/2020.04.15.20066472	No	–	17	13
Sarker	https://doi.org/10.1101/2020.04.16.20067421	Yes	Journal of the American Medical Informatics Association (https://doi.org/10.1093/jamia/ocaa116)	21	7
Borges do Nascimento	https://doi.org/10.1101/2020.04.16.20068213	No	–	6	2
Fontanet	https://doi.org/10.1101/2020.04.18.20071134	No	(Reported on local website)	1654	56
Aguilar	https://doi.org/10.1101/2020.04.19.20071548	No	–	19	1
Williams	https://doi.org/10.1101/2020.04.22.20072124	No	–	155	23
de Souza	https://doi.org/10.1101/2020.04.25.20077396	a	–	61	28
Hornuss	https://doi.org/10.1101/2020.04.28.20083311	Yes	Clinical Microbiology and Infection (https://doi.org/10.1016/j.cmi.2020.05.017)	13	79
Tordjman	https://doi.org/10.1101/2020.04.28.20081687	No	–	12	0
Borobia	https://doi.org/10.1101/2020.04.29.20080853	Yes	Journal of Clinical Medicine (http://doi.org/10.3390/jcm9061733)	109	37
Yin	https://doi.org/10.1101/2020.04.29.20085415	No	–	5	4
Lechien	https://doi.org/10.1101/2020.05.02.20070581	No	–	5	3
Mandić‐Rajčević	https://doi.org/10.1101/2020.05.03.20082818	No	–	6	3
Lechien	https://doi.org/10.1101/2020.05.03.20088526	a	–	6	1
Coolen	https://doi.org/10.1101/2020.05.04.20090316	No		36	41
Parma	https://doi.org/10.1101/2020.05.04.20090902	Yes	Chemical Senses (https://doi.org/10.1093/chemse/bjaa041)	155	40
Iravani	https://doi.org/10.1101/2020.05.07.20094516	No	–	38	3
Lombardi	https://doi.org/10.1101/2020.05.07.20094276	Yes	Clinical Microbiology and Infection (https://doi.org/10.1016/j.cmi.2020.06.013)	7	2
Asseo	https://doi.org/10.1101/2020.05.07.20093955	No	–	14	0
Ortiz‐Prado	https://doi.org/10.1101/2020.05.08.20095943	No	–	320	6
Rivett	https://doi.org/10.1101/2020.05.09.20082909	Yes	eLife (https://doi.org/10.7554/eLife.58728)	28	102
Almazeedi	https://doi.org/10.1101/2020.05.09.20096495	No	–	18	11
Ozturk	https://doi.org/10.1101/2020.05.10.20097535	No	–	11	2
Regina	https://doi.org/10.1101/2020.05.11.20097741	No	–	70	6
Qiu	https://doi.org/10.1101/2020.05.13.20100198	No	–	7	13
McArthur	https://doi.org/10.1101/2020.05.14.20102475	No	–	6	1
Basse	https://doi.org/10.1101/2020.05.14.20101576	a	–	25	4
Olalla	https://doi.org/10.1101/2020.05.18.20103283	Yes	QJM: An International Journal of Medicine (https://doi.org/10.1093/qjmed/hcaa238)	19	3
Boddington	https://doi.org/10.1101/2020.05.18.20086157	No	–	9	3
Mei	https://doi.org/10.1101/2020.05.19.20107102	No	–	7	0
Fafi‐Kremer	https://doi.org/10.1101/2020.05.19.20101832	Yes	EBioMedicine (https://doi.org/10.1016/j.ebiom.2020.102915)	247	23
Lechien	https://doi.org/10.1101/2020.05.20.20106633	Yes	Laryngoscope (https://doi.org/10.1002/lary.28993)	4	2
Izquierdo	https://doi.org/10.1101/2020.05.22.20109959	No	–	90	2
Rojo	https://doi.org/10.1101/2020.05.24.20111971	a	–	185	7
Garibaldi	https://doi.org/10.1101/2020.05.24.20111864	No	–	8	2
Sandri	https://doi.org/10.1101/2020.05.24.20111245	No	–	44	12

Indicates that a paper has been published related to the preprint but with substantial changes.

COVID‐19 = coronavirus disease 2019.

DISCUSSION

The need for rapid dissemination of knowledge surrounding COVID‐19 has led many researchers to utilize preprint servers. MedRxiv states that “pre‐prints are preliminary reports that have not be certified by peer‐review” and that inclusion on the site is not an endorsement of scientific quality. “Preprint” suggests an intention that “print” will follow through a conventional peer‐review process. We looked at 39 papers and found that 1 in 4 had been published at the time of review, between 4.5 to 6 months after posting. Although some preprints likely remain under review and may still be published, at least some of these papers will remain available indefinitely in the public domain in the absence of any peer‐review process.

The results show that papers published in peer‐reviewed journals are more widely cited than those appearing only on preprint servers, suggesting that peer‐reviewed publication remains the most effective avenue to widely disseminate information. Although less frequent citation of preprints may reflect caution, citations of preprints that were subsequently published were often made before peer‐review publication. Almost all non‐published papers had also been cited, some many times. As shown in Table 1, Fontanet (https://doi.org/10.1101/2020.04.18.20071134) was the most widely cited preprint article, receiving more citations than 80% (8/10) of the peer‐reviewed publications.

Four papers were identified through searching that clearly related to the original preprint but had been changed significantly at the time of publication, in terms of significantly increased numbers of included subjects or the analysis. Preprint servers allow preliminary findings of large studies to be published in “real‐time,” whereas final results may take many months to obtain and publish in a peer‐reviewed format; however, it risks premature dissemination of results that may change significantly in the final analysis and those citing the preprint may not be aware if substantial changes subsequently occur. This study provides an early perspective on what will be the evolving role of preprint servers. Further work over a longer time frame is important to understand how preprints are utilized, both by authors and the scientific community.

Preprint servers have played an important role is sharing early insights into COVID‐19. Indeed, Bagheri et al. posted on MedRxiv on March 23, 2020, was 1 of the first written reports of the coincidence of COVID‐19 cases and olfactory dysfunction, and has subsequently been published. However, preprints cannot provide the same level of scientific rigor as peer‐reviewed publications, yet papers may still be highly cited. This highlights the potential pitfalls of preprint servers in which both poor quality or misleading information could become widely disseminated under the guise of rigorous scientific findings.