COVID-19 article retractions in journals indexed in PubMed.

Dear Editor

The COIVD-19 pandemic has produced an unprecedented volume of research, with over 125,000 articles published or released in the first 10 months of the pandemic. In addition, there has been an unusual number of article retractions and withdrawals, with papers being retracted far earlier than the usual time period, which is typically 2 to 3 years. This raises concerns about the publishing process and research quality and asks whether these retractions reflect problematic research or are a by-product of a rush to publish during a pandemic. Potential causes for article retraction include the popularity of pre-print servers, increased scrutiny of COVID-19 research, and expedited publisher reviews. These retractions have occurred in a range of journals, including top-tier journals, indicating that these retractions cannot be attributed solely to lesser-known, lower quality, or predatory journals. While previous studies have examined COVID-19 publications, some studies included articles from a variety of sources (including pre-print servers) and searched multiple databases. Furthermore, given the unique nature of the pandemic, retraction rates should ideally be compared to previous health crises, especially epidemics and pandemics. Since PubMed is a major journal database, is used as an indicator of journal quality, and is arguably the premier repository of biomedical articles, we exclusively searched PubMed indexed journals for retracted COVID-19 articles and for retracted articles from other recent infectious disease outbreaks.

We searched the PubMed database (https://pubmed.ncbi.nlm.nih.gov/) for articles published between 01 November 2019 and 01 August 2021 using the term “COVID-19.” Filters for “Retract Publication” and “Retraction of Publication” were applied and duplicates were omitted. We found 43 COVID-19 retractions or withdrawals from 35 journals (“retraction” here denotes both outcomes). Six journals had multiple retractions. Journal impact factors were obtained from Clarivate Journal Reports™ (https://clarivate.com/webofsciencegroup/solutions/journal-citation-reports/) for the year 2020. These journals had mean and median impact factors (excluding duplicate journals from the list) of 10.8 ± 22.0 and 3.1, respectively; the impact factors ranged from 0.9 to 91.3 (seven journals had no impact factor listed) (Table 1 ). Mean, median, and range of time to retraction were 139.1 ± 107.9 days, 120 days, and 3 - 433 days, respectively. Publication and retraction dates were based on the earliest and most specific date available from PubMed and publishers. Retracted articles had a total of 3470 citations (mean 80.7 ± 240.1; median 6) by 02 Aug 2021. The H-Indices for these journals were obtained from Scimago (https://www.scimagojr.com/) and had mean, median, and ranges of 136.5 ± 221.0, 81.5 (excluding duplicate journals from the list), and 3–1030, respectively (one journal had no H-Index). Welch's t-test was used to compare differences in retraction time between articles retracted in 2020 and 2021.

Table 1

Characteristics of retracted COVID-19 articles and associated journals.

	Mean	Median	Range	Total
Impact Factor*	10.8 ± 22.0	3.13	0.9-91.3	-
H-Index*	136.5 ± 221.0	81.5	3-1030	-
Time to Retraction, days⁎⁎	139.1 ± 107.9	120	3-433	-
Citations⁎⁎	80.7 ± 240.1	6	0-1210	3470

Journals

Articles

We also compared COVID-19 article retractions to articles retracted during previous outbreaks. Searches for “H1N1”, “Zika”, “SARS”, and “Ebola” were performed using the same PubMed filters and for the first 19 months of each epidemic/pandemic: 01 November 2002 – 01 June 2004 (SARS), 01 April 2009 – 01 November 2010 (H1N1), 01 December 2013 – 01 July 2015 (Ebola), and 01 May 2015 – 01 December 2016 (Zika). Timelines for each outbreak were obtained from the World Health Organization (WHO) and the Centers for Disease Control (CDC). Given the unusually high volume of COVID-19 publications, we also compared outbreak publications “to date” (01 Aug 2021)[except for SARS, which was searched 01 November 2002 to 01 November 2019 to avoid overlap with SARS-CoV-2]. This second period is referred to as “Time frame 2”. Percent retractions from previous outbreaks ranged from 0.00 to 0.04% (Table 2 ). The number of articles published (154,562) and retracted (43) during the COVID-19 pandemic was much higher than those in previous outbreaks for both the first 19 months of each outbreak and to date (i.e., 01 Aug-2021). However, the retraction rate of COVID-19 articles (0.03%) was similar to that observed in the H1N1 pandemic for both time frames (0.04%) and is lower than a previously estimated universal rate of retraction (0.04%). Other studies have also observed disproportionately higher publications for COVID-19 compared to other outbreaks and scientific topics , ; our study is the most recent analysis of the retraction rate. For example, an earlier study also searched PubMed and observed a higher retraction rate (0.097%) by 08 June 2020. This higher rate of retraction may be due to an influx of COVID-19 publications in the following year (as of this writing, COIVD-19 publications for 2021 have exceeded the total for 2020). Even though the majority of retractions occurred in 2021 (25; 58.1%), these data suggest retraction rates may normalize over time given increased total rates of publication. Interestingly, there was a statistically significant difference in retraction time between articles retracted in 2020 (n = 18, mean 87.9 ± 63.5 days) and 2021 (n = 25, mean 176.0 ± 118.9 days) (p = 0.003). This may be due to decreased interest in COVID-19 research quality, lower individual article visibility from increased total publications, or articles with less serious or obvious errors being retracted more slowly. Finally, some articles received a high number of citations despite relatively rapid rates of retraction. This observation supports a previous study on the citation of COVID-19 articles post-retraction, can complicate the use of citations as a measure of article quality, and potentially prolongs the impact of problematic research.

Table 2

Comparison of retraction rate between COVID-19 and other outbreaks.

Disease	Time Frame 1*			Time Frame 2⁎⁎
	Total	Retracted	Percentage	Total	Retracted	Percentage
SARS	1884	0	0.00%	8471	1	0.01%
H1N1	4689	2	0.04%	19693	7	0.04%
Ebola	2467	0	0.00%	8644	2	0.02%
Zika	1968	0	0.00%	9569	1⁎⁎⁎	0.01%
SARS-CoV-2	154562	43	0.03%	-	-	-

First 19 months of outbreak- 01 November 2002 – 01 June 2004 (SARS), 01 April 2009 – 01 November 2010 (H1N1), 01 December 2013 – 01 July 2015 (Ebola), 01 May 2015 – 01 December 2016 (Zika), and 01 November 2019 and 01 August 2021 (SARS-CoV-2)

Beginning of outbreak to present- 01 November 2002 – 01 November 2019 (SARS), 01 April 2009 – 01 August 2021 (H1N1), 01 December 2013 – 01 August 2021 (Ebola), 01 May 2015 – 01 August 2021 (Zika)

Article dually listed as an Ebola retraction

In conclusion, many COVID-19 retractions have occurred in PubMed indexed journals. This demonstrates that journals listed in major databases are still susceptible to retractions. There is also a remarkable influx in COVID-19 publications and retractions compared to previous outbreaks, possibly due to global interest, expedited review processes, and journal interest in COVID-19. However, similar rates of retraction to the H1N1 pandemic and universal retraction rates suggest that research quality may be similar to previous outbreaks. While early studies expressed concern about the unusually large number of retractions, the retraction rate may eventually normalize over time. Future studies should assess retraction rates after a more traditional time frame for retractions has passed.

Financial support

None.

Declaration of Competing Interest

None.