Academic Journal Retractions and the COVID-19 Pandemic.

The 2020 COVID-19 pandemic has produced an unprecedented amount of scientific research, with over 100,000 articles on the SARS-COV2 virus or the associated pandemic published within the first year. To effectively disseminate such a large volume of research, some academic journal publishers altered their review criteria, and many articles were made available before undergoing a traditional review process. However, with this rapid influx of information, multiple COVID-19 articles have been retracted or withdrawn. Some researchers have expressed concern that these retractions call into question the validity of an expedited review process and the overall quality of the larger body of COVID-19 research. We examined 68 removed articles and determined that many of the articles were removed for unknown reasons (n = 22) or as duplications (n = 12); 24 papers were retracted for more significant reasons (data integrity, plagiarism, reporting or analysis, and IRB or privacy issues). The majority of removed papers were from the USA (n = 23) and China (n = 19).

Introduction

The COVID-19 pandemic, caused by the SARS-CoV-2 virus, has generated an extraordinary global response, including scientific publications on the pandemic and the associated virus. Within a year of the virus’s discovery, more than 100,000 academic articles regarding the virus or the associated pandemic have been published, with some estimates as high as 200,000.[1] To publish such a large volume of research, some academic journal publishers altered or expedited their review process.[2-4] It was argued that expediting the review process would allow much needed information about the pandemic to be communicated quickly and allows publishers to process a large volume of submissions.[4] Indeed, the time from submission to publication has been significantly decreased during the pandemic.[5] However, the number of COVID-19 retractions has called into question the utility of an expedited (or absent) review process, as well as the quality of the larger body of COVID-19 research.[6-8]

Curated online repositories, such as BioRX and MedRX, have also had a role in rapidly disseminating COVID-19 information by archiving and distributing so-called “pre-prints,” unpublished research articles before and without a traditional review process by a publisher. This gives the scientific community access to research before a sometimes lengthy review and publication process, which can be valuable during a global health emergency. However, several COVID-19 papers have been retracted or withdrawn by the server’s editorial staff or by the authors themselves.[9] And while the pre-print status should serve as a caution to readers about the potentially uncertain quality of the research, these pre-prints are nonetheless available to the scientific community and general public and may become an unintended means of disseminating scientific misinformation. For example, a now highly criticized study alleging similarities between HIV and SARS-COV-2 was first made available as a pre-print on BioRX.[10] The article was soon withdrawn by the authors and subsequently rebutted in several published articles in academic journals and news outlets.[11,12] Nevertheless, in early 2021, this paper was one of the most highly shared and cited articles on social and news media as measured by Altimetric, a data analytics service that tracks the online traffic of academic papers on social media and other websites.[13,14] Thus, despite not being a formal publishing house, pre-print servers have become an important and potentially problematic method of sharing scientific information, especially during a global pandemic in which information is needed rapidly.

Given the rapid influx of COVID-19 publications, changes in the review process in certain journals, the availability of pre-print articles, and the number of COVID-19 retractions, we have reviewed retracted or withdrawn papers from the medical literature to understand these events better. Determining the nature of these retracted papers and the reasons for their retractions can help establish the effectiveness of expedited review processes and the quality of COVID-19 research in general. In addition, since there are several reasons for removal, from fraudulent to duplicate publications, it is important to differentiate between articles committing egregious errors and those with minor ones.

Methods

Retracted, withdrawn, or temporarily removed articles were identified through a PubMed database search with the relevant terms “covid-19” and “retracted”/“withdrawn” and a curated list of such articles from the website Retraction Watch (https://retractionwatch.com/). Articles were designated as either “retracted”, “withdrawn”, or “temporarily removed” based on the status given by the publisher or pre-print server. Inclusion criteria included: (1) research involving COVID-19 (most often identified in the article title or abstract), (2) current/previous retracted/withdrawn/temporary removal status by publisher/pre-print server/authors, and (3) retracted/withdrawn status on or before January 1, 2021. To appropriately distinguish between retractions, withdrawals, and temporary removals, “removals” was used to describe papers removed from circulation for any reason. Bibliographical and methodological information for the article (authors, associated institution, publishing journal) was obtained directly from the article or from the publisher’s or server’s website. Articles published in an academic journal are distinguished from those made available on a pre-print server as “published” or “pre-print”, respectively. All listed institutions for all publishing authors were noted. Associated countries were determined by the country of the author’s designated institution. Information regarding the reasons for the retraction was obtained from the publisher or pre-print website (when possible) but also from published correspondence, rebuttals, and news articles detailing the retraction, as necessary. Only reasons citing statements from the authors or publisher were included. The impact factor of the publishing journal was obtained from InCites Journal Citation Reports by Clarivate™ Analytics for 2019. Journal quartile rankings were determined from the Scimago Journal and Country Rankings. Pre-print status was defined as an article appearing in a designated pre-print server before publication in an academic journal.

Study type was classified according to the following criteria and previously published definitions.[15,16]

Public Health Modeling: Research involving modeling to determine population or public health outcomes.

Biological Modeling/Simulation: Research involving any computer-simulated or generated modeling regarding molecular biology or biochemistry (does not apply to statistical analysis using software).

Clinical Trials: Experimental evaluation of a medical therapeutic or device with human volunteers.

Cross-Sectional: Group of human research subjects examined at a particular point in time, including formal surveys.

Cohort: Observational study involving human subjects followed over time (prospective) or historical exposures, behaviors, etc. (retrospective).

Case-Control: Observation of two groups of patients regarding an exposure or disease outcome with a designated control group using either retrospective or prospective design

Case Report: Examination of a medical case involving a single patient.

Case Series: Examination of multiple medical cases.

Expert Opinion: Articles presenting no original research and providing commentary/opinion of previously published research, current events in science, medicine, or public health, including letters-to-editor, editorial, opinion, viewpoint, commentary articles, and guidelines for medical or scientific practice.

Review: Review, summary, or statistical analysis of previously published studies on a particular topic, including meta-analysis and systemic and literature reviews.

Theoretical: A topic outlining a new theory or providing a mathematical or logical proof defending or rebutting a theory.

Reasons for retraction were classified according to the following categories:

Plagiarism: Duplication of a part or whole of published research, including inappropriate use of data sets gathered and published by other authors.

Duplicated: Duplicate or highly similar article published in another journal by the same authors.

IRB/Privacy: Breaches or lack of IRB protocol, as well as breaches of confidentiality and data privacy.

Methodology: Errors in study design. Includes instances where authors withdrew publication to revise methods.

Data Integrity: Fabricated, inaccurate, or unverified data sets or patients, including survey or public health data that do not reflect current conditions.

Reporting/Analysis: Inappropriately performed analysis or inappropriately drawn conclusions, including papers that have insufficient data or reasoning to support conclusions.

Author Consent: Decisions made without the approval of all authors, including changes to study design or submission of the manuscript.

Other: Reasoning that does not qualify for the above-mentioned categories.

Unknown: Reasons not provided by authors or publishers.

The full text of the article was examined when possible, either from the publishing website or from the hosting pre-print server. However, since many of these articles have been removed from circulation, it was often necessary to obtain a copy of the article through other independent repositories, such as ResearchGate. In the absence of a full text, relevant information regarding study type was determined from the abstract or designated “N/A”.

Results

Sixty-eight papers were identified as having been retracted, withdrawn, or temporarily withdrawn by January 1, 2021. Thirty-seven articles were explicitly based on either public health or clinical data. Of these, cohort studies (n = 13, 35.1%) were the most frequent, followed by case-control studies (n = 5, 13.5%), cross-sectional studies (n = 5, 13.5%), case series (n = 4, 10.8%), case reports (n = 3, 8.1%), and clinical trials (n = 2, 5.4%). Of the 20 papers not explicitly based on public health or clinical data, expert opinion (n = 10, 50%) was the most frequent, followed by biological modeling/simulation studies (n = 6, 30%), reviews (n = 2, 10%), and theoretical analyses (n = 2, 10%) (Table 1). Eleven papers could not be categorized due to unavailability.

Table 1.

Summary of Study Types.

Study type	Number of publications
Cohort	13
Unknown	11
Expert opinion	10
Biological modeling/simulation	6
Case-control	5
Public health modeling	5
Cross-sectional	5
Case series	4
Case report	3
Clinical trial	2
Review	2
Theoretical	2

Twenty-three papers did not have a reason for retraction listed by the authors or publisher. The most frequent reported reason was duplication. Reasons for retraction categorized as “Other” could not be reasonably grouped into a more specific category (Table 2). These reasons included “substantial manipulation of the peer review process,”[17] “language issues,”[18] “serious scientific fraud,”[19] and “to prevent any possible dispute on some expressions.”[20] A contingency table of retractions and study type is presented in Table 3.

Table 2.

Summary of Reasons for Retraction.

Reason for retraction	Number of publications
Unknown	22
Duplication	12
Reporting/analysis	9
Data integrity	7
Other	6
IRB/privacy	5
Plagiarism	3
Methodology	2
Author consent	2

Table 3.

Contingency Table of Reason for Retraction based on Study Type.

Study type	Reason for retraction									Total
	Unknown	Duplication	Reporting/analysis	Data integrity	Other	IRB/privacy	Plagiarism	Author consent	Methodology
Cohort	7	1		2	1	2				13
Unknown	5		2		3				1	11
Expert opinion	3	5		1	1					10
Biological simulation/modeling	2		1				1	1	1	6
Case-control	1		2	2						5
Public health modeling	1		1	1		1		1		5
Cross-sectional	2	2	1							5
Case series		2				2				4
Case report		2	1							3
Clinical trial				1	1					2
Review							2			2
Theoretical	1		1							2
Total	22	12	9	7	6	5	3	2	1	68

Thirty-eight papers were published in journals with an impact factor listed by Clarivate™ with a mean impact factor of 10.5 ± 19.1 (median: 3.3). Twenty-two articles were published in 20 first quartile journals (list available from corresponding author). Nineteen papers were published in pre-print servers (9 in medRxiv, 6 in SSRN, 3 in bioRxiv) and 11 in journals without an impact factor indexed by Calvairate. Three journals published more than one retracted article; these included The Lancet (n = 3), Journal of the American Pharmacists Association (n = 2), and Asian Journal of Psychiatry (n = 2).

In several retractions, the exact date of publication and removal could not be determined. For those with both available dates, the average time from publication (or availability on a server) to removal was 42.7 ± 52.2 days. The most frequent geographic origin for study authors was the United States with 23 papers, followed by China (n = 19), India (n = 5), Italy (n = 4), and France and Greece (n = 3). Several countries had 2 publications (Iran, South Korea, and the UK) and one publication (Australia, Belgium, Jordan, Nigeria, Philippines, Russia, Singapore, South Africa, and Turkey) (Table 4). Seven articles had authors from multiple countries. Articles had an average of 6.7 ± 5.5 authors.

Table 4.

Retracted Publications by Country.

Country	Number of publications
USA	23
China	19
India	5
Italy	4
France	3
Greece	3
Iran	2
South Korea	2
UK	2

The following countries had one publication: Australia, Belgium, Jordan, Nigeria, Philippines, Russia, Singapore, South Africa, Turkey.

Discussion

With a rapid influx of COVID-19 publications and removals, the characteristics of the removed papers can provide important insights into the nature of the removals and the review process. For example, the impact factor of the publishing journals can indicate the quality of the journal and, by implication, the papers it publishes. It might be assumed that removed papers are more likely to be published in lower quartile or unranked journals. However, we found 22 articles published in 20 first quartile journals, including prominent journals such as The Lancet and The New England Journal of Medicine. While these journals are in the minority regarding COVID-19 removals, their influence potentially makes removals from these journals a greater concern for the scientific community.

The country of origin was of interest given the tremendous global response to the COVID-19 pandemic. The countries with the most retractions were the USA (n = 23), followed by China (n = 19), which coincides with studies showing the USA and China as the top producers of COVID-19 research.[16,21] This suggests that the number of retractions may be correlated to the volume of research produced by a country, although scientific standards and journals within countries may also affect this.

The type of research and articles published during the pandemic has also been interesting. A surprising number of expert opinion articles were removed, given that these articles were not presenting original research. In most cases, this was either due to duplication (n = 5) or for unknown reasons (n = 4). The frequency of secondary COVID-19 publications may also contribute to the number of secondary article retractions.[16,21,22]

Reasons for removal varied drastically across studies. Most studies (other than those whose reasons were unspecified) were removed for being duplications. In one instance, this was apparently due to authors violating the publisher’s protocol regarding submission to multiple journals,[23] but most duplications were described as “accidental.”[24,25] Expedited review and publishing processes may have led to duplicate articles being published. More serious removals (ie, those involving errors that significantly affected data or outcomes) involved issues of reporting/analysis (n = 9), data integrity (n = 7), plagiarism (n = 3), and IRB or privacy issues (n = 5). These retractions involved issues such as falsely reported first-hand accounts,[26] failure to obtain IRB approval,[27] and inability to complete an independent data review.[28] In one instance, the authors claimed that IRB approval was unavailable due to university closure, although the study did involve human subjects.[29]

Many reported reasons for removal were vague and provided little information regarding the nature of retraction. For example, 1 article was reportedly retracted for “language issues”[18] (it was subsequently republished).[30] Another paper was reportedly removed for “serious scientific fraud” by the publisher but was later revealed to be an attempt by the authors to expose what they considered a predatory journal.[19] One paper was withdrawn by the authors because their paper referenced another retracted study.[31] There were also 6 republications, suggesting that the error for the removal was not so significant as to invalidate the entire study or compromise the integrity of the author’s research. It is also possible that some removals could have been avoided if a more rigorous review process was used and/or a global urgency for COVID-19 information was not present, particularly those with errors involving duplications or study design. Therefore, removed COVID-19 papers should not be treated equally; the severity of the error can range widely and, in some cases, may be due to “flaws” and not necessarily “fraud”.[26]

In many cases, it was not possible to determine the reason for a paper’s removal, and thus the reasons for and implications of these removals remain unknown. In these instances, publishers often released a statement noting that the article had been removed and indicating where the journal’s policy on removals could be found, without any mention of the reasons for the removals. Authors and publishers have a right to privacy on these decisions and may understandably want to avoid acknowledging specific errors. However, the current environment involving a massive influx of COVID-19 publications, expedited editorial review processes, a noticeable volume of retracted COVID-19 articles, and expressions of concern from the scientific community concerning the quality of the publishing process and the research itself, transparency regarding retractions is needed. Understanding the reasons for the removal allows the scientific community to assess the effectiveness of expedited publishing processes more accurately. It also prevents authors whose work was removed for minor reasons from being equated with those removed for more egregious ones. And while the number and severity of removals do not determine the quality of other, non-retracted studies, it has been shown that these retractions can nonetheless alter the perception of this larger body of COVID-19 research. This may prove especially important during a pandemic in which scientific and medical professionals and the public have an interest in rapid and reliable research.

The major limitation of this review was the availability of reasons for removal provided by the publisher or study authors. Furthermore, some articles and abstracts were no longer available after removal. While this limits information for this study, removing problematic articles from circulation helps prevent the spread of disinformation (although some articles remain available despite removal status).

In summary, we suggest that more transparency regarding the removal process and its rationale will allow the scientific community and public to better assess the integrity of COVID-19 research and expedited reviews. Since some articles may be removed for less severe or minor reasons, they need to be differentiated from those involving more serious errors.