Continued Visibility of COVID-19 Article Removals.

OBJECTIVE: The coronavirus disease 2019 pandemic has produced an unprecedented amount of scientific research as well as a high number of article retractions. Social and news media have been used to disseminate scientific research, and this can include retracted or withdrawn research. This risks the persistence of low-quality research and may contribute to controversial ideas or conspiracy theories.
METHODS: We examined 34 retracted or withdrawn coronavirus disease 2019 articles using alternative metrics.
RESULTS: These articles continued to receive social and news media mentions up to 180 days postremoval, although most mentions occurred within 30 days postremoval. Articles available on preprint servers accounted for 45.5% of total mentions.
CONCLUSIONS: A significant, positive correlation was observed among Scimago Journal Rank, Immediacy Index, and Journal Citation Index, and total article mentions.

Key Points

Retracted and withdrawn articles continue to be mentioned on social and news media postremoval (44.2% of all mentions occurred postremoval).

Articles available on preprint servers accounted for a high number of total and postremoval mentions (45.5% and 74.6%, respectively).

Moderately positive, significant correlations were observed between journal metrics (Scimago Journal Rank Immediacy Index and Journal Citation Indicator) and postremoval mentions.

The coronavirus disease 2019 (COVID-19) pandemic has generated a large number of academic publications, with more than 125,000 articles published within the first 10 months of the pandemic. To accommodate this influx of manuscripts and to rapidly disseminate information, some journals used an expedited review process.[1-3] In addition, preprint servers, or publicly available online repositories that allow manuscripts to become available before peer review, also have had a role in disseminating research[4]; however, the influx in COVID-19 publications has resulted in many retractions and withdrawals, raising concerns about the quality of this research and the publication process.[5-8] In addition, these removals have the potential to circulate long after removal, given the public interest in and access to COVID-19 research. This potentially increases the dissemination of misinformation, which can negatively influence both professional and public understanding of science.

Alternative metrics, or measurements of the impact of an article outside traditional academic and journal citations, have become a useful method of measuring the impact of research. These metrics measure social and news media engagement with science and may indicate the public response to research. These metrics may complement traditional article citation metrics, and some publishers, such as ScienceDirect and Taylor & Francis, list these metrics on their journal Web sites.[9,10] Furthermore, the heightened public interest in the COVID-19 pandemic has resulted in increased engagement with COVID-19 research via social and news media.[11] As such, alternative metrics may help determine public response to removed articles as well. Here, we examine the news and social media attention given to removed COVID-19 articles to gauge this response.

Methods

Retracted or withdrawn COVID-19 articles were identified from a curated list provided by the Web site Retraction Watch (https://retractionwatch.com. The inclusion criteria included removal, defined as article retraction or withdrawal, before June 31, 2021, an identifiable date of removal, and removal for at least 180 days. Alternative metric data were provided by Altmetric, a data analytics organization (https://www.altmetric.com), using the Altmetric Explorer platform, and was extracted for the selected articles on July 31, 2021. Impact factor, Immediacy Index, Eigenfactor, and Journal Citation Indicator were obtained from Clarivate Journal Reports (https://clarivate.com/webofsciencegroup/solutions/journal-citation-reports) for 2020. H-Index and Scimago Journal Rank (2020) were obtained from Scimago (https://www.scimagojr.com/). The Pearson correlation coefficient was used to determine the relation between journal metrics and both postremoval and total mentions, and time to retraction and both postremoval and total mentions. Beall’s List (https://beallslist.net), a curated list of potential predatory journals, was used to identify any such journals included in the search.

Results

We identified 34 articles that met our inclusion criteria. Before removal, the articles were published for an average of 52.9 ± 55.3 days and a median of 30.5 days. The majority of mentions occurred before retraction for Twitter, Facebook, and Reddit, and 30 days postremoval for new media, blog posts, and video (Tables 1–3). A total of 21 articles were published in 19 journals, with an average impact factor of 16.7 ± 22.5, a median of 6.2, and a range of 1.7 to 91.2. No journals were classified as predatory. Journal articles accounted for 77,074 total mentions (54.5%; mean 3503.4 ± 8298.7) and preprint servers for a total of 64,237 mentions (45.5%; mean 5353.1 ± 11,446.9). Total postremoval mentions accounted for 62,390 mentions (44.2% of total mentions), with 46,554 (45.5%) from preprint servers. Twelve articles were solely available on preprint servers before retraction, namely bioRxiv (4) and medRxiv (8). Large variances were observed for each metric, indicating a wide range of engagement across articles. Significant correlations were observed only between Scimago Journal Rank (r = 0.472, P = 0.041), Immediacy Index (r = 0.590, P = 0.017), and Journal Citation Indicator (r = 0.510, P = 0.026), and total article mentions (pre- and postremoval). There was no significant relation between time to removal and both postremoval and total mentions.

Table 1

Total alternative metric mentions for retracted articles

Table 3

Percentage of total alternative metric mentions per time period for retracted articles

Discussion

Retraction and withdrawal can provide an effective means by which the academic community safeguards the quality of published research. Retracted articles may receive significant social and news media attention before and after removal, however. In this study, although articles were removed relatively quickly compared with typical removal timeframes (23.8 months for retracted articles),[12] they nonetheless gained significant attention before retraction. Furthermore, some articles continued to be mentioned postremoval, but most of the mentions occurred with the first 30 days postremoval. This presents several problems. First, even with relatively rapid retraction of problematic articles, these articles may still gain significant attention before removal. This could result in the persistence of problematic ideas despite later retraction. For example, Serghiou et al found that popular retracted articles received more attention regarding the original publication than the retraction, suggesting that those engaged with the article were less aware of or interested in the removal than in the original publication.[13] Second, the availability of problematic research to a wide audience with varying levels of scientific literacy may increase the possibility that research will be misinterpreted and used to promote controversial ideas. A review of COVID-19 conspiracies theories noted how poor research and scientific communication can promote misinformation.[14] In this study, one article proposed that 5G cell communication technology was capable of producing coronavirus in skin cells.[15] Although this article was swiftly retracted 8 days after publication, it generated 3133 Twitter tweets preretraction and 1866 postretraction. Although the intent of many of these mentions is unclear, some mentions noted the retraction status of the article and criticized its publication, whereas others appeared to agree with or promote conspiracy theories regarding 5G and COVID-19. Finally, given that the public response to COVID-19 information (or misinformation) can influence behaviors that affect virus transmission,[16-19] it is important that the public have access to high-quality research to better drive prohealth behaviors.

Certain articles were disproportionately influential and contributed to a majority of social and news media engagement. This is consistent with a study by Jan and Zainab, which found that although articles continue to receive social and news media attention postretraction, a single article out of 10 accounted for the majority of this attention.[20] Furthermore, predicting which articles published in which journals are likely to receive high attention postremoval will be difficult. Although moderately positive correlations were noted for Scimago Journal Rank, Immediacy Index, and Journal Citation Indicator, additional studies with a larger number of articles will be needed to determine reliable trends. Interestingly, the article with the largest total mentions was on a preprint server. Furthermore, preprints had higher mean total mentions compared with journals, and the article with the highest total mentions was a preprint article, suggesting that preprint servers may be the most predictive of social and news media engagement despite lacking traditional journal metrics. Finally, time to removal and the number of article mentions did not have a significant relation, suggesting that visibility may not be determined by the time of retraction.

The nature of pre- and postremoval mentions was not quantified in this study, although some mentions more than 1 year postremoval continued to promote conspiracy theories. Furthermore, the characteristics of users sharing removed articles on social media were not quantified. Future studies should assess the nature of postremoval mentions, how scientific research is so quickly and widely circulated in news and social media, and how news of article removal can be communicated more broadly. Although this study was limited to articles that had been removed for 180 days, the majority of mentions occurred by 30 days postremoval. Because the pandemic is ongoing, however, future studies should be performed after the pandemic has ended to provide a more definitive analysis of COVID-19 removals.

Conclusions

The continued visibility of removed articles poses a problem for scientific integrity because it risks the persistence of flawed research; however, when the review process is expedited, such as during the COVID-19 pandemic, or absent, such as with uploaded preprint articles, problematic research can become widely available. Although methods such as preprint servers can rapidly disseminate research and be useful during an emerging public health crisis, researchers and publishers should be aware that low-quality research may continue to have an impact despite quick removal. Regulation by the scientific community at large, although capable of flagging problematic research, may be insufficient when research is so quickly and widely available to the public. As such, careful peer review of articles before publication or online availability is likely the most effective method at preventing the widespread and persistent dissemination of problematic research.

Table 2

Mean alternative metric mentions for retracted articles