Bibliometric and Altmetric Analysis of Retracted Articles on COVID-19.

BACKGROUND: With greater use of social media platforms for promotions of research articles, retracted articles tend to receive approximately the same attention. We systematically analyzed retracted articles from retractionwatch.com to look at the Altmetric Attention Scores (AAS) garnered over a period of time in order to highlight the role of social media and other platforms in advertising retracted articles and its effect on the spread of misinformation.
METHODS: Retractionwatch.com was searched for coronavirus disease 2019 related retracted papers on November 6th, 2021. Articles were excluded based on lack of digital object identifier (DOI), if they were preprint articles, absent AAS, and incomplete AAS of pre retraction, post retraction, or both scores.
RESULTS: A total of 196 articles were found on the Retraction Watch website of which 189 were retracted papers and 7 were expression of concern (EOC). We then identified 175 articles after excluding those that did not have a DOI and 30 preprint articles were also excluded giving 145 articles. Further exclusion of articles with absent AAS and incomplete AAS resulted in a total of 22 articles.
CONCLUSION: Retracted articles receive significant online attention. Twitter and Mendeley were the most popular medium for publicizing retracted articles, therefore more focus should be given by journals and their Twitter accounts to discredit all their retracted articles. Preprints should be reconsidered as a whole by journals due to the huge risk they carry in disseminating false information.

INTRODUCTION

With the current coronavirus disease 2019 (COVID-19) pandemic, new information is constantly being demanded, received, and overburdened to the point of its reliability being questioned. The rise of the infodemic during COVID-19 has become an increasingly threatening issue as misinformation oftentimes spreads as rapid fire through the press of a button.1 In the era of a technology-versed generation, information can be spread within seconds and made viral within minutes. Consequently, the fragility of using scientific information on an online platform such as social media is extremely risky, especially when such information is dependent on many lives.123 Retracted articles play a huge responsibility in confuting misinformation and curbing the widespread infodemic.4

The COVID-19 race for research publications is still on full speed as researchers are keen to investigate the gray areas in managing this sinister illness.5 The topic is in great demand by journals and time restrictions lead researchers to rush and submit their work, consequently realizing in the post-publication review that inconsistencies were made causing its retraction or correction. However, post-retraction spread of misinformation through social media has been an ongoing concern and the exponential rise in COVID-19 related articles has exacerbated the phenomenon.6 With the tentative and sensitive COVID-19 related information, readers have fallen prey to the infodemic regardless of the attempts made in retraction and correction. Moreover, a lesser known concern is the preprint publication of false research that has increased since the COVID-19 pandemic.7 Preprints are typically unrefined pre-publications of peer-reviewed papers that are placed almost immediately after some superficial screening. This allows information to reach readers quickly while bypassing the long-drawn process required in officially publishing through peer-reviewed journals.

With greater use of social media platforms for post publication and preprint promotions of research articles, retracted articles tend to receive approximately the same attention.2 The generations are quickly picking up on the use of social media thus expanding its exploitation especially during the time of a health crisis.8 This causes the spread of falsified information without warning of its retraction from the journal. Social media plays a huge role in marketing fresh research but retracted articles also receive the same attention. However, retracted articles are usually popular for its misinformation rather than its reason for retraction, building a community of misinformants.9 The issue of retracted articles is greatly overlooked but its consequences are of widespread disinformation as research proves that retracted articles still continue to circulate.10

Hence, we systematically analyzed retracted articles from retractionwatch.com to look at the Altmetric Attention Scores (AAS) garnered over a period of time. AAS is a widely used metric to reflect on societal impact of research output by providing an indicator of the amount of attention that it has received.11 It is a reliable form of calculating the total outreach an article has received including that through social media platforms such as Twitter, Facebook, video uploaders and news.6 We hope to highlight the role of social media in advertising retracted articles and its effect on the spread of COVID-19 linked misinformation.

METHODS

Retractionwatch.com was searched for COVID-19 related papers on November 6th, 2021. All the articles recorded on the website during that day were considered during the search. The exclusion criteria was the lack of digital object identifier (DOI) and if they were preprint articles. Further exclusion was done for absent AAS and incomplete AAS due to lack of pre retraction, post retraction, or both scores. The media platforms that were investigated for attention scores were Altmetric, Mendeley users, Dimensions, Twitter, News outlet, blogs, Facebook, Reddit, and video uploaders.

RESULTS

A total of 196 articles were found on the Retraction Watch website of which 189 were retracted papers and 7 were expression of concern (EOC). We then identified 175 articles after excluding those that did not have a DOI (Fig. 1). A total of 30 preprint articles were also excluded giving 145 articles. There was a further exclusion of articles with absent AAS as these articles were not processed through the Altmetric system at all and resulted in 113 articles. However those without pre retraction, post retraction, or both scores were also excluded giving a total of 23 articles. One article was excluded at the end of this process due to the reason of EOC being revoked by the journal completely.

Fig. 1

Flowchart for search strategy of retracted articles related to coronavirus disease 2019.

DOI = digital object identifier.

*One article was excluded due to the journal completely revoking the initial expression of concern

A table was curated of all the 22 articles to show the attention scores from Altmetric, Mendeley users, Dimensions, Twitter, News outlet, blogs, Facebook, Reddit, and video uploaders (Table 1). The highest pre-retraction and post-retraction attention score are both from Twitter of 14,681 and 2,793 respectively. These scores both belong to the article “The Safety of COVID-19 Vaccinations—We Should Rethink the Policy” that also has the highest AAS (10,294, 2,146) amongst the other retracted articles in Table 1. The highest pre-retraction score for Mendeley is 668 in the article “SARS-CoV-2 Infects T Lymphocytes Through Its Spike Protein-Mediated Membrane Fusion” and post-retraction score is 123 in the article “Chinese Medical Staff Request International Medical Assistance in Fighting Against COVID-19.”

Table 1

Altmetric scores of COVID-19 retracted articles up to November 6th, 2021

Title of the paper		Altmetric score	Mendeley	Dimensions	Twitter	News outlets	Blogs	Facebook	Reddit	Video uploaders	Reason behind retraction
Anal swab as the potentially optimal specimen for SARS-CoV-2 detection to evaluate the hospital discharge of COVID-19 patients	Before retraction	267	54	8	101	21	3	1	0	0	It did not meet the ethical standards of the journal, owing to a lack of informed consent obtained from the patients, prior to publication.
	After retraction	20	1	0	0	2	1	0	0	0
Basic demographic parameters help predict outcomes in patients hospitalized with COVID-19 during the first wave of infection in West Texas	Before retraction	6	53	0	0	0	1	0	0	0	The retraction of the articles follows a request by the authors after their institutional review board found that the previously approved study protocols appear to violate IRB guidelines around prisoner research.
	After retraction	6	1	0	0	0	1	0	0	0
Chinese medical staff request international medical assistance in fighting against COVID-19	Before retraction	2,809	343	20	7,476	38	3	1	5	0	The author of this correspondence says that the account described therein was not a first-hand account, as the authors had claimed, and that they wished to withdraw the piece.
	After retraction	313	123	1	284	13	1	0	2	0
Chloroquine or hydroxychloroquine for COVID-19: Why might they be hazardous?	Before retraction	1,310	179	39	496	113	7	3	2	0	Several concerns were raised with respect to the veracity of the data and analyses conducted.
	After retraction	937	41	17	2,181	1	3	2	0	0
COVID-19 and potential global mortality - revisited	Before retraction	1	49	5	2	0	0	0	0	0	The article was based on very early data and reports from the World Health Organization and Ferguson et al from which the article drew imprecise conclusions.
	After retraction	13	2	0	0	1	1	0	0	0
COVID-19, suicide, and femicide: rapid research using Google search phrases	Before retraction	127	43	2	37	12	1	0	1	0	The author has identified inaccuracies in the methodology.
	After retraction	6	2	0	0	0	1	0	0	0
Effectiveness of surgical and cotton masks in blocking SARS-CoV-2: a controlled comparison in 4 patients	Before retraction	7,094	101	127	10,006	112	18	40	1	7	Authors did not fully recognized the concept of limit of detection (LOD) of the in-house reverse transcriptase polymerase chain reaction used in the study (2.63 log copies/mL), and regret their failure to express the values below LOD as “< LOD (value).”
	After retraction	1,563	8	39	2,711	20	7	3	3	2
Effects of a single dose of ivermectin on viral and clinical outcomes in asymptomatic SARS-CoV-2 infected subjects: a pilot clinical trial in lebanon	Before retraction	808	73	9	1,043	5	3	1	0	3	The authors contacted the editorial office regarding an error between files used for the statistical analysis.
	After retraction	269	0	0	1,004	0	0	0	0	0
Efficacy and safety of acupuncture therapy for asymptomatic infection of COVID-19: a protocol for systematic review and meta-analysis	Before retraction	14	62	4	22	0	0	1	0	0	The article was retracted due to self-duplication.
	After retraction	1	6	1	1	0	0	0	0	0
Family planning in COVID-19 times: access for all	Before retraction	9	21	1	15	0	0	0	0	0	Significant portions of text were used verbatim from several other pieces of published work without proper attribution. (plagiarism)
	After retraction	2	3	0	3	0	0	0	0	0
Human Immunodeficiency Virus (HIV) and outcomes from coronavirus disease 2019 (COVID-19) pneumonia: a meta-analysis and meta-regression	Before retraction	4	29	3	1	0	0	0	0	0	A technical issue caused the accepted version to post online before all plagiarism checks were finalized. Those checks determined that there was too much duplication from previously published sources which prevented the continuance to final publication.
	After retraction	1	3	0	2	0	0	0	0	0
Efficacy and safety of acupuncture therapy for asymptomatic infection of COVID-19: a protocol for systematic review and meta-analysis	Before retraction	14	62	4	22	0	0	1	0	0	The article was retracted due to self-duplication.
	After retraction	1	6	1	1	0	0	0	0	0
Intersectionality and inequalities in medical risk for severe COVID-19 in the Canadian longitudinal study on aging	Before retraction	78	72	6	10	8	1	0	0	0	The article was retracted because the data analysis violated a data use agreement with the Canadian Longitudinal Study (CLSA); the owners of the CLSA had not approved the study.
	After retraction	2	20	1	2	0	0	0	0	0
Mental health burden for the public affected by the COVID-19 outbreak in China: Who will be the high-risk group?	Before retraction	40	517	125	2	4	2	0	0	0	This paper was substantively similar to two previously published papers and therefore constitutes duplicate publication.
	After retraction	9	11	10	1	0	1	0	0	0
Methylene blue photochemical treatment as a reliable SARS-CoV-2 plasma virus inactivation method for blood safety and convalescent plasma therapy for COVID-19	Before retraction	98	48	8	121	1	2	0	0	1	Following publication, concerns were raised regarding Figure in the paper and also authors did not appropriately declare a competing interest regarding their affiliation.
	After retraction	25	15	0	13	1	1	0	0	0
Obesity and mortality of COVID-19. Meta-analysis	Before retraction	198	539	172	158	8	1	3	0	0	The paper was withdrawn by the authors due to inadvertent errors that unfortunately passed unnoticed during the extremely rapid review and publication process at the peak of the COVID-19 pandemic.
	After retraction	2	12	0	4	0	0	0	0	0
Safety and efficacy of favipiravir versus hydroxychloroquine in management of COVID-19: a randomised controlled trial	Before retraction	40	66	13	34	1	1	0	0	0	The authors provided several versions of their dataset. Post-publication peer review confirmed that none of these versions fully recapitulates the results presented in the cohort background comparisons, casting doubt on the reliability of the data.
	After retraction	14	4	1	3	1	1	0	0	0
SARS-CoV-2 infects T lymphocytes through its spike protein-mediated membrane fusion	Before retraction	3,049	668	163	5,062	16	10	9	9	0	After the publication of the article, it came to the authors attention that in order to support the conclusions of the study, the authors should have used primary T cells instead of T cell lines. Additionally, there are concerns that the flow cytometry methodology applied here was flawed. These points resulted in the conclusions being considered invalid.
	After retraction	48	14	52	48	0	2	0	2	0
The safety of COVID-19 vaccinations—we should rethink the policy	Before retraction	10,294	123	5	14,681	27	7	5	34	2	The article was evaluated by the Editor-in-Chief with the support of several Editorial Board Members. They found that the article contained several errors that fundamentally affect the interpretation of the findings.
	After retraction	2,146	9	1	2,793	17	4	1	3	0
Tracking COVID-19 vaccine hesitancy and logistical challenges: a machine learning approach	Before retraction	8	37	2	3	0	1	0	0	0	After the article was published, concerns came to light regarding data use permissions.
	After retraction	8	4	0	1	0	1	0	0	0
Unknown unknowns – COVID-19 and potential global mortality	Before retraction	13	280	54	6	0	1	0	0	0	The article was based on very early data and reports from the World Health Organization and Ferguson et al. from which the article drew imprecise conclusions.
	After retraction	1	2	0	1	0	0	0	0	0
Complete genome sequence of a 2019 Novel Coronavirus (SARS-CoV-2) strain isolated in Nepal	Before retraction	642	560	87	664	11	2	1	1	1	Micobiology Resource Announcements was notified by the Nepal Health Research Council (NHRC) that the authors did not have ethics approval from the NHRC, which the NHRC alleges to be required for such research in Nepal. This Expression of Concern is issued pending the final court decision and will be updated accordingly thereafter.
	After retraction	1	8	0	1	0	0	0	0	0

All the articles under Mendeley had scores greater than or equal to 20 (23 articles) while Twitter had the majority of articles that were scored greater than 100 (11 articles). Dimension, a searching database for research, had the highest pre-retraction score of 172 for the article “Obesity and Mortality of COVID-19. Meta-Analysis” and post-retraction score of 52 for the article “SARS-CoV-2 Infects T Lymphocytes Through Its Spike Protein-Mediated Membrane Fusion.” News outlet’s highest pre-retraction score is 349 for the article, “Unexpected Detection of SARS-CoV-2 Antibodies in the Prepandemic Period in Italy” and post-retraction score is 20 for the article, “Effectiveness of Surgical and Cotton Masks in Blocking SARS-CoV-2: A Controlled Comparison in 4 Patients.” Blogs, Facebook, Reddit, and video uploaders did not have significant scores for comparison.

DISCUSSION

In the span of the first year of the pandemic, more than 100,000 COVID-19 related articles have been published and around 68 of those articles have been retracted as conveyed by Retraction Watch in January, 2021.1213 With more than 6 months gone since January 2021, the number of retracted articles reached a total of 149 which is an increase of more than 50% as shown on Retraction Watch. This shows an alarmingly large increase in retracted articles to which greater attention was possibly brought by the pandemic. COVID-19 attracted enormous attention to the world of research as people were avid to learn about the newly found virus by reading all the latest research articles published.1415 COVID-19 associated articles were found to be of the top 3 most cited articles in 2020 according to the British Medical Journal.16

Social media is a widely used source that is depended on a daily basis for entertainment, news, networking, politics, and even scientific research.1718 It is a heavily based platform to spread thousands and thousands of different subjects, discussions, concepts, and any other matter one can imagine.19 In regards to the COVID-19 retracted articles, social media plays a significant role in spreading the misinformation as healthcare workers, patients, and the mere public are almost completely reliant towards it.20 Although some research was done on the hazards of social media in spreading COVID-19 related misinformation, very little investigation is available on its hazard in spreading COVID-19 related retracted articles.221 Retracted articles are considered more detrimental to the scientific community than misinformation created solely from social media.22 Therefore, an investigation was carried out in our research to evaluate the AAS of retracted articles. In this case, it helped identify the total outreach through these platforms in the post-retraction and pre-retraction period to see the distribution of attention.

Our analysis showed that 39.1% of retracted articles in Table 1 had a post-retraction AAS greater than 20 which is found to be significant according to the Altmetric guidelines.23 Additionally, many of the scores that were greater than 100 stem from Twitter, but the platform with the majority of scores greater than 20 is Mendeley. This proves that citation count is very high on Mendeley which is also reinforced in a similar study done on the citation counts between Mendeley readership and AAS.24 Regardless, Twitter has received twice the amount of attention due to its powerful exposure and spread of information to not just researchers but all groups of people.25

There is no increase in post-retraction scores of individual articles in comparison to pre-retraction scores across all platforms shown in Table 1. As shown in our results, the article that displayed the most attention in both the pre- and post-retraction period was called “The Safety of COVID-19 Vaccinations—We Should Rethink the Policy.” This is a gravely serious topic as countries across the globe are still struggling to vaccinate the entirety of their population.2627 One of the main reasons for this struggle is the “anti-vaxxer” community that believe in the majority of misinformation regarding the side effects and conspiracies behind the use of vaccines.2829 A retrospective analysis done on the power of social media in conveying the side effects of the COVID-19 vaccination supported that most of the misinformants are unreliable sources. Regardless, these sources still influence a number of patients which greatly affect their decision in receiving the vaccine. Therefore, this is a great example of how retracted articles still receive attention through social media and is still a recurring issue that intervenes in herd immunity as been proven by many other research articles. Moreover, video uploaders, Facebook, and blogs did not show significant results to compare with. Video publishers such as Youtube are also known to be a source of misinformation, however, in terms of the spread of retracted articles, it does not pose much of an issue as shown in our results.30

As the generation changes, social media begins to attract those of many age groups especially young adults.31 Studies show that social media can be a reliable source for professional development with students making the majority of this support.32 As the younger generation is influenced by social media, this reliance carries on to future professions thus building a larger impact especially in the healthcare field.3334 A great example of a famous and fraudulent influencer on social media is Joseph Mercola, an osteopathic physician, who has published over 600 articles on Facebook that cast doubt on COVID-19 vaccines since the beginning of the pandemic.35 There has been evidence of social media spreading the misinformation in retracted articles making them so popular that its retraction is deemed insignificant at times.63637 A similar study done on post-retraction AAS shows that most retracted articles and their retraction notice receive media and social media attention.6 In fact, retracted articles tend to receive more media and social media attention than very similar, matched unretracted articles. The cause of this misinformation popularity, without any mention of its retraction, can be partly blamed on journals. Journals can take months to years to retract unreliable research and often fail to even inform the public of its invalid or fraudulent information.38 The author of this discovery, Stylianos Serghiou, explained that the reason for post-retraction popularity is in the key problems that communicate science to the public. He stressed the need to not only correct the scientific record, but as well as the public record, otherwise called social media, following the retraction of an article. Highly popularised articles should especially be given top priority in conveying its fraudulence through social media platforms.39

Preprints are another issue as a whole because the rapid review conducted for their public accessibility is superficial and carry a higher risk of an unsuccessful evaluation.40 During the COVID-19 pandemic, preprints were flourishing due to a rush of novel information that had to be scrutinised in a timely manner. On the other hand, the cost of rushing the publication of such fresh information without other forms of supportive evidence carried many dangers. Since all the initial research done on COVID-19 was the only information available on that topic, readers did not have other mediums to back up the evidence and relied on the first publication they discovered. It led to a number of readers relying on false information and using that to inform the public through social media or citations in their own articles.41 A study conducted on the evolution and impact of preprints during the pandemic showed that majority of the COVID-19 articles are actually preprints which puts an even larger risk of publicising incorrect data and information.41 During our own analysis, we excluded preprints as they are not marketed on social media in the same way as peer-reviewed publications. However, the exclusion process allowed us to find that approximately 17% of retracted articles with DOIs were mainly preprints (Fig. 1).

Moreover, post publication peer-review should be more stringent, and greater attention be paid to concerns raised on social media platforms of scholarly journals by Twitter/Facebook followers.42 Our analysis on Retraction Watch proves that Twitter was the most common used platform in spreading retracted articles with scores spanning from 4 to 2,793 (Table 1). Another study done on the use of social media for academia proved that Twitter is in fact a highly used platform by researchers.43 Many famous twitter accounts have also used their popularity to debunk largely circulated myths such as that about the speed of COVID-19 vaccine production.44 Thus it is possible to circulate and even rectify scientific information in the most accurate manner possible especially giving focus to retracted articles. Modern health journalism is a two-way process, and it is about time to abide by the pace of the new digital era.45 Researches have recommended to use meticulous guidelines, ethics, and professionalism to spread scientific information on social media which is generally a contradiction to the main purpose of social media which is freedom of speech.46 However, during a global crisis, priorities need to be considered and strict limitations should be applied for scientific information.

This particular crisis of the pandemic should set an example for future calamities hence a major implementation should be done to prevent the spread of scientific falsification. The proof of the exponential rise in published articles and consequently its retraction during this crisis is an eye opener to alter the marketing of articles through social media. Journals should specifically work towards broadening their social media use by debunking false information and explicitly publicizing retracted articles as fraudulent. Journal editors play a major role in disseminating information from misinformation so a stronger editorial team is required during this time of unprecedented high submission rates.1

The limitations of our study are that real-time data was not available to access trends in AAS of social media. The variation on social media in a day to day basis after retraction is important to elaborate on the fluctuation and attention received at particular time periods or events. As social media is rapidly changing, the results are also just as variable.

In conclusion, retracted articles do have a tendency to receive a decent amount of attention than one would regard as appropriate especially for information that is false enough to be revoked from the public. The medium of attention most highly used was from Twitter and Mendeley, but a literature review found that debunking fabrication through Twitter is just as possible as spreading it. Therefore, more focus should be given by journals and their Twitter accounts to discredit all their retracted articles. Additionally, a more thorough assessment of articles should be given during the peer-review process regardless of the time restrictions because prevention is better than cure. Preprint articles are also a dubious concept when it comes to publishing accurate and precise research so the whole concept should be reconsidered for novel information.