Dissemination of research during the first year of the coronavirus disease 2019 pandemic.

The rapid and effective dissemination of research during the coronavirus disease 2019 (COVID-19) pandemic is critical if healthcare providers and public health officials are to remain aware of new developments. Several organizations have collected relevant COVID-19 articles to facilitate data sharing, including the World Health Organization1 and the National Institutes of Health (NIH).2 Yet, in this time of remarkable research productivity and social media influence, how peer-reviewed research disseminates to the global community remains poorly understood.

In order to ascertain how published research disseminated during the COVID-19 pandemic, we examined the world’s peer-reviewed literature on COVID-19 through the application of alternative metrics (altmetrics) based on social media engagement and evaluative bibliometrics using citation rates. We utilized almetrics and citation rates to analyze COVID-19 articles indexed in the NIH’s iSearch COVID-19 portfolio2 and the Almetric Explorer3 (study flow chart, figure 1). The iSearch COVID-19 Portfolio contains peer-reviewed COVID-19 articles from PubMed and preprints from several sources. We restricted our analysis to articles and used this dataset to ascertain citation rates. These data were merged by articles’ unique PubMed identification numbers with altmetrics from the Altmetric Explorer, where we ascertained articles’ Almetric Attention Scores (AAS; composite score of social interest that includes mentions in newsfeeds, Twitter, Facebook, and Google, among other sources).4

Figure 1

Study flow chart. WoS, ISI Web of Science.

We analyzed COVID-19 articles in the ISI Web of Science (WoS) Core Collection5 to evaluate article characteristics, including study type, authorship, and funding sources. We queried the ISI WoS Core Collection using the search terms “coronavirus disease 2019”, “COVID-19”, “severe acute respiratory syndrome coronavirus 2”, “SARS-CoV-2”, and “novel coronavirus”. The ISI WoS query was limited to publications from January 2020 to February 2021 (with no language or article type restrictions). Although there are several search engines that can be used for bibliometric analysis, including Scopus, Medline, and Google Scholar,6 7 we selected the ISI WoS Core Collection because this database was classically used by Eugene Garfield, the developer of the impact factor metric, to identify “citation classics”.8

Research data were downloaded from iSearch and merged with Almetric Explorer on February 26, 2021. The results of the ISI WoS query were also downloaded on February 26, 2021. This analysis was based on a total of 87,643 articles in iSearch that were merged with data from the Altmetric Explorer (75,960 (86.7%) published in 2020 and 11,682 (13.3%) published in 2021) and 90,609 articles in the ISI WoS query (82,008 (90.5%) published in 2020 and 8601 (9.5%) in 2021).

COVID-19 articles were published rapidly during the first months of the pandemic, peaking in April 2020, and then plateauing at persistently high rates (figure 2A). Trends in citation rates and AAS mirrored each other during the study period (figure 2B). There were 48 articles in iSearch with unique PubMed identification numbers that were retracted or were retraction notices, corresponding to 34 (0.04%) articles (geometric mean citation rate (95% CI) 7.7 (3.5 to 16.5) and arithmetic mean (SD) 30.3 (80.3) and geometric mean AAS (95% CI) 26.9 (8.5 to 84.9) and arithemetic mean AAS (SD) 1243.3 (3784.0)). Some of these articles received substantial social media attention. For example, among the top 10 articles with highest AAS, one of the articles was retracted and another article was the official retraction notice from the journal that published that article. The top 10 articles with the highest citation rates and highest AAS, after excluding retracted articles, are described in table 1.

Figure 2

Monthly distribution of citation rates and altmetrics for articles indexed in the National Institutes of Health’s iSearch Coronavirus Disease 2019 Portfolio, January 2020 to February 2021. (A) Distribution of citation rates by month of publication. (B) Distribution of monthly article publication, mean citation rates, and mean Altmetric Attention Scores (AAS).

Table 1

Top 10 articles with the highest citation rates and highest Almetric Attention Scores for articles indexed in the National Institutes of Health’s iSearch Coronavirus Disease 2019 Portfolio, January 2020 to February 2021

Publication date	PMID	First author	Article title	Journal	Total citations	AAS
Top 10 articles with highest citation rates
2020-01-28	31 986 264	Huang, Chaolin	Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China	Lancet	11,403	14,215
2020-02-29	32 109 013	Guan, Wei-Jie	Clinical characteristics of coronavirus disease 2019 in China	NEJM	7893	10,360
2020-03-15	32 171 076	Zhou, Fei	Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study	Lancet	6942	13,466
2020-02-08	32 031 570	Wang, Dawei	Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China	JAMA	6586	6413
2020-01-25	31 978 945	Zhu, Na	A novel coronavirus from patients with pneumonia in China, 2019	NEJM	6271	5648
2020-02-03	32 007 143	Chen, Nanshan	Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study	Lancet	5534	4422
2020-02-06	32 015 507	Zhou, Peng	A pneumonia outbreak associated with a new coronavirus of probable bat origin	Nature	4765	6132
2020-02-25	32 091 533	Wu, Zunyou	Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention	JAMA	4582	11,466
2020-01-30	31 995 857	Li, Qun	Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia	NEJM	4027	7042
2020-03-07	32 142 651	Hoffmann, Markus	SARS-CoV-2 cell entry depends on ACE2 and TMPRSS2 and is blocked by a clinically proven protease inhibitor	Cell	3998	4383
Top 10 articles with highest AAS
2020-04-15	32 284 615	Andersen, Kristian	The proximal origin of SARS-CoV-2	Nature Med	1021	35,223
2020-11-19	33 205 991	Bundgaard, Henning	Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial	Ann Intern Med	12	28,937
2021-02-06	33 545 094	Logunov, Denis	Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase three trial in Russia	Lancet	0	27,639
2020-03-18	32 182 409	van Doremalen, Neeltje	Aerosol and surface stability of SARS-CoV-2 as compared with SARS-CoV-1	NEJM	2273	26 117
2020-06-05	32 497 510	Chu, Derek	Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis	Lancet	481	23,966
2020-06-13	32 527 856	Zhang, Renyi	Identifying airborne transmission as the dominant route for the spread of COVID-19	PNAS USA	135	21,657
2020-11-22	33 219 229	Cao, Shiyi	Post-lockdown SARS-CoV-2 nucleic acid screening in nearly ten million residents of Wuhan, China	Nature Comm	3	20,849
2020-05-07	32 371 934	Leung, Nancy	Respiratory virus shedding in exhaled breath and efficacy of face masks.	Nature Med	394	20,847
2020-04-1	3 229 127	Kissler, Stephen	Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period.	Science	604	19,609

Retracted articles were not included in the top 10 lists.

AAS, Altmetric Attention Score; Ann Intern Med, Annals of Internal Medicine; JAMA, Journal of the American Medical Association; Nature Comm, Nature Communications; Nature Med, Nature Medicine; NEJM, New England Journal of Medicine; PMID, PubMed unique identifier; PNAS USA, Proceedings of the National Academy of Sciences of the United States of America.

The ISI WoS query revealed the most common document types were “articles” (47,717; 52.7%), “editorials” (14,491; 16.0%), and “letters” (14,073; 15.5%). The most frequent WoS categories were “Medicine, General & Internal” (11,111; 12.2%), “Public, Environmental & Occupational Health” (7281; 8.0%), and “Infectious Disease” (4790; 5.2%). Over half of all articles originated from four countries: the US (25,312; 27.9%), China (10,535; 11.6%), Italy (8899; 9.8%), and England (8759; 9.7%). The top funding agencies were the NIH, the National Natural Science Foundation of China, and the European Commission.

In this study of the world’s peer-reviewed COVID-19 literature in the first year of the pandemic, we observed a dramatic explosion of research output. With rapid publication of approximately 90,000 peer-reviewed articles that addressed all facets of COVID-19, the global community has been inundated with data. Notably, nearly 60% of the world’s research output originated from four countries that were hit first and hardest by COVID-19. Three countries are geographically distinct from the origination site of the virus, highlighting the global impact of the disease.

We observed that mean citation rates and AAS mirrored each other. Although there is debate about whether altmetrics correlate with citation rates,9 10 this study suggests these metrics have correlated during the pandemic.

While citation rates and altmetrics reflect influence, they do not provide insight into research quality. The explosion of COVID-19 publications has raised legitimate concerns about research quality11 as well as misconduct.12 Many journals, particularly top tier journals, prioritized submissions of COVID-19-related articles, potentially at the expense of other topics, and expedited their peer review and publication. The rush to publish on the part of investigators and journals may have encouraged suboptimal research designs and methods as well as suboptimal peer review. Retracted articles perhaps reflect the most egregious examples of how the push to publish during the pandemic promoted poor-quality research. While the proportion of retracted articles in this study was small, these articles received substantial social media attention as well as high citation rates. This study underscores the need for a novel metric that prioritizes research quality rather than quantity.