Most notable 100 articles of COVID-19: an Altmetric study based on bibliometric analysis.

OBJECTIVE: The purpose of this study is to guide researchers in the COVID-19 pandemic by evaluating the 100 most cited articles of COVID-19 in terms of bibliometric analysis, Altmetric scores, and dimension badges.
METHODS: "COVID-19" was entered as the search term in Thomson Reuter's Web of Science database. The 100 most cited articles (T100) were analyzed bibliometrically. Altmetric attention scores (AASs) and dimension badge scores of the articles were evaluated.
RESULTS: T100 articles were published from January to September 2020. The average citation of the top 100 articles on COVID-19 was 320 ± 344.3 (143-2676). The language of all articles was English. The average Altmetric value of T100 is 3246 ± 3795 (85-16,548) and the mean dimension badge value was 670 ± 541.6 (176-4232). Epidemiological features (n = 22) and treatment (n = 21) were at the top of the main topics of T100 articles.
CONCLUSION: The more citations an article is made, the more it indicates the contribution of that article to science. However, the number of citations is not always the only indicator of article quality. The existence of methods that measure the impact of the article outside the academia to measure the value of the article arises more in an issue that affects the whole world, such as the COVID-19 pandemic.

Introduction

The disease, which was first seen in Wuhan, China, was defined as Corona Virus Disease 2019 (COVID-19) by the World Health Organization and spread all over the world, causing the pandemic [1]. As of September 2020, it has caused approximately 31 million cases and 963 thousand deaths [2]. The virus that causes COVID-19 has been defined as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by the Coronavirus Study Group (CSG) [3]. While the most common symptoms are fever, cough, myalgia, and malaise, rare symptoms such as sputum production, headache, hemoptysis, and diarrhea are also seen [4]. Acute respiratory distress syndrome (ARDS) and poor prognosis due to cytokine storm can be observed in the elderly and people with comorbidities [5].

There is no proven cure for COVID-19. Many drug and vaccine studies are carried out. Treatment is mostly based on symptoms. Oxygen therapy is recommended when respiratory distress develops and oxygenation worsens [6].

Bibliometric analysis is a statistical method that counts the citations received since the date of publication of the article (times cited), which helps to predict its effectiveness in the field of science in which it was written and thus aims to shed light on future research in that field [7, 8]..

Social attention–based Altmetric score, which is an alternative article evaluation system, is determined by the frequency of sharing scientific articles on social media. Each citation a post receives means its contribution to science, which further increases the impact value of the article [9]. However, there may not always be a correlation between Altmetric score and times cited.

Recently, dimension badges (Fig. 1), which are inspired by the altmetric system, are also used. The Dimension badge is a research information platform that brings together information about funding, scientific outputs, policy, patents, and grants [10]. The aim of our study is to inform researchers and shed light on future studies by evaluating the top 100 cited articles of COVID-19, as bibliometric, altmetric, and dimension badges, according to the data obtained from Web of Science (WoS) and PubMed, identifying prominent publications and important issues.

Fig. 1

Example of publication metrics (5th article of top 100 cited articles of COVID-19)

Methods

Study design

The type of our study is a retrospective clinical research and level of evidence is 3 or group B (Scottish Intercollegiate Guidelines Network-SIGN 50) [11].

Data collection and inclusion rules

The data used in this bibliometric citation analysis were obtained from the Web of Science (WoS) Core Collection database (Philadelphia, PA, USA) and PubMed (US National Library of Medicine-National Institutes of Health). We accessed the WoS database (accessed date: August 31, 2020) using the keyword “COVID-19” between the months January and August. Thus, we obtained 27,387 articles and analyzed the first 100 articles (T100) that received the most citations among these results. Articles not related to “COVID-19” were excluded from our study and we included original research articles, editorials, correspondences, review articles, and news. Besides, we use PubMed to access additional data about the study. All authors independently defined the T100 with a consensus. We limited quantitative values to “two or more” and “four or more” to shorten most of the tables. SIGN 50 is used for source information regarding the level of evidence [11]. Altmetric attention scores (AASs) were obtained by downloading the “Altmetric it” function from the Altmetric.com website [12]. The AAS is calculated automatically using an algorithm based on the weighted number of all the attention a research output receives.

Since all articles belong to the year 2020, the average number of citations was not used. In determining the value of the article, we determined the total number of citations through the dimension badge system, which we can call the article analysis package.

Statistical analysis

Descriptive statistical methods were mostly used in this study. All the data obtained were defined as percentage, number, bar chart, or mean ± standard deviation (SD) in the tables. We also analyzed the correlations between total citations, Altmetric scores, and dimensional badges. Pearson’s correlation coefficients were calculated to determine the linear relationship between times cited and Altmetric scores. Beta coefficients were estimated by univariate linear regression analysis. Beta coefficients were estimated by univariate linear regression analysis. All analyses were performed by using SPSS for windows version 23.0. p < 0.05 was considered as statistically significant.

Results

The articles in the T100 were published between January 2020 and September 2020. We found the average citation of the top 100 articles on COVID-19 to be 320 ± 344.3 (143–2676). The 100 most cited articles are listed in order of citation (Apendix 1). All articles are published in English. The most cited article named “Clinical Characteristics of Coronavirus Disease 2019 in China” was first published by Guan et al. in April 2020 and received 2676 citations in 5 months. The first 3 articles with more than 1000 citations were studies conducted in China, especially in Wuhan, where the first cases were seen, and were published in March–April, when the cases became pandemics in the world. The average Altmetric value of T100 is 3246 ± 3795 (85–16,548), and the article with the most Altmetric value is “Covid-19-Navigating the Uncharted” by Fauci et al. with 16,548.

In correlation and regression analysis results, the mean dimension badge value was 670 ± 541.6 (176–4232), and the article with the most dimension badges was “Clinical course and risk factors for mortality of adult in patients with COVID-19 in Wuhan, China: a retrospective cohort study.”

When T100 articles are examined according to the distribution of main topics, epidemiological, viral, clinical features, and treatment of COVID-19 constitute the subject of 78 out of 100 articles (Table 1). Epidemiological features (n = 22) and treatment (n = 21) were at the top of the main topics of T100 articles. In our study, we found that in some of the articles in T100, intersection clusters were formed in main subjects, that is, they included more than one main topic (Fig. 2). When the distribution of T100 by months was examined, 81% of the articles were published in March, April, and May. The most cited cases were detected in April (n = 411) and July (n = 419), when the cases escalated.

Table 1

Classification of the 100 most cited articles according to main subjects and level of evidence*

Main subject	Level 1	Level 2	Level 3	Level 4	Total (= n)
Epidemiologic features	2	1	3	16	22
Treatment	2	2	5	12	21
Clinical features		4	11		15
Viral features		1	5	4	10
Clinical and epidemiologic features		2	3	4	9
Radiologic features			5	1	6
Neurologic features			3	2	5
Transmission dynamics			2	2	4
Clinical and pathological features			2		2
Clinical and radiologic features	1		1		2
Pathologic features				1	1
Clinical and viral features			1		1
COVID-19 on internet				1	1
Economics				1	1
Total (= n)	5	10	41	44	100

*Level of evidence is determined according to the SIGN classification system

Fig. 2

Distribution and intersection points of top 100 cited articles main subject on COVID-19

Two or more articles were published in the top 15 of the journals in which T100 articles were published. In the first 15 journals in which T100 articles were published, two or more articles were published, and 63 (63%) of the T100 articles were published in these 15 journals (Table 2). Most articles were published in Jama Journal of The American Medical Association, New England Journal of Medicine, and Lancet, and the number of articles published was 12, 12, and 7, respectively. The average H-index of 15 journals with two or more COVID-19-related publications is 357.8 ± 300.6 (46–987) [13]. According to the quartile (Q) scores of the journals, 14 of them are Q1, and 1 of them is Q2 [13]. In the list of countries where T100 articles are published most, there are 21 countries, with China (n = 59) in the first place, followed by the USA (n = 31) and England (n = 17) (Table 3). Twenty-two [14] universities with 4 or more publications were included in the list of universities with the highest number of T100 articles (Table 4). The universities, where the most articles were published, were Huazhong University of Science and Technology (n = 14, China), Wuhan University (n = 7, China), and University of Oxford (n = 6, England). We found that the most prominent authors in T100 articles were Lıu Y. with 7 publications and Hu Y. with 5 publications.

Table 2

List of journals in which two or more articles were published

Rank	Journal	Number of articles	H-index*	Quartile score*
1	Jama Journal Of The Amerıcan Medical Association	12	654	Q1
2	New England Journal Of Medicine	12	987	Q1
3	Lancet	7	747	Q1
4	Lancet Infectious Diseases	6	217	Q1
5	Radiology	5	283	Q1
6	International Journal Of Antimicrobial Agents	3	118	Q1
7	Intensive Care Medicine	2	187	Q1
8	International Journal Of Infectious Diseases	2	79	Q1
9	Jama Cardiology	2	46	Q1
10	Journal Of Medical Virology	2	111	Q2/Q3
11	Journal Of The American College Of Cardiology	2	419	Q1
12	Journal Of Thrombosis And Hemostasis	2	165	Q1
13	Lancet Respiratory Medicine	2	93	Q1
14	Nature Medicine	2	524	Q1
15	Proceedings Of The National Academy Of Sciences Of The United States Of America	2	737	Q1

2020 Scimago Journal and Country Rank

Table 3

Geographic of origin of top 100 articles

Countries/region	Numbers
China	59
USA	31
England	17
Italy	13
Canada	7
Germany	6
Japan	6
Netherland	6
Spain	5
France	4
Singapore	4
Switzerland	4
Australia	3
Denmark	3
Saudi Arabia	3
Belgium	2
Greece	2
India	2
Russia	2
South Korea	2
United Arab Emirates	2

Table 4

Institutions of origin with 4 or more of the top 100 cited articles

Rank	Institutions	Numbers
1	Huazhong Univ Sci Technol (China)	14
2	Wuhan Univ (China)	7
3	Univ Oxford (England)	6
4	Univ Hong Kong	6
5	Columbia Univ	5
6	Icahn Sch Med Mt Sinai	5
7	Univ Washington	5
8	Capital Med Univ	4
9	Chinese Acad Med Sci	4
10	Humanitas Univ	4
11	Imperial Coll London	4
12	Minist Hlth	4
13	Shanghai Jiao Tong Univ	4
14	Sun Yat Sen Univ	4
15	Tsinghua Univ	4
16	Univ Milan	4
17	Univ Milano Bicocca	4
18	Univ Toronto	4
19	Univ Utrecht	4
20	Univ Virginia	4
21	Wuhan Jinyintan Hosp	4
22	Zhejiang Univ	4

Results of correlation and regression analysis

There was a moderate positive correlation between times cited and Altmetric scores (r = 0.416; p = 0.001) (Fig. 3). According to univariate linear regression analysis, 17.3% of variation in Altmetric score was explained by time cited (Table 5). One unit increase in time cited resulted in 2.96 increases in Altmetric score. Regression model to estimate Altmetric score was;

Fig. 3

Scatter plot of relationship between Altmetric score and Times cited

Table 5

Univariate linear regression analysis result to estimate Altmetric score by using

	Unstandardized coefficienta
	B	Std. Error	p
Constant	2298.26	503.18	0.001
Average citation per year	2.96	1.07	0.007*

aDependent variable Altmetric score

*Significant at 0.05 level

Discussion

The need for rapid and urgent information on COVID-19 has rapidly increased the number of articles [15]. With new fluctuations, the COVID-19 pandemic seems to continue to negatively affect human health for a long time. Therefore, interest in the COVID-19 issue both in social media and in the academic community will continue to remain alive for a while.

After the first publication on COVID-19 was published on 17 January 2020 [16], the number of publications increased rapidly each week and as of August 31, 27,387 articles were published. While there were 27,387 articles on August 31, 2020, the number of articles published on September 11, 2020, increased to 30,637 and the average number of articles per day was 295.

There are bibliometric reviews of COVID-19 in the literature, including the study examining the Indian publications [17], the study examining orthopedic-related COVID-19 articles [18], the bibliometric analysis of the articles from December to May by Pericas et al. [19], the bibliometric study examining manuscripts between December 1, 2019, and April 20, 2020, by Francesca de Felice et al. [20] and the study examining the top 10 most cited articles by Zhou et al. [21].

In addition, there is also Mohamad Chahrour et al.’s article reviewing 1809 publications on COVID-19 between December and March 2020 [22] and the bibliometric article by Sa’ed H. Zyoud et al., which examines both the nineteen thousand articles published until 19 June 2020 and the 20 most cited articles [14].

There are no studies examining the 100 most cited articles on COVID-19. When we summarize the areas of the 100 most cited articles among the articles on COVID-19, they are distributed as 42 internal medicine, 14 infectious disease, 9 cardiovascular systems, 6 pharmacology and medicine, 5 public health, 5 radiology, and 5 science technologies.

Epidemiological features and treatment of COVID-19 stand out as the main subject in our study. According to Table 1, there are 5 studies at the level of evidence 1 and 10 studies at the level of evidence 2, and 85% of the studies in T100 are at the level of 3 and 4. In fact, this situation shows that the level of evidence of the articles on the T100 list on COVID-19 is not strong. The main reason for the weak level of evidence of the first reported articles is that they are descriptive studies that draw attention to the subject in the style of case series and correspondence for the need for rapid information transfer. Peer-review studies were carried out in later processes. The reason why some articles published at the beginning of the process were retracted may be due to the inability to peer-review the first articles. While the number of articles and citations increased in the first months when the cases increased the most, it stagnated in May and June, when the peak in the number of cases decreased.

The majority of early studies were retrospective studies and case series studies by Chinese authors in Chinese language [23]. In most of the articles, original data was published in English. All T100 articles in our study were published in English. The number of citations increases the interest and value of the article. The altmetric method shows the interest of social media in the article [24]. In our study, in addition to citation, social media interest was determined by calculating the altmetric value of T100 articles. For the first time in this study, a correlation has been established between the altmetric value and times cited regarding COVID-19.

In the study by Konstantinos et al., 140 articles were published until the 14th of March; 5 journals with high impact factor were examined and the total number of citations was found to be 4210 and news-social media comments were found to be 342,000 [25]. The same study suggested that in COVID-19, there is a greater increase in social media interest and citation numbers in a shorter time than expected. In our study, a moderate positive relationship was found between the number of citations and the Altmetric score (r = 0.416; p = 0.001). This may be because all the articles on COVID-19 are heavily shared on social media by scientists and other researchers.

Social media interest can be positive or negative. AAS increases with positive comments as well as AAS increases with negative comments. Therefore, this situation should also be taken into account when evaluating AAS. However, the articles with the highest citation may not receive much attention on social media due to their subject content. The time cite of the article, which has the highest score on the altmetric explorer and deals with non-pharmacologic treatments (social isolation, preventive measures) in COVID-19 [26], is low. While it has a 64,768 Altmetric score, times cited is only 44. It could not be included in the top 100 articles we evaluated in our study.

The total number of citations is displayed on the badge in dimension, and the total citation, recent citation and field citation, relative range, and citation ratios are calculated. Since the studies are in the last 6 months, recent citation and times cited are the same. Since the field citation ratio and recent citation ratio contain the ratios of the last 2 years, these ratios are “0” in our study. Total citation is the total number of times a publication has been cited. The reason why we generally find the total number of citations higher in the dimension according to the number of citations we receive from WoS in our study is that citing publications can be of any publication type, such as articles, chapters, preprints, or monographs. Dimension badge shows that chapters, preprints, and monographs are added to the number of citations besides articles and provides us an analysis package on the subject. As the citation value of the article increases in the dimension, it shows that the article is cited from all types of publications belonging to the fields of interest, as well as medical and health sciences.

In the T100, the article with the second highest number of hits ranked first with the dimension badge score of 4232. This has proven that the value of articles has changed on different platforms.

Conclusion

COVID-19 has quickly become a focus of attention both in the health field and in the scientific literature, as it causes the pandemic. The urgent and rapid requirement for information on COVID-19 has rapidly increased the number of articles [15]. Many published articles highlighting the different aspects of the COVID-19 pandemic have been shared in a short time not only in academia but also on social media. Epidemiological and therapeutic articles have received more citations and have also found a response on social media.

The number of citations and altmetric values of the articles on COVID-19 increased more than expected. In order to contribute to scientific studies on COVID-19, in our study, we determined the number of citations of T100 articles, their main topics, the universities that published them, the scientific journals they were published, their level of evidence, social media interests, and the number of citations-Altmetric score correlations. The number of citations is not always the only indicator of the quality of the article, but reflects the value of the article and its contribution to science.