Investigating of Association between Altmetrics Activity Indicators and Citation Quality Indicators in Iranian Medical Journals.

BACKGROUND: Given the limitations of traditional citation indicators, more indicators are needed to examine the effectiveness and improvement of existing indicators. The present study aimed to investigate the association between Altmetrics activity and quality citation indicators in Iranian journals based on Clarivate Analytics, Scopus, and Medline.
METHODS: The research was carried out using Altmetrics method through scientometric approach. The population of the present study was Iranian medical journals, which were available at three databases of Clarivate Analytics, Scopus, and Medline. In order to obtain quality information, we've used the indicators of CiteScore, SJR, and SNIP at Scopus database as well as the impact factor at JCR database; besides, to find Altmetrics indicators and Altmetrics scores of articles, the Altmetrics explorer database was used.
RESULTS: About 16% of the articles in the reviewed journals were cited at least once in the social media and had Altmetrics scores. Among the reviewed journals, the highest rate of social media coverage was related to the Iranian Journal of Basic Medical Sciences, and the highest Altmetrics score obtained from the mean Altmetrics score of the papers was related to Cell Journal (Yakhteh). A review of quality indicators of journals with Altmetrics coverage and Altmetrics score of journals showed that there was a significant and positive correlation between the Altmetrics score and impact factor. However, any significant association was neither found between the journal's coverage and SiteScore, SNIP, SJR, nor between the journal's Altmetrics score and quality indicators.
CONCLUSIONS: According to the results of this study, policy makers of scientific journals should adopt strategies that bring about social media presence; thus, we will find further Iranian articles in the social media. Copyright:

Introduction

Altmetrics indicators which are derived from the social web are increasingly becoming popular and are used as the primary impact factors. Altmetrics—stands for alternative metrics—means alternative indicators, which is a term to describe web-based indexes to impact the research articles and emphasizes the social media for data collection. The concept was first introduced in 2010 by Priem et al.[1] The web age has provided a new environment for the discussion and dissemination of research information and according to this new environment there is a need for more indicators to examine the impact and to complete the existing indicators.[2] Altmetrics indicators have eliminated several limitations of traditional citation-based indicators,[3] in which other aspects of impact has developed, such as social, cultural, and education impacts, which were neglected by traditional methods of impact assessment.[1] Taylor referred to Altmetrics indicators as a revolution in scientometrics[4]; and Lin and Fenner described it as a new paradigm in research evaluation.[5] Altmetrics focuses on research outputs in social networks such as Facebook, Twitter, blogs, news outlet and reference management tools, and goes beyond citation in analyzing academic activities. Instead of just analyzing the articles of scientific journals, Altmetrics considers the impact of scientific output in different formats.[1] Altmetrics, unlike the citation-based indicators, such as the journal's impact factor or the H-index, evaluates the impact of publication of a single article.[6] The indicators of article level measure the impact of a single article through quantitative indicators, such as counting the number of times that an article is downloaded or shared on Twitter, as well as quality indicators such as media coverage or blog posts by the experts in that field.[7] However, some scientometrics experts consider Altmetrics as a threat to the alignment process and quality control of research activities[8]; nevertheless, pursuing the use of scientific articles in social media means that researchers are able to examine the impact of these products more extensively[9]; hence, along with the citation, these indicators should be investigated and their impact should be considered. Salajegheh and Diari examined the association between citation indices and Altmetrics in 111 medical sciences journals in Scopus database. Altmetrics data was obtained from Altmetrics official website According to their results, there was a correlation between the citation indicators (except the impact factor) and the mean scores of Altmetrics.[10] Goltaji, and Jowkar studied the extent of availability of medical informatics articles in social media through Altmetrics approach. The results showed that the scientific outputs were often used through social networks. In this study, a significant positive association was found between most of Altmetrics indicators and the number of received citations on the Web of Science platform.[11] The study of Erfanmanesh on the relationship between Altmetrics activity and quality of the library and information sciences journals showed that the published articles in higher quality scientific journals were more widely seen on social networks, and attracted more attention of users.[12]

Serati Shirazi and Goltaji carried out a research on Scientific Articles of “Health Literacy” in Social Media. The results of their study revealed that Mendeley and Twitter are the most used social media by health literacy scholars for sharing scientific outputs and there is significant relationship between most alternative metrics and the number of citations in Web of Science.[13]

Zahedi, Costas and Wouters studied the use of Web of Science journals at Mendeley, as well as the articles' citation indexes through the Impactstory website, and reported a moderate correlation between these two indicators.[14] In a comparative study, Cho studied the impact of Korea's research papers in four academic disciplines using the Altmetrics approach. His research population included 383 Korean international research papers in medicine, engineering, social sciences, arts and humanities. The results of this study suggested that the highest number of research articles that were “discussed” in international media such as Twitter was in medicine; and the highest number of research papers that had been stored via reference management tools such as Mendeley was seen in social sciences, arts and humanities.[15]

Articles of medical sciences journals, considering the novelty of the subject in this discipline, are being studied shortly after publication; and the spent time on the publication of the article may reduce the value of the content. Therefore, the use of articles in online social networks can be effective in updating the publications. Hence, it is essential to assess the Altmetrics approach in medical sciences articles, in order to be used in research policies and in evaluating the publications quality.

Since the Altmetric activity of Iranian medical journals has not been studied so far, thus, the objective of this study was to investigate the association between Altmetrics activity and quality citation indicators in Iranian journals on three databases of Clarivate Analytics, Scopus, and Medline.

The results of this study can persuade policymakers that they should pay more attention to the social media and the rules governing them such as limitations and filtering. They should inform everyone especially the scientific community about social networks and the way to be an active one in using them and also their benefits.

In addition, they can encourage researchers to publish their research in online and open access format, and can make research policy to promote faculty members based on their Altmetric score.

Methods

The research was carried out using Altmetrics method through scientometric approach. The research population consisted of Iranian medical journals, which were available on databases of Clarivate Analytics, Scopus, and Medline. In order to obtain quality information, the indicators of CiteScore, SJR, SNIP were used, which were extracted from the Scopus database, and the impact factors of journals were obtained from SJR database. During the research, since the last calculation for these indicators was in 2017, the same year was considered for data collection. The Altmetric Explorer was used to obtain Altmetrics indices and Altmetrics scores.

The Altmetric Explorer database provides the possibility of searching articles in a variety of ways, such as the use of scientific identifiers including the digital object identifier (DOI), and it also provides the articles from a journal using ISSN. Because the aim of this study is to evaluate all articles in medical journals, we used the journals' ISSN.

The Altmetrics score was calculated by giving weight to the scores obtained from the above-mentioned social media and was assigned to the article.[14] For each journal, Altmetrics scores were obtained for all articles. Then, the mean Altmetrics scores for articles of each journal was considered as Altmetrics activity index of the journal and was used in correlation tests. Statistical tests were performed using SPSS 20 statistical software and Spearman's correlation test.

Results

The results of this study showed that in 2017, 15 journals published a total of 1,674 articles. Iranian Red Crescent Medical Journal with 269 articles had the highest number of articles, and DARU Journal of Pharmaceutical Sciences with 25 articles had the smallest number of articles in 2017. As shown in Graph 1, the number of published articles in 2017 in the journals does not follow the equality distribution. In other words, in 1 year according to the journal's size and the consumed time for publishing, each journal published various numbers of articles.

Graph 1

Frequency of articles in the reviewed journals in 2017

The results of reviewing 15 journals using the ISSN on the Altmetrics Explorer showed that only 13 journals were retrieved, for instance, by entering the DOI of the published articles, the information of Arch. Iran. Med. Journal was retrieved, while no information was obtained from DARU Journal. The findings showed that among the 1,674 articles published in these journals (including articles in DARU Journal), there has been paid attention to 276 articles for at least once, which had Altmetrics score. Accordingly, only about 16% of these articles had Altmetrics score. In total, these documents were taken into consideration 685 times, and on average, the share of each of the 256 papers was about 2.7. Among the studied journals, the most frequent Altmetrics coverages were 50.75, 34.38, and 34.24, respectively, in the Iranian Journal of Allergy, Asthma & Immunology, Iranian Journal of Immunology, and Iranian Journal of Basic Medical Sciences.

According to the results of Table 1, the highest presence of articles in social media were mentioned for 649 (94.7) times, among which Twitter (583 times) was ranked first, and Facebook, Google+, and Reddit Posts were in the next ranks long after Twitter. After the social media, the highest presence rate was referred to the category of “other sources” with 19 times of presence (8.2), the presence of resources in Wikipedia was 6 times and they were 11 times mentioned in the videos. News and weblogs ranked third with 17 times of presence (2.4). In this category, news with 11 and weblogs with 16 times mentioning were respectively in the first and second rank. As stated above, the examined articles had the highest levels of presence on Twitter. The geographic distribution of Twitter users showed that the highest number of tweets was from the United States, while England and Spain were in second and third places, respectively, and Iran ranked tenth. Facebook had the most number of posts after Twitter. A survey of Facebook users showed that the highest number of users was from the United States, India, and Spain, respectively; however, no definite user was from Iran.

Table 1

The rate of mentioning the articles in the reviewed journals in various media

Type of Social Network	Mentioned counts percentage	Mentioned counts	Social Network	Mentioned Counts
Social media	94.7	649	Twitter	583
			Facebook	55
			Google+	9
			Reddit Posts	2
News and blogs	2.4	17	News	11
			Weblog	6
Other sources	2.8	19	Wikipedia	6
			Videos	13
			Q & A Posts	0
Policy & Patents*	0	0	Policymaking sources	0
Academic sources	0	0	Peer Reviews	0
			Research Highlights	0

*Policy & patents, policymaking sources, and academic sources were the categories in which the articles in the reviewed journals were not present

Specifically, Hepatitis Monthly Journal, Iranian Journal of Allergy, Asthma and Immunology, Iranian Journal of Immunology, Iranian Journal of Kidney Diseases, Iranian Journal of Pharmaceutical Research, Iranian Journal of Radiology, Iranian Red Crescent Medical Journal, and Urology Journal were only available in social media; and Jundishapur Journal of Microbiology was only present at news and blogs. Other journals were present at more than one category. Table 2 represents the rate of citation to the articles in the reviewed journals in different media.

Table 2

The rate of reference to the articles in the reviewed journals in different media

Journal	Presence	Mentioned Counts Percentage	Mentioned Counts	Social networks	Mentioned Counts
Hepatitis Monthly	Social networks	100%	4	Twitter	4
				Facebook	0
				Google+	0
				Reddit Posts	0
Iranian Journal Of Allergy Asthma And Immunology	Social networks	100%	63	Twitter	63
				Facebook	0
				Google+	0
				Reddit Posts	0
Iranian Journal Of Basic Medical Sciences	Social networks	90.27%	130	Twitter	120
				Facebook	9
				Google+	0
				Reddit Posts	1
	News and blogs	4.16%	6	News	6
				Weblog	0
	Other sources	5.55%	8	Wikipedia	2
				Videos	6
				Q & A Posts	0
Iranian Journal Of Immunology	Social networks	100%	21	Twitter	21
				Facebook	0
				Google+	0
				Reddit Posts	0
Iranian Journal Of Kidney Diseases	Social networks	100%	19	Twitter	17
				Facebook	2
				Google+	0
				Reddit Posts	0
Iranian Journal Of Parasitology	Social networks	97.5%	39	Twitter	38
				Facebook	1
				Google+	0
				Reddit Posts	0
	News and blogs	2.5%	0	News	0
				Weblog	0
	Other sources	0	1	Wikipedia	0
				Videos	1
				Q & A Posts	0
Iranian Journal Of Pharmaceutical Research	Social networks	100%	76	Twitter	56
				Facebook	16
				Google+	4
				Reddit Posts	0
Iranian Journal Of Radiology	Other sources	100%	2	Wikipedia	2
				Videos	0
				Q & A Posts	0
Iranian Red Crescent Medical Journal	Social networks	100%	1	Twitter	1
				Facebook	0
				Google+	0
				Reddit Posts	0
Journal Of Research In Medical Sciences	Social networks	96.29%	130	Twitter	111
				Facebook	14
				Google+	4
				Reddit Posts	1
	News and blogs	0.74%	11	News	0
				Weblog	1
	Other sources	2.96%	4	Wikipedia	1
				Videos	3
				Q & A Posts	0
Jundishapur Journal Of Microbiology	News and blogs	100%	1	News	1
				Weblog	0
Urology Journal	Social networks	100%	27	Twitter	22
				Facebook	5
				Google+	0
				Reddit Posts	0
Archives Of Iranian Medicine	Social networks	90%	45	Twitter	44
				Facebook	1
				Google+	0
				Reddit Posts	0
	News and blogs	8%	4	News	2
				Weblog	2
	Other sources	2%	1	Wikipedia	0
				Videos	1
				Q & A Posts	0

Positively, the most frequent presence in the social media was related to the Iranian Journal of Basic Medical Sciences, from which 63 articles were shared at least once and had Altmetrics score. The articles in this publication were 144 times referenced; the articles in this journal were totally cited 144 times. Following that, 44 articles from Journal of Research in Medical Sciences and 34 articles from Cell Journal (Yakhteh) have been shared at least once, each of which were cited 135 and 102 times, respectively. Both journals ranked second and third, respectively. Table 3 illustrates the number of mentioned articles and the number of mentions in each journal (the presence of each journal).

Table 3

The number of mentioned articles and the number of mentions, along with Altmetrics score of journals

Altmetrics score	Total Altmetrics score	Number of articles	Percentage of articles presence in all articles of the journal	Number of articles	Number of mentioned outputs	Total mentions	Journal title
Iranian Journal of Basic Medical Sciences	144	63	184	34.24	184	142	0.77
Journal of Research in Medical Sciences	135	44	138	31.88	138	102	0.13
Iranian Journal of Allergy, Asthma & Immunology	63	34	67	50.75	67	41	0.61
Iranian Journal of Parasitology	40	27	82	32.93	82	32	0.39
Iranian Journal of Pharmaceutical Research	76	24	188	12.77	188	51	0.27
Cell Journal (Yakhteh)	102	22	79	27.85	79	101	1.27
Urology Journal	27	16	63	25.40	108	8	0.07
Iranian Journal of Immunology	21	11	32	34.38	32	13	0.4
Iranian Journal of Kidney Diseases	19	9	76	11.84	76	12	0.15
Iranian Journal of Radiology	2	2	114	1.75	114	6	0.05
Hepatitis Monthly	4	2	87	2.30	87	4	0.04
Jundishapur Journal of Microbiology	1	1	108	0.93	108	8	0.07
Iranian Red Crescent Medical Journal	1	1	269	0.37	269	1	0.003
Archives of Iranian Medicine	50	19	162	11.73	162	66	0.04

However, since the articles presence was considered in relation to the number of articles in the journal, the highest rate of articles presence (75.50) was assigned to Iranian Journal of Allergy, Asthma & Immunology. 34 articles out of 67 articles from this journal were mentioned at least once. Thus, more than half of this journal's articles were present on social networks. Table 3 exhibits the percentage of each journal's articles which had presence in social media. In Table 3, the mean Altmetrics scores of the examined journals was reported from the mean Altmetrics score of the journals' articles; so that Cell Journal had the highest Altmetrics score with 1.27 Altmetrics score. DARU Journal of Pharmaceutical Sciences, with an impact factor of 2.667, CiteScore of 2.13, and SNIP of 1.001 had the highest rate of these qualitative indicators; however, according to SJR indicator, Jundishapur Journal of Microbiology with 2.297 had the highest rate for SJR index. Table 4 shows the qualitative indicators of the reviewed journals.

Table 4

Citation quality indicators in the reviewed journals

Title	Alt score mean	IF 2017	CiteScore	SNIP	SJR
DARU-Journal of Pharmaceutical Sciences	0	2.667	2.13	1.001	0.528
Cell Journal	1.27	2.363	2.12	0.862	0.692
Hepatitis Monthly	0.04	1.81	1.92	0.841	0.785
Iranian Journal of Basic Medical Sciences	0.77	1.514	1.73	0.753	0.538
Journal of Research In Medical Sciences	0.13	1.391	1.52	0.871	0.593
Iranian Journal of Pharmaceutical Research	0.27	1.372	1.56	0.812	0.502
Archives of Iranian Medicine	0.4	1.254	1.28	0.904	0.61
Jundishapur Journal of Microbiology	0.07	1.233	1.67	1.875	2.297
Iranian Journal of Kidney Diseases	0.15	1.192	1.07	0.687	0.481
Iranian Journal of Allergy Asthma And Immunology	0.61	1.049	1	0.503	0.326
Iranian Journal of Parasitology	0.39	1.043	1.42	0.802	0.656
Urology Journal	0.3	0.88	1.53	0.824	0.543
Iranian Journal of Immunology	0.4	0.787	0.9	0.491	0.376
Iranian Red Crescent Medical Journal	0.003	0.786	1.16	0.827	0.42
Iranian Journal of Radiology	0.05	0.524	0.64	0.498	0.216

Spearman's correlation test was used to investigate the association between citation-based qualitative indicators (impact factor, SJR, SNIP, and CiteScore) and Altmetrics activity of journals including the journal's Altmetrics coverage and Altmetrics score. The results of this test for the journals showed that there was a significant and positive correlation between journals coverage and the impact factor (r = 0.546; P = 0.035), but there was no significant association between journals coverage and CiteScore, SNIP, and SJR. Moreover, there was no significant association between Altmetrics scores and the qualitative indicators of the journals.

Discussion

The present research was carried out aiming to investigate the association between Altmetrics activity and citation quality indicators in Iranian journals on three databases of Clarivate Analytics, Scopus, and Medline, which was conducted through scientometric approach and Altmetrics methodology. From the reviewed journals, Archives of Iranian Medicine and DARU were not retrieved by using ISSN. But by entering the DOI of the articles, the information of Archives of Iranian Medicine was retrieved, but no information was obtained from DARU Journal. This issue should be addressed by journalists and policy makers of the journals; because today, apart from the impact of articles on science, which is calculated by counting citations and calculating the impact factor, the impact of articles in the community is measured by Altmetrics in terms of dimensions other than scientific dimension. Failure to retrieve articles in databases that assess Altmetrics indicators means that the articles are not seen and it is not possible to examine the social impact of the article. To solve this problem, journal should increase their visibility on the web by creating weblogs, ResearchGate, and Linkedin pages and so on.

The findings suggested that just about 16% of these published articles (considering the articles of DARU Journal) had Altmetrics scores. This trend is little for medical journals that should be available to the scientific community and even other people as soon as possible. Social media and cyberspace in general allow scientific content to be readable, regardless of the time-consuming process of publishing the papers. The lack of using this possibility is clearly seen for Iranian medical journals that have been investigated in this study.

According to the results, the examined articles had the highest rate of presence on Twitter, and Facebook had the highest number of posts after Twitter. Social media, especially Twitter, are powerful tools in terms of having Altmetrics data and can be a comprehensive tool for researchers; saving, tagging, and tweeting articles on Twitter can reflect the impact of these articles on users. In other words, increasing social impact has led to an increase in citation.[16] The geographic distribution of Twitter users showed that the highest number of tweets was from the United States, while England and Spain ranked second and third respectively; Iran ranked tenth. The filtering of the Twitter network in Iran can be one of the most important reasons for not using this network as much as it can be used and also in comparison the United States and some other countries such as England the English language as a second language in Iran can be the other reason.

A review of Facebook users demonstrated that the highest number of users was from the United States, then from India and Spain; and no specific user from Iran was found. The results of Goltaji and Jowkar's study showed the active presence of members from the United States, England, Canada, and Spain in social media.[11] The geographic review of tweets about Iranian articles in Erfanmanesh's study showed that the highest number of tweets belonged to the United States, England, and Canada,[8] which was in line with the present study. Erfanmanesh argued that articles published in higher quality scientific journals were more widely present on social networks and attracted more attention. By increasing electronic access, readers can have access documents with less effort, as users are more eager to retrieve documents with less access barriers. According to the principle of least effort, the user tries to achieve the desired result by spending minimum time and effort. One of the grounds provided to achieve this goal is social media. Therefore, it is necessary for researchers to use this environment in order to be seen and effective. Mazov and Gureev suggested that some journals shared all the published articles in their latest volumes manually or mechanically via Twitter.[17] Accordance of the results of investigating the geographical dispersion of tweets confirmed this, so it is essential for the publishers to make serious efforts to inform about the articles.

Absolutely among the reviewed journals, the articles of “Iranian Journal of Basic Medical Sciences” had the highest levels of social media presence. However, in terms of the presence of journal's articles comparing to the number of articles, the highest number of articles was belonged to “Iranian Journal of Allergy, Asthma & Immunology.” The highest Altmetrics score was 1.27, which was belonged to “Cell Journal.” “DARU” journal of pharmaceutical sciences with an impact factor of 2.667, CiteScore of 2.13, and SNIP of 1.001 had the highest rates among these qualitative indicators, but considering the SJR index, “Jundishapur Journal of Microbiology” had the highest rate (2.297). The results of reviewing the relationship between citation-based qualitative indicators (impact factor, SJR, SNIP, and CiteScore) and Altmetrics activity of journals, including Altmetrics coverage and Altmetrics score, indicated that there was no significant relationship between journals coverage and CiteScore, SNIP, and SJR; besides, there was not found any significant relationship between the journals' Altmetrics score and qualitative indicators. These results were not in line with Erfanmanesh's research.[12] But there was a significant and positive correlation between the journals coverage and the impact factor (r = 0.546, P = 0.035), which was consistent with Erfanmanesh's research.[12] The results of Goltaji and Jowkar's study indicated that there was a significant association between the received citations on the Web of Science and Altmetrics indicators of most social media, so that the existence of articles increased the number of citations to those articles.[11] In most studies conducted in the field of Altmetrics, researchers reported a positive and significant relationship between the quality indicator (citation) and the Altmetrics score of the articles, so that the findings of this study were consistent with the results of Zahedi et al.,[14] Mohammadi et al.,[18] Ebrahimy and Setareh,[19] and Li et al.[9] On the other hand, these studies have shown that social media has had a positive impact on increasing the number of citations to articles.

Conclusion

Although there was a reasonable and moderate association between Altmetrics indicators and citations, the quality of the data obtained from the Altmetrics and their effective dimensions which could be reported accurately were not clear; and Altmetrics was still immature. Nonetheless, researchers of the present study believed that the indicators of article and Altmetrics could be used additional to the scientometric indicators in order to investigate the impact of scientific output and the performance of researchers in Iran. On the other hand, according to the results of this study and similar studies, it can be concluded that social media has had a positive impact on the increase of citations to articles. Therefore, policymakers of scientific journals need to provide facilities for the presence in social media in order to see more Iranian articles in social media.

Limitations

Some limitations of the present research were limited coverage of data provided by the Altmetrics Institute and other Altmetrics service providers, as well as the inability to access certain social media within the country due to filtering; these limitations could affect the results.

Suggestions for further studies

Although in this research it was tried to fully review medical journals of three validated databases, it is suggested that this research should be repeated in different times and the process of changes in the Altmetrics indicators should be considered. In the present research, we examined the medical journals regardless of their specialty. It is suggested that for further studies the articles in various disciplines should be reviewed and compared to each other.

Ethics committee approval

Ethics approval was obtained from the Semnan University of Medical Ethics Committee (IR. SEMUMS. REC.1397.044).

Financial support and sponsorship

Research reported in this publication was supported by a grant [number: 1404] from the Semnan University of Medical Sciences, Semnan, Iran.

Conflicts of interest

The authors declare that they have no competing interests.