Impact of Altmetrics in evaluation of scientific journals, research outputs and scientists' careers: Views of editors of high impact anaesthesia, critical care and pain medicine journals.

BACKGROUND AND AIMS: Altmetrics represent the attention of an article drawn from social and mainstream media. The aim of this survey was to investigate the views of editors of high-impact journals on the Altmetric Attention Score (AAS), the number derived from an automated algorithm including a weighted count of mainstream news and social media sources.
METHODS: A questionnaire related to the AAS was sent to the editors of high-impact journals, namely Anaesthesia, Critical Care Medicine (CCM) and Pain Medicine (PM). Eleven questions were related to the possible benefits and flaws of Altmetrics.
RESULTS: Of the 1,381 editors asked, 126 answered. The overall answers showed that 76% of the editors were familiar with Altmetrics, 28% knew how AAS is calculated, 12% believed that AAS should replace traditional bibliometrics, 34% favoured AAS for journal ranking, 40% believed that AAS should be used to assess an article, 44% felt that AAS should be included in researchers' curriculum vitae and 22% felt that it should be considered for grants. Sixty-two percent of editors believed that AAS is vulnerable to manipulations, 60% proposed improvement and 16% abandonment. Positive answers were similar across the fields, except for journal ranking. Fifty-four percent of editors of the CCM journals favoured journal ranking using AAS versus 28% and 26% editors of anaesthesia and PM journals (P = 0.025 and P = 0.006, respectively).
CONCLUSION: A high percentage of editors believed that AAS should be used to assess scholarly output and that it should be included in the researchers' curriculum vitae. Sixty percent of responders supported the improvement of AAS. Copyright: © Indian Journal of Anaesthesia.

INTRODUCTION

The rapid growth of the online literature along with the extensive use of social media gave birth to alternative metrics or Altmetrics. Altmetrics are bibliometric indicators expressing public opinion through social and mainstream media. Several providers in Altmetrics are available using different sources for scholarly output evaluation, PlumX and Altmetric.com being the most frequently used ones. PlumX evaluates the scientific articles in the journals of Elsevier Publications.[1] Altmetric.com is the oldest one. The Altmetric Attention Score (AAS) is an integral number that indicates the attention that a scholarly output receives in real-time as it is derived from social and mainstream media sources. The sources and their weighted counts implemented to calculate the AAS are[2]: News 8, Blog 5, Policy document (per source) 3, Patent 3, Wikipedia 3, Twitter (tweets and retweets) 1, Peer review (Publons, Pubpeer) 1, F1000 1, Syllabi (Open Syllabus) 1, Face book 0.25, Reddit 0.25, Q&A (Stack Overflow) 0.25, Youtube 0.25, Number of Mendeley readers 0, Number of Dimensions, and Web of Science citations 0.

A correlation between traditional bibliometrics and AAS has been found in several studies.[34] Some studies suggest the AAS be considered as an alternative to or an early measure of traditional bibliometrics.[56] According to other studies, social media metrics should not replace traditional bibliometrics,[37] as several sources of social media such as tweets or Facebook are not a measure of quality. In fact, there is no unanimous agreement regarding Altmetrics and their implementation as an assessment tool of research impact.

The aim of the present study was to investigate the views on Altmetrics of editors serving high-impact journals across different—but related—disciplines, such as anaesthesiology, critical care medicine and pain medicine, by e-mailing them a questionnaire.

METHODS

Institutional Review Board approval was waived because the study investigated the views on the impact of alternative metrics in science and did not involve interventions in human subjects or experimental animals. A questionnaire was sent to the editors of high-impact journals of different but related disciplines, such as anaesthesiology, critical care medicine and pain medicine. Subspeciality journals related to obstetric anaesthesia, neurosurgery, cardiovascular anaesthesia, etc., were not included in the study, even though several editors who received the questionnaire could also be editors of such journals.

The journals we considered are—in alphabetical order—as follows:

Anaesthesia journals (impact factor): Acta Anaesthesiologica Scandinavica (2.050), Anaesthesia (5.739), Anesthesia & Analgesia (4.305), Anesthesiology (7.067), British Journal of Anaesthesia (6.880), Canadian Journal of Anesthesia (3.779) and European Journal of Anaesthesiology (4.500).

Critical care medicine journals (impact factor): American Journal of Critical Care (2.105), American Journal of Respiratory and Critical Care Medicine (17.452), Annals of Intensive Care (4.124), Australian Critical Care (2.214), Critical Care (6.407), Critical Care Medicine (7.414), Intensive Care Medicine (17.679), Journal of Critical Care (2.685) and Journal of Intensive Care Medicine (3.142).

Pain Medicine journals (impact factor): Anaesthesia Critical Care and Pain Medicine (2.707), Clinical Journal of Pain (2.893), European Journal of Pain (3.492), Journal of Pain (4.621), Journal of Pain Research (2.386), Journal of Pain and Symptom Management (3.077), Pain (5.483), Pain Medicine (2.513), Pain Physician (3.251), Pain Practice (2.258) and Regional Anesthesia and Pain Medicine (7.015).

We searched the editors’ e-mails either from their PubMed publications (i.e., corresponding author details) or from their departmental or personal web pages. The study was performed between June and October 2020. We created the survey with an online live application survey tool (Survey Monkey)[8] and continued sending the questionnaire repeatedly aiming to receive as many responses as possible until the number of responses was too low. The survey was sent six times before being stopped.

The questionnaire included 11 questions. The first two questions (1, 2) were relevant to the editor’s familiarity with Altmetrics. Questions 3 to 7 were related to the involvement of Altmetrics in the academic world, thus whether scholarly outputs and researches would benefit from obtaining high AAS. Finally, questions 8 to 11 were associated with editors’ opinions about the reliability, manipulations, improvement or abandoning, and possible future of Altmetrics.

The results are presented as a) the numbers of editors retrieved from each journal and the numbers of editors’ e-mails found, b) the numbers and percentages of positive answers for each question per field and overall and c) comparisons of the numbers of positive answers for every single question between the fields.

Normality of distribution was assessed using the Kolmogorov–Smirnov test. Data significantly deviated from a normal distribution (P < 0.001). A statistically significant difference between the numbers of positive answers for every single question between the three fields was determined by the Kruskal–Wallis H test. Post hoc tests were used to detect inter-field differences when appropriate (alpha = 0.05). A statistically significant difference was set at P < 0.05. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) (International Business Machines SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp 2011).

RESULTS

A total of 1,910 editors were identified in the 27 journals included in the study [Table 1]. Eighty-four editors’ e-mails could not be detected. Of the 1,826 detected e-mails, 445 were detected more than once as they belonged to editors included in more than one journal. Responders who reported not being familiar with Altmetrics also answered the 11 questions. All received answers were included in the data analysis.

Table 1

Numbers (%) of editors’ e-mails identified and numbers of overall editors found per journal and per field

	Anaesthesia	Number (%)	Critical Care Medicine	Number (%)	Pain Medicine	Number (%)
1	Acta AnaesthesiologicaScandinavica	23/23 (100%)	American Journal of Critical Care	40/42 (95%)	Anaesthesia Critical Care and Pain Medicine	39/40 (98%)
2	Anaesthesia	47/48 (98%)	American Journal of Respiratory and Critical Care Medicine	49/50 (98%)	Clinical Journal of Pain	51/52 (98%)
3	Anesthesia & Analgesia	148/149 (99%)	Annals of Intensive Care	64/64 (100%)	European Journal of Pain	86/91 (94%)
4	Anesthesiology	59/71 (83%)	Australian Critical Care	29/30 (97%)	Journal of Pain	91/93 (98%)
5	British Journal of Anaesthesia	69/72 (96%)	Critical Care	65/65 (100%)	Journal of Pain Research	36/39 (92%)
6	Canadian Journal of Anesthesia	21/22 (95%)	Critical Care Medicine	119/120 (99%)	Journal of Pain & Symptom Management	79/80 (99%)
7	European Journal of Anaesthesiology	25/25 (100%)	Intensive Care Medicine	147/148 (99%)	Pain	111/111 (100%)
8			Journal of Critical care	65/68 (95%)	Pain Medicine	157/172 (91%)
9			Journal of Intensive Care Medicine	37/41 (90%)	Pain Physician	81/102 (79%)
10					Pain Practice	54/55 (98%)
11					Regional Anesthesia & Pain Medicine	73/76 (96%)
		392/410 (96%)		575/588 (98%)		859/912 (94%)
	Total number (%): 1,826/1,910 (96%)

The views of the editors between the fields did not differ for the questions to be answered, except for the fourth question [Table 2]. In fact, 54% of the editors of the critical care medicine journals believed that Altmetrics should be used for ranking of scientific journals compared to 26% of the editors of pain medicine journals (P = 0.006) and 28% of the editors of anaesthesia journals (P = 0.025) [Table 2].

Table 2

Numbers and percentages (%) of positive answers for each of the 11 questions per field and overall

Questions		Anaesthesia n=35 YES	CCM n=33 YES	Pain Med n=58 YES	Cumulative n=126 YES	χ2 (df=2) P
1	Are you familiar with Altmetrics?	25/35 (71)	27/33 (82)	44/58 (76)	96/126 (76)	1.0 0.604
2	Do you know how the AAS is calculated?	6/35 (17)	10/33 (30)	19/58 (32)	35/126 (28)	2.8 0.250
3	Should Altmetrics replace traditional Bibliometrics?	2/35 (6)	7/33 (21)	6/58 (10)	15/126 (12)	4.1 0.128
4	Should Altmetrics be used for scientific journals’ ranking?	*10/35 (28)	§*18/33 (54)	§15/58 (26)	43/126 (34)	8.3 0.016*
5	Should AAS be used to assess a single scientific output?	16/35 (46)	15/33 (45)	19/58 (33)	50/126 (40)	2.1 0.344
6	Should the cumulative AAS be included in the researcher’s curriculum vitae?	13/35 (37)	17/33 (51)	25/58 (43)	55/126 (44)	1.4 0.490
7	Should a Department’s cumulative AAS be considered to obtain grants?	8/35 (23)	10/33 (30)	10/58 (17)	28/126 (22)	2.1 0.355
8	Is the AAS vulnerable to manipulations?	22/35 (63)	18/33 (54)	38/58 (65)	78/126 (62)	1.1 0.582
9	Do you believe that the AAS has a long time life?	7/35 (20)	10/33 (30)	17/58 (29)	34/126 (27)	1.2 0.549
10	Should AAS be improved?	20/35 (57)	23/33 (70)	33/58 (57)	76/126 (60)	2.2 0.329
11	Should the AAS be abandoned?	4/35 (11)	4/33 (12)	12/58 (21)	20/126 (16)	1.9 0.395

Post hoc analysis for question 4: *P=0.025 Anaesthesia versus Critical Care Medicine, P=0.798 Anaesthesia versus Pain Medicine, §P=0.006 Critical Care Medicine versus Pain Medicine. CCM: Critical Care Medicine, Pain Med: Pain Medicine, AAS: Altmetric Attention Score

DISCUSSION

An overall low percentage of editors responded to the questionnaire with rates of responses similar across the three disciplines. We found no significant differences in the views of editors between the three fields. Seventy-six percent of those who responded were familiar with Altmetrics, and 28% of the respondents knew how the ΑΑS is calculated. We included all given answers in the analysis because data omission is a form of data manipulation, and more specifically data falsification.

Citation counts of an article and journal impact factor are the gold standard for assessing the quality of articles and journals, respectively. Highly cited articles are considered to have an impact on science and mostly produce new knowledge. A number of studies comparing citations with the AAS report a significant positive correlation between these metrics.

Thelwall et al.[9] found strong correlations between citation counts and some AAS posts such as tweets, blog mentions, Facebook wall posts, and mainstream mentions. The correlation between citations and the single component AAS, which represents the weighted counts of all types of sources was not examined.[9] Kunze et al.[10] found citation rates to correlate significantly with the AAS of articles published in five top orthopaedic journals and suggested that AAS may predict citation numbers.

AAS could be used as a predictor of short-term citations counts if Mendeley counts were included. The journal’s impact factor is a stronger predictor of the long-term impact of an article.[11] Web of Science citation counts were found to correlate positively with the AAS of 2,406 articles published in six Public Library of Science (PLOS) journals. These correlations differ across the different disciplines and are more likely to be associated between the high citation counts and high AAS.[3] Investigators invoking these positive correlations show an interest in including cumulative AAS in their curriculum vitae.[12] However, the positive correlations between AAS and citation counts, reported by several studies, do not necessarily mean a cause–effect relationship. Social and mainstream news media are ruled by factors different from those that influence citation counts; therefore, social media cannot substitute citations.[13]

Evaluation regarding scholarly outputs, journal metrics level, and researchers’ performance is related to the quality. The AAS is dominated by the public user, frequently a non-expert, to assess the quality of a single article. The relationship between tweets, the most frequent component in Altmetrics and the F1000Prime peer review post-publication assessments is weak and does not support implementing tweets and AAS for journals or researchers’ scientific evaluation.[14] In another study, Bornmann and Haunschild found a strong relationship between citations and peers’ assessment, but not between the AAS and F1000Prime peers’ assessment.[15] In nine highly cited critical care medicine journals, scholarly outputs with high AAS did not correlate with highly cited ones, indicating that AAS is not a good predictor for citation counts of articles published in these journals.[7]

Sathianathen et al.[6] suggested that Altmetrics could be used as an early measure of a researcher’s work, but their results are derived from high-impact factor journals i.e., the New England of Medicine, the Lancet, Nature, etc. These results may not be applicable to low- or medium-impact journals.

Significantly more editors of the critical care medicine journals favour journal ranking, implementing the AAS compared to editors serving the anaesthesia and pain medicine journals. Fassoulaki et al.[4] reported a significant correlation between AAS and citation counts of five anaesthesia journals. The ranking order by AAS did not follow the order when the five anaesthesia journals were ranked by their traditional impact factors.[4] These data are further strengthened by a recent study proposing an Altmetric impact factor for journal-level metrics of 18 anaesthesia, critical care and pain medicine journals. Ranking of these journals by traditional impact factor does not follow ranking by their Altmetric impact factors.[16] Thus, the impact of Altmetrics when used as an indicator on career prospects, scholarly outputs, and journals’ metric level should not replace the traditional bibliometrics. Nonetheless, including Altmetrics in researchers’ qualifications for hiring, promotions, and grant decisions might be complementary, but not the first-order criterion as public opinion is not an index of quality.

If altmetric.com is used as an indicator of the impact of researchers and outputs in science and journals’ metric level, this should not replace the traditional bibliometrics.

One study summed that traditional impact factor and AAS were found to have a high correlation,[5] whereas another study compared AAS and impact factor, and the authors failed to demonstrate a relationship between these two indicators.[17] A lack of correlation between AAS and traditional impact factors is not surprising as these metrics are calculated by different methodologies, influenced by different factors, and represent different evaluation indices, for scholarly output and authors the first, for journals’ ranking the second.

Citations take a long time to accrue, whereas AAS has a short time lag[18]; therefore, researchers find it convenient to use Altmetrics in their curriculum vitae. In fact, traditional metrics and Altmetrics could be complementary, and both could be implemented into the academic world as metrics.[19] However, it should be noted that Altmetrics and bibliometrics are vulnerable indicators, as they are subjected to manipulations aiming to increase their scores. AAS is the result of public opinion, not dependent on peer review or experts’ opinion. The fact that a large number of journals adopted and include the AAS for each article may even further enhance its manipulation to draw readers’ attention. Flaws, such as manipulations, a short lifetime, lack of peer reviewing and quality assessment, are reasons that many scholars are skeptical about the use of AAS as metrics in the academic context.

The main limitation of the present survey was its response rate (9.12%), which was close but still below the level of 10%. The low response rate may suggest a reluctance of editors to accept metrics derived from public opinion instead of those derived from scholarly outputs and experts’ opinions. The response rates might have been higher if our research had also included journals with lower impact factors and/or was not limited to editors, but addressed to a broader population of researchers. Responders who reported not being familiar with Altmetrics also answered the 11 questions. We included all answers in our data analysis because omitting data is data falsification.

The strengths of this study are that we included editors, who are considered to be experts in bibliometrics. Also, we included a considerable number of high-impact journals covering different but related fields. To our knowledge, this is the first survey addressed to a particular group of scholars (i.e. editors), asking their opinions about the impact of Altmetrics on the academic world.

CONCLUSION

Most of the editors in this survey expressed a positive view for AAS, its use in assessing scholarly output and inclusion in the researchers’ curriculum vitae. A very low percentage of editors believed that the AAS should be abandoned, but the majority proposed that it should be maintained and improved.

Financial support and sponsorship

Departmental funding only.

Conflicts of interest

There are no conflicts of interest.