Examining the Relationship between Altmetric Score and Traditional Bibliometrics in the Pathology Literature.

BACKGROUND: Recently, research data are increasingly shared through social media and other digital platforms. Traditionally, the influence of a scientific article has been assessed by the publishing journal's impact factor (IF) and its citation count. The Altmetric scoring system, a new bibliometric that integrates research "mentions" over digital media platforms, has emerged as a metric of online research distribution. The aim of this study was to explore the relationship of the Altmetric Score with IF and citation number within the pathology literature.
METHODS: Citation count and Altmetric scores were obtained from the top 10 most-cited articles from the 15 pathology journals with the highest IF for 2013 and 2016. These variables were analyzed and correlated with each other, as well as the age of the publishing journal's Twitter account.
RESULTS: Three hundred articles were examined from the two cohorts. The total citation count of the articles decreased from 21,043 (2013) to 14,679 (2016), while the total Altmetric score increased from 830 (2013) to 4066 (2016). In 2013, Altmetric score weakly correlated with citation number (r = 0.284, P < 0.001) but not with journal IF (r = 0.024, P = 0.771). In 2016, there was strong correlation between citation count and Altmetric Score (r = 0.714, P < 0.0001) but not the IF (r = 0.0442, P = 0.591). Twitter was the single most important contributor to the Altmetric score; however, the age of the Twitter account was not associated with citation number nor Altmetric score.
CONCLUSIONS: In the pathology literature studied, the Altmetric score correlates with article citation count, suggesting that the Altmetric score and conventional bibliometrics can be treated as complementary metrics. Given the trend towards increasing use of social media, additional investigation is warranted to evaluate the evolving role of social media metrics to assess the dissemination and impact of scientific findings in the field of pathology. Copyright:

INTRODUCTION

The impact of scientific research has traditionally been assessed by citation number and the publishing journal's impact factor (IF), where IF is defined as the total number of citations within 1 year of all articles published in a journal in the preceding 2 years, divided by the number of articles published in the same 2 years.[1] Social media platforms (such as Twitter, Facebook, blogs, Reddit, and Google+) have provided researchers with a variety of new opportunities for the dissemination and discussion of their work. The near real-time spread of information as it is published online makes social media an attractive mode of delivery for investigators interested in quickly sharing the results of their findings.[2] Investigators within the field of pathology have begun to promote social media use in academic medicine and have noted it to be beneficial for use at academic meetings as well as in recruitment and education.[34]

With the continuing advancements in digital communication and social media for the dissemination of scientific research findings, alternative metrics have been developed to provide a more holistic assessment of their distribution and uptake. In 2010, Altmetric created a novel scoring system (“Altmetric score”) to be used in the evaluation of research dissemination, which integrates the number of “mentions” on various sharing platforms such as Twitter, Facebook, blogs, policy sources, news outlets, Wikipedia, Reddit, online videos, patents, Google+, and research highlight platforms.[567] Each of these media is prescribed specific weights and tallied by the Altmetric tool to provide the overall score as well as a score breakdown by the platform.[7] The Altmetric score is not specific to any one topic or discipline, and a higher overall Altmetric score predicts greater social media distribution. In other fields, Altmetric scores have been more frequently associated with higher citation numbers and less frequently correlated with IF.[89101112] The correlation of the Altmetric score with other, more traditional bibliometrics within the pathology literature is unknown. The aim of this study was to explore the relationship of the Altmetric Score with IF and citation number within the pathology literature.

METHODS

The ISI Web of Knowledge Journal Citation Reports was searched using “Pathology” as the search category, and the top 15 journals based on IF in 2013 were identified.[13] The following journals were included: Annual Review of Pathology-Mechanisms of Disease, Acta Neuropathologica, Journal Of Pathology, Seminars in Immunopathology, Modern Pathology, Disease Models and Mechanisms, Neuropathology and Applied Neurobiology, American Journal of Pathology, American Journal of Surgical Pathology, Journal of Neuropathology and Experimental Neurology, Brain Pathology, Expert Review of Molecular Diagnostics, Journal of Molecular Diagnostics, Laboratory Investigation, and Cancer Cytopathology. The top 10 most-cited articles within the time frame of January 2013 to December 2013 were then identified for each of the 15 chosen journals, using Elsevier's Scopus database.[14] The same procedure was applied to identify the top 10 most-cited articles in 2016 for each of the aforementioned 15 journals to elucidate changes over time in the Altmetric scoring algorithm, shifts in the utilization of various social media platforms as communicators of research, and changes in patterns of scientific research dissemination in the digital medium as a whole. The time points of 2013 and 2016 were chosen as they are within the scope of the “modern social media age” while also allowing sufficient time for citation count equilibration and the accrual of meaningful citation data.[2] Both Scopus and Journal Citation Reports take citation number into account when evaluating article dissemination, so Scopus alone was used to determine the top articles of each journal within each 1-year period since it includes a more inclusive list of articles. Internet searches were performed to determine whether each of the 15 journals maintains a Twitter profile, and the date of the establishment of each Twitter profile as displayed on the Twitter account page was recorded. Twitter profiles were only included in this study if they were dedicated specifically to the journal in question; profiles were not included if they were maintained by a larger publishing body. For example, Annual Reviews maintains a Twitter profile, but Annual Review of Pathology-Mechanisms of Disease does not. In this case, no profile was recorded. The Altmetric bookmarklet tool was used to attain an Altmetric score for each of the 300 individual articles examined.[15] Statistics for this study were performed using Microsoft Excel Version 2004 (Microsoft Corporation, Redmond, WA, USA), and P value threshold of <0.05 was used when determining significance. Pearson's correlation coefficient (r) was used to determine the correlation between variables studied, and coefficient of determination (R2) was used to determine the proportion of the data variance that was accounted for by the determined correlations.

RESULTS

The 150 articles included in the analysis for the 2013 cohort accrued 21,043 total citations (median 115, range 38–947) [Table 1]. These articles had a combined Altmetric score of 830 with a median of 3 (range 0–41). There was a significant positive correlation between citation number for an individual article and journal IF in 2013 (r = 0.228, R2 = 0.052, P = 0.005). While Altmetric score was weakly correlated with citation count (r = 0.284, R2 = 0.081, P < 0.001), it was not correlated with IF (r = 0.024, R2 = 0.0006, P = 0.771) overall. When analyzed individually, only one journal (American Journal of Surgical Pathology) in the 2013 cohort had a strong correlation between the Altmetric score and article citation count [Table 2].

Table 1

Summary data of journals included in analysis

Journal	Median (range)				Journal impact factor (2013)	Journal impact factor (2016)	Age of Twitter account (years)
	Citations per article (2013)	Altmetrics score per article (2013)	Citations per article (2016)	Altmetrics score per article (2016)
Annual Review of Pathology-Mechanisms of Disease	178 (99-303)	4 (0-18)	66.5 (52-164)	2 (0-130)	22.128	26.853	N/A
Acta Neuropathologica	203 (181-245)	6.5 (1-24)	136 (102-3842)	19 (1-676)	9.777	12.213	N/A
Journal of Pathology	170.5 (116-947)	4.5 (1-21)	63 (59-91)	4.5 (1-14)	7.33	6.894	9.83
Seminars in Immunopathology	136 (79-243)	0 (0-21)	53.5 (44-73)	2.5 (0-7)	6.482	5.296	N/A
Modern Pathology	108.5 (88-309)	4 (1-16)	77.5 (59-204)	11.5 (3-22)	6.364	5.728	3.25
Disease Models and Mechanisms	98 (76-459)	7 (1-33)	43.5 (30-104)	19.5 (1-79)	5.537	4.691	10.58
Neuropathology and Applied Neurobiology	101.5 (38-406)	4.5 (0-25)	50 (25-119)	11.5 (2-101)	4.97	5.347	6.5
American Journal of Pathology	113 (102-222)	1 (0-5)	48 (39-291)	8.5 (1-228)	4.602	4.057	6.92
American Journal of Surgical Pathology	129 (110-592)	4 (1-32)	77.5 (70-840)	9 (3-51)	4.592	5.363	8.83
Journal of Neuropathology and Experimental Neurology	69 (57-114)	1 (0-7)	36 (29-68)	8.5 (1-30)	4.372	3.503	N/A
Brain Pathology	85.5 (59-172)	0 (0-18)	52.5 (39-116)	10.5 (1-19)	4.354	6.624	0.25
Expert Review of Molecular Diagnostics	52.5 (40-94)	1 (0-5)	33.5 (23-56)	1.5 (1-46)	4.27	3.1	N/A
Journal of Molecular Diagnostics	114.5 (103-317)	3.5 (0-41)	40 (33-63)	27 (0-421)	3.955	4.526	6.92
Laboratory Investigation	68 (54-211)	0 (0-27)	48 (28-83)	8.5 (1-62)	3.828	4.857	3.25
Cancer Cytopathology	66 (56-83)	4 (0-17)	49 (30-152)	16.5 (4-39)	3.807	3.818	7.5
Summary	115 (38-947)	3 (0-41)	58 (23-3842)	7 (0-676)	6.425	6.858	6.38

N/A: Not available

Table 2

Impact of Twitter account age on Altmetric-citation count correlation, by journal

Journal	Year Twitter created	Correlation coefficient between number of citations and Altmetric score (2013)	P (2013)	Correlation coefficient between number of citations and Altmetric score (2016)	P (2016)
Annual Review of Pathology-Mechanisms of Disease	N/A	−0.306	0.390	−0.142	0.696
Acta Neuropathologica	N/A	−0.270	0.451	0.931	<0.0001
Journal of Pathology	2010	−0.0176	0.962	−0.121	0.739
Seminars in Immunopathology	N/A	0.242	0.500	−0.489	0.152
Modern Pathology	2017	−0.220	0.542	0.0474	0.896
Disease Models and Mechanisms	2009	0.488	0.152	−0.0772	0.832
Neuropathology and Applied Neurobiology	2013	0.361	0.305	0.270	0.451
American Journal of Pathology	2013	−0.138	0.704	−0.226	0.531
American Journal of Surgical Pathology	2011	0.966	<0.0001	0.586	0.0750
Journal of Neuropathology and Experimental Neurology	N/A	−0.141	0.698	−0.205	0.569
Brain Pathology	2020	0.194	0.592	−0.542	0.105
Expert Review of Molecular Diagnostics	N/A	0.154	0.672	0.232	0.520
Journal of Molecular Diagnostics	2013	0.284	0.426	0.755	0.0116
Laboratory Investigation	2017	0.132	0.716	−0.148	0.683
Cancer Cytopathology	2012	0.0993	0.785	0.0147	0.968

N/A: Not available

The same analysis was performed for the 2016 cohort to have a point of comparison across time for Altmetric scores in the pathology literature [Table 1]. The 2016 cohort of 150 articles accumulated 14,679 total citations (median 58, range 23–3842). The Altmetric score was 4066 with a median of 7 (range 0–676). This was approximately 4.9 times more than the cumulative Altmetric score in 2013.

For the 2016 cohort including all data points, there was no significant correlation identified between article citation count and journal IF (r = 0.111, R2 = 0.0124, P = 0.175). There was a significant correlation between citation count and Altmetric score including all data points (r = 0.714, R2 = 0.510, P < 0.0001). There was one outlier in the 2016 article cohort with 3842 citations and an Altmetric score of 676. The article was published in Acta Neuropathologica and detailed the World Health Organization's classification of central nervous system tumors. Interestingly, when this article was removed from the dataset, this relationship did not remain significant (r = 0.114, R2 = 0.168, P = 0.168). Previous bibliometric analyses have excluded outliers when there was an obvious mismatch (extremely high Altmetric with low citation count, or vice versa), but we decided to include this point in our discussion as the article in question scored highly in both metrics.[16] Comparing IF to Altmetric score, neither comparison with (r = 0.0442, R2 = 0.002, P = 0.591) or without the outlier (r = 0.0153, R2 = 0.0002, P = 0.853) revealed a significant correlation. When analyzed individually, two of the journals in 2016 (Acta Neuropathologica and Journal of Molecular Diagnostics) had significant positive correlations between article Altmetric score and citation number [Table 2].

In analyzing the individual components contributing to the total Altmetric score, Twitter accounted for the highest number of mentions in both 2013 and 2016. In 2013, the next highest number of mentions was on Facebook and in patents, while in 2016, news outlets then Facebook were second and third, respectively [Table 3]. Additional analysis was performed in order to evaluate the relationship between the age of the Twitter account, Altmetric score, and citation number. Two-thirds of the pathology journals evaluated had a journal-specific Twitter account at the time the study was conducted [Table 1]. The average age of these accounts was 6.38 years, with the oldest account belonging to Disease Models and Mechanisms (10.58 years) and the youngest account belonging to Brain Pathology (0.25 years). However, analysis of the age of the Twitter accounts and the relationship between Altmetric score and citation number did not reveal a significant correlation in either 2013 (r = 0.368, R2 = 0.135, P = 0.296) or 2016 (r = 0.413, R2 = 0.171, P = 0.235).

Table 3

Pathology article mentions in 2013 and 2016 by social media platform

Source	Article “mentions” in 2013	Article “mentions” in 2016	Percentage change
Twitter	464	1856	300
Facebook	55	149	170.9
Blog	17	33	94.1
Policy Source	8	5	−37.5
News Outlets	26	333	1180.8
Wikipedia	16	14	−12.5
Reddit	2	4	100
Videos	4	2	−50
Patents	52	11	−78.8
Google+	6	8	33.3
Research highlight platforms	6	4	−33.3

CONCLUSIONS

Dissemination of scientific findings through social media is a new reality; our study sought to explore the relationship of increased social media “mentions”, as measured by the Altmetric score, with more traditional bibliometrics such as IF and citation number within the pathology literature. Overall, we found a correlation between the Altmetric score and the citation number, but not the IF. It is also worth noting that the relatively low coefficients of determination suggest that the Altmetric score measures dissemination and impact in a way that cannot be directly substituted by citation count.

These patterns have also been described in other areas of medicine, with trends observed between citation number and Altmetric score in urology and pediatric surgery literature. Interestingly, while the literature of these two fields did not show a significant association between IF and Altmetric score in their earlier cohort, there was a significant positive correlation between these two metrics in the more recent group of pediatric surgery and urology articles.[89] This did not occur in the pathology literature examined, in which there was no correlation between the Altmetric score and IF in either time cohort. This difference suggests that there are variances in Altmetric characteristics and traditional metrics attributable to field type as well as the journal, and it is likely that there are other factors or approaches to dissemination that vary from journal to journal that this study did not specifically investigate.

The evolution of various social media platforms continues to affect how research is seen and by whom it is seen. Researchers are now more than ever able to disseminate information directly through digital means, and this trend is readily visualized when comparing the summed Altmetric scores of the articles included in this study. Although the total Altmetric score increased by a factor of 4.9 between 2013 and 2016 in our study, and Twitter was the highest contributor to the Altmetric score, we also found that the maturity of the Twitter account was not correlated with the Altmetric score. Our findings suggest that users approach Twitter research sharing differently, and there are other factors involved in the relationship between Altmetric score and citation number. As pathologists continue to embrace Twitter as a way to share research findings, the quantity of Twitter mentions integrated in the Altmetric score provides a strong opportunity for measurement and tracking.[34]

Previous studies have demonstrated that early citation number within up to the first 3 years following publication is a poor predictor of citation number after several years.[217] Although Altmetric and traditional bibliometrics are both quantitative methods for measuring research dissemination, this finding provides another potential niche where Altmetrics could be used, as suggested in prior analyses of the urologic and pediatric surgery literature.[89] Given the correlation between article citation count and Altmetric score, it is possible that Altmetric score could be used to predict the eventual citation count of articles within the early time frame, facilitating more rapid prediction of article dissemination. However, more investigation is needed to further assess this finding and the precise role of the Altmetric score.

Since research can be disseminated at ever-increasing speeds through digital platforms, Altmetric's direct sampling of such platforms makes the Altmetric score especially useful during the 1st year after an article's publication. As digital sharing trends continue to change, as evidenced by the increased number of research Twitter mentions and shift in distribution preference from Facebook to news outlets between 2013 and 2016, Altmetric will be able to account for these changes in its algorithm without the lag time seen in more traditional methods.

Our study has several limitations. First, this study solely analyzed the influence of Twitter account age at the time of the writing of this study on article metric correlations due to Twitter's dominance in usage for the spread of scientific research. While Twitter mentions were used as a surrogate for social media presence in this study, it may be true that there are noteworthy associations between other social media platform mentions and future citations. Likewise, there are other alternative metrics that track Twitter data. Our focus on the Altmetric score alone may exclude trends in Twitter that may be visible with other services. Second, our study only analyzed the top 10 most cited articles in each of the journals examined, and the top 15 pathology journals based on traditional measures of impact. It is unknown whether the correlation between Altmetric score and citation count would hold among other, less visible journals or if there would be any new findings regarding social media visibility as measured by Altmetric score and IF. Likewise, it is possible that the results would change outside of these high-IF pathology journals or in journals focusing on specific areas within pathology that are not well represented by our methodology. Differences in the level of webpage visibility and in the case of paid or membership-only journals, universality of access to the articles are also potential limitations that are not assessed in this study. Third, we reviewed two cohorts of articles separated by 3-years' time in this study. As social media usage for scientific purposes is so rapidly evolving, it is difficult to predict when any particular article will approach its near-terminal citation number, so therefore, it is unclear if correlations between the metrics studied would have been more or less robust at different time intervals. Finally, we accounted only for Twitter account age and did not include any measures of account activity. Prior studies have demonstrated a significant correlation between Twitter account activity, as measured by number of “tweets” within a given time period or since account creation, and citation number.[81819] It is also worth noting that other Twitter accounts not operated by the publishing journal may contribute a large portion of an article's Twitter activity. Both of these caveats provide interesting routes for future analysis in a study focused more specifically on Twitter use in the realm of pathology literature.

There remain widespread concerns among investigators about the shortcomings of social media as a method of sharing scientific research.[820] As Twitter, Facebook, and other “mentions” do not go through the same rigorous review processes inherent to peer-reviewed journals, the objectivity and background of the data presented often cannot be evaluated.[21] In addition, all “mentions” are not created equal, even within a single social media platform, and a limitation of this study is the lack of the ability to assess the users of the social media platforms. In a similar method to how journals may encourage self-citation to drive up IF, it is possible that investigators or health-care marketing interests eager to achieve publicity through Altmetric score could “game” the metric by sharing and encouraging associates to share their own work via social media.[202223] Information often risks losing context and expertise as it is distributed further from its primary source.[24] These are issues inherent to social media that the Altmetric score cannot as of yet measure, and the lack of enforced reliability inherent in the measures that make up the score have called into question its value as a bibliometric.[8917]

In summary, we find a correlation between one metric of social media dissemination (Altmetric score), and article citation, but not IF. Thus, while the Altmetric scoring system does not sufficiently overlap with traditional bibliometrics to provide a single replacement metric, there is value to Altmetric as a complimentary method of evaluation. Although it remains unclear whether increased social media exposure leads to increased citation count or if the same factors that lead to a high citation count also lead to high social media visibility, social media has an effect on research uptake regardless of the directionality of the data. More research into the usage and importance of the Altmetric scoring system and other metrics of social media dissemination will amplify our understanding of the evolving impact of this medium as pathology continues to use new online venues to disseminate scientific findings.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.