Correlation Between Altmetric Score and Traditional Bibliometrics in Total Joint Arthroplasty Research.

BACKGROUND: Citation analysis is commonly used to evaluate the impact of academic publications within an area of study. The purpose of this study is to review the publications with the highest Altmetric scores related to total hip arthroplasty (THA) and total knee arthroplasty (TKA) and assess the correlation with traditional bibliometrics.
METHODS: Altmetric scores were obtained for the top articles relating to THA and TKA and organized from highest to lowest scores. The Clarivate Analytics Web of Knowledge database was used to search the top articles by Altmetric score. Articles meeting inclusion criteria were reviewed for various metrics, including number of citations, journal impact factor at the time of publication, and study type and design.
RESULTS: The top THA and TKA publications were cited 3042 times and 7523 times, respectively. The former were published in journals with an average impact factor of 17.861, and the latter in journals with an average impact factor of 15.564. For THA, Altmetric score demonstrated a significant, weakly positive correlation with the number of citations (P = .008). For TKA, Altmetric score was found to have a significant, weakly positive correlation with impact factor of journal (P = .04); however, it was nonsignificant for citation count (P = .11). When one outlier is removed, the correlation with citation count is statistically significant (P = .009).
CONCLUSION: While alternative metrics cannot replace traditional bibliometrics, they may serve a complimentary role in describing the influence of research. Thus, the Altmetric score represents an additional tool to identify the most influential articles to guide learning and evidence-based practice.

Introduction

Traditionally, evaluating scientific research and publishing in terms of quality, distribution, and impact has largely been quantified by bibliometric factors such as article citation number, impact factor of the journal, and the h-index [[1], [2], [3]]. With the recent rise and advancement in digital technology, new metrics have come about to comprehensively assess the utilization and distribution of published research in other ways [2]. While bibliometrics remain the norm in terms of evaluating the long-term impact of a research article, new alternative metrics show promise as means of assessing publication status in a new and possibly more comprehensive light [3].

The Altmetric Attention Score was introduced in 2010 as a tool to evaluate the individual impact of an article, mainly through the attention garnered through online platforms and dissemination [3]. This scoring system was created to complement traditional bibliometrics and tracks the online media presence of an article by measuring and compiling mentions an article receives [3]. This includes Twitter, Facebook, blogs, and various other online media outlets [[2], [3], [4], [5]]. The score reflects how widely an article is “mentioned” in the range of media as described previously, with individual mentions assigned to specific weights. The score is derived from an automated algorithm, representing a weighted count of the amount of attention received [4,6]. Default weightings exist, with news stories and blogs weighing higher than other social media posts, as seen in Table 1 [4,6]. Three main factors are used to calculate the Altmetric score: the volume, sources, and authors [4]. The “volume” of the article increases the score as more people mention it [4]. Each category of mention contributes to the “sources” domain of the score and is graded on a predetermined scale [4]. Finally, the Altmetric score grades the “authors” of each mention by how frequently they discuss scholarly articles [4]. These 3 factors are then combined into a single index [3].

Table 1

Default weights of media platforms for use in Altmetric score calculation [6].

Media platform	Score
News	8
Blog	5
Policy document	3
Patent	3
Wikipedia	3
Twitter (Tweets and Retweets)	1
Peer review (Publons, PubPeer)	1
Weibo (not trackable since 2015, but historical data kept)	1
Google+ (not trackable since 2015, but historical data kept)	1
F1000	1
Syllabi (Open Syllabus)	1
LinkedIn (not trackable since 2013, but historical data kept)	0.25
Facebook (only a curated list of public pages)	0.25
Reddit	0.25
Pinterest (not trackable since 2013, but historical data kept)	0.25
Q&A (Stack Overflow)	0.25
YouTube	0.25
Number of Mendeley Readers	0
Number of Dimensions and Web of Science Citations	0

To our knowledge, there are no prior studies comparing the association between Altmetric score and traditional bibliometric measures within the literature on total hip arthroplasty (THA) or total knee arthroplasty (TKA). Our study sought to determine whether there was any association between the Altmetric score and traditional metrics within this subset of the orthopedic literature.

Materials and methods

As sorted by highest Altmetric scores, the top 100 articles relating to THA and the top 100 articles relating to TKA were obtained and organized from highest to lowest scores. To achieve this, PubMed/Medline and Cochrane Library were searched for articles published using the following search terms: (“Arthroplasty, Replacement, Knee” [Mesh]) OR (Total Knee Arthroplasty) OR (TKA) OR (Total Knee Replacement) OR (TKR) OR (“Arthroplasty, Replacement, Hip” [Mesh]) OR (Total Hip Arthroplasty) OR (THA) OR (Total Hip Replacement) OR (THR). Once a list of the articles organized by Altmetric score were obtained, each individual article was searched using the Clarivate Analytics Web of Knowledge database to obtain bibliometrics. These included impact factor of the journal at year of publication of article, number of citations, and whether or not the article was open access. All listed metrics were retrieved at the same time (March 26, 2020), as these are inherently dynamic measurements. Each article was then analyzed to determine the article type (original, letter to editor, review, clinical practice guidelines or viewpoints). The study type was also determined and assigned a level of evidence as described previously [7]. Articles were excluded if they initially met search criteria but did not relate to THA or TKA.

Statistics for this study were performed with Microsoft Excel, and significant findings were interpreted using a predetermined P value threshold of <0.05. The correlation between the variables studied was determined using Pearson’s correlation coefficient (r), and the coefficient of determination (R2) was used to determine the proportion of variance in the data that were accounted for by the correlations observed. A student’s t-test was performed between the Altmetric attention score and open-access status to assess the relationship. The statistical analysis and methodological procedure were derived from prior research analyzing the correlation between Altmetric score and citations in pediatric surgery and urological literature [2,3].

Results

Total hip arthroplasty

Altmetric scores were recorded for the top 100 articles published with respect to THA. Seventy-five articles met inclusion criteria pertaining specifically to THA. Articles were excluded if they were obtained using the search criteria but did not pertain to the subject studied. Hence, a total number of 75 articles were analyzed for this study. The cumulative total number of citations for all the articles was 3042. These articles were published in journals with an average impact factor of 17.861. The number of citations for each individual article ranged from 0 to 351, with a median of 22. The Altmetric scores for the articles ranged from 118 to 1291, with a median of 235. Altmetric score and number of citations showed a significant, weakly positive correlation (r = 0.302, P = .008) as depicted in Figure 1. The Altmetric score and impact factor of the journal of publication had no significant correlation (r = 0.082, P = .489) as depicted in Figure 2. The coefficient of determination, or R2, was 0.0914 for citation number and 0.0068 for journal impact factor.

Figure 1

Linear regression model for Altmetric Attention score and citation rate in total hip arthroplasty.

Figure 2

Linear regression model for Altmetric Attention score and journal impact factor in total hip arthroplasty.

The majority of the analyzed articles were original articles (85.3%). The breakdown of article type is further shown in Table 2. Retrospective cohort studies were the most common design (36.8%), followed by randomized control trials (16.2%) and prospective cohort studies (10.3%). The research design of the original articles by percentage is depicted in Table 3. There was no significant effect of open access status on Altmetric scores (P = .12). The mean level of evidence was 2.36 (range 1 to 4).

Table 2

Characterization of total hip arthroplasty article type.

Article type	Percentage (%)
Original article	85.3
Letter to editor	1.3
Review	4.0
Clinical practice guidelines	5.3
Viewpoint	1.3
Editorial	1.3

Table 3

Characterization of total hip arthroplasty study type.

Study type	Percentage (%)
Retrospective cohort	36.8
Randomized control trial	16.2
Cross-sectional/survey	8.8
Prospective cohort	10.3
Case control	4.4
Case report/series	2.9
Nonrandomized control trial	5.9
Basic science	5.9
Systematic review	8.8

Total knee arthroplasty

Altmetric scores were recorded for the top 100 articles published with respect to TKA. Ninety-seven articles met inclusion criteria pertaining specifically to TKA. The cumulative number of citations for all the articles was 7523. These articles were published in journals with an average impact factor of 15.564. The number of citations for each individual article ranged from 0 to 3,680, with a median of 13.5. The Altmetric scores for the articles ranged from 73 to 1291, with a median of 141. Altmetric score and number of citations showed a nonsignificant, weakly positive correlation (r = 0.16, P = .10) as depicted in Figure 3a. After excluding the outlier, a statistically significant, weakly positive correlation was delineated (r = 0.27, P = .009), shown in Figure 3b. Altmetric score and journal impact factor additionally had a statistically significant but weak positive correlation (r = 0.21, P = .043) as depicted in Figure 4. The coefficient of determination, denoted R2, was 0.0264 for citation number and 0.0428 for journal impact factor.

Figure 3

Linear regression model for Altmetric Attention score and citation rate in total knee arthroplasty for (a) all data and (b) data excluding outlier.

Figure 4

Linear regression model for Altmetric Attention score and journal impact factor in total knee arthroplasty.

The majority of the articles for TKA were original articles (91.5%). The breakdown of article type is further shown in Table 4. Retrospective cohort studies were the most common design (41.3%), followed by randomized control trials (18.5%) and prospective cohort (9.8%). The research design of the original articles by percentage is depicted in Table 5. There was no significant effect of open access status on Altmetric score (P = .29). The mean level of evidence was 2.28 (range 1 to 4).

Table 4

Characterization of total knee arthroplasty article type.

Article type	Percentage (%)
Original article	91.5
Letter to editor	1.1
Review	3.2
Clinical practice guidelines	1.1
Viewpoint	1.1
Editorial	2.0

Table 5

Characterization of total knee arthroplasty study type.

Study type	Percentage (%)
Retrospective cohort	41.3
Randomized control trial	18.5
Cross-sectional/survey	7.6
Prospective cohort	9.8
Case control	3.3
Case report/series	1.1
Nonrandomized control trial	5.4
Basic science	1.1
Systematic review	7.5
Quality improvement	1.1
Review article	3.3

Discussion

The results of traditional bibliometrics have correlated well with indicators of research quality, such as external funding, scientific prizes, and merits within the scientific community; however, limitations do exist [8]. Conventional citations are slow to accumulate, precluding any meaningful evaluation of impact based on citations within a short time frame [9]. Furthermore, conventional bibliographic measures that can be obtained early, such as journal impact factor, measure impact among one specific type of audience and do not necessarily reflect broader dissemination over time or predict ultimate number of citations [9]. As such, it is commonplace to wait 2 to 3 years from the time of publication to reliably assess the impact of the publication, as studies have observed that this time period is necessary for an article to reach its peak citation count [3,9].

Given these limitations, analyses that go beyond the traditional bibliometric measures, such as Altmetrics, may provide a major complementary benefit by allowing the orthopedic community to measure real-time dissemination of new literature [1,2]. When used in conjunction with traditional metrics, Altmetrics may also assist in both fully understanding the reach of published literature and in identifying impactful articles at an earlier time point [2]. Prior literature has confirmed this complementary function between Altmetrics and traditional bibliometrics within other disciplines [2,3,10,11]. In addition, the score can be useful to journals, publishers, and academic institutions in analyzing the breadth of dissemination of an article, which can strengthen the case for grant applications [1].

To the authors’ knowledge, this is the first study to provide an in-depth statistical analysis of the relationship between Altmetrics analysis and traditional bibliometrics within the orthopedic literature pertaining to THA and TKA. This study demonstrated a significant, weakly positive correlation between Altmetric score and citation number for articles for THA, as well as a significant correlation between Altmetric score and journal impact factor for articles on TKA. After an outlier was removed, the study also demonstrated a significant, weakly positive correlation between Altmetric score and citation number for articles on TKA. Other studies outside the field of orthopedic surgery have had similar findings. Nocera et al. examined the correlation of Altmetric scores with citation count and journal impact factor in urologic literature [2]. They demonstrated that Altmetric scores and citation number showed a weakly positive correlation (r = 0.164) but also that there was no significant relationship between Altmetric score and journal impact factor (r = 0.005) [2]. Similarly, in a study by Chang et al., a weak but positive correlation with number of citations was demonstrated when examining the pediatric surgery literature (r = 0.236) [3]. Taken together, the minimal coefficients of determination within these studies and the present analysis suggest that Altmetrics and traditional bibliometric measures should be used in a complementary manner to determine the overall influence of articles within the orthopedic literature [2].

A recent article similarly demonstrated that the Altmetric score of a publication demonstrated a significant, but weak positive correlation with the citation rate within orthopedic literature [12]. Similarly, Zhang and Earp describe the positive relationship between social media posts and academic citations in recent orthopedic research [13]. This suggests that the Altmetric score could potentially resemble a quantifiable assessment of the publication’s impact and its ability to garner attention within the scientific community [12]. The slight positive correlation with citation count within the THA literature could represent an articles ability to attract more citations and readers through its social media dissemination. Similarly, the growth of social media in the scientific world has also been described [14]. The number of physicians using social media for professional purposes has reached upwards of 90%, allowing for sharing and dissemination of scientific literature [12,14]. In addition, many academic journals have adopted social media profiles to help share and distribute publications, contributing the ever-growing presence of social media in scientific literature [13].

Limitations do exist when using Altmetrics to assess the value of published literature. In comparison to basic science research, clinically oriented publications, such as editorials, tend to have poor correlations with citation counts [15]. The relative weight of significantly heterogenous data that various social media platforms portray is also a concern. As an example, the Altmetric score counts all tweets as equal in weight, and all Facebook content, such as “likes” or “comments,” are weighted equally to “posts.” Some have argued that an initial tweet or Facebook “post” is of more value than a retweet or “like” and that this scoring algorithm does not fully account for this [3]. The use of Altmetrics is further limited by those who have access to and engage with these social media platforms. While the consumers of these types of media are likely to expand over time, the contributions assessed by this measure may not have the ability to replace the validity of an expert’s independent ability to critique the primary literature, as would be captured by measuring citations within peer reviewed literature [16]. In theory, authors or institutions could delegate individuals to help increase social media awareness for certain articles. We were not able to distinguish if articles with higher Altmetric scores received attention because of their own authors or institutions advertising these articles vs those receiving attention through public dissemination. Finally, the Altmetric score is a dynamic entity, accumulating with time. Thus, the findings of the study could be confounded by the year of publication. However, Zhang and Earp describe that while number of academic citations tend to accrue over time, the temporal relationship was not demonstrated with social media posts in their study, describing the concept that social media dissemination of research occurs more rapidly after publication [13]. In recognition of these limitations, previously authors have suggested that it is important to consider Altmetrics as a complement, rather than a substitute, to traditional bibliometric measures [1,3,17].

Conclusion

Alternative bibliographic metrics, such as Altmetrics, are new and rapidly evolving tools. Our study demonstrated that the Altmetric score is only weakly correlated with citation counts in the arthroplasty literature for THA and TKA articles and has no correlation with impact factor of the journal of publication for THA articles but does have a weak correlation with the impact factor of the journal of publication for TKA articles. While there is limited information regarding the concrete relationship between traditional citation counts and novel metrics in many fields, our study demonstrates that attention within social media does appear to have some association with the more traditional measures of impact within the orthopedic literature on THA and TKA. While alternative metrics should not represent a replacement of traditional bibliographic metrics, they may serve a useful, complimentary role in describing the influence of research. Thus, with the continual expansion of the influence of social media, the use of the Altmetric score can serve as an additional tool to identify the most influential articles on total joint arthroplasty to guide learning and evidence-based practice.

Conflict of interest

The authors declare there are no conflicts of interest.