Predicting citation impact from altmetric attention in clinical and translational research: Do big splashes lead to ripple effects?

Publications are commonly used to evaluate the productivity and impact of research programs. Traditional metrics examine publication impact through slowly accumulating academic citations. "Altmetrics" are a new way to describe early publication influence in nonacademic media/community spheres (news, tweets, and blogs). Articles with significant altmetric attention make a big splash of immediate impact, whereas papers with high rates of academic citation reflect ripple effects of influence over time. Past research has found weak associations between altmetrics and academic citations. However, no previous research has focused on clinical/translational research, which aims to translate scientific discoveries to public use. Further, no previous research has assessed the relationship between altmetrics and modern citation impact factors like the National Institutes of Health (NIH)'s Relative Citation Ratio (RCR). It is also unclear whether publication in journals with higher journal impact factors (JIFs) may drive both public attention and academic impact. We investigated whether early altmetric indicators of splash predict citation ripples, beyond the effect of the JIF. For a portfolio of 2188 publications supported by the NIH's Georgia Clinical and Translational Science Alliance from 2007-2020, we collected 2020 Altmetric Attention Scores (AAS), 2020 JIFs, and 2021 RCRs. All three were significantly correlated with one another. Regression analyses revealed that AAS significantly predicts later RCR, controlling for JIF and publication year. Findings indicate that in clinical/translational science, articles that make a big splash of altmetric attention have ripple effects through increased citation influence, which is not entirely due to publication in higher impact journals. Altmetric attention may be a useful early indicator of eventual influence and potential for translational advancement.

WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Past research, in many fields of research, has found weak associations between publication altmetrics and raw academic citation count. It unclear whether any association is due to publication in higher impact journals.

WHAT QUESTION DID THIS STUDY ADDRESS?

We investigated whether, within clinical/translation publications, early altmetric scores predict later citation influence (using the Relative Citation Ratio, a citation impact factor adjusted for field and time since publication), beyond the effect of the journal’s impact factor.

WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Using regression analyses, we found that in clinical/translational science, articles that make a big splash of altmetric attention have ripple effects through increased citation influence, which is not entirely due to publication in higher impact journals.

HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

Altmetric attention may be a useful early tool for gauging the eventual influence and potential for translational advancement of published clinical/translational research.

INTRODUCTION

One way to evaluate the productivity and impact of a research program is through bibliometrics, or analysis of their supported publication portfolios. Bibliometrics describe a pivotal early step in the process of bringing new scientific discoveries to clinical use. Traditional publication impact analyses usually rely on examining academic citations and indices, such as an investigator’s H‐index, which can be important for moving research forward toward eventual translation. , , , , However, academic citation metrics are limited in that they take considerable time to accumulate and reflect influence within scholarly circles only. Altmetrics, on the other hand, are a new, alternative type of bibliometrics that describe the influence of published research in nonacademic spheres, such as media and community discourse (e.g., news articles, blogs, and Twitter), technological patents, and public health policy. Publications with early altmetric attention can be thought of as big splashes with immediately measurable impact in society, whereas papers with high rates of academic citation reflect ripple effects of downstream, accumulated influence within academic fields over time. Taken together, academic citations and altmetric citations can illustrate how publications are “making waves” of social and academic impact that can advance a line of research.

Do big splashes lead to the ripple effects that drive research forward? Articles that elicit early nonacademic attention may be more likely to be noticed by other researchers, which could increase the likelihood that the research will be applied and cited in subsequent research. Indeed, past research in numerous fields has investigated the relationship between altmetric scores and citation counts and found little to no association. , , , , However, no previous research has focused on this relationship in clinical/translational research, which aims to advance discoveries across boundaries from basic science to public use. Dissemination across academic silos and across phases of the translational spectrum is expected to facilitate the interdisciplinary cross‐pollination that is key to translational goals. Thus, altmetric attention that spreads findings beyond the immediate scholarly circle may engender more academic citations among researchers in adjacent fields, clinician‐scientists, and public/community health researchers. Moreover, no previous research has examined the relationship between altmetrics and modern, adjusted citation impact factors like the National Institutes for Health’s (NIH) Relative Citation Ratio (RCR), which is a more sophisticated and accurate measure of citation influence that accounts for field and time since publication. It is also unclear whether any relationship between altmetric attention and citation is simply due to publication in journals with higher journal impact factors (JIFs), which, by definition, are more likely to be read and cited and may drive both public attention and academic impact. In this study, we investigate whether early altmetric indicators of splash predict citation ripple effects, beyond the effect of the journal’s citation impact factor.

METHODS

The Georgia Clinical and Translational Science Alliance (Georgia CTSA) was established by the National Center for Advancing Translational Science (NCATS) of the NIH in 2007 to accelerate clinical and translational education, research, and community engagement to impact health in Georgia and beyond. , The alliance consists of a cross‐institutional collaboration among Emory University, Morehouse School of Medicine, Georgia Institute of Technology, and the University of Georgia, with a collection of interconnected programs charged with supporting and providing relevant services to clinical and translational investigators within those institutions. Researchers who receive support and services are asked to cite the Georgia CTSA grant when publishing resulting research. In mid‐2020, we identified all publications formally acknowledging the Georgia CTSA as having provided support via a PubMed query of all past and present Georgia CTSA NIH grant project numbers (including UL1, KL2, and TL1 grants).

To retrieve JIF values, the list of supported publications was searched in Web of Science’s InCites application at the end of 2020. InCites’ proprietary JIF is an unadjusted measure of typical citation rates for the journals in which each article was published over the previous 2 years. A JIF value of five, for example, means that articles published in that journal in the past 2 years were cited an average of five times.

To retrieve altmetric information, the list of supported publications was searched in Digital Science’s Altmetric Explorer application in mid‐2020. Data included an Altmetric Attention Score (AAS), a combined, rank order index score that uses an algorithm to quantify media and community attention paid to publications, and use of the articles in public documents. Specific components of the AAS reflected in this dataset include references to publications in: news articles, blog posts, Twitter posts, Wikipedia pages, patent applications, government and NGO policy documents, Faculty Opinions (formerly F1000 Prime) peer faculty recommendations, public Facebook posts, and legacy Google+ posts. Although they did not appear in our dataset, the AAS also includes references to publications in Publons, Pubpeer, Weibo, Open Syllabus, LinkedIn, Reddit, Stack Overflow, YouTube, and Pinterest. References vary in their weighted influence on the AAS, depending on the relative reach of the outlet (e.g., news mentions are given the highest weighting when calculating AAS, whereas Facebook and YouTube mentions are given among the lowest weighting).

To retrieve RCR values, the list of supported publications was searched in the NIH Office of Portfolio Analysis’ iCite application in September 2021. The RCR is a field‐normalized metric that calculates the citation impact of an article relative to the average NIH‐funded article in its co‐citation network. RCR data are only available for articles that are at least one calendar year old, which allows time for the accrual of sufficient co‐citation network data. We also collected the iCite total citation counts as of September 2021 for these articles for comparison with the RCR.

The final dataset included 2188 publications with complete data to be included in analyses. To contextualize the publication portfolio we calculated aggregate mean, median, maximum, and totals for journal‐ and article‐level metrics and describe sources for altmetric attention. To test the hypothesis that altmetric attention is predictive of citation influence independent of the effect of the journal’s impact, we first conducted preliminary correlation analyses among the JIF, AAS, RCR, and total citations. We also included correlations with publication year because our data includes articles published across 13 years and because time since publication is related to citation accumulation and may also be related to evolving altmetric activity and tracking over time. We assessed relative effect sizes and statistical significances of the Spearman’s rho rank order correlation coefficients. We then proceeded to conduct hierarchical multiple regression analyses with the JIF as an independent variable, and publication year as a control variable predicting the dependent variable RCR in step 1 of the model. We next added the AAS as a separate independent variable predicting RCR in step 2. We assessed the effect sizes and statistical significances of the standardized regression coefficients (β) for the predictors of the RCR and the total and change in variance explained (Δ R 2) at each step. All analyses were carried out in SPSS version 27.

RESULTS

The portfolio of 2188 Georgia CTSA‐supported publications indexed across PubMed, InCites, Altmetric Explorer, and iCite were published from 2007 to 2020. They appeared in 734 different journals across clinical/translational science fields, with a mean JIF of 7.2. Some of the most frequent journals included PLoS One, Pediatrics, and the Journal of Acquired Immune Deficiency Syndromes. Of these, 1902 (87%) articles had at least one altmetric reference and 2156 (99%) had at least one academic citation. The articles accumulated over 31,000 references in altmetric sources as of 2020, for an average of over 14 references per article (median = 4) and a maximum of 857 references to one article. The average AAS value was 17.1 (median = 3), ranging from 0 to 654. The most common sources for altmetric references were Twitter posts and news stories. Some of the most frequent sources include the Twitter account HIV_Insight, the news outlet MedicalXpress, the blog Physician’s Weekly, and the policy source the World Health Organization. These articles accumulated almost 120,000 academic citations as of 2021, for an average of over 55 citations per article (median = 22) and maximum of 4823 to one article. The average RCR value of 2.8 (median = 1.4) means that these articles have been cited an average of 2.8 times as often as other articles from the same years and fields.

Big splashes lead to ripple effects

We investigated whether articles in the Georgia CTSA‐supported portfolio that make a big splash of public altmetric attention have ripple effects through increased academic citation influence. Table 1 presents a Spearman’s rank correlation matrix for the journal, altmetric, citation impact metrics, and publication year. Results showed that the AAS, JIF, RCR, and total citations were all positively and significantly correlated with one another, with medium effect sizes. Notably, the AAS was more strongly associated with the RCR than with total citations or JIFs. More recent publication year was associated with higher AAS and lower RCR and total citations; it was not associated with JIFs. The correlations among the JIFs, AAS, and publication year were well below the threshold for suspected multicollinearity among predictors.

TABLE 1

Spearman’s rho intercorrelation coefficients among the 2020 JIF, 2020 AAS, 2021 RCR, 2021 total citations, and Publication Year in the Georgia CTSA‐supported publication portfolio

	JIF	AAS	RCR	Total citations
AAS	0.35*
RCR	0.41*	0.41*
Total citations	0.40*	0.27*	0.88*
Publication year	−0.03	0.19*	−0.22*	−0.57*

n = 2,188; *= p < 0.001.

Abbreviations: AAS, Altmetric Attention Score; CTSA, Clinical and Translational Science Alliance; JIF, Journal Impact Factor; RCR, Relative Citation Ratio.

Using hierarchical multiple regression analyses, we found that in step 1 of the model (Δ R 2 = 0.19*), the 2020 JIF positively and significantly predicts 2021 RCR, with a medium effect size (β = 0.42*) and later publication year significantly predicts lower RCR with a small effect size (β = −0.08*). In step 2 (Δ R 2 = 0.03*), adding 2020 AAS as an independent variable, we found that all three significantly predicted 2021 RCR (βs = 0.34*, −0.11*, and 0.20* for JIF, year, and AAS, respectively). Figure 1 depicts a diagram of the final regression model findings.

FIGURE 1

Final regression model results for relationships among publication year, 2020 Journal Impact Factor (JIF), 2020 Altmetric Attention Score (AAS), and 2021 Relative Citation Ratio (RCR), in the Georgia CTSA‐supported publication portfolio; * p < 0.001

DISCUSSION

Using state‐of‐the‐art bibliometric tools, we were able to evaluate both the near‐term splashes of public attention and long‐term ripples of academic influence of clinical and translational science publications supported by the Georgia CTSA. These patterns describe different but interconnected ways in which these publications are being utilized and potentially impacting translation. Altmetrics show how published research is influencing public knowledge and discourse among scientific organizations, researchers in the same or related fields, patients, and the general public. Academic citations show how research is used in scholarly circles that lead to further academic research. Our results indicate that articles with more altmetric attention have more subsequent citation influence, which is not fully driven by publication in higher impact journals. That is, early altmetric attention to published research, in addition to publication in high impact journals, heightens articles’ ability to achieve wider citation influence, paving the way for translation. It may be that when a publication generates early public interest and community discussion it is more likely to come to the attention of other researchers and professionals that have the ability to propel that research forward. The AAS contributes a modest, but significant improvement to the model predicting the RCR. Unlike the JIFs, this provides an article‐level and potentially actionable factor that can be used to foreshadow eventual citation influence.

Limitations and future directions

One limitation of the relatively new AAS is that, whereas extensive, the metrics tracked and included in the AAS cannot be exhaustive of all altmetric attention paid to research articles. Media communication is vast, ever‐evolving, and sometimes ephemeral. Further, although the AAS is available and interpretable much sooner than traditional citation metrics, some components of the score, such as patent and policy references, may appear later and may be more indicative of ripples than splashes. However, even these do sometimes manifest soon after an important article is published, and they are rarer and less strongly weighted than media sources and are therefore less likely to drive AAS scores.

The Georgia CTSA publication portfolio is a large and cross‐cutting sample of clinical and translational research, covering research across 13 publication years and four complimentary research institutions. However, this portfolio did not contain references to all subcomponents of the AAS (e.g., LinkedIn, which is an important social media tool for translational scientists in industry). Future research should examine the roles of different kinds of altmetric data sources and whether these relationships hold in other portfolio samples. Depending upon relevance to a particular publication, different types of altmetric attention may make different contributions to future citation impact.

To our knowledge, this is the first analysis to investigate the relationship between altmetric influence and subsequent academic citation using a modern, adjusted citation impact factor rather than raw citation count. In our sample of clinical and translational research, we found a stronger correlation between the AAS and citations than did a recent meta‐analysis examining this relationship across many fields. This suggests that research focused on translation, which specifically aims to advance discoveries from the bench to the bedside, may benefit more from dissemination through public and community attention. Further, we found that the AAS was more strongly related to the RCR than to total citations in this dataset. As the RCR is a more robust and valid measure of eventual academic influence, especially when measured across fields, this suggests that the AAS, in addition to the JIF and controlling for time since publication, can be used for evaluation purposes as an early gauge of scientific importance and eventual citation impact. Having this kind of early signal is important for decision making regarding allocation of resources, funding research, career advancement, and efficient acceleration of the translational pipeline. In the future, it would be useful to explore other early citation indices, such as the immediacy index, which measures average citations to a journal in the first year after publication, to assess their relationships with the AAS and later citation influence.

Another outstanding question is how the immense media interest in coronavirus disease 2019 (COVID‐19) research, (which was not included in this analysis because RCRs are not yet available), will influence the academic citation impact of that work. Finally, a key question is whether the association between big splashes and ripple effects is causal in nature such that directed efforts to promote research through media and social networks would result in more utilization of that research.

CONCLUSION

Although publications are not an end goal for translational endeavors, taking the further step of assessing how publications are shared and applied shines the light further down the translational pipeline. Our findings indicate that when clinical and translational research is publicly shared and discussed upon publication it is more likely to achieve greater academic citation influence. Thus, altmetric attention may be a useful early bellwether of eventual, long‐term influence and potential for translational advancement.

CONFLICT OF INTEREST

The authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

N.L. and E.N. wrote the manuscript. N.L. and E.N. designed the research. N.L. performed the research. N.L. analyzed the data.