Big splashes & ripple effects: a narrative review of the short- & long-term impact of publications supported by an NIH CTSA pediatrics program.

Background and Objective: This review examines a promising new framework for analyzing outputs of pediatric research in the context of translational advancement. We demonstrate a method for evaluating the impact of an NIH Clinical and Translational Science Award's (CTSA) Pediatrics Program through publications that have emerged from supported research. The Georgia CTSA Pediatrics Program provides training, funding, and infrastructure to ensure that researchers have the resources to advance pediatric health. Internal evaluations found that research supported by this program is exceptionally impactful within the academic community and commands high interest within the lay community. Therefore, we examined the impact of this research in both traditional academic and broader community spheres using bibliometrics-the study of supported publications. Bibliometrics describe a pivotal stage in the translational process of bringing scientific discoveries to clinical/community use and include both academic citations and 'altmetric' or non-academic attention. These complementary approaches combine to shed light on the short- and long-term impact of the research on segments of the translational pipeline, including academic literature, community discourse, technological advancement, and public health policy.
Methods: The authors identified a portfolio of 250 articles supported by the Georgia CTSA Pediatrics Program from 2007-2020. We utilized various bibliometrics to analyze both short-term attention, or 'splash' made by articles, and long-term influence, or 'ripples' made across both academic and public spheres. Key Content and Findings: The short-term splash of the portfolio was indicated through publication in high-impact factor journals, peer faculty recommendations, and Mendeley readership, as well as by early altmetric attention in news stories, blogs, and Twitter posts. The portfolio's long-term ripples were demonstrated by high absolute and relative rates of academic citation and by downstream altmetric influence in public-facing documents, including Wikipedia articles, patent applications, and policy documents. Conclusions: This article reviews a useful bibliometric methodology for illustrating the waves of impact made by pediatric research. Whereas splash provides a picture of early interest in a publication, a preliminary indicator of eventual utility and impact, ripples provide a measure of the cumulative influence of an article over time. Both reflect opportunities for a line of research to advance along the translational spectrum. 2022 Translational Pediatrics. All rights reserved.

Introduction

The aim of this review is to examine a promising new framework for analyzing the outputs of pediatric research in the context of translational advancement. The central question under review is how bibliometrics, or the study of publications as a scientific output, can be used to determine how a line of research is making an impact on the translation of pediatric science from the bench to the bedside, or from clinical research to clinical use and health improvements.

The National Center for Advancing Translational Science (NCATS) of the National Institutes for Health (NIH) supports a nationwide network of Clinical and Translational Science Award (CTSA) hubs charged with accelerating clinical and translational education, research, and community engagement to improve health (1,2). The Georgia Clinical & Translational Science Alliance (Georgia CTSA) was established in 2007 as one of the first of these hubs and consists of a cross-institutional collaboration among Emory University, Morehouse School of Medicine, the Georgia Institute of Technology, and the University of Georgia, with a collection of interconnected programs providing relevant services to investigators within those institutions. One such program is the Georgia CTSA Pediatrics Program, a longstanding effort to transform scientific discovery into solutions that improve pediatric health. The program provides training, seed funding, and infrastructure to researchers to ensure that research projects have the resources needed to move forward. Services include expert consultations, educational and training materials, clinical research space and staff, pilot and training grants, and administrative support.

Regular evaluation of programmatic outputs is essential for understanding the extent that a program is meeting its stated goals, and several qualitative and quantitative evaluation approaches of various CTSA programs have been published thus far, including assessment of supported publications and grants (3-5) and numbers of citations (5-8) accrued over several years. Previous internal evaluation of Georgia CTSA pediatric research output uncovered a body of work that is highly interdisciplinary and particularly impactful within the academic community, compared to similar work at other institutions (9). Pediatrics research is unique among clinical/translation research in its coverage of diverse fields of study and the translation of pediatrics research from bench to bedside often involves significant collaboration and scholarly discourse across disciplines and phases of translational research. Moreover, pediatrics research has a tendency to be of high interest to the broader lay community and internal evaluation showed that pediatrics research received more public/community attention than other specializations (10). Heightened public interest in children’s health may have implications for research priorities and directives. Therefore, it is useful to systematically analyze the impact of translational pediatric research broadly, across both academic and public spheres.

One useful way to understand the impact of a research program is through bibliometric portfolio analysis. Although publications are not themselves an end goal for translational endeavors, bibliometrics do describe a pivotal early stage in the process of translating new scientific discoveries to clinical use. Taking the further step of assessing how publications are shared and applied shines the light even further down the translational pipeline. Past bibliometric evaluations have typically relied upon citations within academic literature only (11-13). This approach is vulnerable to self-citation (11), and usually requires long windows of time, (i.e., at least several years), to observe slowly accumulating influence that does not generally speak to the public and translational impact of the research. Additionally, previous research has typically focused on journal-level, (e.g., journal impact factor, Eigenfactor score) (12,14,15) or investigator-level, (e.g., h-index, g-index) (11,15) impact, which are difficult to compare across diverse areas of pediatric research. Indeed, others have suggested that most current citation factors are ill-suited for comparing across scientific disciplines and that the use of multiple indicators is crucial (16,17). Traditional bibliometrics have been used to provide overviews that focus on long-term influence in particular areas of pediatric research (18-21), and among CTSA-supported programs (8,22). However, this is the first analysis to present complimentary bibliometric tools used together to understand and discuss both short-term attention and long-term influence of publications and shed light on how pediatric research is being used within both academic and public networks.

We first contextualize the research supported by the Georgia CTSA Pediatrics Program by summarizing the content of the publication portfolio that has amassed over 13 years of operations. Identifying the predominant subjects addressed by supported research aids in understanding the focus and diversity of the research. We also describe features of the portfolio that pertain to the translational spectrum. In the central analyses, we delineate both the short-term and long-term impact of the articles according to patterns that reflect ‘splashes’ or short-term, immediate, and emphatic forms of impact, and ‘ripples’ or long-term, accumulating, and enduring forms of impact. We combine established academic citation metrics with non-traditional ‘altmetrics’, or non-academic forms of public attention to publications (23), to form a comprehensive bibliometric review. To assess short-term ‘splash’ we examine bibliometric measures that reflect immediate academic exposure and potential for future citations (i.e., journal impact factor, peer faculty recommendations, and Mendeley reader downloads) as well as altmetrics that reflect early community attention and public interest/engagement (i.e., references in news stories, blog posts, and Twitter posts). To assess long-term ‘ripples’ we examine bibliometric measures of impact on the scholarly literature (i.e., total citations and Relative Citation Ratios calculated over time), as well as altmetrics that reflect downstream influence among the broader public (i.e., references in Wikipedia articles, patent applications, and public policy documents). We summarize overall and average citation impact metrics and altmetrics and provide case illustrations of several articles that have made notably big splashes or had broad ripple effects.

In sum, this article is intended to demonstrate an innovative methodological toolkit for assessing contributions to clinical and translational science that have been made possible by research supported by an NIH CTSA Pediatrics Program. Using complementary approaches, we examine the bibliometric products that have emerged from supported research and their translational impact. Results illuminate the scope and the short- and long-term impact of this research on academic literature, community discourse, technological advancement, and public health policy. We present the following article in accordance with the Narrative Review reporting checklist (available at https://tp.amegroups.com/article/view/10.21037/tp-21-506/rc).

Methods

Data collection

In fall 2019, internal records were queried for instances of program support received by pediatric researchers and other researchers collaborating on pediatric research, since the inception of the Georgia CTSA in 2007. A total of 224 Georgia CTSA investigators (most from Emory University and the affiliated Children’s Healthcare of Atlanta) and 988 instances of support were identified [predominantly (approx. 80%) clinical services and staff from the Georgia CTSA Clinical Research Centers and expert consultation support from the Informatics Program and the Biostatistics, Epidemiology, and Research Design Program]. These investigators were cross-referenced with publications queried from PubMed (24) as having formally acknowledged grant support from the Georgia CTSA (n=2,942). We selected publications that met all of the following inclusion criteria:

Acknowledged a past or present Georgia CTSA NIH grant (UL1 TR002378, UL1 TR000454, UL1 RR025008, KL2 TR002381, KL2 TR000455, KL2 RR025009, TL1 TR002382, TL1 TR000456, TL1 RR025010) as having provided support

Authored by at least one of 224 pediatric researchers from Georgia CTSA member institutions who received program support

Contained clear pediatric content evidenced by pediatric keywords appearing in the journal title, article title, MESH terms, or Web of Science Research Area. Resulting publications were individually inspected for misattributed/aberrant keyword results.

Articles were excluded from analysis if they did not meet all three criteria. A total of 250 (8.5% of all Georgia CTSA-supported publications) articles met inclusion criteria and were included in further analyses (see ). These publications were authored by 93 Georgia CTSA investigators who had received 630 instances of program support.

Table 1

The search strategy summary

Items	Specification
Date of Search (specified to date, month and year)	October 1, 2019
Databases and other sources searched	Internal records, PubMed, InCites, iCite, Dimensions
Search terms used (including MeSH and free text search terms and filters).	Containing clear pediatric content evidenced by pediatric keywords (pediatric/paediatric, child, infant, youth, natal, adolescent) appearing in the journal title, article title, MESH terms, or Web of Science Research Area. Resulting publications were individually inspected for misattributed/aberrant keyword results
Timeframe	Published 2007 through 2019
Inclusion and exclusion criteria (study type, language restrictions etc.)	To be included in analysis, pediatric articles must have acknowledged a past or present Georgia CTSA NIH grant (UL1 TR002378, UL1 TR000454, UL1 RR025008, KL2 TR002381, KL2 TR000455, KL2 RR025009, TL1 TR002382, TL1 TR000456, TL1 RR025010) as having provided support, and been authored by at least one of 224 pediatric researchers from Georgia CTSA member institutions who received program support
Selection process (who conducted the selection, whether it was conducted independently, how consensus was obtained, etc.)	The search was conducted by the first author using methods discussed and approved by all authors
Any additional considerations, if applicable	None

Next, in order to retrieve journal and content information, the list of publications was searched in Web of Science’s (WoS) subscription-based InCites application (25,26) in 2020; 236 indexed publications were found in WoS InCites, yielding a dataset that included the following for each article:

❖ Journal and Journal Impact Factor (JIF), a proprietary InCites metric, which is an unadjusted measure of typical citation rates for the journals in which each article was published. A JIF of five, for instance, means that the articles published in that journal in the past two years have been cited, on average, five times.

❖ Web of Science Research Area, the most granular categorization scheme for research content area available from InCites, which includes 252 subject categories across science, social science, arts and humanities; not all of which are expected to be applicable to clinical/translational pediatric research. The WoS Research Area is usually assigned based upon the content area of the journal in which the article is published. If the journal is general or multidisciplinary (e.g., New England Journal of Medicine, PlosOne, etc) then the article is assigned based upon its cited reference list and only assigned to the general category if no more specific designation can be made. It is typically not feasible to assign a journal/publication to a single category, therefore, up to six research areas may be assigned to a given journal and corresponding articles (27).

Next, in order to retrieve citation and translational feature information, the list of publications was searched in the NIH’s iCite application (28) in 2020. All 250 indexed publications were found in iCite, yielding a dataset that included the following for each article:

❖ The iCite Translational Features Module, which includes (I) percentages of research involving human, animal, and molecular/cellular research as designated through the triangle of biomedicine (29); (II) the Approximate Potential to Translate (APT) (30) a value between 0 and 1 reflecting the predicted percent likelihood that a paper will eventually receive a clinical citation; (III) designations as clinical articles and actual citations by clinical articles to date.

❖ Total academic citation count as of mid-2020.

❖ Relative Citation Ratio (RCR) (31), a state of the art field-normalized citation metric that calculates the citation impact of an article relative to the average NIH-funded paper in its co-citation network. RCR data is only available for publications that are at least one calendar year old and was available for 231 of the focal articles as of mid-2020.

Next, to retrieve author and altmetrics information, the list of publications was searched in Digital Science’s subscription-based Dimensions application (32) in 2020 (Dimensions also offers a free limited-access version). All 250 indexed publications were found in Dimensions, yielding a dataset that included the following for each article:

❖ Co-authors and their affiliated organizations.

❖ Altmetric Attention Score (AAS) (23), a rank-ordered index score, aggregated from a number of specific subcomponents, that reflects media and community attention paid to a published article as well as use of the article in subsequent public documents. Specific subcomponents of the AAS detailed in this study include references to publications in: news articles, blog posts, Twitter posts (33), Wikipedia articles (34), patent applications, policy documents, and Faculty Opinions [formerly F1000 Prime; (35)] peer faculty recommendations. We also included in our analysis reader downloads in the Mendeley Reference Management program (36), which are tracked by Dimensions but are not used in calculating the AAS.

Data analysis

First, in order to contextualize the content of this pediatric publication portfolio before the central analyses, we assessed descriptive characteristics of the portfolio, including the focus and diversity of the research via the frequency distributions of articles published in different journal outlets and representing different WoS Research Areas. To identify major clusters of research carried out in repeated collaborations, we employed network analysis for co-authorship data from Dimensions, using VOSviewer 1.6.15 [Visualization of Similarities (VOS)] (18). Major lines of work were then assessed for content based upon article content and author specialty. We then assessed the translational features of the portfolio, utilizing the metrics included in iCite’s Translational Features module.

In the main phase of analyses, to explore the splashes and ripples made by articles in this portfolio, we examined mean, maximum, and sum totals for journal- and article-level impact factors, Mendeley reader downloads, and Altmetric Attention Score components, describing the sources of altmetric attention. The short-term impact of the portfolio in the academic sphere is reflected by scholarly interest and readership that can be measured soon after publication and heighten potential for future citations. The journal impact factor, which is immediately available as it is based upon citation precedent, is associated with circulation, readership, and average rates of academic citation (27,37,38). Thus, articles published in high impact factor journals will, on average, receive more exposure within the academic community, and the impact factor is often used as a preliminary indicator of the expected influence of articles selected for publication in that journal. Official peer faculty recommendations in Faculty Opinions are reserved for articles thought to be of exceptional importance after qualitative review by a peer-nominated faculty of internationally-renowned researchers. Article downloads on the Mendeley reference management program is a particularly early measure of broad academic interest in a publication from around the globe. Identifying and downloading an article creates potential for that paper to be read, used, and cited in subsequent research. On the other hand, early interest in research in the public sphere is indicated by altmetrics that signify early community discourse on published research. Relevant components of the AAS for short-term impact include references to publications in news stories, blog posts, and Twitter posts shared to followers around the world, all of which tend to peak within days to weeks after publication.

Conversely, the long-term impact of the portfolio in the academic sphere is demonstrated by absolute and relative rates of academic citations accumulated over several years. Aggregate citation counts correspond to the amount of direct impact that the portfolio has had on the academic literature. The RCR corresponds to relative citation influence of articles within the context of their field of study and the time elapsed since publication. Long-term, downstream influence in the public sphere is indicated by relevant AAS components that reflect documented usage of the publications to inform public knowledge. References in broadly used and openly accessible Wikipedia articles reach and inform wide-ranging audiences. References in filed patent applications document usage of research in ways that may advance technology through new inventions or technological processes. References in policy documents from government agencies and NGOs document how research is used to influence public health policy.

summarizes the full metric toolkit of short-term splash versus long-term ripples analyzed in this article. Finally, we used the aforementioned metrics to identify several case example articles, with relatively high AAS values, that reflect high impact splashes and influential ripple effects, and present illustrations of the impact made by these articles.

Figure 1

Framework for bibliometric measures of short- and long-term impact within academic and public spheres.

Discussion

Portfolio summary: focus, diversity, and translational features

Descriptive analyses revealed that 250 Georgia CTSA-supported pediatrics articles were published at a steady rate from the inception of the program in 2007 through 2020. The articles were published in 112 different journals, most frequently Pediatrics (21 articles), The Journal of Pediatrics (20), and the Journal of Allergy and Clinical Immunology (18). The articles represented 44 WoS Research Areas, most often Pediatrics (104 articles), but with the most frequent field-specific areas being Immunology [42], Allergy [27], and Oncology [23]. The 93 Georgia CTSA authors collaborated with more than 1,550 co-authors from 236 different institutions across the US, as well as 16 other countries. The network of most frequent co-authorships (of at least 5 publications; 97 authors) indicated significant collaboration both within and across five major cohesive clusters centering on: premature infants, childhood asthma, liver and kidney disease, and autism.

The research in this portfolio covered a broad range of the translational spectrum from basic and foundational research intended to elucidate correlates, mechanisms, and molecular underpinnings of health problems to applied research intended to improve treatments, therapies, and protocols for particular health problems (39). The iCite Translational Features Module revealed that the majority (89%) of the research involved human subjects. The mean APT value for the portfolio was 0.67, meaning that on average, there is a 67% likelihood that articles in this portfolio will be translated to clinical use by being cited in a clinical article. Consistent with this, 59% of the articles have been cited by clinical articles thus far, and 25% are themselves clinical articles.

Big splashes & ripple effects

summarizes findings for metrics that reflect splashes and ripples among the Georgia CTSA Pediatrics Program’s supported publications. Of the 250 articles, 93% had at least one academic citation and 70% had some form of altmetric attention.

Table 2

Short- and long-term academic and altmetric impact measures for the Georgia CTSA pediatrics program publication portfolio

N=250 publications	Mean	Max	Sum
Overall Altmetric Attention Score (AAS)	26	677	5,228
Splashes: short-term impact metrics
Academic measures:
Journal impact factor (JIF)	8	75	n/a
Peer faculty recommendations	0.1	2	19
Mendeley reader downloads	75	1,029	15,266
Altmetric references in:
News stories	1.5	54	365
Blog posts	0.3	10	73
Twitter posts	18	740	4,494 (to 17 M followers)
Ripples: long-term impact metrics
Academic measures:
Academic citations	43	1,091	10,819
Relative citation ratio (RCR)	3.3	107	n/a
Altmetric references in:
Wikipedia articles	0.02	1	4
Patent applications	0.06	3	16 (15 granted)
Policy documents	0.1	4	33

CTSA, Clinical and Translational Science Award.

Short-term impact in this portfolio is demonstrated by high average JIF, a number of rarely-granted recommendations from Faculty Opinions, thousands of Mendeley downloads and references in news stories (from sources such as Physician’s Briefing, MedicalXpress, and EurekAlert), blog posts (from blogs such as Neonatal Research, Latest BMJ Blogs, and JAMA Author Interviews), and Twitter posts (largely from science and medical professionals and organizations). These metrics have the advantage of typically manifesting at or soon after publication and providing indications of the potential importance and utility of recent, cutting-edge research without a long period of latency.

Long-term impact in this portfolio is demonstrated by thousands of academic citations, a mean RCR value indicating more than 3 times the expected number of citations based on field and publication year, references in Wikipedia articles (for Sepsis, Adrenarche, Sweetened beverage, and one for a study author), patent applications (such as for medications, formulas, and diagnostic tools), and policy documents (from agencies such as the World Health Organization, the Centers for Disease Control & Prevention, and international health agencies). These metrics have the advantage of portraying actual usage of articles in subsequent scholarly and public pursuits that move research along the translational pipeline.

Using this suite of metrics we identified specific publications that represented strong case examples of big splashes and broad ripple effects. summarizes these case examples including a description of how Georgia CTSA Pediatrics Program supported the research and how the article represents a big splash or broad ripple effect, using the metrics in this toolkit. Examples a, b, and c (40-42) exemplify big splashes within both academic spheres (high impact factor journals, Faculty Opinions recommendation, Mendeley downloads) and public spheres (news, blog, and twitter references); examples d, e, and f (43-45) illustrate broad ripple effects across both academic (academic citations) and public spheres (Wikipedia, patent, and policy references).

Table 3

Case examples of publications representing big splashes and broad ripple effects in the Georgia CTSA pediatrics program publication portfolio

Big splashes
(a)	‘Effect of a low free sugar diet vs. usual diet on nonalcoholic fatty liver disease in adolescent boys: A randomized clinical trial’. Schwimmer, et al., JAMA. 2019 (40); AAS =538
	o Authors supported via Georgia CTSA nursing/lab resources and Informatics and Biostatistical consultations• Published in the Journal of the American Medical Association (JAMA), JIF =51.3• Mentioned in 28 news articles, e.g., 2019 New York Times article: ‘To Fight Fatty Liver, Avoid Sugary Foods and Drinks’• Mentioned in more than 600 Tweets to over 2 million Twitter followers, e.g., @DrKristieLeong: “Non-alcoholic fatty liver disease is a growing problem even in children. A recent study found that a reduced #sugar diet over 8 weeks led to significant improvement in markers of liver #health #liverdisease”• Recommended by a faculty member in Faculty Opinions
(b)	‘Tisagenlecleucel in children and young adults with B-cell lymphoblastic leukemia’. Maude, et al. NEJM. 2018 (41); AAS =413
	o Georgia CTSA author (M. Qayed) supported via a pilot grant and a KL2 training grant• Published in the New England Journal of Medicine (NEJM), JIF =70.7• Mentioned in more than 150 tweets to over 1 million followers, e.g.: @dr_mark_russell: “Will be interested to see the impact of CAR-T cell therapy on #PedsICU haem/onc children—anyone seen many children who have received it?” • Mentioned in 8 science blogs, e.g.: 2018 Hematology Times article ‘CAR T-cell therapy produces durable CRs in ALL’• Mentioned in 37 news outlets around the world, e.g., BBC News, Deutsches Arzteblatt, MedIndia• Downloaded by Mendeley reference manager program readers 719 times
(c)	‘Acetaminophen versus ibuprofen in young children with mild persistent asthma’. Sheehan, et al. NEJM. 2016 (42); AAS =332
	o Georgia CTSA author (A. Fitzpatrick) supported via Georgia CTSA clinical research space and laboratory resources and a KL2 training grant• Published in the New England Journal of Medicine (NEJM), JIF =70.7• Mentioned in 41 news stories from 27 outlets, e.g., Physician’s Briefing, Deutsches Arzteblatt, MedicalXpress • Mentioned in 3 academic blog posts, e.g., Vector, BMJ Blogs• Mentioned in more than 190 tweets to over 1 million followers. e.g., @AllergyKidsDoc: “Re-reading great @NEJM article from last yr: No difference in #asthma attacks for kids receiving tylenol vs. ibuprofen”
Broad ripple effects
(d)	‘Consumption of added sugars is decreasing in the United States’. Welsh, et al. Am J Clin Nutr. 2011 (43); AAS =412
	o Authors supported via Georgia CTSA pilot grant • Cited in over 300 academic articles; 10 times the expected citation rate• Cited in multiple CDC health policy reports, e.g., ‘Consumption of Added Sugar Among U.S. Children and Adolescents, 2005-2008’• Referenced in Wikipedia article on ‘Sweetened beverage’
(e)	‘A randomized controlled crossover trial with delayed-release cysteamine bitartrate in nephropathic cystinosis: effectiveness on white blood cell cystine levels and comparison of safety’. Langmon, et al. Clin J Am Soc Nephrol. 2012 (44); AAS =13
	o Georgia CTSA author (L. Greenbaum) supported via clinical research space and laboratory resources• Cited in multiple patents on cysteamine formulation and manufacture: ‘Methods for Storing Cysteamine Formulations and Related Methods of Treatment’, ‘Delayed Release Cysteamine Bead Formation, and Methods of Making and Using Same’
(f)	‘Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network’. Stoll, et al. Pediatrics. 2010 (45); AAS =86
	o Georgia CTSA author (B. Stoll) supported via clinical research services• Cited in over 1,000 academic articles, including citations in 88 clinical articles; 51 times the expected citation rate• Cited in multiple international and CDC health policy reports, e.g., ‘Health, United States, 2015: in brief’• Cited in a patent application: ‘Biomarker Pairs of Preterm Birth’

AAS, Altmetric Attention Score, JIF, journal impact factor.

Strengths and limitations

The limitations of using academic citations to measure impact have long been discussed (11,31,46), and newly updated metrics are periodically introduced to address the limitations of previous iterations (e.g., adding the g-index and m-quotient to the h-index to compare productivity among pediatric faculty) (47,48). In the context of translational goals, academic citations alone are especially limiting because they reflect influence within an insulated academic community that does not necessarily reach the relevant clinicians and patient populations at the bedside. Altmetrics, on the other hand, have the potential to capture some of the impact that research articles have beyond the boundaries of academia. One limitation of the relatively new AAS is that, while extensive, the metrics collected by Dimensions cannot be exhaustive of all community attention paid to research articles. Media communication is vast, ever-evolving, and sometimes ephemeral, and the AAS has been tracking some sources for less than ten years. We intentionally chose some of the most salient and well-tracked media platforms available (i.e., Twitter, Wikipedia), but the AAS includes additional, albeit, limited data from platforms such as Facebook, Google+ and Reddit, which we did not include in our analysis in the interest of more concise interpretability. Further, the content and quality of altmetric attention is not captured in the AAS and can be either positive or negative in nature, or may not have strong relevance for translational advancement. For example, Tweets can be susceptible to gaming by automated bots (49), which further reason to adopt multi-faceted methods for evaluating publications in the short term, which do not rely on singular metrics. Still, the AAS is a useful quantitative tool for identifying articles with significant attention to then examine their specific altmetrics on a more individual or qualitative basis. Indeed, models the inclusion of twitter followers and users when showcasing media influence.

Our overall framework, summarized in , provides a useful blueprint for using bibliometrics to compare/benchmark publication portfolios from different programs or centers, and to communicate findings to leadership and stakeholders as well as the general public. Indeed, demonstrating the impact of a funded program such as an NIH-sponsored CTSA center is critical when advocating for continued funding. The suites of metrics that reflect splashes and ripples leverage cutting-edge altmetrics together with established best practices for evaluating academic citation influence, forming a compelling methodology for illustrating impact, tailored to the specific characteristics of cross-cutting translational pediatric research. A strength of this framework is that it may be flexibly applied to other existing and emerging data sources, (e.g., PlumX metrics, clinical citations, Overton policy citations), as needed, to meet the needs and resources of a given program, such that the ‘academic sphere’ includes use by other researchers within the translational pipeline, the ‘public sphere’ includes use by those outside of the translational science pipeline, ‘short-term’ encompasses the first months and years after publication, and ‘long-term’ generally encompasses more than several years after publication. Although any one metric may yield an incomplete picture (e.g., journal-level impact factors represent averages across articles that may have with widely varying individual citation impact), we suggest that converging evidence of impact is key, whether it be academic and/or public, short and/or long term in nature.

Summary & conclusion

The short-term splashes and long-term ripples depicted by both academic measures and altmetrics tell the story of a pediatrics program’s research impact in ways that go further than traditional citation metrics and inform our understanding of how these research products are being used and translated in both the academic and the broader community. Our analyses characterize specific mechanisms by which this body of research is advancing findings from the bench to the bedside. Taken together, these findings establish the myriad ways that this research has engendered progress across diverse segments of pediatric medicine. Case examples further illustrate specific areas of notable influence.

This review provides a useful new methodological framework for illustrating the waves of impact made by pediatric research, that we advocate for as complementary to traditional publication evaluation metrics. Whether a publication makes a big splash of early interest that foreshadows eventual utility and impact, or shows ripple effects of cumulative influence over time, both reflect opportunities for a line of research to advance along the translational spectrum. In the future it may be useful to use bibliometrics to draw clearer connections between specific program inputs and relevant outputs. Future research could also further explore the connections between splashes and ripple effects. That is, does early attention paid to an article predict long-term influence, or do these forms of influence function separately? Past research has found a weak association between Altmetric score and citations among pediatrics publications (50), but others have identified ‘sleeping beauties’ in pediatric research that only become highly cited and influential after a long period with few citations (51). Additionally, assuming that all forms of altmetric attention are not equal, what are the implications of different types of near-term public attention? It is important to note that much of the altmetric attention observed in this review originated with academic or professional sources, such as a Tweet from a university account, or a blog written by a health professional, which may then be disseminated and re-shared broadly. These activities could bring findings to the attention of other researchers in adjacent fields, to relevant clinicians, or to potential patients or community stakeholders. This observation illustrates the dual responsibility of the scientific community in communicating findings through both scholarly and public channels, with a key distinction in the intended audience. Future research could benefit from qualitatively studying the pathways through which scientific knowledge flows through academic versus altmetric sources to both other researchers and to the public, and the implications of each in terms of potential inter-disciplinarity and translational advancement.

The article’s supplementary files as