Counting publications and citations is not just irrelevant: it is an incentive that subverts the impact of clinical research.

Not everything that can be counted counts

More than one million scientists publish peer-reviewed research each year.1 Health research strives to generate new discoveries or consolidate existing knowledge to benefit the lives of humans. But does published health research impact patients, policy, the economy, or society?

Common metrics that are purported to capture scientists’ contributions to their field include citations generated by peer-reviewed publications, journal impact factor, and indices that combine stand-alone metrics such as publication and citation count (eg, H-index).2 These metrics are frequently used by academic scientists and administrators to (1) inform faculty hiring and promotion, (2) rank grant funding applications, and (3) compare researchers’ perceived productivity.3 However, measures of academic output do not appear to capture the socioeconomic impact of health research, and fixating on academic metrics can lead scientists to neglect other important areas. As federal and international health research funding agencies increasingly demand that research should have impact beyond academia, researchers and academic institutions must adapt. We aim to draw the sport and exercise medicine community’s attention to the concept of research impact, highlight existing ways of assessing research impact, and outline the challenges of measuring research impact.

What is research impact?

Research impact is considered the positive effect, influence, or benefit that research has on a variety of areas beyond academia.4 Although a body of literature exists that evaluates the impact of health research,5 research impact is rarely discussed within the broad field of sport and exercise medicine (including sports physiotherapy/physical therapy, sports and exercise science, sports nutrition, and so on). For the purposes of the current editorial, we introduce and consider the impact of health research in sport and exercise medicine on public policy, the economy, and society (table 1).

Table 1

Distinguishing between research impact and academic output

Category	Definition and example
Policy impact	Policy impact refers to research that informs rules established by an organisation (ie, a policymaker) to govern behaviour.9 Scientists can contribute to health policy by submitting relevant research evidence to policymakers, by helping to develop national and local policies, and by contributing to government enquiries (eg, by serving on expert panels or through consultation exercises). For example, research in Canada reported an elevated risk of injury among PeeWee hockey players in leagues that permitted body checking at age 11–12 years compared with leagues that introduced body checking from 13 to 14 years. These data contributed to Hockey Canada’s policy change to delay body checking until 13–14 years of age.10
Economic impact	Economic impacts of health research include commercialising applied health research, healthcare cost savings through reduced morbidity and mortality as a result of interventions produced by health research, or the monetary value of improved health that is informed by research. In the UK during 2013, multifaceted physiotherapy for low back pain improved quality of life to yield an estimated return on investment in related research of £130 million after accounting for the cost of delivering the intervention.7
Societal impact	Societal impact encompasses many terms such as the third-stream activities, societal benefits, societal utility, public value, and societal relevance of health research.11 Although many initiatives have been developed to evaluate the societal impact of health research,12 13 there is still a gap around standardised measures that have been agreed upon and adopted by the research community at large.13 Good Life with osteoArthritis in Denmark (GLA:D) is a population-based programme that implemented clinical guidelines to treat hip and knee osteoarthritis through patient education and physiotherapist-supervised exercise.14 Twelve months after starting the programme, patients reported doing more physical activity and taking fewer pain medications than before entering the programme. Fewer patients took sick leave during the first year of the programme compared with the year prior to participating in the programme.14
Academic output	Academic output is a measure of scientists’ academic performance and research productivity, and is often conflated with the importance and impact of research. Academic output refers to scientists’ intellectual contributions within academia. Many metrics aim to capture the academic output of a scientist and their research, including document-level (eg, publication count), author-level (eg, number of institutional affiliations), and journal-level (eg, journal impact factor) metrics.2 It is unclear, however, whether academic output relates to research impact. A cluster-randomised controlled trial investigating the efficacy of an injury prevention warm-up on acute knee injuries in female adolescent soccer players has been cited over 350 times in 8 years.15 Since publication in a high-impact factor journal, the article has been viewed over 38 000 times and boasts an Altmetric Attention Score in the 98th percentile compared with outputs of the same age and source. From this impressive academic output alone, it can be unclear how this research has contributed to policy change or how it has positively impacted the economic or societal burden of anterior cruciate ligament injuries.

How is the impact of health research assessed?

More than 20 frameworks aim to understand and evaluate the impact of health research.4–6 Impact assessment frameworks often combine a logic model (that maps the intended flow of research from theory to practice) with a case-study description to reflect the complex, non-linear, and interactive processes through which research knowledge is produced and subsequent impact occurs.4 Research impact frameworks can be highly context-specific, often serving the aims of the health organisations that developed them. Some frameworks aim to link research processes (or research funding) with subsequent positive outcomes, whereas other frameworks emphasise the social interactions and networks that develop between scientists and non-academic stakeholders during a research project.4 Frameworks that evaluate research impact have merit. However, many are hampered by their theoretical underpinning and lack empirical validation. The absence of field-specific frameworks (eg, in sport and exercise medicine) is notable for fields in which scientists receive large grants from major federal and international funding agencies with the expectation of subsequently demonstrating research impact.

Not everything that counts can be counted: why evaluating research impact is challenging

At least three challenges confront the task of reliably evaluating research impact.

There is often a lengthy time lag between the initial dissemination of research and its practical impact.6 In 2018, there was an estimated 16-year lag between the provision of public funding and the impact of research-informed interventions for musculoskeletal disorders in the UK.7 Funding- or publication-to-impact lag is often due to the time interval necessary for knowledge dissemination, effective knowledge translation, and corresponding uptake into clinical practice or policy.

Determining the causal effect of research on the outcome of interest is challenging. The effects of impactful research are likely indirect and incremental; informed by prior research and informing subsequent research that enables impact most often within, and sometimes across, research fields.6

Research impact can manifest gradually and can be cumulative, or even fluctuate, over time. Consequently, the impact of research may partially depend on the timing of the impact assessment.4 6

What gets rewarded gets done. Time for transformational change

Misconceptions about what constitutes research impact abound, buoyed by incentive structures in science that predominantly reward traditional academic output. Such misconceptions and incentives compel scientists to prioritise research that will have impressive academic output at the expense of research that has socioeconomic or policy impact. If scientists are rewarded only for the number of peer-reviewed articles that they publish, their corresponding citations, and their ability to obtain grant funding (as they currently are),8 it is not within scientists’ best interests to proactively enhance the socioeconomic impact of their research.

To transcend academia’s obsession with research quantity, research impact must be adequately measured (by using available quantitative and qualitative tools), and appropriately rewarded and prioritised (by funders and universities). This demands that academic incentives be reconsidered and restructured in a way that motivates researchers to embrace impact as a planned phase of the research process.3 Efforts to inform public policy, perform community-based participatory research, develop authentic partnerships with community stakeholders, and engage patients and the public to formulate and design patient-oriented research need to matter and be measured. This, in turn, will empower scientists to consider more fully how their research can be translated to have a positive impact on the health of individuals, communities, and nations.