Toward a science of translational science.

Translational research as a discipline has experienced explosive growth over the last decade as evidenced by significant federal investment and the exponential increase in related publications. However, narrow project-focused or process-based measurement approaches have resulted in insufficient techniques to measure the translational progress of institutions or large-scale networks. A shift from traditional industrial engineering approaches to systematic investigation using the techniques of scientometrics and network science will be required to assess the impact of investments in translational research.

Introduction

The National Center for Advancing Translational Science recently released a new strategic plan and the Clinical and Translational Science Award (CTSA) program is on the cusp of building a national network focused on clinical trials. In addition to this renewed vision for advancing clinical investigation and the science of translation, in 2016, President Obama signed the 21st Century Cures act into law. This initiative also has a strong focus on lowering the barriers to translation and enhancing therapeutic discovery, development, and delivery. Despite this renaissance of interest in accelerating translational breakthroughs and the substantial investment associated with it, the methods by which to quantify or characterize the conditions for translational success are still nascent. Although it is possible to identify progress along the translational continuum for an individual project, there are insufficient methods to assess whether institutions, networks, or nations are becoming more translational in their scientific activity. In this article we review the ontogeny of translational research as a concept, describe definitions of translation, and propose approaches to measuring translational character in large-scale programs.

Historical Perspectives

Although the practice of translational medicine has existed since at least the time of Galen, it was historically a highly pragmatic exercise rooted firmly in a given physician’s individual experience and informed by the immediate needs of his or her patients. Despite the increasing complexity and specialization of medical science in the following centuries, diagnoses and therapies generally continued to improve with only minimal state support for the development of new applications from basic discoveries. In the United States, Vannevar Bush summed up this general philosophy in the mid-20th century by writing that “scientific progress on a broad front results from the free play of free intellects, working on subjects of their own choice, in the manner dictated by their curiosity for exploration of the unknown” [1]. However, as government support for basic science increased after World War II and through the initial inclusion of a patent clause favoring government monopoly, the ethics and assessment of influencing research direction through funding incentives became more important from an administrative perspective.

With massive federal investment, the scale of academic research and the pace of discovery accelerated dramatically. A number of authors warned of the effects of increasing specialization and academic output on the dissemination of knowledge. To quote Bush again, “…there is increased evidence that we are being bogged down today as specialization extends. The investigator is staggered by the findings and conclusions of thousands of other workers—conclusions which he cannot find time to grasp, much less remember, as they appear” [2]. Although Bush penned those words in 1945, by the turn of the 21st century the problem had only grown in importance, with global scientific output doubling every 9 years [3]. The abundance and increased complexity of publications has made the task of timely translation from laboratory-based research to patient care increasingly difficult at the level of the individual physician, who faces increasing burdens to keep up with new clinical or scientific findings. In 1998, researchers from the RAND Corporation found that only 50% of patients received recommended preventative care and only 60% of those with chronic conditions received recommended care. There were errors not only of omission but also of commission: 30% of patients received contraindicated acute care and 20% received contraindicated chronic care [4]. Clinical scientists are similarly overwhelmed by newly reported findings from the basic sciences. Further increasing this burden is the reproducibility crisis in science, which has proven a perennial concern. John Ioannidis’ 2005 determination that most published research findings are false [5] is by now well known. A 2016 poll reported in the journal Nature suggests that little is improving, with over 70% of the 1500 respondents indicating that they had tried and failed to reproduce another scientist’s experiments, and more than 50% indicating having tried and failed to reproduce one of their own [6]. Results such as these threaten translation and undermine trust in foundational or early clinical findings.

In the year 2000, the Institute of Medicine convened the Clinical Research Roundtable (CRR) to discuss emergent issues in clinical research in response to the Association of American Medical Colleges’ publication of “Clinical Research: A National Call to Action.” In June of 2003, the CRR, noting that the clinical research enterprise was truly in crisis, published steps to improve the translation and dissemination of clinical research, identifying 2 translational obstacles—from basic discoveries to clinical practice and from the clinical identification of “things that work” to broader application—as primary concerns [7]. Only 3 months later, the Committee on the Organizational Structure of the National Institutes of Health, in response to a Congressional request and working under the auspices of the National Research Council and the Institute of Medicine, issued a report [2] recommending, among other items, that the NIH enhance its ability to plan and implement trans-NIH strategic initiatives by further funding and empowering the office of the director for this purpose. This recommendation established the groundwork necessary for the NIH to plan and execute the large-scale initiatives necessary to address systemic problems. In October, only 4 months after the CRR’s report was published and 1 month after the Committee on the Organizational Structure of the NIH made its recommendations, the NIH—fresh from a 5-year doubling of its budget and with a new Director in Dr. Elias Zerhouni—published “The NIH Roadmap,” which described a number of trans-NIH clinical initiatives. One among these was the creation of a national network of regional research centers focused on translation. This program, which would later mature into what we know as the CTSA, would go on to fund more than 60 grantee institutions across the country, representing hundreds of millions of dollars of federal investment on an annual basis. It was during this period that publications targeting the concept of translational research began to proliferate at an exponential rate, increasing by 1800% between 2003 and 2014.

Current Challenges

Although the problem of translational research has received increased interest from both sponsors and academics in the last decade, the literature reveals inconsistent, project-based, and nonquantitative definitions of translation [8-12]. This has resulted in a paucity of methods to grasp large-scale translational processes and, given current national initiatives, suggests the need for a shift in perspective away from the analysis of small subsystems within local institutions toward solutions focused on complex systems.

Lacking guidance from concrete success measures or the theoretical framework by which to pursue them has resulted in literature about translation focusing on relatively small and well-understood tasks: institutional process modeling, definitional activities, administrative process improvements, and regulatory streamlining. Much of this work is informed by individual experience and experience at the level of individual institutions. Implicit in this approach is the assumption that improvements to these various subprocesses will result in positive outcomes consistent with the CTSA program’s strategic goals. However, the social networks and systems that support translational outcomes as we recognize them are complex, nonlinear, often multidisciplinary or interdisciplinary, and temporally diffuse. This complexity has confused the work of many authors, leading Steven Woolf to state simply that “translational research means different things to different people” [8]. In addition to confounding attempts at definitional concision, these characteristics indicate that subprocess improvement alone will be, at best, accidentally sufficient in improving national translational outcomes and, at worst, will result in retrogression or other unintended negative systemic results.

Future Solutions

Narrow performance measurements are important for fine-tuning small-scale interventions, but the scope of change that the CTSA program was created to effect requires global measures of sociological evolution and research impact. Although we have traditionally assessed whether projects are translational, we must now assess whether entire programs are translational. New and transparent scientometric indicators, rather than modified industrial engineering approaches, are critically important.

Researchers in the fields of bibliometrics, scientometrics, informetrics, and network science have already established much of the necessary theory needed to pursue the study of translational science policy and have developed a number of tools and measures that are readily applicable. For example, Boyack et al. [13], building upon earlier work by Francis Narin, have developed a new method by which individual journal articles can be automatically classified according to research level: basic research, clinical investigation, clinical mix, and clinical observation. Surkis et al., recently developed a new model to classify publications along the translational spectrum using similar techniques [14]. These automated approaches to classification can potentially eliminate the time-consuming task of manual review that previously hindered the systematic study of translational trends. Although both Boyack and Surkis focused on publications, the analysis of article texts tells us only of translational claims. Equally important in the study of translational behavior will be the examination of translational intent. Perhaps the most important source of information on translational intent is the corpus of proposal data held by the NIH and the tributary repositories held within research institutions across the country. Representing an enormous trove of almost completely unexplored data, this full-text corpus contains not only a source to deduce the research intentions of funded investigators but—perhaps more importantly—many thousands of unsuccessful proposals.

Coding all of an institution’s biomedical publications according to where they fall on the translational continuum will allow for the identification of trends in basic and translational productivity. However, these publication trends may be confounded by the effects of Congressional budget allocations, NIH funding priorities, the whims of study groups, an investigator’s writing quality, the varied tastes of journal editors, or by simple skewing in the number of journals targeting one area of the continuum over others. The analysis of submitted proposals, particularly of unfunded proposals, will allow for a more complete picture of the changing interests of biomedical researchers. Importantly, these trends will contain valuable information on the effects of large-scale investment such as the CTSA. By examining the texts of submitted proposals as well as their references we can determine surrogates of the translational intent of these proposals and where on the continuum the proposed work is situated. With this information we can map shifts in an institution’s translational interests—regardless of whether or not these interests are realized through eventual funding and regardless of when, where, or how any findings are published. The potential effects of the interventions of a single CTSA hub are necessarily limited. If these interventions can be correlated with increased translational intent, but not with funding success or publication output, then the hub can perhaps consider additional interventions specifically targeting these areas. A similar process could be explored at the national level, with federal response to opportunities for improvement.

The analysis of grant proposals as a surrogate of scientific activity and publications as a surrogate of scientific productivity still yields an incomplete picture that reflects either an institutional perspective or an undifferentiated aggregate of multiple institutions. What will be critically important for initiatives such as the CTSA is the analysis of translational success by virtue of being a network. Highly efficient networks strike a balance between functional segregation (eg, a unique strength of an individual CTSA institute) and global integration (eg, the communication structures and knowledge transfer among the institutes). Although the CTSA has more recently adopted the approach of network science with new language and hub-and-spoke models, the quantitative methods of network science have been applied rarely and only by individual CTSA institutes [15-19]. The current CTSA focus on developing common metrics should be complemented by consideration of the various properties of the nodes (ie, the individual CTSA institutes) and a quantitative definition of the links between them. This step would allow the rich analysis of various network properties, including degree (number of connections of an individual node), path length (steps to transmit knowledge or best practices), modularity (how isolated or siloed the network is), centrality (the extent to which nodes create shortcuts across complex institutional or national networks), and small worldness (an organizational principle that facilitates integration through strong hubs). This network analysis could span from an individual CTSA institute within an institution, to the CTSA network itself, allowing a quantitative approach to grasping the synergy of CTSA interactions that can advance translational progress on the national scale.

Conclusions

This is an exciting time for translational research. However, the investment of resources and effort in advancing the translational character of the biomedical research enterprise must be matched by the methodology to measure success. Leveraging emerging bibliometric approaches and network science techniques will be an important first step to advance beyond the consideration of individual translational research projects to large-scale translational programs at the national level.