Greatest 'HITS': A new tool for tracking impacts at the National Institute of Environmental Health Sciences.

Evaluators of scientific research programs have several tools to document and analyze products of scientific research, but few tools exist for exploring and capturing the impacts of such research. Understanding impacts is beneficial because it fosters a greater sense of accountability and stewardship for federal research dollars. This article presents the High Impacts Tracking System (HITS), a new approach to documenting research impacts that is in development at the National Institute of Environmental Health Sciences (NIEHS). HITS is designed to help identify scientific advances in the NIEHS research portfolio as they emerge, and provide a robust data structure to capture those advances. We have downloaded previously un-searchable data from the central NIH grants database and developed a robust coding schema to help us track research products (going beyond publication counts to the content of publications) as well as research impacts. We describe the coding schema and key system features as well as several development challenges, including data integration, development of a final data structure from three separate ontologies, and ways to develop consensus about codes among program staff.

1. Introduction

The National Institute of Environmental Health Sciences (NIEHS), one of the National Institutes of Health (NIH), is a federal research agency that aims to understand how the environment influences the development and progression of human disease. NIEHS’ Division of Extramural Research and Training supports a wide portfolio of environmental health science research grants. To better organize, track, and communicate the short- and long-term impacts of these research investments, we developed the High Impacts Tracking System (HITS), the first of its kind at NIH.

Scrutiny of federal investments in research has grown steadily during the past two decades (OMB 1993, 2009, 2010, 2012), and research budgets have been flat or declining for several years as well (NIH Office of Budget 2011). These pressures have consequently increased the need for accountability and transparency at NIH, especially as relates to the impacts of our research enterprise. To do this, research administrators need to be able to describe the scientific topics addressed in a grant (e.g. basic versus applied research, disease endpoint, contaminant, or route of exposure), the methodological approaches used by grantees (e.g. model systems, environmental epidemiology, community-based participatory research, or genome-wide association studies) as well as the activities, products, and impacts of research.

NIH is well positioned to quantify and report on both the inputs as well as the direct products (outputs) of research. In recent years the agency has developed databases, coding projects, and portfolio tools that provide valuable information about the funding, topics, approaches, and outputs or products of the research we fund for planning and decision making. Key examples include the following:

IMPAC II (Information for Management, Planning, Analysis, and Coordination system) contains many layers of complex qualitative and quantitative data about the receipt, review, financial oversight, and annual administration of grants.

SPIRES (Scientific Publication Information Retrieval & Evaluation System) provides a method to link publications to the grants that supported them (Boyack and Jordan 2011).

eSPA (Electronic Scientific Portfolio Assistant) provides the ability to create portfolios of research grants, and link/track publications and citation data to those portfolios (Haak et al. 2012).

RCDC (the Research, Conditions, Disease, and Categories) system provides an automated and searchable ‘fingerprint’ of the scientific, disease, and technical terms for each grant, based on its abstract and specific aims.

RePORT (Research Portfolio Online Reporting Tools) gives the public access to non-sensitive grant information, links to publications via the SPIRES tool, and the technical ‘fingerprints’ from RCDC (http://projectreporter.nih.gov/reporter.cfm). This is the only one of these tools that is directly available to the public.

These resources provide an essential foundation for portfolio analyses within the NIH research grant context, but they are not sufficient for tracking research impacts. While identifying and reporting activities, outputs and impacts can be crucial in making the case for program success, tracking and monitoring the results of research presents several challenges. First, there is no common ontology to describe and categorize the outputs and impacts of NIH funded research. Second, there is no infrastructure to facilitate the collection and analysis of impacts. Finally, data to evaluate long-term impacts are not easily available for quantitative analysis, and instead typically require intensive qualitative analysis methods (Orians et al. 2009).

2. Conceptualizing impacts

Before we could design and implement a new data infrastructure for tracking impacts, we first had to address the lack of ontology to discuss, organize, and catalog them. NIEHS has a history of working with staff to develop program logic models that detail the activities, outputs, and impacts of a specific research program (Engel-Cox et al. 2008; Liebow et al. 2009; Orians et al. 2009). This work has defined key terms as follows: inputs are the resources available to a project; funding, people, facilities, equipments, etc. Activities are actions (verbs) that use available resources (research, identify, build, involve, conduct, create, determine, etc.). Outputs are the direct and tangible products of activities. These can include meetings, agendas, maps, publications, reports, websites, twitter feeds, etc. Impacts are the benefits or changes resulting from activities and outputs. Examples include societal benefits such as improvements to human health or the environment, or positive changes in behavior (NIEHS 2012).

In creating our logic models, we identified broad categories of research outputs and impacts, which have formed the basis of our ontological framework. Output categories included publications, patents, curricula, and training materials. Impact categories included changes to knowledge and behavior, advocacy and policy impacts as well as benefits to the community. An ontological approach developed at the Becker Library at Washington University in St. Louis (Sarli et al. 2010) closely aligned with the categories we used in our logic models, and also informed our ontology.

NIEHS has also developed several products and databases that facilitate tracking and monitoring of research impacts. The Partnerships for Environmental Public Health (PEPH) Evaluation Metrics Manual (NIEHS 2012) provides grantees with a methodology for developing metrics for non-bibliometric activities, outputs, and impacts common to PEPH grantees. Concrete examples of metrics are also provided. Additionally, we recently launched CareerTrac at NIEHS; this is a trainee tracking database that follows a broad spectrum of long-term training outcomes. The structure of the trainee accomplishments in NIEHS’ CareerTrac system (https://careertrac.niehs.nih.gov) also informed our impact ontology.

3. Infrastructure gaps

The existing data structures for research programs (IMPAC II, SPIRES, RePORT) at NIH focus mainly on inputs (financial investments, personnel, and expertise), activities (a project’s abstract and specific aims) and outputs (publications and patents); they are not designed to track and monitor impacts. Recent advances and automated linkages between grants and their bibliometric outputs have enabled faster and more routine use of outputs. While collecting bibliometric outputs is helpful, such assessments do not typically include the knowledge content produced by the research. A structured database system that captures the results of the research is needed to enable, assess, and store information about the impacts of our individual investments as well as aggregate and summarize information about these impacts.

Moreover, some of the data collected by the NIH contain valuable information about grant progress and outcomes but are not structured to facilitate full text searching. For example, annual progress reports submitted by research grant personnel may be downloaded individually from IMPAC II and searched manually, but they are not easily searchable within the main Query, View, and Report module. Similarly, annual notes that scientific program officers are required to submit about grant progress and outcomes are not readily available to or searchable by other staff.

4. High Impacts Tracking System

In response to the needs and challenges described above, NIEHS developed HITS, which is an innovative, Web-based application intended to capture and track short- and long-term research outputs and impacts. Although federal agencies place a high value on evidence-based evaluation, we are unaware of any similar systems at NIH or elsewhere in the USA. HITS imports much of its information directly from IMPAC II, including basic grant information such as funding period, grant number, PI name, Institution, etc. In addition, HITS downloads and imports the text contained in progress reports as well as the notes the program and grants management personnel enter about the grant. HITS provides free-form and structured coding of a wide range of portfolio characteristics as well as outputs and impacts. A dynamic query and reporting infrastructure allows users to access real time grant data in a way that has not been possible before. HITS includes funded grants awarded in Fiscal Year 2007 and beyond.

The four main functions of the system are to search existing data, to display results of the search, add data (in the form of tags or notes), and to summarize information in reports. These functions provide the necessary infrastructure that can support many different evaluation needs, which at NIH range from small ad hoc efforts to large-scale formal evaluations. Below, we describe each of these functions in more detail.

4.1 Searching HITS

The search screen is divided into three main areas: grant information (grant number, institution, principal investigator, program officer, etc.); document search, where the user chooses the artifacts that are searched (progress reports, program notes, title, abstract, specific aims, etc.); and tag search, where the user can search on existing tags within the system (Fig. 1). Tags are structured codes that a user assigns to a particular grant and are discussed further in Section 4.3. The search function can be used to either identify grants that have already been linked to certain tags or to identify grants that the user wants to tag.

Figure 1.

HITS search page.

One key objective of HITS is to make progress reports and associated notes accessible and searchable. In order to receive ‘non-competitive’ funding, each NIH grantee submits an annual progress report in a structured PDF format. Program officers follow a standard protocol to review the reports that requires them to assess the progress of the project and enter ‘signoff notes’ to authorize continued funding. We import all of these artifacts, as searchable text, into HITS (Fig. 2).

Figure 2.

Search terms include various artifacts that are imported into the system.

4.2 Displaying results

Results of a particular search are listed in a tabular format (Fig. 3). The format contains a summary of the tags that have been assigned and the artifacts (progress reports, notes, etc.) that are available for that grant. The user can then drill down into the grant information to see grant details or select a group of grants to tag with a ‘batch tag’.

Figure 3.

HITS search results grid.

Selecting a grant from the list produces the grant details page, which displays all the information about that specific grant in one frame. The user can see and access subprojects, tags, program officer and grants management specialist notes, progress reports, publications (imported automatically), and the grant abstract. From here the user can add data about outputs, impacts, dissemination, or other portfolio attributes.

4.3 Adding data for impact analysis

At the heart of HITS are the tags (or codes) we have identified to delineate and describe outputs and impacts (Table 1). Our previous work with logic models and the Becker Model for measuring research impact (Sarli et al. 2010) heavily influenced the content of these codes.

Table 1.

Output and impact codes included in HITS

Outputs	Impacts
Scientific findings Publications Patents Collaborations Animal models Biomarkers Curricula and guidelines Databases and software Measurement instruments and sensors	Improved health/ disease reduction Exposure reduction Policies and regulations Community benefit Economic benefits

Each of the codes has a unique structure. All contain a description, the name of person who entered the code, and the time and date of entry. Each also has additional fields needed to provide appropriate details for the type of code (examples are shown in Figs 4–6).

Output tag: scientific findings.

Output tag: measurement instruments.

Impact tag: policy or regulation.

Another objective of HITS is to enable NIEHS to track specific portfolio characteristics for each grant in the system. We have been engaging in a parallel process to develop and conduct a systematic coding of our active grant portfolio that will enable us to analyze our research in more detail. Having detailed portfolio codes in the same system with a broad range of output and impact codes will allow us to summarize and aggregate across a variety of factors and will provide timely information that can be used in a variety of settings. Portfolio codes include:

Science Type—captures whether the grant includes basic, applied, social behavioral, or translational types of science.

Exposure Agents—captures the chemical or other environmental exposure that is being studied.

Biological Sample—captures the type of sample collected or tested, for example, blood, hair, mucus, or organ/tissue samples.

Lifestage—captures the stage at which the research focuses, for example, pre-conception, prenatal, adolescence, or older adulthood.

Organism—captures the organism studied, such as human, cellular, rodent, fish, etc.

Social Science Research—captures the type of social science topics and methods referenced, for example, health disparities, or community-based participatory research.

Strategic Plan Goals—captures which NIEHS strategic plan goal is addressed by the research.

Solicitation Type—captures whether the grant was submitted in response to a solicitation with specific goals and objectives and funds to support it, such as RFA, or whether it was an unsolicited application.

We also code the grants in terms of how NIEHS disseminates output or impact information. For example, we highlight key papers each month in the ‘Papers of the Month’ feature in the NIEHS newsletter (http://www.niehs.nih.gov/news/newsletter/). Until now, we have not had a permanent tracking system that enables us to know whose research has been cited. Knowing this distribution may help us identify key programs that are consistent producers of important articles as well as identify research areas that would benefit from broader dissemination. Dissemination codes include materials that have been included in press releases, the annual Congressional Justification, in presentations by senior leadership of NIEHS for Congressional Testimony or other audiences, and research that has been featured on the NIEHS Website (https://www.niehs.nih.gov/research/supported/index.cfm) as Papers of the Month; (http://www.niehs.nih.gov/news/newsletter/2012/4/dert/) or for highlights in American Recovery; and Reinvestment Act reports (http://www.niehs.nih.gov/research/supported/recovery/).

4.4 Reporting

To date, we have developed a few reports to summarize data within the system. We expect others to evolve as the system and our approaches mature. The most valuable report is the one that lists ‘Key Findings’ by program officers. During their annual review of progress reports, program officers now use a specific character string—***Key Finding—in their notes to identify important results. Our intent was to make it fairly simple for program officers to identify important results ‘during their normal course of business’. Adding this simple character string while a program officer is completing their review enables him or her to quickly flag something that is important. Program officers do not have to open a separate database (HITS) to make the notation, saving time and effort. NIEHS analytical staff can then search on the ***Key Findings character string at a later time to code the content of the key finding. An additional report lists data associated with the tags in the system, and can be filtered by Fiscal Year, program officer, or tag. A third report provides an indication of system use by displaying a count of the tags created by each program officer. Additional reports will be designed as more data are entered.

4.5 Benefits and challenges

We believe that HITS will significantly improve our ability to document, access, and evaluate the results of NIEHS-funded research quickly and accurately. Until now, key resources have not been searchable or accessible and we have not had a centralized repository for outcome or impact information. Structured coding of research outputs and impacts will facilitate better analysis of our portfolios and what they achieve. A key purpose of HITS is to access and qualitatively code the expert opinions of our program officers, a benefit because we can more easily get a sense of what they feel is ‘most important’. The types of data we collect are very flexible, so we are better able to capture results that are not always reflected in peer reviewed publications, for example, contributions to publicly available datasets for projects such as the NIH Roadmap Epigenomics program (www.roadmapepigenomics.org/data) or developing an environmental health science curriculum for use by elementary school teachers (http://www.k8science.org). The qualitative data that are captured and organized in HITS can then be paired with quantitative measures of impact, such as citation rates or other bibliometric indicators. As a result, research managers at NIEHS will be able to obtain relevant information for strategic planning, to improve the management of scientific research programs, and to document our achievements and impacts to the US Congress and the American people.

We are aware of international interest in tracking research outcomes. For example, in the UK, the Medical Research Council (MRC) has developed an output and outcome tracking system that reports categories similar to HITS, including publications, collaborations, further funding, career progression, dissemination, influence on policy, research materials, and intellectual property. (Researchfish: http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/index.htm). Known as Researchfish, the system acts as an electronic progress report for researchers funded by MRC. Collecting structured data in dedicated systems such as HITS and Researchfish allows funders to aggregate and better analyze the results of research, as well as track trends over time. So we would hope that both the ontology and infrastructure of HITS is potentially transferrable to a range of other organizations that support scientific research investments—foundations, other federal agencies, international organizations, and the like.

Developing the HITS system has also revealed challenges. One issue is that inputs, activities, and outputs are inherently more ‘tangible’ than short- or long-term impacts of research, and they are more immediately traceable back to specific research funding. Researchers and program staff are sometimes less comfortable attributing impacts (i.e. changes to society, economy, or health) to a specific research enterprise. In part, this is because, it often takes several years for substantive changes to occur and the original program may not be in existence by the time the impacts are realized (Guthrie et al. 2005; Orians et al. 2009; Teles and Schmitt 2011; Graham 2012).

Moreover, after funding ends, grantees may no longer have a stake in tracking and measuring long-term impacts. The complexity of an impact also may make it difficult to directly attribute it to a specific grant, or even to a program (Guthrie et al. 2005; Stuart 2007; Teles and Schmitt 2011; Graham 2012). Measuring impacts over longer timeframes also makes it more likely that contextual factors that are beyond the program’s control—such as administrative rules, political climate, or economic recession—are driving the change, and not the programs hoping to have an effect (TBCS 2001; NIEHS 2012). Our own efforts to track long-term impacts has shown that specifically designed data collection may be needed to review longer term impacts of grants and programs that have ended (Orians et al. 2009), and this is the approach that Researchfish appears to be following in the UK. We will need to continue to explore and test methods to encourage long-term reporting, including consideration for what incentives former grantees have to do this. There may be ways to leverage quantitative approaches, such as through bibliometrics and citation rates as well as expert opinion and review of specific program areas.

Another challenge has been the need to work with program staff to develop consensus about what constitutes an ‘impact’ and how to capture it appropriately. At present we are presenting scenarios and sample coding to program staff and obtaining their feedback about how best to describe grant contributions to specific research impacts. The related issue of inter-coder reliability is also likely to emerge as we begin to use the system more intensely in the coming years. As our use of the system evolves, we expect to develop and rely heavily on coding guidelines to help mitigate this concern. In addition, we intend to document our internal and external feedback regularly and update our coding guidelines, teaching ourselves and normalizing the process as we go.

5. Conclusions

As evaluators of government funded scientific research, we recognize that budgetary pressures are here to stay. We expect demands for greater accountability in government only to increase over time. Historically, data infrastructure at NIH has neither provided access to all existing project information nor accommodated the long-term cataloging of impacts. The HITS tool provides an approach that addresses both these needs, and can potentially be expanded to other institutes, to all of NIH and to other funders. HITS is still very much a work in progress; we continue developing appropriate codes for outputs, impacts, and particularly, portfolios. Effective and efficient reporting mechanisms are also evolving. The intent of HITS is to provide systematic infrastructure of impact/output data that will help us define and measure success. There are likely other ways we can integrate HITS with existing processes within the grant administration and evaluation setting to gain access to tacit knowledge within program and grants management staff, and to leverage existing data and information. Clearly, there is no ‘one right answer’ to the challenges we face, but we hope that HITS represents a step forward. We aim to work closely with other NIH institutes and scientific research programs to further explore HITS and its potential applications across the federal government.