Tracking the impact of the National Institutes of Health Clinical and Translational Science Awards on child health research: developing and evaluating a measurement strategy.

Introduction

The National Institutes of Health (NIH) has an extensive history of supporting child health research through a variety of infrastructure grants, including funding for clinical research centers (CRC), institutional clinical research curricula and training and career development programs for child health researchers, pediatric clinical research networks, and education and meeting activities (1, 2). In 2006, the NIH launched the Clinical and Translational Science Award (CTSA) program with the vision that these enhanced institutional infrastructure grants would better catalyze the translation of new scientific discoveries into improved health of human populations (3, 4). Existing pediatric CRCs and some institutional training and career development programs were subsumed into the CTSAs as linked awards; there was no requirement for a separate child health component. The NIH Reform Act of 2006 changed this framework in order to preserve independent funding and infrastructure for pediatric CRCs (5). Subsequent CTSA Request for Applications (RFA) have allowed for the appointment of a pediatric co-principal investigator under a single CTSA who would have direct authority over a separate budget or would otherwise secure institutional independence of pediatric CRCs with respect to finances, infrastructure, resources, and research agenda. As of 2011, 58 of the 60 CTSAs include child health components and 9 have principal investigators who are themselves pediatricians (6). However, none of the CTSAs have chosen to create a separate budget for child health, which makes it difficult to directly identify the resources devoted to support of child health research in individual institutions and to measure the effectiveness of the CTSA program as a whole in supporting child health research over time.

Concurrent with the funding of the first group of 12 CTSAs in 2006, the CTSA Consortium-Child Health Oversight Committee (CC-CHOC) was established as a part of the broader CTSA Consortium governance structure and was charged with the responsibility of facilitating child health research throughout the Consortium (7, 8). A CC-CHOC workgroup was formed to develop and test a measurement strategy to: 1) enable CTSA institutions to uniformly assess their current levels of support, research activity, training of new investigators, and multicenter collaborations related to child health research; 2) track changes in these parameters over time within institutions; and 3) eventually estimate the impact of the CTSA Consortium, as a whole, on child health research. This report describes the initial development and evaluation of the measurement strategy and presents data on specific metrics from CTSA institutions receiving CTSAs during 2006-2009.

Methods

Definitions

The workgroup used the NIH definitions for clinical and translational research (9) and for inclusion of children as participants in research involving human subjects (10) to define child health research as, “all clinical and translational research involving subjects less than 21 years of age that impacts children’s health or addresses childhood diseases, including maternal-fetal research, as well as studies involving infants, children and adolescents.” Research that primarily focused on adults but in which a few older adolescents were enrolled was excluded.

Development of the Metrics

From 2008-2009, workgroup members representing 12 CTSAs participated in monthly teleconferences during which measurement domains and specific metrics were proposed, discussed, and selected. Decisions were made by consensus during all phases of this process. The workgroup reviewed the CTSA Consortium Strategic Goals (11) and chose the following measurement domains, which aligned with these Strategic Goals: I) funding, infrastructure support, and research activity; II) career development of new investigators; III) collaborative research; and, IV) leadership. Workgroup members were then asked to propose specific metrics in each of these domains and then to rank prospective metrics using the following criteria: 1) capacity to utilize current systems to capture existing data; 2) generalizability across most institutions; and 3) ability to identify “new value” attributable the CTSA infrastructure. Metrics that were ranked highly across these criteria were selected for further development. No metrics were added or deleted thereafter. Several members of the workgroup (S.L.B., C.D.S., W.C.H.) collaborated to draft a written description of the metrics with specific definitions of the numerator and denominator for each metric and instructions for data collection. Other workgroup members provided critique of the draft and comments were incorporated into a revised version which was presented to the workgroup in the spring of 2009. During the summer of 2009, members of the workgroup pilot-tested procedures for collecting the required data in 6 of their own institutions and the metric definitions and data collection procedures were revised based on their recommendations. The proposal to implement the data collection strategy for the metrics was reviewed and approved by the CTSA Consortium Steering Committee, a body that consists of all participating CTSA principal investigators and NIH CTSA representatives.

Collection of Metric Data

In November 2009, the final version of the metrics (Table 1) and a worksheet for data collection with instructions were distributed by e-mail to the CC-CHOC member or other relevant contact at each CTSA institution. For each metric, the institution recorded the numeric data and provided qualitative comments, as needed, about how the metric was interpreted and data were collected, as well as any barriers or difficulties encountered in this process. An open-ended field allowed for additional qualitative descriptions of institutional enhancements to promote the conduct of child health research. Given the need to assess administrative financial and IRB databases, data were collected primarily by administrative personnel under the supervision of the CC-CHOC member and either entered directly by each institution using a confidential access code into a Research Electronic Data Capture (REDCap, Vanderbilt Institute for Clinical and Translational Research, VICTR, Nashville, Tennessee) database (12) or transmitted confidentially to VICTR and subsequently entered into the database. Four follow-up e-mails reminders were sent to CC-CHOC members at all institutions, but non-responding institutions were not contacted individually either by e-mail or telephone.

Table 1

Measurement Domains and Description of Metrics

Measurement Domain	Description of Metric a
I. Support, infrastructure, and research activity b
Overall funding	Proportion of CTSA grant dollars allocated in support of CH investigators c, d
Pilot grants	Proportion of CTSA-funded pilot grants allocated in support ofCH research
CRC utilization	Proportion of active inpatient and outpatient studies conducted in the CRC involving pediatric subjects d
Pilot grants	Proportion of CTSA-funded pilot grants allocated in support of CH research
CRC utilization	Proportion of active inpatient and outpatient studies conducted in the CRC involving pediatric subjects d
Research activity	Proportion of active IRB-approved studies that involve pediatric subjects
	Proportion of active IRB-approved and funded studies that involve pediatric subjects
II. Career development of new investigators b	Proportion of mentored clinical and translational research training  awards held by pediatric/CH trainees and faculty e
III. Collaborative research b	Number of multicenter studies involving pediatric subjects in which more than one CTSA participates f
IV. Leadership	Proportion of CSTA leadership positions held by pediatric/CH investigators g

CH, child health; CRC, clinical research center; CTSA, Clinical and Translational Science Award; IRB, Institutional Review Board; NIH, National Institutes of Health

Numeric data values needed to calculate proportions were requested and reported (see Table 2)

Aligned with CTSA Consortium Strategic Goals 1-3 (11)

Including such services as pilot grants, education/training, research infrastructure, and study design services.

Including the following funding sources: NIH, investigator-initiated industry, industry-initiated, foundation, other

Including NIH KL2, K08, K12, and K23 awards, foundation and academic society career development awards

Including studies with co-investigators at other CTSA sites or with collaborative or subcontract arrangements with other CTSA sites

Individuals with substantial influence on directing strategy and resource allocation for the institutional or national CTSA program (e.g., principal investigator, co-principal investigator, program director, associate director, advisory committee member, administrative leader)

Data and Statistical Analyses

REDCap and SPSS, version 18 (IBM SPSS Statistics, formerly Predictive Analytics SoftWare Statistics, Armonk, New York) were used to generate descriptive statistics. Qualitative data were abstracted and synthesized by members of the workgroup (W.C.H., C.D.S., J.W., S.L.B.) and reviewed by the other members. Fisher exact tests were used to compare institutional characteristics between responding and non-responding CTSA institutions.

Results

All 46 institutions that received CTSAs from 2006-2009 received an invitation to participate; 37 (80%) institutions provided responses. Comparing responding and non-responding institutions, there was no evidence of a difference in geographic distribution by region (P=0.96), year of award (P=0.75), and the amount of the CTSA in dollars by quartiles of percentile rank (P=0.33). A children’s hospital was listed as a partner institution and a pediatrician was the CTSA Principal Investigator for 34 (92%) and 5 (14%) responding institutions, respectively, vs. for 6 (67%, P=0.08) and 0 (P=0.57) of non-responding institutions, respectively.

The numeric data needed to calculate each metric were provided by most of the responding institutions (Table 2). Except for the collaborative research metric, data were provided similarly for child health research, investigators, or pediatric subjects (the numerators) and for the CTSA or institution as a whole (the denominators). Data regarding the number of multicenter studies involving more than one CTSA were the most difficult for institutions to provide, both for studies involving pediatric subjects and for studies involving subjects of any age.

Table 2

Institutions Providing Data and Descriptive Data Regarding Metrics

	Value relating to CH research/investigatorsor involving pediatric subjects		Value relating to the CTSA or institutionas a whole
Measurement Domain/Metric a	Numberprovidingdata (%)	Median (range)	Numberprovidingdata (%)	Median (range)
I. Support, infrastructure, and research activity
Overall funding, dollars	36 (97%)	643,379 (69,994 – 17,564,022)	36 (97%)	6,727,057 (1,365,625 – 52,750,544)
Pilot grants, number	35 (95%)	4 (0 – 34)	35 (95%)	20 (0 – 288)
CRC utilization
Active CRC inpatient and   outpatient studies, number	35 (95%)	32 (1 – 263)	35 (95%)	159 (21 – 449)
Research activity
IRB-approved studies, number	35 (95%)	335 (2 – 1596)	31 (84%)	1483 (55 – 6568)
IRB-approved and funded   studies, number	33 (89%)	295 (4 – 983)	27 (73%)	1371 (45 – 5228)
II. Career development of new investigators
Mentored clinical and translational  research training awards, number	33 (89%)	2 (0 – 170)	28 (76%)	48 (0 – 625)
III. Collaborative research
Multicenter studies with ≥1 CTSA  institution participating, number	29 (78%)	3 (0 – 208)	-- b	-- b
IV. Leadership
Leadership positions, number	37 (100%)	4 (1 – 13)	36 (97%)	23 (5 – 75)

CH, child health; CRC, clinical research center; CTSA, Clinical and Translational Science Award; IRB, Institutional Review Board

See Table 1 for description of each metric

Few institutions were able to provide data for the institution as a whole

The numeric data values reported by individual institutions ranged widely (Table 2). Likewise, there was wide variation in the calculated metrics (Figure 1), most notably in the metrics related to active CRC studies, mentored research training awards, and CTSA leadership positions involving child health researchers. There was diversity among the institutions that represented the upper bounds of the ranges, with 6 different institutions represented (the upper bound was represented by 1 institution for the metrics related to overall funding, active CRC studies, and mentor research training awards, 1 institution for both metrics related to research activity, and 4 different institutions for the other metrics). Nonetheless, despite the overall variation, the medians of the metrics were generally consistent (Figure 1, range of medians, 0.11-0.24) and the central portions of their distributions, as represented by the interquartile ranges, were relatively narrow.

Figure 1

Distribution of Metric Proportions across CTSA Institutions

CRC, clinical research center; CTSA, Clinical and Translational Science Award; IRB, Institutional Review Board

Boxplots show the distribution of metric proportions reported by the 37 responding institutions (minimum, 25%, 50%, 75%, maximum). The number and percent of institutions reporting data for each metric is indicated below the metric label. Table 1 provides the full description of each metric. Metric III. Collaborative research (multicenter studies with ≥1 CTSA institution participating) was not included because few institutions were able to provide data for the institution as a whole (see Table 3).

The number of institutions included in each distribution is: overall funding 36, pilot grants 33, active CRC studies 35, IRB-approved studies 31, IRB-approved and funded studies 27, mentored clinical/translational research training awards 27, leadership positions 36.

Challenges in collecting the numeric data were due primarily to issues related to the interpretation of the definitions and the data sources of the metrics (Table 3). Differences between NIH and IRB definitions of child health research and whether to include other sources of support (e.g., supplemental awards, institutional matching funds) were identified commonly. Less common, but more fundamental, were issues of whether studies that relate to child health but do not involve children as research subjects directly should be viewed as child health research, such as healthcare services research and maternal health studies. Inadequate sources of data were a major challenge due to the lack of variables that clearly identify child health research studies and to difficulties synchronizing the records of separate institutional databases, such as institutional review board (IRB) and grant funding databases, to perform queries using the combined data fields from each database.

Table 3

Description of Issues Identified in Collecting Data Regarding the Metrics a

Measurement domain /type of issue	Description of Issue a
I. Support, infrastructure, and research activity
Definitions b	NIH definition defines a child as a person less than 21 years of age, whereas many IRBs define a child as less than 18 years of age
	Support may also include supplemental awards and institutional matching fundsStudies that do not utilize the CRC may benefit from use of other CTSA resources (e.g., study design/biostatistics consultation)
	Studies that do not involve subject contact (e.g. health services research) or contact with children (e.g., maternal health research) may be viewed as CH research
Data Sources c	Support for infrastructure (e.g., facilities, core laboratories) is difficult to trackData fields in grants management databases are insufficient to identify all CH studiesIRB, grants management, and CRC databases are rarely synchronized
II. Career development of new investigators
Definitions b	CH investigators in non-medical schools (e.g., public health, pharmacy) may also receive these awards
Data Sources c	Non-federally funded mentored research training awards are not recorded reliably in existing databases
III. Collaborative research
Definitions b	Sub-contracts and studies funded by non-federal sources may represent additional evidence of collaboration
Data Sources c	Data fields in grants management databases are insufficient to identify the participating sites in multicenter studies, to apportion fractional credit to sites that are identified, and to ascertain whether participating sites are CTSA institutions
IV. Leadership
Definitions b	Positions on external and internal advisory committees and CTSA Consortium-related committees and directorships of core laboratories and training grants may also represent important leadership roles

CH, child health; CRC, clinical research center; CTSA, Clinical and Translational Science Award; IRB, Institutional Review Board; NIH, National Institutes of Health

Variability in how individual institutions interpreted and dealt with the issues identified with respect to definitions (**) and data sources (†) may have resulted in underestimates in individual institutions and greater variability among institutions.

May not be all-inclusive or may be interpreted differently among institutions

May not capture all relevant activity

Institutions provided additional qualitative statements of how the CTSA prompted the development of new institutional strategies and initiatives to enhance child health research, including: establishing multidisciplinary child health research teams or committees to better integrate child health research into the CTSA, enhancing linkages with basic science researchers, and developing specific foci of expertise in child health research; using evidence of CTSA support to obtain additional matching funds to support child health research and new child health investigators; using indirect funds to increase investment in child health research; recruiting child health researchers to compete for pilot grants; establishing a community-based child health research network; and inaugurating an annual pediatric research day event.

Discussion

The metrics presented in this cross-sectional survey illustrate that institutions that received CTSAs from 2006-09 provide substantial support for child health research through overall funding and pilot study grants, CRC utilization, and mentored research training awards for child health investigators. In these institutions, roughly a quarter of all IRB-approved studies include pediatric subjects and nearly a fifth of CTSA leadership positions are filled by pediatricians. These findings may be positively skewed by the trend toward a higher proportion of institutions with a children’s hospital listed as a partner institution among responding vs. non-responding institutions.

Although these findings are generally positive, additional investment by CTSAs in child health research will be required to achieve the major breakthroughs necessary to improve the health of children and those children into adulthood. As reported in this study, the proportions of overall funding and pilot study grants devoted to child health research in CTSA institutions are consistent with the proportion of NIH funding targeted toward child health research reported recently (13). However, the proportion of overall NIH funding for child health research has declined by nearly a third over the past two decades (13). Moreover, leaders in the pediatric research community have identified major unmet needs for child health research that must be addressed in order to improve the future health and well-being of children in the United States (13, 14).

These results should also be viewed in the context of the limitations of this first attempt to assess the impact of the CTSA program on child health research. The lack of pre-award data and the cross-sectional design of this study, which occurred 1 to 4 years after the receipt of the CTSA in the cohorts of institutions receiving awards during 2006-09, do not allow us to assess trends in the support for child health research in relation to the CTSA in individual institutions or across the Consortium as a whole. Although we provided instructions to standardize data collection, issues related to metric definitions and data sources (Table 3) may have resulted in underestimates or greater variability with respect to the metric data reported by individual institutions. We did not perform formal assessments of the reliability of data collection at individual CTSA institutions or the validity of the data in relation to externally verified measures, although we plan to do so in the future. The difficulties we encountered with definitions and data sources for the metrics illustrate general challenges inherent in quantifying the impact of the CTSAs on clinical and translational research in general, and on child health research in particular. Some of these barriers can be addressed easily. For instance, we have already refined the definitions of the numerators and denominators to be more detailed and specific. Other barriers, such as those involving data sources, are likely to require new or revised databases with additional data fields and improved linkages. The difficulty we encountered in capturing information about research collaborations across CTSA institutions from existing grants management databases deserves particular attention because this metric was one of the most difficult to quantitate and yet it uniquely reflects the power of the Consortium.

Nonetheless, the metrics presented in this study provide an important starting point for evaluating the impact of CTSAs on child health research. First, the metrics were developed intentionally to align with the strategic goals of the CTSA Consortium. Therefore, over time, they should reflect enhancements to the clinical and translational research infrastructure that occur in individual institutions and throughout the Consortium. Second, they provide an opportunity for CTSA institutions to assess their current support for child health research in relation to the Consortium as a whole. Third, the metrics provide a standardized mechanism for individual institutions to track the effect of their efforts to enhance child health research over time through changes in either the numeric data values (i.e., the numerators for the metrics), the calculated proportions, or both. Institution-specific tracking of these metrics is important since factors that limit the validity of comparisons across institutions—particularly those related to sources of data—are likely to be less variable within institutions over time. Moreover, by creating an institutional process for measuring child health research efforts, CTSA institutions can examine if they are achieving the child health research goals they set. Fourth, the high response rate indicates there is support for this effort across the Consortium.

To better understand the impact of CTSAs on child health research, we also need better approaches for describing child health research portfolios, including the spectrum of translational research (i.e., T1 through T4) conducted in CTSA institutions(15), research that crosses populations (e.g., women’s health research with outcomes relevant to child health, studies of adult diseases that have their antecedents in conditions affecting children), and research that has an important impact on child health, but does not involve pediatric subjects directly.

As the CTSA Consortium matures, its strategic goals evolve, and the definition of child health research is refined, the metrics used to describe the impact of CTSAs on child health research may need to be adapted. We hope these data will also inform a discussion of what may be considered adequate (or optimal) performance with respect to these metrics. Finally, one of the aspirations of this effort is to develop metrics to evaluate the impact of CTSAs on child health, not just child health research. Indeed, the process of developing such a measurement strategy, in and of itself, demonstrates one of the potential strengths of the CTSA Consortium in facilitating child health research (8).