Outcome evaluation of the National Cancer Institute career development awards program.

The National Cancer Institute (NCI) career development (K) awards program supports investigators to develop their cancer research programs and achieve independence. The NCI Center for Cancer Training conducted a K program evaluation by analyzing outcomes of awardees and individuals who applied to the program but were not funded. The evaluation covered seven NCI mechanisms (K01, K07, K08, K11, K22, K23, and K25) between 1980 and 2008. Descriptive statistics and regression modeling were performed on the full cohort (n = 2,893 individuals, 4,081 K applications) and a comparison cohort described herein. K awardees proportionately received more subsequent NIH grants and authored more publications, and time to first R01 grant was unaffected. Of those not pursuing research, K awardees were more likely to participate in activities signaling continued scientific engagement. The NCI K program had a positive impact, not only on participants' biomedical research careers but also on achieving outcomes significant to the scientific enterprise.

Introduction

The National Cancer Institute (NCI) is committed to supporting the career development of cancer researchers and physician scientists by offering multiple training and career development programs for graduate students through more experienced investigators. Some grant mechanisms are intended to facilitate the transition from mentored scientist to independent investigator or provide protected time for newly independent investigators to develop their research programs, while other grants go directly to research institutions to educate and train predoctoral students and postdoctoral or clinical fellows. The spectrum of research career development opportunities NCI supports reaches scientists, clinicians, and other health professionals conducting basic, translational, and clinical cancer research.

The NCI career development (K) awards program has a longstanding history and has evolved over time to address the changing cancer research workforce needs. NCI K mechanisms have been consolidated or created to target specific scientific disciplines, career paths, or populations shown to be underrepresented in biomedical research. Currently, the NCI Center for Cancer Training (CCT) supports ten K grant mechanisms that vary in discipline, program focus, and applicant eligibility. In fiscal year 2011, the CCT supported 365 K awards at an approximate cost of $65 million.1

Despite its rich history, mixed perceptions exist in the research community regarding the value of the K program experience. These concerns seem to focus predominantly on the K mechanisms that provide early career scientists and clinicians 3 to 5 years of support, with the overarching goal of further developing their cancer research careers or facilitating their transition to independence. The research career successes of K awardees have been questioned, and there is an undocumented perception that participation in these K programs delays an individual’s progression to receipt of their first R01, the NIH gold standard for research independence. In contrast, unpublished informal analyses and assessments have demonstrated success of the NCI K program, but no systematic, comprehensive evaluation of the program had been conducted thus far. Based on findings from a feasibility study, we performed an outcome evaluation of the NCI K awards program to define the population of individuals who applied for NCI K awards, to investigate the reality of the above perceptions, and to quantitatively determine the impact receipt of a K award had on pursuit of a biomedical research career and on contributions to the scientific enterprise. The evaluation examined five individual K award mechanisms currently offered by the NCI CCT (K07, K08, K22, K23, and K25) and two K mechanisms previously phased out by NCI CCT (K01 and K11).2 Mechanisms not included were mentoring awards for more senior investigators (K05, K24), a newly established transition mechanism (K99/R00), and those grants administered by NCI’s Center to Reduce Cancer Health Disparities (a subset of K01, K08, K22, and K23 awards).3 Descriptions of the K mechanisms can be found in Online Resource 1. Evaluation highlights are presented in this report, but a description of the full study and its findings are available at http://cancer.gov/researchandfunding/cancertraining/KAward-Evaluation-Report.

Methods

Data Sources

The NIH Information for Management, Planning, Analysis, and Coordination (IMPAC II) database was the primary source through which information about NCI K applicants was obtained, including basic demographic information and data regarding prior and subsequent NIH grant applications. The Association of American Medical Colleges Faculty Roster and the National Science Foundation’s Doctorate Records File were used to supplement demographic information from IMPAC II and to provide additional fields to augment these analyses. The International Cancer Research Partnership (ICRP) database was used to identify subsequent awards from non-NIH sources, including the Department of Defense Congressionally Directed Medical Research Program, American Cancer Society, Prostate Cancer Foundation, Susan G. Komen for the Cure, California Breast Cancer Research Program, and the Canadian Institutes of Health Research. The Thomson Reuters ScienceWire® grant catalog was used to match NCI K applicant names to grants sponsored by the U.S. Department of Energy (DoE) and National Science Foundation (NSF). Subsequent publications were collected from the National Library of Medicine’s MEDLINE. The ClinicalTrials.gov database was used to match K applicant names to those of key personnel. IMPAC II and data obtained from the Federal Interagency Databases Online (FIDO.gov) database were used to collect information about participation in NIH advisory groups and other Federal advisory committees. The HealthLink/Lodestone database was used to identify K applicants registered as clinical practitioners, while the NIH Employee Directory (NED) was used to identify NIH staff. K applicant membership in selected professional societies, the American Association for Cancer Research (AACR), the American Society of Clinical Oncology (ASCO), and the Federation of American Societies for Experimental Biology (FASEB), was determined through name matches to society member databases. The as-of dates for data sources are noted in Online Resource 2.

Comparison Cohort Derivation

The comparison cohort was constructed using regression-discontinuity design based on K application priority score,4 an NIH-wide measure of application quality. The comparison cohort had equal distribution of K awardees (n = 293) and non-awardees (n = 293) whose priority scores clustered around the funding cutoff, the point at which applicants had comparable chances of their K applications being funded. The comparison cohort excluded applicants with maximal chances of funding and those who were least likely to receive funding. Such a design permits approximation of relatively unbiased estimates of program effects [1].

Outcome Analyses

For all individuals in the study, outcomes were measured starting with the fiscal year following the first NCI K award (for those with awards) or the last unsuccessful NCI K application (for non-awardees). Prior grant support was measured up to the fiscal year preceding the first award or last unsuccessful application.

For individuals who received a subsequent R01 award as of FY 2011, the time to R01 value is the numerical difference between the fiscal year of that first R01 award and the fiscal year after their first NCI K award or their last unsuccessful NCI K application. According to this rule, individuals who received an R01 in the fiscal year immediately following their K award, or last application, would have a time to R01 value of zero.

To measure publication productivity for each individual, the number of publications in each fiscal year after their first NCI K award or last unsuccessful NCI K application was recorded. Several independent but overlapping matching rules were used to identify MEDLINE publication records in which a study applicant appeared as an author. To be considered for matching, the publication date had to be at least 1 year after the application date of the last K application for that applicant. The upper bound for the publication date was April 1, 2011. These rules are presented in more detail in Online Resource 3. The matching process was conservative, favoring precision over recall. This approach results in high confidence in the papers assigned to individuals, but means that some publications for individuals are missed.

Data from other non-IMPAC II data sources used a baseline name match, with corrections applied as necessary based on quality checks of the matching results. The baseline rule required an exact first and last name match, and if a middle name was present in both IMPAC II and the other data source, the first characters in each string had to match. Common names were determined by a weighted average of the first and last name frequencies as measured in IMPAC II, with a threshold determined by examining a sample of names. All data containing personally identifiable information were handled and maintained in conformance with the Privacy Act and applicable legal, regulatory, and NIH policy requirements regarding privacy.

Statistical Analyses

Unless otherwise noted, Fisher’s exact two-tailed tests were performed for statistical analyses, using p values <0.05 as the threshold for significance. All tests were conducted using the statistical computation software R, version 2.13.1.

Results

Demographics

Applicant Demographics

The conceptual framework, or logic model, for the evaluation is presented in Fig. 1. Applicant demographics and characteristics of the K program were analyzed, as well as characteristics of the applicant institutions to provide context. Outcome data in three main areas were collected including subsequent research funding and time to first R01 grant, publication activity, and career appointments or involvement in the biomedical research enterprise. Program impact was assessed with regression modeling on a subset of awardees and applicants, exploring a variety of indicators.

Fig. 1

Logic model of NCI K awards outcome evaluation. The logic model highlights K program inputs, activities, outcomes, and impact, as well as contextual factors. The inputs include the features (demographics) that define applicants to the NCI K program, as well as features of the K mechanisms. Activities include the actions that a funded researcher would take to further their research training and career plans, and context refers to specific features of the past and present environment in which program participants are functioning. Outcomes include measures that might be attributed to participation in the NCI K program and are divided into three broad categories of subsequent funding, productivity, and career appointments. Impact is assessed by comparing outcomes of closely matched cohorts of K awardees and non-awardees and examining proxies of scientific research and engagement

Demographic and career outcome data were collected on K awardees and K applicants who applied to the program but were not funded (non-awardees), spanning seven NCI K mechanisms offered between 1980 and 2008. Table 1 outlines the evaluation cohort5 and provides a summary of application and award volume, numbers of unique applicants and awardees, and percent awarded. Across the seven NCI K mechanisms, there were a total of 4,081 applications, of which 1,224 were awarded since 19806 for a funding award rate of 30 %. The overall success rate for new NCI research project grants in fiscal years 2002–2011 ranged from 12.6 to 28.4 % in comparison [2], with the average being 20.7 %. When examining unique applicants, 1,206 individuals received K awards out of 2,983 applicants, with an applicant funding rate over 41 %.

Table 1

Scope of K mechanisms included in the outcome evaluation

Mechanism	Years analyzed	Total applications	Total awards	Percent awarded, applications	Applicants (primary mechanism)	Awardees	Non-awardees	Percent awarded, applicants
K01	1997–2007	624	153	24.5	479	152	327	31.7
K07	1980–2008	825	274	33.2	562	274	288	48.8
K08	1984–2008	1,638	515	31.4	1,176	514	662	43.7
K11	1987–1996	216	86	39.8	166	86	80	51.8
K22	1998–2008	324	73	22.5	200	57	143	28.5
K23	1999–2008	366	98	26.8	254	98	156	38.6
K25	2000–2008	88	25	28.4	56	25	31	44.6
Total	1980–2008	4,081	1,224	30.0	2,893	1,206	1,687	41.7

We analyzed qualifying degree type, field of study or medical specialty, race and ethnicity, prior NIH grant support, and institutional characteristics of K awardees and non-awardees. The majority of applicants to mechanisms focused primarily on the non-clinical sciences (e.g., K01, K07, and K25) held PhDs, while those with a more clinical focus (e.g., K08, K11, K23) included a larger proportion of applicants with MDs. Individuals with MD/PhDs were predominantly in the K08 and K22 mechanisms. When comparing awardees and non-awardees, no significant differences were found based on degree type. Across mechanisms, degree field of study and clinical specialty of the applicants matched program focus. Of those NCI K applicants with medical degrees, the top three medical specialties were Internal Medicine, Medical Oncology, and Hematology Oncology. Among K07 applicants, the most common degree fields of study were Epidemiology, Clinical Psychology, and Public Health, reflecting emphasis of the K07 mechanism on behavioral aspects of cancer prevention, control, and population science.

No conclusions could be drawn on K applicant race and ethnicity in this cohort due to the relatively large proportion of individuals with unknown or undeclared status (approximately 30 % for all applicants, including 6 % of awardees and 36 % of non-awardees). Race and ethnicity are voluntarily reported on NIH grant applications. Moreover, a complete picture of the contribution of race and ethnicity to K award success and future outcomes would need to include analysis of awards administered by NCI’s Center to Reduce Cancer Health Disparities, which were not included in this evaluation.

Close to half the applicants to the K program had prior NIH training support which included appointments on National Research Service Awards institutional (T) grants, individual fellowships (F), loan repayment contracts (L), and research7 or other NIH support. Applicants with prior T grant support had 1.25× higher odds of receiving a K award (p = 0.0051) compared to applicants without prior T support. For applicants with a prior F award, the odds ratio was higher (1.28×) compared to applicants without prior F support, but was not statistically significant, presumably due to the small number of F awards in the cohort. We tested the impact of having participated in multiple prior training grants or fellowships using a three-group chi-square and found that having more than one training grant or fellowship had no effect on the probability of K award receipt.

Applicant Institution Demographics

We examined two characteristics of the institutions from which NCI K applications were received: whether an institution had an NCI Cancer Center designation, and the level of overall annual NCI funding the institution received during the evaluation period. The NCI Cancer Center designation is granted to institutions with research programs that foster interactions between basic laboratory, clinical, and population scientists through shared services, technologies, and other scientific infrastructure. Across K mechanisms, the majority of applications (71 %) were received from and awards (74 %) granted to institutions with NCI-designated Cancer Centers. Affiliation of an applicant institution with an NCI-designated Cancer Center was associated with improved odds for receiving a K award (1.36×, p = 0.0005). Additionally, applicant institution funding level was strongly correlated with probability of receiving a K award. Applicants from institutions receiving $10 million or more annual NCI funding had 1.64× higher odds of receiving a K award (p = 4.83 × 10−10) than applicants from other institutions.

Gender

Gender distribution and K application success were analyzed to better understand female representation in the NCI-supported biomedical workforce. In 2000, men received 55.2 % of biological and biomedical sciences PhDs and women received 44.8 % [3]. However, data from 2010 show that the proportion of men to women receiving biological or biomedical sciences PhDs was 47.6 to 52.4 %, respectively, demonstrating that the gender gap in this field appears to have closed within the last decade [3]. In contrast, 2008 data show that 55.8 % of MD degrees were earned by men and 44.2 % by women [4]. Across K mechanisms and over almost three decades of K program history, women comprised 37 % of program applicants and 37 % of awardees, indicating that, although women applied in fewer numbers, the proportion of women who received K awards was commensurate with the proportion who applied. When examining the proportion of female and male applicants over the final 2 years of this evaluation, the percentage of female and male applicants more closely reflected the near balance seen in the biomedical workforce data above. The gender similarities in K awardees mirrored the finding in the NIH pool of newly competing R01s for first-time investigators—success rates for men and women are equivalent and any gender disparities in receipt of new NIH awards are attributable to differences in application rates rather than differences in success [5, 6].

We also examined resubmission rates for K applications by gender. Our initial analysis showed that female applicants were 1.2× more likely to resubmit a K application than male applicants (p = 0.03). However, when the data were stratified by mechanism, the higher odds ratio was driven primarily by the K07 applicant pool, in which the majority of K07 awardees (65 %) and non-awardees (63 %) were female. This is consistent with the K07 program’s focus on PhD applicants in areas of prevention, control, and behavioral and population sciences, which have higher percentages of female degree recipients.8 When K07 applicants were excluded, there was no significant difference in resubmission rates by gender.

Subsequent Research Funding

We incorporated a variety of metrics to assess career outcomes and performance of K awardees compared to non-awardees subsequent to the K program. Unless otherwise indicated, outcome results focus on the K program as a whole, aggregated across the seven evaluated mechanisms. For this and several other analyses, we examined K awardees and non-awardees in both the full cohort and a comparison cohort consisting of applicants matched on K application priority score (described in “Methods”). The comparison cohort demographics were similar to the full evaluation population and are presented in Online Resource 4.

One indicator of scientific research independence is the attainment of subsequent NIH funding. As shown in Fig. 2, proportionately more K awardees in the comparison cohort were awarded grants from NIH Institutes and Centers (including NCI) than non-awardees (56 % for awardees, compared to 43 % for non-awardees; p = 0.0038). The awarded grants included P01s, R01s, institutional training grants, U01s, and other research project grants. This difference was also observed in the full cohort of awardees and non-awardees as well (59 % compared to 36 %, respectively) and reflects findings of the NIH-wide mentored K evaluation, which analyzed the K01, K08, and K23 mechanisms [7]. We determined that more K awardees in the comparison cohort received subsequent grants from NCI than non-awardees (43 % compared to 29 %, p = 0.0008), a finding also observed in the full cohort.

Fig. 2

Subsequent NIH grant activity of K applicants. Individuals are represented only once in each bar in the highest category of grant activity achieved. The solid blue bar represents those who received subsequent NIH grant funding, and the red cross-hatched bar indicates those who applied but were unsuccessful. The dotted green and striped purple bars show individuals who had other activity (such as only a type 5 non-competing renewal award) or did not have any subsequent applications to the NIH, respectively. Analysis includes funding from all NIH Institutes and Centers, including NCI. For the comparison cohort, n = 293 for both awardees and non-awardees; for the full cohort, n = 1,206 for awardees and n = 1,687 for non-awardees

Although not depicted in Fig. 2, we analyzed subsequent grant activity by ordering grants into 12 achievement, or “high water mark,” levels as a surrogate for grant funding longevity, and each NCI K applicant was placed into the best level according to their subsequent grant record. For instance, a K applicant who subsequently became a principal investigator (PI) of an institutional training grant (T32) was ranked at a greater high water mark than a K applicant who received an individual R01, since T32 eligibility criteria require the PI to have NIH R01 or R01-like funding. Similarly, a K applicant who subsequently received an R01 was ranked higher than an individual who applied for an R01 but did not receive funding. K awardees in both the full and comparison cohorts were more likely to be at a greater high water mark (e.g., receive R01s, U01s, etc.) than non-awardees (data not shown).

Additionally, we investigated grant funding from Federal agencies including DoE, DoD, and NSF and from funding organizations in the ICRP. Although the numbers of K awardees and non-awardees who received such funding were too small to draw definitive conclusions, it was clear that K applicants successfully competed for cancer-related research funding from sources other than NIH (data not shown).

Time to First R01

One question that has been raised is whether time spent in the K program impedes timely career progression. We explored this notion by calculating the length of time to first NIH R01 grant from receipt of K award or from the last unsuccessful K application (for non-awardees). Overall, the mean time to R01 for awardees and non-awardees was within 1 year of each other and the distributions had substantial overlap. A Mann–Whitney test of median differences for all mechanisms combined in the comparison cohort revealed no significant difference in time to R01 for K awardees and non-awardees (4 and 3.5 years, respectively). The only significant difference in the full cohort was detected in the K01 mechanism, in which non-awardees attained their first R01 in 3 years versus the awardees’ 3.5 years (p = 0.02). Yet, the K01 applicants’ median time to R01 was still shorter than the comparison cohort median for all K mechanisms combined. These findings reflect those seen in the NIH mentored K evaluation [7]. Additionally, the median age at K application for awardees and non-awardees was similar, between 36 and 37 years. Therefore, we gauge the age at first R01 for K awardees and non-awardees to be 40–41 years, in line with the age of NIH first-time R01equivalent investigators [8].

Publications

In addition to grant funding, peer-reviewed publications are a relevant indicator of subsequent research activity. Across all mechanisms combined, a significantly larger proportion of awardees had subsequent research publications than non-awardees (89 versus 59 %, respectively; p = 2.2 × 10−16). In addition to awardees being more likely to publish, among those who published, the average and median number of publications per awardee were significantly higher than non-awardees in the K01, K07, and K23 mechanisms (data not shown). The other K mechanisms showed similar trends, but the data did not reach statistical significance.

Research Activity and Engagement in the Scientific Enterprise

Since the overall goal of the K program is to support individuals as they establish independent cancer research careers, we created a composite outcome to serve as a proxy for being a funded researcher. This composite outcome included the following subsequent activities, with each bearing equal weight: being awarded a competitive grant from the NIH, DoE, or one of the ICRP organizations or being a key personnel on a clinical trial registered at ClinicalTrials.gov. In the comparison cohort, 182/293 (62 %) of K awardees were funded researchers as measured by the above proxy, compared to 149/293 (51 %) of non-awardees (p = 0.0076). This finding was also true in the full cohort, where 797/1,204 (66 %) of K awardees had funded research in comparison to 758/1,685 (45 %) of non-awardees (p = 1.0 × 10−29).9 Therefore, K awardees had significantly improved odds of having subsequent funded research or contributing to research in clinical trials.

It is unreasonable to expect that all K awardees will continue in funded research careers, especially given the current sustained flat Federal investment in biomedical research and limited availability of tenure track faculty positions. The career development opportunities provided by NCI’s K program may prove valuable for those individuals who, although they are not conducting funded research, continue to participate in the biomedical research enterprise in some capacity. To determine whether K awardees continued to work in broader roles in the scientific community, we developed a composite career outcome variable and referred to it as being “engaged” in the scientific enterprise, with each of the included outputs bearing equal weight. All individuals not captured in the above funded research composite were classified as being engaged or not. The engaged proxy included individuals who served on Federal advisory committees or review panels related to biomedical research, published subsequent research papers, matched to the NED, were on the current membership rolls of selected scientific professional societies (AACR, ASCO, or FASEB), or were listed in the HealthLink/Lodestone database as a health practitioner. In the comparison cohort, 102/111 (92 %) of K awardees were subsequently engaged in the scientific enterprise following their K award, compared to 110/144 (76 %) of non-awardees (p = 0.0012). Again, the same trend was seen in the full cohort where 379/407 (93 %) of K awardees were engaged, compared to 665/927 (72 %) of non-awardees (p = 8.2 × 10−21).

It may be argued that inclusion of individuals who are registered health-care practitioners and/or professional society members reflects passive involvement but not active scientific engagement. Therefore, we refined the engaged criteria in two different analyses to include only those who (1) served on Federal advisory committees or review panels or published subsequent research papers, or matched to NED or (2) met the above criteria or were listed as current members of AACR, ASCO, or FASEB. In all analyses, K awardees in both the comparison and full cohorts had significantly improved odds of being engaged in the scientific enterprise compared to non-awardees.

Variation Between K Mechanisms

The results presented thus far have concentrated on K applicants in the full or comparison cohorts across all seven K mechanisms. However, each of the K mechanisms varies slightly in applicant eligibility, field of cancer research, and award provisions and requirements. Subtle differences in awardee and non-awardee outcomes between K mechanisms were revealed by several measures, and a few examples are noted below. In the full cohort, K01, K07, K08, K11, K22, and K23 awardees had greater odds of subsequently receiving an NCI grant (2.29× to 4.34×, with p values ranging from 0.0032 to 9.18 × 10−14). The trend was similar for K25 awardees but did not reach statistical significance. Likewise, in the comparison cohort, K08 and K23 awardees had greater odds (1.67× and 4.91×) of subsequent NCI funding (p = 0.0489 and p = 0.0186, respectively), and the trend was similar for the other mechanisms but did not reach statistical significance. When examining service on Federal advisory committees and review panels, K01 and K07 awardees in the full cohort had the highest odds of committee participation (3.06× and 3.34×, respectively; p = 6.19 × 10−08 and p = 5.23 × 10−12, respectively). Also, in the full cohort, K01, K07, K08, K11, or K25 awardees had significantly higher odds (4.73× to 8.15×) of being engaged compared to non-awardees (p values range from 0.0414 to 2.2 × 10−08). The trend was similar for K22 and K23 awardees but did not reach statistical significance.

Discussion

The sheer size and longevity of NCI’s K program, combined with an absence of past evaluation data, presented an opportunity to examine performance, track program participant outcomes, and better understand whether K program goals were being met through a quantitative, comprehensive evaluation. Overall, we found that participation in the NCI K program had a positive impact on the careers of K awardees. This was not only gauged by traditional factors such as research grant funding and publication productivity but also by activities that signal engagement within the broader biomedical research enterprise such as Federal advisory committee service and membership in scientifically oriented professional societies.

K awardees were more likely than non-awardees to apply for and secure subsequent research funding from the NIH. Additionally, K awardees attained their first R01s in a similar amount of time as non-awardees, indicating that the “protected” time spent on career development activities through the K program did not hamper timely career progression to research independence. Although 59 % of K awardees in the full cohort received an NIH research grant, 84 % of K awardees had some NIH application activity (applied for or received a grant). It should be noted that our data provided a snapshot of grant funding activity over a given time period and included applicants who received their K awards during the late years of this study (e.g., FY06-08), so their K awards had not been completed when grant outcomes were assessed in FY11. Thus, the percent of K awardees receiving subsequent grants may be underestimated, especially for the more recent K awardees. Regardless, our findings are in accord with, or are slightly higher than, the proportion of K awardees who received subsequent NIH funding in the NIH mentored K evaluation [7]. Although it is difficult to make direct comparisons of grant outcomes between evaluations of career development programs supported by other organizations due to differences in evaluation methodologies, data analyses, and program goals, a 2003 evaluation of the Burroughs Wellcome Career Award in the Biomedical Sciences (CABS) program indicated that just over 65 % of CABS recipients had subsequent NIH research support [9]. CABS was one of the first “bridging” awards intended to facilitate the transition to research independence. The K99/R00 is most similar to CABS, so a direct comparison between those two programs would be most appropriate. However, the K99/R00 was not included in our evaluation because the first NCI K99 awards were not made until 2007, and sufficient time has not elapsed to analyze outcomes from this mechanism.

Though the overarching goal of NCI’s K program is to provide support to develop and facilitate an independent cancer research program, approximately one-quarter of K awardees appeared to have pursued other routes in the broader scientific arena. Outside of academia, we found K awardees affiliated with government and industry. Without a discussion group or survey of this subset of K awardees, we can only speculate on the variety of reasons behind these career choices, which may include interests outside of research and teaching, work–life balance, compensation, limited number of available faculty positions, and low NIH funding rates. Regardless of the reasons, we believe it is important to recognize the value of other career pathways—public policy, scientific writing, grants administration, etc.—that use science for the benefit of society [10, 11]. Perhaps, if exposure to these non-traditional options, discussion of more clearly defined career trajectories and models, and training in transferrable professional skills were widespread in graduate school or earlier stages, it would better serve the future workforce to more closely align skills, interests, and careers in biomedical science. This may also divert some individuals from pursuing independent research careers before both NCI and the awardees invest time and resources in the K program. A recent report from the NIH Advisory Committee to the Director Biomedical Workforce Group recommended the NIH support additional training and career development experiences to equip graduate students for various career options beyond the academic research track [12].

From a program perspective, several findings from this evaluation point to opportunities to consider moving forward. Men and women were equally successful in K award receipt. That finding, together with the trend showing the share of full-time full professorships held by women with science and engineering degrees, has risen substantially over the past three decades [13], have positive implications for women in science. However, women still comprise only a little more than 20 % of those holding full-time full professorships [13], so a need to focus on retention strategies to increase female representation in the biomedical research workforce, particularly at the tenure-track faculty stage [14], remains.

In general, K awardees had comparable successful outcomes across the multiple K mechanisms explored in this evaluation. Most often, K awardees’ outcomes showed similar trends (e.g., higher median publications per person than non-awardees), and any differences lie in the magnitude of the change or in the ability to reach statistical significance, which was also influenced by individual mechanism size (applicant pool). Any other differences could be attributed to features of the target population each mechanism attracts (e.g., applicants with MDs compared to PhDs). Thus, the similar outcomes across K mechanisms provide an opportunity to contemplate adjustments to create a simplified and more inclusive program structure. Consolidating mechanisms to remove the barriers created by the discrete mechanisms may offer K applicants more freedom in designing their research and training experiences and would reflect NCI’s message of encouraging interdisciplinary research. As an example, applicants to the K08 and K23 mechanisms are early stage investigators with clinical or health professional degrees who desire to have mentored experiences in basic (K08) or patient-oriented (K23) research. Overall, the two mechanisms have similar objectives, award duration, financial support, applicant eligibility, and commitment. Maintaining a K mechanism to support physician scientist career development is critical in responding to the dearth of qualified physician scientists in oncology [15, 16]. Therefore, using one mechanism that unifies the K08 and K23 applicant pools and invites applications across the cancer research disciplines could be the first step in facilitating seamless opportunities among basic and patient-oriented physician scientists. A previous report on biomedical workforce training by the National Research Council noted that the complex nature of NIH’s K program may discourage applicants [17]. Thus, a closer examination of potential redundancies or gaps in NCI’s current K portfolio is justified.

There are several limitations to the current study warranting discussion. The evaluation was performed in a quantitative fashion by extracting, collating, and analyzing data from extant sources. No new information was collected from K applicants. Therefore, we lack data from participants on their assessment of the value of the K program experience and what they believe were direct effects of participating in the program. We could also gain an understanding of why individuals pursued career paths in various sectors if we surveyed applicants who subsequently became funded researchers or were involved in other ways in the scientific community. Construction of the comparison cohort was performed to minimize effects attributed to selection bias—supporting the best and brightest—but it still remains a challenge to disentangle career development program effects from career successes of highly talented, motivated scientists [18]. Furthermore, we did not have information on whether non-awardees in either the full or comparison cohorts had career development awards sponsored by other Federal or private funding organizations. Thus, it is unknown if the non-awardee cohorts truly had no similar career development program experiences. We also did not directly measure career longevity since grant funding data were snapshots taken in time, but we did investigate grant activity by high water mark ranking, as well as academic faculty rank and proportion of K applicants who achieved Assistant, Associate, or Full Professor status at the time data were collected (data not shown). The majority of K01, K07, K22, K23, and K25 applicants achieved current ranks at the Assistant or Associate Professor level, and half of K08 and K11 applicants were at the Associate or Full Professor level. Across all the K mechanisms analyzed, proportionally more members of the K awardee cohort attained a current rank of Associate or Full Professor than non-awardees.

This evaluation demonstrates the overall successes of NCI K program participants and shows through multiple outcome metrics that, among individuals who were equally likely to receive K funding, those who participated in the K program derived additional benefit in pursuing research careers and participating in the scientific enterprise. The evaluation was the first of its kind in NCI training program history, affording a unique opportunity to retrospectively integrate decades of program data into our reflections of what constitute successful outcomes of training and career development programs. The primary objective of NCI’s K program remains the support of early career cancer scientists and health professionals at critical junctures to promote research independence. Moreover, we recognize the value and necessity of acknowledging additional types of contributions from those who have been scientifically trained. We encourage others to consider adopting such perspectives into training program assessments as well.

Below is the link to the electronic supplementary material.

(PDF 89 kb)