A matched cohort examination of publication rates among clinical subspecialty fellows enrolled in a translational science training program.

PURPOSE: This study examined the effectiveness of a formal postdoctoral education program designed to teach skills in clinical and translational science, using scholar publication rates as a measure of research productivity.
METHOD: Participants included 70 clinical fellows who were admitted to a master's or certificate training program in clinical and translational science from 1999 to 2015 and 70 matched control peers. The primary outcomes were the number of publications 5 years post-fellowship matriculation and time to publishing 15 peer-reviewed manuscripts post-matriculation.
RESULTS: Clinical and translational science program graduates published significantly more peer-reviewed manuscripts at 5 years post-matriculation (median 8 vs 5, p=0.041) and had a faster time to publication of 15 peer-reviewed manuscripts (matched hazard ratio = 2.91, p=0.002). Additionally, program graduates' publications yielded a significantly higher average H-index (11 vs. 7, p=0.013).
CONCLUSION: These findings support the effectiveness of formal training programs in clinical and translational science by increasing academic productivity.

Introduction

Reports from the Institute of Medicine [1], the National Center for Advancing Translational Sciences (NCATS) Advisory Council Working Group [2], and NCATS leadership [3] stress the need for a translational science workforce with the skills necessary to translate discoveries into routine clinical practice with tangible benefits for patients. The National Institutes of Health has created several mechanisms to facilitate the training of investigators in clinical and translational science (CTS), beginning with the K30 Clinical Research Curriculum award in 1999. This program facilitated the creation and implementation of master’s and certificate programs in CTS which are now offered in many academic medical centers. Other important mechanisms including the Roadmap K12 program and the current KL2 and TL1 components of the Clinical and Translational Sciences Award (CTSA) program, funded by NCATS, have been built on this strong curriculum foundation.

The literature addressing the effectiveness of CTS education programs is sparse, especially for clinical postdoctoral trainees. Training programs have been associated with increased research productivity in pediatric clinical fellows [4] and basic scientists [5] compared with control samples of nonmatched peers who were not enrolled in the program under study. Productivity outcomes included publication rate, number of clinical collaborators, confidence in conducting translational research, and interest in a translational science career. Matched controlled comparisons have not been reported.

Since 1999 more than 600 fellows and junior faculty, representing 26 departments, have participated in the master’s and certificate programs at Mayo Clinic. Our primary objectives were to determine whether formal training of clinical fellows in translational science methods resulted in a higher number of publications 5 years post-fellowship matriculation and a faster time to publication of the 15th peer-reviewed publication post program matriculation compared to matched peers who did not complete such training. Secondary objectives included examining whether there was a difference in the impact of post-fellowship publications as measured by the H-index and determining whether there is a difference in the type of professional position taken after fellowship completion.

Methods

This study was approved by the Mayo Clinic Institutional Review Board. Subjects eligible for study in the intervention group included internal medicine (MD) clinical subspecialty fellows in cardiovascular diseases (CV) and gastroenterology and hepatology (GI) who matriculated in either the master’s or the certificate CTS programs at Mayo Clinic between 1999 and 2015. These two divisions were selected as a relatively high proportion of CV and GI fellows participate in the master’s and certificate programs (12% of 316 fellows and 11% of 259 fellows in CV and GI, over the period of study, respectively). Both CV and GI fellowship programs have 1 year of protected research time within the 3-year fellowship program, regardless of matriculation into the master’s or certificate programs.

Each fellow who matriculated into the master’s or certificate programs during this time period was matched with a peer in the same fellowship program at Mayo Clinic, who did not receive this training; these individuals served as matched controls. The matching was performed using the greedy algorithm based on the absolute differences between specified matching criteria to identify the most similar control for each case. In the event, a control was identified as the best match for a case, the control was randomly assigned to one case and the next best control was paired with the second case [6-8]. Matching criteria included fellowship program (CV or GI), gender, years of post-MD graduate training (±1 year for 65 pairs and ±3 years for 5 pairs), age at the time of starting fellowship training (±3 years), and site of fellowship (2 pairs needed to be matched across sites).

Translational Science Training Program Descriptions

Beginning with a K30 curriculum development award in 1999, the Mayo Clinic Center for Clinical and Translational Science (CCaTS) and its precursors developed a comprehensive CTS curriculum with a current catalog of over 50 courses that are directly developed and managed by CCaTS (Supplementary Table S1). This curriculum has been mapped to competencies developed by the CTSA consortium for master’s and certificate scholars. Scholars individualize their own curriculum development plan from among all available graduate school courses, including the CTS curriculum (summarized in Supplementary Table S1).

The master’s degree program is a 24-credit program and the certificate program is a 12-credit program. Both require a mentored research project. An approved thesis is required for master’s graduation; the magnitude of the thesis is sufficient enough to generate two manuscripts, but there is no requirement that manuscripts are actually written. Review articles may contribute to a thesis, however, they do not count as original data and would not count toward the two publication metric. The certificate program requires submission of one manuscript in place of the thesis but does not require publication before conferral of the certificate. Scholars choose between the master’s and certificate programs based on desired intensity of training and available time to dedicate to training. Master’s program applicants are admitted typically with at least one publication and a project that appears likely to yield two publications. Certificate applicants are admitted regardless of publication history and with a project that appears likely to yield one publication. At least 2 years with 1 year of fully protected time is suggested to complete the master’s program and at least 1 year with 50% time is suggested to complete the certificate program. Protected time for both programs is required to meet program milestones including completing coursework. Scholars typically spend 1–4 hours outside of class time per credit hour per week to complete coursework. These programs are available to all postdoctoral scholars and publicized via internal newsletters, websites, and weekly CTS Grand Rounds.

Scholars in the Certificate program are required to complete seven required credits and five elective credits during the 12 months of the program. Required courses include three total credits in research methods and regulations including responsible conduct of research and clinical research protocol development. Protocol development involves writing a research protocol for their mentored project. Formal training in manuscript writing is limited to a required 1-day workshop. Formal training in grant writing is also limited to a 1-day workshop. Scholars are also required to take two credits of clinical epidemiology and two credits of biostatistics. The GI fellows have preselected GI electives they are required to take according to their program curriculum.

Scholars in the master’s program are required to complete 16 required credits and eight elective credits during their 24 months in the program. The additional 8 credits of required courses for the master’s degree include advanced statistics and clinical epidemiology, clinical trials design, and regulatory issues in clinical research. Required courses include 5 total credits in research methods and regulations including responsible conduct of research, clinical research protocol development, regulatory issues in clinical research, and clinical trials: design and conduct. Additionally, master’s scholars are required to take 7 total credits in biostatistics and 4 total credits in clinical epidemiology.

In addition to the curriculum, both programs utilize clinical investigators as mentors who are active in both research and clinical practice [5], with research interests spanning the spectrum of translational science (Supplementary Figure S1). Such mentors serve not only as rich informational resources but also as role models in blending this dual expertise. Mentors from a wide range of disciplines, who work in interdisciplinary teams and model synergistic team science collaboration, are also highly valued (Supplementary Figure S2). These programmatic values align with institutional values, providing important environmental contextual support.

Outcome Measures

The primary measures of academic productivity were (1) the number of publications at 5 years post-fellowship matriculation and (2) time to publication of the 15th peer-reviewed publication post-program matriculation. These were selected as the primary outcomes for several reasons, including the fact that number of peer-reviewed publications serves as a primary consideration in academic promotion and the availability of open-access databases that objectively track this outcome across scientific journals. The Scopus database was utilized for bibliometric source data [9]. Publications for all cases and controls were identified using the search function in Scopus. First name, last name, and Mayo Clinic affiliation during fellowship were used to identify the correct individuals from among multiple individuals with similar names listed in Scopus. If more than one author met these criteria in the initial search, a secondary search used middle initial, department affiliation, or article focus to determine the correct author. All searches were done between January 12, 2018, and January 28, 2018. Articles published in 2018 were not included in the analysis. Data collected for analysis included author list, title, journal, date of indexing (DOI), and document type. Year of fellowship start was used for baseline assessment, for both cases and their matched controls.

Secondary outcome measures included the Scopus H-Index as of January 2018, first-author publication rate in the same manner as total publication rate described above, and whether the fellows entered private practice or an academic position after completion of the program/end of study. The Scopus H-index was calculated according to their proprietary algorithm to provide a more complex measure of scientist impact within a given field. Briefly, this algorithm is based on a set of calculations proposed by Hirsch that assigns a scientist an H-index based on the number of papers that they have published that have at least H number of citations [10]. Professional position post-fellowship completion was assessed using Mayo Clinic Human Resources records (if still at Mayo Clinic) and Web-based searches for those no longer at Mayo Clinic.

Statistical Analyses

Matching criteria were summarized using mean (± standard deviation) or count (percentage) as appropriate. Current position was also summarized using count (percent) and univariate tested using McNemar’s test. Due to skewness toward higher values, the median, interquartile range, and range were presented for continuous outcome measures: number of publications prior to fellowship, number of publications within 5 years of fellowship matriculation, and Scopus H-index. Logistic and linear regression models using generalized estimating equations to account for matched pairs were used to further assess the impact of the training program on binary and continuous outcomes, respectively. The relative effect of the training program was reported as odds ratios (ORs) for binary outcomes and, due to skewness, as a ratio of geometric means (fold-change) for continuous outcomes. Summarizing the percentage of people reaching specific publication counts 15 years after matriculation into fellowship was done via Kaplan-Meier methods. Median time to the events is also presented. The relative effect of the training program was presented using hazard ratios (HR) from conditional proportional hazards models. For the models predicting Scopus H-index and current position, time since matriculation into fellowship was included as a covariate as may be a confounder of these outcomes.

A significance level (α) of 0.05 was used for all tests. Statistical analyses were performed using SAS (version 9.4; Cary, NC, USA) and R (version 3.4.2; Vienna, Austria).

Results

From 1999 to 2015, 28 MD clinical fellows (13 CV and 15 GI) graduating from the master’s program and 42 MD clinical fellows (14 CV and 28 GI) graduating from the certificate program in CTS were included in this study. These 70 fellows were matched successfully with peers who completed the same clinical fellowship at Mayo Clinic with 1 year of protected time but did not participate in the CTS programs. Matching variables at time of fellowship included age, years since medical school graduation, clinical subspecialty area, and Mayo Clinic site. The mean age of both cases and controls at the start of fellowship was 31.0 and 31.2 years; the mean number of years since completion of medical degree was 5.31 and 5.34 years, respectively (Table 1). No significant differences were found in these baseline measures between program subjects and non-program subjects (Table 1). While not a matching variable, the number of publications prior to completing the master’s or certificate programs also did not differ significantly between program and non-program subjects (p=0.771; Table 1).

Table 1

Demographic and professional characteristics of program subjects and matched nonprogram participants

	Total (n=140)	Program subjects (n=70)	Nonprogram subjects (n=70)	p Value
Baseline measures
Mean age at fellowship start (SD)*	31.2 (2.78)	31.0 (2.83)	31.3 (2.74)
Gender, n (%)*
Male	76 (54.3%)	38 (54.3%)	38 (54.3%)
Female	64 (45.7%)	32 (45.7%)	32 (45.7%)
CCaTS program completed, n (%)
None	70 (100.0%)	0 (0.0%)	70 (100.0%)
Master’s	28 (40.0%)	28 (40.0%)	0 (0.0%)
Certificate	42 (60.0%)	42 (60.0%)	0 (0.0%)
Mean years between MD and program start (SD)*	5.34 (2.82)	5.31 (2.84)	5.36 (2.82)
Site, n (%)*
Rochester, MN	138 (98.6%)	70 (100.0%)	68 (97.1%)
Florida or Arizona	2 (1.43%)	0 (0.0%)	2 (2.86%)
Clinical area, n (%)*
Gastroenterology	86 (61.4%)	43 (61.4%)	43 (61.4%)
Cardiovascular diseases	54 (38.6%)	27 (38.6%)	27 (38.6%)
Number of publications prior to fellowship				0.771
Median	1	1	0
Interquartile range	0, 3	0, 4	0, 3
Range	0–35	0–14	0–35
Number missing	3	1	2
Post-fellowship matriculation measures
Number of publications 5 years post-matriculation				0.041 1
Median	6	8	5
Interquartile range	3, 13	4, 15.8	2, 9.75
Range	0–52	0–52	0–32
Number of first-author publications 5 years post-matriculation				0.002 1
Median	3	4	2
Interquartile range	1, 6.25	2, 8	1, 5
Range	0–26	0–26	0–17
Time to publication of 15th peer-reviewed publication post-matriculation				0.001 1
Median (years)	13	10	na
Events	61	40	21
Current H-index				0.013 1
Median	9	11	7
Interquartile range	4, 14.5	7, 16.8	3, 12
Range	0–44	0–44	0–44
Number missing	1	0	1
Current position, n (%)				0.862 2
Private practice	55 (39.3%)	28 (40.0%)	27 (38.6%)
Academic	85 (60.7%)	42 (60.0%)	43 (61.4%)

SD, standard deviation; CCaTS, Mayo Clinic Center for Clinical and Translational Science; MD, internal medicine.

Matching variables.

Wilcoxon signed rank test of the paired difference.

McNemar’s test.

Program participants had a significantly higher median number of publications within the first 5 years following matriculation into fellowship as compared to their nonprogram counterparts (8 vs. 5, p=0.041; Table 1). Furthermore, program participants had a significantly higher median number of first-author publications within this same time period [4] as compared to those who did not complete a CTS training program (4 vs. 2; p=0.002; Table 1).

Univariate Kaplan-Meier analyses demonstrated a significant difference in the rates of total publications within the first 15 years after matriculation into fellowship between fellows who were program participants and those who were not (Fig. 1). The median number of years for program participants to publish 5 peer-reviewed manuscripts was 8 years, as compared to 9 for those did not participate in the program (p=0.115). Half of program participants reached the 15 publication milestone by 10 years post-program matriculation; however, this was not able to be calculated for non-program participants because an insufficient number (29.6%) reached this milestone during the period of observation (p=<0.001). A Cox Proportional Hazards model for reaching the 15 publication milestone indicated that there was a significantly increased likelihood of reaching 15 publications within the assessed time frame among program participants as compared to nonprogram participants in both unadjusted (HR = 2.32, 95% confidence interval [CI] = 1.37–3.94, p=0.002) and matched (HR = 2.91, 95% CI = 1.47–5.77, p=0.002) analyses.

Fig. 1

Kaplan-Meier survival curves of time to 15 publications post-matriculation.

Similarly, univariate analyses of rates of first-author publications in the first 15 years following matriculation into fellowship revealed significant differences between those who were program participants and those who were not (Fig. 2). The number of years it took for half of program participants to publish 5 first-author manuscripts was 6 years, while an insufficient number (28.8%) of nonprogram participants reached this milestone (p=0.004). Neither group had sufficient numbers with 10 first-author publications to allow for the calculation of median time. Cox proportional hazards modeling for reaching 10 first-author publications demonstrated that there was a significantly increased likelihood of reaching 10 first-author publications within the study time frame among program participants as compared to nonprogram participants in both unadjusted (HR = 2.31, 95% CI = 1.09–4.91, p=0.029) and matched (HR = 2.11, 95% CI = 0.96–4.67, p=0.065) analyses.

Fig. 2

Kaplan-Meier survival curves of time to 10 first-author publications post-matriculation.

Secondary Outcomes: H-Index and Post-Fellowship Position

The median Scopus H-index was significantly higher in those who completed the CCaTS master’s or certificate program compared to those that did not (11 vs 7, p=0.013). In a model adjusted for time since fellowship, participants who completed either the CCaTS master’s or certificate program had a 1.48-fold increase in Scopus H-index as compared to controls who did not participate in these programs (95% CI = 1.15–1.90, p=0.002). This magnitude of increased fold difference in median H-index remained consistent in both magnitude and statistical significance after conducting matched analyses (fold change: 1.476, 95% CI = 1.147–1.897, p=0.002).

Of the 70 program participants, 42 (60.0%) transitioned into an academic position after completion of their fellowship, not significantly different from nonprogram participants (43/70, 61.4%; p=0.862; Table 1). In an unadjusted model, there was no significant difference in odds of moving into an academic position versus private practice between those who completed a CCaTS master’s or certificate program and those who did not (OR = 0.92, 95% CI = 0.46–1.82; p=0.806); this finding held in matched analysis (OR = 0.918, 95% CI = 0.49–1.80; p=0.803). All models, including the “unadjusted” model, included time since fellowship as a covariate.

Discussion

The primary objective of the current study was to determine the impact of formal CTS training on research productivity among clinical fellows. Collectively, our data suggest that CTS training at the masters or certificate level was associated with higher publication rates by clinical fellows, compared with peers who did not receive formal training in CTS. This effect extended well past the period of training, at least to 15 years post-matriculation into fellowship program. In addition to publishing more than their fellowship peers, program graduates also published in higher impact journals that are oft-cited in their respected fields as reflected in a higher H-index. Program trainees also had a greater likelihood of research activity within the first 5 years after fellowship matriculation. Notably, baseline publications prior to training were not statistically different between groups, suggesting similar productivity and research experience between participants and controls at baseline.

Some mentors have expressed hesitancy to encourage their trainees to participate in CTS training programs, attributed to a belief that the coursework may slow research productivity during training. Through matched peer-group comparison, we found that translational science trainees were more productive in research both in near and long-term. Knowlton and colleagues [5] also found translational science training to be associated with greater research productivity among basic science PhD students from 2006 to 2010 compared to a nonrandomized group of peers. Their 1-year program integrated closely mentored clinical medicine exposures into a basic science curriculum to promote translational science. Their comparison group consisted of peers from the same basic science programs with comparable date of enrollment and GPA cutoff score. Knapke and colleagues [4] also found increased research productivity among clinical pediatric fellows who completed a master of science degree program in CTS. Their graduates had a higher number of overall publications and first-author publications compared with unmatched peers in the same fellowship over the same time period.

This study is one of the first to examine the impact of a CTS training program on research dissemination among clinical fellows. One of the major strengths of this study is its ability to examine outcomes of a training program over a period of 15 years, and report publication outcomes up to 15 years post-date of matriculation into a clinical fellowship. Another strength of this study is its inclusion of two different options (master’s and certificate) of a translational science training program. A third strength is the utilization of a matched comparison group design [4, 11]. A matched cohort is a more rigorous and practical design, given challenges of randomly assigning intensive postdoctoral training.

Regarding study limitations, like any observational study, the results may be confounded by selection bias, as individuals with greater interests and abilities in research interests may be more likely to pursue research training. While baseline research productivity was not significantly different between cases and controls, suggesting group equivalency in research productivity and experience prior to training, we were unable to collect data regarding baseline measure of research interest, nor were we able to assess scholar-institution research alignment at baseline. As both the CV and GI fellowships are highly competitive and include a research component with 1 year of protected research time, we believe that the scholars’ research interests are likely well aligned with that of the institution. Additionally, cases and controls were equally likely to obtain an academic versus private practice position postgraduation and thus have the opportunity to pursue research and publication. Nonetheless, causal attribution of program effectiveness to explain the observed associations should only be made with caution. Other limitations include insufficient study power to compare the results of master’s versus certificate training; however, the results suggest that hypothesis that more intensive training (master’s) is most effective, as would be predicted. Finally, this is the experience of a single institution, and the results may not generalize to other settings.

In conclusion, CTS training was associated with increased research productivity among clinical fellows up to 15 years post-matriculation into fellowship, compared with clinical fellows who did not pursue formal translational science training. The association extended beyond higher publication rates to also include higher impact of publications within the greater research community. These results may be of interest to mentees, considering enrollment in a translational science program as well as their mentors and training program directors. Additional investigation is warranted to examine the association of training in CTS with other markers of research productivity, leadership, and success. In addition to publication rates and impact scores such as the H-index, outcomes worthy of future study include, funded grants, serving as a grant PI, serving as a mentor on career development grants and otherwise, placement in translational research positions, and subjective measures of interest in translational science and intent to pursue translational science as a career choice.

Study Highlights

The impact of clinical and translational science (CTS) training programs is not often scientifically reported in the literature. In this study, the authors examined whether there are differences in research productivity between clinical fellows who completed a master’s or certificate program in CTS and matched clinical fellows at the same institution who did not complete such a program. Productivity was defined by publication history and current H-index for each participant post-fellowship. This study demonstrates that there is a significant difference in publication rates and H-index post-fellowship between clinical fellows with formal CTS training and those who did not have formal training and supports the necessity of such training in expanding our translational science workforce.