Interventions to increase research publications in graduate medical education trainees: a systematic review.

INTRODUCTION: Competency-based educational models recommend trainee exposure to research, but the best methods for Graduate Medical Education (GME) programs to accomplish this have not been clarified. The objective of this study was to quantify published interventions to generate resident research and compare effectiveness among those interventions.
MATERIAL AND METHODS: A systematic review of English-language articles of studies of GME programs was performed, describing resident research interventions and quantifying the number of publications as an outcome.
RESULTS: The search produced 13,688 potentially relevant articles, and included 47 articles in the final synthesis. Publication effectiveness was calculated as publications per year. The top ten programs for publication effectiveness were compared to others for interventions chosen. Interventions were characterized as research director, protected time, research requirement, research mentor, curricula, research assistant, biostatistician, information technology support, research fund, pay-for-performance plans, and celebration of accomplishments. Total number of different interventions was not significantly associated with primary outcome (r = 0.20, p = 0.18). When comparing the top ten programs to the others, appointment of a research director was statistically more prevalent in those programs (70% vs. 30%, p = 0.02), while presence of a defined curriculum was more common (90% vs. 57%, p = 0.052) but not statistically significantly.
CONCLUSIONS: Leadership interventions (directors, curricula) are associated with successful GME research efforts.

Introduction

Residency training programs are expected to provide residents with exposure to research opportunities [1]. Reported benefits of resident exposure to research include increased satisfaction with training [2, 3], increased lifelong learning and analytical skills [4, 5], improved patient care [6], and increased likelihood of pursuing academic careers [7-9]. Furthermore, these residents are more likely to choose careers in research [4], which may be important because of the predicted shortage of clinician-investigators in the United States [10].

Despite these benefits, many barriers to resident research have been identified. These include lack of resident and faculty time for research [11-13], availability of mentors [14, 15], absence of research curricula [14, 15], paucity of funding for research [14, 16], and lack of perceived value of research to residents [17]. In addition, interventions to increase research take a significant amount of time to show benefits; residents acquiring research skills may not see success before leaving residency. For more than 50 years [18], United States GME programs have attempted to navigate these challenges and with the transition to the Accreditation Council on Graduate Medical Education’s (ACGME) Next Accreditation System, they are now being measured on their effectiveness to stimulate resident research. This shift requires residencies to enumerate their department’s annual publication outcomes, but programs were given the latitude to develop their own process to achieve those outcomes [19].

In order to determine what programmatic interventions lead to improvement in resident research production, we performed a systematic review of the literature to identify and compare published studies of programmatic interventions in residency training which included number of publications as an outcome of their study. Our intention was to both quantify published interventions in GME and to compare effectiveness in order to allow others to develop or update their resident research programs.

Material and methods

This systematic review was conducted according to PRISMA guidelines [20]. We sought to answer two questions: (1) what interventions have been performed in GME to increase resident scholarly activity, and (2) how do outcomes compare across interventions?

Broad inclusion criteria were used to capture any reported interventions at the GME level intended to improve resident scholarly activity. English-language studies were included that described any interventions on Graduate Medical Education trainees whose outcomes included peer-reviewed publications, so interventions could be compared across programs. We created a standard measure to quantify publications across programs (“publication effectiveness”), which we defined as publications/year. All published works in peer-reviewed journals, including letters, case reports, case series, meta-analyses or randomized trials but not abstracts or presentations were counted in the numerator. Publication rates were averaged across the length of time of the study. Studies without an intervention or not reporting program-level publication outcomes, cross-sectional studies that did not report outcomes over time, and studies of podiatry and pharmacy programs were all excluded.

Study identification

A research librarian (A.S.) designed a search strategy (Appendix 1) to search MEDLINE, EMBASE, CINAHL, EMBASE, PsychExtra, and ERIC databases, and from hand search of reference lists of identified publications for synonyms of resident physicians (e.g. medical resident, Graduate Medical Education) and scholarly activity (e.g. publishing, authorship, scholarly activity, journal article). No beginning date was selected, and the search was last performed December 17, 2015. Additional studies were sought from hand searches of authors’ reference lists.

Study selection

Two authors (A.D., R.Z.) independently screened all titles and abstracts for inclusion. In the event of a disagreement or insufficient information in the abstract, the full text was reviewed independently and in duplicate. Conflicts were resolved by consensus and a third reviewer (R.A.). The full protocol can be accessed at http://readinghospital.libguides.com/ld.php?content_id=20513770.

Data extraction

A data extraction table was developed by iterative testing. Data were extracted independently by two authors in duplicate for all variables and conflicts resolved by consensus. The authors abstracted information on type of residency program, number of publications, and years of study. Interventions were coded for analysis into the following 11 categories, and unadjusted intra-class correlation coefficients (ICC) were calculated. Categories were: research director (ICC, 0.91) protected time (ICC, 1.0), research requirement (ICC, 0.86), research mentor (0.71), research curriculum (0.86), research assistant (ICC, 0.85), biostatistics support (ICC, 0.85), information technology support (ICC, 0.86), research funds (ICC, 0.91), pay-for-performance (ICC, 0.88), and research symposium (ICC, 0.95). Risk of bias was independently assessed by two authors using tools developed by the Cochrane Collaboration (ACROBAT-NRSI) [21]. Risks of bias were categorized as due to confounding (ICC, 1.0), selection (ICC, 1.0), measurement (ICC, 0.98), departure from intended interventions (ICC, 0.95), missing data (ICC, 0.99), measurement of outcomes (ICC, 0.94), bias in selection of reported outcomes (ICC, 0.96), and overall risk (ICC, 0.91) as compared to an ideal randomized controlled trial, and were judged to be low, moderate, serious or critical risk.

Data synthesis

No studies were performed using randomized, controlled designs; therefore no meta-analysis was performed. Chi-square (χ2) analyses were performed to compare frequency of interventions between the top ten programs (the 10 programs with the highest rates of publications/year) and all other programs in the study. Effectiveness of individual interventions was analyzed by comparing means of primary outcome among programs with or without that intervention by Mann-Whitney U tests (since data were non-parametric). Pearson correlations were taken to measure the primary outcome against total number of interventions, total curricular hours and total protected research time respectively. SPSS software, version 17.0 (Chicago, IL) was used for analyses.

Results

We identified 13,688 potentially relevant articles from our initial search, of which 707 were found to be duplicates. From these, we found 123 records requiring full-text review. Thirteen authors were contacted for more information, with 9 responding. Forty-seven articles were included in our final synthesis (Figure 1). Interventions were studied over an average of 5.9 years (range: 1–19) (Table I). Internal Medicine was the most common residency specialty (n = 14), followed by Family Medicine (n = 7) and General Surgery (n = 6) (Table I). Publication effectiveness ranged from 0 to 68.1 publications/year.

Figure 1

PRISMA diagram of study inclusion

Table I

Records arranged in descending order by most to fewest publications per year

Author	GME type	Study length [years]	#Pub	Pub/yr	RD	Prot. time	Prot. time [day/year]	Req	Ment	Curr	RA	BS	IT	Fund	P4P	Cele	Total
Robbins 2013	Orth	6	409	68.17		•	60.00		•	•	•		•	•			6
West 2011	IM	7	397	56.71	•	•	20.00		•	•				•			5
Manring 2014	Orth	4	136	34.00	•			•	•	•							4
Kanna 2006	IM	2	59	29.50	•	•	3.33	•	•	•						•	6
Kohlwes 2011	IM	5	85	17.00						•							1
Papasavas 2013	GS	2	32	16.00	•			•	•	•	•	•				•	7
Alweis 2015	IM	14	204	14.57	•	•	20.00	•	•	•		•			•	•	8
Ruiz 2011	IM	2	26	13.00	•	•	23.33	•		•				•		•	6
Kolade 2014	IM	5	57	11.40				•									1
Sakai 2014	Anes	5	50	10.00	•					•						•	3
Fisher 2010	GS	2	18	9.00				•	•								2
Khurana 2015	Neur	1	9	9.00		•	30.00	•	•								3
Millis 2004	PMR	4	35	8.75		•	4.30	•	•	•		•		•		•	7
Segal 2006	Orth	19	129	6.79		•	62.50				•	•					3
Roane 2009	Psy	5	32	6.40	•			•	•	•				•		•	6
Eckert 2015	Anes	4	25	6.25		•	10.00	•								•	3
Tsai 2003	Psy	11	68	6.18		•	76.25	•	•	•				•			5
Hoedebecke 2014	FM	1	6	6.00	•				•							•	3
Kohlwes 2006	IM	4	22	5.50		•	19.50		•	•							3
Panchal 2014	ER	3	16	5.33					•	•	•	•		•		•	7
Torres 2015	Orth	6	31	5.17				•	•						•		3
Chang 2013	ENT	7	32	4.57		•	15.00								•		2
Thomas 2000	GS	9	40	4.44		•	50.00										1
Chung 1999	GS	3	13	4.33				•		•			•				3
Lampman 2003	GS	14	59	4.21	•				•	•	•			•		•	6
Hepburn 2003	IM	5	21	4.20	•	•	13.33	•	•	•			•	•		•	8
Seehusen 2009	FM	1	4	4.00				•									1
Kurahara 2012	Ped	7	26	3.71				•	•	•						•	4
Crownover 2008	FM	2	7	3.50				•		•							2
Potti 2003	IM	10	35	3.50		•	13.33	•		•							3
Durning 2004	IM	5	17	3.40	•												1
Kichler 2014	GS	2	6	3.00				•	•	•							3
McCabe 1975	ENT	8	23	2.88		•	45.00	•	•	•				•			5
Liese 1998	FM	5	10	2.00	•					•						•	3
Penrose 2012	OB	3	6	2.00	•				•								2
Alguire 1993	IM	10	19	1.90	•	•	23.00		•	•				•			5
Carek 2011	FM	10	15	1.50		•	7.50	•	•	•		•		•			6
Tucker 1975	ENT	6	8	1.33		•	20.00	•	•					•			4
Ables 2010	FM	7	9	1.29				•								•	2
Macpherson 1995	IM	1	1	1.00		•	0.54		•								2
Lovejoy 2008	Ped	10	9	0.90		•	20.00	•	•	•				•			5
Desbiens 2008	IM	10	8	0.80				•			•			•		•	4
Mezzacappa 2012	Psych	5	4	0.80		•	25.00		•	•	•	•		•			6
Mayo 2015	GI	13	10	0.77		•	40.00	•	•	•		•				•	6
DeHaven 1994	FM	5	3	0.60	•			•	•	•	•						5
Fancher 2009	IM	3.5	2	0.57	•	•	17.17	•	•	•		•		•			7
Tintara 2007	OB	4		0.00				•									1

GME – Graduate Medical Education, Res – Resident; #Pub – total number of publications, RD – Research Director, Prot. time – Protected Time, Req – Research Requirement, Ment – Research Mentor, Curr – Curriculum, RA – Research Assistant, BS – Biostatistical Support, IT – Information Technology Support, Fund – Research Fund Available, P4P – Pay for Performance Fund, Cele – Celebrate Accomplishments, FM – Family Medicine, IM – Internal Medicine, Anes – Anesthesia, Peds – Pediatrics, GS – General Surgery, GI – Gastrointestinal Disease Fellowship, ER – Emergency Medicine, Psy – Psychiatry, ENT – Otorhinolaryngology, Neur – Neurology, Orth – Orthopedic Surgery, OB – Obstetrics and Gynecology. *Calculated protected work days per year, assuming 8 h per work day, 40 h per work week, 20 work days per block or month, and 50 work weeks per year.

Research director

Eighteen records described appointing a research director (RD). Four programs described protected time for the RD, which ranged from 0.1 to 0.25 full-time equivalents (FTE). Of the 18 programs, ten appointed the RD from within the faculty, five hired from outside the faculty (four Ph.D. scientists, one postdoctoral associate) [22-26], two appointed a resident as the RD [27, 28], and one could not be determined. The RD duties included approval and oversight of research projects, granting elective time to trainees, editing abstracts and manuscripts, and assisting with Internal Review Board (IRB) submissions and grant applications.

Protected time

Twenty-four records described protected time for resident research. Allotted time varied from 0.5 to 76.25 work days per year (mean: 25.8 days), assuming 8 h per work day, 40 h per work week, 20 work days per block or month, and 50 work weeks per year. Of the records that described allocation of days, fourteen programs did so in consecutive days or block format [13, 27, 29–40], while six programs intermixed research time into clinical time throughout the year [41-46]. Six programs provided protected time contingent on the approval of a research proposal [27, 36, 39, 46–48]. A non-significant correlation was seen between amount of time granted and primary outcome (r = 0.18, p = 0.39).

Research requirement

Thirty records described a research requirement for residents, and of them 11 tied this requirement to graduation [13, 25, 29, 31, 34, 40, 46, 49–52]. Three programs described creation of a scoring system for measurement of completion of required scholarly activity [29, 49, 51]. Most interventions with a stated research requirement stipulated that residents at least present a poster or abstract at a departmental research day or scientific meeting (which were not captured in this study’s primary outcome), while 4 required submission of a manuscript to a peer-reviewed journal [25, 31, 34, 52]. One study from the Royal Thai College of Obstetricians and Gynecologists required all trainees in the country to submit a publication to a peer reviewed journal in order to complete training [52].

Research mentor

Thirty records described appointing research mentors. Six programs appointed mentors based on research expertise [35, 40, 41, 46, 53, 54]. Most programs assigned the mentor to the resident, while five programs required the resident to choose a mentor [35, 40, 50, 55, 56]. The resident-mentor relationship and reporting structure were generally not described. Two programs required at-least-monthly meetings of the resident and mentor [41, 42]. One record described a financial incentive for faculty based on effective mentorship [57].

Curricula

Thirty records described research curricula. Of those described, the allotted didactic teaching time of research-related topics varied from 7.5 to 75 hours per year (average: 23 h). When described, the most commonly taught topics, in descending order of frequency, were research methods (n = 14), scientific writing (n = 7), and biostatistics (n = 6). Other taught topics included literature searching, IRB structure, and research ethics. Four programs redesigned a regularly occurring journal club to focus on critical appraisal of the literature [23, 29, 31, 55, 57].

Research assistant

Eight records described the addition of a research assistant. Four of them utilized a research assistant to assist with completion and submission of abstracts, manuscripts, grant applications, and IRB submissions [35, 57–59]. Two programs utilized a registered nurse to obtain informed consent for studies and provided continuity among resident research studies [23, 36].

Biostatistician

Six records described the availability of a biostatistician for resident research initiatives. Three records provided funding through the department [29, 40, 44], and two through university affiliation [41, 48].

IT support

Three records identified the use of information technology support. Specific details of these interventions were not well described. One program invested in an information technology program that included a simulation laboratory to develop surgical techniques and to complete anatomical research [35].

Research fund availability

Seventeen records described the availability of a research fund for residents. Six programs described internal funding and program-sponsored grants to support research-related expenses [23, 31, 35, 44, 56, 57], and one obtained NIH R25 grant funding [46]. Thirteen described funds to cover travel expenses for residents to present findings at national meetings [23, 27, 31, 32, 35, 41, 44, 47, 48, 50, 56–58].

Pay-for-performance incentives

Three records described pay-for-performance incentives. Two described incentive pay based on production of any scholarly activity [29, 30], and one placed discretionary funds at risk pending compliance with a research requirement [54].

Celebration of accomplishments

Seventeen records described celebration of research accomplishments. Thirteen programs described an annual resident research day or grand rounds to present research findings [13, 24, 27, 39, 40, 45, 50, 57–62]. Ten programs awarded top performers in research, either by judging presentations or singular recognition for year-long efforts [13, 23, 27, 31, 39, 40, 45, 55, 59, 60]. Three programs displayed research publically, by displaying in the department [29], announcing in a newsletter [28], or listing on the resident website [62]. Three programs participated in regional research symposia [31, 39, 58].

Comparison of effectiveness of interventions

There was no significant difference in mean primary outcome for any single intervention (Table II). Total number of interventions was not significantly associated with primary outcome (r = 0.20, p = 0.18). When comparing top ten programs to the others, the presence of a research director was statistically more likely when compared to the others (70% vs. 30%, p = 0.02), while a defined curriculum was more common but not statistically significant (90% vs. 57%, p = 0.052) (Table III). No other single intervention was significantly more present in the top ten programs when compared to the others.

Table II

Relationship between individual interventions and primary outcome (publications/year)

Intervention category	Intervention described		Intervention not described		P-value (MW-U)
	Mean (publications/year)	SD	Mean (publications/year)	SD
Research director	11.69	14.72	6.74	12.40	0.182
Protected time	10.53	17.26	6.66	7.49	0.907
Research requirement	6.87	8.05	11.75	19.57	0.499
Research mentor	10.32	16.28	5.66	4.65	0.947
Curriculum	10.98	16.22	4.50	3.39	0.413
Research assistant	12.84	22.92	7.77	10.48	0.923
Biostatistician	6.12	5.98	9.23	14.62	0.720
IT support	25.57	36.90	7.48	10.37	0.390
Research fund available	10.80	19.82	7.41	7.99	0.438
Pay for performance	8.10	5.61	8.67	13.81	0.367
Celebration of accomplishments	7.81	7.28	9.10	15.97	0.250

Table III

Comparison of top ten publishing programs to other published programs

Intervention	Presence or absence of that intervention	Presence in top ten publishing programs		Presence in comparator group		P-value
		Count	%	Count	%
Research director	Absent	3	30.0	26	70.3	0.020
	Present	7	70.0	11	29.7
Protected time	Absent	5	50.0	18	48.6	0.940
	Present	5	50.0	19	51.4
Research requirement	Absent	4	40.0	13	35.1	0.776
	Present	6	60.0	24	64.9
Research mentor	Absent	4	40.0	13	35.1	0.776
	Present	6	60.0	24	64.9
Curriculum	Absent	1	10.0	16	43.2	0.052
	Present	9	90.0	21	56.8
Research assistant	Absent	8	80.0	31	83.8	0.778
	Present	2	20.0	6	16.2
Biostatistician	Absent	8	80.0	30	81.1	0.939
	Present	2	20.0	7	18.9
IT Support	Absent	9	90.0	35	94.6	0.598
	Present	1	10.0	2	5.4
Research fund available	Absent	7	70.0	23	62.2	0.647
	Present	3	30.0	14	37.8
Pay for performance	Absent	9	90.0	35	94.6	0.598
	Present	1	10.0	2	5.4
Celebration	Absent	5	50.0	25	67.6	0.305
	Present	5	50.0	12	32.4

Risk of bias assessment

Studies were retrospective cohort studies that nearly uniformly were not able to control for prior resident research skills or the degree and depth of exposure to individual interventions. These limitations made overall risk of bias (as compared to a randomized controlled trial) to be at least serious for all records. Eleven were judged to be at critical risk for bias, often due to methods of collecting and reporting publication outcomes. No sensitivity analyses were performed as the studies were of similar quality levels.

Discussion

Our study identified scholarly activity interventions across a wide range of graduate medical education programs. We identified curricular interventions, mentoring programs and research requirements as the most common interventions described. We identified that appointment of a research director to lead research efforts and development of a formal curriculum appeared to be factors that were more prevalent in the most successful programs, confirming calls from prior authors for emphasis on these leadership elements [14, 15, 17]. A recent systematic review found that research tracks, protected time, research directors, and curriculum had a positive effect on scholarly activity. However, these authors included cross-sectional studies (which we excluded, since effects of interventions could not be followed over time). Those authors also defined their scholarly activity outcomes more broadly than we did, by including any participation in research (including abstracts) as a positive finding [63]. As our intent was to deliberately compare effectiveness among interventions, we limited our scope to records that explicitly defined their publication effectiveness. Similar to their work, we were unable to isolate the effect of any single intervention as nearly all studies used concurrent interventions that may have confounded their effectiveness.

As few of the programmatic interventions we characterized contained control groups other than historical controls, the interventions were likely confounded by the prior research skills of the participating residents, the interest and quality of the faculty, and the quality of the curricular instruction. Absolute differences in publication rates before and after interventions may have also been helpful to ascertain the impacts of interventions, but very few records included pre-intervention data, so historical comparisons across programs were not possible. Most interventions were multi-modal, so the independent impact of any one intervention was difficult to independently ascertain. Because we were not able to control for either program size or number of exposed residents, interventions in larger programs may have appeared more efficacious than those in smaller programs, even if similar in quality; larger programs may also have had more resources to create more interventions to stimulate research, making program size an unmeasured confounder in interpretation of our results. The actual amount of individual resident exposure to curricula and mentoring at the level of the individual program was unquantified in most records. Publications could not be independently verified in our study, as we relied on what the authors reported; there is the potential for one publication with many authors to have been counted more than once in those publications, potentially inflating the impact of the interventions. Publication bias may have limited the number of records with interventions that were not successful in increasing publication rates. Given that interventions to increase publication rates take a long time to bear fruit, the timeframes of many of the studies may have been too short to ascertain the full benefits of their interventions.

In conclusion, an understanding of research methods is believed to be important to residents in training, for better patient care, lifelong learning and career satisfaction [2-5]. Training programs should regularly evaluate their effectiveness in teaching these skills. Leadership in research may be a key factor in improving resident research, as evidenced by the higher prevalence of research directors and defined curricula in top publishing programs. Publication effectiveness appears to be an important metric for programs to demonstrate effectiveness of teaching programs to the ACGME [19]. As interventions that work at one location may not work in another, tracking effectiveness of interventions using standard measures such as publication efficiency over time is important for both individual programs and the field of education in general, and should be tracked to advance this field of study.

Conflict of interest

The authors declare no conflict of interest.