Building a Sustainable National Infrastructure to Expand Research Mentor Training.

An evidence-based research mentor training (RMT) curricular series has been shown to improve the knowledge and skills of research mentors across disciplines and career stages. A train-the-trainer model was used in the context of several targeted approaches aimed at sustainability to support national dissemination of RMT and expand the network of facilitators prepared to implement the curricula. These infrastructure elements included 1) an expansion initiative to increase the number of trained facilitators able to deliver train-the-trainer workshops nationwide; 2) adaptation of RMT curricula for multiple audiences and career stages to increase accessibility; 3) implementation resources to support facilitators and help them overcome implementation barriers; and 4) standardized evaluation of training. This approach to dissemination and implementation has resulted in the preparation of nearly 600 trained facilitators, a large percentage of whom have implemented mentor training for more than 4000 graduate student, junior faculty, and senior faculty mentors. Implications for and challenges to building and sustaining the national dissemination of RMT are discussed.

INTRODUCTION

Quality mentoring is an important contributor to success of researchers across all disciplines and career stages. It has been linked to enhanced trainee productivity, research self-efficacy, and career satisfaction (reviewed in Pfund ; National Academies of Sciences, Engineering, and Medicine, 2017). Effective mentoring relationships have been shown to also influence the confidence of trainees’ ability to successfully conduct research, which is a key predictor of persistence in science, technology, engineering, and mathematics (STEM; Byars-Winston ). Mentors themselves also benefit from effective mentoring relationships, reporting an increase in productivity, a sense of fulfillment through knowledge and skill sharing, and increased self-awareness (Dolan and Johnson, 2009). Despite mentoring’s positive impact, not all mentoring relationships are equally effective. In fact, students from historically underserved backgrounds report receiving less mentoring than their nonminority peers (Helm ; Thomas, 2001; Morzinski and Fisher, 2002; Ginther ; Beech ). Unfortunately, most mentors do not receive training on how to optimize their mentoring relationships (Keyser ; Silet ; Pfund ).

The need for more equitable access to quality mentoring has prompted many agencies, academic institutions, and professional societies to invest resources to develop, test, and promote best practices in mentorship. One such approach is the well-studied Entering Mentoring, a process-based professional development curriculum (hereafter referred to as research mentor training [RMT] that has been tested via a randomized controlled study [Pfund , 2014a] and adapted for use across many disciplines and career stages [Sorkness ; Pfund ]). The Entering Mentoring curriculum covers mentoring domains such as maintaining effective communication, aligning expectations, addressing equity and inclusion, assessing understanding, promoting professional development, and fostering independence. Mentors who participate in Entering Mentoring–based training report skill gains across mentoring domains as well as changes in their mentoring practices. Mentees of participating mentors also report increases in their mentors’ skill gains and positive changes in their mentoring practices (Pfund ).

In 2015, the outcomes of initial efforts to build national capacity for this evidence-based approach were published (Pfund ). These expansion efforts were based on a train-the-trainer approach of dissemination that is often used by personnel in fields as diverse as education, public health, workforce development, nutrition, primary care, and occupational safety (Clarke and Dede, 2009; Pearce ; Yarber ; Baldwin ) to expand the network of researchers exposed to an intervention and promote its use (Hutchinson and Huberman, 1994; Rabin and Brownson, 2012). As shown in Figure 1, this model for nationally disseminating RMT posits that increasing the number of train-the-trainer workshops (referred to as facilitator training [FT]) will increase the number of facilitators prepared to implement RMT locally, thus increasing the number of mentors experiencing evidence-based RMT.

FIGURE 1.

Model for building a national capacity for research mentor training.

During the FT workshop, participants build the confidence and skills to facilitate RMT in their local contexts. The workshop was designed using core elements of high-quality train-the-trainer workshops (Baldwin ) such that it gave attendees 1) exposure to Entering Mentoring through experi­encing modules and receiving copies of the full curriculum; 2) practice facilitating the curriculum and receiving feedback; 3) opportunity to design an implementation plan to support the organization and delivery of the curriculum in local contexts; and 4) access to resources and a peer network to support implementation over time.

This paper describes the infrastructure developed to sustain and support the continued expansion of the national network of facilitators trained to offer RMT using the model shown in Figure 1. Empirical research examining this FT model has focused upon its effectiveness for increasing participant knowledge, skills, and confidence to implement targeted interventions (Pfund ). However, follow-through implementation in local contexts is often inconsistent (Ray ), even among the most effective interventions. Identified barriers to implementation include lack of time, resources, rewards, expertise, and confidence to implement (Hutchinson and Huberman, 1994; Henderson and Dancy, 2007; Henderson ; American Association for the Advancement of Science, 2011; D’Avanzo, 2013).

The infrastructure approaches described here incorporate strategies to overcome barriers to implementation and support a national propagation of RMT. Approaches include 1) developing a Master Facilitator Initiative to train experts to deliver FT, 2) adapting RMT curricula for multiple audiences and career stages, 3) providing implementation resources to address implementation barriers, and 4) developing a standardized evaluation tool to evaluate RMT. Each infrastructure approach is described and data are shared to show national impact and contribution to the expansion of RMT nationwide. These approaches reflect dissemination and implementation factors known to promote utilization and high-quality implementation.

METHODS

Data were collected to assess the efficacy of the infrastructure approaches using a variety of methods, including online registration and applications, posttraining evaluation and implementation surveys, Google Analytics, and internal and online tracking databases. Data types are described below, and the specific infrastructure approach the data were used to assess is noted. The infrastructure approaches themselves are described in detail in the next section.

Master Facilitators

Experts who deliver FT are referred to as “master facilitators” (approach 1). Master facilitators are required to submit an online application, which captures information on demographics, professional background, facilitation experience, and evidence of facilitation skills/effectiveness (Supplemental Material).

FT Application

Individuals interested in attending FT workshops are required to complete an application survey in which they share information on their demographics, professional backgrounds, facilitation experience, and plans to implement RMT (approach 3) (Supplemental Material).

FT Evaluation

FT attendees are required to complete an evaluation survey after participating in the training. Surveys are administered in both paper format and online through University of Wisconsin–Madison (UW-Madison) Qualtrics Software. Data reported in this article were collected from 281 FT participants (73% response rate). Attendees rated the value of each individual component of the FT workshop on a Likert-like scale, with 1 = not at all valuable, 2 = somewhat valuable, 3 = valuable, and 4 = very valuable. Several FT evaluation surveys also used a five-point scale with 5 = extremely valuable. For reporting in this article, the 4 and 5 categories were combined (approach 1). Attendees also retrospectively rated their confidence in facilitation skills, comparing their confidence before and after the FT workshop on a seven-point Likert scale, with 1 = not at all confident, 4 = moderately confident, and 7 = extremely confident. Differences in ratings before and after FT were determined by Wilcoxon signed-rank test (approach 3). Evaluation surveys also contained open-ended questions regarding attendees’ intent to implement RMT, what additional resources might be needed for RMT implementation, what improvements could be made to the FT workshop, and any peer connections made or support used during the workshop. These open-ended responses were qualitatively analyzed using a thematic analysis (approach 3; Boyatzis, 1998; Braun and Clarke, 2006).

Google Analytics and Online Databases

Website usage data were collected through Google Analytics for three websites that hosted curricula and implementation resources (approach 2). Website curricula downloads are tracked by requiring users to create profiles to access and download RMT curricula. This information is stored on the website portal and downloaded into a spreadsheet. Data were reported on www.researchmentortraining.org, which was active from 2009 to 2016; https://ictr.wisc.edu/mentoring, which was active from 2013 to 2016; and the currently active website, www.cimerproject.org, from April 2016 to January 2018.

Internal Tracking Databases

Internal tracking databases are used to record national trainings facilitated by master facilitators and local trainings led by trained facilitators. Information collected includes names of facilitators, dates of training, location of training, curricula used for the training, competencies covered in the training, and number and career stage of workshop participants (approach 1). Tracking databases are also used to record whether facilitators use internal evaluation surveys hosted through UW-Madison Qualtrics to evaluate and assess the effectiveness of their training(s) (approach 4).

Implementation Survey

Implementation surveys are electronically administered to FT participants annually to collect information on their implementation practices and any supports or barriers they have experienced since attending FT. Participants also retrospectively rate their confidence in implementing, thinking back to before they attended the FT workshop and “now” after implementing RMT (1 = not at all confident, 4 = moderately confident, 7 = extremely confident) and whether or not they felt they were prepared to facilitate after attending the training (1 = a little prepared, 2 = a moderate amount, 3 = a lot prepared, 4 = a great deal prepared; approach 3). Participants also rate the helpfulness of the FT workshop in preparing them to implement training locally (1 = extremely unhelpful, 2 = unhelpful, 3 = neither, 4 = helpful, 5 = extremely helpful; approach 3). Additionally, participants are asked to report the number of training workshops they facilitated each year since attending the FT workshop and to rate the overall quality of their local implementations on a Likert scale with 1 = very low, 2 = low, 3 = average, 4 = high, 5 = very high (see Impact of Sustainability Infrastructure). Data reported in this article were collected from 144 of 477 FT participants (30% response rate) who completed the 2016 implementation survey.

INFRASTRUCTURE APPROACHES AND SUPPORTING DATA

Approach 1: Develop a Master Facilitator Initiative to Increase the Number of Trained Facilitators

Whether or not an intervention has enough human capital to scale efforts is an important factor in determining whether it will be sustainable (Barker ). For the Entering Mentoring–based RMT curricula to be nationally disseminated (Handelsman ), a group of experts trained to deliver FT workshops were needed to increase the number of prepared facilitators to implement RMT (Figure 1; Ray ). Until 2015, a small team of five UW-Madison experts led the majority of the FT workshops (Pfund ). A Master Facilitator Initiative was developed as part of the National Research Mentoring Network (NRMN) to increase the number of experts across disciplines and institutions (McDaniels ; www.nrmnet.net). The primary goal of this initiative was to develop a group of experts who could lead mentorship trainings across the country on behalf of NRMN. However, a subset of this group was also trained to lead FT workshops, which are multiday, multifaceted events requiring substantial preparation (Figure 2).

FIGURE 2.

Using a Master Facilitator Initiative to build a national capacity for research mentor training.

To become a master facilitator, individuals must submit an application, which is reviewed internally. Requirements for master facilitators include participation in both RMT and FT, evidence of effective facilitation of RMT, commitment to equity and inclusion, and the ability to nurture self-reflection and learning in support of more equitable and efficacious mentoring relationships. Evidence of effective facilitation is collected from their training participants through a centralized evaluation (approach 4). To support the master facilitators, a professional development program was designed to encourage the continued growth of their expertise and ensure quality of facilitation. Key components of this professional development program include monthly newsletters, webinars, online learning communities, and annual in-person meetings.

Since the Master Facilitator Initiative was launched in 2015, 38 master facilitators have been designated, and the number of experts prepared to lead or co-lead FT workshops increased from 5 to 22. Enlarging the overall group of experts and increasing the number of experts capable of delivering FT enabled more FT workshops to be held nationally, thus increasing the number of facilitators trained to implement RMT. Overall, these efforts have resulted in 24 FT workshops in which 597 facilitators have been trained (Table 1). These FT workshop were held in a variety of settings, including public universities (UW-Madison, University of Cincinnati, Portland State University, University of Maryland, University of Minnesota, University of Puerto Rico, University of San Diego, Georgia State University), private institutions (Vanderbilt University, Boston University, Boston College), medical schools (Albert Einstein College of Medicine, Rutgers New Jersey Medical School, Medical College of Wisconsin), the Big Ten Academic Alliance, the annual conference of the Society for Chicanos/Hispanics and Native Americans in Science (SACNAS), and the Annual Biomedical Research Conference for Minority Students (ABRCMS).

TABLE 1.

Building capacity for research mentor traininga

Year	Number of master facilitators	Number of master facilitators able to deliver FT	Number of FTs	Number of trained facilitators	Number of RMT workshops led by trained facilitators
2010–2011	5	5	1b	38b	33c
2012–2013	9	9	9b	165b	60c
2014–2017	38	13	14	394	317c
Total	38	22	24	597	410c

aData source: internal tracking spreadsheets.

bData reported in Pfund .

cData from 2016 implementation survey (144 respondents).

FT workshop attendees represent a wide range of career stages as reported on the FT application and evaluation surveys. FT attendees were 24.2% (n = 82) full professors, 15.6% (n = 53) associate professors, 11.8% (n = 40) assistant professors, 10.9% (n = 37) training program directors, 9.1% (n = 31) deans or administrators, and 15.9% (n = 54) other (i.e., scientists, postdoctoral fellows, instructors). FT attendees were 66% (n = 213) female and 33% (n = 106) male, with 3 (0.9%) preferring not to report. FT attendees mostly self-identified as white (n = 189, 63%) and non-Hispanic (n = 203, 70.5%), with 40 (13.3%) identifying as Black or African American, 4 (1.3%) as American Indian or Alaskan Native, 29 (9.7%) as Asian, 6 (2.0%) as Native Hawaiian or Pacific Islander, 24 (9.4%) as Puerto Rican, 7 (2.7%) as Mexican or Chicanx, 3 (1.2%) as Cuban, 16 (6.3%) as other Hispanic, and 14 (4.7%) as unknown or not reported. Additionally, the FT attendees represent 152 academic institutions.

Data collected from the 2016 implementation survey indicate that FT participants are implementing RMT. Of the 144 survey respondents, 109 (76%) indicated that they facilitated RMT since attending the FT workshop, 21 indicated they did not implement but have plans to do so, and 14 reported they had not implemented training. The 109 participants who indicated they implemented training reported facilitating 410 RMT workshops (Table 1).

Approach 2: Adapt RMT Curricula for Multiple Audiences and Career Stages and Increase Accessibility

One of the key factors in promoting sustained dissemination is the extent to which material is adaptable, accessible, and available (Hutchinson and Huberman, 1994; Rabin and Brownson, 2012). Such resources reduce the time that frontline implementers, in this case the facilitators, need to prepare for implementation in their local settings (Froyd, 2007). Entering Mentoring was originally developed to improve the effectiveness of graduate student and postdoctoral mentors working with undergraduate researchers (Handelsman ). Over time, requests increased for curricular adaptations that retained the efficacy of the original curriculum, yet were relevant for multiple audiences and career stages. In response to these requests, new RMT curricula and modules were developed (Table 2). Through these efforts, Entering Mentoring was adapted for research mentors, graduate students to senior faculty, who work with trainees across various career stages and disciplines, across STEM (Pfund ). From 2010 to 2012, the curriculum was adapted for research mentors of junior faculty trainees engaged in clinical and translational, biomedical, clinical and behavioral, and community-engaged research (Pfund , 2014c; Sorkness ; Asquith ; House ). In 2016, the curriculum was further adapted for research mentors of trainees engaged in the social sciences (Robert and Asquith, 2017).

TABLE 2.

RMT curricula and module expansions

Complete curriculaa	Career stage
Entering Mentoring (1st ed.)	Mentors of undergraduate and graduate students
Entering Mentoring (2nd ed.)	Mentors of undergraduate and graduate students
Mentor Training for Clinical and Translational Researchers	Mentors of junior faculty and postdoctoral fellows
Mentor Training for Biomedical Researchers	Mentors of junior faculty and postdoctoral fellows
Mentor Training for Clinical and Behavioral Researchers	Mentors of junior faculty, postdoctoral fellows, and graduate students
Mentor Training for Community-Engaged Researchers	Mentors of junior faculty and postdoctoral fellows
Mentor Training for Social Science Researchers	Mentors of junior faculty and postdoctoral fellows

aComplete curricula and modules are available through www.cimerproject.org via download or by request.

New modules on focused topics were also developed, including: “Promoting Mentee Research Self-Efficacy” (Byars-Winston ; Butz ), “Enhancing Work–Life Integration” (Durbin and Meagher, 2015), and “Culturally Aware Mentoring” (Byars-Winston ). These new curricula and modules have been beta-tested, and assessment scales have been developed to assess training outcomes. RMT curricula and modules are often adapted and expanded for new audiences with support from cross-institutional partnerships and external funding.

Although the Entering Mentoring curriculum was adapted for multiple audiences and career stages, it also had to be adaptable for use in multiple formats. For example, the original Entering Mentoring curriculum was initially used in an 8-week summer seminar for graduate students and was not easily adjusted for other types of implementations (Handelsman ). Curricula in this series were not only revised for use across multiple audiences, but also to allow for shorter, modular implementation to increase adaptability. This adjustment allowed facilitators to implement different parts of the curriculum for use in their local contexts, customizing the content to the needs of the populations they serve and accommodating different training durations. Revisions included new, user-friendly facilitator guides with step-by-step implementation instructions, learning objectives, additional activities, and suggested time allotments to provide more clarity for implementation and strategies for adaptation.

Increased accessibility and availability of the curricula and resources were made possible through an open-access website developed to house Entering Mentoring complete curricula and modules for those mentoring junior faculty, postdoctoral fellows, graduate students, and undergraduate students (www.cimerproject.org). On the website, trained facilitators (and other interested parties) can download full curricula and complete modules or create customized curricula by “mixing and matching” modules and activities depending upon the content of interest, career stages, and research background of their local audiences. The website also contains planning and recruitment information, facilitator guides, evaluation tools, and other resources to promote easy implementation. New curricula and modules are made available on the website as they are developed, allowing facilitators access to new content.

Website usage data suggest that trained facilitators, as well as others, are accessing these resources to support their implementation efforts. To date, three websites have housed Entering Mentoring–based curricula. The first website, www.researchmentortraining.org (now inactive), allowed users to download complete and customized curricula. More than 4000 documents were downloaded from this site. The second website, https://ictr.wisc.edu/mentoring, which has since been reworked and no longer hosts curricular materials, allowed 397 unique individuals to download complete or customized curricula. The CIMER website (www.cimerproject.org), launched in 2017, hosts all of the full and customizable curricula.

Approach 3: Provide Implementation Resources to Address Barriers to Implementing RMT

Efforts to build a sustainable infrastructure to expand RMT should account for characteristics of institutions (local contexts for implementation), individuals (participants in FT), and the curriculum itself, from which barriers and enablers to implementation emerge (Ray ; Nilsen, 2015). The existence of these factors and associated barriers and enablers were considered when creating and disseminating resources for implementation.

As previously reported, despite increases in skills and confidence, FT workshop participants reported perceived barriers to implementing RMT in their local contexts (Pfund ). Subsequent qualitative data collection with thematic analysis (Boyatzis, 1998; Braun and Clarke, 2006) of recent FT workshop participants’ (N = 123) responses to the FT evaluation survey question “Please describe any additional resources you would need to effectively implement research mentor training?” indicates that institutional support (22%) and trainer support (21%) continue to be important for successful implementation. While 15% of respondents indicated they were ready to implement and did not need any additional support or resources, the majority of participants described needing assistance for planning and organization, supported institutional time for implementing training, consultation with experienced facilitators, and access to evaluation tools. Other less common themes included unknown or to be determined support (15%), access to training materials such as via website (15%), lack of professional time (5%), further participation in FT and/or RMT (5%), and other (2%).

In an effort to increase facilitation preparedness (Cohen and Ball, 2007) and the likelihood that facilitators would implement RMT, FT workshops were restructured in 2015 to focus more attention on addressing barriers to successful implementation. As part of the restructure, a new session on strategic planning was added so participants could learn about implementation resources and develop customized implementation plans. During this session, participants are provided with an implementation planning worksheet that includes a series of questions to consider regarding recruitment, training design and audience, resources for implementation, anticipated challenges, and “first next steps” upon returning to their home institutions (Table 3).

TABLE 3.

Implementation planning worksheet example questions

Recruitment
Who is your target audience for research mentor training on your campus? What recruitment strategies will you use?

Who do you need to convince that mentor training is worth the time and investment?

What information could be used to convince those stakeholders?

Which training do you plan to implement (career stage, disciplinary context)?

In what time frame and format will your training(s) be delivered (e.g., academic year, summer, multiple 1–2 hour sessions or single workshop, credit/noncredit)?

Will your training be integrated or serve an existing program (e.g., career development series for junior faculty, graduate or fellows training program), or will it stand alone?

What barriers to implementation of research mentor training might you face at your institution?

What strategies might be taken to overcome any barriers or challenges to implementation?

Barriers to implementation were also addressed through a targeted recruitment strategy that was enforced to ensure that FT participants have peer implementation support during and after the FT workshop. For example, participants from the same institution are encouraged to attend the FT together and requests to host FT workshops with a regional impact are given high priority. Priority registration is also given to applicants who indicate specific plans to implement RMT within the next year.

To ensure quality of FT workshops led by master facilitators, data are collected in the FT evaluation surveys. As in workshop satisfaction data previously reported (Pfund ), the majority of FT participants report components of the FT workshop as valuable. The three new FT components integrated since the 2015 article (Pfund ) are also favorably rated (Table 4).

TABLE 4.

Satisfaction with new or adapted FT workshop componentsa

Implementation and resources components	n	Average ratingb	SD
Data/evidence for recruitment	269	3.41	0.765
Website overview	261	3.34	0.792
Drafting a training implementation plan	257	3.50	0.730
Evaluation measures of research mentor training	247	3.518	0.7147

aData source: FT evaluation survey.

bWorkshop components were rated on a Likert scale: 1 = not at all valuable, 2 = somewhat valuable, 3 = valuable, 4 = very valuable.

During the FT workshop, time is set aside for networking, and participants are encouraged to work with attendees from the same or similar institutions during the strategic planning session to develop implementation plans. This approach to cohort building enables facilitators from similar institutions or disciplines to collaborate with one another and create capacity to connect after they return to their institutions (Froyd, 2007). Several open-ended questions were added in the FT evaluation survey to assess whether adding strategic planning components and opportunities for networking and peer support were helpful. In response to the question “If applicable, describe the connections you made during this workshop (e.g., Did you make connections with multiple attendees? Do you anticipate maintaining connections with attendees outside the workshop? Was the mix of attendees valuable?),” 114 responses were analyzed to discern the value and nature of connections made during the FT workshop. Overall, the data indicate that FT workshop design and activities promoted networking and informed participants’ implementation plans. Overwhelmingly, 98% of the survey respondents reported that they made connections with multiple attendees at the workshop and, of those, 100% indicated those interactions were valuable. Of the 112 participants who indicated that valued connections were made, 73% explicitly mentioned the merits of institutional connections: either meeting with colleagues at their own institution, other institutions, or both.

Additional data were collected to assess whether efforts to address barriers to implementation and increase facilitator preparedness were effective, including data collected in the FT evaluation survey and 2016 implementation survey. In the FT evaluation survey, participants retrospectively reported increased confidence in aspects related to implementation (Table 5), which agrees with previously reported data (Allen and Nimon, 2007; Pfund ). In the implementation survey, participants reported confidence gains comparing how they felt before attending the FT workshop and after they implemented RMT (Table 6).

TABLE 5.

Self-reported confidence gains before and after attending FTa

	Before			After
Learning objective	n	Mean	SD	n	Mean	SD	Differenceb
To use the available RMT curricula and supporting resources	276	3.06	1.440	279	4.99	1.281	p < 0.000
To describe evidence to support the effectiveness of RMT	278	2.96	1.542	279	4.67	1.414	p < 0.000
To facilitate RMT using the process-based approach	276	3.04	1.496	277	4.94	1.327	p < 0.000
To recruit mentors to participate in training	274	3.22	1.486	274	4.52	1.415	p < 0.000
To implement RMT at your home institution	275	3.17	1.599	277	4.95	1.425	p < 0.000
To use metrics and tools to assess the effectiveness and impact of RMT	277	2.92	1.552	277	4.47	1.488	p < 0.000
To understand the implementation process	161	3.41	1.723	161	5.58	1.273	p < 0.000

aData source: FT evaluation survey.

bDifferences in ratings before and after FT were determined by Wilcoxon signed-rank test. Confidence was rated on a seven-point Likert scale: 1 = not at all confident, 4 = moderately confident, 7 = extremely confident.

TABLE 6.

Self-reported confidence gains in ability to implement RMTa

Item	n	Mean	SD	Differenceb
Confidence before attending the FT	96	3.29	1.450	p < 0.000
Confidence now after implementing	95	5.93	1.024

aData source: 2016 implementation survey.

bDifferences in ratings before FT and after implementing were determined by Wilcoxon signed-rank test. Confidence was rated on a seven-point Likert scale: 1 = not at all confident, 4 = moderately confident, 7 = extremely confident.

In the implementation survey, participants also retrospectively rated their preparedness to implement RMT, reflecting back to how they felt after attending the FT workshop. Of the 98 survey respondents, 29 (30%) reported feeling “A great deal prepared,” 30 (31%) “A lot prepared,” 33 (34%) “A moderate amount,” and 6 (6%) “A little prepared.” Participants also answered how helpful the FT workshop was in helping them prepare for implementation. Of the 94 survey respondents, 91 (97%) reported the FT workshop was either extremely helpful or helpful.

Approach 4: Develop a Standardized Approach to Evaluate the Quality of RMT Implementations Nationally

Critical components in ensuring the quality of any implementation and dissemination effort are standardized evaluation and tracking tools (Hutchinson and Huberman, 1994; Barker ) and processes for using these data for the continuous improvement of the intervention effort as a whole (Baldwin ).

To increase capacity for ongoing assessment of intervention efficacy, both in terms of the quality of the curricula and facilitation efforts, an assessment tool was developed to test skill gains across mentoring domains (Fleming ) and made available to facilitators for evaluating RMT workshops at their home institutions. This Mentoring Competency Assessment was first used with facilitators in 2010 and then became freely available on an online Web portal after it was validated (Fleming ).

In 2016, a research and evaluation team was developed to standardize and systematize data collection and evaluation metrics for workshops led by master facilitators and trained facilitators using Entering Mentoring–based curricula. The main goal of this effort is to collect nationwide data to inform future implementation of and research on RMT. To incentivize use of assessment tools, the team provides customization of core assessment surveys; evaluation surveys include standardized questions so that aggregate data can be reported and compared but also allow for inclusion of site- or training-specific questions to meet the needs of the user. Assessment surveys are available for all types of implementations, no matter the length or venue.

The centralized evaluation system allows for data collection at multiple points in the dissemination process and for large-scale comparison of all FT workshops as well as the subsequent RMT. For example, FT participants are surveyed immediately after training and annually to track implementation efforts. As facilitators implement RMT locally, the research and evaluation team provides evaluation services for the RMT. Core data include satisfaction with the training, perceived skill gains and perceived quality of mentoring gains, intent to make changes in their mentoring relationships, mentoring background, and demographics (Supplemental Material). As of Summer 2017, this centralized evaluation system has been used to collect data from more than 200 events and 2000 participants (Table 7).

TABLE 7.

Standardized evaluation of traininga

	Master facilitators		Trained facilitators
Type of training	Number of events using centralized evaluation	Number of participant surveys	Number of events using centralized evaluation	Number of participant surveys
FT	23	293	—	—
RMT	77	879	63	561
Research mentee training	17	175	15	150
Total	117	1347	78	711

aData source: internal tracking spreadsheet.

IMPACT OF SUSTAINABILITY INFRASTRUCTURE

To increase the national dissemination of evidence-based RMT for mentors across career stages and disciplines, a multicomponent infrastructure was developed and continues to be refined to sustain and support the continued expansion of a national network of trained facilitators capable of offering RMT. Data in this article provide evidence for the effectiveness of the FT model and infrastructure approaches to increase the number of trained facilitators implementing RMT at their respective institutions (Figure 3).

FIGURE 3.

Model components and impact of infrastructure approaches for national dissemination of research mentor training.

Through the centralized evaluation and internal tracking systems, implementation and evaluation data are being collected from master facilitators, trained facilitators, and RMT participants across the nation. Using the implementation survey, trained facilitators report the number of RMT workshops they have implemented since attending the FT workshop (Figure 3). Trained facilitators were not asked to provide the number of training participants in their workshops in the implementation survey; however, these data are captured through internal tracking databases for trained facilitators using centralized evaluation. These databases indicate that, on average, 10 participants attend RMT workshops led by trained facilitators. Given that trained facilitators reported implementing 410 RMT workshops, it is reasonable to assume that more than 4000 research mentors have been trained.

Although not included in this article, initial analyses from RMT participants indicate high workshop satisfaction and increases in mentoring skills gains, suggesting that the vast majority of trained facilitators are effective. Further, self-reported data in the implementation survey indicate that the majority of trained facilitators rate their implementation quality as very high or high with 16 (17%) reporting the quality as very high, 49 (51%) as high, 29 (30%) as average, 2 (2%) as low or very low.

DISCUSSION

Over the past several decades, federal agencies have invested in projects that support the dissemination of evidence-based practices aimed at diversifying the students, faculty, and other researchers in STEM fields (National Institute of General Medical Sciences, 2011). Of particular concern in recent years has been the realization that the biomedical research workforce in the United States does not mirror the nation’s demographic diversity (Valantine and Collins, 2015; Pfund ). In response, the National Institutes of Health (NIH) announced the awardees of the $250 million NIH Diversity Consortium, a collaborative designed to develop and test interventions that engage trainees, including those from historically underserved backgrounds, and prepare them to thrive in NIH-funded careers (Valantine and Collins, 2015; Pfund ). One of the consortium members, the NRMN, aims to increase the diversity of scientists in NIH-funded research specifically through evidence-based mentorship and professional development programming (www.NRMNet.net; https://commonfund.nih.gov/diversity/Initiatives; Guerrero ; Jones ; Sorkness ).

In 2014, UW-Madison received federal funding to lead the Mentorship Training Core of NRMN and serve as a national hub to deliver evidence-based training programs designed to help mentors and mentees engage in productive, culturally responsive mentoring relationships. Several interventions have been developed and tested to address the need for equitable access to quality mentoring. A team from UW-Madison developed one such intervention—the Entering Mentoring RMT curriculum described in this article. Although this intervention and others have been well studied, it is critical that these approaches are also effectively disseminated. A previous publication (Pfund ) described the first step in supporting dissemination of the Entering Mentoring curriculum, which included development and testing of an FT workshop aimed at increasing the number of individuals able to effectively deliver Entering Mentoring to a larger audience.

This article builds upon the previously published work and describes the infrastructure that was subsequently developed and its impact, with support of NRMN funds, to promote both a sustainable expansion of FT workshops and the success of FT workshop participants as they implement the Entering Mentoring curricula in their local contexts. This approach was guided by the literature on dissemination, implementation, and scale-up (Hutchinson and Huberman, 1994; Cohen and Ball, 2007; Rabin and Brownson, 2012; Barker ), with a focus on increasing the accessibility and adaptability of the curricula, developing resources targeted at overcoming barriers to implementation, providing opportunities for connection among implementers, and monitoring implementation quality across local contexts. The four infrastructure approaches described in this paper include:

An expansion initiative to increase the number of expert facilitators able to deliver FT workshops nationwide

Accessible RMT curricula adapted for multiple audiences and career stages

Implementation resources to support facilitators and help them overcome implementation barriers

Standardized evaluation of training

Strong evidence supporting each of these infrastructure approaches and their national impact is reported for 597 trained facilitators, 30% of whom have reported on their implementation efforts. These facilitators self-report facilitating RMT in their local contexts for more than 4000 graduate student, junior faculty, and senior faculty mentors. There is evidence that this number is underreported by trained facilitators, as we know some have implemented through use of evaluation and did not complete the implementation survey.

Evidence also exists that diffusion, a more passive, unplanned dissemination process is taking place (Rogers, 2003; Rabin and Brownson, 2012). For example, website curricula downloads and evaluation requests indicate that individuals who have not participated in FT are implementing Entering Mentoring–based RMT. Some of these individuals have participated in RMT themselves, while others have discovered materials through national outreach efforts supporting the accessibility and usability of the Entering Mentoring curricula. Given the low implementation survey response rate and limited knowledge on implementation efforts of individuals who have not attended an FT workshop, other factors should be explored to further examine the impact of these approaches on increasing facilitator implementation.

Although this evidence supports the dissemination model, more data must be collected to understand why some FT attendees are not implementing and what infrastructure needs to be developed to address implementation barriers. Currently, data on implementation barriers and supports are being collected annually, including information on factors that may have impacted initial decisions/ability to implement training, ways institutions have supported efforts to implement training, and any institutional/organizational barriers encountered during the planning process. Initial analyses show commonalities among FT participants regarding individual and institutional barriers, such as lack of time and lack of administrative support, as well as the need for planning and organization assistance, supported institutional time for implementing training, consultation with experienced facilitators, and access to standardized, readily available evaluation tools. Future analyses can further examine these barriers and supports to understand what differences exist, if any, among different subgroups of facilitators identified by race/ethnicity, career stage, gender, and other variables.

It is additionally critical to address the facilitators who are using Entering Mentoring–based curricula but have not attended an FT workshop. As mentioned earlier, website tracking shows that individuals who have not attended FT or used centralized evaluation are downloading RMT curricula. Additional recruitment and outreach efforts need to be implemented so that these individuals can benefit from the infrastructure and resources developed.

Several efforts are underway to provide additional support to increase the number of facilitators prepared and motivated to implement training. In 2017, an NRMN Facilitator Certification Program was implemented to recognize facilitators who have demonstrated individual commitment to the local dissemination of RMT. Eligible facilitators receive a letter of recognition and exclusive access to curated resources. This recognition provides facilitators institutional leverage for implementing training efforts at their institutions or organizations. Other resources for trained facilitators, including online communities and opportunities for additional professional development and networking, are also in development.

Additional resources to support implementation across a range of contexts that are aligned with the needs of programs and institutions are essential for continued growth and scale-up, including resources for both RMT and research mentee training. New RMT curricula and modules are continuously in development and made available to facilitators on www.cimerproject.org. In addition to new material for research mentors, development of research mentee training curricula is also needed to empower trainees to get the most out of their research experiences. Entering Research, a research mentee training curriculum designed for undergraduate students in STEM fields (Balster ; Branchaw ), began revision in 2016 with support from NRMN. The updated curriculum, Entering Research, Version 2, is designed for use with both undergraduate and graduate students with varying levels of research experience. The new version includes revised and new activities and is organized thematically to allow for modular implementation. Once revisions are complete, the curriculum will be available via the website (www.cimerproject.org). Tailored FT workshops are being offered using a train-the-trainer approach to promote national dissemination.

Limitations

Several possible limitations are acknowledged in this research. First, much of the data on facilitator confidence, preparedness, and implementation efforts are self-reported by FT participants. However, data were also collected from the subsequent mentors who participated in training offered by these facilitators. These mentors largely indicated satisfaction with the training, comparable to previously reported data (Pfund ). Internal tracking databases and use of standardized evaluation also support the success of this model to increase the number of trained facilitators who implement training. Second, only 30% of respondents completed the implementation survey, which suggests that those who have not implemented RMT are not completing the survey. This limits data collection on implementation barriers for this group, providing a biased sample. Future data collection should focus on collecting data from those who have not implemented RMT.

Future Directions

The infrastructure described here is part of an important model for building a sustainable national infrastructure to broadly disseminate evidence-based RMT. However, several challenges still exist and must be addressed. It is important to ensure quality control of RMT as it is broadly implemented among master facilitators and trained facilitators. Centralized data evaluation has allowed for ongoing monitoring and testing of diverse implementations, while contributing to a national data set. Metrics will be designed and validated to assess new curricula and modules as they are developed. A new assessment platform is under development for use on the CIMER website (www.cimerproject.org). This platform will provide facilitators access to curated scales and assessment tools while compiling data for ongoing research on mentoring relationships and training effectiveness. It will also allow for inquiry into training effectiveness across diverse factors such as demographics of facilitators and RMT participants. This streamlined evaluation process will enhance ongoing monitoring of implementation quality and accommodate deeper investigation into the barriers and supports that impact facilitators’ decisions to implement training.

Most importantly, efforts are needed to both support and further understand institutionalization of RMT in local contexts, with a particular focus on how to effectively change institutional cultures to explicitly value and reward mentoring. To that end, now that a critical mass of trained facilitators exist within certain institutions, efforts to understand the impact of that critical mass and the role that academic leaders (deans, chairs, directors) play in supporting successful implementation can be investigated. In addition, significant potential exists for richer examination of the strategies new facilitators use upon returning to their home regions and/or institutions to create buy-in and support for RMT implementation. A conceptual lens that shines the light on facilitators as institutional change agents will be explored (Wall, 2009).

As funding mechanisms emerge for the dissemination of interventions that increase effective research mentoring, the infrastructure described here can be leveraged by institutions striving to address the calls for optimization of mentoring relationships for trainees across career stages and disciplines. The model we present can be used for national dissemination of other training interventions.

Click here for additional data file.