Building the Next Generation of Researchers: Mentored Training in Dissemination and Implementation Science.

PROBLEM: Dissemination and implementation (D&I) science provides the tools needed to close the gap between known intervention strategies and their effective application. The authors report on the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) program-a D&I training program for postdoctoral or early-career cancer prevention and control scholars. APPROACH: MT-DIRC was a 2-year training institute in which fellows attended 2 annual Summer Institutes and other conferences and received didactic, group, and individual instruction; individualized mentoring; and other supports (e.g., pilot funding). A quasi-experimental design compared changes in 3 areas: mentoring, skills, and network composition. To evaluate mentoring and D&I skills, data from fellows on their mentors' mentoring competencies, their perspectives on the importance of and satisfaction with mentoring priority areas, and their self-rated skills in D&I competency domains were collected. Network composition data were collected from faculty and fellows for 3 core social network domains: contact, mentoring, and collaboration. Paired t tests (mentoring), linear mixed models (skills), and descriptive analyses (network composition) were performed. OUTCOMES: Mentors were rated as highly competent across all mentoring competencies, and each mentoring priority area showed reductions in gaps between satisfaction and importance between the 6 and 18 months post-first Summer Institute. Fellows' self-rated skills in D&I competencies improved significantly in all domains over time (range: 42.5%-52.9% increase from baseline to 18 months post-first Summer Institute). Mentorship and collaboration networks grew over time, with the highest number of collaboration network ties for scholarly manuscripts (n = 199) in 2018 and for research projects (n = 160) in 2019. NEXT STEPS: Building on study findings and existing literature, mentored training of scholars is an important approach for building D&I skills and networks, and thus to better applying the vast amount of available intervention evidence to benefit cancer control.

Problem

Dissemination and implementation (D&I) science—the study of the processes and factors that are associated with the successful integration of evidence-based interventions within various settings—provides the tools needed to close the gap between scientific knowledge about how to prevent and control cancer and the application of effective interventions for individuals and populations.[1] Accelerated progress toward closing this gap requires new approaches for training D&I scholars. Mentoring is a critical component of training, as it can increase research productivity and collaboration.[2]

Additionally, transdisciplinary teams and social networks foster mentoring, training, and scientific collaboration.[3] Social network analysis is the mapping and measuring of relationships and flows between people, groups, and organizations.[4] Using social network analysis allows researchers to go beyond examining single, dyadic, mentoring relationships to assess: the structure of mentoring networks, information exchange, diagnostic leverage points (e.g., connecting a mentee who is isolated), and whether certain mentoring patterns (e.g., high levels of peer-to-peer mentoring) lead to increased research skills.

To assist in building the next generation of researchers, the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) program was created. The MT-DIRC program was a D&I training program for postdoctoral or early-career scholars (MDs, PhDs) in cancer prevention and control and was, to our knowledge, the first of its kind. This program was based on the pedagogical philosophy that group training and one-on-one mentoring are key components of training success. In this Innovation Report, we describe the program’s 2 aims: to build human capital for D&I science via direct skill building and evidence-informed mentoring and to develop a collaborative network of faculty and fellows.

Approach

The MT-DIRC program was a 2-year training institute in cancer D&I research, which trained 58 fellows between 2014 and 2019.[5] Driven by our core competencies (i.e., 43 unique D&I competencies in 4 domains; see below), we developed MT-DIRC in 2013.[6] Fellows in the program attended 2 annual Summer Institutes where they received didactic, group, and individual instruction. Fellows also attended broader national D&I conferences, received research pilot funding, and were provided with 11 webinar sessions from 2015 to 2019.

We assigned a mentor to each fellow to provide individualized mentoring over the 2 years of the program. Faculty mentors were trained via an asynchronous, self-paced component[7]; a synchronous online component[8]; and a face-to-face training session at the Summer Institutes.

Evaluation design

We used a quasi-experimental (pre/post) design to evaluate MT-DIRC using data on mentoring, skills, and network composition.

Measures

To evaluate mentoring, we asked fellows to rate the skills of their mentors using the Mentoring Competency Assessment (MCA), a validated 26-item skill inventory that evaluates 6 mentoring competencies (maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, fostering independence, and promoting professional development), using a 7-point anchored Likert scale (where 1 = not at all skilled and 7 = extremely skilled).[9] We also collected data on fellows’ perspectives on the importance of and their satisfaction with 8 mentoring priority areas (skill building, sharing resources and infrastructure, performance feedback, providing opportunity, career planning, professional networking, professional socialization, and providing emotional support), using a 5-point anchored Likert scale (where 1 = very unimportant/very dissatisfied and 5 = very important/very satisfied).

To evaluate skill change, we developed a D&I skills survey to assess fellows’ self-rated achievement of 43 D&I competencies within 4 domains (definitions, background, and rationale; theory and approaches; design and analysis; and practice-based considerations), using a 5-point anchored Likert scale (where 1 = not at all skilled and 5 = extremely skilled).[6]

To evaluate network composition, we collected data on 3 core social network domains (contact, mentoring, and collaboration) from both faculty and fellow program participants.[4] For the contact domain, we provided participants with a list of MT-DIRC fellows and faculty and asked them to report on who they knew and, on average, the frequency of their direct contact with those individuals. For the mentoring domain, we provided participants with a list of individuals they had contact with and asked them to indicate individuals who had mentored them or been mentored by them since the previous Summer Institute. For the collaboration domain, we asked participants to indicate their engagement in any of the following activities since the previous Summer Institute with individuals with whom they had contact: joint research projects, scholarly manuscripts, grant writing, teaching, or presenting research.

We collected data on skills at baseline and 6 and 18 months after completing their first Summer Institute (2014–2018). We collected data on mentoring at 6 and 18 months post-first Summer Institute (2014–2018). We collected network composition data each year from 2015 to 2019. All surveys are publicly available (see https://sites.wustl.edu/mtdirc/resources/webinar-archive).

Statistical analysis

For mentoring, we subtracted satisfaction scores from importance scores to create a mean “gap” for each priority area. These gaps allowed for satisfaction to be examined relative to inferred expectations based on the rated importance of a priority area. Mean differences in gaps were assessed between the 6-month and 18-month time points with paired t tests. We also compared 6-month and 18-month MCA ratings with paired t tests.

We assessed changes in D&I skills for each fellow with linear mixed models to examine the fixed effect of time on skill change.

Social networks were analyzed with descriptive statistics that examined changes in the connectivity of the networks and patterns of connections between roles and disciplines over time.

This study was approved by the institutional review board at Washington University in St. Louis (#201303122).

Outcomes

Due to professional or personal circumstances, 3 of the 58 fellows left the program before attending their second Summer Institute and thus were excluded from all analyses. Response rates on the demographics, mentoring, and skills surveys across the 3 time points ranged from 96.4% (53/55) to 100% (55/55).

Participant characteristics

Overall, fellow characteristics were similar across 4 cohorts (i.e., the 2014–2017 cohorts; see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B25). The majority of fellows were female (43/55, 78.2%), White (41/52, 78.8%), and were from the United States (35/42, 83.3%). Fellows’ cancer research focused mostly on prevention (27/55, 49.1%), followed by treatment (12/55, 21.8%), and survivorship (10/55, 18.2%). Most fellows had access to a local D&I mentor (38/53, 71.7%).

Mentoring

Fellows rated their assigned mentors as highly competent across all MCA items (range: 5.95–6.24, on a 7-point scale; 2014–2018). Overall competency ratings between the 6-month and 18-month surveys did not significantly differ (mean difference = 0.17, 95% confidence interval [CI] –0.08 to 0.42), though ratings in all competency areas decreased slightly.

Gaps between satisfaction with and importance of mentoring (both measured on a 5-point scale) ranged between 0.60 and −1.08 among the 8 mentoring priority areas in the 6 months post-first Summer Institute survey and between 0.31 and −1.10 in the 18 months post-first Summer Institute survey (2014–2018), with positive values indicating higher ratings of importance than satisfaction and negative values indicating higher ratings of satisfaction than importance and a possible range of −4 to 4. Each of the 8 mentoring priority areas showed reductions in gaps within the 12-month time period between the 2 post-first Summer Institute surveys, though gaps for mentors’ sharing resources and infrastructure and professional networking were the only 2 priority areas that were significantly reduced (mean difference = 0.37, 95% CI 0.04 to 0.69 and mean difference = 0.32, 95% CI 0.00 to 0.64, respectively). Two priority areas, professional socialization and providing emotional support, remained below the equal threshold of 0, indicating that although fellows were highly satisfied with these mentoring priority areas, they did not rank these priority areas as highly in terms of importance.

Skill change

Fellows’ self-rated skills in D&I competencies (on a 5-point scale) improved significantly in all 4 domains areas over time (2014–2018, Figure 1). Average competency in the definitions, background, and rationale domain increased from 2.85 at baseline to 3.70 at 6 months after the first Summer Institute, and 4.06 at 18 months after the first Summer Institute [F(2,104.02) = 80.58, P < .001)], a 42.5% increase. Average competency in the theory and approaches domain increased from 2.49, to 3.33, to 3.78 from baseline, to 6 months, to 18 months [F(2,104.32) = 115.48, P < .001)], a 51.8% increase. Average competency in the design and analysis domain increased from 2.38, to 3.25, to 3.64 [F(2,104.03) = 102.86, P < .001], a 52.9% increase, and in the practice-based considerations domain increased from 2.82, to 3.59, to 4.06 [F(2,104.19) = 92.64, P < .001], a 44.0% increase, from baseline, to 6 months, to 18 months.

Figure 1

Changes in Mentored Training in Dissemination and Implementation Research in Cancer fellows’ self-rated dissemination and implementation (D&I) skills over time, grouped by 4 competency domains, 2014–2018. Valid responses from survey participants on skills data for the different time points were: baseline (n = 55), 6 months post-first Summer Institute (n = 52), and 18 months post-first Summer Institute (n = 53). Competencies were rated on a 5-point anchored Likert scale (where 1 = not at all skilled and 5 = extremely skilled).

Network composition

A total of 68 people participated in MT-DIRC from 2015 to 2019, including 13 faculty and 55 fellows. Response rates on the social network composition survey ranged from 100% (68/68) in 2017 to 89.7% (61/68) in 2019. The average number of mentor relationships that fellows reported receiving via the program ranged from 4.1 in 2015 and 2019 to 5.8 in 2018. The average number of collaborations for fellows established through the program ranged from 5.4 in 2015 to 7.9 in 2018.

The mentorship and collaboration networks grew over time (Figure 2). For mentorship, nodes (members) are connected with an arrow if either or both members indicated providing mentorship to or receiving mentorship from the other over the previous year, with the arrows pointing in the direction of those who received mentorship. It was possible to have bidirectional mentoring. The size of the nodes corresponds to out-degree (i.e., the number of members to whom they provided mentorship). For collaboration, nodes are connected with a line if either or both members indicated any of the 5 types of collaboration over the previous year. The size of the nodes corresponds to degree (the number of members with whom they are connected). Faculty (black nodes) tended to provide more mentorship and have more collaborations than fellows (gray nodes) and are therefore more centrally located in the networks.

Figure 2

Mentoring and collaboration networks among Mentored Training in Dissemination and Implementation Research in Cancer fellow and faculty participants, 2015 and 2019. The arrows point in the direction of those who received mentorship; it was possible to have bidirectional mentoring. For mentorship, nodes (members) are sized by the number of members to whom they provided mentoring. For collaboration, nodes are sized by the number of members with whom they are connected. Faculty (black nodes) tended to provide more mentorship and have more collaborations than fellows (gray nodes), leading to more central positions in the networks. Network sizes for the different time points were: 2015 (n = 38) and 2019 (n = 68).

The overall MT-DIRC network grew as additional faculty and fellows joined the program in 2016 and 2017, then held steady in 2018 and 2019 (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B25). The contact network (at least at the yearly level) had the highest connectivity of the 3 core social network domains, ranging from an average number of connections (average degree) of 20.09 in 2019 to 30.40 in 2016. Modularity is a measure of the extent to which ties occur within subgroups in the network (range: −0.5 to 1, where higher values indicate ties mostly occurring within subgroups and lower values indicate ties mostly occurring between subgroups). Discipline modularity was near zero for all networks, indicating a random pattern of contact, mentoring, and collaboration between disciplines. Role modularity demonstrated a slight negative pattern for mentoring, which makes sense given that faculty were expected to provide mentorship to fellows while fellows were not expected to provide mentorship to each other. Degree centralization is the extent to which a network has either one or a few individuals with many connections while everyone else has a smaller number of connections (range: 0–1, where 0 = everyone has the same number of connections and 1 = a “star” network where one person is connected to everyone else and everyone else is only connected to that central person), and betweenness centralization is the extent to which a network is dependent on one or a few individuals to keep everyone else connected (range: 0–1, where 0 = everyone is connected to everyone else and 1 = a star network as described above). Degree centralization was generally higher than betweenness centralization, with a few people having a high number of connections. Betweenness centralization was somewhat elevated in the last 2 years (2018 and 2019) for collaboration, as one faculty member connected to many other people who were not otherwise connected to each other.

Members of the MT-DIRC network reported several types of collaboration over time, with the 3 most common types being joint research projects, scholarly manuscripts, and grant writing (see Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/B25 and Figure 3). Publishing or writing a scholarly manuscript was the most common type of collaboration for all years except for 2019, peaking in 2018 with 199 network ties. Engaging in a joint research project was the second-most common type of collaboration, peaking in 2019 with 160 network ties. Role modularity was generally low across all types of collaboration, with the highest value being the one for presentations in 2015 (Q = 0.263), showing that these types of collaborations were not restricted to within-faculty or within-fellow relationships. The percentage of collaborations between individuals that were of more than one type (e.g., engaging in a joint research project, presenting that research) grew steadily over time, from 26.4% (33/125) in 2015 to 37.8% (104/275) in 2019.

Figure 3

Collaboration networks for the 3 most common types of collaboration (joint research projects, scholarly manuscripts, and grant writing) among Mentored Training in Dissemination and Implementation Research in Cancer participants by collaboration type and year, 2015–2019. Nodes (members) are sized by the number of members with whom they are connected. Publishing or writing a scholarly manuscript had the most network ties for all years, with the exception of 2019 when engaging in a joint research project had more network ties. Network sizes for the different time points were: 2015 (n = 38), 2016 (n = 55), 2017 (n = 68), 2018 (n = 68), and 2019 (n = 68).

Next Steps

As the only large-scale, multiyear, D&I mentored training program focused on cancer prevention and control of which we are aware, MT-DIRC outcomes on mentoring, skills, and network composition from 3 time points we present here suggest that it may be an effective model. While other D&I training programs also include some type of evaluation, many only include short-term follow-up. Our outcomes show significant increases in self-rated D&I skills and overall MT-DIRC social network growth over time.

Mentoring is a core element for success in D&I research,[4,5] yet most senior scholars learn mentoring skills more from on-the-job training than from systematic inquiry and reflection.[2] The MT-DIRC program’s application of evidence-informed mentoring[5,6] was based on what we learned over the course of the program and on existing literature[4,7,8]; thus, we:

Built a mentoring network outside of the fellow’s home institution;

Fostered peer-to-peer mentoring;

Applied a tested mentorship curriculum;

Established trusting relationships and provided positive feedback to fellows;

Conducted regular, scheduled communication (at a minimum, once a month);

Developed mentoring contracts/compacts (i.e., an agreement that outlines expectations, the mentoring process, and time frames) and individual development plans; and

Applied mechanisms for feedback and improvement.

Use of a social network perspective in the development of the MT-DIRC program allowed for more efficient and effective transfer of training, with the network itself serving as an important outcome. Regular examination of the social network identified fellows who were isolated, allowing us to seek ways to better connect them to other members.[10] Examining measures such as modularity enabled our understanding of the mix of mentoring relationships between faculty and fellows.

There are other spin-off benefits and impacts from our collaborative network. For example, 3 core faculty developed the first comprehensive textbook on D&I with multiple chapters coauthored by MT-DIRC faculty members and fellows.[1] The long-term impacts of a program such as MT-DIRC lie in the increased uptake of evidence-based practices and ultimately better prevention and patient care. While the time window presented here is too short to examine such extensive systems changes, research being pursued by MT-DIRC fellows is already showing practice impacts. For example, one fellow helped to secure funding through their state health agency responsible for overseeing the Medicaid health program to provide technical assistance and support to 15 coordinated care organizations (CCOs), including education on evidence-based interventions and best practices for their uptake. Early results show increases in cancer screening rates across these CCOs.

The main limitation of our program evaluation is that it relies on self-reported data on skills and mentoring. It is also difficult to comprehensively measure the myriad of factors that can lead to progress in D&I research. Research productivity and impact are likely to be related to a complex mix of mentoring, training, environment, and timing factors.

To move cancer prevention and control forward, thus applying the vast amount of available intervention evidence, D&I capacity needs to be enhanced and must demonstrate “evolvability” (i.e., the ability to take up new methods and questions). Early outcomes suggest the MT-DIRC model has an impact on D&I skills and social networks and seems to show early signs of practice changes, and thus may be a promising blueprint for other D&I training programs.

Acknowledgments:

The authors thank faculty members Maureen Dobbins, Matt Kreuter, Shiriki Kumanyika, Enola Proctor, and Kurt Stange. They also appreciate the administrative support of Linda Dix, Mary Adams, and Cheryl Valko at the Prevention Research Center, Washington University in St. Louis.