Developing Competencies for Health System Impact: Early Lessons Learned from the Health System Impact Fellows.

BACKGROUND: The Health System Impact (HSI) Fellowship program provides highly qualified post-doctoral fellows studying health services and policy research (HSPR) with opportunities for experiential learning, enriched core competency development and mentorship from senior-level leaders within health system organizations. Its overall aim is to prepare post-doctoral fellows with the research and professional skills, experiences and networks to make meaningful and impactful contributions in careers in academic and applied health system settings.
OBJECTIVE: This study examined whether this HSI Fellowship program has contributed to the development of enriched core competencies in HSPR.
METHODS: A competency assessment tool was developed and administered to the 46 fellows and their health system and academic supervisors from the inaugural HSI Fellowship cohort. Fellows' self-assessments at baseline, three months and 12 months were analyzed, along with supervisors' assessments at three and 12 months. Descriptive analyses were used to examine competency development over time. Differences by gender and between supervisor and fellow ratings were analyzed.
RESULTS: HSI fellows' self-assessments indicate that they strengthened their skills in all 10 enriched core competencies. Supervisors' assessments of the fellows' competencies also improved from baseline to 12 months. Gender differences at baseline disappeared by the 12-month assessment.
CONCLUSION: The HSI Fellowship provides an opportunity to develop the full suite of enriched core competencies, particularly in competency domains that are not currently emphasized in HSPR doctoral curriculum.

Background

An enriched core competency framework for health services and policy research (HSPR) doctoral and post-doctoral training was introduced in Canada in 2016 (Bornstein et al. 2018; CHSPRA TMWG 2015). Its goal was to align the competencies emphasized in doctoral curriculum with the skills needed to maximize the impact of health services and policy research (HSPR) graduates in the diverse range of employment sectors and roles they are entering based on employment trends (see McMahon et al. 2019b) and emerging career opportunities, such as embedded scientist roles within organizations seeking to be learning health systems (Reid 2016). The enriched core competency framework maintains emphasis on rigorous scholarly training and research and analytic skills but also introduces a core suite of professional skills that are demanded in the non-academic labour market and necessary to effect change and make an impact in applied health system settings (Figure 1).[1]

Figure 1.

Enriched core competencies for health services and policy research

The inaugural cohort of 46 Health System Impact (HSI) fellows was the first to pilot a training program for the enriched core competencies. The HSI Fellowship program provides fellows with an experiential learning opportunity within a health system organization (defined as a public, private, or not-for-profit organization based in Canada with a mandate focused on health, such as a ministry of health, regional health authority, health service delivery organization, health quality council, public health unit, health charity, health professional association, consulting firm with a health-focused mandate, etc.) where they spend the majority of their time, as well as protected time for academic research at a Canadian university; co-supervision and mentorship from a health system leader within the health system organization and an academic supervisor with expertise in HSPR at the university; a professional development training allowance; and participation in a national cohort of fellows and health system and academic leaders that includes an annual in-person National Cohort Retreat and quarterly webinar training sessions in enriched core competencies (for a full description, see CIHR-IHSPR [2016]; McMahon et al. [2019a]; and McMahon and Tamblyn [2019c]). Each of these program elements is deliberately aligned with the suite of enriched core competencies and provides opportunities for fellows to complement their research skills with professional competencies and enhanced skills.

At the outset of the fellowship, fellows identify three enriched core competencies from the suite of 10 (see Figure 1) to target for development. Fellows create a professional development plan (PDP) and meet with their health system and academic supervisors to discuss their mentorship and support. To track competency development over time, fellows use a standardized framework to self-assess their competencies at the start of their fellowship (baseline) and two subsequent time points. Each fellow's health system and academic supervisor also assesses the fellow's competency strength and development.

The competency assessments provide an opportunity to learn whether the HSI Fellowship is contributing to the development of enriched core competencies and preparing a new cadre of doctoral graduates equipped with the research and professional skills to make an impact in a broad range of employment sectors and roles.

The objective of the present study was to analyze whether the inaugural cohort of HSI fellows' competencies evolved over the first year of the fellowship, if development occurred primarily in the self-identified target areas, and to assess the degree of alignment between fellows' and supervisors' competency assessments.

Methods

To measure the fellows' development of the enriched core competencies over the first year of the HSI Fellowship, the CIHR HSI Fellowship Professional Development Plan and Competency Tracking Tool assessments were analyzed.

The competency tracking tool was created by the Canadian Institutes of Health Research Institute of Health Services and Policy Research (CIHR-IHSPR) in collaboration with the Training Modernization Working Group of the Canadian Health Services and Policy Research Alliance (CHSPRA) based on a review of the literature and an environmental scan of existing competency assessment tools used in relevant fields. [2] The draft tool was presented to a group of deans and directors of 10 of Canada's HSPR doctoral training programs and health system leaders from across the country[3] for feedback and refinement prior to finalizing. It was introduced to the HSI Fellows and their supervisors via an orientation webinar in the first month of the fellowship prior to its implementation for baseline assessment. Completion of the competency assessments was a mandatory requirement of the HSI Fellowship.

Competency assessments were completed using an online survey platform at three defined time points: at baseline, three months and 12 months (see Figure 2 for a timeline overview). Questions and response scales were consistent in all assessments to allow for the analysis of change over time and comparisons between fellow and supervisor ratings. At baseline, fellows were asked to:

Figure 2.

Competency assessment timeline

identify their top three competencies for development over the course of their fellowship;

rate the strength of their competence on a Likert-type scale (range: 1 to 5) in each of the 10 competencies relative to their perception of others' competence in their peer group; and

discuss their PDP with their health system and academic supervisors, who were required to indicate how they intended to help the fellow achieve his or her goals.

At three months, fellows and their health system and academic supervisors completed the assessment (separately). Fellows were asked to indicate which competencies they had the opportunity to strengthen over the past three months and, for each competency, to rate how much they felt they learned (response options: a lot, a fair bit, a little bit, nothing). Similarly, health system and academic supervisors rated the extent of their fellow's development in each of the competency domains (response options: a lot, a fair bit, a little, not at all) and rated the overall strength of the fellow's competence in each domain using the 5-point Likert-type scale.

At 12 months, fellows and supervisors rated the fellows' competency strength in each domain using the 5-point Likert-type scale. Additionally, fellows were asked to rate the importance of nine program-specific enablers to their competency development. These included: (1) being embedded in a health system organization, (2) mentorship from a health system leader, (3) working on an impact-oriented program of work of high relevance to a health system organization, (4) mentorship from the academic supervisor, (5) co-mentorship (team-based approach) from the health system and academic supervisors, (6) fellow-to-fellow interactions, (7) protected time for academic research, (8) the professional development training allowance and (9) the National Cohort Retreat and quarterly webinar training sessions.

Descriptive analyses were used to assess competency development over time for all fellows, as well as by gender. Gender differences were assessed using a two-sample t-test, and differences between the three-month and 12-month assessments were assessed using a paired-sample t-test in the sample of fellows for whom both a baseline and a 12-month assessment were available (n = 38/46 [82%]). The magnitude of change over time was assessed by calculating the difference in self-rated ability at baseline and 12 months, expressed as a per cent change of the difference relative to baseline. The same methods were used to assess differences between health system and academic supervisor ratings and differences between the supervisors' three- and 12-month ratings. Analyses and figures were conducted using Excel.

Results

Overall, all 46 fellows completed the baseline and three-month competency assessments and 38 completed the 12-month assessment. Of the nine non-responders, five had transitioned into employment positions or were on leave at the time of the 12-month assessment. Seventy-eight of 92 supervisors completed the three-month assessment (84.8%), and 69 completed the 12-month assessment (75%). For all but two fellows, at least one of the health system and academic supervisors completed a three- and a 12-month assessment.

At baseline, fellows were asked to identify their top three competencies to develop. Figure 3 illustrates some consistency in fellows' top two selections, which were leadership, mentorship and collaboration (identified by 60% of fellows) and change management and implementation (identified by 49% of fellows). There was heterogeneity thereafter in fellows' self-identified target competencies.

Figure 3.

Fellows' baseline target competencies and opportunity to develop competencies over time

Figure 3 also shows that although fellows identified three competencies to develop at the outset of their fellowship, by the three-month mark, approximately 80% of fellows reported that they had the opportunity to strengthen their competence in all 10 domains, and by 12 months, 85% had. Even competencies that were not prevalent selections at the start of the fellowship witnessed considerable development over time. For example, at baseline, only 15% and 17% of fellows identified interdisciplinary work and networking, respectively, as a top three target competency for development, but by 12 months, more than 97% of fellows reported that they had the opportunity to strengthen their skills in these domains.

Table 1, available here, presents fellows' self-assessed ratings of their strength in each competency domain at baseline and 12 months. At baseline, the mean overall cohort rating was highest in the analysis of data, evidence and critical thinking and lowest in the change management and implementation competencies. By 12 months, the mean cohort rating in all 10 competencies had significantly increased (p < 0.01), ranging from a 9.5% increase in the analysis of data competency to a 44% increase in the change management and implementation competency. Next to change management, self-assessed competence in understanding health systems and the policy-making process exhibited the greatest increase over time (31.2% increase), followed closely by networking (28.8% increase) and dialogue and negotiation (29.2% increase).

At baseline, the mean competency assessment rating for female fellows (n = 38) was higher than the mean for male fellows (n = 9) in six of the 10 domains, although statistically significantly higher in only two: project management and networking (see Table 1). At 12 months, there were no statistically significant gender differences in competency assessment ratings.

Table 2, available at available here, presents the supervisor ratings of their fellows' competency strength in each domain at three and 12 months, overall and by supervisor type. At both time points, the mean overall supervisor ratings were highest in networking and analysis of data, evidence and critical thinking and lowest in change management. As with the fellows' self-assessments, the overall supervisor assessments of fellows' competency strength increased significantly over time in each domain (p < 0.01), ranging from a 12.2% increase in dialogue and negotiation to a 22% increase and in the analysis and evaluation of health policies and programs. When assessed separately by supervisor type, health system supervisor assessments increased significantly over time in all 10 competency domains and academic supervisor assessments increased significantly in all but two domains: project management and dialogue and negotiation.

Differences between health system and academic supervisor competency assessments were not statistically significant except for the three-month assessment of the leadership competency and the 12-month assessment of interdisciplinary work where, in both cases, the mean academic supervisor assessment was higher (see Table 2).

Table 3 compares the fellow and overall supervisor competency assessments at 12 months and shows that in seven of 10 competencies, the supervisors' assessment of the fellows' competency strength is higher than the fellows' self-assessment; however, the differences are not statistically significant.

Table 3.

Fellows' and supervisors' competency assessments at 12 months

Competency	Fellows Mean (SD) (n = 38)	Supervisors Mean (SD) (n = 69)	Δ Mean, baseline to 12 months (fellows)	Δ Mean, 3 to 12 months (supervisors)
Leadership, mentorship and collaboration	3.81 (0.61)	4.00 (0.73)	0.61	0.5
Change management and implementation	3.50 (0.95)	3.45 (0.93)	1.07	0.41
Project management	3.68 (0.93)	3.96 (0.93)	0.59	0.46
Interdisciplinary work	4.21 (0.70)	4.13 (0.82)	0.61	0.49
Networking	4.03 (0.72)	4.26 (0.72)	0.9	0.53
Dialogue and negotiation	3.76 (0.79)	3.67 (0.93)	0.85	0.4
KT, communication and brokerage	3.76 (0.71)	3.97 (0.82)	0.76	0.55
Analysis and evaluation of health policies/programs	3.82 (0.77)	3.94 (0.94)	0.33	0.71
Analysis of data, evidence and critical thinking	4.14 (0.66)	4.30 (0.77)	0.43	0.58
Understanding health systems and the policy-making process	3.74 (0.89)	3.87 (0.78)	0.89	0.68

KT = knowledge translation.

Table 4 shows that of nine defined HSI Fellowship program enablers designed to support fellows' competency development, fellows rated being embedded in a health system organization (mean rating 4.72, SD 0.57) and having a dedicated professional development training allowance (4.73, SD 0.56) as the top two enablers, followed by mentorship from the health system supervisor (4.35, SD 0.98). All nine enablers were rated at least moderately important (mean rating of 3 or higher).

Table 4.

Fellow ratings of the importance of nine program elements designed to enhance competency development

Program element	Mean rating (SD)
The professional development training allowance	4.73 (0.56)
Being embedded in a health system organization	4.72 (0.57)
Mentorship from my health system supervisor	4.35 (0.98)
Working on an impact-oriented project of high importance to a health system organization	4.32 (0.94)
Mentorship from my academic supervisor	4.24 (0.95)
Co-mentorship (team-based approach) from my health system and academic supervisors	4.12 (1.19)
Protected time for my academic research	3.91 (1.26)
National Cohort Retreat and quarterly webinar training sessions	3.84 (1.09)
Fellow-to-fellow interactions	3.35 (1.25)

Note: Ratings are based on a 5-point Likert-type scale.

Discussion and Conclusion

This study examined whether the HSI Fellowship contributed to the development of enriched core competencies in HSPR, if development occurred primarily in fellows' self-identified target competencies and the extent to which fellows' and supervisors' competency assessments align. The study's overall goal is to advance knowledge about the effectiveness of embedded fellowship training programs in fostering the development of competencies that are demanded in the labour market and needed to create a cadre of research-savvy health system leaders with the skills to drive health system innovation and transformation.

This study found that HSI fellows strengthened their skills in all 10 enriched core competencies over the first year of the fellowship. Fellows' baseline assessments were, in general, lower in the professional competency domains that are not yet emphasized in doctoral training programs (e.g., change management) and higher in the traditional research and analytic competencies that are the focus of doctoral curriculum (e.g., analysis of data). However, fellows' self-assessment of their competencies improved from baseline to 12 months in both the professional and research competencies, as did the supervisors' assessments of the fellows' competencies. Improvements were observed in fellows' top three targeted competencies and in less-targeted competencies. Gender differences at the baseline assessment disappeared by the 12-month assessment. Overall, these findings suggest that the HSI Fellowship provides an opportunity for exposure to and development of the full suite of enriched core competencies and, in particular, in competency domains that are not currently emphasized in HSPR doctoral curricula.

According to the fellows, being embedded in a health system organization, having a dedicated professional development training allowance, receiving mentorship from a health system leader and working on an impact-oriented project of high importance to a health system organization were key enablers to their development of the enriched core competencies. These program elements are unique to the HSI Fellowship program when compared to conventional fellowships and are not yet systematically offered within HSPR doctoral training programs. To ensure the long-term sustainability of training modernization that maximizes doctoral graduates' impact in a broad array of employment sectors and roles, it will be important for Canada's university-based HSPR doctoral training programs to institutionalize the enriched core competencies within their curriculum, provide trainees with opportunities for field immersion within health system organizations and consider greater involvement of health system leaders in graduate training and mentorship.

This study has strengths and limitations. A notable strength is the use of a competency assessment tool to study and learn whether and how the program is working. The tool was administered at baseline and two subsequent time points to allow for assessment of change over time and was administered to fellows and their supervisors to allow for analysis of alignment in ratings. A limitation is the lack of a control group, such as a cohort of fellows funded through a conventional research fellowship, to test for a causal relationship between the intervention (the fellowship) and the outcomes (the competency development). Additionally, the assessments are based on fellow self-report and supervisor report and not on a validated evaluation of competencies. Potential bias could arise if fellows or supervisors underassessed early on in the fellowship in order to demonstrate improvement over time or, similarly, if they inflated their assessments at the 12-month mark in an effort to show improvement and program success.

Moving forward, analyzing the career trajectories of HSI fellows will be critical to understanding whether the program is achieving its objective of preparing doctoral graduates for success and impact in a broader array of employment settings, both within and beyond the academy. Emerging evidence based on the career transitions of the first cohort of one-year fellows (n = 24) is promising. HSI fellows have secured academic roles within universities as well as research and managerial roles within health system organizations. Examples include applied research scientist within public sector organizations, managerial positions within not-for-profit and healthcare delivery organizations and renewed fellowship positions. Although it is in its early days, the HSI Fellowship program appears to be successfully fostering the development of the enriched core competencies in HSPR and preparing fellows for careers in academic and applied health system settings.