Program directors' reflections on national policy change in medical education: insights on decision-making, accreditation, and the CanMEDS framework.

BACKGROUND: Outcomes of national policy change impact all levels of the organizational hierarchy. The medical education literature is sparse on how reflections from program directors (PDs) on past large-scale policy changes can inform future policy initiatives. To fill this gap, we conducted a national survey on PDs' perceptions of, and reflections on, decision-making in medical education, accreditation procedures, and the CanMEDS framework implementation.
METHODS: The survey was distributed to former Canadian specialty medicine PDs (N = 684). Descriptive analysis was performed on quantitative data, thematic analysis was performed on qualitative comments, and comparisons between the quantitative and qualitative findings were performed to identify areas of convergence and/or divergence.
RESULTS: A total of 265 (38.7%) former PDs participated. Quantitative analysis revealed that 52.8% of respondents did not feel involved in decision-making regarding policy changes, 45.1% of respondents did not feel prepared to assess the CanMEDS Roles, and PDs were divided on the reasonableness of accreditation documentation. Qualitative analysis produced four themes: communication, resources, expectations of outcomes, and buy-in. Nine sub-themes were also identified. A high level of convergence was identified across the content, with only four areas of divergence identified.
CONCLUSIONS: Our findings have the potential to inform future policy and/or accreditation changes. Without the lens of those charged with overseeing the implementation, policy evaluation and quality improvement will remain uninformed. PDs, therefore, bring unique insights into our understanding of national policy changes, and without the voices of these frontline implementers, the true success of policy change implementation will be hindered.

Introduction

Policy change is an inevitable part of any medical education system. Such changes are often complex and may impact all levels of the organization.[1],[2] Reflections on past policy implementation are critical to understand how to best introduce future changes. These reflections should ideally come from agents across all organizational levels, focusing on the successes of the intended outcomes achieved and the unintended consequences, both positive and negative. National policy changes are common in medical education; for example, the Accreditation Council for Graduate Medical Education’s (ACGME) Outcome Project,[3] the Royal College of Physicians and Surgeons of Canada’s (RCPSC) CanMEDS implementation,[4] and the international movement towards competency-based medical education (CBME).[5] Changes in these educational systems are often followed by reflections on the successes of implementation; however, the evaluation of unintended outcomes are equally important.[6],[7] Some have built upon these perspectives, applying change management strategies from the business literature; however, despite these reflections, future initiatives rarely run as intended.[8] It is therefore timely to consider other perspectives, including those discussed within the business literature, to provide additional lenses on implementation strategy.

The change management literature has highlighted the importance of focusing on all organizational levels for the successful implementation of a new policy.[9],[10] In postgraduate medical education (PGME), organizational levels include: accrediting bodies, specialty committees, deans, chairs, program directors (PDs), faculty, and trainees. Each level has unique hierarchies, pathways of communication, and perceptions which impact policy change. Furthermore, each level is included within different stages of national policy development and thus yields unique perspectives. The current literature describes five stages of policy development: agenda setting, policy formation, adoption/decision-making, implementation, and evaluation.[11] Complexity science[12] argues that these stages may be non-linear and not always discrete in execution. With these assumptions, it can be understood how insights from some stages, such as PDs involved in the implementation stage, may or may not be fed back to stakeholders in earlier stages, such as accrediting bodies involved in the agenda setting or policy formation stages. Lessons from the business and policy literature suggest that, in order to understand the success of a policy change initiative, feedback and perceptions must include insights from those outside of the highest hierarchical levels.[11],[13] Understanding policy change implementation within and across each level will help elucidate the nuanced challenges faced by agents in each level, which will in turn help assess the outcomes of the implementation. This approach may decrease the social distance across levels and thus promote acceptance; however, there is limited literature capturing diverse stakeholders’ perspectives on the successes and barriers of large-scale change initiatives in medical education. Two examples exist that have assessed the impact of the local environment.[10],[14] One examined how local constraints prevent innovations from being transferred from a previous setting and instead cause a negative transformation of the innovation.[14] The second examined how to include manipulation checks as a model to better understand the learning conditions when an implementation is synthesized across diverse settings.[10]

Previous reflections on policy change implementation in medical education have primarily focused on two levels: accrediting bodies and local constraints. In PGME, the frontline implementers of national policy change are PDs. PDs therefore constitute a unique sample of organizational stakeholders whose perceptions represent a valuable and critical lens to the success of the policy implementation stage. This lens will help identify strategies for implementation, which may improve future changes, such as the international movement towards CBME.

The purpose of this study was to reflect on one large-scale policy change implementation: the introduction of the CanMEDS framework into Canadian PGME programs and its associated changes to accreditation standards.[6] Similar to the ACGME’s core competencies, the CanMEDS framework, implemented in 1997, is an outcomes-based CBME framework that requires residents to demonstrate mastery of specific competencies before progressing to independent practice.[3],[4],[6],[15] While previous reviews of implementation have been completed, this previous research has largely focused on specialty- or context-specific issues and not the broader issues of national policy change implementation.[6],[16],[17] To fill this knowledge gap, our cross-sectional study included a retrospective review of the successes and barriers of a large-scale policy change from the perspective of a national body of former PDs across all Canadian medical specialty programs. Our methodological approach was guided by the following research questions:

What are PDs’ perceptions of the implementation of the CanMEDS framework across Canadian residency training programs and its associated effects on accreditation practices?

What are PDs’ reflections on their level of involvement in decision-making regarding the implementation of the CanMEDS framework?

Findings from the current analyses will help inform future initiatives and policy changes, including the identification of key areas of focus to improve implementation across diverse settings.[18]

Methods

This research sought to investigate PDs’ experiences and overall satisfaction with their role. The Hamilton Integrated Research Ethics Board reviewed the scope of this research and granted REB exemption.

The authors developed an anonymous 70-question online survey. Survey development was guided by the current literature and informed by content experts in medical education. The questions were categorized to capture content in three stages of the PD role: prior to entering the PD role, during the PD role, and after leaving the PD role. The survey included quantitative (Likert scale) and qualitative (open textbox, no character limit) questions addressing PDs’ needs and experiences. Consistent with recent recommendations concerning the use of open-ended survey data in medical education research, we “conceptualize[d our] data and their analysis a priori.”[19] The use of both qualitative and quantitative methods was required to gain a complete understanding of PDs’ perceptions of the three stages of a PD.[20],[21] The survey was reviewed by experts and piloted before distribution to all former Canadian specialty medicine PDs (N = 684) who were within five years of leaving their PD role. The survey was open for two months commencing in February 2014. PDs in family medicine were not included in this study because these PDs are affiliated with a unique accrediting college separate from the RCPSC.

Due to the scope of the larger survey, this paper focuses solely on the middle stage of a PD during which the PD was active in their role. The relevant questions related specifically to the implementation of the CanMEDS framework and its associated changes to accreditation standards (five qualitative open-ended/free-text questions [see Appendix B] and 15 quantitative Likert scale questions). The other questions included within the larger survey addressed responsibilities before entering and after leaving the PD role. Analysis of these questions will be included within subsequent publications to provide a comprehensive picture of the PD role across all of its stages.

Research design and data analysis

A cross-sectional survey design was utilized and involved the concurrent collection and separate analysis of quantitative and qualitative data. Findings from each analysis were subsequently compared to identify areas of convergence and/or divergence.[21] The quantitative questions were used to measure satisfaction and agreement, the qualitative questions explored the PDs’ views in more personal depth, and the comparison of results between the quantitative and qualitative data facilitated a richer understanding of large-scale policy implementation.[22] The collection of both quantitative and qualitative data in the current research was necessary to fully investigate PDs’ perceptions and personal experiences on accreditation, decision-making, and national policy change.

All phases of analysis were completed independently to prevent bias. Descriptive statistical analysis was performed on the quantitative data after qualitative analysis was completed.

Following thematic analysis procedures, two authors read all narrative comments to become familiar with the data while also completing reflexive journaling to explore and document personal assumptions and biases.[20],[22],[23] This process of reflexivity was important for promoting credibility and ensuring that final thematic interpretations were an authentic representation of the participants’ reported perceptions and experiences, without influence from the researchers’ personal assumptions or biases.[24] Two authors independently conducted first-level coding of the narrative comments using a data-driven coding procedure to identify initial codes.[20],[25] They then performed second-level coding independently to organize their codes into potential themes before comparing their identified themes and refining them through discussion.[26] The use of a thematic analysis protocol ensured internal validity (i.e., credibility) and, by extension, trustworthiness of the methodological design, data, and the interpretation of findings.[27],[28] Final themes were reviewed by all authors to establish a consensus opinion that the thematic definitions accurately represented the “essence” of each theme (i.e., to promote confirmability and authenticity of the findings).[20],[27],[28] The process of analyzing the qualitative data was facilitated through the use of QSR NVivo (https://www.qsrinternational.com/nvivo/home).

The final step in the analysis was the triangulation and integration of the quantitative and qualitative results. Two authors explicitly sought areas of convergence and divergence across the findings.[22]

Results

A total of 265 (38.7%) invited PDs completed the survey. Respondents represented a diverse sample across all RCPSC-recognized specialties/subspecialties, with pediatrics (24.2%), internal medicine (16.2%), and surgery (15.1%) representing the most common specialties/subspecialties. The respondents included PDs from institutions across Canada. Table 1 summarizes the participants’ demographics.

Table 1

Basic and professional demographics.

Characteristic	Value
Age, no. (%)
25-3435-4445-5455-65>65	1 (0.4)55 (20.8)124 (46.8)67 (25.3)14 (5.3)
Gender, no. (%)
MaleFemale	149 (56.2)111 (41.9)
Duration (years) as PD
Mean (SD)MedianRange	6.2 (3.5)6.00.5-21
Duration (years) between faculty appointment and PD appointment, no. (%)
0-56-1011-15>15	126 (49.4)69 (27.1)28 (11.0)25 (9.8)
Institution, no. (%)
Dalhousie UniversityMcGill UniversityMcMaster UniversityMemorial UniversityNorthern Ontario School ofMedicineQueen’s UniversityUniversité de SherbrookeUniversity of AlbertaUniversity of British ColumbiaUniversity of CalgaryUniversité LavalUniversity of ManitobaUniversité de MontréalUniversity of OttawaUniversity of SaskatchewanUniversity of TorontoWestern University	14 (5.6)15 (6.0)21 (8.4)6 (2.4)2 (0.8)10 (4.0)10 (4.0)15 (6.0)24 (9.6)25 (10.0)12 (4.8)21 (8.4)12 (4.8)17 (6.8)9 (3.6)25 (10.0)13 (5.2)

Percentages are given relative to each demographic category. Non-responses are not represented in this table.

Qualitative analysis

Thematic analysis of respondents’ narrative comments produced four themes and nine sub-themes.

Theme 1: Communication regarding the details of the policy change

Communication was identified as a critical component of large-scale policy implementation across all organizational levels. PDs commented on the necessity for bidirectional communication between themselves and agents higher in the organization. Complications arose when the “expected trickle down of communication did not happen as expected and resulted in some lack of perceived transparency.” When the rationale, objectives, and guidelines were not explicitly shared from higher levels, the PD was left “feel[ing] quite alone working on this initiative,” unsure how to navigate through unclear expectations. This lack of transparency was alleviated when the PD “could relay information back and forth” between the program level and higher organizational levels.

We also identified three sub-themes of communication.

PDs were concerned with the lack of direct communication with and from the RCPSC. One PD shared that:

Changes to accreditation standards [were] not communicated to PDs… during my time as PD, resident safety was introduced, but unless one were to visit the [RCPSC] on a regular basis and read the latest ‘blue book,’ one would not be aware of such changes.

Without sufficient communication from the top of the hierarchy, PDs felt uninformed with respect to policy change expectations. Many PDs likewise expressed a feeling that information shared from the top occurred through a “very top down” process, with feedback from PDs “never [being] responded to and never seem[ing] to have any impact on policy.”

Each RCPSC-recognized specialty/subspecialty has an associated specialty committee that is responsible for advising the RCPSC on “specialty-specific content issues.”[29] Communication with the specialty committee was identified by PDs to be the preferred conduit to the program level by the RCPSC. PDs supported that the “specialty committee was fantastic… [and] was the main source of information” to them, while also noting that they “were more frequently informed of decisions [by] the specialty committee than truly involved in decision-making.” While it appears that PDs received meaningful information from their specialty committee, there existed a desire for PDs to be more involved in decisions that affected them at the specialty and program levels.

PDs discussed the need for clear and transparent communication within their institutional hierarchy. This included communication with other professionals within their specialty and department. PDs noted that it “took a lot of collaboration and creativity working with colleagues to develop materials at [their] center.” In the absence of such internal communication, “the largest support as PD came from other PDs in the same specialty across the country, more so than PDs of other programs within the same university.” Hence, PDs within the same institution generally expressed a need to support one another to help advance their progress.

When communication at the expected formal levels did not yield the desired transparency, alternatives were developed through grassroots communication. These included communication with former PDs and PDs from other specialties to facilitate information sharing “[to avoid] reinventing the wheel.”

Theme 2: Teaching and evaluation resources

Resources are a critical factor for large-scale policy change. With the CanMEDS framework implementation, PDs acknowledged that the “CanMEDS Roles are important,” but noted that they received “no help,” resulting in the CanMEDS Roles being “poorly taught [and] poorly assessed.” PDs similarly endorsed the necessity for accreditation, but they expressed that “[they] needed and continue to need real life guidance” which they did not always feel was available. The provision of appropriate and sufficient resources was a strong facilitator driving successful accreditation and CanMEDS framework implementation. This theme was present in sufficient depth to yield two sub-themes.

Successful implementation of accreditation standards and the CanMEDS framework required the provision of sufficient resources. Potential barriers to achieving resources occurred at the RCPSC, PGME office, and program levels. PDs commented that “there was no time or support to 'implement' the [CanMEDS] Roles into the curriculum” at the program level, and “that accreditation [is] becoming focused on increasingly minute details that have little influence on the learning or lives of residents, but require great demands on programs.”

PDs expressed concern that expectations of implementation exceeded their capabilities and resources. When challenges arose, PDs commented that they received “no help from PGME with respect to produc[ing assessment] tools,” and that there existed a general “lack of concrete assistance from [the RCPSC] in terms of helping PDs with the process” of accreditation. These quotes highlight PDs’ needs to receive more resources from higher levels to meet the expectations of implementation.

A shared reflection identified across the themes of resources and communication was a lack of influence. PDs expressed that “there should be more transparency in what the RCPSC specialty committee does and what role the PDs have in their decision-making. When it is good, it is good. When it is bad, it is dysfunctional.” Although PDs identified potential solutions to this concern, the “heavy oversight” from agents higher in the educational hierarchy “slow[ed] down every decision and impose[d] unjustified restrictions [on PDs] to implement changes and adapt program requirements.”

Tension existed between a desire for clear, standardized accreditation expectations and allowance for programs and specialties to adapt expectations to meet local needs. This tension was identified through divergent perspectives of wanting the RCPSC to provide and implement resources in a standard process across institutions and programs versus those who expressed the need for the implementation to allow freedom of adaptation to suit local needs. Many PDs desired resources to be “adapted uniformly across the country rather than requiring each university to re-invent the wheel.” Other PDs felt they could adapt their existing activities into “multiple CanMEDS categories” that fit their local needs.

The desire for standards also existed in the area of accreditation where many PDs commented that “the accreditation process does not permit sufficient flexibility for local issues to be accommodated.” There was also an identified “lack of consistency in accreditors” which led to “objectives that were satisfactory in one place when borrowed not [being] approved.”

Having a set of standardized objectives and resources would help each program meet implementation expectations. PDs offered that “involv[ing] the specialty committee and all PDs together to develop national objectives” for a program before the implementation of an organizational change would promote consistency across levels of the organization and would facilitate the understanding and achievement of expectations; however, this inflexibility runs the risk of ignoring the differences across programs, with one PD noting that “large businesses and small businesses are treated very differently in Canadian law, [so] why are small programs and large programs held to the same [standards]?”

When resources were sparse or unavailable, PDs often relied on connections with other PDs to share objectives and teaching/assessment resources to facilitate implementation and accreditation in their own program. These connections helped PDs implement the CanMEDS framework and navigate the accreditation process without any available standardized resources provided from agents higher in the organizational hierarchy.

Theme 3: Expectations that the policy change will yield meaningful outcomes

Expectations of outcomes was identified by PDs to represent a desire to have clear evidence to support how a large-scale policy change would improve learner and patient outcomes. PDs expressed concern with ensuring that core values were being addressed after the policy change and that rationale for expected action was evident, with one PD stating, “I think we forgot the reason why accreditation is important: to improve [the] quality of doctors.” When those who are expected to implement change on the ground remain unclear about outcome expectations, support may be hard to gather. PDs discussed the impactful nature of clear expectations across two sub-themes.

PDs expressed concern with inconsistencies in the accreditation process: “everyone is doing it differently, no clarity in the process… how do you do it… can we do it differently in different areas to get the same results?” There were similar concerns about the operationalization of the CanMEDs Roles, with one PD identifying “this [as] the greatest problem; although required to assess the CanMEDS Roles, there are no valid assessment tools for this… the assessments turn out to be irrelevant anyway since the results of evaluations, positive or negative, have no impact on the resident or the program.”

Issues of operationalization were also identified regarding the CanMEDS framework implementation. PDs had specific concerns regarding operationalizing the framework within their specialty. One PD reflected on the “values of each [Role]: what do they mean in my specialty?” This was echoed by another PD who wrote that “the CanMEDS Roles are not specific to each program. Making them ‘fit’ the program is the challenge.” PDs also struggled with conceptualizing expectations during the initial policy change: “when CanMEDS [was] rolled out... it appeared that equal ‘weight’ was being given to each of the Roles, a ridiculous concept… fortunately, over time, the concept of the Roles having ‘relative weight’” was realized. As they turned to implementation, PDs expressed difficulty translating traditional teaching methods to the Intrinsic Roles, reporting that it was “difficult to teach professionalism, collaboration, and collegiality in a didactic format.” Similarly, “some Roles don't lend themselves well to traditional models of assessment,” with one PD noting that they “didn't get a lot of feedback from staff about the non-medical expert realms.” These quotes suggest that PDs require additional support to operationalize the changes within their own context.

A second sub-theme was the need for evidence regarding large-scale changes in practice and/or expectations. Some PDs questioned the accreditation process as a driver for change, noting that “many of the changes currently being enforced by the [RCPSC] are not going to necessarily produce better trainees.” One PD reported not seeing a link between the policy change and the expected outcomes, writing that, “despite all the CanMEDS teaching, [they] have seen an obvious rise in unprofessional behaviour among residents over [their] seven years as PD.”

Theme 4: Buy-in to the new policy from others

PDs highlighted key components of buy-in to make initial and sustainable changes, especially when obstacles arose. One PD described a feeling of isolation, stating that “other faculty did not understand clearly the CanMEDS Roles and did not help at all. Nobody in my group seemed interested and helped me.” Two sub-themes were also identified to expand this theme.

PDs experienced challenges of buy-in from trainees, faculty, and agents in their local hospital. One PD noted that “all the CanMEDS Role teaching [was done] at the expense of surgical teaching. It was felt to be a complete waste of time by both faculty and residents.” This highlighted the reluctance of some to value and incorporate the Intrinsic CanMEDS Roles into clinical practice. Some PDs identified that a “fundamental issue was convincing the faculty that… they actually were [already] teaching and evaluat[ing] a lot of different competencies on a regular basis,” attempting to dispel notions of a paradigm shift to their regular duties. While pushback was common, PDs recognized that “the regular occurrence of external accreditation [was] a powerful driver for hospitals and health authorities to make necessary changes,” and to inspire buy-in from others at the program level.

This sub-theme represented how accreditation occurred independently of local context. PDs expressed concern that accreditation “is very dependent on the visitor’s capacity to understand and interpret local issues,” including the cultural nuances of the specialty. One PD described how their “program was assessed by a psychiatrist and a pediatrician”, and the PD did not “think [these accreditors] could understand or appreciate the unique stressors of [a surgical specialty] or how demanding it is on every level.” A similar issue arose when program size was not considered. Many PDs from smaller programs highlighted that a “one- to two-year program [was] treated like [a] five-year program.” Finally, PDs often reported that authentic review of their program was not possible when the accreditation process allowed for the overamplification of a single dissenting voice. For instance, one PD commented that, “under the current system, all it takes is one unhappy or emotionally unstable trainee to set up a cascade of issues.” Another PD similarly described that “residents use the [accreditation] process as an opportunity to control the program, and bring up complaints that they see, rightly or wrongly, as unfair.”

“In theory, accreditation, both internal and external, should check on a program to evaluate the learning environment. In [some PDs’] experience, it is a white wash and window dressing.” PDs seemed to express that accreditation does not consider local context (i.e., rural vs. urban locations), nor does it adapt the process according to program size. “Accreditation gives a snapshot view of a program at a given time,” and a fully authentic program evaluation may not be possible. Once recommendations are received, “there is no room for PDs to request assistance in effecting change.” PDs desired more authentic and program-specific accreditation processes in order to holistically promote program quality and ultimately enhance resident training.

Quantitative analysis

Descriptive quantitative analysis (summarized in Table 2) was performed across three concepts: decision-making, the accreditation process, and the CanMEDS framework implementation. Results were generally positive, with most respondents reporting an understanding of CanMEDS teaching and assessment requirements; however, most respondents reported that they were not informed of usable CanMEDS assessment tools. Nearly all questions probing the accreditation process yielded positive results, with most respondents agreeing that accreditation is a fair and important process to promote quality improvement and effect change. While most respondents agreed that they were informed of policy decisions made by the RCPSC and their specialty committee, 52.8% did not feel involved in decision-making at the RCPSC and 46.8% did not feel involved in policy decisions affecting residency education. Overall, only 51.1% of respondents somewhat agreed that the CanMEDS framework implementation has improved residency training.

Table 2

Summary of the quantitative results.

Question	Agreement, no. (%)
	No		Yes
Do you feel you were adequately informed about the requirements for teaching the CanMEDS Roles?	54 (24.2)		169 (75.8)
Do you feel you were adequately informed about the requirements for assessing the CanMEDS Roles?	100 (45.1)		122 (54.9)
Do you feel you were adequately informed about usable tools and methods for teaching the CanMEDS Intrinsic Roles?	141 (64.7)		77 (35.3)
Do you feel you were adequately informed about usable tools and methods for assessing the CanMEDS Intrinsic Roles?	162 (75.0)		54 (25.0)
Question	Agreement, no. (%)
	Not At All to A Little	Somewhat		A Lot to Extremely
How much has the formal implementation of teaching and assessing the CanMEDS Roles in your program affected the quality of residency training?	68 (31.1)	112 (51.1)		39 (17.8)
Statement	Agreement, no. (%)
	Disagree	Neutral		Agree
The accreditation process is a fair mechanism for evaluating program quality.	40 (17.6)	50 (22.0)		137 (60.4)
Accreditation is an important way to ensure ongoing quality improvement.	14 (6.1)	15 (6.5)		201 (87.4)
Documentation for the accreditation process is reasonable.	80 (34.9)	62 (27.1)		87 (38.0)
Accreditation was a driver for the implementation of the CanMEDS Roles.	15 (6.6)	29 (12.7)		184 (80.7)
The accreditation process sets reasonable standards for residency programs.	28 (12.2)	49 (21.4)		152 (66.4)
I was informed about policy changes made by the RCPSC.	17 (7.5)	40 (17.6)		170 (74.9)
I was involved in decision-making regarding policy changes made by the RCPSC.	115 (52.8)	46 (21.1)		57 (26.1)
I had a voice on policy affecting residency education.	102 (46.8)	51 (23.4)		65 (29.8)
I was informed about decisions made by my specialty committee.	18 (7.9)	27 (11.9)		181 (80.1)
I was involved in decision-making at the level of the specialty committee.	49 (21.9)	44 (19.6)		131 (58.5)

Percentages are given relative to each question/statement. Non-responses are not included in these analyses.

Areas of convergence and divergence between the qualitative and quantitative results

Analysis revealed a high level of convergence between the qualitative and quantitative findings (see Appendix A, Table 3). For example, PDs’ agreement with the statement ‘I was informed about policy changes made by the RCPSC’ yielded a better understanding of the emotionality experienced by those PDs who felt uninformed about the policy change as a result of poor communication; however, comments like “[the policy change process] is very top-down. This is what the decision is and now you have to find a way to implement it into your program” helped reconcile the quantitative finding. It appears, then, that although the majority of PDs reported that they were informed of the policy change, the qualitative comments revealed dissatisfaction with the way in which this information was conveyed.

Table 3

Summary of the areas of convergence and/or divergence between the qualitative and quantitative results.

Question/Statement	Most Endorsed Option		Qualitative Quote	Converge or Diverge
Do you feel you were adequately informed about the requirements for teaching the CanMEDS Roles?	75.8% Agree		“CanMEDS Roles are important but NO HELP = poorly taught, poorly assessed”	Divergent
Do you feel you were adequately informed about the requirements for assessing the CanMEDS Roles?	54.9% Agree		“CanMEDS Roles are important but NO HELP = poorly taught, poorly assessed”	Convergent
Do you feel you were adequately informed about usable tools and methods for teaching the CanMEDS Intrinsic Roles?	64.7% Disagree		“More formal teaching time spent on Intrinsic competencies. Residents aren't interested; they would rather attend seminars on ‘medical’ topics”	Convergent
Do you feel you were adequately informed about usable tools and methods for assessing the CanMEDS Intrinsic Roles?	75.0% Disagree		“The non-medical expert Roles are notoriously difficult to discretely identify and quantify.”	Convergent
How much has the formal implementation of teaching and assessing the CanMEDS Roles in your program affected the quality of residency training?	51.1% Somewhat Agree		“[D]espite all the CanMEDS teaching, I have seen an obvious rise in unprofessional behaviour among residents over my 7 years as PD”	Convergent
The accreditation process is a fair mechanism for evaluating program quality.	60.4% Agree		“There is a significant risk that [the] site visit will lead to poor decisions based on a minority of highly vocally negative residents and/or a surveyor that has biases that are undeclared”	Divergent
Accreditation is an important way to ensure ongoing quality improvement.	87.4% Agree		“Accreditation was the only time I could orchestrate changes at the individual hospital level”	Convergent
Documentation for the accreditation process is reasonable.	38.0% Agree		“The amount of documentation required is unreasonable for the presurvey questionnaires”	Convergent
Accreditation was a driver for the implementation of the CanMEDS Roles.	80.7% Agree		“The major action was just to take the Roles and force them into what we were already doing. This was just to satisfy Royal College requirements. There was no time or support to 'implement' the Roles into the curriculum”	Divergent
The accreditation process sets reasonable standards for residency programs.	66.4% Agree		“Standards may be variably applicable to different programs, and it may be difficult to nuance this during an accreditation”	Divergent
I was informed about policy changes made by the RCPSC.	74.9% Agree		“[D]uring my time as PD, resident safety was introduced, but unless one were to visit the RC on a regular basis and read the latest ‘blue book’, one would not be aware of such changes. Changes need to be communicated better to all those who need to know, such as PDs”	Convergent
I was involved in decision-making regarding policy changes made by the RCPSC.	52.8% Disagree		“Ya, maybe things have changed over the last 4 years, but man, things were thrust upon me rather than incorporated with me. But hey, I was just a PD of a small subspecialty program”	Convergent
I had a voice on policy affecting residency education.	48.8% Disagree		“[T]he move towards CBE was dictated to the PDs and the drivers of the CBE movement definitely had a ‘we'll leave you behind if you do not follow’ undertone in most conversations”	Convergent
I was informed about decisions made by my specialty committee.	80.1% Agree		“As chair of the PD's committee at the specialty committee, I felt that I was much better informed about what was going on at the specialty committee level and could relay information back and forth between the two groups”	Convergent
I was involved in decision-making at the level of the specialty committee.	58.5% Agree	“[T]here was a disconnect between how much the actual PDs were informed and engaged depending on the actual Chair of RCPSC Specialty committee”		Convergent

Analysis also identified four areas of divergence (see Appendix A, Table 3). For example, to the question, ‘Do you feel you were adequately informed about the requirements for teaching the CanMEDS Roles?’, the majority of PDs agreed; however, a different picture was gained from the qualitative comments. It is possible that PDs recognized the general requirement that they were expected to teach the CanMEDS Roles, but the details of the specific expectations surrounding CanMEDS teaching were not fully understood.

Discussion

This cross-sectional study provided insights into the nature and process of large-scale policy change in medical education from the perspective of those charged with implementation, especially as they differ from those agents involved in earlier stages of the policy development.[11] Reflections of a resource-intensive, large-scale educational transition resulted in strong emotions, which were expressed in the qualitative comments. The qualitative data provided a framing for the quantitative data to focus discussion into three concept areas: decision-making, the accreditation process, and the CanMEDS framework implementation. There was a high degree of convergence between the qualitative and quantitative results, allowing for a better understanding of the phenomena.

The quantitative analysis demonstrated that most PDs understood the policy itself, as well as the expected implementation requirements; however, the qualitative analysis revealed that many respondents were unable to apply the requirements to their own context. Indeed, nearly half of the respondents did not believe that residency training improved as a result of the CanMEDS framework implementation. This low endorsement rate was surprising given that implementation began in the late 1990s. Perhaps this finding reflects a lack of appreciation for the specialty-specific nuances across residency training programs.[30] In fact, many respondents reported dissatisfaction with the applicability of all CanMEDS Roles to their specialty, highlighting discordance among some PDs regarding this “one size fits all” approach.

This framing of needs for the unique application of standards and policies to each specialty signifies the importance of program-level considerations in national policy initiatives, which has thus far been missing from quality improvement evaluations of policy in medical education. The observed tension between standardization and adaptability represented how most PDs wanted policy implementers to have specific standards they could understand and strive towards. Indeed, a recurrent message regarding implementation observed in the qualitative data was the lack of a perceived “fit” of the CanMEDS framework across residency training programs. Understanding what “fit for the program” means, and how this “fit” is or is not supported by evidence, seems to be a driving force behind PDs’ expectations of how implementation and/or accreditation would benefit their program.

PDs wanted boutique-level applications where the cultural nuances of their specialty and practice setting were represented in the directives. Additionally, with these contextual adaptions, it is critical to understand how and why implementation differed. Specifically, evaluators would need to know if the different implementation events truly were comparable or if they resulted in transformations.[14] The themes resulting from the qualitative analysis could be used to inform evaluation using the lens of implementation science to understand what aspects of context and setting were perceived to have impacted implementation.

While most PDs highlighted the necessity of the accreditation process, many also noted areas for improvement across organizational levels and stages of implementation. Though accreditation can drive change, PDs did not feel they had a voice when change occurred, nor that the process was always responsive to the unique qualities of their local context.

This study highlighted the difficulties of communication, resources, expectations of outcomes, and buy-in across the organizational hierarchy, as well as between those who plan and implement policy. Perceptions of PDs included a feeling of powerlessness in being involved in, or informed of, decisions. Rather than the expected trickle down of information, PDs described novel approaches to receiving information from the accrediting bodies and higher level committees. The implementation of the CanMEDS framework is only one example of the rapidly changing field of medical education. In an era where change is quite common, this paper provides a unique understanding of barriers, novel pathways, and unexpected outcomes from the perspective of frontline agents.

Conclusion

The findings from this study can inform future educational transitions internationally as the field of medical education expands to meet changing societal needs. This paper comes at a crucial time when many PDs and other frontline faculty are faced with implementing new policy. To understand what leads to success or poor implementation, policy evaluators must engage agents throughout the organizational hierarchy, especially those tasked with implementation and adapting policy to the local context, in order to inform successful outcomes. In this way, collaboration amongst the levels of the organization will promote the effective flow of information to increase confidence and participation in implementation processes, accreditation, and decision-making. Lessons learned from education leaders—including frustration, identified issues with implementation, and challenges with program accreditation—offer opportunities to inform better participation at all organizational levels. Without the lens of those charged with implementation, quality improvement and/or policy evaluation will remain uninformed. From locally shared work-arounds to national resources, the perspectives gained from PDs’ engagement with the policy change will support successful shifts in future educational transitions across all levels of the organizational hierarchy.