Program director and resident perspectives of a competency-based medical education anesthesia residency program in Canada: a needs assessment.

PURPOSE: In July 2015, the University of Ottawa introduced a competency-based medical education (CBME) postgraduate program for anesthesia. Prior to program implementation, this study aimed to identify Canadian anesthesiology program directors perceptions of CBME and residents' opinion on how the program should be designed and perceived consequences of CBME.
METHODS: This two-phase, qualitative study included semi-structured interviews with Canadian anesthesia program directors (Phase I) and a focus group interview with residents enrolled in the University of Ottawa time-based anesthesia program (Phase II). Both phases sought to gauge participant's perceptions of CBME. Interviews were recorded, transcribed verbatim and thematically analyzed.
RESULTS: Data was combined to protect anonymity of the six participants (three program directors and three residents). Participants spoke about the perceived advantages of CBME, the need to establish definitions, and challenges to a CBME program highlighting logistical factors, implications for trainees and the role assessment plays in CBME.
CONCLUSION: These findings will inform CBME implementation strategies in anesthesia programs across the country, and may assist other residency programs in the design of their programs. Furthermore, our findings may help identify potential challenges and issues that other postgraduate specialties may face as they transition to a CBME model.

Introduction

Current postgraduate medical education has been criticized for being largely unchanged from the time-based apprentice model adopted over a century ago [1]. The public is demanding more accountability from the medical profession [2] and the current model has been accused of producing physicians who are ill-equipped to deal with an increasingly complex healthcare system [3]. As a result, there is a growing global trend towards alternative models of education such as competency-based models of medical education, also called competency-based medical education (CBME) [4,5,6,7]. CBME programs identify the competencies that are required by an independently practicing healthcare professional and structure a training program to develop, assess, and maintain those competencies [5].

In 2011, the Future of Medical Education in Canada Postgraduate Project identified the need to shift from the traditional apprenticeship model to CBME [8]. Pilot data from the University of Toronto Orthopedics program suggests that many residents can finish training faster than the mandatory duration of traditional training, which has the potential to optimize the costs of residency training if these findings can be scaled up across specialties and countries [9].

In July 2015 a CBME postgraduate program for anesthesia residents will be introduced at the University of Ottawa in Canada. The Royal College of Physicians and Surgeons of Canada named the new CBME programs “Competency by Design.” There is, to our knowledge, currently no other postgraduate CBME program in anesthesia in Canada and only a few in the world that use the model to reduce the time required for train a specialist. Prior to program implementation, this study aimed to identify (1) Canadian anesthesiology program directors perceptions of CBME and (2) residents’ opinion on how the program should be designed and perceived challenges and consequences of CBME.

Subjects and methods

1. Ethics approval

The Ottawa Health Science Network-Research Ethics Board (OHSN-REB #20130282-01H) has approved the study.

2. Study population

All 17 directors of postgraduate anesthesia programs across Canada and current anesthesia residents (n=50) from the University of Ottawa were invited to participate by email. We recruited program directors through a contact at the Association of Canadian University Departments of Anesthesia (ACUDA), who forwarded recruitment information to members and sent out repeated follow-up requests to potential participants on our behalf. Residents were recruited via email sent by the administrative staff with the university’s anesthesia department. Three emails were sent and all interested participants were followed up with until they provided a response. Participants provided written consent and gave permission to have their data published.

3. Study design

This two-phase, qualitative interview study was conducted prior to CBME program implementation. In Phase I, all 17 Canadian anesthesia program directors were invited to participate through the ACUDA. A trained research assistant conducted semi-structured individual interviews in person or over the phone with participants. Recorded interviews were transcribed verbatim and analyzed concurrently in an iterative process using qualitative methods. Phase I data was used to develop a question guide for Phase II interviews.

Phase II was conducted after Phase I was completed. In Phase II, residents enrolled in the University of Ottawa anesthesia program were invited to participate in a focus group interview. The focus group aimed to solicit resident’ opinions about CBME as a model for residency training in anesthesia and their perceived challenges and consequences of CBME. The focus group interview was transcribed verbatim.

The interviews and focus group followed the developed interview guides (Appendixes 1, 2). All transcripts (individual and focus group) were imported into NVivo 10 (QSR International, Melbourne, Australia) and thematically analyzed using the constant comparison method to synthesize data [10,11]. Employing the first three steps of the purposeful constant comparison approach [12] comparisons occurred within a single interview, between interviews within the same group and between different groups. Each interview was reviewed in its entirety and open coded to label each passage with an adequate code. After reviewing each interview, interviews within the same grouping (i.e., program directors) were compared and the coding strategies were combine to form an overall coding strategy. Since there was only one resident focus group, this step was not completed for this group. The combine coding strategy from the program director group was used to inform the development of the question guide for the resident focus group. After all interviews were complete and analyzed by group, interviews from the two different groups (i.e., program directors and residents) were compared to capture the overall similar and different perceptions of a CBME program.

Results

For Phase I, following the recruitment methods described in the previous section, three out of the 17 program directors were engaged. For Phase II, a question guide based on Phase I data guided one focus group interview with three current anesthesia residents (current traditional 5-year time-based program). Findings from all interviews have been combined to protect anonymity. Our goal was not to compare program directors and residents perspectives but rather to present a general overview of the perceptions and opinions of CBME prior to CBME program implementation.

The emerging themes from participants focused on the need for definitions, the advantages of CBME, and challenges (i.e., program administration, implications for trainees, and assessment) in a CBME program (Table 1).

Table 1.

Sample of Program Director and Resident Perceptions to Implementing a CBME Program

Theme	Quotation
Definitions	"I think the idea [of CBME] is to teach people in order to meet a predefined, specific standard. Whereas now, the predefined, specific standard is 'can you pass the Royal College exam the first time in 5 years.'" (Resident)
	"[...] what is a specialty of anesthesia? And off-service rotations, we all feel that they add the ability for us to be perioperative physicians. We're not just intra, in the Operating Room, providing a general anesthetic. We are so much more than that." (Program director)
Advantages	"You have more direction over your own learning and your own pace." (Resident)
	"I think a great opportunity with competency-based education is this is going to be good for the person who is falling behind and identifying them earlier, giving them some things to strive for, giving them some great feedback. But, what about the outstanding individuals? Is it going to allow them to move quickly to proficiency." (Program director)
Challenges: program administration	"Nothing should stop you from saying a resident has achieved competency in a certain area, even if it's before the block ends. The question is, what do we do with that week or two that is left? How can we best use that?" (Program director)
Challenges: assessment	"It’s too bad we don't get evaluated more often because I think a lot of the softer things we talk about, like communicator, professional, and things like that, we recognize the staff are very poor at evaluating these things. I think our patients could somehow play a role in that. I think that would take some of the responsibility off the staff and you'll have people who are more appropriate to evaluate that actually evaluate it." (Resident)
	"I say, okay, this is a good one for us to do together...And they may [do] this intubation...the little nuances of how to do things safely is there. And they may feel, okay, I've intubated, therefore, I’m competent. Well, not really." (Program director)

CBME: Competency-based medical education.

1. Need for a CBME vocabulary

Before a CBME program is implemented, participants emphasized the need to form common definitions and to create a new vocabulary to guide the program. The word competency was perceived as ambiguous. Discussion of competence led to questions about incompetence or dyscompetence and how these terms are defined. Participants questioned if proficiency and competency meant the same thing; however, the majority of participants defined competency in terms of skill mastery. There were concerns about whether the definition considered overall confidence or simply the ability to perform a skill consistently. Residents spoke to the need for redefined objectives of training and standard training requirements using a competency lens and for an updated set of objectives for subspecialties.

“I think the idea [of CBME] is to teach people in order to meet a predefined, specific standard. Whereas now, the predefined, specific standard is ‘can you pass the Royal College exam the first time in 5 years.’” (Resident)

Participants raised concerns about how to define the scope of anesthesia practice and what training is required to be an anesthesiologist. Some reconciled this professional scope based on location of practice (e.g., rural, city, teaching hospital).

“[...] what is a specialty of anesthesia? And off-service rotations, we all feel that they add the ability for us to be perioperative physicians. We’re not just intra, in the Operating Room, providing a general anesthetic. We are so much more than that.” (Program director)

2. Advantages of CBME

Participants highlighted the advantages of CBME citing: early identification of those who are advanced or require extra assistance; an accelerated program for those who excel and plan for further subspecialty training; flexibility in completing a more personalized program; regular feedback; a sense that trainees would be better prepared to take their Royal College exams; a system that could expedite the accreditation of foreign trained international medical graduates; and the benefits (financial and time) of having a more efficient training system.

“You have more direction over your own learning and your own pace.” (Resident)

Echoing findings from others [6,13], participants also explained that a CBME program would facilitate feedback to the trainees and allow for setting goals and outlining strategies to address weak areas.

“I think a great opportunity with competency-based education is this is going to be good for the person who is falling behind and identifying them earlier, giving them some things to strive for, giving them some great feedback.” (Program director)

3. Challenges of CBME

1) Program administration

Participants discussed potential administrative challenges. These concerns included administrative burden for program coordinators; issues with scheduling rotations; unpredictable changes in program size; curriculum development; the development of personalized learning programs to match different learning speeds; and available resources. There were a number of discussions about how to fill in time if someone achieves competency early in a rotation.

“Nothing should stop you from saying a resident has achieved competency in a certain area, even if it’s before the block ends. The question is what do we do with that week or two that is left? How can we best use that?” (Program director)

Another challenge voiced was the need for administrators to ensure that those with an accelerated program were exposed to an appropriate case mix, which they perceived as required to making a competent physician.

“Now, as they become more experienced and they see more cases, do more cases, do some medicine, do some critical care, do all of this, they get better and better and better at it.” (Program director)

Program directors also raised concern over the need to ensure faculty commitment, buy-in, and participation. This reflects other CBME programs that emphasize faculty engagement and support as critical elements for successful educational reform [6]. With respect to providing and supporting faculty development, there were questions raised as to how faculty will be trained to provide alternative forms of assessment.

2) Implications for trainees

Concerns were raised about the implications for trainees participating in a CBME anesthesia program, such as trainees in the existing system (time-based) may be at a disadvantage; there may be additional pressure on trainees to perform; the intensity of the CBME program is unknown; it is possible that trainees may focus on getting through the program as fast as possible and pass the assessment rather than recognizing competency on the continuum to mastery through lifelong learning; there may be stigma for those who are part of the new program; CBME program participants may be faced with additional administrative responsibilities; there may be implications for future employment with varied numbers of graduating residents each year; and it may take longer to complete the program.

“I think it’s going to create a stigma. And I think, too, there is a big step up in income when you finish and you get to go out and practice. If, I tell you, you need extra time, in a way, I’m costing you money.” (Program director)

While participants cautioned that there could be gaps in experience if people advanced too quickly, while others noted that the time-based system also results in missed opportunity to be exposed to rare cases.

“It’s good and it’s bad. I think that’s the entire purpose of competency is that we will get to see all those cases ...there were some cases that I was supposed to see that I didn’t see. So, I guess there is a mixed...I think time is important. There should probably be a minimum amount of time on a sub-specialty.” (Resident)

Furthermore, residents emphasized the value social support provided by being part of a cohort and questioned if the trade-offs were worth it.

“It just sounds like there is more competition amongst residents. I think that is an issue that should be looked at closely because I think residents are the ones that get you through. Especially for us, it’s so supportive. So, to lose that, to have some competition, and focus on time, and you don’t want to finish later because then when you’re applying for a job you have to say ‘I was in competency and it took me 6 years because I wasn’t competent.’” (Resident)

3) Assessment/evaluation

The implications of a CBME program on assessing competency were discussed by all participants. Questions were raised about the degree of competency required, what is an acceptable standard, minimal standard, and how to determine if someone has mastered a skill or achieved competency.

“I say, okay, this is a good one for us to do together...And they may [do] this intubation...the little nuances of how to do things safely is there. And they may feel, okay, I’ve intubated, therefore, I’m competent. Well, not really.” (Program director)

Participants discussed the need for shared definitions of competency among faculty evaluating trainees. The capacity for faculty to provide rigorous and frequent assessment to ensure that trainees achieve competence before progressing to further stages of training was brought into question. There were concerns expressed about subjectivity and validity of assessment, as well as the relevance of existing tools and checklists. Overall, participants were unclear about the exact delivery and structure of assessment in a CBME context. A large component of the interviews highlighted the grey areas of assessment and participants raised questions about how existing frameworks and tools (i.e., Canadian Medical Education Directions for Specialists [CanMEDS], objective structured clinical examination [OSCE]) will be integrated into this form of curricula and how technical and non-technical skills will be assessed. Furthermore, some participants questioned if the faculty have reached the standard setting for competence they will be assessing.

Frequency of observation, assessment and feedback were discussed. Both program directors and residents noted that the processes for identifying and assisting struggling residents in a CBME program are unclear. One important area of assessment where ambiguity was highlighted was communication skills. In the current model, program directors reported that it was challenging to assess the acquisition and evaluation of communication skills. Residents suggested that more frequent, consistent, personalized feedback from the same staff could improve assessment of trainees. This is supported by the literature, which emphasizes the value of direct observation and workplace based assessment to provide meaningful formative feedback and opportunities for growth [14]. One suggestion to improve feedback was to involve assessment from team members (e.g., nurses and nurse educators) as well as patients.

“It’s too bad we don’t get evaluated more often because I think a lot of the softer things we talk about, like communicator, professional, and things like that, we recognize the staff are very poor at evaluating these things. I think our patients could somehow play a role in that. I think that would take some of the responsibility off the staff and you’ll have people who are more appropriate to evaluate that actually evaluate it.” (Resident)

Discussion

Stakeholder perspectives have revealed potential unintended consequences and concerns relating to the implementation of a CBME program. Echoing the literature, one of the greatest concerns raised by participants was the need for shared definitions of competency and related terms when designing a CBME curriculum [15]. Even though participants highlighted potential challenges for program administration and trainees as well as logistical issues with providing assessment, they also spoke to the merits of a CBME program (i.e., a personalized program to match trainee skill; regular feedback; improved preparation for exams; and a more efficient training system). Thus, based on our data, we hypothesize that many stakeholders (program directors and residents) will recognize the value of a CBME program and support the implementation of CBME for anesthesia residency, but consider CBME to be a nuanced topic with many perceived advantages and challenges. If we extrapolate further form our data, we may hypothesize that by addressing the perceived barriers and reinforcing the perceived strengths identified by stakeholders, program administrators and policy makers will facilitate the successful implementation of a CBME curriculum for anesthesia residency.

For a program to be successful, it is imperative that all parties involved work within a common framework that has shared definitions based on up-to-date evidence. Although our findings reinforce existing knowledge about the perceived advantages and challenges of CBME, these findings suggest that new developments in CBME are not being effectively transmitted to stakeholders. For example, participants and previous research identified institutional difficulties in organization and evaluation of CBME, as well as the challenges with breaking the complex construct of the competent practitioner into distinct competencies [15]; however, there have been a number of advances in these areas [16,17,18]. In particular, it should be noted that concerns about assessment and the best way to approach competency have been largely improved by the development of frameworks such as CanMEDs [7], and by breaking competencies down into entrustable professional activities (EPAs) [4,17,18]. EPAs are essentially “units of competencies in context.” As trainees progress throughout their careers, they have to demonstrate competency in all EPAs via completion of a series of modules with several milestones in each of them [19]. The use of EPAs and milestones has been proposed as the way forward to help address issues with assessment and curricula development in CBME programming [16]. The anesthesia program directors and residents included in this study did not systematically convey these concepts suggesting that it may be necessary to implement additional strategies to translate updated information about CBME programming to help curb concern among stakeholders.

Several hypotheses may explain why study participants may be unaware of the developments in CBME. It is possible that there the literature on the advanced stages of planning and implementing CBME programs still needs more development or that literature alone may not be the best approach to disseminate evidence to stakeholders. Alternative dissemination strategies such as workshops, rounds, and podcasting may contribute to fill the gap. Therefore, we suggest that it is imperative that programs engaging in CBME share their experiences—both positive and negative—using multiple formats of dissemination, so that the entire community can learn from it. This study partially contributes to addressing this gap. These findings will inform CBME implementation strategies in anesthesia programs across the country, and may assist other residency programs in the design of their programs. Furthermore, our findings may help identify potential challenges and issues that other postgraduate specialties may face as they transition to a CBME model.

The opinion of program directors nationally is key to the acceptance and generalized of the anesthesia CBME program at the national level. Unfortunately, we had a low response rate with 18% of program directors and 6% of residents participating. Our limited success in recruiting program directors may suggest a lack of interest in implementing the new CBME program. If true, there could be a number of challenges in the implementation of the new Royal College CBME curricula. Although low recruitment prevented data saturation, qualitative research is generally viewed as hypothesis-generating research (as opposed to hypothesis-driven), and results are not expected to be generalizable. Our findings provide a contextualized understanding of stakeholder experience.

In conclusion, as more postgraduate specialties programs switch to a CBME model across Canada and internationally [20], our findings may help guide them on the potential challenges and issues that they may face.