Creating a National Competency-Based Curriculum for Orthopaedic Surgery Residency: The Canadian Experience.

The Royal College of Physicians and Surgeons of Canada (RC) is responsible for setting standards of training and certification in all specialty disciplines of medicine and surgery. In 2011, the RC planned to launch a competency-based medical education (CBME) program named “Competence by Design” (CBD)[1]. The goal of CBD was to transform all postgraduate (postgraduate medical education [PGME]) medical and surgical specialties, through all stages of training and practice, into a CBME model.

The first step in this process was to apply a CBME framework to PGME[2]. Recognizing the significant change that CBD would bring to residency training and the support required to implement such change, the RC planned to have all specialties implement CBD over an 8-year period. The first training programs to implement CBD did so in 2017; orthopaedic surgery was scheduled to implement CBD in July 2020[3]. This paper outlines how a CBME model for residency training in orthopaedic surgery was developed in 2 pilot programs before the initiation of CBD, summarizes how a national CBD model was created, and discusses the challenges that are expected to occur during its implementation.

Previous Experience at the University of Toronto and Queen's University

Unlike other specialty training programs creating a CBD model, orthopaedic surgery already had 2 internationally renowned Canadian training programs that had pioneered the development and implementation of CBME. In July 2009, the Division of Orthopaedic Surgery at the University of Toronto was the first to implement a pilot surgical training program that was CBME-based for a subset of residents entering training, known as the Competency-Based Curriculum (CBC)[4,5]. Queen's University also adopted a CBME training model for its entire PGME system in July 2017[6]. Designed by local faculty and internationally renowned educators, these programs incorporated all the essential elements of CBME (Table I). Early experience with the pilots was positive; in Toronto, the CBC has been the sole mode of training and assessment in the training program since the 2013 to 2014 academic year[5].

These 2 training programs have become instructive paradigms for those interested in transforming a conventional curriculum into a CBME framework, particularly as it relates to curriculum design and enhanced assessment and feedback processes. The new frameworks for residency education have led to better training outcomes (as defined by performance on surgical skills examinations, time and cost to complete training, and licensing examination pass rates), delivered in a more efficient manner[5]. Despite their successes, both programs were aware that once the RC-initiated CBD model was designed for orthopaedic surgery nationally, it would replace their novel programs.

Developing a National Orthopaedic Surgery CBD Curriculum: The Specialty Committee Experience

The development of the national CBD curriculum and assessment plan was tasked to a group of academic and nonacademic orthopaedic surgeons and representatives from the RC. This group, known as the Orthopaedic Surgery Specialty Committee (SC), included program directors from all residency training programs, community surgeon representatives, subspecialty representatives, and a RC writer and clinician educator (who ensured consistency with RC policies and standards).

Using a generic template on how to implement CBD from the RC, which contained the aforementioned key elements of the Toronto and Queen's models, the SC was tasked to develop the CBD curriculum and assessment plan over the course of three 3-day workshops in 2018 and 2019. Input and advice from the Toronto and Queen's programs was used during these workshops. The RC organized and funded the workshops. Each workshop followed a structured approach developed by the RC specialties unit[7,8], with 5 key components. First, the SC was to use the CanMEDS 2015 roles as an organizing framework of competencies (Table II, which outlines the CanMEDS roles and the American College of Graduate Medical Education, ACGME core competencies)[9,10]. Second, the standard postgraduate year designation of training was to be transformed into 4 stages of postgraduate training (Table III). Although residency training programs would continue to be delivered over a 5-year period, they would have the flexibility to allow their trainees to spend as much time as necessary in each stage to acquire the relevant competencies within that period. No trainee would graduate in less than 5 years of training. If a trainee was not found to be able to acquire the necessary competencies in that period of time by the resident Competence Committee (CC; see Appendix 1, http://links.lww.com/JBJSOA/A361, for details on the function and structure of a CC[11]), the resident would go on remediation; funding for remedial time would have to be obtained from the PGME office. Third, for each stage of training, a set of entrustable professional activities (EPAs) (key tasks of the discipline that a resident must be observed to perform competently) and milestones would need to be created to cover the breadth of the relevant CanMEDS competencies that had to be acquired[12]. Note that the RC definition of “milestones”[12,13] differs from the ACGME definition of “milestones”[14]. A RC milestone refers to the individual skills that are needed to perform an EPA; an ACGME milestone describes performance levels residents are expected to demonstrate for skills, knowledge, and behaviors in the 6 competency domains. Fourth, the requirements for successful EPA achievement had to be defined, and EPA assessment forms had to be created. These EPAs would be used by each training program's CC to determine how a trainee was progressing through the program. EPA achievement, along with other forms of assessment administered by the program (such as end-of-rotation assessment forms and AAOS in-training examination results), would determine whether a trainee could progress to the next stage of training. It would be the decision of the CC to determine promotion in the training program. Fifth, although the SC would create a common set of EPAs, each residency training program would have the flexibility to self-determine the location, content, and duration of individual rotations and develop additional forms of assessment to determine stage progression. It is important to note that even if a resident were to successfully complete training, the ability to acquire certification to practice independently would only occur by passing the standard written and oral RC specialty examination at the end of the fifth year of training.

The design of the CBD program for orthopaedic surgery had to take into account the parallel CBD training program in which all of its junior residents were taking part: as of July 2018, the RC declared that all residents entering surgical residency training in Canada must complete a 2-year “Surgical Foundations” (SF)[15] CBD curriculum. This curriculum was designed to ensure that surgical trainees would acquire the relevant knowledge, skills, and attitudes underlying the practice of all surgical disciplines. The SF curriculum was designed to take place during the first 2 years of residency training and was to run at the same time as the home surgical specialty training program. Successful completion requires achievement of EPAs specific to the SF curriculum and passing a multiple-choice examination that covers the content of the curriculum.

Workshop 1

The first workshop focused on 3 tasks. The first was to define the specific set of knowledge and skills in which a graduate of an orthopaedic residency training program should be competent. The second was to define the required training experiences of a resident and the EPAs a resident must perform competently to be considered eligible for certification. The third was to define the stage of training at which each training experience should be provided and the EPAs that would have to be completed. It is important to note that the EPAs created would not assess all the skills and knowledge required of a graduate; they would only assess the skills and knowledge related to what were believed to reflect the more important clinical presentations in a specific rotation.

Workshop 2

The second workshop focused on finalizing the required and recommended training experiences and EPAs for each orthopaedic subspecialty, the method of assessment for each EPA, and the number of observations of acceptable performance required to “achieve” each EPA. The number of observations was decided by expert opinion of the SC members but was not evidence-based. Because residents commonly perform discontinuous patient care (a patient seen in the outpatient clinic might not be the same patient on whom a resident operates), discrete clinic or emergency department EPAs and procedural EPAs were developed. Limits were placed on the number of EPAs assigned to each orthopaedic subspecialty to keep the number of assessments on a specific rotation manageable while still ensuring a meaningful assessment of competence. The SC also considered limits to the total number of EPAs required within the entire CBD plan. Owing to the large number of subspecialties and different surgical procedures performed by orthopaedic surgeons in all anatomic regions, the SC opted to maintain a workable total number of EPAs (a total of 47) by grouping clinical assessments common to many subspecialties and defining key procedures for each subspecialty and stage of training (see Appendix 2, http://links.lww.com/JBJSOA/A362). Each training program was given the flexibility to assign specific EPAs to each clinical rotation.

An assessment plan was developed for each EPA, specifying the need for direct vs. indirect observation, the context of observation, the number of achieved observations required, and the assessment form to be used. Each training program was given the flexibility to use any of a number of various entrustment scales (a scoring system that assesses how much an observer would trust a resident to perform the EPA task independently) for the nonprocedural EPAs although the O-SCORE[16] entrustment scale was recommended for the procedural EPAs. Although clinical fellows and senior residents would be able to complete the evaluation of some EPAs, supervising clinical faculty would be required to complete the evaluation of most EPAs on a rotation. Although residents would be responsible for completing all EPAs in a clinical rotation, the program director and CC would closely monitor EPA completion to ensure that all requisite EPAs would be completed by the end of each rotation.

Workshop 3

The third workshop focused on developing CanMEDS-specific milestones for each EPA[17]. Each EPA would only have the relevant CanMEDS roles incorporated in it (not all CanMEDS roles would be assessed in each EPA). See Appendix 3, http://links.lww.com/JBJSOA/A363, for a sample EPA. Again, in CBD language, EPAs are the “key tasks” of a discipline, whereas milestones are observable markers of a resident's ability in performing the task. Although EPAs form the overall basis of assessment, the associated milestones are used by faculty to assess how the trainee performs in each step of the EPA. It is anticipated that when a trainee reviews their milestones performance on an EPA with a supervisor, the quality and detail of feedback provided during the EPA review will be enhanced because specific discussions can be made on each milestone.

The SC also finalized the required training experiences that are necessary for a program to provide to a trainee[18]. Many types of experiences (e.g., global surgery electives where a resident would spend time on a clinical rotation in a developing country to learn about orthopaedic care in that environment) were not included as a national standard because of the significant variability in the ability of programs to provide them.

In addition to these discussions, the SC also debated what other modalities could be used by the training programs to assess medical knowledge. It was agreed that in addition to EPAs, other assessment tools (including end-of-rotation forms and multiple-choice examinations, such as the Orthopaedic In-Training Examination, OITE) could be used at the discretion of the program. One assessment modality the SC discussed was the use of surgical case logs. Although case logs are part of a number of other international CBME surgery curriculums, they typically track exposure rather than competence. The SC agreed that programs could make their own decision whether to use them, but they were not mandatory.

At the conclusion of workshop 3, a CBD document suite consisting of the draft competencies document[19], training experiences, EPAs, and standards of accreditation[20] was ratified by the SC and submitted to the RC specialties unit for review. The final versions of these documents were approved by the RC Specialty Standards Review Committee in December 2019. This approval allowed all training programs to go ahead with the implementation of CBD in July 2020 and apply it to all incoming residents who were to begin their clinical training in the 2020/2021 academic year. Residents who existed in the training programs before this start date would continue to train along the standards set before the CBD plan. It is important to note that although the RC approved the CBD plan, each training program still had the ability to determine its readiness for implementation by July 2020. To do this, each program director worked with her or his residency program committee and the local PGME office to ensure that implementation could occur. Implementation could be delayed if significant concerns about feasibility were identified. All training programs affirmed their readiness for the CBD launch in July 2020.

Challenges with Implementation

Despite the approval of all programs to implement CBD in July 2020, the SC was aware that several significant challenges in implementation at the program level will likely occur.

One challenge relates to the implementation of the CBD assessment plan by the training programs. In some programs, resident rotation and assessment schedules will be changed so as to provide the necessary clinical and nonclinical experiences at each stage of training. Although revising schedules is painstaking, all SC members, including the program directors, know that by doing so an appropriate environment for learning and assessment will be created.

The second challenge relates to educating faculty and residents on the CBD plan. The SC recommended that local workshops be designed to assist faculty and trainees in understanding how to incorporate EPAs and assessments into their daily workflow. It is believed that the mapping of EPAs to specific rotations will help faculty better understand the assessment and feedback approach for trainees.

The third challenge relates to training programs being able to have the necessary technical and administrative support to implement CBD. An electronic platform to manage the high volume of resident assessments, with an efficient and user-friendly technology, is required. This platform must be able to provide the assessment tools to residents and faculty easily and allow uncomplicated reporting and retrieval to all stakeholders (residents, CC, and program directors) for decision-making. In addition, PGME offices and residency programs must identify sufficient administrative support to implement and maintain these platforms; this will vary depending on the size of the training program. In its survey of program directors, the SC has found that all programs have sufficient support to implement the CBD model.

The fourth challenge involves a change in the culture of residency training programs to adopt and embrace the CBD model. The experience at UofT and Queen's University showed that residents are accepting this change because the new curriculum provides increased quantity and quality of performance assessment and feedback. In addition, residents understand that academic promotion is based on the completion of EPAs. Faculty acceptance can be more difficult; increased work-based assessments add to an already busy workday. To manage this, frequent individual meetings with faculty to discuss issues with implementation should be helpful, particularly for those faculty who at first do not want to participate or for those who do not complete evaluations on a timely basis. Although such meetings will demand more time from training program leaders, such discussions can be used to show how CBD leads to a more directed and intentional resident assessment process, allowing trainees who are in difficulty to be better identified. Such meetings should also emphasize the point that although a faculty member might be filling out an EPA, this does not necessarily determine the ultimate fate of a trainee. It is the CC that, in reviewing all assessment data on a trainee, determines whether a resident has met the competencies of a specific stage of training for academic promotion.

Discussion

The outlined approach the SC took to design the implementation of the CBD curriculum and assessment plan provides insights to others who might be interested in doing the same in their own jurisdiction. It is important to note that the RC CBD initiative is Canadian, with outcomes aligned with the current standards of orthopaedic surgical practice in Canada. This plan reflects the competencies expected of what a trainee graduating from a residency training program would be expected to have if entering a general community-based practice in Canada. As such, the EPAs are not intended to reflect the full scope of orthopaedic subspecialty practice. Most reflect the procedures commonly performed by community orthopaedic surgeons in Canada. Because many members of the SC are community-based surgeons, their input was sought as the CBD plan was developed to ensure its relevance to community practice. It is not anticipated that the application process for fellowship training or the desire for graduates from the residency training programs to pursue subspecialty clinical fellowships will be altered by the CBD curriculum.

It is also worth emphasizing that although the RC and the SC were responsible for setting the CBD curriculum and providing final certification, all individual residency training programs were given the flexibility to deliver the curriculum and assessments to meet their local needs. This flexibility was critical because it promoted individual program cooperation and acceptance.

As with any major change in the way medical education is delivered, the outcome of this initiative will only become clear with time. Taking into account the experience of faculty, trainees, and program administrators, a SC review of the initial implementation of CBD is planned to occur 2 years after its introduction. The SC expects faculty and residents to give feedback on the new curriculum and assessment plans; adjustments will be necessary to improve the plan. It is only through an iterative process that this novel system, providing more focused feedback to trainees in a workplace-based environment, will be able to realize its goal of improving the quality of patient care provided by trainees and graduates of orthopaedic surgery residency training programs.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A361, A362, A363).

TABLE I

The Key Competency-Based Medical Education Elements That Were Incorporated into the University of Toronto and Queen's University Training Programs

University of Toronto[4]	Queen's University[5]
• An optimized curriculum relevant to what an orthopaedic surgery resident would need to know to enter practice	• Stage-specific EPAs and corresponding assessment
• Explicit expectations or milestones	• Systems attestation that all residents met required training experiences
• The intensive use of EPAs	• Web-based trainee portfolios that deliver and record assessments
• An intensive assessment process of all relevant CanMEDS roles	• Extensive faculty training in the implementation and evaluation of EPAs
• The intensive use of simulation to teach and assess trainees
• An optimized, modular curriculum where a trainee would acquire the necessary competencies in basic (junior) rotations before progressing to advanced (senior) rotations
• Attestation of competence (that the trainee has the ability to perform the requisite tasks of practice effectively) by supervising faculty

EPAs = entrustable professional activities.

TABLE II

An Overview of the 2015 CanMEDS Roles and the ACGME Core Competencies

CanMEDS Roles	ACGME Core Competencies
Medical expert	Patient care and procedural skills
Scholar	Medical knowledge
Professional	Professionalism
Collaborator	Practice-based learning and improvement
Communicator	Interpersonal and communication skills
Leader	Systems-based practice

TABLE III

Competence by Design Stages of Residency Training

Royal College Training Stage	Previous Time-Based Training Equivalent
Transition to discipline	First 4 months of PGY-1
Foundations of discipline	To end of PGY-2
Core of discipline	PGY-3, PGY-4, and beginning of PGY-5
Transition to practice	Final months of PGY-5

PGY = Postgraduate Year