Implementation of entrustable professional activities into fellowship council accredited programs: a pilot project.

OBJECTIVE: The Fellowship Council (FC) is transitioning to a competency-based medical education (CBME) model, including the introduction of Entrustable Professional Activities (EPAs) for training and assessment of Fellows. This study describes the implementation process employed by the FC during a ten-month pilot project and presents data regarding feasibility and perceived value.
METHODS: The FC coordinated the development of EPAs in collaboration with the sponsoring societies for Advanced GI/MIS, Bariatrics, Foregut, Endoscopy and Hepatopancreaticobiliary (HPB) fellowships encompassing the preoperative, intraoperative, and postoperative phases of care for key competencies. Fifteen accredited fellowship programs participated in this project. The assessments were collected through a unique platform on the FC website. Programs were asked to convene a Clinical Competency Committee (CCC) on a quarterly basis. The pilot group met monthly to support and improve the process. An exit survey evaluated the perceived value of EPAs.
RESULTS: The 15 participating programs included 18 fellows and 106 faculty. A total of 655 assessments were initiated with 429 (65%) completed. The average (SD) number of EPAs completed for each fellow was 24(18); range 0-72. Intraoperative EPAs were preferentially completed (71%). The average(SD) time for both the fellow and faculty to complete an EPA was 27(78) hours. Engagement increased from 39% of fellows completing at least one EPA in September to 72% in December and declining to 50% in May. Entrustment level increased from 6% of EPAs evaluated as "Practice Ready" in September to 75% in June. The exit survey was returned by 63% of faculty and 72% of fellows. Overall, 46% of fellows and 74% of program directors recommended full-scale implementation of the EPA framework.
CONCLUSION: A competency-based assessment framework was developed by the FC and piloted in several programs. Participation was variable and required ongoing strategies to address barriers. The pilot project has prepared the FC to introduce CBME across all FC training programs.

Since the introduction of the Halsted residency model over a century ago, the North American approach to surgical training traditionally has been largely time-based and case-based. In other words, after completing a set amount of time (years) and performing a set number of procedures the trainee is deemed to be ready to practice independently. However, there has been growing dissatisfaction with this model as 23% of graduating chief residents report not feeling prepared for working independently, with the majority voluntarily extended their training through fellowships [1-4]. Increasing specialization, work hour restrictions, decreased autonomy, increasing public demands for transparency and concerns for patient safety have fueled the drive away from a time-based to a competency-based framework [1, 2]. Competency based medical education (CBME) is seen as an answer to these challenges as the focus is shifted towards outcomes with the goal of all trainees obtaining competency in the essential domains of practice [5]. CBME emphasizes the abilities of the trainees while de-emphasizing time-based training to promote greater learner-centeredness and increased engagement in their own training [5].

The Fellowship Council (FC) is an association of program directors and specialty societies that accredits fellowship training programs to ensure uniformity and to maintain a standard of excellence for fellowship trained surgeons. Beginning in 2015, the Fellowship Council (FC) began a transformation to a CBME framework [6-9]. In this approach, a set of EPAs (Entrustable Professional Activities) are defined and provide a practical and intuitive way to assess competency based on the level of supervision [10, 11]. The EPAs are the core outcomes of the fellowship training program. An EPA is a real task or responsibility that may be delegated (entrusted) to the trainee when they demonstrate sufficient competence [10, 11]. EPAs describe observable work-place based behaviors of the fellow at increasing levels of entrustment from critical deficiency, to ready for independent practice. They do not evaluate knowledge, skills and attitudes in isolation, but rather evaluate the integration of these milestones at the point of care [10-12]. They are used to provide frequent “micro-assessments” across multiple clinical contexts and conditions, allowing for a broader picture of the fellow’s performance compared to the quarterly assessments used currently.

However, the move to a competency-based model is a true culture shift in surgery. Challenges in implementation may include lack of faculty and trainee “buy-in”, inability to find the time to submit EPAs, lack of training and simply forgetting to complete the assessments due to an already busy workflow [6-9]. Fellowship-level EPA assessment is also different from residency-level EPA. Fellows are at their terminal training and entrustment in their chosen area of focus has real consequences for the safety of the population. This report is the first of its kind to examine the EPA framework in a post-residency surgical cohort.

To prepare for wider implementation, the FC initiated a pilot project in 15 volunteer North American fellowship programs. The goal was for these programs to implement a competency-based framework including creating a novel web-based platform to facilitate real-time workplace assessments. This study describes the implementation process employed by the FC during a ten-month pilot project and presents data regarding feasibility and perceived value.

Methods

Creation of a competency-based framework for FC fellowships

The EPAs used in the pilot were created by the sponsoring societies (The Americas Hepato-Pancreato-Biliary Association (AHPBA), American Hernia Society (AHS), American Society for Metabolic and Bariatric Surgery (ASMBS), Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) and The Society for Surgery of the Alimentary Tract (SSAT)) through a Unified Standards Task Force established by the FC. The EPAs encompass the key components of the FC-accredited GI surgical fellowships including Abdominal Wall (AW), Bariatric (BAR), Flexible Endoscopy (FE), Foregut (FOR) and Hepatobiliary (HPB) and include pre-, post-, and intra-operative observable behaviors. Each is mapped to ACGME core competencies that can be found on the FC website (https://www.fellowshipcouncil.org/wp-content/uploads/_pda/2020/03/EPA-MappingChart-MASTER.pdf). For example, for fellowships that include foregut surgery, one EPA is “Evaluation and Management of a Patient with a Diaphragmatic Hernia.” This EPA includes multiple ACGME core competencies [13] spanning patient care, medical knowledge, professionalism, and interpersonal communication, but in practice we do not split them apart when we consider if we would “trust this fellow” to care for a patient with a diaphragmatic hernia.

In total, 8 AW, 5 BAR, 9 FE, 5 FOR, and 10 HPB EPAs were developed (Table 1; current list at https://www.fellowshipcouncil.org/epas/). These were then mapped to the various fellowship categories. At a minimum, each program was expected to use all the EPAs for their fellowship type as these reflect the core components of practice in the respective specialty field but could use any additional EPAs if relevant to their specific program (Table 2). Assessments are mapped to 5 levels of entrustment with behavioral descriptions provided for each level (Table 3).

Table 1

Complete list of EPAs at the initiation of the pilot project

Content Area	EPA
Abdominal Wall	1. Identify and prepare patients for complex hernia repair: prehabilitation (pending)
Abdominal Wall	2. Evaluate and manage patients with acute and chronic loss of domain
Abdominal Wall	3. Evaluate and manage patients with groin pain
Abdominal Wall	4. Evaluate and manage patients with parastomal hernias
Abdominal Wall	5. Evaluate and manage patients with recurrent or inguinoscrotal hernias
Abdominal Wall	6. Evaluate and manage patients with midline incisional or recurrent ventral hernias
Abdominal Wall	7. Evaluate and manage patients with contaminated hernias
Abdominal Wall	8. Evaluate and manage patients with hernias in non-central locations
Bariatric	1. Evaluate and manage a patient with clinically severe obesity (pending)
Bariatric	2. Identify patients with clinically severe obesity eligible for primary surgical therapy and perform the procedure
Bariatric	3. Evaluate and manage patients with acute complications of surgical weight loss procedures
Bariatric	4. Evaluate and manage a patient with weight regain after metabolic and bariatric procedures
Bariatric	5. Evaluate and manage patients with chronic complications of surgical weight loss procedures (pending)
Flex Endo	1. Sedation and monitoring of patients undergoing flexible endoscopy
Flex Endo	2. Endoscopy in the patient with surgically-altered GI tract anatomy
Flex Endo	3. Evaluation and management of the patient requiring advanced tissue resection/transection/ablation
Flex Endo	4. Evaluation and management of obstructing GI tract processes
Flex Endo	5. Evaluation and management of GI tract bleeding
Flex Endo	6. Evaluation and management of partial and full thickness GI tract defects
Flex Endo	7. Evaluation and management of the patient requiring submucosal or translumenal endoscopy
Flex Endo	8. Evaluation and management of patients with pancreatico-biliary diseases
Flex Endo	9. Evaluation and management of a patient needing complex endoscopic enteral access
Flex Endo	10. Evaluation and endoscopic management of the patient with gastroesophageal reflux
Foregut	1. Evaluation and management of a patient with GERD
Foregut	2. Evaluation and management of a patient with barrett’s esophagus
Foregut	3. Evaluation and management of a patient with a diaphragmatic hernia
Foregut	4. Evaluation and management of a patient with an esophageal motility disorder
Foregut	5. Evaluation and management of a patient after failed anti-reflux/hiatal hernia surgery
HPB	1. Evaluation and management of a patient with a solid pancreatic mass
HPB	2. Evaluation and management of a patient with a cystic pancreatic mass (pending)
HPB	3. Evaluation and management of a patient with severe acute pancreatitis
HPB	4. Evaluation and management of a patient with a chronic pancreatitis
HPB	5. Evaluation and management of a patient with biliary obstruction (pending)
HPB	6. Evaluation and management of a patient with a liver or gallbladder mass (pending)
HPB	7. Evaluation and management of a patient with a bile duct injury
HPB	8. Evaluation and management of a surgical patient with cirrhosis and portal hypertension (pending)
HPB	9. Evaluation and management of a patient with duodenal and periampullary diseases (pending)
HPB	10. Multidisciplinary evaluation and management of benign and malignant hepato-pancreato-biliary diseases (pending)

Table 2

Minimum required EPAs by fellowship category

Fellowship category	EPA core set (Required)
Bariatric	Bariatric 1–5
Flex Endo	Flex Endo 1–10
HPB	HPB 1–10
Advanced GI/MIS	Foregut 1–5 & Abdominal Wall 5–6
Advanced GI	Flexible—TBD

Table 3

Entrustment levels

Level 0 (Deficient Fellow)	Fellow is trusted to observe only
Level 1 (Average Entering Fellow)	Fellow is trusted to diagnose and manage with direct supervision and coaching
Level 2 (Early Developing Fellow)	Fellow is trusted to diagnose and manage with indirect supervision for simple cases
Level 3 (Later Developing Fellow)	Fellow is trusted to diagnose and manage with indirect supervision for complex cases
Level 4 (Practice Ready)	Fellow is trusted to execute the EPA without supervision but with the availability of oversight as needed

EPA assessment platform

To facilitate collection and feedback of EPA micro-assessments, a web-based platform developed uniquely for the FC-accredited fellowships and accessible through the FC website was created as part of this pilot. An assessment could be initiated by either the fellow or faculty member with the other member of the pair alerted to complete their part via a SMS (Short Messaging System) and/or e-mail link. The faculty and the fellow then both complete an evaluation choosing the entrustment level for that specific case as well as providing comments regarding what the fellow did well and where improvements could be made. The assessments take 1–3 minutes to complete and remain available for 72 h from the time of initiation to allow for the second evaluation. Once the assessment is completed by the faculty member, the evaluation can be viewed by the fellow and Program Director (Fig. 1).

Fig. 1

Example of a completed EPA

Clinical competency committee (CCC)

Each participating program was asked to establish a Clinical Competency Committee (CCC). This committee integrates the information from the EPA micro-assessments, clinical practice, and any other available assessment data to make recommendations based on the fellow’s progress towards competency. Guidelines for establishing the CCC were outlined at the start of the pilot project. This included the composition with a minimum of 3 members and would be chaired by the Program Director (PD) or Associate Program Director (APD) and could include non-physician members that could provide objective input (i.e., a nurse navigator or nurse practitioner that works with the fellow). Every fellow would be discussed quarterly with more frequent meetings as needed. Once the CCC reviewed and discussed each fellow’s portfolio, including the EPA assessments, a status recommendation was made (progressing as expected, not progressing as expected, or failure to progress) along with any action that would be required (i.e., monitor/modify learning plan, formal remediation). A summary of this meeting was then conveyed to the fellow by the PD.

Pilot project

Members from the Unified Standards Task Force were invited to participate, as were the FC Executive Committee members and Education/Curriculum Committee members. Participants attended webinars and conference calls beginning in November 2019, with the pilot project initially designed to run between January and July 2020. Due to the COVID-19 pandemic, this timeline was delayed and restarted with the 2020–21 fellowship year. A webinar for fellows and faculty was held in August 2020 to educate the participants in the pilot project. This webinar included information on CBME and EPAs, and how the pilot project was going to proceed. This webinar also ensured all participants had access to and were instructed on how to use the EPA platform.

The members of this pilot project including PDs, APDs, fellows and program coordinators were invited to a monthly zoom meeting to track the number of EPA assessments completed, assess the quality and ease of completion of these assessments, troubleshoot, and provide guidance for implementation. These meetings were approximately 1 hour in length with a total of 8 meetings over the year (September 29, 2020, to June 7, 2021). The focus of these meetings was on developing strategies to increase the completion of EPAs as well as to beta test and improve the web-based platform. The meetings started with a summary of each program’s progress with discussion of the barriers to uptake. Early on the meetings focused on ensuring that participants knew how to access and complete the EPAs, examining the rate of completed paired assessments, and understanding barriers to completing these assessments. Later in the year, focus shifted to examining which EPAs were being completed and at what rate. The progression of the fellow’s level of entrustment was also discussed at these later meetings.

Evaluation of the pilot program

Feasibility

Metrics for defining a successful pilot were defined a priori. The first goal was to be able to collect a relevant number of assessments. This was initially defined to as > 50% of operative cases. However, linking EPAs to the case log proved to be challenging given lack of patient identifiers to link cases logged to EPAs, therefore this was re-defined to using all the EPAs required for the fellowship category and ensuring there were increasing number of EPAs each month in the applicable phases of care (pre-, intra-, and post-operative). A second goal was for faculty members to be completing the assessments, in addition to the PD or APDs. A third goal was to track how the EPA entrustment level changed over the year and whether it reflected being practice-ready at the end of the fellowship.

Survey

To evaluate the value of the EPA program, a survey was distributed to all fellows and faculty at each of the 15 participating programs in October 2020. The survey consisted of two demographic questions, 4 multiple choice questions pertaining to current use/awareness of the pilot project, 8 items examining the use and value of EPAs (scored using a five-point Likert scale), and four open-ended questions where respondents could provide their thoughts on potential benefits and barriers for a competency framework. The survey was then redistributed near the completion of the pilot program. The content was very similar to the initial survey but also asked if the participant (faculty or fellow) recommended wider implementation of the EPA framework. This final survey also asked for feedback on the value of the CCC. The final survey was sent out twice to all participants and two reminders were subsequently sent to non-respondent programs. Data presented are mean and standard deviation (SD) with ranges where appropriate.

Results

The pilot project took place from the end of August 2020 to June 30, 2021. At the start of the pilot there were 16 programs; however, one program dropped out mid-pilot and their data were not included in the final assessment. The 15 participating programs included 18 fellows and 106 faculty. Program designations included: Advanced GI/MIS (6), Advanced GI/MIS/Bariatric (3), Advanced GI/MIS/Bariatric/Foregut (2), Advanced GI/MIS/Foregut (1), Foregut/Flex Endo (1), Advanced GI (1) and HPB (1).

In total, there were 655 micro-assessments initiated with 429 (65%) completed by both fellow and faculty. The fellows initiated 87% (569/655) of the assessments. The average (SD) time for both the fellow and faculty to complete an EPA was 27(78) hours. The average (SD) number of EPAs completed for each fellow was 24(18); range 0–72. Intraoperative EPAs were preferentially completed (71%) compared to the preoperative (17%) and postoperative (12%) phases. The two most frequently completed EPAs were BAR-02 and FOR-03 with 89 and 71 completed EPAs, respectively. Figure 2 displays the distribution of completed EPAs across all phases of care.

Fig. 2

Total number of each EPA completed during the pilot for each phase of care. FE-01, -03, -06, and FE-10 were not used. Note that BAR-05 and HPB-06 were added during the pilot project and HPB-05, -08, -09 were not available during the pilot (see online Appendix 1 for EPA development timeline). AW Abdominal Wall, BAR Bariatric, FE Flexible Endoscopy, FOR Foregut, HPB Hepatobiliary

Entrustment level increased throughout the year from 6% of EPAs evaluated as “Practice Ready” in September to 75% in June. Figure 3 demonstrates the cumulative scores achieved by fellows throughout the pilot year month by month.

Fig. 3

Cumulative level of entrustment achieved by the fellows throughout the pilot year. Level 0—Deficient Fellow, Level 1—Average Entering Fellow, Level 2—Early Developing Fellow, Level 3—Later Developing Fellow, Level 4—Practice Ready

Engagement increased from 39% of fellows having at least one completed EPA in September to 72% in December and then declined down to 50% in May. Over half of the attendings participated; 66% (70/106) were sent at least one EPA throughout the pilot project, and 82% of those attendings subsequently completed the EPA. However only 14% of all attendings initiated EPAs. Most EPAs were completed by PDs or APDs (67%).

Participation in the monthly zoom calls was variable with a range of 7–17 participants per call. Of the fellows, 6/18 (33%) attended at least one call while 24/30 (80%) of PDs and APDs attended at least one call.

Survey

At the end of the pilot, a survey was sent to 112 total recipients with a response rate of 63% (59/94) from faculty and 72% (13/18) from fellows. In total, 88% (52/59) of attendings and 100% of fellows who responded to the survey were aware that their program participated in this pilot project. Of the fellows, 69% (9/13) stated they were completing EPAs at least once a week with faculty stating they were completing them infrequently or weekly at the most. There were 9/13 (69%) fellows and 43/55 (78%) faculty agreeing that the feedback from the clinical competency committee was valuable (Table 4). However, 7/13 (54%) of fellows felt that implementation of EPAs was an added burden to their workload. Of the fellows, 6 (46%) recommended full implementation compared to 35 (64%) of attendings including 17 (74%) of PDs/APDs. Fellows did not feel that EPAs improved their confidence in their ability to start practice in contrast to attendings who felt that EPAs were useful in determining if fellows were ready for independent practice as demonstrated by the fact that 6/13 (46%) of fellows disagreed with the statement compared to only 9/57 (16%) of attendings).

Table 4

Results of the survey distributed at the end of the pilot

		Strongly Disagree (1)	Disagree (2)	Neither (3)	Agree (4)	Strongly Agree (5)	Total	Average
EPAs were easy to implement in my fellowship program	Fellow	2 (15%)	2 (15%)	3 (23%)	4 (31%)	2 (15%)	13	3.15
	Faculty	1 (2%)	9 (16%)	15 (27%)	26 (46)%	5 (9%)	56	3.45
The EPA assessment platform is easy to use	Fellow	2 (15%)	3 (23%)	4 (31%)	2 (15%)	2 (15%)	13	2.92
	Faculty	3 (5%)	4 (7%)	16 (29%)	23 (41%)	10 (18%)	56	3.59
I was able to complete an EPA in a timely fashion	Fellow	1 (8%)	4 (31%)	2 (15%)	4 (31%)	2 (15%)	13	3.15
	Faculty	1 (2%)	6 (11%)	14 (25%)	25 (45%)	10 (18%)	56	3.68
EPAs are not an additional burden to my workload	Fellow	3 (23%)	4 (31%)	2 (15%)	3 (23%)	1 (8%)	13	2.62
	Faculty	4 (7%)	9 (16%)	27 (48%)	13 (23%)	3 (5%)	56	3.04
The feedback/discussion from the competency committee was valuable	Fellow	1 (8%)	0 (0%)	3 (23%)	6 (46%)	3 (23%)	13	3.77
	Faculty	0 (0%)	2 (4%)	26 (47%)	17 (31%)	10 (18%)	55	3.64
I recommend the Fellowship Council implement the EPA framework for all fellowships	Fellow	4 (31%)	2 (15%)	1 (8%)	4 (31%)	2 (15%)	13	2.85
	Faculty	3 (5%)	2 (4%)	15 (27%)	24 (44%)	11 (20%)	55	3.69

Data expressed as n (%); bolded values signify responses that tended more towards agreement on average, whereas the italics signify the responses with a tendency towards disagreement on average

Additional free-text comments regarding the benefits and barriers encountered during the pilot were submitted by 53 participants (40/59 faculty and all 13 fellows). When faculty were asked what benefits implementing EPAs had on their programs only 7 commented that there was no or very little benefit; others felt it allowed for more frequent and structured feedback by formalizing the process. The comments from the fellows were more divided with some stating that there was “more frequent feedback and great for documenting progress. The comments were the most helpful with specific things to work on...” and others commenting that “It created an additional burden to an already hectic schedule. It was merely a checkbox that I had to beg the faculty to complete…I did not receive any valuable feedback from it.” Some faculty felt that “The EPA framework is duplicative with what we already do. We ultimately found it time consuming, non-productive, and it didn't offer any additional value to what we were doing.” while others felt it improved their feedback as seen in this comment: “I like the granular descriptors which forced me to think critically about my specific ratings and justify them, which in turn facilitated documenting specific feedback to the fellow.” This survey as well as the monthly zoom meetings highlighted that development of an EPA specific app might be beneficial. A summary of the barriers and benefits expressed by the summative comments of the final survey can be found in Table 5.

Table 5

Benefits/barriers of EPAs identified in the final survey

Barriers	Benefits
Finding time to do an additional task	Increased communication
Forgetting to initiate/complete EPAs	Formalized/regular/immediate feedback
Disappointment when a self-evaluation was created, and the EPA timed out before attendings could give feedback	Clearer objectives
Limited variety of EPAs	More input from a variety of attendings
Education of "off-service” faculty	Ability to monitor growth/development more frequently
Technical issues with the platform	Ability to compare fellow/faculty assessments
Lack of a mobile app	Increased accountability for consistent education/training

Discussion

This study is the first to evaluate a competency-based model for post-residency surgery fellowships training based on novel EPA assessments. While the EPA-based framework can be beneficial for fellowships, widespread adoption will require education to drive changes in culture and practice that will take time. Even with a group of highly motivated program directors and fellows, compliance was variable. However, the EPAs demonstrated the expected graduation of responsibility to “practice ready” in the overall cohort. The competency committee was also seen as a helpful addition to training by both fellows and faculty. Overall, the majority of faculty endorsed implementation of an EPA framework across the FC fellowships.

Implementation of new programs, especially ones where the assessment construct and theory differ from what surgeons are used to, comes with an expected learning curve and an adjustment period. A pilot program implementing EPAs in the University of Wisconsin’s general surgery training program reported similar results in terms of variability of utilization and rate of completion of EPAs as in the present study [7]. Similar barriers were identified including difficulty developing the habit of completing EPAs when EPAs only exist for certain operations, not enough time, forgetting to complete assessments, low faculty response rate, and some learners/faculty not finding the evaluations useful in this early experience [7]. Despite this, a high volume of assessments was collected over their pilot project. The success of their experience was attributed to “high-level departmental buy-in” and additional personnel to assist with development of a mobile application and educational support to achieve behavioural change [7]. Furthermore, half (52%) of their assessments were submitted by residents demonstrating a shared commitment by faculty and learners in comparison to 87% of assessments being initiated by fellows in our experience. Similar to our findings, other surgical programs noted a lack of representation in pre and post op EPAs in favor of intra-operative EPAs [14]. For fellowship level, this perhaps is not necessarily a negative as many individuals seek fellowship to improve their surgical technique [2, 15].

Oregon Health and Science University’s medical school underwent a shift to a competency-based, time-variable curriculum in 2012 [16]. A manuscript published about their journey highlighted the resistance to change from certain faculty who “fought to preserve the status quo” in contrast to the support from the medical students who were embracing the change. They outlined lessons learned including expected resistance, the need for perseverance and grit, appropriate leadership, and risk tolerance [16]. In contrast to their findings, our faculty, especially the program directors, were more in favor of the change than the fellows. Queens University implemented CBME in their family medicine program starting in 2009 [17]. They generated a list of facilitating factors for implementation including many of the techniques that can be employed by the fellowship council in the future such as faculty development to build skills and confidence as expert assessors, boot camps for residents to optimize self-regulated learning and minimizing time and effort involved (app for hand-held device, structure, and intuitiveness of tool). They also highlighted the cost of setting up an EPA platform of approximately 600 h with upkeep/improvement cost of 150–250 h/year [17]. Ease of use of the FC website on a mobile device remains cumbersome and a common suggestion was that development of an app would facilitate this technical aspect, however, there may be financial constraints associated with its development.

Despite demonstrating the overall feasibility of introducing EPAs into Fellowship Programs, this study suggests that additional stumbling blocks can be expected with wider implementation. This pilot included motivated volunteer participants, many of whom were involved in the writing and creation of EPAs or part of the FC leadership and may not be representative of all fellowship programs. Even with this carefully selected group, we had a highly variable rate of EPA submissions ranging from 0–72 per fellow. A proposal to link the EPA assessment to the required case log is an area that the fellowship council will investigate to potentially streamline this workflow for busy fellows. This would allow for automatic generation of an EPA when the fellow logs an index case for the fellowship type. This change would affect the intra-operative EPA response, but not the other phases of care.

The faculty buy-in was also variable with only 33% of EPAs being completed by faculty other than the PD or APD. This was despite the unique mentorship of the monthly calls and significant support provided by the Fellowship Council, which will not be feasible to reproduce on a yearly basis with each new cohort of fellows. The success of EPAs will require institutional champions to drive this practice change and provide faculty development required to disseminate wider understanding of the potential benefits of this learner-centered, outcomes-based training model. With the larger scale roll out, the FC will be providing monthly reports of the number of EPAs submitted and quarterly summations of the gaps in EPAs completed to help provide streamlined feedback to fellows and program directors. This type of dashboard that displays the micro-assessment data in a meaningful way to the trainee and faculty as well as provides predictive analytics for how the trainee is expected to perform (i.e., expected entrustment level) has proved very valuable in the University of Wisconsin experience (R Minter, personal communication).

Conclusion

An assessment framework based on EPAs was developed by the Fellowship Council and integrated successfully in programs participating in a pilot project. Participation was variable but improved over the year as strategies to address implementation challenges were developed. Successful implementation will require increased education on the value of EPAs to increase buy-in as well as continued improvements in the web-based technology to facilitate ease of use. The pilot project has prepared the Fellowship Council to introduce CBME using EPAs on a wider scale and to continue to study its impact and make improvements as needed.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 17 KB)