Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training.

Entrustable professional activities are an intuitive form of workplace-based assessment that can support competency-based medical education. Many entrustable professional activities have been written and published, but few studies describe the feasibility or implementation of entrustable professional activities in graduate medical education. The frozen section entrustable professional activit was introduced into the pathology residency training at the University of Vermont for postgraduate year 1 at the start of their training in frozen section. The feasibility of the entrustable professional activit was evaluated based on 3 criteria: (a) utilization, (b) support of frozen section training, and (c) generating data to support entrustment decision about residents' readiness to take call. The entrustable professional activit was well utilized and satisfactory to residents, faculty, pathologists' assistants, and Clinical Competency Committee members. Most members of the Clinical Competency Committee agreed they had sufficient data and noted higher confidence in assessing resident readiness to take call with the addition of entrustable professional activit to the residents' assessment portfolio. Residents did not endorse it helped them prepare for call; however, the interruption to frozen section training due to the COVID-19 pandemic was a significant contributing factor. The frozen section entrustable professional activit is a feasible addition to pathology resident training based on utilization, support of training, and generation of data to support entrustment decisions for graduated responsibilities. The implementation and integration of the entrustable professional activit into pathology training at our institution is described with discussion of adjustments for future use.

Introduction

As graduate medical education (GME) shifts toward competency-based assessment, practical tools, and guidelines on implementation are necessary to support resident education. Entrustable professional activities (EPAs) are concrete tools to support learning and assessment of the day-to-day activities in residency.[1] Entrustable professional activities support learning by laying out clear expectations of residents (via the knowledge and skills statements) and anchoring the assessment in entrustment, which closely aligns with graduated responsibilities in GME.[2,3] Entrustable professional activities map to milestones and contribute to the semiannual review process by the Clinical Competency Committee (CCC).[4,5]

Implementing EPAs into a training program is no small feat. It starts with development, revision, evaluation, and finally, formal adoption of an EPA into a curriculum.[6] Numerous medical specialties have published examples of EPAs for their field (including 5 in the field of pathology[7-11]), but only a handful of studies have rigorously evaluated EPAs, described how EPAs are integrated into a training program, or documented the use around graduated responsibilities.[12-19] Peters et al[3] describe tips for successful incorporation of EPAs into training, including using the EPA to guide teaching and providing focused feedback, which is helpful, however, more study is necessary. As with any good assessment, EPAs need to be evaluated on their feasibility, validity, reproducibility, equivalence, effectiveness, and acceptance.[20] The goal of this study is to build upon the development of EPAs in pathology[7] by describing the implementation of an EPA in early frozen section training to support learning and assessment.[17]

At the University of Vermont, Pathology residents begin taking call in a time-dependent manner—halfway through postgraduate year 2 (PGY-2) year, after completion of core rotations. The CCC is responsible for deciding if a resident is competent to take call but is currently operating with limited data, often relying on “gut feelings” (personal communication with CCC Chair, February 2019). In this study, Pathology PGY-1 Pathology residents at the University of Vermont were assessed during their frozen section training by pathologists’ assistants (PAs) and faculty using an EPA assessment tool “Performing an intraoperative consultation and frozen section.” This feasibility study evaluates the EPA based on 3 factors (a) utilization, (b) support of frozen section training, and (c) generation of data for the CCC to support entrustment decisions, specifically, readiness to start taking call.[16,19,21] The implementation and integration of the EPA into pathology training at our institution is described with discussion of adjustments for future use.

Materials and Methods

Setting

The University of Vermont Medical Center is a medium-sized academic center with approximately 38 000 surgical cases per year and an average of 3.2 intraoperative consultations (frozens) per day (range 0-8). At the University of Vermont Medical Center, frozen coverage is partially subspecialized, with up to 4 different attendings interpreting frozens in a day.

Residents rotate through surgical pathology (SP) in 4-week blocks and are scheduled on a 2 different subspecialty service in 2-week segments. Frozen section training is introduced as a dedicated week in the third SP block of the PGY-1 year. Residents work closely with a PA at the beginning of the week to practice technical skills and learn the full frozen section process. By the end of the week, residents are handling the technical skills with minimal supervision or assistance. After this week, residents continue frozen section training one day per week during subsequent SP blocks. Pathology residents begin taking call in a time-dependent manner—halfway through PGY-2 year, after completion of core rotations.

Residents on SP are assessed with an end-of-rotation evaluation (by faculty) and a 360° evaluation (by PAs). All evaluations are housed on the GME online platform (New Innovations). In 2018 to 2019, the end-of-rotation evaluation was based on knowledge and skills statements from the grossing, frozen, and SP reporting EPAs (adapted from McCloskey et al[7]). With the adoption of Milestones 2.0 in 2019, the end-of-rotation evaluation was changed to include pertinent Milestone 2.0 subcompetencies, including Patient Care 5 (Intraoperative consultation, including Frozen Section). All assessments include the option of “unable to assess” if the faculty did not observe a behavior.

Study Cohorts

The educational cohorts in this study included 2 consecutive years of anatomic and clinical pathology residents at the University of Vermont Medical Center—the Class of ‘22 (n = 5) and the Class of ‘23 (n = 4). A timeline is shown in Table 1. Both classes had a dedicated frozen week in the third SP block (SP3). The Class of ‘22 spent 1 day per week participating in frozen sections in their subsequent SP rotations. All members of the Class of ‘23 were able to complete their frozen section week, but due to the COVID-19 pandemic, subsequent 1-day frozen training was put on hold (asterisk in Table 1).

Table 1.

Timeline of Frozen Section Training in Surgical Pathology.

Timeline		Resident rotation	Assessment time point	Survey time point
2018-2019 Academic Year	July-October 2018	Enter Class of ‘22, Complete SP1-SP2 rotations
	November 2018 to February 2019	Class of ‘22 complete full Frozen Week in SP3
	March-June 2019	Class of ‘22 does SP4 with 1 frozen d/wk	Class of ‘22 self-assessment
2019-20 Academic Year	July-October 2019	Enter Class of ‘23, Complete SP1-SP2 rotationsClass of ‘22 does 1 frozen d/wk in SP4-6
	November 2019 to February 2020	Class of ‘23 does SP3 with full Frozen Week	Class of ‘23 asks for EPA assessmentsClass of ‘22 self-assessment	CCC surveyed about Class of ‘22
	March 2020	Class of ‘23 does 1 frozen d/wk in SP4 and beyond*	Class of ‘23 asks for EPA assessments*
	June 2020	Semiannual ReviewCCC reviews EPA formative and summative assessments	Class of ‘23 self-assessment	Residents, PAs, Faculty surveyed; CCC surveyed about Class of ‘23

Abbreviations: CCC, Clinical Competency Committee; EPA, entrustable professional activity; SP, surgical pathology rotation.

* Did not occur after March 15, 2020, due to COVID pandemic.

The Class of ‘23 has the EPA incorporated into their frozen section training, while the Class of ‘22 utilized the 2018-2019 end-of-rotation assessment (with knowledge and skill statements). Surveys to evaluate feasibility were send to the Class of ‘23, faculty and PAs who filled out at least one paper copy of the EPA assessment tool, and members of the CCC.

Design and Implementation of EPA Assessment Tool

The formative and summative EPA assessment tools are available as (Supplemental Appendix 1A and B), which are adapted from the frozen section EPA published by McCloskey et al[7] and included entrustment language from 10 Cate.[1] Formative assessments consist of knowledge and skill statements anchored in competency followed by a global entrustment rating. Summative assessments are composed of a global entrustment rating.

The Class of ‘23 residents were given an orientation on EPAs in August 2019, including an information sheet about the study. Immediately prior to their scheduled frozen week, each resident received paper copies of the EPA assessment tool. Residents were instructed to ask for 5 assessments throughout the week, each on a specific case, and it could be filled out by a PA or faculty member. Since PAs and Faculty may observe different aspects of a case, not all knowledge and skill statements of the EPA needed to be checked off on a single formative assessment.

A dedicated Faculty/PA development session was planned for fall 2019; however, this training coincided with a department-wide move to a new electronic medical record system requiring significant training. A dedicated Faculty/PA development session was not feasible at that time. Instead, prior to the designated frozen section week, the author individually met with faculty on clinical service and PAs to provide a brief orientation to the EPA assessment tool and how it would be incorporated into workflow.

All formative assessments were reviewed by the rotation director (B.H.B.), and a summative entrustment level was assigned for each resident prior to the semiannual CCC review. All formative and summative assessments were given to the CCC for semiannual review.

Evaluation of Feasibility

The feasibility of the EPA in our training program was evaluated on 3 main principles: (a) Do residents utilize the EPA? (b) Can the EPA support frozen section training? and (c) Can the EPA generate data for the CCC to make entrustment decisions, specifically, readiness to start taking call?

To assess whether residents (a) utilized the EPA, the number of EPAs completed for each resident, and by whom, is reported. To assess whether (b) the EPA supports frozen section training, anonymous surveys were collected and managed using REDCap electronic data capture tools hosted at the University of Vermont.[22,23] Surveys were sent to the Class of ‘23, faculty and PAs who filled out at least one paper copy of the EPA assessment tool, and CCC members in June 2020. Survey questions focused on ease of use, usefulness, facilitation of feedback (residents, faculty, PAs), understanding of the frozen section process (residents only), and improvement in frozen section education (faculty and PAs only).

To assess whether (c) the EPA generated data for the CCC to make entrustment decisions (c), the CCC members were asked what data they reviewed, if they had sufficient information to assess resident’s readiness to take call, and their confidence in that assessment. These 3 survey questions were asked December 2019 (reflecting on review of the Class of ‘22, just before starting call) and June 2020 (reflecting on review of the Class of ‘23). All surveys were planned to be repeated in December 2020, but due to the interruption to frozen training from June to December 2020, these surveys were not administered (Table 1). Survey questions are listed in Supplemental Appendix 2.

All residents completed the EPA as a self-assessment during their semiannual review process. The Class of ‘23 self-assessments in December 2020 is compared to the Class of ‘22 self-assessment in December 2019, which represents the same time point in training. The Class of ‘23 was also asked if the EPA helped them feel ready to start call.

Institutional Review Board Review and Funding

This study was deemed exempt from institutional review board review based on survey-only study. All participants were reminded survey completion was optional.

Results

Survey Responses

All 4 residents in the Class of ‘23 completed the survey (100%). Across all 4 of the resident frozen weeks, a total of 9 different faculty could have participated in frozen section. Seven faculty filled out paper EPA assessments tools and were surveyed (it is unknown if the other 2 faculty were asked to fill out an EPA assessment tool). Four of the 7 faculty completed the survey (57%). All 4 PAs who train residents in frozen section skills completed at least one EPA assessment tool and received the survey. Two (50%) of the 4 PAs completed the survey. The CCC has 6 members; 4 (67%) CCC members completed the survey in December 2019 and 5 (83%) CCC members completed the survey in June 2020. Results are reported below as the percentage of respondents who agreed or strongly agreed to the question, unless otherwise specified.

Utilization

Residents requested an average of 4 formative EPAs during their frozen week (range: 3-5). Two residents requested EPAs from both PAs and faculty, 1 requested EPAs from faculty only, and 1 requested EPAs from PAs only (Table 2). Three trainees requested EPAs during their frozen week. One trainee (resident 1) requested EPAs several weeks after their frozen week (after a reminder email). One summative assessment was completed on each resident in Spring 2020, based on review of all formative EPAs for each resident.

Table 2.

Number of Formative Assessments Completed on Each Resident.

Resident	Total # EPAs completed	# competed by PA	# Competed by Faculty
Resident 1	3	3	0
Resident 2	4	2	2
Resident 3	5	2	3
Resident 4	5	0	5

Abbreviation: EPA, entrustable professional activity.

Support of Frozen Section Training

Residents, faculty, and PAs responded favorably to survey questions regarding ways the EPA supported resident training. Seventy-five percent of residents found the EPA easy to use, 50% found it useful for learning, 50% received useful feedback about their frozen skills, and 75% reported it helped them understand the frozen section process (Figure 1A). Notably, one resident disagreed that the EPA was easy to use or that they received useful feedback, noting it was cumbersome to complete because PAs more frequently taught and assessed the technical aspect (knowledge and skill statements #2-4) and faculty assessed the interpretive and communication aspects (knowledge and skill statements #5-6). The residents’ suggestions for improvements included have the formative assessment readily available for timely completion at the time of frozen and separating the EPA in the technical aspect and the interpretive aspect.

Figure 1.

Satisfaction survey results for (A) residents, (B) faculty, (C) pathologists’ assistants, and (D) Clinical Competency Committee.

One-hundred percent of faculty found the EPA easy to use, useful for teaching, and that it helped them provide useful feedback to the resident; 50% reported it improved resident education compared to previous training and 50% stated they were unable to make that comparison (Figure 1B). Both PA respondents agreed that the EPA was easy to use, and one agrees it was useful for teaching, helped provide useful feedback, and improved resident education compared to previous training; the other PA was neutral on those topics (Figure 1C). One PA commented that residents were not always consistent in asking for a formative EPA assessment, with some asking after a frozen case and some asking after their week of frozen training was completed.

Entrustment Decision

One hundred percent of CCC respondents agree that the EPA was easy to understand and 80% agreed EPAs were useful in assessing resident readiness to start call (Figure 1D). When making this assessment, the CCC utilized the following information: EPA formative assessment (40%), EPA Summative assessment (60%), end-of-rotation evaluation (80%), 360-degree evaluation (60%), resident peer evaluation (20%), experience working with the resident on frozen section (0%), and experience working with the resident on a different service (80%); 1 (20%) CCC member wrote in that they used the resident EPA self-assessment.

Comparative survey of the CCC showed that in December 2019, none (0%) agreed that they had sufficient information to assess residents’ readiness to take call, while in June 2020 (with the addition of the EPA in the residents’ assessment portfolios) 80% agreed they had sufficient information (Figure 2A). The average CCC member’s confidence in assessing readiness to take call (on a sliding scale of 1-10), went from 3.8 to 6.1 between December 2019 and June 2020 (insufficient data to calculate statistical significance; Figure 2B). Comments by CCC members in December 2019 noted “more assessment and faculty input is needed” and “there is very little objective data on performance.” Comments in June 2020 noted that “more information was provided than previously,” “EPAs on frozen section were very helpful with respect to frozen section call,” and “the EPA self-assessment allowed resident to express their level of comfort and confidence in performing frozens.”

Figure 2.

Clinical Competency Committees survey responses regarding having sufficient information to assess residents (A) and confidence in assessing readiness to take call (B) in December 2019 (before entrustable professional activities [EPAs]) and in June 2020 (after the addition of EPAs).

The Class of ‘23 was asked if the EPA helped them feel ready to start taking call. Only one (25%) respondent agreed, 2 (50%) were neutral, and 1 (25%) strongly disagreed (Figure 3A). When comparing the self-assessed entrustment level prior to starting call, 25% of the Class of ‘23 felt they needed “reactive supervision (attending available within minutes)” when performing intraoperative consultations, as compared to 80% of the Class of ‘22 felt they needed “reactive supervision (attending available within minutes)” at a similar point in training (Figure 3B). The other residents from both classes chose “direct/close supervision (attending in room).”

Figure 3.

Residents’ confidence in stating call. The Class of ‘23’s survey response (A) and comparisons between the Class of ‘22 and Class of ‘23 self-assessment of entrustment level for frozen section (B).

Discussion

This study showed that the frozen section EPA is a feasible addition to resident training at the University of Vermont. The EPA was filled out by both PAs and Faculty. Most residents agreed the EPA helped them understand the frozen section process, suggesting residents found the frozen EPA to be a useful tool to support learning and education, rather than purely for assessment. The knowledge and skills statement of an EPA can layout a roadmap to competence, which is particularly helpful when teaching a new skill.[3] In contrast, only half the faculty and PAs agreed the EPA improved resident frozen section training. This may be due to the limited experience with the EPA or the lack of faculty/PA development to describe why EPAs are useful and how they can support training. Although faculty development was not incorporated into the implementation of EPAs, faculty, and PAs rated residents near the same entrustment level. This may be a reflection of the limited use of the EPA for residents in their first week of frozen section training. Expansion of the EPA to later training may not show such precision of global entrustment rating without faculty development. A larger EPA study would benefit from consistent faculty development prior to implementation of EPAs.[24]

The Formative and Summative EPAs provide useful data to the CCC, boosting the CCC’s confidence in assessing resident’s readiness to take call. No members of our CCC participate in intraoperative consultation as part of their daily work, and so it is not surprising that none utilized “personal experience working with a resident on frozens” as part of their assessment. The prior method of frozen assessment was significantly limited. For the Class of ‘22, the end-of-rotation assessment (based on knowledge and skills statements) had “unable to assess” selected for the frozen section items the vast majority of the time. The primary issue at hand was that it was rare for a faculty to work with a resident on frozens and on subspecialty SP service during the same block. As the CCC went from having “very little objective data” to “more information than was provided previously,” it is clear the EPA is a welcome part of the assessment portfolio. This highlights the advantage of a stand-along assessment of frozen section at our institution (not necessarily the content of the assessment) to document frozens skills in a robust way and provide data to the CCC. However, educators should proceed with caution, as a common flaw with assessment is that educations often conclude their decisions are valid after evaluating limited evidence.[25] Robust assessment systems that are optimized and tailored for an individual program’s workflow are essential as CCC in other fields make decisions around graduated responsibilities.[26,27] Further study—after our CCC has more experience with the EPA and other EPAs—may demonstrate the value of such data to support entrustment decisions.

In contrast to the CCC, the Class of ‘23 residents did not feel as confident in starting call and rated themselves lower on the entrustment scale (“direct/close supervision (in the room)”) compared to the Class of ‘22 (“reactive supervision (attending available within minutes)”). At the time of the self-assessment, at least 9 months had passed since any member of the Class of ‘23 had had formal training in frozen section due to the interruption from the COVID pandemic. This gap in training is likely a contributing factor to the lower self-entrustment rating, which was noted by at least one member of the Class of ‘23 (personal communication). Additional experience with the EPA is necessary to determine whether it supports resident confidence in their stills prior to attaining graduated responsibilities, such as taking call.

Integrating the EPAs into the workflow for consistency and efficiency was highlighted in the survey twice. The resident who requested the EPA after the frozen week ended up only received EPAs from 3 PAs emphasizes the importance of this point. The paper forms were practical to complete quickly in the fast-paced environment of intraoperative consultations and can be stored in the Frozen Room for easy access. Reminding trainees to request an EPA at the time of (or near to) the case being evaluated is vital, particularly at our institution where frozen sections are subspecialized and a trainee may work with up to 4 attendings in one day. Some programs require one EPA is filled out at the end of the workday, which is feasible when a trainee works with one attending all day (and has the added advantage of longitudinal experience; Diane Kowalski, Yale University, personal communication). In a resident-driven assessment system, selection bias is a possible confounding factor. Using a more systematic approach (eg, first case of the day or all frozens on one day of the week) may control for this bias better. All resident end-of-rotation assessments are completed in online platforms, and this study did not address the practicality of integrating an e-version of the assessment into workflow.

The biggest limitation of this feasibility study pertains to the small sample size at a single institution, which does not address feasibility principles of adaptation and expansion.[21] Frozen section training can vary greatly between institutions, thereby requiring nuanced ways to incorporate EPAs into training. The Frozen EPA is better aligned with workflow and frozen section training at our institution than the previous end-of-rotation resident assessment methods, supporting its continued use; however, this may not be the case for all institutions. In addition, this study only looked at an EPA for one on-call skill and should be only extrapolated with caution to other common on-call tasks such as rapid on-site cytology evaluations or any clinical pathology on-call tasks. As this study focused on PGY-1 residents, it applies less to senior trainees and more advanced graduated responsibilities, such as independent evaluation of frozen sections. The areas of adaptation and expansion will be further evaluated in a National EPA pilot, which will trial 4 EPAs (3 of which pertain to common on-call skills) at 10 pathology residency programs across the United States.

In conclusion, this feasibility study demonstrated successfully that the EPA can support resident training in frozen section at the University of Vermont and provide useful data to the CCC to support entrustment decisions. Although this study is not a robust evaluation of the frozen EPA in training, it offered useful lessons for implementing EPAs and demonstrated sufficient utility for continued use and study. Further study of additional EPAs at the University of Vermont and other pathology residency training programs will further inform best practice for using EPAs to support learning, assessment, and achieving graduated responsibility.

Click here for additional data file.

Supplemental Material, sj-docx-1-apc-10.1177_23742895211041757 for Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training by Bronwyn H. Bryant in Academic Pathology

Click here for additional data file.

Supplemental Material, sj-docx-2-apc-10.1177_23742895211041757 for Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training by Bronwyn H. Bryant in Academic Pathology