Experience with key indicator cases among otolaryngology residents.

OBJECTIVE: To describe the resident experience with respect to key indicator cases for each year of training. STUDY
DESIGN: Multi-institution, cross-sectional assessment.
METHODS: Using an electronic survey, current otolaryngology residents were solicited to complete a survey regarding their experiences with the key indicator cases to that point. The survey was sent to this cohort in the winter of 2017-2018.
RESULTS: Three hundred and three residents responded, with 293 completing the survey. Twenty-three percent were PGY1, 19% PGY2, 21% PGY3, 18% PGY4, and 19% PGY5 or higher. The majority of residents progress from resident assistant as a PGY2, to resident surgeon as a PGY3 and self-assessed competent surgeon as a PGY4 for the majority of the key indicator cases. Less than 50% of the surveyed PGY5 residents had reached independent practice in all the key indicator cases, with stapedectomy (16%), rhinoplasty (18%), and paramedian forehead flap (14.5%) being the cases least frequently performed independently. Ninety-five percent of the respondent residents felt their program provided adequate training, but 20% of the respondents were either unsure or believed that they would be unable to perform all the key indicator cases by the completion of their training.
CONCLUSIONS: The majority of otolaryngology residents feel confident in their training, but experience with certain cases lags behind and may not currently be taught as resident level cases. These findings raise the question of whether the current key indicator cases are the best option for assessing breadth and depth of residency training. LEVEL OF EVIDENCE: NA.

INTRODUCTION

How should we train a competent, autonomous surgeon? This is a difficult question to answer. In the past, surgical training was a master and apprentice model of education, where the surgical trainee practiced under a master surgeon with gradually increasing independence and progressively less oversight. With the changes in both the climate of medical practice and medicine itself, trainees operating without oversight have essentially disappeared. The longitudinal master–apprentice relationships have fragmented secondary to subspecialization and increasing faculty size.1 In some specialties, this has led to surgeons who are not prepared for independent practice at the conclusion of residency.2 Efforts have been put forward to better assess and measure autonomy and progression to competence in surgical residency. In late 2017, the Royal College of Surgeons of Canada released their Competence by Design program, which utilizes the Ottawa surgical competency operating room evaluation (O‐SCORE, Table 1).3, 4 This scale is meant for supervising surgeons to use in assessing a trainee's proficiency as he or she progresses from novice to independent practice.

Table 1

Ottawa surgical competency operating room evaluation.3

Level	Descriptor
1	"I had to do" Requires complete hands on guidance, did not do or was not given the opportunity to do
2	" I had to take them through" Able to perform tasks but requires constant direction
3	" I had to prompt them from time to time" Demonstrates some independence, but requires intermittent direction
4	" I needed to be in the room just in case" independence but unaware of risks and still requires supervision for safe practice
5	" I did not need to be there" Complete independence, understands risks and performs safely, practice ready

In 2013, the American Council for Graduate Medical Education (ACGME) selected 14 representative cases to use as markers to verify that each otolaryngology residency had adequate breadth and depth of surgical training.5 Beginning with the 2014 class, the expectation was that all graduating residents would have been the primary surgeon for each of these specified surgeries a certain minimum number of times (Table 2). The ACGME stipulated that these minimum numbers were evidence of experience, but not necessarily competence. This statement raises at least two questions: “what is the resident experience with these cases?” And “do residents become competent at performing these cases by graduation?”

Table 2

American Council for Graduate Medical Education key indicator cases and resident surgeon descriptions.5

Category	Procedure	Minimum
Head and Neck	Parotidectomy	15
	Neck Dissection	27
	Oral Cavity resection	10
	Thyroidectomy/Parathyroidectomy	22
Otology	Tympanoplasty	17
	Mastoidectomy	15
	Stapedectomy/Ossiculoplasty	10
Facial Plastics	Rhinoplasty	8
	Mandible/Midface Fractures	12
	Flaps and Grafts	20
General/Pediatrics	Airway ‐ Pediatric and Adult	20
	Congenital Neck masses	7
	Ethmoidectomy	40
	Bronchoscopy	22
Resident Surgeon	Performs ≥ 50% of the operation with the attending physician or resident supervisor, including the key portions of the procedure.
Resident Assistant	Performs < 50% of the operation, or ≥ 50% of the operation but not the key portions of the procedure.
Resident Supervisor	Instructs/assists a more junior resident during a procedure in which the junior resident performs ≥ 50% of the operation, including the key portions of the procedure; the attending physician acts as an assistant or observer.

METHODS

After a review of both the otolaryngology and graduate medical education literature, we designed our questionnaire (Supporting Appendix A). We selected what the authors felt were characteristic cases from each of the key indicator subheadings. For the broader categories of flaps and grafts and facial trauma, we selected two cases. We then created a modified version of the O‐SCORE to reflect the trainee's experience with the key indicator cases, an example of which can be seen in Figure 1. Each of the four questions is meant to correlate with the progression from assistant surgeon, to primary surgeon, to competent surgeon upon self‐assessment, and then actual independent surgeon, a presumed indicator of supervising surgeon's assessment of competence. We also queried demographic information of the respondent resident including but not limited to year in training, size of training program, region of program, presence of fellows, and planned career path. At the conclusion of the survey, the responding resident was asked to assess his or her surgical training and if he or she would feel comfortable performing the key indicator cases in unsupervised practice. Approval from the WVU Institutional Review Board (#1710823746) was obtained.

Figure 1

Modified Ottawa surgical competency operating room evaluation for parotidectomy.

The program coordinators from the 107 allopathic otolaryngology programs were solicited by email using REDCap.6 The program coordinators were asked to send a link for the anonymous survey to their residents. Solicitations for the survey were sent out four times between November 2017 and February 2018.

Responses were compiled and evaluated for statistical significance with RStudio Version 1.1.383 (RStudio IDE, Boston, Massachusetts). Significance was determined with P values less than .05. Wilcoxon Rank‐Sum and Kruskal–Wallis tests were used for comparisons where appropriate, and a Bonferroni correction was applied when appropriate.

RESULTS

Of a potential 1,500 residents, 303 unique responses were received with 293 completing the survey, for approximately a 20% response rate (Table 3). We did not ask for any confirmation from the program coordinators if they received the email or if they sent the survey to their residents, and so we are unsure if all 1,500 residents were queried. Respondents represented all five regions and all 5 years of surgical training (Table 3).

Table 3

Demographics of Responding Residents.

Total complete responses	293
Region	N (%)
Midwest	80 (27.3)
Northeast	56 (19.1)
Southeast	89 (30.4)
Southwest	33 (11.3)
West	35 (11.9)
Residents per class	N (%)
1	14 (4.9)
2	63 (22.0)
3	123 (42.9)
4	62 (22.6)
5	22 (7.7)
Year in training	N (%)
PGY1	68 (23.2)
PGY2	55 (18.8)
PGY3	62 (21.2)
PGY4	53 (18.1)
PGY5	55 (18.8)
Program fellows	N (%)
Yes	193 (65.6)
No	101 (34.4)
Fellowship plans
Yes	193 (65.9)
No	100 (34.1)

Two‐thirds of the respondents planned to pursue fellowship training (Table 4). From the first to second year of residency, the percentage of respondents who did not plan on pursuing a fellowship tripled, from 13.2% to 41.8% (P = .001). After that jump from first to second year, the percentage that planned to pursue fellowship remained between 57% and 65% for each PGY class.

Table 4

Plans to Pursue Fellowship by Postgraduate Year.

	No	Yes
	N (%)
PGY1	9 (13.2)	59 (86.8)
PGY2	23 (41.8)	32 (58.2)
PGY3	22 (35.5)	40 (64.5)
PGY4	22 (41.5)	31 (58.5)
PGY5	23 (42.6)	31 (57.4)

Of the responding residents, 94.4% felt that their residency provided adequate surgical training, with 3.6% unsure, and 2% feeling as though their program did not provide adequate training. Eighty percent of the respondents felt that they could complete all the key indicator cases at the conclusion of residency, with 13% unsure, and 6.3 percent believing that they could not (Table 5). The most common procedures residents did not feel comfortable completing were stapedectomy/ossiculoplasty and rhinoplasty. Of note, there was no correlation between when a resident was exposed to or became proficient in any of the key indicator cases and whether or not he or she felt comfortable performing the key indicator cases. Similarly, we did not find any correlation between when a resident was exposed to or became proficient in any of the key indicator cases and the presence of fellows in his or her program or the resident's plans to pursue a fellowship. The only difference found was when comparing for regions. There was a significant difference for the question “Have you performed bronchoscopy for foreign body removal independently (without a more experienced surgeon scrubbed)?” Respondents from the West agreed to this statement in 13.0% of cases as compared to 78.9% from the Southwest, 35.8% from the Midwest, 29.9% from the Southeast, and 22.2% from the Northeast (P < .001).

Table 5

Residents comfort with key indicator cases and plans for fellowship training.

Able to perform all key indicator cases
	N (%)
Yes	203 (80.6)
Unsure	33 (13.1)
No	16 (6.3)
Residents applying for fellowship	Able to perform all key indicator cases
	N (%)
	No	Yes
Yes	71 (80.7)	132 (80.5)
Unsure	8 (9.10)	25 (15.2)
No	9 (10.2)	7 (4.3)

Reviewing the specific procedures by year of training, as shown in Figure 2 and Supporting Appendix B, exposure to the key indicator cases begins during PGY1, with over 50% of the PGY1 respondents reporting assisting with 5 of the 15 key indicator cases, and a minority assisting with the other 10 cases. By PGY3, more than 50% of the respondent residents have been primary surgeon for all but three procedures: stapedectomy, rhinoplasty, and paramedian forehead flaps. Greater than 50% of PGY4 residents feel confident in their ability to perform all the key indicator cases independently with the exceptions of parotidectomy (47%), rhinoplasty (19%), tripod repair (40%), paramedian forehead flap (28%), and stapedectomy (11%). More than 50% of the mid‐year PGY‐5 residents reported never independently performing a stapedectomy, tympanoplasty, mastoidectomy, rhinoplasty, open reduction and internal fixation of a zygomaticomaxillary complex fracture, paramedian forehead flap, bronchoscopy with foreign body removal, or congenital neck mass excision.

Figure 2

Key indicator cases compared by postgraduate year and percentage of all respondents.

DISCUSSION

The purpose of this study was to 1) describe the current experience of residents regarding the key indicator cases and 2) ascertain if graduating residents feel that they are competent in performing these procedures by the conclusion of residency. In this study, we surveyed current otolaryngology residents to assess their operative experience in these cases. We selected the ACGME key indicator cases because they are used as a benchmark for breadth of surgical training by the ACGME. We found that by the middle of the PGY2 of residency, most residents have at least assisted with the majority of the key indicator cases. By the middle of the PGY3 year, the majority of residents have been primary surgeon for most key indicator cases. By the middle of the PGY4 year, the majority of residents feel comfortable performing most of the key indicator cases. Despite these findings, certain cases lag behind the rest, specifically stapedectomy/ossiculoplasty, rhinoplasty, and midface trauma reconstructions. Furthermore, less than half of the midyear PGY5 residents had been entrusted with the autonomy to perform the key indicator cases independently. When comparing resident demographics by the key indicator cases, the experience was fairly homogenous, with the exception of rigid bronchoscopy, which differed significantly between the western region and the southern region. About 20% of the respondents were either unsure or believed that by the time they graduated, they would not be able to perform all the key indicator cases independently. When comparing resident demographics by the key indicator cases, the experience was fairly homogenous, with the exception of rigid bronchoscopy.

The markers for competence in this study were both self‐assessed confidence and whether or not a supervising surgeon allowed the resident autonomy to perform the case independently. Unsurprisingly, self‐assessed competence outpaced independence granted by the supervising surgeon in regard to these procedures. Resident self‐assessment of competence has been shown to be fraught with error.7 In fact, poor performers are often more confident and overestimate their abilities while better performers often underestimate their capabilities.8, 9, 10, 11 Despite this inconsistency, when attending surgeons have been surveyed about why they provided residents autonomy, one of the key factors is resident confidence.12 Other important factors in providing autonomy include observed clinical skills of a resident, self‐confidence of the attending, the complexity of the case, context of the case, and the relationship between the trainee and the supervisor.12, 13 In the allotted section for comments, a few of the respondents noted that it was hospital policy that the attending must be present and scrubbed in the operating room when residents were operating. These policies may have limited the percentage of chief residents that reached our definition of surgical autonomy. This practice of increased oversight reduces lapses in quality and safety, but may undermine the trainees’ development of full responsibility.14

The tools we currently have to assess competence are limited.15 Until recently, the surgeon‐mentor had a long period of supervision and observation of a trainee. This form of mentored surgical education has been abandoned in otolaryngology training, with a few exceptions.16 Since ascertaining competence is so difficult, we often will use surrogate markers, such as case logs, procedural checklists, and summative evaluations.17, 18, 19, 20, 21, 22, 23, 24, 25 Case logs can be inaccurate, and deciding whether an individual was the assistant surgeon, resident surgeon, or supervising surgeon is open to interpretation.26, 27 For example, to meet the key indicator for ossiculoplasty and stapedectomy, a resident must complete 10 cases as primary surgeon, where they perform greater than 50% of the case in addition to the key portions of the procedure.28 The national average number of logged cases for these surgeries is 19.6 with a median of 17. While the national average is nearly double the requirement, less than 50% of the respondents felt comfortable with this procedure. As one of the responders commented, “[The] key indicators should be re‐evaluated. [Ossicular–chain reconstruction] is a good example [why the key indicator cases should be reconsidered;] we do a high volume but calling yourself the resident surgeon is a stretch in most cases.” This comment and others ultimately raise the question, if the key indicators are markers of breadth and depth of training, do we expect graduating residents to be competent in these procedures? Based on the results of this study, graduating residents may not be leaving residency feeling competent in certain key indicator procedures. We as otolaryngologists should re‐evaluate these key indicator cases as a marker of breadth of resident training, because while residents participate in these cases, some may not be treated as “resident level” cases.

Secondary to the design of this study, there are limitations to these results. In this study, we surveyed otolaryngology residents via their program coordinators. We did not ask for a receipt from the program coordinators nor did we request confirmation that the survey was sent on to the residents. As such, we have no way of knowing how many of the potentially 1,500 otolaryngology residents received the survey. Second, because the response rate was an estimated 20%, it is difficult to say if those that responded did so because they had strong emotions motivating them to respond. In this anonymous study, we asked each respondent to answer based upon what his or her experience had been in residency up until that point. We asked the residents to tell us what they had accomplished rather than when they first performed or assisted with specific cases in an effort to limit recall bias. Despite this effort, we are reliant upon each resident's memory for this study and this is subject to bias. Furthermore, due to this design, we cannot evaluate for causation of any exposure or outcome. For example, we cannot evaluate if the presence of fellows within a program led to earlier or later exposure to a particular surgery. To answer this or similar questions, a prospective assessment of otolaryngology trainees would be better suited. Future works to assess the impact on exposure and program demographics are needed.

CONCLUSION

In this survey of otolaryngology residents, we found that by the midpoint of PGY4, most felt they could perform the majority of the key indicator cases. Experience with ossiculoplasty/stapedectomy and rhinoplasty lags behind the other key indicator cases, and are the most common procedures that PGY5 residents felt they may not be able to perform independently by graduation. Twenty percent of the respondents had concerns about whether they would be able to perform all the key indicator cases at the completion of their residency. These data can be used to inform discussions about which surgeries best define competence in otolaryngology. Further investigations into this topic are needed; but using the results of this study, the academic otolaryngology community should consider re‐evaluating the role of the current key indicator cases in residency training.

Appendix A is the Redcap survey that was electronically sent to residents across the country.

Click here for additional data file.

Appendix B is a tabulated form of the raw data from the survey stratified by procedure and resident year.

Click here for additional data file.