Tracking operative autonomy and performance in otolaryngology training using smartphone technology: A single institution pilot study.

BACKGROUND: In the era of duty hour restrictions, otolaryngology residents may not gain the operative experience necessary to function autonomously by the end of training. This study quantifies residents' autonomy during key indicator cases, defined by the Accreditation Council for Graduate Medical Education. STUDY
DESIGN: Prospective cohort study.
METHODS: Faculty and residents at a large academic institution were surveyed on the surgical autonomy trainees should achieve for otolaryngology key indicator surgeries at each training level. Residents and faculty used the mobile application "System for Improving and Measuring Procedural Learning" (SIMPL) between December 2017 and July 2018 to log trainees' operative autonomy during cases on a validated four-level Zwisch scale, from "show and tell" to "supervision only."
RESULTS: The study included 40 participants (23 residents and 17 attendings). The survey response rate was 83%. In surveys, residents overestimated the autonomy PGY5 residents should achieve for parotidectomy, rhinoplasty, thyroid/parathyroidectomy, and airway procedures compared with faculty (P < .05). Using SIMPL, 833 evaluations were logged of which 253 were paired evaluations for key indicator cases. Comparing survey predictions with actual cases logged in SIMPL, residents and faculty overestimated the autonomy achieved by senior trainees performing mastoidectomy (PGY5, P < .05) and ethmoidectomy (PGY4/5, P < .05); both felt that senior residents should operate with between "passive help" and "supervision only" whereas residents actually had "passive help." Residents overestimated their autonomy during rhinoplasty (PGY5, P = .017) and parotidectomy (PGY5, P = .007) while attendings accurately expected chief residents to have "passive help."
CONCLUSIONS: Resident surgical autonomy varies across otolaryngology procedures. Multicenter studies are needed to elucidate national trends. LEVEL OF EVIDENCE: 2.

INTRODUCTION

The balance between encouraging the development of resident surgical autonomy and providing appropriate supervision remains a key point of discussion for surgical educators and regulating bodies that provide oversight to resident training. Rising concern for patient safety, duty hour restrictions, and increased expectations for faculty supervision, have created a shift in surgical education from the Halstedian model of “see one, do one, teach one” to a more regulated system of supervised training and evaluation.1 With the time constraints of duty hour restrictions, the educational emphasis has shifted away from the amount of time spent in the hospital to the quality of the training experiences.2 Concurrently, rising financial pressures on hospitals may limit trainee participation as previous investigations of operative cases involving trainees in otolaryngology and other specialties suggest that resident teaching is correlated with longer operative times.3, 4, 5 Increasing operative times may also have ramifications for patient safety, particularly in the pediatric population or for patients with medical comorbidities.6

Faculty must balance the desire to optimize the care of their current patients with the need to ensure the safety of trainees' future patients by simulating autonomous practice and encouraging the acquisition of surgical competence.7 Limited data exist in the otolaryngology literature about the impact of these challenges on the acquisition of resident operative autonomy and the surgical confidence of graduating residents. There is increasing concern, however, that surgical trainees across subspecialties may not receive sufficient training to become confident and competent surgeons by graduation.8 In this study, we sought to quantitatively measure otolaryngology resident autonomy and compare this with the expectations of residents and faculty.

MATERIALS AND METHODS

This study was approved by the Massachusetts Eye and Ear Human Studies Committee. Data were collected from a single large academic otolaryngology residency program. One month prior to the initiation of the study, participants were oriented to the smartphone phone application “System for Improving and Measuring Procedural Learning” (SIMPL) at a standardized 1‐hour presentation, during which they were familiarized with the interface, the structured questions, and the use of the Zwisch scale for rating operative autonomy. The training session includes video demonstrations of different levels of autonomy in the operating room. In the field of General Surgery, research has shown that a 1 hour session is sufficient for users to reliably learn how to use the Zwisch scale, which has been highly correlated with other instruments including the Operative Performance Rating System and the Ottawa Surgical Competency Operating Room Evaluation.9

Survey administration

In the first phase of the study, participants were sent a Qualtrics survey (Provo, Utah) asking them to predict how much autonomy residents at different level of training might expect while performing 14 otolaryngology key indicator cases as determined by the Accreditation Council for Graduate Medical Education (ACGME) on the Zwisch scale (Table 1). Minimum procedure numbers for key indicator cases have been determined by the ACGME and are tracked by all residency programs to ensure adequate operative experience offered over the course of residency training.10 The full survey is in Supporting Information Appendix A.

Table 1

Zwisch scale of operative autonomy

	Attending behaviors	Resident behaviors	Autonomy scale level
Show and tell	Does majority of key portions as the surgeon Narrates the case, anatomy, skills	Opens and closes First assists, observes	1
Active help	Leads actively for >50% of critical portion of case Identifies key anatomy, optimizes the surgical field Coaches technical skills	Actively assists Practice component skills	2
Passive help	Follows resident's lead for >50% of critical portion Coaches for polish/refinement	Recognizes transition points Accomplishes next steps	3
Supervision only	Gives no unsolicited advice for >50% of critical portion Monitors progress and patient safety	Mimics independence Recovers from most errors	4

Source: Adapted from Kozin et al.25

SIMPL

In the second phase of the study, faculty and residents used SIMPL to assess resident performance in cases over 7 months. The Procedural Learning and Safety Collaborative is a nonprofit organization that aims to improve the evaluation of operative experiences.11 The culmination of their research has been the novel smartphone application SIMPL that allows residents to receive timely feedback for every case performed. Following an operative procedure, the trainee initiates the evaluation by documenting the date and the type of operation; the resident and faculty are then notified to complete same evaluation consisting of the three structured questions using the application's minimalist interface (Figure 1). Assessments expire in 72 hours as research suggests that later assessments do not accurately reflect performance.12 Participants were asked to start their evaluations as soon as possible after each case to reduce recall bias. Faculty/residents cannot see the other party's evaluation until both have been submitted.

Figure 1

Screenshot of the SIMPL interface. This is the first out of three structured questions, which asks for a rating of resident autonomy on the Zwisch scale. Clicking on the italicized “i” icon brings the user to a table describing the different levels on the scale. SIMPL, system for improving and measuring procedural learning

Three structured questions on SIMPL log a resident's operative autonomy achieved, their overall performance, and the case complexity. In this study, we focused on the question of operative autonomy, which was tracked on a four‐level Zwisch scale. The Zwisch scale has been validated to reliably describe trainee surgical autonomy.13, 14 The first level is “show and tell,” during which the attending performs the majority of the critical portions of the case as the resident observes. Higher levels on the Zwisch scale suggest increasing autonomy (“active help” and “passive help”) and at the highest level of “supervision only,” the resident safely simulates independent practice and the attending observes without offering significant input on surgical decision‐making (Table 1). The second and third structured questions of SIMPL log a resident's performance on a 5‐level scale (from “critical deficiency” to “exceptional performance”; Table 2) and case complexity on a 3‐level scale (easiest 1/3, average 1/3, hardest 1/3 of cases for a given type of procedure). Note that for cases performed at the “show and tell level,” no assessment of performance was logged. Software from the SIMPL application was modified from a General Surgery platform to focus on specific otolaryngology cases, as previously described.25

Table 2

Operative performance scale

	Performance descriptor	Performance scale level
Unprepared/critical deficiency	Resident is poorly prepared to perform this procedure	1
Inexperienced with procedure	Resident appears inexperienced in performing this procedure with frequent problems with technique, execution, forward planning	2
Intermediate performance	Resident is at an intermediate stage of development; performance of procedural elements is variable but acceptable for the amount of experience with procedure	3
Practice ready performance	Resident is ready to perform this operation safely and independently assuming resident consistently performs procedure in this manner	4
Exceptional performance	Resident performs above the level expected of graduating residents	5

Source: Adapted from Kozin et al.25

Statistical methods

Descriptive statistics were used to examine SIMPL and survey data. For survey data, Mann‐Whitney U tests were used to compare levels of autonomy expected by residents and faculty for key indicator cases. For SIMPL, primary outcomes were the average Zwisch scale ratings reported by faculty and residents. For confidentiality purposes, analysis was not performed if there were either fewer than two residents at the same training level who performed the procedure or if there were fewer than five procedures performed by residents at the same training level. Paired comparisons were between resident and faculty ratings were made with Wilcoxon signed‐rank test; where the comparison was significant, the rank correlation was calculated using the difference proportion of the two rank sums.15 Comparisons of survey predicted autonomy and actual autonomy achieved logged on SIMPL were made with Mann‐Whitney U tests. Correlations between resident autonomy and other factors (resident performance, resident level) were calculated with Spearman's rank correlation coefficient. Statistical analyses were performed using MedCalc (Ostend, Belgium) and Graphpad Prism (La Jolla, CA).

RESULTS

Study demographics

Twenty‐three residents and 17 faculty members used the SIMPL smartphone application between 1st December 2017 and 30th June 2018. Residents spanning a range of training experience (PGY1‐5) and faculty representing all major subspecialties of otolaryngology participated.

Survey on resident autonomy

The response rate for the survey was 83% (82% attendings, 83% residents). For all ACGME selected key indicator cases, participants responded that increasing autonomy should be given with higher levels of training. For all cases, residents and faculty agreed that PGY5 residents should operate at a level between “passive help” and “supervision only.” In the survey, residents overestimated the autonomy that should be achieved by PGY5 residents compared with faculty for 4 of 14 key indicator procedures: parotidectomy (P = .012), rhinoplasty (P = .0053), adult and pediatric airway procedures (P = .0063), and thyroid/parathyroidectomy (P = .025). For remaining key indicator case in the survey, residents and faculty predicted the autonomy of PGY5 residents to be at similar levels (Figure 2).

Figure 2

Expected levels of resident autonomy. A, For most key indicator cases, residents and faculty agreed on the level of autonomy expected. B, For a small number of key indicator cases (4 of 14), residents and faculty had different expectations on the level of autonomy achievable by PGY5. *P < .05, **P < .01 for comparisons at PGY5. Error bars show standard error of the mean (SEM)

Over the study period, 833 total evaluations were logged. Attending response rate to resident‐initiated evaluations was 75% across all cases. All participants logged at least one case during the study. The median number of assessments logged per resident was 14 and the median number logged per attending was 20.16 Seventy‐four percentage of residents logged at least 10 cases, and 76% of faculty logged at least 10 cases.

Case complexity

Case complexity was rated as “average” by participants in 74% (n = 620) of all cases logged. Fewer cases were rated as the easiest 1/3 of cases (8%, n = 70) or the hardest 1/3 of cases (17%, n = 143). There was no significant association between level of resident training and the complexity of cases rated by attendings (Spearman's correlation P > .05), suggesting that although senior residents were not disproportionately being exposed to more complex cases.

Autonomy

Across all 833 evaluations, autonomy was associated with level of resident training (Spearman's correlation r = 0.51, P < .0001) and resident level of performance (r = 0.47, P < .0001). Autonomy was inversely associated with level of complexity (r = −0.11 P = .0014).

Excluding unilateral SIMPL evaluations, there were 356 paired evaluations, of which 253 were for ACGME selected key indicator cases (71%). The most common key indicator procedures were ethmoidectomy (24%, n = 60), rhinoplasty (15%, n = 39), parotidectomy (14% n = 35), bronchoscopy (10%, n = 25), and mastoidectomy (6%, n = 15) (Table 3). Attendings' ratings of resident autonomy for rhinoplasty and ethmoidectomy were significantly higher compared with residents (Wilcoxon signed‐rank test P = .026 and P = .039, respectively). The rank correlation of the two comparisons were r = 0.65 and r = 0.46, respectively, indicating higher ratings from attendings. There were no statistically significant differences in autonomy ratings logged in SIMPL by faculty compared with residents for other ACGME selected key indicator cases.

Table 3

Otolaryngology key indicator cases logged in SIMPL

Key indicator cases	Paired evaluations logged (%, total n = 253)
Ethmoidectomy	60 (24%)
Rhinoplasty	39 (15%)
Parotidectomy	35 (14%)
Bronchoscopy	25 (10%)
Mastoidectomy	15 (6%)
Neck dissection	13 (5%)
Oral cavity resection	12 (5%)
Flaps and grafts	11 (4%)
Airway	11 (4%)
Tympanoplasty	10 (4%)
Stapedectomy/ossiculoplasty	9 (4%)
Congenital neck masses	7 (3%)
Thyroidectomy/parathyroidectomy	4 (2%)
Mandible/maxilla (facial trauma)	2 (1%)

Abbreviation: SIMPL, system for improving and measuring procedural learning.

Comparing survey predicted and actual autonomy

Comparing survey responses of predicted autonomy to actual autonomy logged on SIMPL, for most procedures (10 of 14, 71%), no statistically significant difference could be detected between residents or faculty's surveyed expectations and the level of autonomy achieved in the operating room. For two procedures (mastoidectomy and ethmoidectomy), residents and faculty both overpredicted the level of autonomy achieved by senior residents in the survey (Table 4, Figure 3A). For two procedures (parotidectomy and rhinoplasty), residents overpredicted the level of autonomy senior residents would achieve in the survey, while attendings' predictions were similar to actual data logged on SIMPL (Table 4, Figure 3B,C).

Table 4

Surveyed expectations for resident autonomy compared with ratings logged on SIMPL for key indicator cases (mean Zwisch levels)

Key indicator cases	Resident surveyed expectations of autonomy	Resident ratings of autonomy on SIMPL	Comparison	Attending surveyed expectations of autonomy	Attending ratings of resident autonomy on SIMPL	Comparison
Ethmoidectomy
PGY2	2.1	2.0	P > .05	1.8	2.0	P > .05
PGY4	3.4	2.4	P < .0001	3.1	2.7	P < .0001
PGY5	3.8	3.0	P = .0001	3.6	3.1	P = .024
Rhinoplasty
PGY2	1.8	1.3	P > .05	1.6	1.7	P > .05 P > .05
PGY5	3.7	3.0	P = .017	3.0	3.0
Parotidectomy
PGY3	2.5	2.0	P = .044	2.0	2.3	P > .05
PGY4	3.2	2.5	P = .009	2.8	2.4	P > .05
PGY5	3.7	2.7	P = .007	3.2	3.0	P > .05
Bronchoscopy
PGY2	2.5	2.9	P > .05	1.9	2.7	P > .05
Mastoidectomy
PGY5	3.8	3.1	P = .002	3.5	2.9	P = .029
Neck dissection
PGY3	2.3	1.6	P > .05	2.1	2.2	P > .05
Flaps and grafts
PGY2	1.9	1.7	P > .05	1.8	1.7	P > .05
Airway
PGY2	2.2	1.8	P > .05	1.6	2.0	P > .05
Stapedectomy/ossiculoplasty
PGY5	3.6	3.1	P > .05	3.3	2.4	P > .05

Abbreviation: SIMPL, system for improving and measuring procedural learning.

Figure 3

Comparing expected and actual levels of autonomy. A, For a small number of procedures such as ethmoidectomy, residents and faculty all over predicted the level of autonomy achieved by senior residents on surveys. B and C, For parotidectomy and rhinoplasty, residents overestimated the level of autonomy achieved by senior residents while attendings' surveyed expectations were more realistic. *P < .05, **P < .01 for comparisons at PGY5. Error bars show SEM

Performance

Faculty felt that PGY5 residents performed cases well and were practice ready or nearly practice ready for all cases where this analysis could be conducted (Figure 4). Residents appeared to rate themselves at slightly lower levels of performance but this comparison was significant only for mastoidectomy (Wilcoxon signed‐rank test, P < .05).

Figure 4

Performance of senior residents. Faculty felt that PGY5 residents performed cases well and were practice ready or nearly practice ready for all cases where this analysis could be conducted. Error bars show SEM

DISCUSSION

This is the first prospective study in otolaryngology to compare survey predicted and actual levels of resident surgical autonomy logged using a mobile‐based application. In this single‐center pilot study, we found that residents and faculty reported different expectations of autonomy for a number of key indicator cases in surveys, for which residents are prone to overestimate the level of independence achieved by their last year of training. Comparing predicted autonomy as reported in surveys with actual independence achieved in the operating room logged on SIMPL, residents overestimated their autonomy as senior residents for at least four of 14 key indicator cases while attendings overestimated this for at least two of these same procedures.

Although it would be ideal for residents to consistently simulate independent practice before graduation, this study shows that this might not always be achieved for complex otolaryngology surgeries. For some procedures, such as parotidectomy and rhinoplasty, residents and faculty disagreed on the level of autonomy that should be allotted to trainees, which might be due to the fact that these surgeries have higher risk complications or cosmetic implications. In these cases, it is possible that barriers to autonomy exist because of little room for surgical error.

For other procedures, an autonomy gap exists despite faculty and residents' agreement on surveys that senior residents should operate with supervision only, such as for ethmoidectomy and mastoidectomy. Further inquiry into why this autonomy gap exists is necessary to determine if it is the expectations or the actual practice that should be addressed. One factor this current study could not adequately control was the natural stratification of cases by resident level that leaves senior residents with a disproportionate number of difficult cases. Although there was no significant correlation between the complexity of cases assigned and postgraduate year level, this would be difficult to discern because attendings rated the majority of cases as “average complexity” for the given type of procedure. Additionally, it is interesting to note that, although both ethmoidectomy and mastoidectomy procedures utilize microscopes or endoscopes that allow attendings to watch residents perform the cases, the field of vision is limited and attendings have no tactile feedback, making such cases particularly difficult to teach.17 Therefore, perhaps these procedures are inherently less likely to be “supervision only” as attendings may feel the need to periodically take over the instruments and inspect residents' work. In this study, cases that use endoscopes or microscopes accounted for 31% (n = 256) of total assessments including 19% (n = 159) that were ethmoidectomies or mastoidectomies. Future survey or qualitative studies should investigate why these types of procedures are less likely to result in high resident operative autonomy.

Outside of these explanations, there are a variety of case‐specific factors that affect how residents and faculty negotiate autonomy in the operating room. In this study, performance and training level were correlated with autonomy but other factors such as supervising surgeons' experience teaching specific procedures, prior operative interactions between attendings and residents, as well as residents' personal attributes are also potential contributing factors.18 Case details such as patient comorbidities or patient preference may also limit resident participation. Resident participation may also increase operative time and hospital costs,3 and attending surgeons may also feel varying degrees of personal responsibility to their patients who expect their surgeons to participate in their procedure.19

Although patient safety is of paramount importance, it is desirable to maximize residents' operative experiences to train the next generation of surgeons. This current study begins to characterize the autonomy gap in otolaryngology and offers vocabulary through the Zwisch scale with which residents and faculty can negotiate participation in the operating room. The act of logging resident experiences with an emphasis on tracking autonomy and performance may cause faculty and residents to be more mindful of educational opportunities in the operating room. In future studies, a more detailed understanding of the factors that unduly limit resident learning may further improve the quality of surgical training.

It is beyond the scope of this study to determine whether the level of autonomy senior residents logged correlates with readiness for independent practice. Notably, although residents in this study did not always operate with “supervision only” by the end of training, the performance ratings given to PGY5 residents are reassuring, as faculty found them consistently “practice ready.” Although the current study sample was powered to study only a handful of ACGME selected key indicator cases, the data suggest residents perform a variety of otolaryngology procedures well by the end of training. The further question of how autonomy as a resident relates to performance as a junior attending will be critical. Specific information about graduates' abilities to confidently perform general otolaryngology procedures during early years of independent practice has not been examined. Tracking experiences and outcomes in early years of independent practice may be necessary to understand the impact of autonomy during training.

While this is the first study in otolaryngology to track resident surgical autonomy using a mobile application, its findings are similar to those in general surgery. A study of general surgery residents at a single residency program found that autonomy logged using SIMPL was significantly less than surveyed expectations for chief residents for many core procedures (eg, laparoscopic appendectomy/cholecystectomy/partial colectomy, hemorrhoidectomy, open inguinal hernia repair).20 A multicenter trial of 14 general surgery programs describing the progression of trainee operative autonomy during training found that residents in their final 6 months of training reached meaningful autonomy in “core procedures” defined by the American Board of Surgery in 77.4% of cases.21 These authors concluded that residents are not universally ready to independently perform some common procedures by graduation. However, it is currently unclear how much autonomy is actually necessary for residents to achieve competency, and how many observations are needed to make generalizable assessments about the degree of autonomy attained.21, 22 A follow‐up study examining cases logged using SIMPL at 14 general surgery programs found that the factors that correlated with autonomy were (in order of the strength of the predictor) quality of a resident's operative performance during the case, typical autonomy granted by the supervising surgeon, case difficulty, and trainee experience level.23

Study limitations

The primary limitation of this study is the single institution design that reflects the opinions and experiences of a limited number of residents and faculty. We were unable to obtain the quantity of evaluations needed to perform analysis on all the types of procedures that otolaryngology residents perform beyond common key indicator cases. Moreover, as a single center study, generalizations to other residency programs may or may not be valid as the case mix, surgical staff, and teaching ethos may differ at programs across the country.

The limited sample size in this study made it difficult to track and analyze autonomy at all different levels of residency for all key indicator surgeries. For example, at the study institution, key indicator otologic cases are performed primarily by PGY4 and five residents on their otology rotations. Small numbers of certain procedures logged (eg, facial trauma cases, thyroid/parathyroidectomy, and congenital neck masses) precluded statistical analyses for those cases. Due to the limited sample size, we are also not able to stratify the data to analyze other questions such as whether PGY5 residents are able to achieve autonomy at the end of their rotations or at the very end of residency. Stratifying such data from future multicenter trials may also reveal how a multitude of other factors affects surgical autonomy.

Additionally, the data logged could be confounded by recall biases. It can be difficult for subjects to separate individual components of complex cases from the larger surgery performed; for example, a senior resident who performs an ethmoidectomy during an operation with a difficult frontal sinus dissection may evaluate their autonomy and performance based on the memory of the more difficult portion of the case rather than solely on the ethmoidectomy. This is a limitation inherent to research dependent on participants' evaluations of each other rather than on objective third party assessments.

Furthermore, although inter‐rater reliability has been studied for SIMPL in the field of general surgery,9 this has not been assessed for otolaryngology faculty. Future research will determine if similar findings of inter‐rater reliability are noted in otolaryngology. Videos used for frame‐of‐reference training at the start of this study also used general surgery procedures; future studies should develop training materials specific to otolaryngology.

In the survey study, a possible limitation is question fatigue causing respondents to potentially respond to each procedure with the same level of autonomy for a given PGY level. However, looking at individual survey responses, only one participant out of 17 gave “straight‐lined” answers for each PGY year while the remaining subjects varied responses for different procedures at each training level.

Last, this study has not explored the ways in which SIMPL could be used to complement other tools useful for the assessment of surgical trainees. These tools include Objective Structured Assessment of Technical Skills (OSATS) and other modalities of evaluations including ACGME Milestones surveys,24 MedHub, or New Innovations. Ideally, programs would use SIMPL to inform other evaluations and allow programs to deliver both granular and general feedback to residents. Future studies using alternative assessment tools and standardized independent observers may be valuable in providing validity evidence to support the use of SIMPL.

CONCLUSIONS

This single‐center pilot study finds that otolaryngology residents and faculty do not always agree on the level of surgical autonomy that residents should expect to achieve during training. Additionally, the degree of actual surgical autonomy granted to trainees varies depending on the procedure performed. Multicenter studies will be critical to explore the factors that impact operative autonomy during residency, understand the degree of autonomy necessary to ensure competency by graduation, and determine if simulated autonomy during training translates into improved outcomes during the early years of independent practice.

DISCLOSURES

Dr Brian George and Dr Jordan Bohnen are unpaid, volunteer members of the Procedural Learning and Safety Collaborative, a nonprofit surgical education consortium based in Boston, MA. This work was completed in collaboration with the Procedural Learning and Safety Collaborative.

Appendix A Full survey

Click here for additional data file.