Flexible Bronchoscopy Simulation as a Tool to Improve Surgical Skills in Otolaryngology Residency.

OBJECTIVE: To evaluate the benefits of simulation to teach flexible bronchoscopy. STUDY
DESIGN: A prospective cohort study to assess the bronchoscopic skills of residents in an otolaryngology training program using a commercially available bronchoscopy simulator.
SETTING: Tertiary care otolaryngology residency program.
METHODS: Thirty-two otolaryngology residents and 4 expert faculty across 2 academic institutions were assessed on 3 flexible bronchoscopy tasks: diagnostic bronchoscopy, foreign body removal, and tracheal lesion biopsy. Performance was evaluated with a modified version of the validated Bronchoscopy Skills and Tasks Assessment Tool. At 1 of the 2 academic institutions, an additional tool was implemented to evaluate the simulator.
RESULTS: There was a correlation between postgraduate training year and time taken to complete tasks, including bronchoscopy, foreign body extraction, and passing through the glottis (P < .001, P = .04, and P < .01, respectively). There was a significant difference between residents and faculty laryngologists for a range of skills and tasks, including percentage of time in middle lumen, contact with bronchial walls, inadvertent esophagus entry, and biopsy of healthy tissue (P < .001, P = .003, P < .001, and P < .001). Additionally, increasing postgraduate level was correlated with a higher percentage of time in the center of the lumen and reduced time to task completion (P = .05 and P < .001). Of 32 residents, 20 evaluated the simulator on its realism, with an average score of 4.1 of 5.
CONCLUSION: The commercially available flexible bronchoscopy simulator provides a valid assessment of bronchoscopic skill and is a useful tool for practicing bronchoscopy in a safe, controlled environment. LEVEL OF EVIDENCE: Individual cohort study.

Flexible bronchoscopy is a valuable skill for the otolaryngologist in the clinical evaluation of the subglottic and tracheal airway. Resident exposure with rigid bronchoscopy is typically more extensive and frequently used for removal of foreign bodies; resident training in flexible bronchoscopy, however, is an often-overlooked component of otolaryngology education. It is difficult to instruct residents on flexible bronchoscopy because (1) it is performed in an awake patient in the outpatient setting, (2) it can be uncomfortable or poorly tolerated, and (3) patients may experience significant anxiety during the airway procedure. This anxiety may be heightened if there is the perception that a trainee is performing the procedure and being guided through it by a faculty member. Ideally a procedure in an awake patient such as this is performed as efficiently and accurately as possible, to produce the least potential discomfort.

Over the past 10 to 20 years there has been a shift in the paradigm of resident education and supervision. Duty hour restrictions have limited the amount of time that residents can be in the hospital such that they may miss being exposed to emergency procedures. The early experiences of many procedures have shifted from on-site “learning by doing” to simulations from task trainers or complex scenarios. Simulation training offers residents an opportunity to practice their skills in a safe and stress-reduced environment without the consequence of patient harm.

Procedural tasks and acquisition of technical skill require the combination of cognitive and psychomotor skills. Repeatedly performing simulation tasks has been shown to improve psychomotor skill. A growing body of evidence suggests that clinical skills acquired in medical simulation may be transferred to improved patient care outcomes. Uptake of virtual reality and simulation in laparoscopy, gynecology, ophthalmology, and emergency medicine has shown improvements in performance in specific procedural tasks and in the operating room.[4-8]

With regard to otolaryngology, virtual reality and simulations are being utilized more and more for resident education. Virtual temporal bone dissection and sinus surgery allow for safe practice, familiarization with surgery, and the acquisition of increasingly technical skills. The data are promising that these simulations improve surgical skill.[9-12] To date, the evidence of simulation in otolaryngology is limited mainly by study participant numbers and poor description of blinding, randomization, and allocation concealment procedures.

The literature on simulation, training, and operative skill is overwhelmingly positive. Simulation-based medical education has been shown to be better than traditional clinical medical education for specific clinical skills. Limitations of simulation include cost of equipment, space for equipment and simulation practice, and high investment of time by the supervisor as well as the trainee. As technology improves, so do the realism of simulators and the ability to reduce cost. Ideally a simulation tool for a training program would (1) allow for repeated assessments to document growing competence, (2) use competency-based behavioral anchors to offer interrater reliability confidence, and (3) assess all of the components of the procedure to facilitate formative feedback and summative competency evaluation. There are a number of components to procedural competency for a given procedure: knowledge of instruments used, anatomy, indications for procedure, possible complications, proper monitoring, patient safety, technical skills, and clinical judgment. At present there is the validated Bronchoscopy Skills and Tasks Assessment Tool (BSTAT), which was developed by the Pulmonary and Critical Care Medicine Division at the University of California–Irvine and designed to evaluate diagnostic bronchoscopy with the pulmonologist in mind.

The aim of this study is to show that the flexible bronchoscopy simulator (GI-BRONCH Mentor; 3D Systems) is a realistic simulation tool that can accurately measure bronchoscopy skill level and be a useful tool for resident education.

Materials and Methods

A prospective cohort of otolaryngology residents across all postgraduate training levels at 2 institutions, UCLA (University of California–Los Angeles) and Stanford University, were recruited to perform a bronchoscopy simulation. The study protocol was submitted and categorized as exempt by each institute’s Institutional Review Board.

Recruitment

The flexible bronchoscopy simulation was offered to all residents in each program regardless of training level. All simulations were supervised by a fellowship-trained laryngologist. The timing of these simulations took place over a 1-month period. The exclusion criterion for participation was prior experience with bronchoscopy simulation.

Simulation

Participants were asked to perform 3 tasks on a commercially available flexible bronchoscopy simulator system (GI-BRONCH Mentor). The simulator consists of a proxy flexible bronchoscope, robotic interface device, computer with simulation software, and monitor (Supplemental Figure S1, available online). The virtual patient responds in real time to the trainee’s instrumentation of the equipment with coughing, and the simulator provides force feedback, which enhances the realism. Vital signs are reported throughout the exercises. At the outset participants must select periprocedural medications for topical anesthesia. Each participant performed 3 tasks: flexible bronchoscopy, biopsy of a tracheal lesion, and removal of a foreign body ( ). The training group was compared with 4 expert faculty otolaryngologists according to time to task completion as well as other parameters in the modified bronchoscopy evaluation. Residents were asked to perform these tasks but were not given any other instructions. During the bronchoscopy, the airways were inspected with a flexible bronchoscope to the subsegmental bronchus level. The scope was ideally centered in the lumen to avoid wall collision and red-out (ie, when the scope is pressed against the mucosa). Orientation was difficult, and systematic inspection of the bronchial system was essential to secure a complete inspection and to reduce repeated inspection of subsegmental bronchi, thereby reducing the procedure time and the risk of wall collisions and red-out. In the biopsy of a tracheal lesion, participants were expected to read a case presentation, view several slices of a computed tomography scan, and determine the goal of the exercise. Residents were able to view this computed tomography scan prior to instrumentation of the airway and hence before they began the timed portion of the task to locate the lesion. To establish proficiency, the participants had to biopsy 2 distinct areas. The resident chose which instrument they wanted to use to perform the biopsy (alligator cup forceps, oval cup forceps, and oval cup with needle); once chosen, a generic wired instrument was placed into the working port of the bronchoscope with the assistance of the examiner. The third and final task, retrieval of an aspirated foreign body, was completed in a similar manner. Participants were asked to read the case presentation and view a plain film chest radiograph to determine the position of the foreign body. To complete the task, they were asked to complete a flexible bronchoscopy and successfully remove the foreign body. Similar to the previous scenario, options for grasping the foreign body are provided, including alligator forceps, snare, basket, and cup forceps. Once chosen, a generic wired instrument is placed through the bronchoscope working port. For all cases, the timer starts once the resident places the bronchoscope into the mouth of the simulator.

Figure 1.

Screenshots: (A) the bronchoscope virtually passes through the larynx, (B) the bronchial lesion that participants were asked to biopsy, and (C) the foreign body that participants were asked to remove.

Evaluation

To evaluate resident performance, a modified version on the validated BSTAT was used, referred to as the Flexible BSTAT (FBSTAT; ). This modified the prior evaluation tool and made it more reflective of the skills required for bronchoscopy in otolaryngology. Questions involving the numeric identification of distal bronchi and detailed nomenclature of lesions and secretions were removed. Questions relating to practical tasks were added, such as scope positioning while passing instruments, the use of the correct instrument, and the number of times that the foreign body was dropped. The results of the FBSTAT is a combined score from subjective assessment from the supervising laryngologist as well as specific objective data collected by the simulation, including percentage of time that the scope was centered in and in contact with the lumen.

Figure 2.

Flexible Bronchoscopy Skills and Tasks Assessment Tool. This rubric was used to evaluate each study participant. The listed variables represent the most important components of bronchoscopy in the otolaryngology patient.

In addition, a simulation evaluation was developed to identify the simulation task’s degree of realism. Based on a total of 5 questions and a Likert scale of 1 to 5, ranging from “not realistic at all” to “felt like the real thing,” the questionnaire focused on specific aspects of the simulation, such as instrumentation, the screen, and timing. This was developed at Stanford University and was completed only by the residents at this institution.

Statistics

The performance scores on the FBSTAT were analyzed by postgraduate year (PGY) level. The results of each resident were compared within the resident cohort as well as alongside the expert faculty. Statistics were performed on Stata/IC 13.1 (StataCorp). Pearson correlation models were generated when appropriate. Additionally, comparisons of group means were analyzed with the Student’s t test.

Results

A total of 32 otolaryngology residents were recruited to perform the bronchoscopy simulation: 12 of 24 possible residents from UCLA and 20 of 26 from Stanford. No residents were excluded from this study.

Residents were evaluated with the FBSTAT variables throughout the bronchoscopy exercises ( ). The simulator gave each resident performance feedback at the conclusion of each task. The resident group included residents with varied postgraduate experience, ranging from PGY-1 to PGY-5 ( ).

Table 1.

Postgraduate Training Level of Residents in Simulation Study.

Postgraduate year	No. of participants
1	6
2	9
3	9
4	6
5	2
Board-certified laryngologist	4

Total participants included 32 residents and 4 experts who were used as a comparison.

Comparing Experts and Residents

Results for FBSTAT performance scores for residents and faculty experts are presented in . When compared with expert faculty, residents spent a significantly lower percentage of time in the midlumen of the airway during the basic flexible bronchoscopy (P < .001) and the airway biopsy (P < .001). Residents performed basic flexible bronchoscopy in greater time than the expert faculty, 400 seconds as opposed to 127 seconds (P = .001). They also examined fewer airway segments during the airway biopsy task (P < .001) and were more likely to biopsy normal tissue (P < .001). Residents were additionally more likely to inadvertently enter the esophagus while trying to pass the scope through the larynx (P < .001). During the foreign body removal, residents spent more time in contact with the airway wall (P < .001), took longer to remove the foreign body (P = .005), and were more likely to drop it (P = .003).

Table 2.

Performance for Residents and Expert Laryngologists on Bronchoscopy Simulation Tasks.

Bronchoscopy task	Resident	Expert	P value
Bronchoscopy
Time spent in midlumen, %	24	48	<.001
Time with scope contacting wall, %	41	17	.003
Total time taken for task, s	400	127	.001
Biopsy
Time spent in midlumen, %	33	52	<.001
No. of segments examined	7	20	<.001
Inadvertent healthy tissue biopsies	1	0	<.001
Inadvertent esophagus entry	1	0	<.001
Foreign body extraction
Time to extract foreign body, s	356	174	.005
No. of times foreign body was inadvertently dropped	2	0	.003
Time with scope contacting wall, %	16	3	<.001
Time to pass through the glottis, s	54	5	<.001

Values are presented as means. Student t tests compared the performance for residents and expert laryngologists for bronchoscopy tasks, showing a significant difference across all tasks. Statistical significance set at α < 0.05.

Results Based on Postgraduate Year

There was a significant correlation between PGY and time to complete the bronchoscopy task, with higher level of training associated with shorter duration to complete the task (R2 = 0.05, P < .001; ). The higher PGY status was also associated with a higher percentage of time in the center of the lumen (P = .05). Similarly, increasing PGY level was associated with a shorter time to complete the foreign body extraction (R2 = 0.13, P = .04; ). While there was a trend for less dropping during the foreign body task, there was no statistical difference between PGY level and number of foreign body drops (P = .12; ).

Figure 3.

Procedure time is significantly related to postgraduate year (PGY; R2 = 0.36, P = .03). Senior residents performed basic bronchoscopy faster than junior residents.

Figure 4.

Time to pass the larynx is significantly related to postgraduate year (PGY; R2 = 0.41, P = .03). Senior residents were able to pass the scope through the glottis faster than junior residents.

Figure 5.

Despite a trend, there was no statistical difference between postgraduate year (PGY; R2 = 0.51, P = .12) and number of foreign body drops.

Of 32 residents, 20 evaluated the simulator on its realism (all residents from Stanford University). The average rating was 4.1 of 5. There was no correlation between PGY level and realism rating (P = .40). Of the 2 Stanford expert laryngologists, 1 evaluated the simulator for realism and rated it 3 of 5.

Discussion

Bronchoscopy remains one of the cornerstones of airway evaluation. For decades, development of bronchoscopy skills has been based on training on real patients, often compromising patient safety and comfort. The development of a realistic bronchoscopy simulator and otolaryngology-specific tasks allows for the acquisition of skill in a safe and controlled environment and without the risk of patient harm. The ability to perform an effective bronchoscopy in an awake patient requires that it be efficient, comfortable, and accurate, and this requires practice and familiarity. The availability of a realistic bronchoscopy simulator can enable this skill acquisition and prepare residents for the real thing.

This simulator allows junior residents to develop a knowledge of airway anatomy and the functions of the glottis that may be otherwise difficult to attain early in otolaryngology training. The simulation environment provides a safe low-stress learning environment. Due to the 24-hour availability of the simulation laboratory, residents are able to practice their bronchoscopy skills at their discretion. Feasibility is an important feature to account for the integration of a bronchoscopy simulator into resident education. In an unsupervised practice setting, the scheduling of practice involves only the physician in training. In such a setting, the use of the bronchoscopy simulator is flexible and feasible with regard to schedule. However, if feedback is required from a faculty or fellow, the supervised practice setting requires that physicians in training and instructors coordinate their schedules to be able to attend teaching or practice sessions at the same time. The GI-BRONCH Mentor provides real-time feedback and collects data on performance, such as contact with walls and percentage in the center of the lumen. This feedback reduces the reliance on a supervising physician.

Training on virtual reality simulators like the one used in this study allows for unsupervised, self-directed learning. The simulator provides constant feedback, objective measurements of acquired skills and the opportunity of unlimited repetition of procedural maneuvers in a changing and safe environment. The results of our study indicate that this simulator, the GI-BRONCH Mentor, is a viable and reliable tool for assessing current skill levels. This simulator provides a promising foundation that reflects real life, with expert faculty performing better than residents (as we might expect given the clearly delineated skill level) but tangible differences noted even by the resident PGY level.

It has been suggested that bronchoscopy skills acquired though simulation training transfer readily to clinical skills, and in this study the translation of skills from simulation to clinical practice remains to be demonstrated. Also to be determined, however intuitive, is whether repeated use of the simulator translates to better performance. Without a comparison of pre- and postsimulation bronchoscopic skills, we cannot comment on skill attainment for real-life bronchoscopic skills or simulation-specific skills, and this is a limitation of our study. The study would have been strengthened by assessing pre- and postsimulator bronchoscopy skill or by having >1 time point for the simulator use. A second potentially delayed bronchoscopy simulation may have been able to show change in bronchoscopy skill over time. Interesting to note is the difference in the realism rating between the resident and faculty assessment of the simulators. Admittedly it is hard to extrapolate from a single expert evaluation, but it is interesting that the expert trained in years of bronchoscopy found it less realistic than less experienced physicians (residents). Exposure to flexible bronchoscopy can be variable and relatively limited for a junior resident, with increasing hands-on experience with time. Given the relatively heavy junior resident weighting of this study (24 of 32 residents were PGY-3 and below), this does bear some significance when considering the validity of the realism assessment. A more detailed evaluation of the GI-BRONCH Mentor by numerous experts in the field should be a next step prior to any attempts in incorporating it into a resident education curriculum, particularly with our pulmonology, critical care, and anesthesia colleagues. Additional limitations include the incorporation of simulator realism evaluation at a single institution only. There was also a relatively small cohort size overall (32 residents, 4 expert faculty). We attempted to improve generalizability by making it a 2-site study.

The development of a modified otolaryngology specific evaluation of bronchoscopy skills (FBSTAT) allows for standardized objective evaluation. It provides concrete parameters for the otolaryngologist in training to improve, as well as a language with which to discuss strengths and weaknesses. While promising, the FBSTAT requires further analysis before being considered a valid evaluation tool.

Conclusion

Training on this simulator may improve the participant’s confidence in terms of handling the bronchoscope and may serve as preparation prior to patient examination. However, the performance quality of complex procedures must be obtained by repetitive practice in a real-life clinical setting. The GI-BRONCH Mentor by 3D Systems is an excellent training device. The FBSTAT serves as a useful evaluation tool for flexible bronchoscopy for the otolaryngologist, yet additional validity studies are needed.

Author Contributions

Chloe Santa Maria, carried out the simulation, data collection, data analysis and manuscript preparation; Chi-Kwang Sung, participated in the simulation, data collection, encouraged resident participation; Jennifer Y. Lee, designed the evaluation metric, contributed to study design, manuscript preparation; Dinesh K. Chhetri, simulation design, data collection, analysis and manuscript preparation; Abie H. Mendelsohn, simulation design, data collection, analysis and manuscript preparation; Karuna Dewan, simulation design, data collection, analysis and manuscript preparation.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.

Click here for additional data file.

Supplemental material, sj-jpg-1-opn-10.1177_2473974X211056530 for Flexible Bronchoscopy Simulation as a Tool to Improve Surgical Skills in Otolaryngology Residency by Chloe Santa Maria, Chi-Kwang Sung, Jennifer Y. Lee, Dinesh K. Chhetri, Abie H. Mendelsohn and Karuna Dewan in OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation