Impact of a Multimodal Simulation-based Curriculum on Endobronchial Ultrasound Skills.

Background: Currently there is no consensus on ideal teaching method to train novice trainees in EBUS. Simulation-based procedure training allows direct observation of trainees in a controlled environment without compromising patient safety. Objective: We wanted to develop a comprehensive assessment of endobronchial ultrasound (EBUS) performance of pulmonary fellows and assess the impact of a multimodal simulation-based curriculum for EBUS-guided transbronchial needle aspiration.
Methods: Pretest assessment of 11 novice pulmonary fellows was performed using a three-part assessment tool, measuring EBUS-related knowledge, self-confidence, and procedural skills. Knowledge was assessed by 20 multiple-choice questions. Self-confidence was measured using the previously validated EBUS-Subjective Assessment Tool. Procedural skills assessment was performed on Simbionix BRONCH Express simulator and was modeled on a previously validated EBUS-Skills and Task Assessment Tool (EBUS-STAT), to create a modified EBUS-STAT based on internal faculty input via the Delphi method. After baseline testing, fellows participated in a structured multimodal curriculum, which included simulator training, small-group didactics, and interactive problem-based learning sessions, followed by individual debriefing sessions. Posttest assessment using the same three-part assessment tool was performed after 3 months, and the results were compared to study the impact of the new curriculum.
Results: The mean knowledge score improved significantly from baseline to posttest (52.7% vs. 67.7%; P = 0.002). The mean EBUS-Subjective Assessment Tool confidence scores (maximum score, 50) improved significantly from baseline to posttest (26 ± 7.6 vs. 35.2 ± 6.3 points; P < 0.001). The mean modified EBUS-STAT (maximum score, 105) improved significantly from baseline to posttest (44.8 ± 10.6 [42.7%] vs. 65.3 ± 11.4 [62.2%]; P < 0.001). There was a positive correlation (r = 0.81) between the experience of the test participants and the modified EBUS-STAT scores.
Conclusion: This study suggests a multimodal simulation-based curriculum can significantly improve EBUS-guided transbronchial needle aspiration-related knowledge, self-confidence, and procedural skills among novice pulmonary fellows. A validation study is needed to determine if skills attained via a simulator can be replicated in a clinical setting.

Endobronchial ultrasound–guided transbronchial needle aspiration (EBUS-TBNA) is recommended as the first-line procedure for the diagnosis and mediastinal staging of lung cancer by multiple medical societies. However, debate still exists on methods to effectively train and measure EBUS-TBNA performance in trainees (1–4, 6). Compared with the short and steep learning curve for conventional TBNA (7–9), the learning curve for EBUS-TBNA is longer and more complex, resulting in the need for extensive training and experience (4–6, 10). Currently, EBUS-TBNA proficiency is either judged by procedure volume or determined by direct observation, both of which vary widely between institutions worldwide (11). Volume-based certification may be arbitrary, because individuals learn at different speeds, and it is problematic for smaller institutions that serve an insufficient number of patients to meet the threshold for each trainee (12). In observation-based assessment, the lack of a structured protocol introduces potential bias of supervisors because it relies on the individual supervisor’s level of expertise and experience. A 2015 CHEST consensus statement suggested changing to a system that assesses skill acquisition and knowledge by incorporating tools like simulation (13–15). Despite this, a national survey of U.S. pulmonary fellowship program directors revealed only 30% of programs used a structured assessment strategy to evaluate EBUS competency (16).

Currently, there is no consensus on the ideal teaching method for EBUS training in novice learners. Most pulmonary fellowships currently use the traditional apprenticeship model for EBUS-TBNA training, supplemented with some secondary teaching methods (i.e., literature review, videos, and didactics). However, this training model has significant disadvantages in detecting and addressing gaps in trainees’ knowledge and skill, due to overreliance on trainees to recognize their deficiencies and the supervisor’s ability to identify those gaps and teach accordingly. During EBUS-TBNA, the patient’s safety takes precedence, resulting in the supervisor often taking over and interrupting the trainee’s experience or assessment. Trainee involvement during EBUS can also increase procedural duration, anesthesia dose, and complication rates (17, 18). Simulation-based training can reduce the learning curve that novice operators need to conduct an independent, successful EBUS-TBNA (19). EBUS simulators can also accurately discriminate between operators at different skill levels and experience and is one of several methods recommended to assess trainees and help achieve proficiency, complementing traditional apprenticeship training models (20, 21).

Based on this, we wanted to 1) develop a comprehensive assessment of EBUS performance of pulmonary fellows in training at our institution; 2) implement a new multimodal simulation-based EBUS training curriculum; and 3) test the effectiveness of this training curriculum.

Methods

Curriculum Development

For this project, we formed an internal EBUS Expert Committee (EEC) with four teaching faculty physicians in the Division of Pulmonary and Critical Care at our tertiary level hospital and training program. Members of the EEC were considered at the expert level, because each independently performed more than 200 EBUS-TBNA procedures (20). One senior pulmonary fellow designated in the Clinician-Educator track developed this study with the EEC and two simulation-center medical directors. The Clinician-Educator track fellow had received training and experience in other simulation-based assessments and debriefing before this study. Knowledge gaps among our trainees were identified through a needs-assessment survey of the teaching faculty. Based on this survey, we identified deficiencies such as anatomical identification of segmental bronchi and lymph node stations and best practices for procedural technique. We then targeted these deficiencies via a newly created EBUS curriculum to improve EBUS-TBNA–related knowledge, self-confidence, and procedural skills. This study was declared exempt by the institutional review board.

Learning Outcomes and Measures

Three learning outcomes were measured to evaluate the efficacy of the EBUS curriculum: knowledge, self-confidence, and procedural skills. Multiple-choice questions (MCQs) were used for EBUS-related knowledge assessment. Twenty-five MCQs were selected from an online pool of previously validated MCQs by expert bronchoscopists (The Essential EBUS Bronchoscopist, access at http://www.bronchoscopy.org) to address topics per the faculty’s needs-assessment survey (lymph node station anatomy, lung cancer staging, and EBUS procedure technique). To refine the MCQ tool, it was tested on a focus group of the three EEC faculty mentors and three graduating pulmonary fellows (post graduate year [PGY] VI), who were trained in the traditional apprenticeship method. Each of the three graduating pulmonary fellows performed at least 35 EBUS-TBNA procedures at the time of testing. The questions were rated by difficulty based on the focus group’s responses. The difficulty index or ρ of a test item is the proportion of a group of test-takers who responded incorrectly. For example, ρ = 90% is a very difficult question, whereas ρ = 10% is very easy. Questions that had either ρ ⩾ 75% among experts or ρ ⩽ 25% among the graduating PGY-VI class were eliminated. The resulting 20 MCQs were used for the knowledge assessment of the learners in this study.

We included subjective self-assessments as a measurable outcome because the EBUS–Subjective Assessment Tool (EBUS-SAT) (see online supplement) has been previously validated as a tool to measure the change in EBUS skills (22). It also allows trainees to provide feedback on the curriculum. In EBUS-SAT, trainees rate their ability to perform 10 different EBUS-related tasks using a 5-point Likert confidence scale.

Our objective assessment was modeled on the EBUS Skills and Tasks Assessment Tool (EBUS-STAT), a validated 10-item assessment tool developed in 2012 (23). Permission was obtained from one of the tool’s creators (Dr. Henri Colt) before modifying it for our study. Two validated objective assessment tools exist to evaluate EBUS-TBNA: the EBUS Assessment Tool and the EBUS-STAT (22, 23). We chose the latter as a template because it contained more relevant details of TBNA assessment for a novice trainee, was designed as a potential screening tool to assess fundamental skills, and was flexible to be modified per local requirements (24). Certain items in the EBUS-STAT fail to discriminate between novices and experts (24). Therefore, we developed a modified version (mEBUS-STAT) that replaced some items to include tasks measurable by the BRONCH Express simulator (Figure 1) and elements reflective of our institutional practice, while adherent to evidence-based practices of EBUS-TBNA (25). Using the Delphi survey method, the EEC reached a consensus after three survey cycles on the final modifications of the objective assessment tool. The differences between the original EBUS-STAT and the mEBUS-STAT we used are summarized in Table 1.

Figure 1.

BRONCH Express simulator dual-screen display. A = endobronchial ultrasound bronchoscopy view; B = white light bronchoscopy view.

Table 1.

Differences between the original Endobronchial Ultrasound Skills and Tasks Assessment Tool (EBUS-STAT) and the modified EBUS-STAT

Original EBUS-STAT 10-Point Assessment Tool	Modified EBUS-STAT
1. Able to maneuver the scope through the upper airway into the trachea without trauma or difficulty (5 points for single item tested)	Not included*†
2. Able to maneuver scope using white light bronchoscopy within a tracheobronchial tree without trauma (4 points, no partial points)	Not included†
3. Ultrasound image obtained without artifacts (5 points, no partial points)	Not included‡
4. Identify major mediastinal vascular structures (4 points per item)	Retained
5. Identify lymph node station (Select 3 targets, 5 points each)	Modified (trainee must identify all 10 targets, 2 points each)§
6. Able to demonstrate EBUS processor functions (2 points for each item)	Not included‖
7. Performance of EBUS-TBNA (1 point each, target 15 points)	Modified (target 20 points, 2 points each)¶
8. Image analysis: CT scans (1 point each, target 10 points)	Retained
9. Image analysis: EBUS views (1 point each, target 10 points)	Retained
10. Decision-making tasks (2 points each, target 10 points)	Retained

Definition of abbreviations: CT = computed tomography; EBUS = endobronchial ultrasound; EBUS-STAT = Endobronchial Ultrasound Skills and Tasks Assessment Tool; EBUS-TBNA = EBUS-guided transbronchial needle aspiration; EEC = EBUS Expert Committee.

Not reflective of local institutional practice (most of our EBUS-TBNA procedures are performed after endotracheal intubation).

Noted to be not sensitive in previous studies.

Difficult to measure in a simulator; easy to get clear images with minimal skill required for scope manipulation.

EEC agreed that to complete trainee assessment on one module, trainees would be expected to identify all 10 lymph node stations (station 2R, 2L, 4R, 4L, 7, 10R, 10L, 11Rs, 11Ri, and 11L).

Difficult to measure in a simulator; unable to modify gain and depth in EBUS simulator software.

EEC agreed on 20-step TBNA process.

The mEBUS-STAT has two parts: knowledge (image recognition and decision-making tasks) and technical skills assessment (lymph node identification, mediastinal vascular structure anatomy, and TBNA technique). The simulation case scenario for TBNA technique assessment was the same for all participants and met the objectives of “simple, obvious” mediastinal adenopathy (see online supplement). The TBNA assessment was limited to one case, because it accomplished our goals within a 1-hour duration. To maintain consistency of the assessment method, the Clinician-Educator fellow guided the learners through the session with a standard script (see online supplement) but otherwise did not interfere. Any verbal or manual intervention was tracked. Feedback was only given during targeted debriefing sessions to validate the trainee’s accomplishments while highlighting opportunities for improvement. Trainees also completed a questionnaire to assess demographic information such as year of training and EBUS-TBNA experience. The trainee assessment was conducted in a controlled environment in the simulation lab. All simulator case modules were developed by 3D Systems and preloaded into the bronchoscopy simulator.

Before implementation, the EEC approved the grading for the final three-part assessment (MCQs, EBUS-SAT, and mEBUS-STAT). Before this project none of the trainees had been assessed formally and were all trained under the apprenticeship model. Approximately 6 months was necessary to perform the needs assessment, develop the assessment tools, and finalize the educational interventions for this curriculum.

Participants and Procedures

Eleven pulmonary fellows (six PGY-V and five PGY-VI) were the target of this study. After their baseline assessment, each participant successfully attended a didactic lecture, problem-based learning (PBL) session, and individual debriefing session over 4 weeks (Table 2). The one-on-one debriefing practice session targeted the deficiencies in their individual EBUS-TBNA technical skills on the simulator.

Table 2.

Endobronchial ultrasound–guided transbronchial needle aspiration assessment and multimodal simulation-based curriculum

Step	EBUS Course	Allotted Time (min)
1	Knowledge assessment (20 MCQs) with trainee self-assessment (EBUS-SAT)	15
2	1:1 Objective skills assessment on the simulator (mEBUS-STAT)	45
3	Didactic lecture by faculty	60
4	PBL cases in a flipped classroom model (3 cases)	60
5	1:1 EBUS simulation debriefing	60

Definition of abbreviations: EBUS = endobronchial ultrasound; MCQ = multiple choice questions; mEBUS-STAT = modified Endobronchial Ultrasound Skills and Tasks Assessment Tool; PBL = problem-based learning; SAT = Subjective Assessment Tool.

Didactics addressed evidence-based practices related to EBUS, including indications, lung cancer staging, lymph node station anatomy, TBNA performance, and slide preparation (26). A flipped classroom model was implemented for the PBL session, because it is better for learner retention than a passive learning model (27–30). PBL sessions facilitate the learners to remain engaged during the session and take ownership of their self-learning, and faculty act as facilitators to challenge the trainees’ thinking without dictating it (31, 32). The cases for the PBL session were chosen after consulting with the EEC and based on the baseline MCQ testing of the learners and included the following topics: lung cancer staging, the role of mediastinoscopy, assessing TBNA results, and positron emission tomography–negative adenopathy (see online supplement). During the PBL session, fellows perform a literature review and discuss their answers in a brief PowerPoint presentation as a small group, with faculty moderating the sessions.

After 3 months, all 11 fellows were retested using the three-part EBUS assessment. All fellows were allowed access to the BRONCH Express simulator to practice, in addition to performing supervised EBUS-TBNA procedures on actual patients in the interval between baseline and posttest.

Equipment

The simulator used in our study was the BRONCH Express (3D Systems) that consisted of a proxy bronchoscope, a proxy EBUS biopsy needle tool, an interface that tracks equipment movements, and a monitor displaying the computer-generated endoscopic and ultrasound images (Figure 1).

Statistics

Results are reported using mean ± SD. Paired two-tailed t test analyses were conducted with the trainees’ posttest scores compared with their baseline scores. For statistical analysis, SPSS was used (version 20; IBM Corp). The significance level was defined as P < 0.05.

Results

We conducted a single cohort pretest-posttest study in July 2019 and included PGY-V (n = 6) and PGY-VI (n = 5) pulmonary fellows from a single-center tertiary care training program. At baseline pretest assessment, basic demographic data showed that 9 of the 11 participating fellows had assisted in fewer than five EBUS-TBNA procedures under the traditional apprenticeship model. The remaining two fellows were involved in fewer than 10 EBUS-TBNA procedures.

Knowledge Assessment

The mean MCQ score improved significantly from a baseline of 52.7–67.7% posttest (10.5 ± 1.4 vs. 13.5 ± 1.6; P = 0.002). To assess the effectiveness of the educational intervention for the group as a whole, the total class-averaged gain (g) was calculated (g = 31.7%). A minimum significant increase of 30% is considered for rating the intervention as effective (33).

Subjective Self-Assessment

The mean EBUS-SAT confidence scores improved from 26 ± 7.6 to 35.2 ± 6.3 points posttest (maximum score, 50; P < 0.001).

Objective Skills Assessment

The mean mEBUS-STAT score improved from 42.7% to 62.2% posttest (44.8 ± 10.6 vs. 65.3 ± 11.4; maximum score, 105; P < 0.001). The learners improved significantly in the bronchoscopic technical skills portion from 38.4% to 62.9% (30.7 ± 9.6 vs. 50.4 ± 10.9; maximum score, 80; P < 0.001). Construct validity for the mEBUS-STAT tool assessment was supported by a positive Spearman correlation between the experience of the operators taking the test (number of EBUS procedures performed or assisted) and the mEBUS-STAT scores when tested with expert members of the EEC and trainees (r = 0.81). The EEC members had a mean mEBUS-STAT of 76.5% (81.3 ± 3.2). Using the contrasting group method, a pass score of 75 was established (34). At the baseline objective assessment, 0% of learners passed, but at the posttest, 27% passed.

Discussion

Pulmonary fellowship programs commonly determine bronchoscopic competency either by an arbitrary number of supervised procedures or subjective evaluations, despite evidence that it should be evaluated in a more formalized, objective fashion (35). We modeled our EBUS curriculum based on the three-step approach to EBUS-TBNA training proposed by the European Respiratory Society: learning the necessary anatomy, simulation-based training, and supervised performance (36, 37). Simulation training alone has been shown to improve technical skills in EBUS-TBNA (24, 38). The multimodal simulation-based curriculum we used rapidly improved EBUS-TBNA technical skills and knowledge among novice pulmonary fellows. It is notable the posttest mEBUS-STAT scores of the novice learners (n = 11) after the 3-month curriculum were similar to the graduating class (PGY-VI, n = 3) who underwent 3 years of traditional apprenticeship training (62.2% vs. 64.8%; P = 0.69), despite the novice learners’ involvement in much fewer EBUS-TBNA procedures during posttest (<15 among novice trainees vs. >30 procedures among graduates). This suggests that the addition of a standardized simulation-based training may shorten the learning curve for EBUS-TBNA compared with the traditional apprenticeship training model (22, 39). Despite improvement in technical skills, only 27% of learners achieved a passing score on the mEBUS-STAT posttest. It is unclear at this time if the low passing score is from the lack of sufficient self-regulated practice and/or training on the simulator (trainees spent an average of 30 min in self-regulated training on the EBUS simulator), low procedural volume (<15 by posttest), or suboptimal teaching methods in our training curriculum.

Strengths and Limitations

Because multiple teaching approaches were used, it is difficult to discern which modality proved to be the most effective: small group didactics, flipped classrooms and PBL, or the hands-on EBUS simulation training. Currently, there is a lack of consensus on the best method for EBUS training (20–22, 40, 41). The flexibility to individualize teaching to learners’ specific skill or knowledge gaps through one-on-one interaction during the debriefing session and hands-on simulation training were strengths of our curriculum.

Several limitations may influence the interpretation of the findings in this study. First, this is an observational study with a small sample size at a single institution—a common problem among medical education research. Other studies using bronchoscopy simulators had between 6 and 16 participants (22, 42, 43). Expanding the curriculum to learners from other programs is needed to validate these tools externally. We considered a crossover cohort study design with the PGY-V class in the educational intervention group and PGY-VI as the control. However, during baseline testing, both classes reported novice level procedural experience based on procedure logs (<10 procedures) and similar scores on their baseline subjective and objective assessments. We therefore decided to provide the multimodal training program to all 11 participating fellows. A second limitation is using a single unblinded proctor for assessment because of scheduling conflicts involved in having a second proctor. The Clinician-Educator fellow was the proctor for the assessment and debriefing sessions because of the limited availability of the EEC faculty to proctor these sessions, given their clinical responsibilities. Anonymous grading was not feasible, because items on the mEBUS-STAT require direct observation. To minimize proctor bias, a scripted protocol was followed during every assessment session, with documentation of any verbal cues (see online supplement). Third, we considered that acquiescence bias or learners’ unrestricted procedural experience on real patients during the 3-month intervention period before posttest could lead to confounding results. This is unlikely, because the EBUS procedure logged by all 11 learners only increased by an additional average of five cases or fewer by the time of the posttest. The improvement in posttest scores is likely out of proportion to the experience gained by five or fewer additional EBUS cases. Fourth, using the same questions and case scenarios for baseline testing and posttest can introduce test–retest bias. We minimized this with the 3-month washout period, collecting all completed knowledge tests and not providing the MCQ answers to learners after the pretest. Fifth, there can be issues with realism and transfer with all types of simulation training. For example, lymph nodes were easier to identify on the simulator than real patients because of a more defined separation between nodes and vessels. Because additional tools such as a stylet or syringe are not available with the BRONCH Express, we supplemented the debriefing session with a real EBUS needle tool to bridge the realism gap. We were unable to assess learners during real patient cases because of scheduling conflicts and the limited availability of the proctor. Still, all faculty commented on a notable improvement in the bronchoscopic skills of the learners who went through this curriculum compared with the prior graduated classes who went through the traditional apprenticeship model. Finally, although it would be desirable for every novice operator to receive simulator training before patient exposure (44), EBUS simulators are a fragile and expensive resource, which may limit their widespread use. The Simbionix BRONCH Express costs $25,000 to purchase. In addition to equipment costs, conducting simulation training is time intensive and requires dedicated trained personnel and/or teachers. Although an assessment based solely on simulator-generated metrics does not require additional resources from busy educators, a simulator cannot replace a comprehensive curriculum to achieve all learning objectives. Simulation-based education can be an excellent tool to train novice fellows to acquire technical skills, complementing traditional apprenticeship training to improve procedural skills rapidly, but further studies are needed to test the validity of this curriculum.

Conclusions

A multimodal simulation-based EBUS curriculum can improve novice learners’ EBUS-related knowledge, self-confidence, and technical skills. A validation study is needed to determine if skills attained via a simulator can be replicated in a clinical setting.