A Multicenter Study Assessing Interventional Pulmonary Fellow Competency in Electromagnetic Navigation Bronchoscopy.

Background: Current medical society guidelines recommend a procedural number for obtaining electromagnetic navigational bronchoscopy (ENB) competency and for institutional volume for training. Objective: To assess learning curves and estimate the number of ENB procedures for interventional pulmonology (IP) fellows to reach competency.
Methods: We conducted a prospective multicenter study of IP fellows in the United States learning ENB. A tool previously validated in a similar population was used to assess IP fellows by their local faculty and two blinded independent reviewers using virtual recording of the procedure. Competency was determined by performing three consecutive procedures with a competency score on the assessment tool. Procedural time, faculty global rating scale, and periprocedural complications were also recorded.
Results: A total of 184 ENB procedures were available for review with assessment of 26 IP fellows at 16 medical centers. There was a high correlation between the two blinded independent observers (rho = 0.8776). There was substantial agreement for determination of procedural competency between the faculty assessment and blinded reviewers (kappa = 0.7074; confidence interval, 0.5667-0.8482). The number of procedures for reaching competency for ENB bronchoscopy was determined (median, 4; mean, 5; standard deviation, 3.83). There was a wide variation in the number of procedures to reach competency, ranging from 2 to 15 procedures. There were six periprocedural complications reported, four (one pneumomediastinum, three pneumothorax) of which occurred before reaching competence and two pneumothoraces after achieving competence.
Conclusion: There is a wide variation in acquiring competency for ENB among IP fellows. Virtual competency assessment has a potential role but needs further studies.

Procedural medical education has become increasingly challenging with the introduction of new technologies, procedures, and methods to be used to care for patients. Primary to the shifting landscape of procedural medical education is the question of how one achieves competency in the use of a new technology or procedure. In the 1970s, the World Health Organization described a competency-based curriculum model to deliver medical education with more recent adoption by graduate medical education, which shifted to competency-based medical education (CBME) (1). Mandatory to CBME is an assessment system consisting of trained faculty and a validated assessment measure (2, 3). Both the American Thoracic Society and CHEST recommend the use of validated assessment tools when available, which is in line with CBME (4, 5). Some examples of procedures performed by thoracic surgeons and interventional pulmonologists include rigid bronchoscopy, endobronchial ultrasound (EBUS), and electromagnetic navigational bronchoscopy (ENB). These procedures are routinely performed by these subspecialists and have become a part of expected fellow training. Current procedural training recommendations that exist for interventional pulmonology (IP) fellows have been based on expert opinion recommendations, particularly as they pertain to ENB, with an expected minimum of 20 procedures over a 12-month fellowship (6).

Development of learning curves is essential to providing trainers with learner rates of knowledge acquisition and application because previous studies have shown significant variations in learner skill acquisition (7). In this regard, efforts to quantify the learning curves that exist among learners of each unique advanced/interventional bronchoscopic modality have been undertaken (8, 9). Despite this, these data are incomplete as they pertain to peripheral lung navigation/ENB. Using a tool previously validated in a different population, we performed a multicenter prospective evaluation of IP fellows to better understand the competency learning curve(s) for ENB (10).

Methods

This study was approved by the Johns Hopkins Internal Review Board (NA00158082) and by boards at all participating study centers (see the data supplement). Oral consent was obtained from all participating IP fellows, who agreed to have up to 20 ENB procedures observed for scoring using an assessment tool by their faculty members and video recorded for independent review.

The studied cohort consisted of 2 consecutive academic classes of IP fellows at 16 academic programs in the United States. Six of the 16 medical centers participated in only 1 year of the study. All fellows had previously graduated from a full 3-year pulmonary and critical care fellowship, had performed more than 150 flexible bronchoscopy procedures, and had no prior experience with the Veran ENB system (SPiN Drive/PERC; Veran Medical Technologies). An initial survey conducted obtained IP fellow demographic information and variables related to potential procedural learning (prior procedural experience, sex, age, and hand dominance). All fellows watched a standardized introduction video on the ENB system at the start of their fellowship.

Before study initiation, the investigators were oriented to the protocol and electronic scoring of a tool previously validated in a different but similar population (7). The investigator faculty orientation included a full day of live training to review 1) study and consenting protocol; 2) training on the assessment tool, including mock scoring of cadaver-based cases; and 3) best practices for procedural instruction. The present site investigators then also trained additional faculty at their home institutions who were not present for the orientation in a similar fashion.

The assessment tool consisted of four domains: 1) procedural planning, 2) equipment setup/registration, 3) navigation to the target lesion, and 4) biopsy performance (see the data supplement). The assessment score was modified to be completed electronically (smartphone or personal computer) and divided into four domains (total score, 4–16; competency ⩾12), with each domain scored separately from 1 to 4. Competency was defined as three consecutive procedures that had an overall competency score ⩾12 and a minimum score of 3 in each domain.

The first 20 ENB cases performed by the learner were observed and scored using the assessment tool electronically on a smartphone or computer immediately after each case. To differentiate IP fellow performance from the faculty, the first attempt at the procedure was performed by the IP fellow with only verbal coaching unless there were safety concerns. Data collected included the aforementioned measure as well as a comprehensive global rating scale (GRS) of procedural competency (scale, 1–4; competent, ⩾3), and intra-/periprocedural adverse events.

An independent review of the video-recorded cases was performed by two blinded expert users of ENB procedures (>300 prior ENB cases). Only the first attempted biopsy was scored using the assessment tool by both the faculty and observers. All sites and cases were blinded and randomly reviewed, including the study site and temporal procedural order.

Statistical Methods

For demographic data, comparisons were performed using the Wilcoxon test and analysis of variance for continuous and categorical data, respectively. All categorical variables concerning competency were abridged to binary outcomes (competent or not competent) because the assessment of competency was the primary goal of this study. Fleiss’ kappa coefficient and percentage agreement were determined for the relationship of binary outcomes of competent versus noncompetent. The Spearman correlation coefficient was also determined for correlation between observer assessment raw scores. A simultaneous quantile regression was used to estimate the characteristics of the quartile of fellows’ performance to develop quartile learning curves. All statistical analyses were performed using STATA IC version 14.1 (StataCorp).

Results

We recruited 26 IP fellows from 16 medical centers in the United States over the course of 2 years (July 2018 to June 2020) to take part in the study (Table 1). Six of the 16 medical centers participated in only 1 year of the study. A total of 184 ENB procedures were available for review with assessment. Not all procedures were recorded to the 20th procedure for each fellow; however, all fellows had complete information for at least three procedures beyond their competent index procedure. The raw assessment scores correlated (rho = 0.8776) well between both independent observers. There was a high correlation between determination of competent and noncompetent procedures between the faculty compared with blinded reviewers’ assessments (Table 2), with an overall high interrater agreement (kappa = 0.7074; confidence interval, 0.5667–0.8482; standard error, 0.0713). The faculty GRS and their own assessment scoring for competent and noncompetent had a perfect correlation.

Table 1.

Baseline characteristics

Characteristics	IP Fellows (n = 26)
Age, yr, mean (SD)	33.1 (2.3)
Prior ENB experience*	6
Female sex, % (n)	23.1% (6)
Left-handed (n = 4)	15.4% (4)
Prior bronchoscopy	26 (>150)
PGY level	PGY 7 (20)
	PGY 8 (5)
	PGY 9 (1)

Definition of abbreviations: ENB = electromagnetic navigational bronchoscopy; IP = interventional pulmonology; PGY = post-graduate year.

Experience with a different ENB system.

Table 2.

Blinded reviewer agreement compared with faculty assessment

	Blinded Reviewer 1	Blinded Reviewer 2
ENB noncompetent	78.6%	75.2%
ENB competent	93.1%	98.7%

Definition of abbreviation: ENB = electromagnetic navigational bronchoscopy.

All IP fellows obtained competency in the ENB procedure studied. The number of procedures before achieving competency for IP fellows had a variable range for ENB (Figure 1) (mean, 5; median, 4; standard deviation, 3.83; range, 2–15). When evaluating learning curves, simultaneous quantile regression showed significant differences between quantile groups. When considering learning curve quartiles, all four learner quartiles show upward-sloping curves and ultimate achievement of competency (Figure 2). It should be noted, however, that the learning curve showed the first quartile achieving competency in a median of 2 (range, 1–2) procedures, whereas quartiles 2 and 3 had similar slopes with competency achieved at a median of 3 (range, 2–4) and 4 (range, 4–6) procedures, respectively. Finally, quartile 4 was shown to have a much shallower initial slope consistent with a requirement to reach a median of 9 (range, 6–15) procedures before competency. There were six periprocedural complications (3.8%) reported in the study: four (one pneumomediastinum, three pneumothorax) before reaching competency and two pneumothoraces after achieving competency. No significant differences were found in subgroup analysis, including hand dominance (P = 0.41), prior ENB experience (P = 0.66), post-graduate year level (P = 0.55), or sex (P = 0.27).

Figure 1.

Number of procedures to reach competency for individual IP fellows. IP = interventional pulmonology.

Figure 2.

Learning curve in various quartiles. Competent scores are evaluations scores ⩾12. ENB = electromagnetic navigational bronchoscopy.

Discussion

This study represents the first multicenter prospective evaluation of ENB learning and the curve associated with the development of competency in this procedure. During the study, all participating IP fellows achieved our self-defined competency level in performing ENB using the platform involved. When considering individual performance, the number of procedures needed before achieving competency was variable, with some fellows attaining competency within 2 procedures and others after more than 12. This would suggest that a single procedural value is not sufficient to capture the range of ENB procedures needed to adequately train all learners to the level of defined competence. As a matter of policy, training directors should ensure a procedural evaluation and/or err on a higher number of procedures during training to ensure competency. In our study, 15 ENB procedures would have been sufficient for all trainees to reach the competency threshold, which is slightly lower than the institutional minimum number of ENB procedures (20) to maintain accreditation of an IP fellowship. By tracking trainee performance over time, training directors may have the opportunity to identify struggling or exceptional learners, allowing them to alter their training for optimal performance. As an example, fellows who have not reached ENB competency by their fourth ENB procedure would be placed at the 50th percentile or lower when compared with their peers (7). These fellows could potentially benefit from a different instructional approach or additional training opportunities throughout their fellowship to change their course of learning. In contrast, a fellow who reaches competency early (fewer than four ENB procedures) might benefit from more challenging cases to prevent arrested development and focus on expertise development for their remaining training period.

As new, increasingly complex procedures continue to be introduced into the field of pulmonary medicine, competency evaluations need to be incorporated into training environments to assess skill acquisition, growth, and maintenance. A unique aspect of our study is the use of video replay of the procedure to further support our results and confirm competency. This should be considered in the context of a very small sample size of only two blinded video reviewers with obvious limitations. Video assessment is currently being used in European certification for EBUS and for IP certification in China (11, 12). Our study supports consistent results for competence between live and video-recorded assessments. As part of continued medical education, physicians out of training or considering new procedural credentialing where there may not be an objective observer may find remote/virtual assessment an option. Although our study showed a very strong correlation between the faculty scorer and the independent video observers, additional studies are needed to further validate these findings and this approach. For example, the video observer cannot observe direct communication between staff members and portions of the preprocedural preparation. However, these limitations could be overcome with additional technology and/or standardized protocols.

Our definition of competency (requiring three consecutive competent scoring procedures) was rather conservative. Although almost half of the fellows were able to perform competently on a single procedure followed by a noncompetent procedure on their way to eventual competency, we had no cases in which fellows performed competently on two consecutive procedures and then failed on the third. In future research, investigators may consider using two consecutive competent procedures to define overall competency. Although we followed the IP fellows for 20 consecutive cases, we observed a plateau after reaching competency without another steep slope in performance scores. This lack of a second steep slope within 20 cases suggests that mastery performance of ENB procedures likely requires significantly more experience or indicates weakness in our assessment tool. Our assessment tool has never been validated for IP fellows, only for physicians already in practice, and we did not use an expert group in this study, because it was not the intention of the tool to measure expert performance (13). This likely would need a completely different assessment tool because experts are known to skip steps in certain scenarios and need to be assessed by other experts, preferably using a GRS, which may have better construct validity and reliability over assessment instruments (14, 15).

Our study was performed in a controlled, formalized learning environment that may not be applicable outside a structured IP training program. In addition, although there are other navigational and robotic bronchoscopy systems available, this metric has not been validated for use with these other platforms. However, there are studies that support a similar learning curve for comparable guided bronchoscopy technologies (16, 17). Although the overall concept of a transbronchial needle aspiration remains the same, the different technology platforms may stress different skill sets, which may be more or less transferable from the platform we studied. Until we explore a study on the transfer of skill sets between technologies, this question will remain unanswered. Analogous to other competency assessment tools for bronchoscopy and EBUS, our study also has variation in these technologies from various manufacturers over time that was not validated on the corresponding assessment tools (18). Future studies may consider modifying our metrics to evaluate the time to competency in a similar study.

We included sex and hand dominance in the demographics to see if there was a suggestion of bias in the training of ENB procedures with various neurocognitive pathways of learning. If the data on hand dominance and/or sex were significant, they may have suggested that our current training styles or ENB design may not fit different biological learners. Ultimately, we did not find any significant difference in outcomes on any of our demographic measures, which is consistent with other studies (16, 19).

A limitation of our study was clearly the sample size, despite the study being a multicenter study. In addition, there may be biases in a research environment: Learners may be more motivated to perform better and faculty more critical when formally evaluating performance despite our countermeasures. The perfect correlation between the GRS and our metrics for competency may be partially explained by a bias in that both were completed at the same time, or the “halo” effect. However, despite this potential bias, we did not see any disagreements that would have indicated a limitation of the assessment test. Another potential limitation is the lack of correlation with procedural outcomes other than complications. Having results or accuracy of the procedure would strengthen its validation but at the same time introduce confounders such as nodule size and bronchus sign. Also, there are controversies on the method to determine accuracy for ENB bronchoscopy.

Conclusion

There is a wide variation in acquiring competency for ENB among IP fellows. Virtual competency assessment has a potential role but needs further studies.