Learning curves in ERCP during advanced endoscopy training: a Canadian multicenter prospective study.

Background and study aims  Growing emphasis on quality and patient safety has supported the shift toward competency-based medical education for advanced endoscopy trainees (AETs). In this study, we aimed to examine Canadian AETs learning curves and achievement of competence using an ERCP assessment tool with strong evidence of validity. Methods  This prospective study was conducted at five institutions across Canada from 2017-2018. Data on every fifth procedure performed by trainees were collected using the United Kingdom Joint Advisory Joint Advisory Group of Gastrointestinal Endoscopy (JAG) ERCP Direct Observation of Procedural Skills (DOPS) tool, which includes a four-point rating scale for 27 items. Cumulative sum (CUSUM) analysis was used to create learning curves for overall supervision ratings and ERCP DOPS items by plotting scores for procedures performed during training. Results  Eleven trainees who were evaluated for 261 procedures comprised our sample. The median number of evaluations by site was 49 (Interquartile range (IQR) 31-76) and by trainee was 15 (IQR 11-45). The overall cannulation rate by trainees was 82 % (241/261), and the native papilla cannulation rate was 78 % (149/191). All trainees achieved competence in the "overall supervision" domain of the ERCP DOPS by the end of their fellowship. Trainees achieved competency in all individual domains, except for tissue sampling and sphincteroplasty. Conclusions  Canadian AETs are graduating from fellowship programs with acceptable levels of competence for overall ERCP performance and for the most specific tasks. Learning curves may help identify areas of deficiency that may require supplementary training, such as tissue sampling. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Introduction

Advanced endoscopy fellowships have traditionally been taught through an apprenticeship model, wherein trainees perform procedures under the supervision of an experienced endoscopist 1 . In this model, the supervisors’ subjective assessment and procedural volume have been used as surrogates for competency. A growing emphasis on ensuring graduating trainees are ready to practice effectively has supported the shift toward competency-based medical education (CBME) for advanced endoscopic procedures. CBME allows for documentation of learner acquisition of the requisite skills needed for independent practice while offering formative feedback to the trainee based on their measured performance throughout the training period 2 3 . Accurate, timely, and meaningful assessment tools for advanced endoscopy trainees (AETs) are critical to track achievement of cognitive and technical milestones 4 .

There is growing concern over use of arbitrary procedural volume thresholds in advanced endoscopy training. Learning curve analyses have reported that the number of endoscopic retrograde cholangiopancreatography (ERCP) procedures to achieve competence in bile duct cannulation can range from 79 to more than 300 5 . These findings support the notion that uniformly applying volume thresholds neglects the variable rates at which trainees learn. Recent reports have revealed that some trainees fail to reach competency by the end of their fellowship 6 7 . Given the higher rates of potentially serious complications with ERCP compared to other endoscopic procedures 8 , efforts to systematically document trainees’ competence and demonstrate readiness for practice are needed.

Canadian advanced endoscopy training programs do not currently have uniform curricular or assessment requirements 9 . Recommendations from the American Society for Gastrointestinal Endoscopy (ASGE) suggest the use of a task-specific competence assessment tool to facilitate grading of technical and cognitive skills continuously throughout fellowship training 4 10 . Several such tools, including The endoscopic ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) and the ERCP Direct Observation of Procedural Skills (DOPS) tool, have been used to evaluate trainees in the United States and United Kingdom 11 12 . In this study, we examined Canadian AETs’ learning curves and achievement of competence using an ERCP assessment tool with strong evidence of validity.

Methods

This multicenter prospective study of AETs was conducted at five academic institutions for the 2017–2018 training cycle. All participants provided informed consent. This study received approval from the research ethics board at each participating site.

Setting

The five sites in this study were St. Michael’s Hospital in Toronto, University of British Columbia, University of Calgary, University of Alberta, and the University of Ottawa. The advanced endoscopy fellowships at these institutions are focused on ERCP and EUS. AETs eligible to participate if they were enrolled in an advanced endoscopy program and had completed at least two prior years of core gastroenterology or five years of general surgery training. Prior to the start of the fellowship, AETs completed a questionnaire on prior ERCP experience.

Assessment protocol

Trainees were assessed throughout their fellowship year. After completion of an initial 25 ERCPs to allow for trainee orientation, attending endoscopists assessed every fifth procedure and provided formative feedback to trainees. Data on the procedure, trainee, rater, and site were collected prior to each assessment.

Assessments were conducted using the United Kingdom Joint Advisory Joint Advisory Group of Gastrointestinal Endoscopy (JAG) ERCP DOPS tool, a direct observation assessment tool that grades the technical, cognitive, and integrative skills for ERCP 11 . Prior to the study period, all attending endoscopists underwent a five-hour training session during which the reviewed the ERCP DOPS items, scores, and anchors, assessed example cases, and discussed ratings among each other. Trainers were asked to complete assessments immediately after procedures and provide specific feedback based on ERCP DOPS domains.

This formative tool was developed through multidisciplinary consensus and has strong validity evidence for direct observational assessment of ERCP 11 . It contains 27 task-specific items within six domains: pre-procedure, intubation and positioning, cannulation and imaging, selected therapies, post-procedure, and non-technical skills. Assessors rated each item and the level of supervision needed and overall level of supervision on a four-point scale based. Assessors also graded the difficulty of the each procedure as grade 1 (deep cannulation of duct of interest via main papilla, biliary sampling, and biliary stent removal or exchange), grade 2 (biliary stone extraction ≤ 10 mm, treatment of bile leaks or extrahepatic strictures, and prophylactic pancreatic stent insertion), grade 3 (biliary stone extraction ≥ 10 mm, minor papilla cannulation, treatment of pancreatic strictures, removal of pancreatic duct stones ≤ 5 mm, and treatment of strictures in hilum or above), or grade 4 (removal of internally migrated pancreatic stent, removal of pancreatic stones ≥ 5 mm, removal of intrahepatic stones, and ERCP after Whipple or Roux-en-Y bariatric surgery) using an ASGE framework on procedural complexity 13 .

Outcomes

The primary outcome was overall performance on the ERCP DOPS, rated on a four-point rating scale ranging from 1 (maximal supervision), 2 (significant supervision), 3 (minimal supervision), to 4 (competent for independent practice) 12 . Maximal supervision was selected if the supervisor undertakes the majority of the tasks/decisions & delivers constant verbal prompts while competent for independent practice was selected if no supervision was required ( Supplementary file 2 ). Secondary outcomes were the performance of individual skills noted in the ERCP DOPS tool, including technical (e. g. selective cannulation, sphincterotomy, biliary stenting and tissue sampling) and non-technical (e. g. communication and teamwork, situational awareness, leadership, and judgment and decision-making) domains.

Statistical analysis

We used cumulative sum (CUSUM) analysis to create learning curves for overall supervision ratings and for individual elements of the ERCP DOPS by plotting successful outcomes of consecutive procedures performed each month over the duration of the training program 14 . CUSUM allows for continuous monitoring of a trainee’s performance and detection of deviations from predefined standards 15 16 17 . When used in training programs, it can enable and earlier recognition of deficiencies and provision of feedback to address them 18 19 20 .

For the CUSUM calculation, successful outcomes are given a score of s , and unsuccessful outcomes are given a score of 1- s . We defined competence (a successful outcome) as an ERCP DOPS rating of 3 (minimal supervision) or 4 (competent for independent practice) for ≥ 80 % of the procedures in a month, which is often the goal of ERCP training programs 21 22 . The value of s is determined by a predefined acceptable failure rate (p0), which represents the failure rate for competent practitioners, and the unacceptable failure rate ( p1, where p1 – p0 represents the maximum acceptable level of human error ), which is typically two to five fold higher than p0 14 .

We used a p0 score of 0.2, which has been used in a previous ERCP learning curve analysis 23 and is in line with the 80 % goal success rate of ERCP programs 21 22 , and a p1 score of 0.5. Decision limits were calculated based Type I (false-positive) error rate (α), and the Type II (false-negative) error rate (β) of 0.1 each 14 , and the above failure rates. Based on learning curve plots, if the CUSUM curve crosses the higher decision limit (unacceptable failure) from below, the trainee has reached the preset unacceptable failure rate. If the CUSUM curve crosses the lower decision limit (competence) from above, the trainee has achieved competence. STATA statistical software was used for analysis.

Results

Eleven trainees were invited and all participated in this study. The median number of cases trainees had performed prior to their fellowship was 50 (IQR 25–400).

Our study sample consisted of 261 ERCP procedures evaluated using the ERCP DOPS tool. The actual number of procedures performed by each trainee was higher given that every fifth procedure was evaluated. The median number of evaluations by site was 49 (Interquartile range (IQR) 31–76) and by trainee was 15 (IQR 11–45). Based on the ASGE procedural complexity framework, 42 procedures (16 %) were rated grade 1, 163 rated grade 2 (63 %), 52 rated grade 3 (20 %), and three rated grade 4 (1 %). There were 191 (73 %) native papilla cases. The overall cannulation rate by trainees was 82 % (241/261), and the native papilla cannulation rate was 78 % (149/191). Baseline characteristics are summarized in Table 1 .

Summary of procedure characteristics.

Characteristic	n (%) (N = 261)
Overall difficulty of cases
Grade 1	42 (16 %)
Grade 2	163 (63 %)
Grade 3	52 (20 %)
Grade 4	3 (1 %)
Successful cannulation by trainees	214 (82 %)
Number of cases with a native papilla	191 (73 %)
Successful cannulation of native papilla by trainees	149 (78 % of native papilla cases)

Learning curves

Using a predefined competence threshold of 80 % of the procedures in a month receiving a “minimal supervision” or “competent for independent practice” rating, all 11 (100 %) trainees reached competence for overall supervision by the end of their fellowship ( Fig. 1 ).

Fig. 1 

Cumulative sum analysis learning curve of overall level of supervision, tissue sampling, and wire management with the upper solid line signifying the unacceptable failure threshold, and the lower solid line representing the competence threshold.

Trainees reached competency in all pre-procedural, intubation and positioning, post-procedure, and endoscopic non-technical skills items by six months ( Supplementary file 1 ). Within the cannulation and imaging domain, trainees reached competency for selective cannulation at 12 months, wire management at six months, sphincterotomy at six months, stone therapy at six months, and stenting at 12 months ( Fig. 1 ). Trainees did not reach competency for tissue sampling or sphincteroplasty during their fellowship. ERCP DOPS items and the percentage rated as competent throughout the fellowship year are detailed in Table 2 .

Competence in ERCP DOPS items by 3-month intervals.

Item, %	Percentage of assessments rated as competent stratified by time
	First quarter	Second quarter	Third quarter	Fourth quarter
Overall supervision	58	83	91	96
Pre-procedure
Indication	92	93	100	100
Risk	91	93	100	100
Preparation	91	90	100	100
Equipment check	86	92	99	100
Consent	92	93	100	100
Sedation and monitoring	95	92	99	100
Intubation and positioning
Intubation of esophagus and duodenum	91	90	100	100
Visualization and position relative to ampulla	80	90	96	100
Patient comfort	92	92	99	100
Cannulation and imaging
Selective cannulation	70	73	89	95
Wire management	81	90	97	97
Image quality and interpretation	81	93	96	99
Decision about appropriate therapy	83	93	99	99
Sphincterotomy	77	89	86	93
Sphincteroplasty	54	75	91	73
Stone therapy	77	97	94	93
Tissue sampling	77	71	100	65
Stenting (metal and plastic)	79	75	97	73
Actions to minimize pancreatitis	85	88	100	100
Complications	96	95	94	100
Post-procedure
Report writing	90	92	100	100
Management plan	89	93	99	100
Endoscopic non-technical skills
Communication and teamwork	89	95	99	100
Situational awareness	88	95	96	100
Leadership	85	95	97	100
Judgment and decision making	85	95	99	100

ERCP, endoscopy retrograde cholangiopancreatography; DOPS, direct observation of procedural skills.

Discussion

We evaluated learning curves using CUSUM analysis for ERCP among AETs at five Canadian advanced endoscopy programs. Using the ERCP DOPS tool, all of the trainees achieved competency for overall procedure performance within their fellowship year. Trainees achieved competence in all non-technical domains as well, including communication, teamwork, situational awareness, leadership, and judgment and decision-making. With respect to technical skills, competence was achieved in all tasks related to cannulation and imaging except for tissue sampling.

Previous studies on ERCP learning curves using the TEESAT are concordant with our findings of trainees achieving competence in ERCP during their fellowship 6 12 . Trainees achieved biliary cannulation rates of over 80 % by the end of their fellowship in keeping with the goals of most endoscopy training programs 22 . While trainees did not meet the 85 % and 90 % thresholds suggested for practicing endoscopists by the British Society of Gastroenterology and ASGE respectively 21 22 , a report tracking AET performance suggested that they achieve > 90 % cannulation by the end of their first year of independent practice, even if they did not achieve competence during their fellowship 24 .

Our learning curve data provide important insight into development of competency for specific technical and non-technical skills. Trainees reached the competency threshold for sphincterotomy and stone therapy prior to selective cannulation, in keeping with a learning curve analysis by Ekkelenkamp and colleagues 7 . Additionally, trainees in our study achieved competency in non-technical skills early in their fellowship. Deficiencies in these skills are associated with adverse patient events 25 26 . Non-technical skills, such as situational awareness, judgment and decision-making, and teamwork help endoscopy teams (e. g. trainee, supervisor, endoscopy nurse) understand their roles, anticipate and respond to unexpected or challenging circumstances, and prevent errors through open communication 27 . While comparing our findings to other studies is difficult as most ERCP tools focus only on technical and cognitive skills 4 28 29 , our findings may reflect ongoing efforts to formalize and integrate non-technical skills training into core gastroenterology curricula 27 30 .

Importantly, of the technical skills important to practicing independently, trainees did not reach the competency threshold in tissue sampling or sphincteroplasty. ERCP brush sampling of biliary strictures to date has been considered a low-complexity task in ERCP 31 , however the few studies that have examined learning curves for this skill suggest that this is not the case. Results the United Kingdom demonstrate that ERCP tissue sampling is one of the last specific tasks where trainees achieved competency during their training 11 . Non-diagnostic sampling of biliary strictures can lead to further costly and more invasive interventions such as cholangioscopy, EUS-guided biopsy, percutaneous biopsy and unnecessary surgery 32 33 . Trainees in our study also struggled with sphincteroplasty, in keeping the United Kingdom-based study 11 , where learners did not reach competency by 300 procedures. ERCP sampling and sphincteroplasty are not highly emphasized in ERCP current training curricula despite being core skills for this procedure 10 . Advanced endoscopy training programs consider supplementing live training with simulators or animal models 34 35 36 37 . Supervisors may also benefit from offering targeted feedback and help trainees generate learning plans for these tasks. This can be enabled through the use of direct observation tools which include tissue sampling and sphincteroplasty such as the DOPS 28 29 . Additionally, training programs can create more resources to highlight evidence-based approaches to achieving competency for sphincteroplasty and tissue sampling to improve diagnostic yield 38 .

Our study has several important limitations. First, our relatively small sample size of procedures limited our ability to evaluate trainees competence based on more stringent definitions of competence and acceptable failure rates. The sample size also precluded meaningful regression analyses to identify trainee characteristics associated with success. Second, we did not account for factors which may have introduced bias, such as trainee prior ERCP experience, success rates of native vs. non-native papillae, and how cases of various difficulty were assigned to trainees by their supervisors. Third, we did not track clinical outcomes of patients included in this study and attempt to correlate them to ERCP DOPS scores. Fourth, we applied a competency threshold for cannulation to all other ERCP DOPS domains. While this is intuitively acceptable, it is not ground in any data. Fifth, this study did not include all advanced endoscopy programs in the country, limiting the generalizability of our findings. Sixth, certain advanced techniques such as precutting or double-guidewire cannulation are often not part of advanced endoscopy training and thus were not evaluated in this study. Finally, as with other studies using subjective assessment methods to evaluate learning curves, unmeasured biases, such as knowing that trainees were approaching the end of their training, may have impacted supervising endoscopists and their assessments.

Conclusions

Despite these limitations, our study shows that Canadian AETs are graduating from fellowship programs with acceptable levels of competence for overall ERCP performance and for the majority of specific intra-procedural tasks. We also add to the growing body of literature in ERCP training that supports the shift away from volume-based training and toward assessing well defined competencies using pre-established thresholds. With the understanding that trainees will acquire skills at different rates 39 , the incorporation of assessment tools to generate individual learning curves can help identify deficiencies, enable goal-directed and actionable feedback, and allow trainees to generate learning plans 40 . Learning curves may also help identify areas in which many trainees are deficient and that may require supplementary training, such as tissue sampling. In addition, aggregate data from multi-institution samples of trainees can establish competency thresholds at a national level to be used for credentialing purposes 18 41 . Meaningful assessment practices with validated tools can improve learning for trainees and help ensure they achieve the knowledge and skills needed for high-quality advanced endoscopic care.