A Simple Evaluation Tool (ET-CET) Indicates Increase of Diagnostic Skills From Small Bowel Capsule Endoscopy Training Courses: A Prospective Observational European Multicenter Study.

Small bowel capsule endoscopy (SBCE) has become a first line diagnostic tool. Several training courses with a similar format have been established in Europe; however, data on learning curve and training in SBCE remain sparse.Between 2008 and 2011, different basic SBCE training courses were organized internationally in UK (n = 2), Italy (n = 2), Germany (n = 2), Finland (n = 1), and nationally in Germany (n = 10), applying similar 8-hour curricula with 50% lectures and 50% hands-on training. The Given PillCam System was used in 12 courses, the Olympus EndoCapsule system in 5, respectively. A simple evaluation tool for capsule endoscopy training (ET-CET) was developed using 10 short SBCE videos including relevant lesions and normal or irrelevant findings. For each video, delegates were required to record a diagnosis (achievable total score from 0 to 10) and the clinical relevance (achievable total score 0 to 10). ET-CET was performed at baseline before the course and repeated, with videos in altered order, after the course.Two hundred ninety-four delegates (79.3% physicians, 16.3% nurses, 4.4% others) were included for baseline analysis, 268 completed the final evaluation. Forty percent had no previous experience in SBCE, 33% had performed 10 or less procedures. Median scores for correct diagnosis improved from 4.0 (IQR 3) to 7.0 (IQR 3) during the courses (P < 0.001, Wilcoxon), and for correct classification of relevance of the lesions from 5.0 (IQR 3) to 7.0 (IQR 3) (P < 0.001), respectively. Improvement was not dependent on experience, profession, SBCE system, or course setting. Previous experience in SBCE was associated with higher baseline scores for correct diagnosis (P < 0.001; Kruskal-Wallis). Additionally, independent nonparametric partial correlation with experience in gastroscopy (rho 0.33) and colonoscopy (rho 0.27) was observed (P < 0.001).A simple ET-CET demonstrated significant improvement of diagnostic skills on completion of formal basic SBCE courses with hands-on training, regardless of preexisting experience, profession, and course setting. Baseline scores for correct diagnoses show a plateau after interpretation of 25 SBCE before courses, supporting this number as a compromise for credentialing. Experience in flexible endoscopy may be useful before attending an SBCE course.

INTRODUCTION

Small bowel capsule endoscopy (SBCE) has become the standard investigation to establish a diagnosis in small bowel disease. Several studies have already demonstrated that experience in SBCE is an important factor for correct diagnosis.

In an Italian study, 75 consecutive video segments of capsule endoscopy were prospectively evaluated by 8 investigators.[1] Agreement with the gold standard of an external investigator was moderate for lesion detection (kappa 0.48) and for final diagnosis (kappa 0.45). Endoscopists with a higher yearly number of capsule endoscopies had a better agreement. Active bleeding and angiectasias had a better agreement than polyps, ulcers, and erosions. Size estimation of a lesion showed the poorest agreement. In a small series on 10 videos showing villous atrophy and 10 controls, investigators with experience in capsule endoscopy had a sensitivity of 70% and a specificity of 100% with a complete interobserver agreement (kappa value 1). Those with limited experience had a poor interobserver agreement (kappa 0.2).[2]

It has well been demonstrated that, in SBCE, experience of the investigator is important for attaining a reliable result.[3] For example, in an animal training model, polyp detection rates and sizing accuracy improved with endoscopic experience, but dedicated training was required for best assessing the size of the lesion.[4] It had also been observed that prior endoscopic experience enables trainees to interpret SBCE findings more accurately than medical students without experience in endoscopy. However, reliability of interpreting the detected pathology was suboptimal and warranted additional focused training.[5]

Training and continued education for SBCE are currently not standardized, differ between countries and medical centers, and definition of competency has not yet been universally established. The 2005 American Society for Gastrointestinal Endoscopy (ASGE) guidelines stated that “specific measures for competency … should be rapidly adopted in credentialing processes as they are developed,” and that “formal training during GI fellowship must include both didactic tutoring and an adequate case volume.”[6] These recommendations have recently been refreshed including 8 hours hands-on training courses followed by supervised SBCE reading in a postfellowship setting.[7] To fulfill this requirement, 1 center in the United States has recently introduced a formalized tool for assessing SBCE teaching to GI trainees within their fellowship program.[8] To date, no formal methods of assessing the impact of SBCE training courses with participants from different professional groups have been published. We developed a simple evaluation tool (ET-CET) to assess the effect of a hands-on training intervention on the SBCE diagnostic skills of a large number of participants attending formal SBCE training courses. Primarily, the results of this tool before (baseline) and after SBCE courses were used to evaluate the didactic impact. Secondly, the varying experience in SBCE of consultants, fellows, and endoscopy nurses attending these training courses was correlated to baseline level of knowledge and skill. Influence of experience in flexible endoscopy on ET-CET performance was also evaluated.

METHODS

SBCE Basic Training Courses

Between 2008 and 2011, a number of SBCE training courses were organized by different European centers, based on a similar 8-hour curriculum. The courses were designed to impart basic knowledge of SBCE, including application in clinical practice, use of equipment and software, patient care, and recognition and interpretation of small bowel pathology. All courses provided a combination of didactic lectures and practical computer based training, using a wide range of clinical cases.[9] Two delegates shared 1 computer working station to review SBCE videos using the Rapid Reader 6 software for PillCam SB and PillCam SB2® (both Given Imaging, Yoqneam, Israel) in 12 courses, and the Olympus EndoCapsule® system (Olympus, Tokyo, Japan) in 5 courses. There were between 3 and 6 experienced trainers per course, resulting in an average of one trainer per 4 to 6 delegates for close supervision. Hands-on evaluation of SBCE videos comprised about half the course time. After an introduction to the software, the delegates evaluated a mix of complete SBCE videos and case sequences to develop skills in the detection, description, and interpretation of lesions, including their clinical relevance. Interactive discussion was actively promoted to enhance learning. Clinical case presentations included background history, capsule findings and correlated enteroscopic, radiological, surgical, and histological results. Training videos were categorized as normal, bleeding, inflammation, tumors and polyps, unusual findings, variants of normal and look-alikes. The course program had been harmonized before this study by the course directors through sharing their common training experience by meeting, telephone conferences, and participation as faculty members of other courses in this group.

Delegates of the SBCE training course were predominantly physicians (eg, gastroenterology consultants or gastroenterology fellows) and to a lesser extent endoscopy nurses who were training to preread SBCE videos. The courses took place in the UK (n = 2), Italy (n = 2), Germany (n = 2), and Finland (n = 1) with an international audience, and in Germany (n = 10) for a national audience (Table 1).

TABLE 1

Characterization of Courses in Terms of Venue, Setting, System Used for Training and Number of Delegates (Percentage of Entire Cohort)

Evaluation Tool

This is a simple evaluation tool for SBCE training courses (ET-CET). Ten short videos were included in this tool. Half of the videos showed typical findings of small bowel pathology like angiectasia, polyp, tumor, diverticulum, villous atrophy. The other half comprised normal small bowel and landmarks including papilla, retrograde view of the pylorus, as well as variants of normal such as lymphatic cyst, large veins, lymphoid hyperplasia (Fig. 1). Correct description of lesions was defined in accordance with Capsule Endoscopy Standard Terminology (CEST).[10] However, accepted synonyms were also defined, considering that SBCE is not able to provide histology and that appropriate diagnoses may be provided in natural language. For example, as well as angiectasia, terms such as angiodysplasia, arterio-venous malformation (AVM) were scored, and instead of submucosal tumor, the terms neuroendocrine tumor, carcinoid, gastrointestinal stromal tumor, mass, or tumor were also accepted.

FIGURE 1

(A–J) Examples of still images extracted from test videos. Upper row: Olympus EndoCapsule—relevant findings (A, angiectasia; B, villous atrophy; C, flat adenoma); normal and variants of normal (D, focal lymphangiectasia; E, papilla of Vater). Lower row: Given PillCamSB2—relevant findings (F, Meckel diverticulum; G, Peutz-Jeghers polyp; H, submucosal tumor); normal and variants of normal (I, lymphoid hyperplasia; J, retrograde view of pylorus).

The short videos of 20 seconds had been exported as .avi or .mpeg files from selected SBCE cases using the software features of the appropriate software. During courses with the Given System, only PillCam videos were used for both training and the evaluation tool, and for Olympus courses, EndoCapsule videos, respectively. An initial feasibility testing before the beginning of the study found the number of 10 videos to be a good compromise between inclusion of a representative amount of findings and a tolerable additional time during the courses. The videos were selected from a database of anonymized SBCE studies, based on their educational value. These were from patients who, after informed consent, had undergone routine SBCE in a clinical setting for different indications. As no patients were involved in our study, approval from an ethics committee was not necessary.

Following a short introduction to the test, an anonymous pair of questionnaires matched by identical random numbers was handed to each delegate. The first sheet sought information regarding profession, as well as the extent of previous endoscopic experience in gastroscopy, colonoscopy, enteroscopy, and SBCE investigations by predefined categories.

The 10 short video files were then presented centrally by projector in the lecture room to all course delegates at the beginning and at the end of each course; both times, the videos were played twice before presenting the next video. Delegates were asked to document any lesion detected and the supposed diagnosis, and were asked for the clinical relevance of the lesions by stating if any further diagnostic or therapeutic consequences should be recommended on the basis of the findings seen (answered by ticking yes or no). The participants were unaware of the percentage of pathological and normal videos included. Directly after finishing the final video of the baseline test, the first questionnaire was collected from each delegate before the course commenced. Course sessions then began immediately to avoid any discussion between delegates on the findings. Pre- and postcourse evaluation was done with identical video cases presented in a modified random sequence. Once all postcourse questionnaires were completed and collected, the findings and correct answers were discussed with all delegates.

Delegates were stratified according to profession and categories of previous endoscopic experience. Delegates who returned only the precourse questionnaire were included in the baseline evaluation of influence of endoscopic experience on ET-CET performance. A per protocol (PP) analysis included all delegates who had provided both precourse and postcourse questionnaires.

Statistics

Scores for correct diagnosis were summed up (maximum of 10) as well as for correct classification of clinical relevance of the findings (maximum of 10) before and after the course. Scores of the ET-CET were tested for normal distribution by Shapiro–Wilk test, presented as medians with interquartile range (IQR) and analyzed by using nonparametric tests as appropriate (Mann–Whitney U test, Kruskal–Wallis H test, and Wilcoxon test). A P-value of <0.05 was considered as significant. For partial correlation, Spearman rank correlation coefficient rho was calculated. Analyses were performed with SPSS Statistics version 12.0 for Windows (IBM, Ehningen, Germany). A nonparametric regression tree analysis with Bonferroni correction of P values was applied to evaluate the influence of profession (physician/nurse) on ET-CET scores, independent of previous experience in flexible and capsule endoscopy (R software version 2.14 party package; The R Foundation for Statistical Computing, Vienna, Austria). Due to small number and heterogeneity in the subgroup of other professions this group was not compared with physicians or nurses.

RESULTS

The SBCE course evaluation tool was applied in 4 European countries during 17 SBCE training courses in 8 different locations: Finland (Helsinki), Germany (Augsburg, Berlin, Frankfurt, Hamburg), Italy (Rome), and UK (Durham, London) between 2008 and 2011 (Table 1). A total of 294 trainees attended the courses (mean 17.3 delegates/course, range 7–30). Two hundred fifteen of these used the Rapid Reader for PillCam SB (73.1%) and 79 the EndoCapsule software (26.9%). One hundred sixteen delegates (40%) had no previous experience in SBCE, 97 (33%) had performed 10 or less procedures. Between 11 and 25 previous SBCE readings were reported by 27 (9%), 26 to 50 by 24 (8%), 51 to 100 by 18 (6%), and more than 100 by 12 (4%).

There were 233 physicians attending the training courses (79.3%), 48 (16.3%) delegates were endoscopy nurses, and 13 (4.4%) had other professions, such as physiology lab assistants or technicians. Evaluation forms from 268 course participants (91.2%) were complete and were included to statistical PP analyses. Twenty-six forms were only applicable for the baseline pre course test, as the corresponding postcourse evaluation was missing.

Before starting the SBCE training course, the median number (score) of correct diagnoses for all participants was 4 (IQR 3) out of a maximum 10. Following the training, a median of 7 (IQR 3) correct diagnoses was stated (P < 0.001; Wilcoxon test) (Table 2). Classifications of findings into relevant or irrelevant were correct in a median of 5 (IQR 3) out of 10 cases pre training and in 7 (IQR 3) after the course, respectively (P < 0.001) (Table 3).

TABLE 2

Median ET-CET Scores (Interquartile Range—IQR) for Correct Diagnosis (Maximum 10 Each) Before and After the Course According to Preexisting Experience of Delegates in SBCE

TABLE 3

Median ET-CET Scores (Interquartile Range—IQR) for Correct Classification of Relevance of Findings (Maximum 10 Each) Before and After the Course According to Preexisting Experience of Delegates in SBCE

Subgroup analysis showed a significant increase in the median score of correct diagnoses after the course for physicians by 2 (IQR 2; P < 0.01, Wilcoxon test) and for nurses also by 2 (IQR 3; P < 0.01). There was also a significant increase in the number of correct classifications of relevance observed for both, physicians (1, IQR 3; P < 0.01) and nurses (1, IQR 2; P < 0.01). Increase in median number of correct diagnosis (P = 0.363) or classification of relevance of lesion (P = 0.835) was not different between physicians and endoscopy nurses.

Increase of scores for correct diagnosis was not depending on previous SBCE experience. However, those delegates with high SBCE experience started with higher median scores and finally achieved better results after the course. Subgroups are shown in Tables 2 and 3. Only for the very experienced (>100 SBCEs) this increase was not significant. Regarding classification of relevance of lesions, the increase in median scores after the course was significant in the groups with <10 SBCEs and showed only an insignificant trend in the groups with preexisting experience of >10 SBCEs (Table 3). There was no difference in improvement of correct diagnosis and lesion classification for type of capsule system, course venue, or year.

Baseline scores before the course differed significantly between less and more experienced SBCE training course delegates (n = 294): correct diagnoses were given in a median of 3 (IQR 3) of 10 cases in the nonexperienced group (no previous SBCEs), but in 7 (IQR 1) cases (P < 0.001, Kruskal–Wallis test) of the most experienced delegates (>100 SBCEs) (Fig. 2A), and correct classification for clinical relevance was given in 5 (IQR 3) versus 6 (IQR 3), respectively (P < 0.001) (Fig. 2B). Baseline scores for correct diagnosis reach the maximum value of 6 in the subgroup of delegates who had performed 11 to 25 SBCEs before the course. A plateau in the learning curve above this level might be assumed as the following subgroups with increasing experience (26–50 SBCE; 51–100 SBCEs, and >100 SBCEs, respectively), show no further improvement in baseline levels. Only the most experienced subgroup of delegates (>100 SBCEs; n = 12) had a higher baseline score of 7 (IQR 1) for correct diagnosis (Fig. 2A), while the PP analysis (n = 8) again showed a baseline of 6 (IQR 1) (Table 2). Nonparametric partial correlation eliminating the influence of experience in SBCE as control variable demonstrated preexisting experience in gastroscopy (Rho = 0.33, P < 0.001) (Fig. 3A) and in colonoscopy (Rho = 0.27, P < 0.001) as an independent factor for higher baseline scores of correct diagnosis (Fig. 3B).

FIGURE 2

(A and B). Baseline scores of all 294 delegates for correct diagnosis (A) and correct classification of relevance of lesion (B) according to previous experience in SBCE (P < 0.001, Kruskal–Wallis). Box plots—median (horizontal black line), interquartile range (IQR—box), range (thin vertical line), and outliners (dots with numbers).

FIGURE 3

(A and B) Baseline scores of all 294 delegates for correct diagnosis according to previous experience in gastroscopy (A) and colonoscopy (B) (P < 0.001, Kruskal–Wallis). Box plots—median (horizontal black line), interquartile range (IQR—box), and range (thin vertical line).

Median baseline scores of correct diagnosis were 3 (IQR 3) for nurses versus 5 (IQR 3) for physicians. Scores for correct classification of relevance of lesion were 5 (IQR 4) for nurses and 5.5 (IQR 3) for physicians. However, when creating a nonparametric regression tree with Bonferroni correction of P values (Fig. 4), only in the group of delegates with experience of >500 gastroscopies, an independent influence of profession could be observed (P < 0.001). In this subgroup, nurses had a median score of 3 (IQR 2), without an additional significant effect of capsule experience. Conversely, physicians with no capsule experience reached a score of 5 (IQR) and those with any amount of capsule experience of 6 (IQR; P < 0.001).

FIGURE 4

Nonparametric regression tree as calculated by repeated partial correlation with Bonferroni correction of P values to evaluate the influence of profession (nurse vs. physician) on baseline diagnosis scores in the ET-CET.

DISCUSSION

Formation of competence in SBCE requires an understanding of the indications, risks, and limitations of the method. SBCE training may be incorporated into a GI training program with a credentialed endoscopist reviewing the findings of the trainee. It might also be imparted in a nonfellowship setting that may include completion of a hands-on course, for example, covering a minimum of 8 hours of hands-on training and Continuing Medical Education (CME) credit, endorsed by a responsible gastrointestinal society.[7] In 2005, the ASGE Standards of Practice Committee published training goals for SBCE that included amongst others either formal training in SBCE during GI fellowship or completion of a hands-on course with a minimum of 8 hours of CME credit.[6]

Tests

To evaluate increase of diagnostic skill during accredited SBCE training courses, we developed a simple evaluation tool for SBCE training (ET-CET) and prospectively implemented it into basic SBCE training courses that were held in several European countries. Secondary aims were to evaluate the influence of experience in SBCE and flexible endoscopy on baseline test results, as well as comparing performance between physicians and nurses.

ET-CET was found to be a valuable tool independent of the SBCE software and hardware platform. Evaluation of the tool has reliably demonstrated significant increase of diagnostic skills in SBCE, independent of the previous experience and profession of the course attendant, and of the SBCE system used for the training.

Other centers have also developed training evaluation tools. The Capsule Competency Test (CapCT), developed by the Mayo Clinic, applied multiple choice questions related to indication, 8 short video clips and still images, and evaluation of a complete SBCE video including definition of landmarks, detection of lesions, and incorporating the findings into a patient management plan. Trainees were allowed to participate in the CapCT after 5 supervised SBCE readings. Participants with experience of more than 20 prior SBCE studies had significantly higher CapCT results than those with less experience. Competency was defined as a CapCT score of 90% or higher of the mean staff score.[8]

This training evaluation tool for SBCE was more complex than our ET-CET and was mostly aimed at gastrointestinal fellowship trainees. Our ET-CET is simple and can be implemented into CE training courses for physicians and nurses. As the primary aim of our study was to assess the effect of formal training courses the ET-CET was performed twice, before and after each course.

Effect of SBCE Training

Existing data on the effect of SBCE training is sparse and conflicting. In an Italian single center trial, 17 physicians reviewed 12 videos with a total of 26 clinically relevant findings as identified by 3 experts. The mean detection rate for significant findings was low with a moderate interobserver agreement with the Reference Standard. No difference in the detection rate with different levels of SBCE experience could be observed. After participation in a 10-hour expert tutorial there was no improvement of detection rate and agreement with reference standard.[11]

In contrast, a UK single center trial found a positive effect of an E-learning SBCE training course which included 30 SBCE video clips and multiple choice questions. Mean performance for 14 students was significantly different from 14 trainees and from 4 experts. After completion of the computer-based SBCE training module, mean performance increased significantly for students and for trainees.[12]

The positive results of the E-learning course correspond with our observation of an increase in mean scores for correct diagnosis and determination of clinical relevance in a large group of participants of several formal training courses, with lectures and closely supervised hands-on training. Thus formal training by attending an organized course seems useful, especially as a UK survey showed a demand for training in capsule endoscopy by 67% of gastroenterology trainees.[5]

Similarly, a recent US survey in third year graduating fellows found that the curricular need was met for SBCE for only 42%, in contrast to 100% for colonoscopy.[13]

The strength of our study is the inclusion of a large number of participants (physicians and nurses) and courses organized by several course leaders at different venues and countries using both Given and Olympus systems. Results show an improvement in scores for correct diagnoses and differentiation between relevant and irrelevant/normal findings after participation in an 8-hour formal hands-on SBCE course. Even delegates with advanced experience in SBCE improved their results during a structured course (Tables 2 and 3). Similar findings in this study for different courses, trainers, and systems suggest that a common curriculum across national borders as applied for these courses may be useful.

Threshold for Credentialing Competency

Some societies (ASGE and KSGE) have published credentialing guidelines which specify a number of supervised SBCE studies as the minimum threshold for assessing competence. During a Korean trial, 12 trainees with endoscopic but without SBCE experience interpreted 1 SBCE each week over 15 weeks. After 10 weeks, a kappa coefficient >0.5 was obtained for accordance with an expert, considered sufficient to assume competency.[14]

Establishing a threshold number of studies was beyond the remit of this study. However, a plateau in the scores for correct diagnosis above 25 SBCE studies corresponds with the results of the CapCT described by Rajan et al.[8] This group found significantly lower scores for trainees with up to 20 SBCEs when compared with expert readers, but not for the group with experience of more than 20 SBCEs (n = 7) in a single center setting. Hence, completion of at least 20 SBCE studies was demanded by the authors before assessing competency. The largest number of SBCE studies previously performed by a participant of this study was 35. However, in our multicenter study, 18 participants had already evaluated between 50 and 100 SBCEs, and 12 more than 100 SBCEs. It may be speculated that further increase of SBCE experience to an expert level might further improve the baseline scores above the assumed plateau. In the baseline analysis delegates with more than 100 SBCEs (n = 12) were the only subgroup reaching a median baseline score of 7 (Fig. 2A). However, this was not confirmed in the PP analysis (8 delegates with >100 SBCEs; Table 2). Furthermore, our study did not further differentiate the numbers of previously performed SBCEs in this subgroup with >100 SBCEs. However, Graepler et al[15] found that experts with more than 400 SBCEs tended to provide more precise size estimations suggesting an ongoing learning curve even after reading many studies. Our mean scores for the entire group of delegates after completing the SBCE courses of only 7 out of 10 possible additionally demonstrate the potential for continuing improvement. As our study was focused on basic SBCE training courses, further assessment of the learning curve up to an expert level requires future research.

Experience in Flexible Endoscopy

Data on the influence of previous experience in flexible endoscopy on performance in SBCE interpretation is limited. In a UK study, 10 gastroenterology trainees with some experience in flexible endoscopy performed better than 5 medical students considering correct determination of gastric emptying, recording of true positive findings and correct diagnosis in 10 SBCE videos.[5] In an animal model, polyp detection rates and sizing accuracy during SBCE improved with endoscopic experience, but training to improve performance in these measures for novices as well as experts was demanded for overall moderate competency.[4] In contrast, the Mayo Clinic reported no influence of previous experience in flexible endoscopy on the result of the CapCT.[8]

In our multicenter study, there was a significant nonparametric partial correlation (P < 0.001) in baseline ET-CET scores for correct diagnosis related to categories of experience in gastroscopy and colonoscopy (Fig. 3), independent from preexisting SBCE experience.

Hence, previous experience in standard endoscopy seems useful before starting a structured SBCE training course.

Endoscopy Nurses

Similar competency in SBCE has been described for physicians and endoscopy nurses trained in SBCE in more detailed comparisons. However, these reports were only based on single persons in each group.[16-22]

Application of the simple ET-CET in our study allowed inclusion of a large number of endoscopy nurses (n = 44) with varying SBCE experience. Subgroup analysis found better baseline score of correct diagnosis for physicians than for nurses only in the group with experience of >500 gastroscopies (Fig. 4). Nevertheless, both groups improved their ET-CET results similarly by attending a course. Thus, participation of endoscopy nurses in formal training courses as part of training for pre-reading of SBCE seems appropriate. In the UK, some nurses have extended reading skills to an advanced level, enabling independent image interpretation.[23] However, within other health system models outside the UK, such extended scope of practice is unlikely and, supervision of SBCE interpretation by a physician would be necessary.

SUMMARY

In summary, formal training courses in SBCE are useful in increasing diagnostic skills, even for participants with advanced experience, for physicians and nurses, and independent from course venue or SBCE system. Analysis of baseline test results according to capsule endoscopy experience suggests a minimum of 25 SBCE as a feasible compromise for assessing competency.

LIMITATIONS

This simplified test focuses only on interpretation of short video sequences without assessing theoretical knowledge or ability to detect lesions in full SBCE studies. There is no evidence how the results translate into performance in clinical practice. Furthermore, this ET-CET is rather designed for assessing basic skills than expert experience.