Development and validation of metric-based-training to proficiency simulation curriculum for upper gastrointestinal endoscopy using a novel assessment checklist.

BACKGROUND/AIMS: : This study aimed to design a structured simulation training curriculum for upper endoscopy and validate a new assessment checklist.
MATERIALS AND METHODS: A proficiency-based progression stepwise curriculum was developed consisting of didactic, technical and non-technical components using a virtual reality simulator (VRS). It focused on: scope navigation, anatomical landmarks identification, mucosal inspection, retro-flexion, pathology identification, and targeting biopsy. A total of 5 experienced and 10 novice endoscopists were recruited. All participants performed each of the selected modules twice, and mean and median performance were compared between the two groups. Novices pre-set level of proficiency was set as 2 standard deviations below the mean of experts. Performance was assessed using multiple-choice questions for knowledge, while validated simulator parameters incorporated into a novel checklist; Simulation Endoscopic Skill Assessment Score (SESAS) were used for technical skills.
RESULTS: : The following VRS outcome measures have shown expert vs novice baseline discriminative ability: total procedure time, number of attempts for esophageal intubation and time in red-out. All novice trainees achieved the preset level of proficiency by the end of training. There were no statistically significant differences between experts' and trainees' rate of complications, landmarks identification and patient discomfort. SESAS checklist showed high degree of agreement with the VRS metrices (kappa = 0.83) and the previously validated direct observation of procedural skills tool (kappa = 0.90).
CONCLUSION: : The Fundamentals of Gastrointestinal Endoscopy simulation training curriculum and its SESAS global assessment tool have been primarily validated and can serve as a valuable addition to the gastroenterology fellowship programs. Follow up study of trainee performance in workplaces is recommended for consequences validation.

INTRODUCTION

The traditional apprenticeship model of education in endoscopy has relied mainly on learning through observation and performing under supervision. This approach of training has depended for many years on the duration of training and volume of procedures (threshold numbers) as metrics/surrogates for accomplishment of clinical expertise.[1]

Simulation-based training provides the opportunity of learning in a safe, low risk environment and as such allows the trainees to learn from their own mistakes.[23] Simulation in endoscopy has shown reasonable accepted face validity, however evidences for construct validity were reported in many studies as good performance measures related mainly to procedural time.[4]

Evidences exist from multiple systematic reviews that simulation-based endoscopy (SBE) training (e.g., virtual reality (VR)) can be educationally effective in preparation of novices in diagnostic esophagogastroduodenoscopy, colonoscopy and/or sigmoidoscopy before the conventional patient-based endoscopy training.[45678] Results show that SBE training accelerates the learning curve of the novice trainees, improves performance and it has proved to be transferable to clinical practice.[91011]

However, King et al.[7] concluded in their systematic review that “the optimal SBE training program has yet to be developed”, a conclusion that agrees with that of a Cochrane review[5] as well.

Many efforts reported from different international scientific societies have focused on either the development of the cognitive component of the training curriculum (e.g., Fundamentals of Laparoscopic Surgery (FLS) or the development of evaluation tools for the assessment of gastrointestinal (GI) endoscopy technical and non-technical competencies (e.g., American Society for Gastrointestinal Endoscopy (ASGE)).[12]

Up to date, we are aware of only few standardized validated curricula for SBE endoscopic training. In response to the need for a standardized approach for GI endoscopy simulation training in the Saudi gastroenterology fellowship programs, a task force of subject-matter experts was formed to develop a comprehensive simulation-based curriculum for the “fundamentals of GI endoscopy” (FGE). This newly developed curriculum was piloted as a mandatory fundamental GI course conducted a week ahead of the start of the national gastroenterology fellowship program and prior to any patient-based GI endoscopy encounter. The aims of this manuscript are to: 1. Describe the developed curriculum in terms of competencies, general and specific objectives, educational and learner assessment methods; 2. Provide construct validation evidence of the developed curriculum tasks, with identification of which VR simulator (VRS) metrics are capable of discriminating between novices and experts and; 3. Validate the new innovative assessment endoscopic skill checklist developed by the task force group.

MATERIALS AND METHODS

Study design

Phase I: Development of the course curriculum

A task force team was formed in December 2016, which consisted of national experts who were involved in previous pilot GI simulation courses (MA, NA, FB, AH), experienced endoscopists (FS, TT, AJ,), and simulation-based education experts (NK, YA, RS). A comprehensive metric-based-training-to-proficiency curriculum was designed by the working group using Khamis et al. (2016) model.[13] The model and its worksheet were developed by integrating the criteria of educational effectiveness of simulation into the steps of curriculum development. It addresses the three competency domains of knowledge, technical and non-technical skills.

The main aim of the FGE course is to introduce the learners to the fundamental principles of upper GI endoscopy. The competencies, general objectives, outcome measures, and common errors that should be avoided are shown in Tables 2 and 3. These were developed through serial meetings by the task force team. The team has used a modified Delphi consensus process[14] in the duration between December 2016 till January 2018 to establish the content validity of the developed course. The Delphi panel included a group of 11 multinational experts; eight endoscopists (MM, NA, FB, AH, FS, TA, AJ, PI) and three simulation education experts (NK, YA, RS). To assure content validation, task deconstruction and review of the literature were used to develop the initial course content in a face-to-face meeting. Then two rounds of email voting were used to reach final consensus.

Table 2

Competencies, General objectives, and Outcome Measures of the Developed FGE Curriculum

Competencies and Goals
1. Knowledge*
General Objectives:
Understand and apply the basic fundamentals of GI endoscopy.
Understand indications and contraindications for the basic endoscopic procedures
Describe appropriate patient preparation
Recognize and know how to manage common complications related to basic endoscopic procedures
Identify and discuss available diagnostic alternatives
2. Cognitive and Perceptual Skills**
General Objective:
Apply the basic fundamentals of upper GI endoscopy with appropriate clinical judgment and intra-procedure decision making
Outcome Measures and Metrics
I. Patient selection
Assess the indications risks and contraindications for appropriate patient selection for the procedure
II. Pathology identification and interpretation
-Recognize abnormal findings and correctly interpret them
-Use landmarks to identify the specific location of the abnormal finding
-Independently identify correct therapeutic tool and settings appropriate for the pathology encountered.
III. Management of patient discomfort during the procedure
-Assess continuously and select the appropriate management for patient discomfort during the procedure
IV. Diagnostic and therapeutic decision making and problem solving:
-Integrate relevant sensory cues into an accurate perception of current situation (situation awareness) for:
Navigation
Anatomical constraints
Optimal air insufflations
Mucosal visualization
Identification of mucosal abnormalities
Correct equipment functioning
-Use clinical reasoning to take procedural decisions and solve problems that arise during the procedure.
-Integrate patient clinical information and endoscopy findings into a management plan
3. Technical Skills
General Objectives:
-Demonstrate improved motor skills and dexterity that form the fundamental basis for the performance of upper GI endoscopy.
Outcome Measures and Metrics
Prepare for an aseptic technique (wash hands, wear gloves, ....).
Assure that all required equipment is available before starting the procedure.
Demonstrate the process of endoscope navigation, tip deflection and torque
Demonstrate the technique for retroflexion
Demonstrate the ability to adequately evaluate the mucosa and the skill to target lesions endoscopically
4. Other Non-technical Skills (in addition to the cognitive & perceptual Skills)
General Objectives:
I. Communicate effectively with and demonstrate ethical behavior towards patients
II. Adopt professional behavior and demonstrate continuous self-development skills.
Outcome Measures and Metrics
I. Communication and ethical behaviour with the Patient
Introduce self to the patient
Adopt ethical behavior when dealing with the patient
Ensure the correct patient by checking name, date of birth against arm band and notes
Explain the indications, procedure and potential complications to the patient/his relatives to obtain informed consent
Involve the patient in the decision-making process
Provide patient education regarding upper GI endoscopy
II. Professionalism and Continuous Self-development
Make decisions as based on practice guidelines
Demonstrate responsibility and accountability for own performance and decisions.
Recognize the limits of knowledge and skills and identifies when to ask for senior assistance
Reflect accurately on self-knowledge, skills and attitudes post procedure.
Evaluate the training session and facilitator performance.
Develop with the facilitator a plan for improvement of performance.

*Adopted from SAGES FES didactic course (www.fesprogram.org/fes-didactic/) with little adaptation, **Derived and adapted from ASGE Assessment of competency in endoscopy the work of Sedlack et al.[11] and Zupanc et al.[12]

Table 3

List of the Common Errors for Gastrointestinal Endoscopy Reached by a Consensus of the Subject Matter Experts Designing the Course

Not paying attention to room setup, tools and instruments needed

Not properly assessing the patient's anesthesia risk (airway, ASA score, co- morbidities)

Inappropriate selection of sedations or medications according to patient condition

Not identifying the patient in need for prophylactic antibiotics.

Poor monitoring for patient's vital signs and oxygenation (pre-, during and post-procedure)

obtaining consent without proper explanation for the patient (procedure should be explained, risk and complications clarified, and the alternative mentioned).

Not paying attention to insert the endoscope under direct visualization and to navigate the endoscopy tip upwards around the base of the tongue to place it at the level of the glottis. This will avoid:

Tracheal intubation and injury

Trauma to hard palate

Injury and perforation of upper esophagus/or piriform fossa

Inability to identify the anatomical landmarks (Z line, pylorus and duodenal bulb) due to inability to maintain luminal view and direction.

Pushing the scope against resistance or without seeing the lumen

Inducing of severe pain and discomfort

Not willing to quit or ask for help

Ineffective use of air, water and suction resulting in luminal poor visualization and its sequelae. Inability to reach the 2nd part of the duodenum due to ineffective right-hand maneuver.

Insufficient tissue sampling (proper targeting of lesion and taking at least 4 samples of each lesion).

The outcome measures and content of the didactic part of the cognitive module of the curriculum were adapted from the validated Fundamentals of Endoscopic Surgery (FES) didactic online course (http://www.fesprogram.org/fes-didactic/).

Simulator

The AccuTouch® Endoscopy Simulator (Immersion Corp., CA, and USA) was used by the task force team for the administration of the technical skill part of the fundamentals of endoscopy training.

Training

The cognitive module was delivered through a series of interactive lectures, videos, and workshops that were developed and validated by the curriculum development team. This module introduces the trainees to the world of endoscopy, starting with lectures addressing the indications/contraindications of endoscopy, consent taking, levels of sedation, and how to stratify patients prior to sedation administration. These lectures were followed by a video on disinfection of the scopes then an interactive workshop on scope “parts, tools, and accessories”. This module prepared the trainees to be able to plan the endoscopy procedure in a stepwise fashion that mimics real life settings.

Phase II: Construct validation of the modules and setting proficiency benchmarks

A total of 15 subjects (5 experts and 10 post-graduate year-4 internal medicine residents; PGY4) were recruited for the study. An experienced endoscopist was defined as having performed more than 500 upper GI endoscopies and 200 colonoscopies. The 10 novice trainees (PGY4 residents) represented a subset of the residents who were accepted for a GI fellowship program in Saudi Arabia for the year 2018 and reported no previous endoscopy performance or completion of endoscopy training/rotation. Participants were also excluded if they previously attended upper endoscopy SBE training.

In December 2017, each of the 5 expert endoscopists completed two rounds on the simulator performing an upper GI endoscopy for a case of dyspepsia. The results of the simulator-generated 12 outcome measures and metrics of the procedure were recorded. These 12 outcome measures are: (1) Time: total procedure time, time to upper esophageal sphincter intubation, time to pylorus intubation, (2) Identification of anatomical landmarks: gastro-esophageal-junction, gastric fundus, angularis-incisura, major papilla and 3rd part of the duodenum, (3) Esophageal intubation: esophagus intubated with/without patient swallowing and number of attempts to intubate the esophagus, (4) Total amount of air inflated: total amount of air and air left, (5) Others: minimum depth of scope insertion, time in red-out and maximum force exerted by endoscope, (6) Pathology identification: visualization, location and lesion type, (7) Patient discomfort: the percentage of patient non-discomfort throughout the procedure, amount of time of mild and moderate discomfort, and (8) Complications: perforation, tracheal intubation and laryngeal edema. The level of proficiency required to pass each of the tasks of the VR simulator exercise was determined as two standard deviations (SDs) below the mean of the scores of the experts in two consecutive repetitions of the tasks.

From 20th to 25th of February 2018, the 10 PGY4 residents with no previous experience in endoscopy participated in the mandatory fundamentals of endoscopy course. As described before, they all received an orientation of the VR simulator followed by completion of the cognitive module. All trainees were required to pass a post-cognitive module multiple choice question (MCQ) test to determine eligibility to start the VR simulation technical (psychomotor) and non-technical skills training module. Following this they participated in a hands-on workshop for identification of scope parts, tools and accessories with demonstration of the steps of performing the upper GI endoscopy as shown in Figure 1. The technical module of the simulation course was facilitated by three expert endoscopists. Training on the communication and ethical behavior with the patient (non-technical skills components) was performed before the virtual simulation tasks, where the course facilitator played the role of simulated patient.

Figure 1

Flow chart

Trainees were allowed to practice with direct supervision and coaching. Avoidance of the common and critical errors (e.g. esophagus perforation, tracheal edema) was re-emphasized.

Deliberate repetitive practice was then permitted for all trainees where they repeated each of the two VR simulator tasks (basic upper endoscopy and a diagnostic biopsy) with formative feedback. Each of the trainees had to achieve technical proficiency in two consecutive trials of each task before proceeding to the next module/task.

Phase III: Assessment

Assessment of the cognitive module

For the cognitive skills, 10 MCQs were answered by the experts once before the course. The same 10-item MCQ test was then administered to the trainees before and after the cognitive module as a pre-and post-test. The difference between the pre- and post-test scores was used to assess the knowledge gained as a result of the didactic component.

Assessment of the technical and non-technical skills

Two of the authors NA and MA have developed a new assessment tool named Simulation Endoscopic Skill Assessment Score (SESAS) [Table 1] in January of 2018. Another author, NK has reviewed the task deconstruction of the procedure. SESAS came as a modification of the Direct Observation of Procedural Skills (DOPS) tool that has been developed by the Joint Advisory Group (JAG) for gastrointestinal endoscopy in the United Kingdom.[15] DOPS is a global assessment tool for gastrointestinal endoscopic skills and has been used by the Saudi fellowship committee to evaluate the psychomotor skills of the trainees for a few years. However, it lacks comprehensive competency assessment, with little procedure-related cognitive skill assessment. Even though different programs are using this assessment tool worldwide, up to our knowledge, it has not been validated as a benchmark for competency thresholds and the trainees can be scored as passed, even if some simulator benchmarks were not achieved. The new assessment tool SESAS was developed using task deconstruction where we have put our efforts to improve the profession's quality metrics and competency assessment for diagnostic upper GI endoscopy. All the experts who participated in the curriculum-design validated the content of the newly developed checklist through modified Delphi consensuses using the face-face followed by two email rounds methodology[14] that we have previously described for the curriculum content validation. Table 4 shows each checklist task and its relevant VRS metrics.

Table 1

Upper GI endoscopy ‘Simulation Endoscopic Skill Assessment Score’ (SESAS) Checklist

Tasks	Done	Not Done
1. Handles Skillfully the Endoscopy
Adequate fine tip control with proper navigation and torques of the scope
Intubate the oesophagus from 1st attempt under direct visualization without trachea intubation
Appropriate use of the water and suction with minimal air left in the lumen
2. Maintains Luminal View/Inserts in Luminal Direction
Able to maintain the lumen, with appropriate landmarks identification (gastroesophageal junction, incisura, major papilla, fundus)
3. Completes Procedure in Reasonable Time
Perform the endoscopy independently within 6 minutes with time to pylorus not more than 4 minutes
No/Minimal patient’s discomfort in 90% of the procedure time
4. Diagnostic Ability
a. Adequately visualizes the mucosa identifies/interprets pathology
Able to visualize most of the mucosa with proper photo documentation
Recognize and interpret the abnormalities with minimal prompting
b. Safe use of biopsy forceps techniques
Able to perform the biopsy with limited coaching
5. Commitment of Critical Errors

Table 4

Each SESAS Checklist Task and its Equivalent from the VRS Metrics

Checklist task	Relevant virtual reality simulator metric
Adequate fine tip control with proper navigation and torques of the scope	Percentage of time in red-out
Intubate the oesophagus from 1st attempt under direct visualization without trachea intubation	Number of attempts to intubate oesophagus
Appropriate use of the water and suction with minimal air left in the lumen	a. Total amount of air (cc)b. Amount of air left (cc)
Able to maintain the lumen, with appropriate landmarks identification (gastroesophageal junction, incisora, major papilla, fundus)	Landmarks identification (gastroesophageal junction, incisora, major papilla, fundus).
Perform the endoscopy independently within 6 min with time to pylorus not more than 4 min	Time metricsa.Total procedure time (min)b. Time to pylorus (min)
No/Minimal patient discomfort in 90% of the procedure time	Percentage of time of patient discomfort
Able to visualize most of the mucosa with proper photo documentation	Anatomical landmarks visualization
Recognize and interpret the abnormalities with minimal prompting	Pathology identification (gastric/duodenal ulcers, erosions, esophagitis, mass)
Safe use of biopsy forceps techniques Able to perform the biopsy with limited coaching	a. Complications like perforationb.Open force by bulling into working channel (oz/3.15N)

Technical, perceptual and decision making components of the non-technical skills were assessed during every simulation exercise using the SESAS and the relevant simulator metrics. The proficiency benchmarks for each task were based on the mean of the scores of the two trials performed by the five experts. The set level of proficiency for novice trainees was two standard deviations below the mean of the experts as per the methodology described by Gallagher AG and O'Sullivan.[16] Dialogic bi-directional feedback on communication, ethical behavior of the patient and professional and continuous development was provided to the trainees in a debriefing session following the simulation. Outcome measures and metrics of these non-technical components are mentioned in Table 2.

Statistical analysis

Descriptive statistics were computed for continuous variables, including minimum and maximum values, means, standard deviations, as well as 95% confidence intervals (CIs) and frequencies for categorical variables when appropriate. The Pearson's chi-squared test, t-test and, where appropriate, the Fisher's exact test were used.

R Studio was used for analysis using the R statistical language. A statistical significance threshold of P = 0.05 was adopted.

Cohen's kappa coefficient was used to assess the degree of agreement between the novel checklist SESAS and metrics generated by VRS and it was used as well to assess the inter-rater reliability between SESAS and DOPS.

RESULTS

Participants

A total of 5 experts and 10 inexperienced GI fellows were recruited for the current study. All experts met the criteria indicated in the methodology section. The characteristics of the participating experts are shown in Table 5.

Table 5

Characteristics of the GI Endoscopy Experts (n=5)

Characteristic	Mean±SD	Median (Min.– Max.)	IQR (25th-75th)
Experience in practice (in years)	9.6±4.742	10.0 (4-15)	4.75-14.25
Number of procedures (per year)	930.4±259.193	1012 (600-1200)	660-1160
Total number of procedures/expert’s center (per year)	8350.0±4088.704	7350 (3900-15000)	5450-11750
Simulation experience (in years)	1.4±0.516	1.0 (1.0-2.0)	1.0-2.0

Knowledge competency

All 15 participants answered the 10 MCQs. The mean score of the 5 experts was 9.0. All of the 10 inexperienced trainees completed the baseline MCQs pre-test. The mean score for the pre-test for this group was 2.7. After attending the cognitive module lectures, videos and hands-on workshop, all the inexperienced trainees re-answered the same MCQ as a post-test. The median MCQ pre- and post-test scores were 2.7 vs. 7.8 (P < 0.01).

Endoscopic technical and non-technical skills

The differences in performance between experienced and inexperienced (novice) endoscopists were compared.

The inexperienced trainees took significantly longer procedure time and time to pylorus compared with the experts (P < 0.01) although they were better than the experts in intubation of the esophagus with patient swallowing (P = 0.026) and amount of air left in stomach (P < 0.001). On the other hand, the experts needed a statistically significant smaller number of attempts to intubate the esophagus (P = 0.05). After training, there were no statistically significant differences between both trainees and experts regarding the rate of major complications (tracheal intubation, laryngeal edema), identification of landmarks, and patient discomfort. Table 6 demonstrates that the novices have reached most of the proficiency benchmarks as compared to experts.

Table 6

Comparison between the Two Trials’ Performance of Experts and Trainees Using Metrics Generated by Virtual Reality Simulator (VRS)

a. Time Metrics (min.)
Task Timing in minutes	Novice (n=10) Mean±SD Median M (Min.-Max.) IQR (25th-75th)	Expert (n=5) Mean±SD Median (Min.-Max.) IQR (25th-75th)	*P
Total procedure time	13.307±3.836	3.769±0.961	0.0001
	14.13 (5.44-21.54)	4.075 (2.29-5.16)
	10.178-15.333	2.835-4.4675
Upper Esophageal	3.391±2.117	1.01±0.911	0.001
Sphincter	3.15 (1.2-9)	0.715 (0.27-3.3)
Intubation	1.393-4.396	0.3975-1.35
Pylorus intubation	7.721±2.814	1.873±0.489	0.0001
	7.555 (2.28-13.3)	2.055 (1.10-2.47)
	5.71-9.975	1.4675-2.1975
*By Mann-Whitney U test
b. Complications
Complication	Novice (n=10) n %	Expert (n=5) n %	*P
Perforation location	0 (0%)	0 (0%)	-
Intubated trachea	1 (5%)	0 (0%)	0.667
laryngeal edema	1 (5%)	0 (0%)	0.667
*By Fisher's exact test
c. Identified anatomical landmark
Landmark	Novice (n=10) n %	Expert (n=5) n %	*P
GEJ	20 (100%)	9 (90%)	0.333
Gastric fundus	20 (100%)	10 (100%)	0.999
Angular incisura	17 (85%)	10 (100%)	0.281
Major papilla	19 (95%)	10 (100%)	0.667
3rd part duodenum	6 (30%)	3 (30%)	0.656
*By Fisher's exact test
d. Esophageal intubations
Task	Novice (n=10) n %	Expert (n=5) n %	P
Esophagus intubated with patient swallowing	18 (90%)	5 (50%)	0.026*
Number of attempts to intubate esophagus	1.3±0.47 1 (1-2)	1.0±0.0 1 (1-1)	0.05**
*Fisher's exact test. **Mann-Whitney U test
e. Amount of air inflated and left
	Novice (n=10) Mean±SD Median (Min.-Max.) IQR (25th-75th)	Expert (n=5) Mean±SD Median (Min.-Max.) IQR (25th-75th)	*P
Total amount of air inflated (cc)	1330.7±740.031	1758.4±526.956	0.095
	1144.5 (374-2670)	1723 (1025-2689)
	717.25-2123.5	1357.0-2191.5
Amount of air left (cc)	62.65±27.561	93.0±7.645	0.001
	74.5 (16-100)	95 (81-100)
	30.25-81	85.5-100
*By Mann-Whitney U test
f. Pathology Identification
Task	Novice (n=10) n %	Expert (n=5) n %	*P
Visualization	20 (100%)	10 (100%)	0.9999
Location Identification	19 (95%)	10 (100%)	0.9999
*Fisher's exact test
g. Other metrics
Metric	Novice (n=10) Mean±SD Median (Min.-Max.) IQR (25th-75th)	Expert (n=5) Mean±SD Median (Min.-Max.) IQR (25th-75th)	*P
Minimum depth of scope insertion (in cm)	73.25±8.20	72.4±4.742	0.774
	69.0 (64-92)	73.0 (65.0-78.0)
	68.0-80.25	67.75-77.25
Time in red-out (in min)	2.2175±1.129	5.235±15.029	0.001
	1.875 (1.02-5.10)	0.395 (0.20-48.0)
	1.34-2.9975	0.3-0.745
Percentage of total time in red-out	16.45±6.074	18.5±15.96**	0.536
	16.0 (7.0-28.0)	12.0 (5.0-60.0)
	11.25-20.25	10.75-23.5
Maximum force exerted by endoscope (oz/2.46 N)	9.6755±1.509	8.883±1.4979	0.113
	9.35 (7.14-11.34)	8.54 (7.35-11.3)
	8.4175-11.33	7.488-10.2125
*By Mann-Whitney U test. **Outlier excluded
h. Patient discomfort
Percentage of time of patient discomfort	Novice (n=10) Mean±SD Median (Min.-Max.) IQR (25th-75th)	Expert (n=5) Mean±SD Median (Min.-Max.) IQR (25th-75th)	P*
No discomfort	95.3±3.0796	94.80±4.131	0.9999
	96.0 (86.0-99.0)	96.0 (88.0-100.0)
	95.0-97.75	91.25-98.25
Mild	4.40±3.169	3.30±2.163	0.548
	4.0 (1.0-14.0)	3.0 (0.0-6.0)
	2.25-5.0	1.75-5.25
Moderate	1.50±0.527	1.90±2.558	0.428
	1.5 (1.0-2.0)	1.0 (0.0-7.0)
	1.0-2.0	0.0-3.0

*By Mann-Whitney U test

The following are the simulator metrics that showed evidence of construct validity (baseline discriminative ability between novices and experts):

(1) Time: total procedure time, time to upper esophageal sphincter intubation, time to pylorus intubation, time in red-out

(2) Esophageal intubation: number of attempts to intubate the esophagus.

(3) Occurrence of perforation as a complication.

Validation of the novel checklist

A total of 10 novices were assessed for their endoscopic skills at the end of the training using the new checklist (SESAS) by two independent evaluators and the degree of agreement between the new checklist and metrics generated by the VR simulator was found to be high (kappa = 0.83) [Figure 2].

Figure 2

Comparison of performance of experienced and inexperienced endoscopists on VRS tasks and SESAS which was done at the end of training

The degree of agreement and inter-rater reliability between SESAS and DOPS was found to be excellent (kappa = 0.90) [Figure 3].

Figure 3

Comparison of performance of experienced and inexperienced endoscopists based on DOPS and SESAS at the end of training

DISCUSSION

The need for a comprehensive, standardized curriculum for GI endoscopy for gastroenterology fellowship programs was identified resulting in the commissioning of a task force to develop a 'Fundamentals of GI Endoscopy' curriculum using simulation-based education. The primary outcome of the current study was to develop a FGE simulation course that can improve novice performance in GI endoscopy prior to patient encounter. The second outcome was to determine which of the VR simulator metrics demonstrate evidence of construct validity. This is in addition to demonstrating the efficacy of the newly developed checklist (SESEAS). These outcomes were demonstrated throughout the components of the curriculum – knowledge, technical skills, and non-technical skills using a proficiency-based progression (PBP) methodology.[16]

In skills training, it is very important to assess patient safety, as measured by occurrence of critical errors. In the current study, no major critical error (e.g. perforation) was reported by any of the trainees following the technical skills training. Also, although experienced endoscopists intubated the oesophagus in less time and less attempts (on the first attempt), most of novices intubated the oesophagus within two attempts. Time is not a critical factor, and with experience, the novices will continue to improve intubation skills without jeopardy to patient safety. Of major importance was the ability to successfully enter and examine the stomach and duodenum, and all were able to enter the pylorus and identify the Papilla of Vater (one of the hallmarks of a successful upper endoscopic procedure) by the completion of training. Finally, but of expected minor consequence, the novices have actually evacuated more air upon completion of the procedure than experienced endoscopists. This theoretically results in less post procedural patient discomfort.

So, the current single-arm study demonstrates that novices trained on this curriculum can reach a pre-set proficiency benchmark level of safe performance. As such, the trainee is expected to be able to perform safely an endoscopic procedure on a true patient under an experienced endoscopist's minor active then passive supervision. Despite the claim that achieving proficiency in simulation-based training constitutes “prima fascie” evidence for the competency of the novice, we are currently planning for a follow up study at the workplace to collect evidence for consequences validity.

Our new checklist-based global assessment tool, (SESAS), that was developed based upon and compared to the internationally used DOPS checklist, has demonstrated comparability to the novices' performance results that are automatically generated during training on the AccuTouch® GI endoscopy simulator. We hope that providing the simulation-based education community with a global assessment tool, that incorporates but does not solely rely on simulator outcome measures and metrices of construct validity, would lead to more objective assessment of trainee performance.

Strengths of the current study include: (1) following a validated stepwise approach for simulation-based curriculum development that is reached by international consensus[13] and is currently being used for curriculum development in multiple countries,[17](2) providing a model for the incorporation of simulator metric outcome measures and metrics of proved construct validity for objective assessment. Limitations to the study include the small number of participants and whether these results would transfer to other available GI simulators. Replication of these results by other independent researchers would strengthen the efficacy of the curriculum. Follow up studies for the transfer of competencies of the simulation trained fellows to the workplace are highly recommended.

CONCLUSION

The 'Fundamentals of GI Endoscopy' curriculum is a comprehensive, proficiency-based, standardized endoscopic training curriculum that has been primarily validated and can serve as a valuable addition to the gastroenterology fellowship programs. The evidence reported herein supports the effectiveness of the simulation curriculum for learning, the reliability and validity of the new SESAS as an assessment tool, and the importance of using the metrics-based methodology to engender patient safety. Workplace follow up study of trainee performance is recommended for consequences validation.

Financial support and sponsorship

This project was funded by an educational grant from the Saudi Gastroenterology Association.

Conflicts of interest

There are no conflicts of interest.