Objective assessment of gastrointestinal endoscopy skills using a virtual reality simulator.

BACKGROUND: This study was carried out to validate the role of virtual reality computer simulation as a method of assessment of psychomotor skills in gastrointestinal endoscopy. We aimed to investigate whether the GI Mentor II computer system (Simbionix Ltd.) was able to differentiate between subjects with different experience with GI endoscopy.
METHODS: Twenty-eight subjects were included in the study. They were divided into 3 groups according to their experience with GI endoscopy: experienced [group 1, performed > 200 endoscopic procedures, (n = 8)] residents [group 2, performed < 50 endoscopic procedures, (n = 10)] and medical students [group 3, never performed GI endoscopy, (n = 10)]. All participants received identical pretest instruction on the simulator. Assessment of endoscopic skills was performed during a simulated colonoscopy and was based on parameters measured by the computer system: time, percentage of mucosa surface examined, efficiency of screening, time with a clear view, excessive local pressure, pain, time with pain, loop formation, and total time with a loop.
RESULTS: Significant differences in performance existed between surgeons in the 3 groups. Experienced surgeons demonstrated best performance parameters, followed by the residents and the medical students. Significant differences in time (Kruskal-Wallis test, P < 0.001), percentage of mucosa surface examined (P = 0.001), efficiency of screening (P = 0.001), time with a clear view (P = 0.001), pain experienced (P = 0.004), time with pain (P = 0.012), loop formation (P < 0.001), time with a loop (P < 0.001), and excessive local pressure (P = 0.001) were demonstrated. Significant differences existed between group 1 and 2 and 1 and 3 (Mann-Whitney test, P < 0.05). Differences between groups 2 and 3 did not reach statistical significance (P > 0.05).
CONCLUSIONS: The VR simulator was able to differentiate between subjects with different endoscopic experience. This indicates that the GI Mentor measures skills relevant for gastrointestinal endoscopy and can be used in training programs as an assessment tool.

INTRODUCTION

Training in new surgical procedures requires hands-on experience, which traditionally has been performed on live patients or by using anesthetized animals or synthetic models. The introduction of new technologies has revolutionized surgical education over the last decade, and virtual reality (VR) simulators are becoming more widespread as tools of teaching and assessment of surgical and endoscopic skills.[1,2] VR simulation has a number of advantages compared with the conventional teaching tools, ie, the training process is safe, standardized and feasible. Additionally, VR simulators allow quantitative and completely objective assessment of performance.[3]

Training in endoscopy in a virtual environment is thought to be a good alternative to classical bedside teaching, but without its adverse effects, such as patient discomfort, risk of perforation, and longer examination time. However, before introducing a VR system in training and assessment practice, studies must investigate the construct validity as well as the ability to transfer skills acquired in the virtual environment into the operating room. Thus, the objective of this study was to validate the role of a virtual reality computer system for assessment of psychomotor skills in gastrointestinal endoscopy. We aimed to investigate whether the GI Mentor II computer system (Simbionix Ltd, Cleveland, OH, USA) was able to differentiate between subjects with different experience with gastrointestinal endoscopy.

METHODS

Setting

The study was carried out at the endoscopy unit of a surgical department at a teaching hospital.

The Simulator

The GI Mentor II (Simbionix Ltd, Cleveland, OH, USA) consists of a mannequin with bodily orifices for upper/ lower endoscopy, a Pentax ECS-3840F endoscope for upper/lower gastrointestinal tract examination, guide-wire devices and foot pedals, all attached to a computer system and monitor. During movement of the endo-scope within the mannequin, the endoscopist gets tactile feedback comparable to that in in vivo examinations.

The software includes different patient cases simulating both diagnostic and therapeutic scenarios. Thus, the system can help the student identify different pathological findings, biopsy techniques, and become familiar with the equipment, while the computer registers objective performance parameters as described below.

Procedures

Assessment of endoscopic skills was performed during a simulated colonoscopy and based on the following parameters: time to complete the examination, percentage of mucosa surface examined, efficiency of screening, time with a clear view, excessive local pressure, pain (due to excessive air insufflation or pressure), time with pain, loop formation, and total time with a loop. These parameters were calculated and registered by the computer system.

Twenty-eight subjects were included in the study. They were divided into 3 groups according to their experience with gastrointestinal endoscopy: experts [group 1, performed >200 endoscopic procedures, (n=8)] residents [group 2, performed <50 endoscopic procedures, (n=10)] and medical students [group 3, never performed gastrointestinal endoscopy, (n=10)]. All participants received identical pretest instruction on the simulator and had one practice session on the simulator prior to the assessment trial. No participants had previous experience with endoscopic simulation.

Statistics

Data was analyzed using SPSS 10.0 software package (SPSS, Chicago, IL, USA). The Kruskal-Wallis, nonpara-metric test was used to assess the difference between the 3 groups regarding all performance parameters. P≤0.05 was considered statistically significant. Data represent median (range) values unless stated otherwise.

RESULTS

Significant differences in performance were found between surgeons in the 3 groups, as illustrated in . Experienced endoscopists (group 1) performed better in all the measured parameters, followed by the residents (group 2), and the medical students (group 3).

Significant differences for each of the following parameters were demonstrated: Time (Kruskal-Wallis test, P<0.001), percentage of mucosa surface examined (P=0.001), efficiency of screening (P=0.001), time with a clear view (P=0.001), pain experienced (P=0.004), time with pain (P=0.012), loop formation (P<0.001), time with a loop (P<0.001), and excessive local pressure (P=0.001).

Significant differences existed between groups 1 and 2 and groups 1 and 3 (Mann-Whitney test, P<0.05). Differences between groups 2 and 3 did not reach statistical significance (Mann-Whitney test, P>0.05).

DISCUSSION

At present, most surgical educators agree that objective assessment of technical skills should be incorporated into modern training programs. In current practice, the assessment of surgical skills is often unstructured, subjective, and may be biased. A generally accepted, standardized and objective method for quantitative measurement of technical skills does not exist.[3] Recent developments in computer technology have provided virtual reality simulators that may give optimal conditions for training and evaluation of laparoscopic and endoscopic skills. The computer models allow repeated practice of procedures away from the OR, which is considered a suboptimal training environment because of high costs and the high stress factor. Furthermore, performance is not biased by anatomic variations or physiologic response as found in animals, and the scenario for testing is easily reproducible and suitable for examination purposes. Initial validation studies have indicated that several systems can provide objective and reliable assessment of technical skills; however, extensive evidence is needed before such systems can be implemented in general clinical practice.[3-5]

The present study demonstrated that the performance of surgeons on the VR simulator was proportional to their endoscopic experience. Thus, experienced endoscopists were faster, visualized a higher percentage of the mucosa, had fewer “red-outs,” and caused less discomfort for the virtual patient compared with the residents and the medical students. Furthermore, the experienced endoscopists showed much less intersubject variability, ie, they all performed well. These differences indicate that the simulator measured skills relevant for gastrointestinal endoscopy.

Comparison of the performance scores between the 3 groups. Group 1 consists of experienced endoscopists (performed >200 endoscopic procedures), group 2 consists of residents (performed <50 endoscopies), and group 3 medical students (never performed endoscopy). Horizontal bands indicate medians, boxes indicate 25th and 75th percentiles, and whisker lines indicate the highest and lowest values for (a) time, (b) percentage of mucosa examined, (c) efficiency of screening, (d) time with a clear view, (e) time with pain, (f) time with a loop.

Our study showed no statistically significant difference in performance between beginners and residents with little endoscopic experience. This may be because the residents had not acquired sufficient proficiency with this examination, and more experience is needed before differences from the novices can be detected. Few previous studies have demonstrated the construct validity of VR simulation in GI endoscopy.[6,7] Our results are consistent with these findings.

CONCLUSION

The present work demonstrates that the GI Mentor endos-copy simulator can successfully distinguish between experienced and novice endoscopists and confirms the construct validity of the computer system. The virtual reality trainer is thus a promising tool for assessment of endoscopic skills and could potentially be incorporated into an endoscopic training and assessment program. Future work should investigate the learning curves on the simulator and assess whether skills acquired in the virtual environment can be transferred to the endoscopy ward.