Validation of laparoscopy and flexible ureteroscopy tasks in inanimate simulation training models at a large-scale conference setting.

OBJECTIVE: Simulation based training with training models is being increasingly used as a tool to help trainees mount the learning curve. However, validation studies of surgical simulators are often limited by small numbers. We aim to evaluate the feasibility of validating simulation-training tasks in laparoscopy and flexible ureteroscopy (FURS) rapidly at a large-scale conference setting for residents.
METHODS: Seventy-six urology residents from various Asian countries were assessed on their laparoscopic and FURS skills during the 14th Urological Association of Asia Congress 2016. Residents performed the peg transfer task from the fundamentals of laparoscopic surgery (FLS) and completed inspection of calyces and stone retrieval using a flexible ureteroscope in an endourological model. Each participant's experience (no experience, 1-30 or >30 procedures) in laparoscopy, rigid ureteroscopy (RURS) and FURS was self-reported.
RESULTS: Median time taken to complete the laparoscopic task decreased with increasing laparoscopic experience (209 s vs. 177 s vs. 145 s, p=0.008) whereas median time taken to complete the FURS tasks reduced with increasing FURS experience (405 s vs. 250 s vs. 163 s, p=0.003) but not with RURS experience (400.5 s vs. 397 s vs. 331 s, p=0.143), demonstrating construct validity. Positive educational impact of both tasks was high, with mean ratings of 4.16/5 and 4.10/5 respectively, demonstrating face validity.
CONCLUSION: Our study demonstrates construct and face validities of laparoscopy and FURS simulation tasks among residents at a conference setting. Validation studies at a conference setting can be an effective avenue for evaluating simulation models and curriculum in the future.

Introduction

Surgical education has evolved since the Halstedian model of graded responsibilities with gradual progression of skills to a current model of accelerated acquisition of skills [1]. In this era of rapidly evolving technologies and expanding surgical fields, surgical trainees are increasingly required to learn more in a shorter period of time [2]. Learning has also been challenged by restrictions on residency training hours [3] and emphasis on operating room efficiency. Moreover, it is no longer acceptable to train on patients for a number of important reasons—patient safety, ethical, medico-legal and fiscal. As such, there is a move toward simulation training to achieve competency of technical skills outside of the operating room.

Simulator based training allows the opportunity to address many limitations of modern surgical training. The fundamentals of laparoscopic surgery (FLS) and European training in basic laparoscopic urological skills (E-BLUS) programs are well established and extensively validated training curricula in laparoscopic surgery [4]. In endourology, simulators are also particularly well suited for its purpose due to the use of camera systems in a closed cavity by a single surgeon.

The rapid technological advances in ureteroscopes have led to expanding indications for flexible ureteroscopy (FURS) [5]. However, there is a significant learning curve to achieve proficiency [6,7]. Flexible ureteroscopes are also expensive to upkeep especially if mishandled by novices. Therefore, a variety of low and high fidelity ureteroscopy training models have been developed and they have been validated to different extents [8] but limited by small numbers. In this context, we aimed to evaluate the feasibility of validating simulation-training tasks in both laparoscopy and flexible ureteroscopy rapidly at a conference setting for residents.

Materials and methods

Urology residents from Asian countries who attended the urology residency course in conjunction with the 14th Urological Association of Asia Congress (July 2016) were asked to perform laparoscopy and FURS simulation tasks on inanimate simulation models. All participants consented to participate and were briefed on the validation tasks as well as the methods of assessment prior to commencement. A baseline questionnaire was given to the participants before the start of the tasks to collect basic demographic information and their self-reported experience with prior laparoscopy, rigid ureteroscopy (RURS) and FURS procedures as a first surgeon (no prior experience, 1 to 30 procedures or >30 procedures).

The laparoscopic and FURS simulation tasks were chosen for their feasibility to be completed within the time constraints of the conference yet with sufficient complexity to show construct validity. Comparing the two simulation tasks allowed for construct validity of these training models to be tested at the same sitting.

The laparoscopic peg transfer task was chosen from the validated FLS and E-BLUS programs (Fig. 1). This task involved the use of two Maryland dissectors to move six rubber rings across a pegboard and back in a laparoscopic box trainer. Timing was started when the first ring was grasped and ended when the last ring was transferred to the original position.

Figure 1

Peg transfer task.

The FURS tasks were performed using flexible fibreoptic ureteroscopes (URF-P6, Olympus, Tokyo, Japan) in a previously validated high fidelity endourology skills training model (Advanced Scope Trainer, Mediskills Ltd, Northampton, UK) [9,10] with a 12/14 Fr ureteral access sheath (UroPass, Olympus, Tokyo, Japan) in place (Fig. 2). Two sets of laparoscopic box trainers and endourology skills training model were used. Participants had to complete two FURS tasks sequentially and were assessed on time taken for each task. In the first task (inspection of calyces), participants had to inspect all calyces within the model. Timing was started when the flexible ureteroscope was inserted and ended when all the calyces were inspected. This was followed by the second task (stone retrieval). This task involved manipulating the flexible ureteroscope to the lower pole calyx and using a nitinol basket to retrieve a stone from that calyx to outside of the model. Participants were aided by a passive assistant who only opened and closed the basket upon receiving instructions to do so. Timing was started when the ureteroscope was inserted and ended when the stone was retrieved out of the model.

Figure 2

Advanced Scope Trainer™. Reproduced with permission from Mediskills Limited.

After completion of the study, participants were also asked to evaluate the positive educational impact of the laparoscopy peg transfer task as well as the two FURS tasks using a 5 point Likert scale, ranging from 1 “Strongly disagree” to 5 “Strongly agree”.

Data were analyzed using Stata version 13 (StataCorp LLC, Texas, USA). Descriptive statistics were used to analyze the study participants' demographics. Spearman's correlation was employed to evaluate the performance with continuous and ordinal variables like age and training year. To evaluate construct validity, the Kruskal–Wallis test was used to compare performance between different participant experience levels.

Results

Over a 3 h afternoon session during the course, 76 participants from 16 Asian countries completed the questionnaires and the laparoscopic and FURS simulation tasks. The data for laparoscopy experience were missing in one participant. The study demographics of the 76 participants are shown in Table 1. There were 61.3% and 55.3% of participants who had no prior experience in laparoscopy and FURS respectively whereas 61.8% of the participants had performed >30 RURS procedures.

Table 1

Participant demographics.

Characteristic	n=76
Age, mean±SD, year	32.3±4.81
Training year, mean±SD	3.4±1.42
Post-graduate year, mean±SD	6.2±3.20
Right-handedness, n (%)	69 (92)
Previous experience in laparoscopy, n (%)
0	46 (61.3)
1–30	21 (28.0)
>30	8 (10.7)
Previous experience in RURS, n (%)
0	8 (10.5)
1–30	21 (27.6)
>30	47 (61.8)
Previous experience in FURS, n (%)
0	42 (55.3)
1–30	23 (30.3)
>30	11 (14.5)

SD, standard deviation; RURS, rigid ureteroscopy; FURS, flexible ureteroscopy.

There were no significant correlations between participant's age and training year with time taken to complete the laparoscopy (r=0.04, p=0.71 and r=−0.06, p=0.65) or FURS (r=0.04, p=0.70 and r=−0.08, p=0.51) tasks.

Participants felt that the positive educational impact of both laparoscopy and flexible ureteroscopy tasks were high, giving mean ratings of 4.16±0.77 and 4.10±0.89 respectively, demonstrating face validity.

The performance of the participants in the laparoscopy and FURS tasks are shown in Table 2. Median time taken to complete the laparoscopy task decreased significantly with increasing experience in laparoscopy (p=0.008) procedures but not RURS (p=0.778) or FURS (p=0.069) procedures. Median total time taken to complete the FURS tasks decreased significantly with increasing experience in FURS (p=0.003) procedures but not laparoscopy (p=0.441) or RURS (p=0.143) procedures. This demonstrates good construct validity for the well-established laparoscopy peg transfer task as well as the FURS tasks.

Table 2

Comparison of performance in laparoscopy and flexible ureteroscopy simulation tasks by differing experience levels.

	Median time taken for laparoscopy task, IQR, s	p-Valuea	Median total time taken for FURS tasks, IQR, s	p-Valuea
Laparoscopy experience
0	209 (162–253)	0.008	380 (241–431)	0.441
1–30	177 (156–233)		309 (164–456)
>30	145 (130.5–159.5)		319 (156–388)
RURS experience
0	213 (189.5–230.0)	0.778	400.5 (328.5–423.0)	0.143
1–30	177 (150–258)		397 (252–494)
>30	183 (155–245)		331 (176–431)
FURS experience
0	213 (171–251)	0.069	405 (309–466)	0.003
1–30	162 (125–237)		250 (183–430)
>30	162 (150–200)		163 (75–384)

RURS, rigid ureteroscopy; FURS, flexible ureteroscopy; IQR, interquartile range.

Calculated using Kruskal–Wallis test.

Discussion

The European Association of Urology Section of Urolithiasis (EULIS) had previously published a consensus statement promulgating the importance of simulation training and curriculum development for trainees throughout Europe. The direction was towards preliminary validation of the curriculum at European Urology Residents Education Program (EUREP) training events due to the scale involved [11]. At the same time, collaborations between various European Association of Urology sections have led to the development of the endoscopic stone treatment step (EST-s1) [12] and its subsequently validation by Veneziano et al. [13]. The EST-s1 curriculum consists of four standardized tasks: Flexible cystoscopy, rigid cystoscopy, semi-rigid URS and FURS. The FURS task utilizes the K-box trainer (Porgés-Coloplast, France) and entails a timed visualization of all 12 calyces within the trainer followed by extraction of scope with sheath out of the trainer. The first component of the FURS task used in our study (inspection of calyces) is similar to task 4 of the EST-s1 curriculum. At the time of the conduct of our study, the EST-s1 curriculum was yet to be well established. Although we utilized a different endourology skills trainer, our findings further confirmed the results of the validation study by Veneziano et al. [13] and also showed the feasibility of performing such validation studies with a large number of Asian study participants at a conference setting.

The metrics used in the standardized assessment of a trainee need to be measured using standardized tasks and endourology task trainers. Our findings suggest that these tasks could also be validated in another FURS trainer. Other FURS specific bench models that have been validated include the Uro-Scopic trainer (Limbs and things, UK) [14], Adult Ureteroscopy Trainer (Ideal Anatomic Modeling, USA) [15], K-Box (Porgés-Coloplast, France) [16,17], Cook URS model (Cook Medical, USA) [18] and Endo-Urologie-Modell (Karl Storz, Germany) [19]. Chou et al. [20] demonstrated no difference between the use of a bench model (Uro-Scopic trainer) and a Virtual reality simulator (UroMentor, Simbionix, Israel) for training novices in ureteroscopy. Looking to success stories in laparoscopic simulation training with the FLS and E-BLUS programs, the attention should now be directed at the validation of specific tasks, reliable assessment tools and curriculum development instead of developing expensive new models [21]. Matsumoto et al. [22] demonstrated similar training efficacy using a low cost, low fidelity ureteroscopy trainer compared with a high cost, high fidelity trainer. In our study we tested the participants' ability to manipulate the FURS to inspect calyces and retrieve stones as part of the larger skill set involved in mastery of FURS. Other tasks that can be practiced and assessed include deployment of ureteric stents and repositioning of stones.

Face validity in simulation is a subjective measure and reflects the realism and educational impact of the simulator as evaluated by learners or non-experts. Construct validity, on the other hand, assesses the ability of the simulator to measure what it is supposed to measure, in this case the ability to distinguish between levels of experience. In other words, experienced trainees perform better on the simulator in comparison to inexperienced trainees. Our study has confirmed face and construct validities for the well established laparoscopic task as well as for the two FURS tasks.

Discriminant or divergent validity is another subtype of construct validity. It helps to establish construct validity by demonstrating that the construct to be measured or evaluated is different from other irrelevant constructs. In our study, we have demonstrated that the ability to perform well on the laparoscopic simulation task is correlated with laparoscopic experience only and not with either RURS or FURS experience. More importantly, we also demonstrate that the ability to perform well on the FURS simulation tasks on this training model is correlated with only FURS experience and not RURS experience. This implies that experts in RURS do not perform better in FURS. Even though both are endoscopic procedures involving the ureter, the skill sets required for each are distinct from each other. This further highlights the need to develop FURS-specific training models and tasks to train the set of skills that are unique to FURS.

There are potential limitations to the conduct of validation studies in this setting. The two flexible ureteroscopes used in the study were significantly damaged during the study due to the mishandling by novice participants. Educating participants on proper handling and care of the scopes before the start of the program is an important learning point from the study as the repair and amortization costs of flexible ureteroscopes may be prohibitive for widespread use in an education setting. However, with the advent of disposable ureteroscopes, there may be scope for development of cost-effective ureteroscopes specifically for training use. In order to facilitate testing large numbers of participants at a conference, the simulation model used needs to be hardy and yet of a relatively low cost.

The use of self-reported operative experience is recognized as a limitation to the study design due to the possibility of recall bias. Although the use of logbook records retrieved from medical records would be more accurate, this would be challenging to collect from a large group of participants from various countries in a conference setting. This has also been recognized in other validation studies [13,23]. We acknowledge that the categorization of experience levels using 0, 0–30, >30 procedures in our study is arbitrary. Although a learning curve of 60 cases has been suggested based on small single surgeon series [24], the participants in our study were at varying levels of training (mean 3.4 years). Hence a smaller number was chosen to better discriminate between experience levels within residents at an earlier stage of training.

Time is also an important limiting factor in such a setting, therefore the tasks developed for these studies were not overly complex and were able to be completed within a reasonably short time, yet achieving the ability to demonstrate various types of validity. Matsumoto et al. [25] first described the use of a Global Rating Scale (GRS) in an endourological simulation environment with the aims of capturing the flow and logical progression of the procedure, with subsequent modification by White et al. [15] for flexible ureteroscopy simulation training. It entails seven domains that are graded by an observer with a Likert scale 1 to 5 to a total score ranging from 7 to 35. It may not be an appropriate tool of assessment in this setting as it is complex and includes less relevant domains like “respect for tissue” and “flow of procedure”. The metrics for assessment should be clearly defined with time limits for tasks and without the use of complex assessment tools in order to maintain the standards of objective assessment in a resource-limited setting.

Conclusion

Our study demonstrates construct and face validities of laparoscopy and FURS simulation tasks using inanimate bench models among residents at a conference setting. Validation studies at a conference setting can be an effective avenue for evaluating simulation models and simulation training curriculum in the future.

Author contributions

Study concept and design: Jirong Lu, Tricia Kuo, Ho Yee Tiong.

Data acquisition: Jirong Lu, Karthik Thandapani, Tricia Kuo.

Data analysis: Jirong Lu, Karthik Thandapani.

Drafting of manuscript: Jirong Lu, Karthik Thandapani.

Critical revision of the manuscript: Tricia Kuo, Ho Yee Tiong.

Conflicts of interest

The authors declare no conflict of interest.