National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients.

Background: Transurethral resection of bladder tumours (TURBT) is the initial diagnostic treatment for patients with bladder cancer. TURBT is not an easy procedure to master and simulator training may play a role in improving the learning curve. Objective: To implement a national training programme for simulation-based mastery learning in TURBT and explore operating theatre performance after training. Design setting and participants: From June 2019 to March 2021, 31 doctors at urology departments in Denmark performed two pretraining TURBT procedures on patients, followed by proficiency-based mastery learning on a virtual reality simulator and then two post-training TURBTs on patients. Outcome measurements and statistical analyses: Operating theatre performances were video-recorded and assessed by two independent, blinded raters using the Objective Structured Assessment for Transurethral Resection of Bladder Tumours Skills (OSATURBS) assessment tool. Paired-sample t tests were used to compare pretraining and post-training analyses and independent t tests for between-group comparisons. This trial is registered at ClinicalTrials.gov as NCT03864302. Results and limitations: Before training, novices had significantly lower performance scores in comparison to those with intermediate experience (p = 0.017) and experienced doctors (p < 0.001). After training, novices significantly improved their clinical performance score (from 11.4 to 17.1; p = 0.049, n = 10). Those with intermediate experience and experienced doctors did not benefit significantly from simulator training (p = 0.9 and p = 0.8, respectively). Conclusions: Novices improved their TURBT performance in the operating theatre after completing a proficiency-based training programme on a virtual reality simulator. Patient summary: We trained surgeons in an operation to remove bladder tumours using a virtual reality simulator. Novice doctors improved their performance significantly after the training, but the training effects for more experienced doctors were minimal. Therefore, we suggest the introduction of mandatory simulator training in the residency programme for urologists.

Introduction

Transurethral resection of bladder tumours (TURBT) is the initial diagnostic treatment for patients with bladder cancer (BC). BC is common, with 570 000 new patients diagnosed worldwide annually, and more than half of these patients will experience recurrence of BC [1]. BC is two-faced: non–muscle-invasive bladder cancer (NMIBC) is recurrent but has minimal malignant potential, while muscle-invasive bladder cancer (MIBC) is lethal if left untreated and has a significant mortality rate even after radical cystectomy [2]. The three objectives with TURBT are cancer clearance, correct histopathological diagnosis, and avoidance of adverse events [3]. Complete tumour eradication and adequate tissue sampling for histopathological diagnosis are paramount for accurate risk stratification in BC [3]. Although the anatomy and technical armamentarium seem simple, TURBT is a highly complex procedure [4].

Inexperience has a negative impact on the quality of TURBT. Detrusor muscle (DM) presence and overall survival are lower for patients operated on by residents in comparison to consultants [5], [6]. In addition, recurrence rates and readmission rates are higher for surgical residents than for consultants [7], [8]. These findings underscore that TURBT is not an easy procedure to master and that residents in their initial learning phase need substantial support to ensure that patient safety is not endangered [9].

Surgical curricula should strive to ensure trainees are proficient in surgical skills before they progress to clinical performance, and a complete curriculum should be developed and evaluated for intended (and unintended) effects [10]. Kern’s six-step approach [10] is a model that facilitates complete curriculum development. It includes six distinct steps that are closely interrelated although not in a strictly linear fashion:

Step 1: Problem identification and general needs assessment;

Step 2: Targeted needs assessment;

Step 3: Goal and objectives;

Step 4: Educational strategies;

Step 5: Implementation; and

Step 6: Evaluation and feedback.

Step 1 is already covered regarding TURBT, as adverse effects have been identified as a problem stemming from classical apprenticeship training [5], [6], [7], [8], and a national assessment of general needs included TURBT on the final list of urology procedures that should be practised in a simulation-based environment [11]. Furthermore, it has been established that simulation-based training (SBT) is an excellent educational strategy for skills training in TURBT (step 4) [11], [12], and goals and specific measurable objectives of SBT have been defined (step 3) [13]. However, scientific evidence is lacking regarding who to train (step 2) and when the SBT curriculum should be implemented (step 5) and the proposed effects evaluated (step 6).

Optimally, this approach should be applied to patient-related procedures using an assessment tool with validity evidence [14].

The aim of our study was to implement a national SBT TURBT training programme and explore which surgeons improved their clinical patient performance after training to a predefined proficiency level on a simulator.

Materials and methods

Ethics

The ethics committee of the Zealand Region deemed this study to be exempt from approval (REG-008-2018).

Study population

Doctors at urology departments in three major hospitals in Denmark were invited to participate. Surgeons of all experience levels were invited, ranging from first-year residents to consultants. Participants were volunteers and gave informed consent before inclusion. Pretraining data for the cohort have been reported previously [14].

Study design

Pretraining testing

Before simulator training, the participants performed two TURBT on patients (with tumours ≤3 cm). An expert investigator (urologist) was present. The investigator could act as a supervisor on request by the participant or if the investigator deemed it necessary. The two performances were video-recorded.

Participants were restricted to performing only TURBT related to the study during their inclusion period.

Simulator proficiency training

All participants filled out a questionnaire on demographics, including age, sex, title, and the number of previous TURBT operations performed. The TURB Mentor virtual reality (VR) simulator (Simbionix/Surgical Science, Gothenburg, Sweden) was used in this study. A standardised introduction to the simulator interface and equipment was followed by a warm-up session on the simulator limited to 15 min to ensure familiarity.

After the warm-up session, the participants performed simulator training as mastery learning, whereby individual training continues until each trainee reaches a predefined level of proficiency [13]. The trainees received computerised feedback and formative feedback from the urologist. We previously gathered validity evidence for the simulator test according to the contemporary framework for validity and defined the proficiency level using an established standard-setting method [13]. Each participant continued training until they reached the computerised proficiency level on three consecutive performances in each of the three TURBT cases on the simulator. Training sessions were limited to a maximum of 2 h per session, or less if the participant requested it because of fatigue. Participants repeated training sessions (minimum 1 d apart) until they reached proficiency.

Post-training testing

After simulator training, the participants performed two TURBT (tumours ≤3 cm) on patients with the investigator present. The two post-training performances were video-recorded. The final TURBT was performed within 30 d of the simulator training.

Outcome and statistical analysis

Video recordings of pretraining and post-training TURBT procedures were assessed by two blinded video raters using the Objective Structured Assessment for Transurethral Resection of Bladder Tumours Skills (OSATURBS) assessment tool [14]. The participating doctors were aware of the scoring items in the OSATURBS tool (insertion of scope, bladder filling, diagnostic cystoscopy, interpretation and strategy, instrumentation, finalisation, haemostasis, progression, and overall impression). Videos were presented to the raters in a randomised order to mask experience levels and procedure order (ie, raters were blinded to operator experience and whether it was a pretraining or post-training procedure). Paired-sample t tests were used for pretraining and post-training analyses and independent t tests for between-group comparisons. Correlations are reported as Pearson’s r. Significance was defined as p < 0.05. SPSS version 28.0 (IBM, Armonk, NY, USA) was used for statistical analysis.

Results

In total, 32 doctors were enrolled; one participant in the novice group did not perform post-training TURBTs and was excluded. Ten novices (<10 TURBT), nine individuals with intermediate experience (11–49 TURBT), and 12 experienced doctors (≥50 TURBT) were included (Table 1). The data collection period was, on average, 17 d per participant (range 2–66). Data were collected from June 2019 to March 2021 at three hospitals in Denmark.

Table 1

Demographic data and clinical performance scores before and after training

Variable	Novice group(0–10 TURBT)	Intermediate group(11–49 TURBT)	Experience group(≥50 TURBT)
Participants (n)	10	9	12
Mean age, yr (range)	30.7 (27–36)	34.2 (29–41)	35.6 (31–43)
Female, n (%)	5 (50)	8 (89)	7 (58)
Mean TURBT experience, procedures (range)	2 (0–6)	26 (15–40)	100 (50–200)
Mean total operating time on simulator, min (SD)	95.5 (34.2)	76 (15.3)	59.4 (16.5)
Mean score before training (SD)	11.4 (9.3)	21.1 (7.8)	25.0 (4.1)
Mean score after training (SD)	17.1 (7.1)	20.9 (5.7)	25.35 (4.0)
p value, paired-sample t test	0.0049	0.9	0.8
Mean difference in OSATURBS score (95% CI)	5.75 (0.02–11.5)	−0.25 (−6.1 to 5.5)	0.31 (−2.5 to 3.1)

CI = confidence interval; OSATURBS = Objective Structured Assessment for Transurethral Resection of Bladder Tumours Skills; SD = standard deviation; TURBT = transurethral resection of bladder tumour.

There was good correlation between OSATURBS scores (Fig. 1) in the two pretraining (Pearson’s r = 0.71; p <0.001) and the two post-training procedures (Pearson’s r = 0.66; p < 0.001). Thus, the performance scores were calculated as the mean for the pretraining TURBT and the post-training TURBT.

Fig. 1

The Objective Structured Assessment for Transurethral Resection of Bladder Tumours Skills (OSATURBS) tool for assessment of surgical skills in transurethral resection of bladder tumour. The scale ranges from a minimum of 0 to a maximum of 36, as scores are recoded as 1 → 0, 2 → 1, 3 → 2, 4 → 3, and 5 → 4 for all nine items.

The novices had significantly lower pretraining performance scores in comparison to individuals with intermediate experience (p = 0.017) and experienced doctors (p < 0.001).

The clinical performance of novices significantly improved after simulator training (from 11.4 to 17.1; p = 0.049, n = 10). However, neither the intermediates nor experienced groups improved significantly after simulator training (Table 1).

Figure 2 illustrates the effects of previous TURBT procedures and sensitivity to simulator training. Again, most novices had positive training effects, and some individuals had considerable improvements after simulator training (+22.5 points).

Fig. 2

Effect of simulator training. Change in performance score for transurethral resection of bladder tumour (TURBT) after simulator training according to previous clinical experience.

The post-training scores did not differ significantly between the novice and intermediate groups (mean difference 3.76, 95% confidence interval [CI] −2.9 to 10.0; p = 0.2), but was significantly inferior for the novice group in comparison to the experienced group (mean difference 8.3, 95% CI 3.3–13.3; p = 0.003; Fig. 3).

Fig. 3

Clinical performance score before and after simulator training according to experience level.

Discussion

In this prospective study of doctors performing TURBT, simulation training significantly improved the operating theatre performance of novices. This is the first study to report an improvement in clinical performance after a TURBT simulator training programme.

Training programmes can be evaluated from different perspectives. The evaluation framework developed by Kirkpatrick in 1959 is widely used to assess the effects of training interventions in a range of industries [15]. The Kirkpatrick model has four levels of educational effects:

Level 1: reaction (eg, questionnaires on trainees’ satisfaction with the programme);

Level 2: learning (eg, procedural checklist test scores in a simulated setting);

Level 3: behavioural change (eg, assessment of surgical skills for clinical performance); and

Level 4: results and outcomes with benefits for patients/the organisation (eg, patient satisfaction, rates of postoperative complications and readmissions) [15].

Our study provides unique evidence for the higher levels of the Kirkpatrick hierarchy. Results for Kirkpatrick level 1 have been reported for VR SBT in TURBT, with good overall trainee satisfaction and improved self-confidence [16], and Kirkpatrick level 2 evaluation revealed improved simulator scores after SBT training in TURBT [17]. Following on to the next level, our findings provide essential documentation of skills transfer from the simulation environment to clinical performance.

Different quality indicators for TURBT have been described [3]. A bladder diagram, description of tumour size, count and completeness of the resection, DM presence in the resected tumour specimen, and the recurrence rate at first follow-up cystoscopy are essential quality indicators of TURBT [3]. Future research should explore SBT and its effects on patient-related outcomes.

In this study we implemented a national standardised programme of mastery learning for TURBT (Kern’s step 4, educational strategy and step 5, implementation). Mastery learning has several implications for training effects. Overall, mastery learning ensures that all trainees reach the same minimum standard. By contrast, training defined according to quantitative measures (such as training time or the number of procedures performed) ensures that only the amount of training and not a defined proficiency level—or quality—is accomplished, with a risk of large variations in actual proficiency. According to the Danish national specialist programme in urology, TURBT competence is achieved after approximately ten TURBT procedures in the first year, followed by 15 procedures in the following 5 yr of residency [18]. There is an association between surgical volume and outcome; however, there are substantial variations in the volume needed to reach proficiency, and a curriculum based on the number of procedures would be unnecessary for some surgeons and, most importantly, would not be sufficient for some surgeons to reach proficiency [19]. Agreement on standards across institutions is crucial for the reliability and generalisability of a training programme, and one way to account for variations in learning is to implement proficiency-based training programmes [20].

Few standardised TURBT programmes have been reported. de Vries et al [21] conducted a validation study on a physical TURBT simulator and assessed performance in terms of procedure time, resection completeness, bladder perforations, and a global rating by video raters, and found that this approach could discriminate between different experience levels. The authors suggested the introduction of a time-defined training programme in urological residency but did not report data to support their statement [21]. Notably, the authors contributed with necessary knowledge via a needs analysis on the procedural steps and technical and nontechnical pitfalls in TURBT, which can support the design of a proficiency-based curriculum in TURBT (Kern’s step 2). In contrast to our patient-free SBT approach, two studies reported on changes to the classical apprenticeship model. Pycha and Palermo [22] suggested a step-by-step progression programme for clinical procedures and emphasised the importance of undisturbed communication between the trainee and supervisor. The authors found that complication rates decreased on inclusion of a supervisor with better performance and concluded that teaching TURBT in the clinical setting is challenging. Brausi et al [23] described an organisational intervention comprising routine flexible cystoscopy and a bladder diagram for diagnosis, video recordings of all TURBTs under the supervision of a senior urologist, monthly teaching meetings, and pathology assessed by a urologist for shared decision-making and surgeon feedback. They found improvements in DM presence and recurrence rates for specialists and trainees. Whereas both of these groups focused on improving teaching in the patient-related setting, our training programme focused on enhancing surgical skills in a patient-free environment, acknowledging that classical apprenticeship in TURBT compromises patient safety [6], [7], [8].

We assessed clinical performance using the OSATURBS tool with validity evidence reported according to Messick’s validity framework [14]. An established standard-setting method was used to determine the pass/fail score necessary for proficiency-based training, which is unfortunately often overlooked in research on surgical simulation [24]. Robust validity evidence should be explored using contemporary frameworks as recommended by the American Psychological Association in 1999 and endorsed internationally [20]. Despite these recommendations, a recent systematic review on surgical simulation and surgical skills found that only 6.6% of 498 studies reported contemporary validity evidence. The reporting quality was even poorer in urological studies (only 1.2% used the modern framework for validity evidence) [25]. As the urological community embraces competency-based medical education, we must report gold-standard evidence to support our assessments of future surgeons [26].

At the beginning of the learning curve, the effect of each case performed is high, but the gain decreases with subsequent cases. Accordingly, we found that the inexperienced doctors improved after the VR training. As Figure 2 illustrates, there was a positive training effect in the novice and intermediate groups, with a breaking point at approximately 20 procedures. SBT did not change the performance for the most experienced doctors in the intermediate group or for the experienced group, which may be explained by the level of complexity of the simulator cases. This provides important knowledge on who to train (Kern’s step 2) when developing a complete TURBT curriculum. Our findings are in line with those of Thomsen and colleagues [27], who investigated VR simulation training in cataract surgery and found positive effects on clinical procedures performed by novice and intermediate surgeons who had performed up to 75 cataract procedures on patients. Other studies on transfer have primarily assessed inexperienced residents and demonstrated improvements in clinical performance for different surgical procedures [28]. Neumann et al [17] conducted a randomised trial comparing VR SBT (intervention) to an instructional video session and found that VR training had a greater effect on simulator scores. However, the study participants were medical students, and therefore the results are difficult to extrapolate to the suggested target for a TURBT VR SBT programme, namely residents in urological apprenticeship.

Our study has some limitations that need to be taken into account. Clinical assessments are prone to several biases, including rater bias. Therefore, we included video rater assessments to diminish the risk of social biases. Konge et al [29] explored the assessment of endoscopic ultrasonography and fine-needle aspiration in mediastinal staging of non–small-cell lung cancer. They found that direct observation resulted in 10% lower scores for residents and 10% higher scores for consultants when compared to blinded video-based assessments [29]. In a study on assessment of flexible cystoscopy, Dagnaes-Hansen et al [30] also found significant differences between direct and video-based assessments of urologist performance. These findings are in concurrence with our results and emphasise that even in a controlled research set-up and conducted in good faith, direct observation assessments are prone to bias.

The sample size in our study could result in type II errors. However, despite the low number of participants, we found significant positive training effects for novices. Variations in complexity for the procedures may confound our results. To account for TURBT case variation, we used strict inclusion criteria for TURBT cases and used mean performance scores for two cases before training and two cases after training. However, differences in case complexity could still influence the scores, and future studies should consider including assessment of each case, for example, by using the novel Bladder Complexity Checklist developed by a Delphi panel of international specialists and stakeholders in TURBT and bladder cancer treatment [4]. However, the strong correlations for the two pretraining procedures and the two post-training procedures indicate that the variation in complexity did not affect the effect sizes. Finally, the cost of the VR simulator could be a prohibitive factor for implementing a mastery learning programme in countries with limited financial resources.

On the basis of the existing literature and our findings, we suggest a future training programme on mastery learning in TURBT consisting of (1) theoretical education, (2) proficiency-based simulator training, (3) supervised clinical performance with video assessment until proficiency is achieved, followed by (4) continuously supervised procedures of increasing complexity, and (5) regular video assessments.

Future studies should explore the effects of proficiency-based SBT on quality indicators in TURBT, including DM presence, complication rates, and the recurrence rate at first follow-up cystoscopy.

Conclusions

Novices improved their TURBT performance in the operating theatre after completing a proficiency-based training programme on a VR simulator. Simulator training had no significant effect on the performance of more experienced surgeons.

: Sarah H. Bube had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Bube, Azawi, Konge, Hansen, Dahl.

Acquisition of data: Bube, Kingo, Madsen, Vásquez, Norus, Olsen.

Analysis and interpretation of data: Bube, Azawi, Konge.

Drafting of the manuscript: Bube.

Critical revision of the manuscript for important intellectual content: Bube, Azawi, Konge, Hansen, Dahl, Kingo, Madsen, Olsen, Norus, Vásquez.

Statistical analysis: Bube, Konge.

Obtaining funding: None.

Administrative, technical, or material support: Bube, Olsen.

Supervision: Azawi, Konge.

Other: None.

Sarah H. Bube certifies that all conflicts of interest, including specific financial interests and relationships and affiliations relevant to the subject matter or materials discussed in the manuscript (eg, employment/affiliation, grants or funding, consultancies, honoraria, stock ownership or options, expert testimony, royalties, or patents filed, received, or pending), are the following: None.

None.