Lessons Learned and New Challenges: Re-evaluation of End-User Assessment of a Skills-Based Training Program for Urology Trainees.

OBJECTIVES: To re-assess the perceived benefit and relevance of simulation sessions to Victorian urology trainees and to identify areas for potential improvement. SUBJECTS AND METHODS: All trainees attending skills training sessions between 2011 and 2016 were asked to complete a structured questionnaire at the completion of the session. The questionnaire included 11 topic areas ranging from the year of surgical training to degree of usefulness of the session, including several sections for free-text response to offer more detailed feedback. Sessions were examined both individually and collectively to assess end-user satisfaction with the structure and content of the program.
RESULTS: In total, 24 individual skills sessions were held over the 6-year period, with a total of 355 attendees. Of these, 331 attendees completed the majority of the questionnaire, a response rate of over 93%. Overall 88% of the surveyed attendees stated that they had both the support of their supervising consultant and the flexibility of workload to attend the session; 90% of trainees felt that there was adequate reading material provided prior to the skills session, an improvement from 76% in the previous study period; and 97% of those surveyed felt that the existing session structure was appropriate and the same proportion found the sessions both useful and interesting, compared with just 63% in the previous study period. Analysis of individual topics demonstrates some variability in outcome measures, but for nearly every assessed parameter, greater than 90% of participants agreed that the session fulfilled the expected criteria. New topics developed since the 2011 analysis, including renal transplant and vascular repair, also had high levels of satisfaction. The practical models used have been refined and achieved higher scores than those in the previous assessment period.
CONCLUSION: The urology skills-based training program has been well received by the surveyed trainees and is now embedded and accepted as part of the Victorian training program. The format of the sessions has matured and the overall rating, both individually and collectively, was high. There has been a clear increase in satisfaction across most areas assessed when compared with previous feedback. Despite this, there remain areas that can be improved, such as the amount and quality of available equipment and the inclusion of video demonstrations of operative techniques.

Introduction

Surgical training in the modern health care system faces a variety of competing demands and challenges. The need for competent and experienced surgeons requires a rigorous and practical training program, which must be balanced by the right of the public to receive safe and effective surgical care.[1] Further demands on surgical training programs have resulted from the concurrent push for safer working hours in addition to the rapidly expanding variety of techniques and technologies that trainees are expected to develop proficiency in during their formative years.[2] This is especially true of urology, where the advances in endourology, robot-assisted techniques, and laparoscopic-assisted techniques are particularly pertinent.

Simulation and model-based training can provide surgical trainees with a platform for the safe introduction of surgical skills prior to exposure to a live patient.[3] It also has the capacity to provide on-demand experience in techniques required for rarely seen pathology, thus reducing the volume of cases required to be seen to gain proficiency in these less common procedures. The benefits even extend to time efficiency, and simulation training, therefore, has financial implications for health services. Laparoscopic surgical training using box models has been found to reduce operating time and improve surgical performance in trainees with limited prior laparoscopic experience.[4]

The Victorian training section of the Urological Society of Australia and New Zealand (USANZ) developed a surgical skills simulation program for urology trainees in 2004. The program is held at the Royal Australasian College of Surgeons (RACS) simulation laboratory in Melbourne, Victoria. The program underwent an evaluation process in 2009, the results of which were published in a study in 2011.[5] The findings at that time were that the simulation sessions were of value to urology trainees but had several areas of potential improvement.[5] Of particular concern was the need for appropriate models and the recommendation that modern techniques be covered by the program such as photo-selective vaporization of the prostate and renal transplantation.[5]

The structure of the skills sessions has remained consistent throughout the lifetime of the program. Prior to the session, trainees are given pre-reading information relevant to the session. The session itself comprises a half day (4-5 h). It begins with a didactic lecture relevant to the covered topic, followed by various practical workshops with a variety of synthetic, ex-vivo animal models and cadaveric specimens. The entire skills lab program consists of 12 sessions run over a 3-year period, with 3 to 5 sessions being conducted per year. The sessions remain mandatory for trainees in the core clinical training years 3 to 5 (of a 6-year training program).

The purpose of this study is to re-evaluate the perceived benefit of the simulation-based skills program to Victorian urology trainees and to assess the response to feedback following the previous study. It will also explore potential areas of improvement.

Subjects and Methods

All urology trainees attending the skills training sessions between 2011 and 2016 were asked to complete a structured questionnaire at the completion of the session. There is no existing validated tool for evaluation of surgical workshops, and therefore a questionnaire was developed to appraise various aspects of the program. This was identical to the questionnaire used in the previous study and consisted of 11 questions, with each garnering information on a different aspect of the session. It included demographic information concerning the year of surgical training, the level of support from the supervising consultant, the adequacy of the format of the sessions, and how useful and interesting the session was. Four of the questions consisted of multiple parts, requiring between 5 and 8 individual responses based on a 5-point Likert-type scale. These sections questioned the trainee’s opinion of the pre-session communication and learning materials, the logistics of the session, the effectiveness and availability of tutors, and the appropriateness of models and equipment. Several of the questions included free-text response to elucidate any potential areas for improvement, general comments about the session and future sessions that the trainee felt would be of benefit.

Sessions were examined both individually and collectively to assess end-user satisfaction with the structure and content of the program.

The data are presented using descriptive statistics.

Results

Across the 6 years included in this study, there were 24 skills workshops held and 355 individual attendances, of which 331 completed the session questionnaire, resulting in a response rate of 93%.

Consultant support for trainee attendance has improved in recent years (83% in the previous study period vs 98% in the current study), as has the flexibility with which trainees have been allowed to adjust operative lists to attend the sessions (89% vs 59%).

Overall 97.6% of attendees supported the current format of the skills workshops. Just 5% of surveyed attendees felt that the pre-session preparation material and communication prior to the workshop was inadequate; 97.6% of trainees agreed, either strongly or somewhat, that the lecture content, training models, supervision, catering, and venue were adequate. Almost 99% of trainees agreed with the adequacy of session inclusiveness, effectiveness, and involvement in the course, which were evaluated in question 6. Only 44 of the included 2637 question responses (1.7%) disagreed with any element of question 7, concerning the quality and involvement of tutors and industry representatives.

Trainees were asked to make an overall assessment of the skills lab session by indicating 1 of the following 4 comments:

They found the session both interesting and useful.

They found the session interesting but not useful to training.

They found the session to be useful and a good idea but thought that it was not taught or run well.

They found the session neither interesting nor useful for training, and felt that the time could be better spent.

Almost 98% of surveyed trainees found the sessions both interesting and useful, with a further 1.5% finding them at least interesting, if not useful to their training. Just 1 of 330 completed surveys stated that the trainee found the session neither interesting nor useful. The individual session responses ranged from 81% of trainees finding them both interesting and useful to 100% (see Table 1). A rating of 100% was seen in 20 of the 24 sessions (83%), a significant improvement on the previous study period (6%).

Table 1.

Workshop topics 2011-2016 with trainee ratings.

Year	Skills workshop	Rating[a]
2011	Bowel usage in urology: conduits and neobladder	100
	Laparoscopic nephrectomy and adrenalectomy	100
	Laparoscopic knot tying/transperitoneal and retroperitoneal approach to pyeloplasty	100
	Penile and artificial urinary sphincter prostheses	100
	Urethroplasty	100
2012	Vasectomy and vasectomy reversal	100
	Partial nephrectomy: open and laparoscopic	100
	Ureteroscopy and flexible pyeloscopy for stones and tumors	81
	Vascular injury, vascular repair, and renal transplantation	100
2013	Percutaneous nephrolithotomy	100
	Lower third ureteric surgery including psoas hitch and Boari flap	100
	Laparoscopic nephrectomy and adrenalectomy	91
2014	Bowel usage in urology: conduits and neobladder	100
	Urethroplasty	100
	Laparoscopic knot tying/transperitoneal and retroperitoneal approach to pyeloplasty	100
	Laser surgery for Benign Prostatic Hypertrophy (BPH)	100
2015	Vasectomy and vasectomy reversal	85
	Partial nephrectomy: open and laparoscopic	93
	Ureteroscopy and flexible pyeloscopy for stones and tumors	100
	Vascular injury, vascular repair, and renal transplantation	100
2016	Percutaneous nephrolithotomy	100
	Lower third ureteric surgery including psoas hitch and Boari flap	100
	Laparoscopic nephrectomy and adrenalectomy	100
	Stress Urinary Incontinence (SUI)	100

Total percentage of surveyed trainees who found the workshop both interesting and useful.

As a general rule, the feedback from these sessions is of a more positive nature than in the previous evaluation. More of the surveyed trainees found the sessions both useful and interesting (98% vs 63%).

Trainees remain satisfied with the current format of the sessions, and there has been a response in recent years to the request for increased background anatomy during the introductory lecture and in the pre-reading material. Of particular note is the improvement in the perceived quality of models. During the previous evaluation, 29% of responders felt that the quality of the simulations was not adequate, compared with less than 1% in the current study. There also appears to have been a marked improvement in the clarity of learning objectives.

Although it was previously an area of strength, there has also been a small improvement in the percentage of trainees who felt that the tutors were adequately prepared (99.7% vs 83%) and enthusiastic (99.7% vs 83%).

A common response during the previous study period was that the models could be improved to further enhance the sessions. It appears from the above satisfaction ratings that this has occurred. However, improved quality of models remained a common request in the free-text response sections of the questionnaire. New sessions, such as laser surgery for the prostate, have been included because of trainee requests in the previous study and were generally well received in this evaluation period.

Some pertinent suggestions from trainees during this study period included the use of video demonstrations of surgical techniques, the increased availability and quality of instruments/equipment, the pairing of junior trainees with more senior colleagues, and the use of clinical cases in the pre-reading and during the initial teaching session.

Some requests for future sessions that have not already been implemented include robotically-assisted surgery, scrotal surgery, and the surgical approach to renal trauma.

Discussion

Simulation-based surgical skills training remains an important feature of modern urology education and has the capacity to develop both technical and non-technical skills.[1] It is recognized as an effective method to develop the necessary surgical skills in a time-efficient, safe, and cost-effective way. The program developed by USANZ in 2004 was well received by the Victorian urology trainees who attended and evaluated the initial 5 years of its existence.[5]

The results of this study confirm the ongoing acceptance and effectiveness of the simulation program in the Victorian urology training system. Not only is there ongoing support from trainees but there appears to have been several key improvements since the previous evaluation period. Almost all facets of the program received increased approval ratings by the urology trainees who attended. They were seen to be more interesting and useful and there was widespread satisfaction with the format of the sessions. This improvement could be explained by multiple factors. The variation in methodology between the 2 studies may have resulted in a selection and/or response bias, due to the retrospective nature of the initial survey conducted in 2009. This difference in methodology between the current and previous study also meant that direct statistical comparison between the responses would not be valid. Those trainees who held strong opinions of the skills sessions may have been more likely to respond to the survey in 2009, and the lengthy time period between some of the sessions and the evaluation may have resulted in response or recall bias. Despite this, it remains likely that some of this difference is explained by the ongoing improvement in the selection and use of training models in response to feedback, as well as the inclusion of several requested topics and the removal of sessions that were not well received in the previous evaluation period.

Despite the obvious benefits of simulation-based training, there remains a paucity of data concerning its validity and educational impact for urological surgical skills.[6] Some evidence exists regarding the validity and benefit of simulation training for specific skills, such as retrograde endoscopic intra-renal surgery using a combination of biological, non-biological, and live animal models.[7] Further evidence is therefore required into the objective benefits of simulation training, specifically regarding its translation to patient outcomes.

Urological practice is a dynamic process and the training system for future urologists must remain current and flexible to meet these demands. Advances in technology, particularly in robot-assisted surgery, place extra demands on trainees and their training programs. Fortunately, the increased sophistication and availability of technology provide a mechanism with which to increase the variety and efficiency of skill acquisition. The current USANZ program uses a variety of models in their simulation sessions. However, there is an obvious gap in the use of virtual reality and in access to robot-assisted surgery simulations, with the cost of providing these educational resources likely to be a major limiting factor. The use of virtual reality simulation for robotic surgery in Australia has already been described in the literature and is increasingly available in the American surgical training system.[8,9] These advances are moving so quickly that it is now augmented reality training that is being discussed as the future of surgical skill acquisition.[10] A systematic review on the use of virtual reality training for laparoscopic surgery found that operating times were significantly reduced, even when compared with training programs that included box trainer models, such as those used in the USANZ sessions.[11] This gap will need to be addressed if the Victorian program is to remain relevant and useful in the future.

There also remains an opportunity to extend the program from its current focus on technical operative skills to include the training of non-technical skills, such as leadership, teamwork, and communication. Simulation has been recognized in the literature as a valuable tool for the development of such skills.[1,2,12]

Simulation training remains an important aspect of modern surgical education. It is regarded as a time-efficient and safe way of complementing the traditional “apprenticeship” model used in the training of surgeons for the past century or more. It is only going to increase in importance as time pressures, safety requirements, and changes to working hours continue to affect training time. This study found that a simulation-based surgical skills program for Victorian urology trainees was found to be both interesting and useful for clinical practice, with the flexibility to adapt to changing requirements and by utilizing user feedback.