Does quality assured eLearning provide adequate preparation for robotic surgical skills; a prospective, randomized and multi-center study.

PURPOSE: In particular after the onset of the COVID-19 pandemic, there was a precipitous rush to implement virtual and online learning strategies in surgery and medicine. It is essential to understand whether this approach is sufficient and adequate to allow the development of robotic basic surgical skills. The main aim of the authors was to verify if the quality assured eLearning is sufficient to prepare individuals to perform a basic surgical robotic task.
METHODS: A prospective, randomized and multi-center study was conducted in September 2020 in the ORSI Academy, International surgical robotic training center. Forty-seven participants, with no experience but a special interest in robotic surgery, were matched and randomized into four groups who underwent a didactic preparation with different formats before carrying out a robotic suturing and anastomosis task. Didactic preparation methods ranged from a complete eLearning path to peer-reviewed published manuscripts describing the suturing, knot tying and task assessment metrics.
RESULTS: The primary outcome was the percentage of trainees who demonstrated the quantitatively defined proficiency benchmark after learning to complete an assisted but unaided robotic vesico-urethral anastomosis task. The quantitatively defined benchmark was based on the objectively assessed performance (i.e., procedure steps completed, errors and critical errors) of experienced robotic surgeons for a proficiency-based progression (PBP) training course. None of the trainees in this study demonstrated the proficiency benchmarks in completing the robotic surgery task.
CONCLUSIONS: PBP-based e-learning methodology is an effective training method avoiding critical errors in the suturing and knotting task. Quality assured online learning is insufficient preparation for robotic suturing and knot tying anastomosis skills. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04541615.

Introduction

The outbreak of COVID-19 has had a profound and almost certainly enduring impact on how health care professionals work, interact, are educated, and trained [1]. It impacts probably most on trainees in procedure-based interventional disciplines such as surgery and in particularly involving the use of advanced technologies, e.g., surgical robotics, to perform the procedure. In response, there appears to be a precipitous rush to implement virtual and online learning strategies in surgery and medicine [2, 3]. There are no good arguments against the value of virtual and elearning approaches to support education and training in medicine. The function of these technologies should however be to augment and support conventional education and training activities [4]. For example, eLearning should be used to prepare a learner for a practical skills training course in which they are about to engage. Such an approach prior to attending the course ensures that the trainee knows what to do, how to do it and with which devices [5]. On the course the trainer has quantitative evidence that the trainee has this knowledge and can therefore concentrate their efforts on supervising the trainee in how to integrate their knowledge with skill implementation in the real world or a simulated environment. The latter part of this education and training sequence has been severely disrupted by the COVID-19 pandemic. We also have concerns that some educationalists envision that virtual and online learning strategies alone will suffice for preparing trainees for clinical practice. All of medical practice, including internal and non-invasive medicine, involves the use of learning technical skill whether it is for taking, a blood pressure, an electrocardiogram, phlebotomy, angiography, endovascular procedures or minimally invasive surgery. Currently, there appears to be an optimistic view that much of the learning can be achieved on online with virtual and observational approaches to learning [6].

In this study, we assessed the performance of trainees and their readiness to implement robotic suturing and knot tying. The study participants received preparation methods, ranging from a complete eLearning path to peer-reviewed published papers describing the suturing and knot tying task and the associated performance metrics and training benchmarks [7, 8]. All of the participants were then required to complete an assisted but unaided robotic vesico-urethral anastomosis task that had been benchmarked on experienced robotic surgeons’ performance for a proficiency-based progression (PBP) training course [9]. The primary outcome was the percentage of trainees who demonstrated the proficiency benchmark after their training. The secondary outcome was the performance scores of trainees (i.e., procedure steps completed, errors and critical errors).

Methods

We evaluated the participants’ ability to perform a robotic suturing, anastomosis and knot tying task, on the “Venezuelan Chicken model” [7]. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee(Institutional Review Board approval from Onze Lieve Vrouw Hospital, Aalst, Belgium) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Each subject signed an informed consent to participate in this study. The development and validation of performance metrics based on PBP methodology, the standardized preparation of the model, the robotic setup and the anastomosis technique for this task have been reported elsewhere [7].

A total of 48 medical students, mean age of 21.9 years old (range 19–26), of Belgian nationality, with a particular interest in robotic surgery were enrolled in the study and randomized in four different groups of 12 subjects each. One participant of group 3 was unable to take part in the study for health reasons and was therefore excluded. The final number of participants was 47. The demographic data, personal characteristics and data on the surgical experience of the participants were collected in detail and are presented in Table 1.

Table 1

Demographic data, personal characteristics and data on the surgical experience of the participants

	Group 1	Group 2	Group 3	Group 4	Total
Num. of participants	12	12	11	12	47
Gender (num (%))					Num (%)
Male	6 (12, 8)	5 (10, 6)	4 (8, 5)	6 (12, 8)	23 (48.9)
Year of medical school (num (%))
Bachelor	8 (17)	4 (8, 5)	6 (12, 7)	5 (10, 6)	23 (48, 9)
Masters	4 (8, 5)	8 (17)	5 (10, 5)	7 (14, 9)	24 (51)
Dominant hand (num (%))
Right	11 (23, 4)	10 (21, 2)	9 (19.1)	7 (14, 9)	37 (78, 7)
Wear glasses (num (%))
Yes	5 (10, 6)	6 (12, 8)	7 (14, 9)	4 (8, 5)	22 (46, 8)
None	7 (14, 9)	8 (17)	7 (14, 9)	7 (14, 9)	29 (61, 7)
< 1 h per week	2 (4, 2)	1 (2, 1)	2 (4, 2)	2 (4, 2)	7 (14, 9)
Num. of participants	12	12	11	12	47
Gender (num (%))					Num (%)
Male	6 (12, 8)	5 (10, 6)	4 (8, 5)	6 (12, 8)	23 (48.9)
Year of medical school (num (%))
Bachelor	8 (17)	4 (8, 5)	6 (12, 7)	5 (10, 6)	23 (48, 9)
Masters	4 (8, 5)	8 (17)	5 (10, 5)	7 (14, 9)	24 (51)
Dominant hand (num (%))
Number of robotic procedures observed (num (%))
None	7 (14, 9)	6 (12, 8)	5 (10, 6)	6 (12, 8)	24 (51)
Number of robotic procedures acted as bedside assistant (num (%))
None	12 (25, 5)	12 (25, 5)	11 (23, 4)	11 (23, 4)	46 (97, 8)
Number of laparoscopic procedures observed (num (%))
None	6 (12, 8)	5 (10, 6)	3 (6, 4)	2 (4, 2)	16 (34)
Number of laparoscopic procedures acted as bedside assistant (num (%))
None	12 (25, 5)	11 (23, 4)	11 (23, 4)	9 (19.1)	43(91, 4)
Number of open procedures observed (num (%))
None	5 (10, 6)	4 (8, 5)	3 (6, 4)	2 (4, 2)	14 (29, 7)
Number of open procedures acted as assistant (num (%))
None	8 (17)	7 (14, 9)	6 (12, 8)	8 (17)	29 (61, 7)
Surgical robotic courses attended (num (%))
Yes	0 (0)	1 (2, 1)	0 (0)	1 (2, 1)	2 (4, 2)
No	12 (25, 5)	11 (23, 4)	11 (23, 4)	11 (23, 4)	45 (95, 7)
Drive licence (num (%))
Yes	12 (25, 5)	10 (21, 2)	10 (21, 2)	11 (23, 4)	43 (91, 4)
No	0 (0)	2 (4, 2)	1 (2, 1)	1 (2, 1)	4 (8, 5)

Regarding the surgical experience of the participants, only three (6.4%) of them had observed more than five robotic surgical procedures before participating in the study, only one had performed a procedure as a bedside-assistant, none had experience as a console operator.

Randomization and characteristics of the study groups

Matched subjects were randomized using © 1998–2020 RANDOM.ORG into four groups. The participants enrolled in the different groups underwent a didactic preparation with different formats before carrying out the practical activity. Trainees were randomized into (i) Full PBP group who received information on how to optimally perform the ORSI chicken anastomosis task and the material was delivered online via the ORSI e-learning platform. This group had access to the material two weeks before the practical training. The participants enrolled in this group were required to study the material to a pre-defined proficiency benchmark before training progression. (ii) eLearning group received the exact same information as Full PBP group on how to optimally perform the ORSI chicken anastomosis task in an eLearning fashion, but they were not required to study the material to a pre-defined proficiency benchmark. The time they spent online and effort expended were logged in details thanks to the characteristics of the platform used. The participants of this group had free access to the practical module, whatever the level of completion and the score obtained in the eLearning module was. (iii) Traditional group received face-to-face lectures in the training center, before starting the practical module, on how to perform the chicken anastomosis task. The lectures were not different from what they would normally receive during a traditional surgery training course. The content of the face-to face lecture is the same as in the eLearning course. The participants of this group had free access to the practical module after the theoretical course. (iv) The Apprenticeship group who received the exact same educational content as the elearning material on robotic suturing and knot tying but the information was contained in a series of peer-reviewed published papers describing the suturing and knot tying task including the performance metrics used for the proficiency assessment.

All groups were evaluated with the same assessment method at the end of the theoretical module, but only the Full PBP group was required to reach the benchmark before proceeding to the technical skills assessment.

Procedure: eLearning

The online learning content for knowledge acquisition deployed for the OSSET study was hosted on the Bridge (by Instructor) Learning Management System (LMS). This LMS was procured with the principal purpose of supporting the development of online proficiency-based progression (PBP) online learning courses.

The Suturing and Knotting course materials were developed from video footage of task performance captured and narrated at the ORSI Academy. This content was then subjected to an instructional design process which ensured that course administrators had the prerequisite number of data points for learning analytics and real-time analysis of learner performance and achievement of the proficiency benchmark.

The OSSET curricula were mapped to a relevant ‘Programme’ within the Bridge LMS. In Bridge nomenclature, a programme is a series of courses, and for this study, each step of the robotic suturing, anastomosis and knot tying was a course in the programme. Each course contained a combination of text content for each step, error and critical error and video footage of the step being completed in accordance with the metrics. The advantage of this approach was twofold—the learner was presented with a sequential stepwise curriculum with prompts for successful completion of each step and to commence the next. In terms of learning analytics, instructional designers, course administrators and subject matter experts had multiple and redundant data points to assess learner engagement and achievement. Course analytics were further supplemented by thoroughgoing video analytics facilitated by Instructure Studio. Number of video views per learner and to what minute and second of each video were captured and analyzed. This provided the study with a rich data set documenting the learner journey. It also provided researchers with a real-time view of trainee performance, and in the PBP group, who had met the benchmark in the online assessment and so could proceed to the skills demonstration phase at ORSI.

The assessment used video extensively; videos of correct task performance but also incorrect practice displaying errors were included as it is important that the learner can identify what not to do as readily as what to do. The assessment was built in Articulate Storyline 360, rendered as a SCORM 2004 package, and uploaded to Bridge. This was to provide researchers with options for multimedia-based assessment and feedback options to provide the learners with prompts to review the curricula content before reattempting the assessment.

The proficiency benchmark for the eLearning module was 94%. This was based on the objectively assessed performance of experienced robotic surgeons on the exact same assessment.

Prior to completing the assessment of the robotic suturing and anastomosis technical skills, all subjects were required to complete the assessment component of the eLearning module, whether they had access to or required to study the eLearning module or not. All of the subjects had access to the exact same information but in different formats, e.g., the Apprenticeship Group (Group iv) had access to the published papers on how the task was optimally performed, the steps of the task, the errors and critical errors.

Procedure: robotics assessment

Afterward, subjects were trained in ORSI Academy, Melle, BE. All had simulation training on the da Vinci® Skills Simulator (DVSS) before robotic surgery training proper. They performed the robotic suturing, anastomosis, and knot tying task on the chicken model with the Da Vinci Si or X or Xi platforms. Their performances were recorded and stored for blinded assessment by 5-pairs of surgeon raters trained to score the metrics with an IRR (agreements/agreements + disagreements) ≥ 0.8, consistently [10, 11]. The validated benchmarks were (1) all five steps of the procedure were completed in a maximum time of 25 min, (2) A maximum of 10 general errors and (3) no critical errors [7].

Statistical analysis

Trainees were required to meet all three benchmarks, and differences between the mean performance of each group were compared for statistical significance with the proficiency benchmarks using one-sample t tests in SPSS (Version 26, Chicago, Illinois, US).

Results

The PBP group (i.e., Group i) was obligated to meet the proficiency benchmark before attending the Skills assessment in ORSI; they were therefore considerably more engaged with the curricula. Using time spent as a proxy for engagement, the PBP group spent on average 55.7 min in the eLearning program. The eLearning Group (i.e., Group ii) without any assessment requirement and attending benchmark spent on average ~ 22 min in the program. Group iii (the Traditional group). The Conventional Group (Group iii) had the exact same material presented to them during (social distanced) face-to-face lectures in ORSI Academy which lasted ~ 150 min. We have no way of reliably estimating the amount of time the Apprenticeship Group (Group iv) spent on the didactic information they were sent (i.e., the peer-reviewed and published papers detailing the task).

The PBP Group (Group i) achieved the highest score (Fig. 1) on their objectively assessed knowledge on how to optimally perform the task. On average, they scored 97% on the eLearning assessment. Not surprisingly they also showed the least performance variability. Their scores were statistically significantly better than the other three groups (F (3, 42) = 16.98, p < 0.000). Group ii (the eLearning group) performed better (82%) than Groups iii (78%) and iv (76%) but they also showed the greatest performance variability. None of these differences were statistically significant.

Fig. 1

The mean and 95% confidence intervals of scores for the four groups on the eLearning module completed before they were assessed on their technical performance of the robotic suturing and anastomosis task

Figure 2a–c shows the mean and 95% confidence intervals (CI) of the performance of each of the four groups relative to the proficiency benchmark for the metric. For procedure steps metrics, none of the groups demonstrated the proficiency benchmark. Trainees completed 40–60% fewer procedure steps than the requirement. All these differences were statistically significant (Table 2). For the procedure error metrics, none of the groups met the proficiency benchmark either. For the PBP, the eLearning and apprenticeship groups, the difference was between 35 and 68% worse than the proficiency benchmark and all the differences were statistically significant (see Table 2). The traditional trained group was only 15% away from demonstrating the benchmark, and the difference between the mean and the benchmark was not statistically significant. None of the trainees in the PBP group made a critical error when they performed the task which means they met the proficiency benchmark for this metric. They however failed to meet the overall proficiency benchmark. 83% of the PBP group failed to meet the proficiency benchmark on the other two metrics (i.e., procedure steps and/or errors). The other three groups all failed to meet the benchmark for the Critical Errors metric by 45–67%, and all these differences were statistically significant (see Table 2).

Fig. 2

a–c The mean and 95% CI of procedure steps, errors and critical errors made by the four groups of trainees on the robotic surgery vesico-urethral anastomosis model relative to the proficiency benchmark for each performance metric. Also shown are how far off the proficiency benchmark performance was

Table 2

The mean difference between the scores of the four groups compared against the proficiency benchmark for each of the performance metrics

	Meandifference	95% confidence interval of the difference	t	df	Prob
Procedure steps (proficiency benchmark = 5)
PBP	− 2.25	− 3.07	− 1.43	− 6.051	11	0.000
eLearning	− 2.17	− 3.21	− 1.12	− 4.57	11	0.001
Lectures	− 2.82	− 3.85	− 1.79	− 6.08	10	0.000
Apprenticeship	− 3.42	− 4.37	− 2.46	− 7.86	11	0.000
Procedure errors (proficiency benchmark = 10)
PBP	3.50	0.84	6.16	2.90	11	0.015
eLearning	6.75	3.34	10.10	4.43	11	0.001
Lectures	1.36	− 2.12	4.84	0.87	10	0.403
Apprenticeship	4.67	0.30	9.03	2.35	11	0.038
Critical errors (proficiency benchmark = 0)
PBP
eLearning	0.67	0.35	0.98	4.69	11	0.001
Lectures	0.45	0.10	0.81	2.89	10	0.016
Apprenticeship	0.50	0.17	0.83	3.317	11	0.007

Discussion

The role of e-learning in improving surgical confidence, knowledge, and skillset was previously reported in the literature, (ref) however, to date, there was no comparison of a simple e-learning application versus PBP-based e learning application with subsequence practical assessment of task performance. Based on this point, our study is aimed to provide a prospective evaluation and comparison of this methodology of training with PBP-based e-learning methodology.

All the trainees in the PBP group demonstrated the proficiency benchmark (i.e., 94%) on the eLearning module, and their scores were significantly better than the other three groups. Despite having access to the exact same material, Group ii (the eLearning group) performed significantly worse than the PBP Group. Without an assessment to focus minds and the benchmark barring progress to skills assessment, online learning is a checkbox exercise to be overcome.

Despite the PBP group being optimally prepared (i.e., with task knowledge), none of them demonstrated the proficiency benchmark on the performance of the robotic suturing and anastomosis task. Indeed, none of the trainees in this study demonstrated the proficiency benchmarks, i.e., (i) completed the procedure (ii) with < 10 errors and (iii) no critical errors. A fundamental tenet of PBP skill training is that the trainee must reach all the metric-based proficiency benchmarks before training progression [4, 5]. Although the PBP group demonstrated the proficiency benchmark on the online curriculum and made no critical errors, none of them were able to complete the procedure. Overall, across all four groups, only five subjects (out of 47) completed the procedure. Being able to complete the procedure is a non-negotiable and fundamental aspect of interventional and procedure-based medicine. Furthermore, the procedure must be completed safely and effectively [4].

The results from this study show that no trainee met all or even most of the proficiency requirements. The PBP Group demonstrated the proficiency level of the knowledge requirement on the eLearning modules. Interestingly, they were also the only group where none of the trainees made a critical error. This is probably because of the explicit way critical errors were treated in the online training material, i.e., enactment of a critical error was an automatic failure to demonstrate the proficiency benchmark.

In the PBP methodology, errors are performances that deviate from optimal performance and probably should not be done by the trainee. In contrast, critical errors are performances that compromise the safety of the patient or the integrity of the procedure. Not all PBP training programs have a ‘0’ critical error proficiency benchmark. For example, Angelo et al. [12] allowed the trainees to make ≤ 3 performance errors and no more than one of them could be a critical error. In contrast, O'Herlihy et al. [13] permitted no critical errors in their skills training program. Proficiency benchmark definition is based on the results of the Delphi and construct validity studies [7]. The proficiency benchmark is usually based on the mean of the objectively assessed performance of the experienced practitioners on the same task [4, 5, 14]. This approach can however be augmented by the experienced practitioners and educators based on their experience.

Our results do not detract from the attractiveness of virtual or eLearning methodologies, and we would argue strongly that they should be a fundamental part of an efficient and quality assured approach to surgical skills training. The COVID-19 pandemic has very clearly demonstrated the vulnerabilities of the current medical education and training approach. Efficient and effective education and training in medicine with homogeneous training outcomes are not a new aspiration [15]. The pandemic has unambiguously demonstrated a fundamental vulnerability of the current paradigm. The Halstedian training paradigm for skills training has not survived for more than a century by accident [16]. With all of its inherent problems such as reliance on fortuitous learning opportunities and assessment subjectivity, it has produced good doctors for more than a century. The clinical landscape is however changing significantly with fewer opportunities to learn the practice of surgery and medicine [17-19]. These changes mean that surgical educators must choose education and learning strategies that are of proven effectiveness and not just opt for tools that ‘might’ be effective [20].

The results from the study reported here demonstrate very clearly that eLearning strategies on their own they are insufficient to prepare a trainee for the implementation of newly learned skills in a clinical context. It therefore behooves medicine to identify strategies that will work better than the approaches that we have identified here. Furthermore, these approaches must be realistic and not simply aspirational. Strategies need to be grounded in the realistic requirement of an approach to skills training that are deliverable expeditiously with at least some quantitative validation evidence [21]. The impact on medical education and training of the COVID-19 pandemic shows no signs of abating. Viable and effective strategies need to be identified and an indispensable part of this process will also be demonstrating what does not work.

Conclusion

PBP-based e-learning methodology is an effective training method avoiding critical errors in the suturing and knotting task. Quality assured online learning is insufficient preparation for robotic suturing and knot tying anastomosis skills.