Feasibility, effectiveness and transferability of a novel mastery-based virtual reality robotic training platform for general surgery residents.

BACKGROUND: The annual number of robotic surgical procedures is on the rise. Robotic surgery requires unique skills compared to other surgical approaches. Simulation allows basic robot skill acquisition and enhances patient safety. The purpose of this study was to evaluate the feasibility, effectiveness, and transferability of a mastery-based curriculum using a new virtual reality (VR) robotic simulator for surgery resident training.
METHODS: Nineteen PGY2s and 22 PGY4s were enrolled. Residents completed a pretest and posttest consisting of five VR and three previously validated inanimate tasks. Training included practicing 33 VR tasks until a total score ≥ 90% ("mastery") was achieved using automated metrics (time, economy of motion). Inanimate performance was evaluated by two trained, blinded raters using video review metrics (time, errors, and modified OSATS). Outcomes were defined as: curriculum feasibility (completion rate, training time, repetitions), training effectiveness (pre/post training skill improvement), and skill transferability (skill transfer to validated inanimate drills). Wilcoxon signed-rank and Mann-Whitney U tests were used; median (IQR) reported.
RESULTS: Thirty-four of 41 residents (83%) achieved mastery on all 33 VR tasks; median training time was 7 h (IQR: 5'26″-8'52″). Pretest vs. post-test performance improved (all p < 0.001) according to all VR and Inanimate metrics for both PGY2 and PGY4 residents. Significant pretest performance differences were observed between PGY2 and PGY4 residents for VR but not inanimate tasks; no PGY2 vs. PGY4 posttest performance differences were observed for both VR and inanimate tasks.
CONCLUSION: This mastery-based VR curriculum was associated with a high completion rate and excellent feasibility. Significant performance improvements were noted for both the VR and inanimate tasks, supporting training effectiveness and skill transferability. Additional studies examining validity evidence may help further refine this curriculum.

The use of robotic-assisted surgery (RAS) has grown rapidly in recent years, with one study finding an 8.4-fold increase in robotic utilization rates for common general surgery procedures between 2012 and 2018 and concomitant decreases in open and laparoscopic utilization rates [1]. Currently there is a significant need for surgeon training, especially at the resident level [2]. While RAS offers advantages relative to open and laparoscopic approach such as elimination of tremor, facilitated use of the non-dominant hand, and 3D visualization, specific skill acquisition is needed to adapt to the lack of haptic feedback and system control features. Indeed, mastery may be difficult to achieve, with learning curves lasting up to 80 cases for complex operations such as robotic pancreaticoduodenectomy [3, 4].

Simulation curricula have proven invaluable in affording skill acquisition, enhancing patient safety for laparoscopic surgery, and producing substantial cost savings [5-10]. Similarly, numerous simulation-based robotic training curricula have been associated with favorable results [11-15]. Given the expenses associated with both the overall robotic system as well as the limited-use instruments, many curricula have favored the use of virtual reality (VR) technology for robotic training [16-19]. Recently, Intuitive Surgical introduced a new VR robotic simulator called SimNow® (Intuitive Surgical Operations, Inc.; Sunnyvale, California) which uses the da Vinci Xi surgeon console and computer software to foster basic skill acquisition on 33 drills. Features include modular curriculum design, automated performance metrics, and customizable benchmarks for each drill. Data supporting the use of this new platform are limited and published studies are lacking. Using the SimNow simulator, our team designed a mastery-based curriculum for surgery resident training and implemented training over two academic years. The purpose of this study was to evaluate outcomes including curriculum feasibility, training effectiveness, and skill transferability.

Materials and methods

The new VR curriculum was implemented over two academic years (2019–2021) at the University of Texas Southwestern (UTSW) Medical Center for training surgery residents in basic robotic skills. A total of 41 residents were enrolled, including PGY2 (n = 19) and PGY4 (n = 22) residents. Data were collected prospectively and analyzed retrospectively under an exempt study approved by the UTSW Institutional Review Board. Training and testing were performed in the UTSW Simulation Center using the SimNow® simulation software accessed through a Da Vinci Xi surgeon console (VR tasks) and a Da Vinci Xi robotic system (inanimate tasks).

VR curriculum

The UTSW curriculum was based on a previous curriculum implemented for surgical oncology fellows at the University of Pittsburgh Medical Center (UPMC) by Hogg, et al. that used a similar but older VR simulator platform (Backpack®, Intuitive Surgical Operations, Inc.; Sunnyvale, California) [15]. The UPMC curriculum also included three inanimate drills to evaluate transferability of skills acquired on the VR simulator to a real environment. As previously described, these include: Ring Rollercoaster (RRC), Around the World Needle Driving (ATW), and Interrupted Suture (IS) (Fig. 1). Similar to the UPMC curriculum, the UTSW curriculum used a combination of VR and inanimate tasks and a pretest, training, and posttest design. Pretest and posttest consisted of five SimNow® VR and three UPMC inanimate tasks. The VR tasks (Fig. 2) were Around the World Needle Driving (vATW), Big Dipper Needle Driving (BD), Ring Roller Coaster 4 (RRC4), Knot Tying (KT), and Three-Arm Relay 3 (TAR3). The first 4 tasks were chosen due to similarity with the inanimate tasks. RRC4 and vATW are identical to RRC and ATW, respectively, and when taken together, BD and KT incorporate the movements required to complete IS. TAR3 was chosen because it was perceived to be a complex task that would likely discriminate improvement between pretest and posttest. Training included practicing all 33 SimNow® VR tasks until a composite score equal to or exceeding 90 (out of 100 maximum) was achieved (defined as mastery). Although residents were encouraged to reach mastery for all 33 tasks, they were not required to do so in order to posttest. The order of task completion was not stipulated.

Fig. 1

Inanimate tasks used for pretest and posttest

Fig. 2

Virtual reality tasks used for pretest and posttest

Protected time was made available during a month-long rotation with relatively light clinical duties for PGY2 and PGY4 residents. Residents were encouraged to complete the curriculum at their own pace within that month. This design was intended to stagger trainees to maximize access to the VR simulator and facilitate completion of the curriculum. Before beginning the curriculum, each resident completed an online training module through the da Vinci Surgery Community website designed to orient trainees to the robotic console and a pre-curriculum survey which captured basic demographic information, past robotic experience, and attitudes toward RAS.

Performance metrics

Performance on the VR tasks was evaluated using metrics automatically generated by the SimNow® software; these metrics included composite score, completion time, and economy of motion (the distance the instruments traveled to complete the task). Performance data were downloaded from the da Vinci Surgery Community website. Inanimate tasks were video recorded to facilitate subsequent assessment by video review. Performance was evaluated according to completion time, errors, and a modified Objective Structured Assessment of Technical Skills (OSATS) similar to the UPMC methodology [15]. This tool has been used in the robotic inanimate setting by some of the authors and has been validated in different fashions previously [15, 20–22]. Moreover, the OSATS confers the advantage of evaluating the time component as well as the economy of motion and instrument handling. These metrics could be similarly assessed in the VR training as well as in the OR. Consequently, in order to be consistent in our work and to allow future comparison between different training and practicing environments, we decided to use the same grading tool that has been effectively used in the past [23]. The OSATS scale involved a combination of 5-point Likert scales in six categories for a maximum of 30 points; categories included respect for tissue, time and motion, instrument handling, knowledge of instruments, use of assistance, and knowledge of procedure. These metrics were assessed by two blinded graders who received specific training in OSATS evaluation. At the end of their training both graders independently scored 10 inanimate drills and achieved high interrater reliability (Spearman: ρ = 0.84, p < 0.001).

Statistical analysis

For each of the 33 tasks in the curriculum, descriptive statistics were used to analyze attempts and time to mastery. The VR task “KT” was removed from the analysis of the pretest and posttest due to the likelihood of committing a critical error that ended the drill early, resulting on pretest in an appropriately low score but misleadingly low time and economy of motion. Pretest and posttest performance metrics were examined for normality by constructing histograms and performing Wilks-Shapiro test. Improvement between pretest and posttest was analyzed by summating each metric, across the four remaining VR tasks, and across the 3 inanimate tasks, and comparing metric totals for the pretest and posttest (total score, total time, total economy of motion, total errors). Paired t test was used to analyze improvement of the entire cohort between pretest and posttest. Wilcoxon matched signed-rank test was used to separately analyze improvement of PGY2s and PGY4s between pretest and posttest. Comparisons of performance between PGY2s and PGY4s at pretest and posttest were made using Mann–Whitney U test. Interrater reliability for the graders reviewing the inanimate drills was assessed with Spearman Rank correlation. All tests employed 2 tails and p < 0.05 was considered to be significant.

Results

Demographics

From the 41 residents enrolled, pre-curriculum surveys were collected from 32 (78%). Twenty-seven residents (84%), including 14 PGY2s and 13 PGY4s, reported prior robotic simulation experience with VR or inanimate drills with a median of 3.75 h of prior practice. Self-reported simulation experience did not correlate with performance in any metric on the VR or inanimate portions of the pretest or the time to achieve mastery on the 33 tasks in the curriculum.

Feasibility

Thirty-four residents (83%) achieved mastery on all 33 VR tasks. The number of days required to complete the curriculum, defined as the time between the pretest and posttest, ranged from 14 days to 10 months with a median of 1 month (IQR: 0.7–4.8 months). Thirteen residents (32%) required longer than 4 months to complete the curriculum. The console time required to achieve mastery on each task ranged from 2 h and 26 min to 13 h and 24 min, with a median of 6 h and 55 min (IQR: 5′26″-8′52″). Table 1 shows a breakdown of the curriculum by task with the percentage of residents that achieved mastery on each task along with the associated number of repetitions and training time.

Table 1

Trainee performance on each of the 33 VR tasks

Task	% Achieving mastery	Attempts required			Time required (min)
		Min	Median	Max	Min	Median	Max
4th arm cutting	100	1	1	2	0.5	1	3
30-degree scope swap Xi	97.5	1	2	7	1	4.5	18
Anterior needle driving ATW	100	1	1	5	2	3.5	18.5
Anterior needle driving horizontal	100	1	2	6	2	6	19
Anterior needle driving vertical	97.6	1	1	12	1.5	2.5	37
ATW needle driving	97.6	2	5	16	4.5	16	49.5
Big dipper needle driving	100	1	5	16	3.5	27	89
Camera 0	97.6	1	2	16	2	5.5	48
Clutch	100	1	1	2	0.5	0.5	1.5
Combo exercise	100	1	3	8	6.5	21	60
Energy pedals 1	100	1	2	3	1.5	3.5	5
Energy pedals 2	100	1	1	3	1.5	2.5	7
Instrument playground	100	1	1	1	0.5	2.5	36.5
Knot tying	100	1	3	14	1	5	33.5
Posterior needle driving ATW	100	1	1	7	2	3	27
Puzzle piece dissection	97.5	1	3	7	4.5	17	55
Railroad track	100	1	2	6	2.5	8.5	21.5
Ring rollercoaster 1	100	1	1	14	1	2	38
Ring Rollercoaster 2	100	1	4	13	3	21	58
Ring rollercoaster 3	95.1	1	2	18	1.5	7	71
Ring rollercoaster 4	95.1	1	5	39	9	37	223.5
Ring rollercoaster 5	100	1	1	7	1	1.5	15
Ring tower transfer	100	1	1	5	1	2.5	23.5
Running suture	100	1	1	5	1	1	9.5
Sea spikes 1	100	1	3	15	1	5.5	40
Sea spikes 2	100	1	7	59	1.5	13.5	135
Sea spikes game	89.7	4	19	62	8	38	124
Three arm relay 1	100	1	1	8	1	2.5	22.5
Three arm relay 2	95.1	1	5	24	2	16.5	82.5
Three arm relay 3	97.6	2	6	15	10	35	85.5
Vessel energy dissection	97.6	1	3	9	1	5	28.5
Wrist articulation 1	100	1	2	9	1	2.5	11.5
Wrist articulation 2	100	1	3	7	1	4.5	14.5

Effectiveness

Compared to the pretest, improved performance was observed on the posttest in every VR metric on every task when considering all trainees as a group, when considering only the PGY2s, and when considering only the PGY4s (all p < 0.001). Considering all trainees as a group, median total score on all 4 VR tasks increased from 82.5 (IQR: 35.8–141.5) to 353 (IQR: 318–366.25) out of a maximum of 400 (p < 0.001), median total time decreased from 26 min (IQR: 22.9–33.1 min) to 12.5 min (IQR: 11.7–13.8 min) (p < 0.001), and median total economy of motion decreased from 2,250 cm (IQR: 1696–2841 cm) to 1400 cm (IQR: 1294–1532 cm) (p < 0.001). For the PGY2s, median total score increased from 47 (IQR: 15–104.5) to 353 (IQR: 319–366) (p < 0.001), median total time to completion decreased from 28.9 min (IQR: 25.8–36.2 min) to 12.6 min (IQR: 11.6–13.3 min) (p < 0.001), and median total economy of motion decreased from 2,539 cm (IQR: 2165–2917 cm) to 1,399 cm (IQR: 1294–1502 cm) (p < 0.001). For the PGY4s, median total score increased from 133 (IQR: 63–182) to 348 (IQR: 305.3–365.3) (p < 0.001), median total time to completion decreased from 22.9 min (IQR:19.0–27.8 min) to 12.5 min (IQR: 11.9–13.7 min) (p < 0.001), and median total economy of motion decreased from 1,853 cm (IQR: 1678–2531 cm) to 1403 cm (IQR: 1302–1,559 cm) (p < 0.001). Figure 3 shows a comparison of pretest and posttest performance on the SimNow® tasks for the entire group, the PGY2s alone, and the PGY4s alone.

Fig. 3

Comparison of SimNow® performance at pretest and posttest. A All residents; B PGY2s; C PGY4s

Statistically significant differences were observed on the pretest between PGY2s and PGY4s in total score (p = 0.0036) and total time (p = 0.0027). On the posttest, however, PGY2 performance was not significantly different from PGY4 performance for both total score (p = 0.535) and total time (p = 0.562). Lower median total economy of motion was observed in the PGY4s relative to the PGY2s on the pretest (p = 0.0538), but this difference did not reach statistical significance. No difference was found in median total economy of motion between PGY2s and PGY4s on the posttest (p = 0.1461). Figure 4 shows a comparison of PGY2 and PGY4 performance on the SimNow® tasks at the pretest and posttest.

Fig. 4

Comparison of SimNow® performance between PGY2s and PGY4s at pretest (A) and posttest (B)

High interrater reliability was observed between the two graders in both errors (Spearman: ρ = 0.642, p < 0.001) and OSATS (Spearman: ρ = 0.614, p < 0.001), so averages of the two graders’ scores were used for analysis of the inanimate drills. Similar to the VR tasks, improved performance was observed on the posttest on each inanimate task according to all metrics when all trainees were considered as a group (all p < 0.005). Median total time to complete all three tasks decreased from 15.5 min (IQR: 14.2–17.8 min) to 12.4 min (IQR: 10.2–14.3 min) (p < 0.001), median total errors decreased from 5 (IQR: 3.5–7.5) to 2 (IQR: 1–3) (p < 0.001), and median total OSATS increased from 61 (IQR: 53–66) to 75 (IQR: 68–80.5) out of a maximum of 90 (p < 0.005). When considering only the PGY2s, improved performance was observed on each inanimate task according to all metrics (all p < 0.043). For the PGY2s, median total time decreased from 17.5 min (IQR: 14.9–19.8 min) to 12.6 min (IQR: 11.0–13.8 min) (p < 0.001), median total errors decreased from 5.5 (IQR: 3.5–7.5) to 2 (IQR: 1.5–2.5) (p < 0.001), and median total OSATS increased from 61 (IQR: 53–66) to 75 (IQR: 70.5–80.5) (p < 0.001). When considering only the PGY4s, improved performance was observed on each inanimate task according to all metrics except time on RRC (pretest: 5.0 min [IQR: 4.0–5.7 min]; posttest: 4.3 min [IQR: 3.3–5.2 min]; p = 0.756). Improvement was observed in the remaining RRC metrics and all metrics for the other 2 tasks (all p < 0.025). For the PGY4s, median total time to completion for all 3 tasks decreased from 14.8 min (IQR: 13.2–15.6 min) to 11.7 min (IQR: 10.1–14.3) (p = 0.014), median total errors decreased from 4.8 (IQR: 2.9–7.8) to 2.3 (IQR: 1–3.1) (p < 0.001), and median total OSATS increased from 60.6 (IQR: 56.4–66.1) to 70.3 (IQR: 66.6–80.8) (p < 0.001). Figure 5 shows a comparison of pretest and posttest performance on the inanimate tasks for the entire group, the PGY2s alone, and the PGY4s alone.

Fig. 5

Comparison of inanimate tasks performance at pretest and posttest. A All residents; B PGY2s; C PGY4s. The scores shown are averages of the scores assigned by each grader

Statistically significant differences were observed between the PGY2s and PGY4s on the pretest for median total time (p = 0.006), but no differences were observed between the two groups on the posttest. There were no significant differences observed between the PGY2s and PGY4s on either the pretest or the posttest for total errors (pretest: p = 0.531; posttest: p = 0.433) or total OSATS (pretest: p = 0.433; posttest: p = 0.637). Figure 6 shows a comparison of PGY2 and PGY4 performance on the inanimate tasks at pretest and posttest.

Fig. 6

Comparison of inanimate tasks performance between PGY2s and PGY4s at pretest and posttest. A PGY2s vs PGY4s at pretest; B PGY2s vs PGY4s at posttest. The scores shown are averages of the OSATS scores assigned by each grader

Discussion

Moving the initial phases of robotic surgical skills training from the operating room to the simulation center is beneficial to shorten the learning curve, minimize OR times and cost, and enhance patient safety. Our simulation center is equipped with a dual-console Xi System and the SimNow® VR platform. We designed our curriculum to use the VR system in an effort to allow surgery residents to acquire basic robotic skills without needing to use actual robotic equipment, which is associated with additional costs for instruments and supplies. Publications describing curricula using the SimNow® system are lacking; we therefore, opted to use methodology previously described for a curriculum using a similar VR system and inanimate transferability tasks [15].Our study demonstrated that PGY2 and PGY4 residents with minimal RAS exposure can effectively train on the SimNow® VR platform to improve their skills in both the virtual and inanimate environments in a reasonable amount of time. This transferability of skills from a virtual to a real environment confirms the positive implications of the use of VR simulation as a major initial component of robotic surgical training.

Eighty-three percent of the residents were able to achieve mastery on every task in the curriculum. This is a relatively high completion rate. Hogg et al. enrolled 17 surgical oncology fellows to undergo a similar VR training curriculum with 94% of completion rate, yet only 24% were able to achieve mastery on every task. These results were attributed to the fact that achieving proficiency was not mandatory in order to proceed with the post-test [15]. Another trial by Kiely et al. enrolled 14 residents and attendings to complete a dVss VR training curriculum but had a completion rate of 36%. This low completion rate was attributed to the short training times available for some of the participants [24]. Several factors likely were responsible for our high curriculum completion rate. Having dedicated training equipment that was accessible 24/7 in the simulation center made access readily available without using clinical equipment. Having protected training time as well as a structured curriculum with clear expectations seemed pivotal as well. Indeed, most residents completed the curriculum in approximately one month, which corresponded to the duration of the clinical rotation we selected for scheduling of this training due to its relatively light clinical demands. However, 31% of residents required more than 4 months to complete the curriculum; but many of these residents either had their training interrupted by a mandatory shutdown of the simulation center due to the COVID-19 pandemic or elected to begin the curriculum before their designated rotation with protected time. Median completion time was 7 h. Considering all these aspects, our data support feasibility of this VR curriculum.

Our data showed that the SimNow® training platform was effective at improving performance in the VR environment. Statistically significant improvements from baseline were observed at the posttest in all metrics on each task. Median total score on all 4 VR drills increased by more than 4 times (83 to 353), median total time halved (26 to 13 min), and median total economy of motion considerably decreased (2250 to 1400 cm). These results support the use of a proficiency-based curriculum that challenges trainees to reach a certain level of expertise rather than to simply perform a pre-determined number of repetitions. Another advantage of this platform is that it gives residents the opportunity to practice using the actual da Vinci Xi® surgeon’s console, introducing them to the unique ergonomics of this robotic system, and preparing them to use the same system in real environments.

Importantly, we found that skills acquired through VR training were transferable to the inanimate environment. This finding has been previously described for prior robotic VR platforms [25, 26] but no studies had examined transferability for the SimNow® platform. After performing VR training, residents showed major improvements in total time, total errors, and total OSATS on the inanimate drills; these improvements were demonstrated when analyzed as one cohort as well as when analyzed as individual classes. While the PGY4s had significantly lower completion time compared to PGY2s on the pretest, both groups demonstrated improvement, and the PGY difference was not present on the posttest. Specific analysis of PGY2 performance showed significant improvement for all metrics on all three inanimate tasks. Similar results were found for the PGY4s, except for a nonsignificant improvement in time for RRC. The fact that both junior and senior residents had a comparable performance at completion of the VR curriculum suggests that skill acquisition was independent of the previous level of clinical training. These data further support early implementation of robotic surgical training during residency.

This study has several limitations. First, this study was a retrospective review of data collected for quality improvement purposes and did not follow a prospective experimental design. Nevertheless, our study involved a larger number of participants than previously published studies and represents new data for this simulator; additionally, many of our findings were consistent with results associated with other VR systems [15, 27–30]. Second, our curriculum used automated metrics produced by the VR system but these metrics have not yet been investigated for validity evidence. Given the positive results of our initial experience, we intend to pursue such studies. Moreover, the passing score used was a pre-defined benchmark of overall score > 90%, established by the manufacturer, SimNow® (Intuitive Surgical Operations, Inc.; Sunnyvale, California). There isn’t previously reported data or publicly available information regarding these passing thresholds. This value does not seem to be equivalent and consistent for all tasks. Consequently, our team will be working in the future to define the content and construct validity of each individual VR drill. Third, transferability of skills was studied on the inanimate environment and might not correlate to the real operating room environment. Future endeavors in this direction must be taken to understand how VR simulation improves trainees’ skills while operating. Lastly, this was a single-institution study and replication of these results at other institutions will be important to establish generalizability of our findings.

In conclusion, this is the first study to evaluate feasibility, effectiveness, and transferability of the new SimNow® VR platform. Our findings documented that completion of the mastery-based VR training curriculum using this platform was feasible for a large majority of learners in a reasonable amount of time and was effective in significantly improving robotic skills that were transferable to a real robotic environment. Additional validation studies may allow further refinement of this VR robotic curriculum.