What is going on in augmented reality simulation in laparoscopic surgery?

BACKGROUND: To prevent unnecessary errors and adverse results of laparoscopic surgery, proper training is of paramount importance. A safe way to train surgeons for laparoscopic skills is simulation. For this purpose traditional box trainers are often used, however they lack objective assessment of performance. Virtual reality laparoscopic simulators assess performance, but lack realistic haptic feedback. Augmented reality (AR) combines a virtual reality (VR) setting with real physical materials, instruments, and feedback. This article presents the current developments in augmented reality laparoscopic simulation.
METHODS: Pubmed searches were performed to identify articles regarding surgical simulation and augmented reality. Identified companies manufacturing an AR laparoscopic simulator received the same questionnaire referring to the features of the simulator.
RESULTS: Seven simulators that fitted the definition of augmented reality were identified during the literature search. Five of the approached manufacturers returned a completed questionnaire, of which one simulator appeared to be VR and was therefore not applicable for this review.
CONCLUSION: Several augmented reality simulators have been developed over the past few years and they are improving rapidly. We recommend the development of AR laparoscopic simulators for component tasks of procedural training. AR simulators should be implemented in current laparoscopic training curricula, in particular for laparoscopic suturing training.

Minimally invasive surgery (MIS) has been accepted worldwide as the main treatment approach for many various pathologies, because of its known advantages over open procedures. However, performing laparoscopic procedures demands very specific capabilities of the surgeon, which can only be gained through extensive training [1]. To master these skills, the trainee needs to develop an understanding of the spatial relationship and the related hand manoeuvres required to manipulate instruments and tissue in a two-dimensional video rendering of a three-dimensional operation field. Developing these skills before entering an operating room enables more focused and efficient performance, which minimizes time in the operating room and enhances patient safety [2, 3]. For this purpose multiple surgical simulation systems became available to train laparoscopic skills prior to performing actual surgery in the clinical setting.

The different kinds of simulators used for training purposes are: traditional box trainers, virtual reality (VR), and augmented reality (AR) simulators. Traditional box trainers have realistic haptic feedback during procedures, but an expert observer must be on hand to assess performance. VR simulators provide explanations of the tasks to be practised and objective assessment of the performance; however they lack realistic haptic feedback. AR simulators retain realistic haptic feedback and provide objective assessment of the performance of the trainee.

Previous studies [4-8] have shown that realistic haptic feedback is fundamental for good laparoscopic training and results in significantly improved skills transfer to the trainee compared with training without haptic feedback [4, 5]. A simulation system that provides unbiased and objective assessment of performance (rather than just speed) could help training, complement knowledge-based examinations, and provide a benchmark for certification [1].

Augmented reality laparoscopic simulators provide both realistic haptic feedback and objective assessment after the performance. By retaining both of these important training properties in this simulator system, these could be potent training tools for current surgical training curricula. Therefore this study provides an overview of the augmented reality simulation technique and the available simulators.

Methods

Pubmed searches were performed to identify articles with combinations of the following key words: “laparoscopic”, “simulation”, “trainers”, “Augmented Reality”, and “hybrid”. Further articles were obtained by manually searching the reference lists of the identified papers.

The identified companies or research groups that we found to have produced an augmented reality laparoscopic simulator were asked to participate in this study. They each received the same questionnaire asking for a description and features of their augmented reality laparoscopic simulator. The items in the questionnaire covered: features, modules and tested skills, properties for assessments, haptic (force) feedback, most important aspects, and shortcomings. The final part of the questionnaire contained questions on validation of their simulator and costs of the hardware and software.

Results

Simulators

Seven simulators that fitted the definition of augmented reality were identified during the literature search. All of the corresponding manufacturers or research groups were approached to complete the questionnaire, asking them to cooperate with this study and inform us about the features of their simulator. Five of the approached producers returned a completed questionnaire; one simulator appeared to be VR and was therefore not applicable for this overview. The results of the returned questionnaires are summarized in Tables 1 and 2.

Table 1

Features of the augmented reality simulators

Feature		ProMIS	Blue Dragon	CELTS	LTS3e
Modules and tasks of the simulator	Basic skills:
	○ Navigation/ coordination	X	X	X	X
	○ Touching	X	X	X	X
	○ Grasping	X	X	X	X
	○ Stretching/ traction	X	X	X	X
	○ Translocation	X	X	–	X
	○ Other:		All laparoscopic skills can be measured
	Advanced skills:
	○ Clip application	X	X	–*	–
	○ Transection/ cutting	X	X	–*	X
	○ Dissection	X	X	–*	–
	○ Diathermia	X	X	–	–
	○ Suturing	X	X	X	X
	○ Knot tying	X	X	X	X
	○ Other:	Hand-assisted/laparoscopic colectomy	All procedural component tasks		Cannulation
Recorded parameters	○ Time	X	X	X	X
	○ Path length	X	X	X	–
	○ Smoothness	X	X	X	–
	○ Economy of movement	X	–	X	–
	○ Errors	X	–	–	X
	○ Other:	Hand dominance	Tool/tissue interaction. Opening/closing of instrument	Instrument orientation, ambidexterity
Feedback	○ Progression curve of recorded parameters
	○ Real playback of the task	X	–	X	X
	○ Virtual playback of the task	X	–	–	–
	○ Other:	X	–	–	–
			Overview of measurements
Need for observer	○ Is an ‘expert’ observer needed for evaluation of the performance of the tasks?	No	No	No	Yes
	○ An “expert” observer is only needed for feedback/ help with problems?	Yes	Yes	Yes	Yes
	○ Trainees can train and evaluate modules without an “expert” observer.	Yes	Yes	Yes	Yes
Instructions	○ Written instruction of the task on the screen	Yes
	○ Demonstration video	Yes	No	No	Yes
	○ Spoken instruction during the task	Yes	Yes	No	Yes
	○ Guiding lines on the screen during the task	Yes	No	No	No
	Other:	Animation to illustrate the task	No	No	No
Validation	Is the simulator completely validated? If not, what part is?	Yes	Yes	No	Under research

Features of the augmented reality simulator according to their manufacturers

* Clipping and transsection could be performed with small changes to the simulated skin, but are not part of the original skills set

Table 2

Assessment methods and important aspect of the augmented reality simulators

Feature	ProMIS	Blue Dragon	CELTS	LTS3e
Assessment	• Time	Markov Model	• Time	• McGill metrics
	• Path length	• Force	• Ambidexterity,
	• Economy of movement	• Tension	• Instrument orientation,
	• Hand dominance	• Velocity	• Economy of motion
	• Task-specific errors
	• Self-assessment form
Most important aspects	• Combination of physical reality and virtual reality in the same unit.	• Objective assessment of MIS skills using the Markov model	• Automatically record performance	• Realistic physical exercises
	• Enables real instruments and real haptics		• Use of real instruments	• Knot integrity exercise using electronic tensiometer
	• Can adapt to a curriculum		• Allowing knot to be assessed
Shortcomings	None	No progression curve of the performance assessment	No commercial partner to date	Absence of anatomical representations

Assessment methods and important aspect of the augmented reality simulator according to their manufacturers

ProMIS

ProMIS combines the virtual and real worlds in the same system: users learn, practise and measure their proficiency with real instruments on physical and virtual models. It comprises a number of modules designed to develop and evaluate surgical proficiency. The simulator comprises a mannequin-type bodyform linked to a laptop computer (running Windows XP). Inside the bodyform, a vision-tracking system enables tracking and measuring of real surgical instruments (and hand movements). By marking each instrument, the vision tracking system can identify the position, direction and velocity of the left instrument, right instrument and camera at any time. There is an unlimited degree of freedom and tactile feedback while performing tasks. The training modules may be physical tasks on “trays”, VR tasks or a combination of both. Real instruments, trocars and port placement are used on physical tissue.

The metrics and assessment (Tables 1 and 2) presented are based on data gathered by the tracking cameras. There are learner and group management tools to follow the progress of the trainee.

The core bodyform unit has inner and outer molded torso casings, between which a model “skin” is placed. There is a sliding drawer on the front side for the placement of the trays. A universal serial bus (USB) foot pedal is used during the performance of the task, to go onto the next step. The base has the option to tilt, enabling the bodyform to be tilted forwards and backwards by up to 45º. The Dell XPS laptop computer, connected to the ProMIS bodyform, runs Windows XP, and has a 15” screen or equivalent, with a hard-drive of at least 60 GB, 1 GB random-access memory (RAM), and a 6800 GFORCE TOGO graphics card.

CELTS

The computer-enhanced laparoscopic training system (CELTS) is a prototype laparoscopic surgery simulator that uses real instruments, real video display and laparoscopic light sources with synthetic skin and task trays to permit highly realistic practice of basic surgical skills. Since instruments and displays are real, actual suturing can be performed without the need to create software models of suture or needle behaviour, for instance. An embedded metrics algorithm automatically scores each user for both right and left hand on five critical indicators of surgical skills.

A five-point graphical scale of trainee performance is used to compare with expert performance, using an automatic integral algorithm. The database is infinitely expandable so statistical variation/reliability improve with each use. Performance is measured longitudinally using trainee log-in data and all performance data can be transmitted wirelessly to faculty mentor using integral transmission hardware.

LTS3e

The LTS3-e (LTS) is a relatively low-cost augmented reality simulator capable of training and assessment of technical laparoscopic skills of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Fundamentals of Laproscopy (FLS) program. The LTS3-e is essentially an electronic evolution of the McGill inanimate system for training and evaluation of laparoscopic skills (MISTELS) and offers a few more tasks. It provides validated physical reality exercises assessed electronically with validated McGill metrics. The system possesses sensors embedded in physical modules, which capture performance data, permitting computer-based electronic scoring. The software can store user information and compare performance over time. The transportable unit (dimensions 50 × 37 × 27 cm, weight 15 kg) consists of an enclosure containing a built-in PC (running Windows XP Professional, Pentium processor, 100 GB hard drive, DVD-RW drive, 120–240 VAC power), video display (15” screen, 1024 × 768 display), electronic carousel, digital video camera, cold light illuminator, DVD, and wireless keyboard. The LTS also has a tensiometer to verify knot security with a disruptive force of 1 kg in one of the suturing exercises.

Blue DRAGON

The Blue DRAGON (University of Washington, Seattle, WA, USA) is a system for acquiring the kinematics and the dynamics of two endoscopic tools along with the visual view of the surgical scene. This is an assessment method for the performance when training in a realistic setting, e.g. on a box trainer, animal model or clinical setting. The assessment of the performance is based on the placement of the instrument and the tool–tissue interaction during the task. This research group has recently produced a new prototype simulator system, the Red Dragon [9], which is not only for the simulation of laparoscopic skills, but can also be used for assessment in the clinical setting. This simulator has not yet been validated, but will be commercially available, produced by Simulab, under the name “Edge”.

The assessment of Blue Dragon is based on the Markov model [10-18], decomposing a surgical task into symmetric finite states (28 states) where the left and right hands are represented by 14 states each. These states correspond to a fundamental tool–tissue interaction based on the tool kinematics and associated with unique force, torque and velocities signatures. These measurements are given at the end of the performance as an overview in a table or as a three-dimensional (3-D) graphic of the path travelled by the instruments. In addition to data acquisition, the synchronized view of the surgical scene is incorporated into a graphical user interface that displays the data in real time.

The Blue Dragon includes two four-bar passive mechanisms attached to real laparoscopic tools, translating the laparoscopic tool’s rotation in the ports. These mechanisms are equipped with three classes of sensors: position sensors (multiturn potentiometers, Midori America Corp.) for measuring the positions, orientations and translation of the two tools attached to them. In addition, two linear potentiometer (Penny & Giles Controls Ltd.) measure the laparoscopic handle and tool tip angles during the performance. Three-axis force/torque sensors (ATIMini sensor) are located at the proximal end of the laparoscopic tools’ shaft, and inserted into the tools’, handles providing binary indication of any tool–tissue contact.

Discussion

Augmented reality laparoscopic simulation

Augmented reality is a term also used in diagnostic and treatment techniques, where an overlay of the anatomy can be given, or visual cues of specific landmarks, which were previously scanned with computed tomography (CT) or magnetic resonance imaging (MRI). In this study we focus only on augmented reality in laparoscopic simulation.

Augmented reality is the essential link connecting the virtual with the real world. Virtual information is added to the real world. Augmented reality arose from the need to exploit optimally virtual data coming from simulations. Augmented reality simulation is the combination of physical (real) and virtual reality in one system (Fig. 1). This enhancement of physical training in laparoscopic simulation can be accomplished with overlays of anatomical representations or by objective assessment at the end of the performance. Another approach to augmented reality is the visual pathway of the instruments, which can be shown during playback of performance.

Fig. 1

Properties of the different simulation techniques used in laparoscopic training

A major advantage of the AR laparoscopic simulator over the VR simulator is that it allows the trainee to use the same instruments that are currently used in the operating room. The simulator provides realistic haptic feedback because of the hybrid mannequin environment in which the trainee is working, which is absent in VR systems. This simulator offers a physically realistic training environment that is based on real instruments interacting with real objects.

The physical task is combined with demonstration videos on the screen, and the performance of the trainee is recorded for subsequent replay. Because AR simulators are a learning system on their own, there is no need for an expert laparoscopic surgeon to be on the scene to guide the trainee. Therefore AR simulation is a good way for trainees to practise their laparoscopic skills in their free time.

Validation of AR simulators

Multiple studies [19-23] have been published validating the effectiveness of ProMIS in training and assessing laparoscopic skill. ProMIS has shown construct validity for orientation, dissection and basic suturing tasks in several independent studies [20, 22, 23]. Face validity has also been shown in the study by Botden et al. [24], in which surgeons give favourable ratings to the suturing module with regards to realism and haptic feedback. Other articles have been published using the ProMIS for general skills acquisition [25] and comparing ProMIS AR with VR simulation [22]. In a comparison of AR with VR laparoscopic suturing [22], the ProMIS AR simulator was preferred by far over the VR simulator for training suturing skills. The study by Nerula et al. investigated the assessment system of the ProMIS simulator for assessing skills with robotic instruments [19]. This shows the wide variety of teaching laparoscopic skills the ProMIS can be used for.

The CELTS simulator has not been thoroughly validated because the focus of this research group is on the development and improvement of new simulator systems. However, several studies have been published introducing the CELTS laparoscopic simulator, showing construct and some face validity [26-29]. Maithel et al. showed construct validity of the CELTS simulator by comparing the assessment of the performance of junior and senior residents. They concluded that computer-enhanced video trainers (augmented reality) may offer an improved interface while incorporating useful multidimensional metrics, but that further work is needed to establish standards for appropriate skills assessment methods and performance levels for using these simulators [26, 27]. Stylopoulos et al. concluded from their studies that the CELTS simulator provided educational feedback by identifying key factors, such as depth perception, smoothness of motion and instrument orientation, which contributed to the overall score. Assessment based on these parameters could distinguish the trainee from the expert [28, 29].

Currently, no studies have been published on the LTS–3e laparoscopic simulator, because validation research is still in progress.

There are several studies [10, 11, 13–16, 18] showing the usability of the Markov model during laparoscopic training, using the Blue Dragon laparoscopic simulator. Rosen et al. have researched the Markov model within the assessment system extensively and concluded that the major differences between different skills levels were shown in terms of the types of tool–tissue interactions being used, transitions between the tool–tissue interactions being applied by each hand, the time spent performing each tool–tissue interaction; overall completion time, and the variable force/torque magnitudes being applied by the subjects trough the laparoscopic instruments [10, 15–17].

Benefits of augmented reality simulation

As shown in this overview, several types of augmented reality simulators currently are on the market, ranging from relatively simple box trainers with a separate assessment method to more advanced simulators with demonstration videos, overlays during the performance and the essential assessment of the performance. Still, there are further improvements that could be made to make these simulator systems more suitable and complete for implementation in current training curricula for laparoscopy. Demonstration videos and the provision of formative feedback during training could help surgical residents more to train their laparoscopic skills. In current training on traditional box trainers an expert observer must be on hand to provide feedback and assess the performance. Both VR and AR systems provide objective measurements of performance, but lack meaningful assessment protocols. However, AR simulators additionally offer realistic haptic feedback. For laparoscopic suturing training, for example, AR is the best choice for a simulation system, as haptic feedback during practice is mandatory for good skills transfer to the trainee [7, 30–33], and providing feedback will guide and motivate trainees to practise these difficult laparoscopic skills until they have reached specific goals [34-37].

Augmented reality simulation has great potential in the training of component tasks of procedural training, especially for procedures that require realistic haptic feedback during training. Such procedures are bariatric surgery and colon surgery, in which anastomoses are frequently made and therefore suturing skills are necessary.

Cost efficacy

The costs of both the hardware platform and the software of an AR simulator are comparable to the costs of a VR simulator, as VR simulators have become less expensive over recent years. This results in a tendency for the costs of AR and VR simulator systems to equalize. The costs of an AR simulator can be divided into three parts: the hardware platform, the software packages and consumables on which to practise the tasks. Both AR and VR simulators offer package deals for several software modules together with the corresponding hardware and, for AR, consumables. In AR, however, costs of consumables vary considerably, depending on the module for which they are designed, ranging from suturing tissue to abdominal landscapes for colon surgery. Therefore, the costs of an AR simulator strongly depend on the modules one desires to practise in the laparoscopic training curricula.

Conclusion

Several augmented reality simulators have been developed over the recent years, and they are improving rapidly. The advantage of AR over VR is that they offer realistic haptic feedback, like traditional box trainers, while additionally providing objective assessment of performance. Our recommendation for the future is the development of augmented reality laparoscopic simulators for component tasks of procedural training such as laparoscopic suturing, and improvement of the assessment methods. For basic skills, however, VR has previously been proven a valid training method.

Augmented reality simulators are a potent new modality of laparoscopic simulator system that should be implemented in current laparoscopic training curricula.