Usability assessment of Versius, a new robot-assisted surgical device for use in minimal access surgery.

OBJECTIVES: Versius is a teleoperated surgical robotic system intended for use in minimal access surgery. This study aimed to validate the safety and effectiveness of the Versius user interface in the hands of trained users and identify and address the causes of any user errors.
DESIGN: Surgical teams completed a commercially representative training program over 3.5 days. After training was completed, the usability-related aspects of the system were assessed.
SETTING: A simulated operating room using a cadaveric model. PARTICIPANTS: Surgical teams consisting of a lead surgeon, assistant surgeon, scrub nurse and circulating nurse. MAIN OUTCOME MEASURES: Usability-related aspects of the system were assessed through the completion of critical and non-critical tasks. A critical task was defined according to the Food and Drug Administration's definition, as a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care.
RESULTS: In total, 17 surgical teams participated in the study and all were experienced in laparoscopic surgery. The number of robotic surgeries performed by the participants per month ranged from 0 to 100. Surgical specialties were similarly represented from obstetrics and gynecology, colorectal, urology and upper gastrointestinal. No critical task failures were observed. Of all the tasks completed, 98% were recorded as a pass or a pass with difficulty.
CONCLUSIONS: These results demonstrate that in a simulated clinical setting, Versius can be safely used by both laparoscopically and robotically trained healthcare professionals. These results support the progression to assessment of Versius in preclinical studies. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Minimal access surgery (MAS) is associated with a number of benefits over open surgery but is difficult to perform and has a steep learning curve.

A surgical robot system, Versius, has been developed that aims to address some of the barriers to MAS uptake by incorporating feedback from end-users throughout the design process.

The usability of the new system has been validated and can be safely used by both laparoscopically and robotically trained healthcare professionals.

It is hoped that the usability of the system will reduce the barriers to MAS and there will be greater uptake of the technique.

The results from this study have led to several, minor remedial alterations, including hardware component upgrades associated with the head-up display and modification of training protocols.

The next steps of preclinical testing will evaluate safety and effectiveness of Versius when performing a range of procedures across a number of surgical specialties in cadavers and live animal models.

Introduction

Minimal access surgery (MAS) offers several advantages over open surgery, including reduced postoperative pain, less blood loss, a reduced risk of postoperative complications and infections leading to a reduction in hospital stay.1 2 Although advantageous for patients, MAS procedures are associated with a prolonged learning curve and impose significant biomechanical and cognitive strain on the surgeon. Compared with surgeons performing open surgery, surgeons performing MAS are more likely to experience muscle fatigue and injury, particularly of the upper limbs, head and neck.3–5 Robot-assisted laparoscopy aims to address these difficulties, for example, by improving ergonomics and reducing the physical stress for surgeons while also maintaining the benefits of MAS afforded to the patient.6–8

Versius is a new teleoperated surgical robotic system intended for use in performing robot-assisted MAS on a patient in an operating theater. The system comprises the Versius surgeon console, a Versius visualization bedside unit (BSU), up to four Versius instrument BSUs, Versius endoscopes and camera, instruments, cables and sterile drapes. The surgeon interacts with the system through the hand controllers and feedback on the surgeon console, including the surgeon head-up display (HUD), which displays the three-dimensional (3D) video from the endoscopic camera together with a display overlay (figure 1). The bedside team access controls and feedback on the visualization and instrument BSUs, and view a two-dimensional version of the endoscope feed and display overlay on an auxiliary display.

Figure 1

Overview of the Versius surgical robotic system. (A) Schematic overview of Versius. (B) An image of the operational setup of Versius. BSU, bedside unit.

Versius was developed with the user and patient central to the design. Throughout the development of Versius, end-user feedback was used to refine the design to ensure it met user needs. Versius has been designed to mimic the articulation of the human arm, which together with the wristed joint of the instruments provide 7 degrees of freedom at the instrument tip, allowing greater surgical access compared with standard laparoscopic surgery.9 The open console was designed to allow easy communication between the surgeon and their team, supporting the execution of non-operative tasks, overcoming the challenges associated with a closed console design.10 The system was also designed with improved ergonomics; for example, the hand controllers and console were designed to accommodate a range of operating hand sizes and allow the surgeon to reorient their hands to an ergonomic position, regardless of operating angle, while the system provides a flexible working environment (surgeons can sit or stand). Similar consideration was also given to the cognitive and sensory demands on the surgeon. For example, the console screen height adjustment provides better posture but also better perception of the 3D feature.

Currently available robot-assisted surgical systems can require specialized infrastructure (eg, specifically designed operating theaters), and therefore can limit the flexibility with which they are deployed.11 Versius is a modular system; each instrument and visualization arm is attached to its own wheeled cart to form a compact and mobile BSU, removing the need for dedicated theaters and making it easy to move between operating rooms. The modest size and mobility of the system also makes the adoption of hybrid manual-robotic procedures simple, which could help to reduce conversion time from robotic to manual/open surgery in an emergency.

Usability testing help reveal opportunities to make medical devices easier, safer, and more efficient to use. It serves as a primary means of demonstrating that the intended users of a medical device can safely and effectively perform critical tasks for the intended uses in the expected use environments.12 Safety, in the context of this study relates to use safety and is defined as: ‘freedom from unacceptable use-related risk’, as per the Food and Drug Administration’s (FDA) definition.13 As outlined by the FDA, an essential component to usability testing and validation is the comprehensive identification and categorization of user tasks, leading to a list of critical tasks. They are defined as tasks, that if performed incorrectly or not performed at all, would or could cause serious harm. Thus, usability testing should be designed (1) so that all critical tasks are performed during the evaluation, (2) the test participants represent the intended users of the device, (3) the device user interface represents the final design and (4) the test conditions are sufficiently realistic to represent actual conditions of use. Furthermore, the study should facilitate discussion of the critical tasks in order to perform analyses of the root causes of any identified use errors.13

This study was designed in line with recommendations from the FDA on applying human factors and usability engineering to medical devices13 and the IDEAL (Idea, Development, Exploration, Assessment, Long-term study) collaboration for surgical research.14 The study aligned with Stage 0 of the IDEAL-D framework and aimed to: (1) validate the safety and effectiveness of the Versius user interface, in the hands of trained users and (2) identify and address the causes of any user errors.

Methods

Study design

The study was structured to include evaluation after a representative training program for Versius. All study participants completed a 3.5-day long residential training program, representative of commercial training. The study took place at the AdventHealth Nicholson Center where training took place in a classroom setting initially and then in a simulated clinical operating theater environment. Following training, study participants completed an evaluation exercise by running through a simulated surgical workflow (online supplementary figure 1). The study employed a simulated use cadaveric method, using cadavers with a range of body mass indices. The study moderator prompted the participants through the workflow. Data collection was mostly observational; as participants worked through the use scenarios they were assessed against 288 predefined critical tasks. Although some scenarios were staged by the study moderator, the surgical teams operated autonomously and worked through surgical flows on their own initiative.

Tasks were identified and predefined through a risk management procedure, which included a use failure modes and effects analysis (UFMEA), based on the international standard, ‘ISO 14971:2019 Medical devices – Application of risk management to medical devices’. Briefly, risk is defined as the combination of the probability of occurrence of harm and the severity of that harm (ranked as negligible, through minor, serious, critical, to catastrophic). However, in the UFMEA the criticality of tasks were based on severity only, such that, even if occurrence is ‘improbable’ but severity is greater than ‘negligible’, the task is critical because it has the potential to cause harm if not performed or performed incorrectly, as per the FDA’s definition.13

Study participants

The recruitment for the surgical team members was blinded and completed by an independent recruitment specialist (Schlesinger Group). Each surgical team constituted four different users who had never previously worked together: a lead surgeon (S) representing a board certified lead surgeon, an assistant surgeon (AS) representing a suitably and medically qualified bedside assistant, a scrub nurse (SN) representing a member of nursing staff operating within the sterile field, and a circulating nurse (CN) representing a member of nursing or technical staff operating outside the sterile field. In compliance with the American National Standard published by the Association for the Advancement of Medical Instrumentation HE75: 2009 (HE75), a minimum of 15 participants from each distinct user group were recruited.12

Participants were selected from surgical specialties that would reflect all groups likely to use Versius. These included: upper gastrointestinal (GI), obstetrics and gynecology (OB/GYN), urology and colorectal. All participants had previous experience in laparoscopic surgery while experience with robotic surgery varied.

Task assessment

Participant’s actions and reactions were captured using a study data capture matrix. Each surgical team member was assessed through the completion of role-specific tasks categorized as either critical or non-critical for safe and effective use (online supplementary table 1). A critical task was defined as per FDA guidance: A user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care.13 Tasks were divided into 20 subgroups, called scenarios, based on the type of tasks being assessed (online supplementary table 1). Completed tasks were recorded as pass (ie, acceptable completion of task), pass with difficulties or use error (ie, failed to acceptably complete the task). A task failure was considered as such, only if a task was failed by more than four users in the same user group. This provides a 95% confidence limit that 90% of users, in the same user group, can perform all critical tasks successfully following completion of a representative training course (as set out in the HE75 guidelines).12 Audio and video capture were used during the study for later review and analysis to ensure all use errors were captured (with full participant consent). A knowledge test was used for tasks that were difficult to observe and/or simulate. For a complete list of the tasks assessed, see online supplementary table 2).

Following completion of the workflow, participants were interviewed on a one-to-one basis. Observers asked about participants’ perception on their use of the system, discussed any difficulties and use errors observed, in order to establish the root cause of these. The observers had an opportunity to gather the participants’ overall impression of the device regarding ease or difficulty of use and their ability to complete critical tasks without difficulty.

Study observers

Each study participant was closely monitored by a study observer, who was trained and led by experienced human factors engineers. Observers were trained over 10 sessions, which included the use of data capture matrices, integrity and accuracy of data, reviewing all tasks, anticipated use errors and acceptance criteria, interviewing techniques and root cause analysis. Every observer was required to complete and pass an assessment at the end of each training session.

During observation, observers used the data capture matrix to collect information on each task relevant to the specific surgical team member they were observing. The design of the capture matrix was tested and refined in previous studies. All observers stood quietly in the simulated environment collecting data. At points during the simulation, the moderator called for short breaks so that observers could consolidate the data for that part of the simulation, which were video recorded from two different angles.

Patient and public involvement

Patients were not involved in the design of this study or the drafting of this manuscript as this study describes the technical development and preclinical testing of a robotic surgical system.

Ethical approval for cadaver study

The study was conducted at the AdventHealth Nicholson Center, 404 Celebration Pl, Celebration, FL 34747, USA. The Nicholson Center is a licensed institution and CMR Surgical complied with all local regulatory and institutional ethical requirements when conducting this study and using cadavers.

Results

Participant demographics

In total, 17 surgical teams (T1 to T17) completed the training program and were subsequently evaluated during the validation day (table 1). Surgical specialties were similarly represented, with four OB/GYN, colorectal, and urology lead surgeons recruited, and five upper GI specialists. Most participants (58/68; 85.29%) had some level of previous experience with robotic surgery while 10/68 (14.71%) had no previous experience (data unavailable for one CN). The number of robotic surgeries performed by the participants per month ranged from 0 to 100 and all participants were experienced in laparoscopic surgery.

Table 1

Surgical team demographics

	Surgeon	Assistant surgeon	Scrub nurse	Circulating nurse
Specialty, n (%)
OB/GYN	4 (24)	–	–	–
Upper GI	5 (29)	–	–	–
Colorectal	4 (24)	–	–	–
Urology	4 (24)	–	–	–
Laparoscopic procedure per month, n (%)
0–9	5 (29)	7 (41)	1 (6)	0 (0)
10–20	8 (47)	4 (24)	8 (47)	5 (33)
21–30	3 (18)	3 (18)	2 (12)	3 (20)
>31	1 (6)	3 (18)	6 (35)	7 (47)
Robotic experience, n (%)	13 (76)	15 (88)	17 (100)	13 (87)
Robotic procedure per month, n (%)
0–9	12 (71)	7 (41)	6 (35)	6 (38)
10–20	3 (18)	5 (29)	5 (29)	4 (25)
21–30	2 (12)	2 (12)	3 (18)	2 (13)
>31	0 (0)	3 (18)	3 (18)	4 (25)
Glove size, median (min–max)	7.5 (6–8)	6.5 (6–8.5)	6.5 (6–8)	7 (5.5–7.5)
Gender, n (%)
Male	12 (71)	5 (29)	3 (18)	3 (19)
Female	5 (29)	12 (71)	14 (82)	13 (81)
Height (cm), mean (SD)	176.93 (10.44)	166.37 (8.32)	169.53 (7.31)	164.68 (9.78)
Handedness, n (%)
Left	1 (6)	0 (0)	0 (0)	0 (0)
Right	15 (88)	17 (100)	14 (82)	15 (94)
Ambidextrous	1 (6)	0 (0)	3 (18)	1 (6)
Hearing issues, n (%)
Yes	2 (12)	0 (0)	0 (0)	1 (6)
No	15 (88)	17 (100)	17 (100)	15 (94)
Wears glasses or lenses, n (%)
Yes	9 (53)	6 (35)	10 (59)	9 (56)
No	8 (47)	11 (65)	7 (41)	7 (44)
Color blind, n (%)
Yes	2 (12)	2 (12)	1 (6)	0 (0)
No	15 (88)	15 (88)	16 (94)	16 (100)
Issues with hands, n (%)
Yes	0 (0)	1 (6)	0 (0)	0 (0)
No	17 (100)	16 (94)	17 (100)	16 (100)

GI, gastrointestinal; OB/GYN, obstetrics and gynecology.

Overview of task completion

Surgical teams performed 11,633 tasks in total. Of these, 7501 were critical for safe and effective use of Versius, while 4132 were non-critical (figure 2A). No critical task failures were observed (ie, no more than four use errors were observed on any critical task in each user group). Of all the tasks completed, 98% were recorded as a pass or a pass with difficulty (figure 2B). Across all user groups, 174 use errors were identified during simulated use and drills but did not constitute any task failures.

Figure 2

Summary of tasks undertaken during simulated use and drills. (A) In total 11,633 tasks were completed between the 17 surgical teams, representing 7501 critical and 4132 non-critical tasks. (B) Summary of task completion by user group. A 95% confidence limit was met, with greater than 90% of users within the same user group successfully completing all tasks.

Root cause analysis of use errors

Critical tasks associated with most use errors are detailed in table 2. The largest source of error (15 use errors) stemmed from participants not mentioning, when asked, the possibility of trapping thin cables when the BSU skirt is raised (Task 76: What should you be careful of when unbreaking the BSU?). Although this task was classified as critical, this specific use error was not considered critical in the risk analysis. During the simulated use and drills, the most frequent use error (eight use errors) when combining all user groups, was failing to maintain sterility; followed by six use errors for participants failing to apply the brake to the carts after pushing all BSUs aside due to a simulated console alarm. Again, this use error was not critical, despite being part of a critical task. For all root cause analyses of use errors see online supplementary table 3.

Table 2

An all user group combined root cause analysis of critical tasks with more than four use errors*

Task	Root cause	User (n)
		S	AS	SN	CN
Task 127: Miscellaneous errors arising during general surgical procedure	HUD component required replacing	2	1	1	–
	Sudden loss of insufflation due to center staff accidentally turning gas off	–	1	–	–
Task 158: Apply brake to cart	Users forgot to apply brake to cart	–	1	2	1
	User thought task was performed by another user	–	–	2	–
Task 76/Q9-1: What should you be careful of when unbraking the BSU?	Users failed to mention possibility of trapping thin cables when BSU skirt is raised	2	4	2	3
	User cognizant of correct answer but failed to mention it in their answer	–	2	1	1
Task 239/Q15: If the BSU is disconnected from the surgeon console while in surgical mode, what will happen to the instrument assigned to that arm?	Users stated they did not learn about it during training	3	1	–	–
	User could not recall the correct answer but stated it was covered in training	–	–	–	2
Task 196/Q16: What do these icons mean? (hand detect)	Users either could not remember the icons being discussed in training or were not paying attention to the particular icons in question	–	2	–	2
	User confused icons between two operation modes	1	–	–	–
Task 334: Maintain sterility	User unknowingly touched robotic arm trying to squeeze between arms	–	2	–	–
	User did not take it as a real procedure and small OR size	–	1	2	–
	User became distracted during use of Versius	1	–	2	–

*All root causes of error were resolved following system design updates or modifications to the training protocol.

AS, assistant surgeon; BSU, bedside unit; CN, circulating nurse; HUD, head-up display; OR, operation room; S, surgeon; SN, scrub nurse.

Design recommendations

All participants were able to use the system successfully. Although no critical task failures were observed, the results of this study highlight areas of potential improvement where some features deserved further consideration for future development: principally concerning the user interface (eg, drape fastenings), training (eg, what is emphasized during training) and system issues (eg, updates to the HUD). Recommendations to address these issues are listed in table 3.

Table 3

Key recommendations based on findings from simulated use and drills

Category	Recommendation	Status
User interface	Add label to console screen buttons	Updated
	Improve drapes and drape connectors	Updated
	Increase light intensity of orientation arrow	Updated
User manual	Add direct references to troubleshooting in the user manual	Planned for future version
	Expand and review the index in the instrument and accessories manual so that readers find terms more easily	Planned for future version
	Emphasize how to change from trainer to surgical mode on console	Implemented*
Training	Emphasize drape attachment technique	Implemented
	Emphasize optimal BSU positioning	Implemented
	Emphasize cart orientation	Implemented
	More time for bedside team at the console to improve their understanding of using the hand controllers and menu navigation	Implemented
	More time for surgeons at the bedside to improve their knowledge of the arm mode map and general use of the BSUs including sleep button, brake button orientation	Implemented
	Emphasize docking features can be stowed under console arms	Implemented
System issues	Improve console’s brake	Updated
	Improve camera connection	Updated
	HUD internal component upgraded	Updated

*A new chapter on Versius Trainer will be added to the user manual.

BSU, bedside unit; HUD, head-up display.

All but two recommendations have been addressed by either updating the design or assembly instructions of specific parts, or by updating a related part. In addition, recommendations have been addressed by implementing changes to the instructions for use, training materials and the training protocol.

The only recommendations not yet implemented are those pertaining to expanding the index in the instruments and accessories manual and adding cross-references in the user manual. These will be added in future versions of the manuals; no negative impact to usability is expected as it will simply improve the navigation of both manuals.

Discussion

This study assessed the usability of Versius, to validate the safety and determine if the user interface had acceptable use-related risk, in the hands of trained intended users. No critical task failures were observed, demonstrating that after undertaking the Versius training program, all users can use the device effectively and safely, including both laparoscopically and robotically trained healthcare professionals.

Six critical tasks resulted in several remedial design recommendations. Task 127 (miscellaneous errors arising during general surgical procedure) was a system issue (HUD flickering resolved by replacing a component with a one of a higher specification) and was fixed during the study. The specific use errors observed during Tasks 158 (apply brake to cart), 76 (knowledge test: What should you be careful of when unbraking the BSU?), and 239 (knowledge test: if the BSU is disconnected from the surgeon console while in surgical mode, what will happen to the instrument assigned to that arm?) were not critical according to the UFMEA and therefore did not mandate a change in the training protocol. However, to increase user awareness and better enable the successful completion of these tasks, the training protocol was modified. Task 196 (knowledge test: What do these icons mean?) is very unlikely to result in serious failure, even though the Task is classified as critical. Finally, Task 334 (maintain sterility) represents a potential residual risk as three use errors were caused by participants’ inattention resulting in their backs or armpits touching areas of the sterile drape.

Robotic surgical systems typically fix robotic arms to a single cart, and in one system the robot is located over the patient.15 These designs can present challenges with maneuverability, restrict physical space, and inhibit access to the patient. Use errors associated with these maneuverability limitations have been reported. For example, surgeon obscured vision caused by the docked robot over the patient has led to reports of unrecognized tissue compression from the robotic arms.16 The modular system and open console design of Versius allows the surgeon and bedside clinical team good visibility and access to the patient, helping to avoid similar use errors. During robot-assisted procedures, surgeons must rely on experience and visual cues for safe and effective surgery. No procedure-related use errors were observed with surgeons’ use of the HUD, suggesting that Versius’ 3D video from the endoscopic camera together with the display overlay, facilitate careful handling of tissues.

Versius includes several advantageous design features. For example, the open console was designed to facilitate clear communication between the surgeon and the surgical team and allow the surgeon to sit or stand. The elbow of the instrument arm can be moved by surgical personnel while Versius is in surgical mode facilitating better access to the patient and the cart design of the BSUs was intended to be maneuverable, easy to set-up and operate, while occupying minimal space. Careful consideration was given to the design of the simulated use and drills, to validate the usability of Versius’ features. For instance, numerous tasks assessed the transport, positioning, braking and draping of the BSUs to demonstrate their ease of use and maneuverability. Communication between members of the surgical teams was checked during general surgical procedures and by tasking the surgeon with requesting instrument changes. No related critical task failures were observed with any of the tasks undertaken, validating that these design features can be used safely.

Study limitations

The merits of simulation training are well recognized and can provide the trainee with an immersive experience, being able to replicate substantial aspects of the real world.17 While this study was conducted in a realistic, simulation environment, including the use of cadavers, ultimately, it will not provide the full experience of a live procedure. Similarly, although cadavers have the advantages of human anatomy and real-size experience, cadaveric tissue has poor handling fidelity compared with live tissue and lacks the live operative experience of dealing with bleeding.18 However, following completion of this study, Versius has been successfully used to complete over 500 live procedures in humans in India and the United KIngdom. Finally, some participants reported being anxious at being assessed on a system new to them, which may have impaired their performance.19 20

Conclusions

Having considered all use-related residual risks and in the absence of any further concerns, it is concluded that the usability of Versius has been validated in a simulated clinical setting. In addition, the results of this study indicate that the system can be proficiently operated by both laparoscopically and robotically trained healthcare professionals after undertaking the Versius training program.