Laparoscopic surgery skills evaluation: analysis based on accelerometers.

BACKGROUND AND
OBJECTIVE: Technical skills assessment is considered an important part of surgical training. Subjective assessment is not appropriate for training feedback, and there is now increased demand for objective assessment of surgical performance. Economy of movement has been proposed as an excellent alternative for this purpose. The investigators describe a readily available method to evaluate surgical skills through motion analysis using accelerometers in Apple's iPod Touch device.
METHODS: Two groups of individuals with different minimally invasive surgery skill levels (experts and novices) were evaluated. Each group was asked to perform a given task with an iPod Touch placed on the dominant-hand wrist. The Accelerometer Data Pro application makes it possible to obtain movement-related data detected by the accelerometers. Average acceleration and maximum acceleration for each axis (x, y, and z) were determined and compared.
RESULTS: The analysis of average acceleration and maximum acceleration showed statistically significant differences between groups on both the y (P = .04, P = .03) and z (P = .04, P = .04) axes. This demonstrates the ability to distinguish between experts and novices. The analysis of the x axis showed no significant differences between groups, which could be explained by the fact that the task involves few movements on this axis.
CONCLUSION: Accelerometer-based motion analysis is a useful tool to evaluate laparoscopic skill development of surgeons and should be used in training programs. Validation of this device in an in vivo setting is a research goal of the investigators' team.

INTRODUCTION

Laparoscopic surgery has many advantages over open surgery, even when it comes to complex procedures or low-incidence diseases.[1,2] However, surgeons must overcome their own technical difficulties, and it is essential to acquire specific new skills; therefore, the traditional surgical learning method seems not to be the optimal one.[3-5]

Laparoscopic surgery requires certain skills that cannot be taught simply by apprenticeship. Minimally invasive surgery (MIS) requires a high degree of eye-hand coordination. The training program for MIS techniques at our center includes bench models and simulation. It has been shown that the skills acquired in the laboratory are subsequently transferred to the real environment.[6] Technical skills assessment is considered an important part of surgical training. Subjective assessment is not appropriate for training feedback, and there is now increased demand for objective assessment of surgical performance. The time required to complete a specific task is not necessarily a reliable measurement of accuracy and efficiency.[7,8]

Therefore, the use of ranking scales, recording the penalties and mistakes made during the procedure, could contribute to enhancing the reliable assessment of surgeons' performance.[9] Video analysis can define the path of an instrument's movement and has the advantage of not interfering with the procedure performed. However, the methods reported are not yet standardized, availability is limited, and the process involves complex recording systems and image analysis.[10] The assessment of movement patterns can be performed with different systems that consist mostly of specialized and expensive equipment.[11]

In this research, we proposed and assessed motion analysis using low-cost and readily available equipment, such as the fourth-generation iPod Touch (Apple, Inc, Cupertino, California). This device has accelerometers in 3 axes (x, y, and z), permitting assessment of the quality of movements made by a surgeon while performing a task. There are no similar reports in the literature or previous experience with the use of this type of equipment to determine MIS skills.

MATERIALS AND METHODS

iPod Touch

The iPod Touch, designed and distributed by Apple, has a motion sensor to measure linear acceleration in 3 axes (x, y, and z). It includes a receiver element and a serial interface capable of providing any particular information obtained, which can be stored in memory. Using the Accelerometer Data Pro application (Wavefront Labs, http://www.wavefrontlabs.com), information related to acceleration and orientation in 3 dimensions is obtained, allowing memory storage and subsequent transmission or transfer via the Internet as independent data files (comma-separated values), which can be imported into Excel (Microsoft Corporation, Redmond, Washington) for tabulation and analysis.

Methods

This was a cross-sectional study. The performance of 2 groups of individuals with different skill levels in laparoscopic surgery was compared. The first group of 5 individuals (novices; postgraduate year [PGY] 1) had no experience in laparoscopic surgery in vivo, and the second group of 5 surgeons (experts; PGY 3). PGY 3 surgeons at our center have performed 40 cases of laparoscopic surgery and have been trained and evaluated in unanimated and animal models.

The task that was performed in this study corresponds to task 3 of laparoscopic common bile duct exploration 4-task training model described at our center. During this task, the surgeon in training must place a T tube in a simulated common bile duct. It has been extensively validated and proved to have great ability to distinguish among individuals with different skill levels in MIS[12] (.

Laparoscopic common bile duct exploration 4-task training model: task 3.

The exercise was carried out with the device (iPod Touch) placed on the wrist of the operator's dominant hand ( using an armband case (iArmBands). A video explaining the task was shown to all participants, and they were allowed to perform initial, unassessed trials with the model.

The iPod Touch 4G device on the surgeon's dominant-hand wrist.

For each study group, the time to complete the task, average acceleration, and maximum acceleration in each axis (x, y, and z) were measured. The nonparametric Student t test was used to determine the difference between novices and experts.

RESULTS

Ten exercises were measured; 5 novices and 5 experts were studied during the performance of this specific task. Participants expressed no discomfort in placing the device on their dominant-hand wrists, and no limitation of freedom of movement was reported for any individual.

The analysis of the data obtained shows representative graphics of acceleration patterns in each axis (. The data related to average acceleration and maximum acceleration are summarized in .

Acceleration patterns in a determined axis, showing differences between novices and experts.

The average acceleration analysis showed statistically significant differences between groups in both the y (P = .04) and z axes (P = .04). Accordingly, maximum acceleration demonstrated an ability to distinguish between novice and expert individuals on the y and z axes (P = .03, P = .04). The analysis and comparison of the study variables for the x axis showed no significant differences between groups.

DISCUSSION

The increasing complexity of surgical procedures has led to the development of new and safe training methods based on bench models and simulators. Training in safe environments allows surgeons to learn from their own mistakes without threatening patient safety. After practice in inert and ex vivo models, surgeons have better performance in the operating room. This transfer of training increases efficacy and reduces failures and complications of surgery.[13]

Given the need to implement a scheduled training program for teaching laparoscopic surgery, we have described multiple training models, and the impact of some of these on skill development has been demonstrated. However, the evaluation parameter used in most of the studies is time to complete a specific task, which is not necessarily a reliable measurement of either accuracy of efficiency.[12,14]

Determination of the moment at which a surgeon acquires the new skills to face real situations requires objective evaluation techniques. The study of the “economy of movement” has been proposed, based on the premise that an experienced surgeon will perform with better accuracy and make fewer rough or unnecessary actions.[15]

Several authors have demonstrated that the acquisition of psychomotor skills can be determined by analyzing the motion of instruments.[5,8] According to a recent review by Chmarra et al,[16] at least 16 devices for objective assessment of surgical skills have been developed worldwide during the past decade. Datta et al[17] used an electromagnetic tracking device to determine the positions and movements of instruments. The final drawing obtained was called “movement signature,” which allowed clear differentiation among individuals with different levels of experience.

Rosen et al[18] designed an electromechanical unit known as the Blue Dragon to measure movements made by the hands and the force that is applied to instruments. Cristancho et al[19] described their own experience with the use of electromagnetic systems, emphasizing the importance of movement study as an objective and practical parameter. Likewise, Yamaguchi et al[20] emphasized the importance of assessing psychomotor skills by detecting movements when a complex task in laparoscopic surgery, such as tying an intracorporeal suture knot, is performed.

Multiple tracking devices have been used in recent years to objectively assess psychomotor skills in MIS. However, most are expensive, which is a major limiting factor for their application in our environment.

Acceleration is related to precision. Novice surgeons make sudden movements that are registered as wider positive or negative variations in acceleration. The proposed method demonstrated the ability to distinguish between individuals with different skill levels (PGY 1 vs PGY 3) on the y and z axes. The laparoscopic common bile duct exploration training model task 3 involves few movements in the horizontal plane, which would explain why no difference was documented in movements made on the x axis. However, this fact will be discussed in future related research studies. The time to perform the task was shorter in the expert group, but we have previously shown that this parameter is not sufficient to distinguish among surgeons with different skill levels.[14]

The use of accelerometers in the iPod Touch device proposed in this report is not found in the literature. Therefore, the addition of this evaluation method to training programs requires validation. Such a validation process should include many aspects, such as accessibility, easy data interpretation, and the ability to differentiate between expert surgeons and untrained individuals. This study had the limitation of a small sample size (5 novices vs 5 experts); a validation protocol with a larger number of individuals, performing a different task with movements requirement in all 3 dimensions, will be carried out soon at our center to provide stronger evidence.

The validation of this method is intended to contribute positively to laparoscopic surgery training programs. This method could be used to assess progress over time, ensuring our staff's training without compromising patient safety. The aim is the achievement of novel surgical techniques with high effectiveness and low morbidity.

CONCLUSIONS

Accelerometer-based motion analysis appears to be a useful tool to evaluate laparoscopic skill development of surgeons and should be used in training programs. Validation of this device in an in vivo setting is a research goal of our team.

Table 1.

Results: G force

Variable	Novices	Experts	P
x axis
Average acceleration	0.0355 ± 0.0056	0.0282 ± 0.0078	.1184
Maximum acceleration	0.2388 ± 0.0597	0.2023 ± 0.0727	.3938
y axis
Average acceleration	0.0187 ± 0.0035	0.0138 ± 0.0027	.0411*
Maximum acceleration	0.1234 ± 0.0426	0.0724 ± 0.0162	.0367*
z axis
Average acceleration	0.0249 ± 0.0045	0.0179 ± 0.0049	.0481*
Maximum acceleration	0.1772 ± 0.0619	0.0954 ± 0.0427	.0413*

Data are expressed as mean ± SD.

Statistically significant.