Monty A Aghazadeh1,2, Isuru S Jayaratna3, Andrew J Hung3, Michael M Pan2, Mihir M Desai3, Inderbir S Gill3, Alvin C Goh4. 1. Department of Urology, Methodist Institute for Technology, Innovation, and Education, Houston Methodist Hospital, 6560 Fannin Street, Suite 2100, Houston, TX, 77030, USA. 2. Scott Department of Urology, Baylor College of Medicine, Houston, TX, USA. 3. USC Institute of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA. 4. Department of Urology, Methodist Institute for Technology, Innovation, and Education, Houston Methodist Hospital, 6560 Fannin Street, Suite 2100, Houston, TX, 77030, USA. acg622@gmail.com.
Abstract
BACKGROUND: We demonstrate the construct validity, reliability, and utility of Global Evaluative Assessment of Robotic Skills (GEARS), a clinical assessment tool designed to measure robotic technical skills, in an independent cohort using an in vivo animal training model. METHODS: Using a cross-sectional observational study design, 47 voluntary participants were categorized as experts (>30 robotic cases completed as primary surgeon) or trainees. The trainee group was further divided into intermediates (≥5 but ≤30 cases) or novices (<5 cases). All participants completed a standardized in vivo robotic task in a porcine model. Task performance was evaluated by two expert robotic surgeons and self-assessed by the participants using the GEARS assessment tool. Kruskal-Wallis test was used to compare the GEARS performance scores to determine construct validity; Spearman's rank correlation measured interobserver reliability; and Cronbach's alpha was used to assess internal consistency. RESULTS: Performance evaluations were completed on nine experts and 38 trainees (14 intermediate, 24 novice). Experts demonstrated superior performance compared to intermediates and novices overall and in all individual domains (p < 0.0001). In comparing intermediates and novices, the overall performance difference trended toward significance (p = 0.0505), while the individual domains of efficiency and autonomy were significantly different between groups (p = 0.0280 and 0.0425, respectively). Interobserver reliability between expert ratings was confirmed with a strong correlation observed (r = 0.857, 95 % CI [0.691, 0.941]). Experts and participant scoring showed less agreement (r = 0.435, 95 % CI [0.121, 0.689] and r = 0.422, 95 % CI [0.081, 0.0672]). Internal consistency was excellent for experts and participants (α = 0.96, 0.98, 0.93). CONCLUSIONS: In an independent cohort, GEARS was able to differentiate between different robotic skill levels, demonstrating excellent construct validity. As a standardized assessment tool, GEARS maintained consistency and reliability for an in vivo robotic surgical task and may be applied for skills evaluation in a broad range of robotic procedures.
BACKGROUND: We demonstrate the construct validity, reliability, and utility of Global Evaluative Assessment of Robotic Skills (GEARS), a clinical assessment tool designed to measure robotic technical skills, in an independent cohort using an in vivo animal training model. METHODS: Using a cross-sectional observational study design, 47 voluntary participants were categorized as experts (>30 robotic cases completed as primary surgeon) or trainees. The trainee group was further divided into intermediates (≥5 but ≤30 cases) or novices (<5 cases). All participants completed a standardized in vivo robotic task in a porcine model. Task performance was evaluated by two expert robotic surgeons and self-assessed by the participants using the GEARS assessment tool. Kruskal-Wallis test was used to compare the GEARS performance scores to determine construct validity; Spearman's rank correlation measured interobserver reliability; and Cronbach's alpha was used to assess internal consistency. RESULTS: Performance evaluations were completed on nine experts and 38 trainees (14 intermediate, 24 novice). Experts demonstrated superior performance compared to intermediates and novices overall and in all individual domains (p < 0.0001). In comparing intermediates and novices, the overall performance difference trended toward significance (p = 0.0505), while the individual domains of efficiency and autonomy were significantly different between groups (p = 0.0280 and 0.0425, respectively). Interobserver reliability between expert ratings was confirmed with a strong correlation observed (r = 0.857, 95 % CI [0.691, 0.941]). Experts and participant scoring showed less agreement (r = 0.435, 95 % CI [0.121, 0.689] and r = 0.422, 95 % CI [0.081, 0.0672]). Internal consistency was excellent for experts and participants (α = 0.96, 0.98, 0.93). CONCLUSIONS: In an independent cohort, GEARS was able to differentiate between different robotic skill levels, demonstrating excellent construct validity. As a standardized assessment tool, GEARS maintained consistency and reliability for an in vivo robotic surgical task and may be applied for skills evaluation in a broad range of robotic procedures.
Authors: Martin N Jonsson; Mahmood Mahmood; Tomas Askerud; Henrik Hellborg; Stig Ramel; N Peter Wiklund; Magnus Kjellman; Gunnar Ahlberg Journal: J Endourol Date: 2010-11-29 Impact factor: 2.942
Authors: Melina C Vassiliou; Liane S Feldman; Christopher G Andrew; Simon Bergman; Karen Leffondré; Donna Stanbridge; Gerald M Fried Journal: Am J Surg Date: 2005-07 Impact factor: 2.565
Authors: Patrick A Kenney; Matthew F Wszolek; Justin J Gould; John A Libertino; Alireza Moinzadeh Journal: Urology Date: 2009-04-10 Impact factor: 2.649
Authors: William T Lowrance; James A Eastham; Caroline Savage; A C Maschino; Vincent P Laudone; Christopher B Dechet; Robert A Stephenson; Peter T Scardino; Jaspreet S Sandhu Journal: J Urol Date: 2012-04-11 Impact factor: 7.450
Authors: Stéfanie A Seixas-Mikelus; Andrew P Stegemann; Thenkurussi Kesavadas; Govindarajan Srimathveeravalli; Gughan Sathyaseelan; Rameela Chandrasekhar; Gregory E Wilding; James O Peabody; Khurshid A Guru Journal: BJU Int Date: 2010-10-04 Impact factor: 5.588
Authors: Luise I M Pernar; Faith C Robertson; Ali Tavakkoli; Eric G Sheu; David C Brooks; Douglas S Smink Journal: Surg Endosc Date: 2017-04-14 Impact factor: 4.584
Authors: Steven M Monda; Jonathan R Weese; Barrett G Anderson; Joel M Vetter; Ramakrishna Venkatesh; Kefu Du; Gerald L Andriole; Robert S Figenshau Journal: Urology Date: 2018-02-05 Impact factor: 2.649
Authors: Andrew J Hung; Thomas Bottyan; Thomas G Clifford; Sarfaraz Serang; Zein K Nakhoda; Swar H Shah; Hana Yokoi; Monish Aron; Inderbir S Gill Journal: World J Urol Date: 2016-04-22 Impact factor: 4.226
Authors: Hyeong Won Yu; Jin Wook Yi; Chan Yong Seong; Jong-Kyu Kim; In Eui Bae; Hyungju Kwon; Young Jun Chai; Su-Jin Kim; June Young Choi; Kyu Eun Lee Journal: Surg Endosc Date: 2017-08-25 Impact factor: 4.584