Vivek Datta1, Simon Bann, Mirren Mandalia, Ara Darzi. 1. Department of Surgical Oncology and Technology, Imperial College, 83 Sumatra Road, West Hampstead, London NW6 1PT, United Kingdom. v.datta@imperial.ac.uk
Abstract
BACKGROUND: Technical skills assessments are being increasingly used in surgical residency programs, with the objectivity and validity of several techniques well established. However, many of these methods are labor and time intensive, limiting their feasibility. This study aims to compare more efficient techniques of skills appraisals with an established gold standard. METHODS: Thirty surgeons completed 2 previously validated laboratory-based surgical models: small bowel anastomosis and vein patch insertion. Gold standard evaluation was the Objective Structured Assessment of Technical Skills (OSATS) method. "Efficient" techniques used were (1) quality of final product (FP); (2) snapshot assessment (SS), in which task performance was edited to a 2-minute sound bite and scored with OSATS; and (3) the surgical efficiency score (SES), a combination of final product quality and hand-motion analysis. All human observer evaluations used retrospective video analysis with 3 trained observers. Nonparametric tests were used to analyze the results. RESULTS: With respect to small bowel anastomosis, correlations with OSATS were as follows: FP 0.341 (P=.07), SS 0.577 (P<.001), and SES 0.842 (P<.001). For vein patch insertion, the correlations were as follows: FP 0.545 (P=.001), SS 0.609 (P<.001), and SES 0.700 (P<.001). Interobserver concordance was high for both models with respect to FP (Cronbach's alpha 0.80 for small bowel anastomosis and 0.84 for vein patch insertion). With respect to SS, interobserver reliability was high for vein patch insertion (Cronbach's alpha 0.80) but only moderate for small bowel anastomosis (0.59). CONCLUSIONS: The surgical efficiency score and snap shot assessments both show significant correlations with the traditional OSATS appraisals and suggest that skills assessment can be made more feasible. Correlations were closer with the former and interobserver concordance more variable with the latter, suggesting the surgical efficiency score as the most reliable of the methods evaluated.
BACKGROUND: Technical skills assessments are being increasingly used in surgical residency programs, with the objectivity and validity of several techniques well established. However, many of these methods are labor and time intensive, limiting their feasibility. This study aims to compare more efficient techniques of skills appraisals with an established gold standard. METHODS: Thirty surgeons completed 2 previously validated laboratory-based surgical models: small bowel anastomosis and vein patch insertion. Gold standard evaluation was the Objective Structured Assessment of Technical Skills (OSATS) method. "Efficient" techniques used were (1) quality of final product (FP); (2) snapshot assessment (SS), in which task performance was edited to a 2-minute sound bite and scored with OSATS; and (3) the surgical efficiency score (SES), a combination of final product quality and hand-motion analysis. All human observer evaluations used retrospective video analysis with 3 trained observers. Nonparametric tests were used to analyze the results. RESULTS: With respect to small bowel anastomosis, correlations with OSATS were as follows: FP 0.341 (P=.07), SS 0.577 (P<.001), and SES 0.842 (P<.001). For vein patch insertion, the correlations were as follows: FP 0.545 (P=.001), SS 0.609 (P<.001), and SES 0.700 (P<.001). Interobserver concordance was high for both models with respect to FP (Cronbach's alpha 0.80 for small bowel anastomosis and 0.84 for vein patch insertion). With respect to SS, interobserver reliability was high for vein patch insertion (Cronbach's alpha 0.80) but only moderate for small bowel anastomosis (0.59). CONCLUSIONS: The surgical efficiency score and snap shot assessments both show significant correlations with the traditional OSATS appraisals and suggest that skills assessment can be made more feasible. Correlations were closer with the former and interobserver concordance more variable with the latter, suggesting the surgical efficiency score as the most reliable of the methods evaluated.
Authors: Anne-Lise D D'Angelo; Drew N Rutherford; Rebecca D Ray; Shlomi Laufer; Calvin Kwan; Elaine R Cohen; Andrea Mason; Carla M Pugh Journal: Am J Surg Date: 2015-01-14 Impact factor: 2.565
Authors: Sayra M Cristancho; Antony J Hodgson; O N M Panton; Adam Meneghetti; Garth Warnock; Karim Qayumi Journal: Surg Endosc Date: 2008-12-31 Impact factor: 4.584
Authors: J B Pagador; F M Sánchez-Margallo; L F Sánchez-Peralta; J A Sánchez-Margallo; J L Moyano-Cuevas; S Enciso-Sanz; J Usón-Gargallo; J Moreno Journal: Int J Comput Assist Radiol Surg Date: 2011-08-14 Impact factor: 2.924
Authors: Amir Baghdadi; Ahmed A Hussein; Youssef Ahmed; Lora A Cavuoto; Khurshid A Guru Journal: Int J Comput Assist Radiol Surg Date: 2018-11-20 Impact factor: 2.924
Authors: Aneeq Zia; Yachna Sharma; Vinay Bettadapura; Eric L Sarin; Thomas Ploetz; Mark A Clements; Irfan Essa Journal: Int J Comput Assist Radiol Surg Date: 2016-08-27 Impact factor: 2.924
Authors: David P Azari; Lane L Frasier; Sudha R Pavuluri Quamme; Caprice C Greenberg; Carla M Pugh; Jacob A Greenberg; Robert G Radwin Journal: Ann Surg Date: 2019-03 Impact factor: 12.969
Authors: Anne-Lise D D'Angelo; Drew N Rutherford; Rebecca D Ray; Shlomi Laufer; Andrea Mason; Carla M Pugh Journal: Am J Surg Date: 2015-11-11 Impact factor: 2.565