Celine Yeung1, Brian Carrillo2,3, Victor Pope4, Shahob Hosseinpour1, J Ted Gerstle2,3, Georges Azzie5,6. 1. Faculty of Medicine, University of Toronto, Toronto, ON, Canada. 2. Division of General and Thoracic Surgery, The Hospital for Sick Children, Toronto, ON, Canada. 3. Centre for Image-Guided Innovation and Therapeutic Intervention, Toronto, ON, Canada. 4. Division of Otolaryngology, The Hospital for Sick Children, Toronto, ON, Canada. 5. Division of General and Thoracic Surgery, The Hospital for Sick Children, Toronto, ON, Canada. georges.azzie@sickkids.ca. 6. Centre for Image-Guided Innovation and Therapeutic Intervention, Toronto, ON, Canada. georges.azzie@sickkids.ca.
Abstract
BACKGROUND: Previous investigators have shown that novices are able to assess surgical skills as reliably as expert surgeons. The purpose of this study was to determine how novices and experts arrive at these graded scores when assessing laparoscopic skills and the potential implications this may have for surgical education. METHODS: Four novices and four general laparoscopic surgeons evaluated 59 videos of a suturing task using a 5-point scale. Average novice and expert evaluator scores for each video and the average number of times that scores were changed were compared. Intraclass correlation coefficients were used to determine inter-rater and test-retest reliability. Evaluators were asked to define the number of videos they needed to watch before they could confidently grade and to describe how they were able to distinguish between different levels of expertise. RESULTS: There were no significant differences in mean scores assigned by the two evaluator groups. Novices changed their scores more frequently compared to experts, but this did not reach statistical significance. There was excellent inter-rater reliability between the two groups (ICC = 0.91, CI 0.85-0.95) and good test-retest reliability (ICC > 0.83). On average, novices and experts reported that they needed to watch 13.8 ± 2.4 and 8.5 ± 2.5 videos, respectively, before they could confidently grade. Both groups also identified similar qualitative indicators (e.g., instrument control). CONCLUSION: Evaluators with varying levels of expertise can reliably grade performance of an intracorporeal suturing task. While novices were less confident in their grading, both groups were able to assign comparable scores and identify similar elements of a suturing skill as being important in terms of assessment.
BACKGROUND: Previous investigators have shown that novices are able to assess surgical skills as reliably as expert surgeons. The purpose of this study was to determine how novices and experts arrive at these graded scores when assessing laparoscopic skills and the potential implications this may have for surgical education. METHODS: Four novices and four general laparoscopic surgeons evaluated 59 videos of a suturing task using a 5-point scale. Average novice and expert evaluator scores for each video and the average number of times that scores were changed were compared. Intraclass correlation coefficients were used to determine inter-rater and test-retest reliability. Evaluators were asked to define the number of videos they needed to watch before they could confidently grade and to describe how they were able to distinguish between different levels of expertise. RESULTS: There were no significant differences in mean scores assigned by the two evaluator groups. Novices changed their scores more frequently compared to experts, but this did not reach statistical significance. There was excellent inter-rater reliability between the two groups (ICC = 0.91, CI 0.85-0.95) and good test-retest reliability (ICC > 0.83). On average, novices and experts reported that they needed to watch 13.8 ± 2.4 and 8.5 ± 2.5 videos, respectively, before they could confidently grade. Both groups also identified similar qualitative indicators (e.g., instrument control). CONCLUSION: Evaluators with varying levels of expertise can reliably grade performance of an intracorporeal suturing task. While novices were less confident in their grading, both groups were able to assign comparable scores and identify similar elements of a suturing skill as being important in terms of assessment.
Keywords:
Laparoscopic; Novice evaluators; Suturing skill; Video assessment
Authors: D J Scott; R V Rege; P C Bergen; W A Guo; R Laycock; S T Tesfay; R J Valentine; D B Jones Journal: J Laparoendosc Adv Surg Tech A Date: 2000-08 Impact factor: 1.878
Authors: Matthew R Thomas; Thomas J Beckman; Karen F Mauck; Stephen S Cha; Kris G Thomas Journal: J Gen Intern Med Date: 2011-03-03 Impact factor: 5.128
Authors: Daniel Holst; Timothy M Kowalewski; Lee W White; Timothy C Brand; Jonathan D Harper; Mathew D Sorensen; Mireille Truong; Khara Simpson; Alyssa Tanaka; Roger Smith; Thomas S Lendvay Journal: J Endourol Date: 2015-05-26 Impact factor: 2.942
Authors: Lee W White; Timothy M Kowalewski; Rodney Lee Dockter; Bryan Comstock; Blake Hannaford; Thomas S Lendvay Journal: J Endourol Date: 2015-08-24 Impact factor: 2.942