Amin Madani1, Jordan Gornitsky2, Yusuke Watanabe3, Cassandre Benay2, Maria S Altieri4, Philip H Pucher5, Roger Tabah2, Elliot J Mitmaker2. 1. Department of Surgery, McGill University Health Centre, 1650 Cedar Avenue, Rm D6-257, Montreal, QC, H3G 1A4, Canada. amin.madani@mail.mcgill.ca. 2. Department of Surgery, McGill University Health Centre, 1650 Cedar Avenue, Rm D6-257, Montreal, QC, H3G 1A4, Canada. 3. Department of Gastroenterological Surgery II, Hokkaido University, Sapporo, Japan. 4. Department of Surgery, Stony Brook University Medical Center, Stony Brook, NY, USA. 5. Department of Surgery and Cancer, Imperial College London, London, UK.
Abstract
BACKGROUND: Errors in judgment during thyroidectomy can lead to recurrent laryngeal nerve injury and other complications. Despite the strong link between patient outcomes and intraoperative decision-making, methods to evaluate these complex skills are lacking. The purpose of this study was to develop objective metrics to evaluate advanced cognitive skills during thyroidectomy and to obtain validity evidence for them. METHODS: An interactive online learning platform was developed ( www.thinklikeasurgeon.com ). Trainees and surgeons from four institutions completed a 33-item assessment, developed based on a cognitive task analysis and expert Delphi consensus. Sixteen items required subjects to make annotations on still frames of thyroidectomy videos, and accuracy scores were calculated based on an algorithm derived from experts' responses ("visual concordance test," VCT). Seven items were short answer (SA), requiring users to type their answers, and scores were automatically calculated based on their similarity to a pre-populated repertoire of correct responses. Test-retest reliability, internal consistency, and correlation of scores with self-reported experience and training level (novice, intermediate, expert) were calculated. RESULTS: Twenty-eight subjects (10 endocrine surgeons and otolaryngologists, 18 trainees) participated. There was high test-retest reliability (intraclass correlation coefficient = 0.96; n = 10) and internal consistency (Cronbach's α = 0.93). The assessment demonstrated significant differences between novices, intermediates, and experts in total score (p < 0.01), VCT score (p < 0.01) and SA score (p < 0.01). There was high correlation between total case number and total score (ρ = 0.95, p < 0.01), between total case number and VCT score (ρ = 0.93, p < 0.01), and between total case number and SA score (ρ = 0.83, p < 0.01). CONCLUSION: This study describes the development of novel metrics and provides validity evidence for an interactive Web-based platform to objectively assess decision-making during thyroidectomy.
BACKGROUND: Errors in judgment during thyroidectomy can lead to recurrent laryngeal nerve injury and other complications. Despite the strong link between patient outcomes and intraoperative decision-making, methods to evaluate these complex skills are lacking. The purpose of this study was to develop objective metrics to evaluate advanced cognitive skills during thyroidectomy and to obtain validity evidence for them. METHODS: An interactive online learning platform was developed ( www.thinklikeasurgeon.com ). Trainees and surgeons from four institutions completed a 33-item assessment, developed based on a cognitive task analysis and expert Delphi consensus. Sixteen items required subjects to make annotations on still frames of thyroidectomy videos, and accuracy scores were calculated based on an algorithm derived from experts' responses ("visual concordance test," VCT). Seven items were short answer (SA), requiring users to type their answers, and scores were automatically calculated based on their similarity to a pre-populated repertoire of correct responses. Test-retest reliability, internal consistency, and correlation of scores with self-reported experience and training level (novice, intermediate, expert) were calculated. RESULTS: Twenty-eight subjects (10 endocrine surgeons and otolaryngologists, 18 trainees) participated. There was high test-retest reliability (intraclass correlation coefficient = 0.96; n = 10) and internal consistency (Cronbach's α = 0.93). The assessment demonstrated significant differences between novices, intermediates, and experts in total score (p < 0.01), VCT score (p < 0.01) and SA score (p < 0.01). There was high correlation between total case number and total score (ρ = 0.95, p < 0.01), between total case number and VCT score (ρ = 0.93, p < 0.01), and between total case number and SA score (ρ = 0.83, p < 0.01). CONCLUSION: This study describes the development of novel metrics and provides validity evidence for an interactive Web-based platform to objectively assess decision-making during thyroidectomy.
Authors: Lawrence W Way; Lygia Stewart; Walter Gantert; Kingsway Liu; Crystine M Lee; Karen Whang; John G Hunter Journal: Ann Surg Date: 2003-04 Impact factor: 12.969
Authors: Amin Madani; Yusuke Watanabe; Melina Vassiliou; Liane S Feldman; Quan-Yang Duh; Michael C Singer; Daniel T Ruan; Roger Tabah; Elliot Mitmaker Journal: Surgery Date: 2015-10-02 Impact factor: 3.982
Authors: F Rulli; V Ambrogi; G Dionigi; S Amirhassankhani; T C Mineo; F Ottaviani; A Buemi; P DI Stefano; M Mourad Journal: Acta Otorhinolaryngol Ital Date: 2014-08 Impact factor: 2.124
Authors: Hamzeh Naghawi; Johnny Chau; Amin Madani; Pepa Kaneva; John Monson; Carmen Mueller; Lawrence Lee Journal: Tech Coloproctol Date: 2022-05-03 Impact factor: 3.699
Authors: Amin Madani; Babak Namazi; Maria S Altieri; Daniel A Hashimoto; Angela Maria Rivera; Philip H Pucher; Allison Navarrete-Welton; Ganesh Sankaranarayanan; L Michael Brunt; Allan Okrainec; Adnan Alseidi Journal: Ann Surg Date: 2020-11-13 Impact factor: 13.787