Ronit Patnaik1, Nicholas E Anton2, Dimitrios Stefanidis3. 1. Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA. Electronic address: patnaikr@uthscsa.edu. 2. Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA. Electronic address: nanton@iu.edu. 3. Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA. Electronic address: dimstefa@iu.edu.
Abstract
BACKGROUND: Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters. METHODS: The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups. RESULTS: The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001). CONCLUSIONS: Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.
BACKGROUND: Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters. METHODS: The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups. RESULTS: The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001). CONCLUSIONS: Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.
Authors: B Joseph Elmunzer; Catharine M Walsh; Gretchen Guiton; Jose Serrano; Amitabh Chak; Steven Edmundowicz; Richard S Kwon; Daniel Mullady; Georgios I Papachristou; Grace Elta; Todd H Baron; Patrick Yachimski; Evan L Fogel; Peter V Draganov; Jason R Taylor; James Scheiman; Vikesh K Singh; Shyam Varadarajulu; Field F Willingham; Gregory A Cote; Peter B Cotton; Violette Simon; Rebecca Spitzer; Rajesh Keswani; Sachin Wani Journal: Gastrointest Endosc Date: 2020-07-30 Impact factor: 9.427