Benjamin Zendejas1, Raaj K Ruparel2, David A Cook3,4. 1. Department of Surgery, Mayo Clinic College of Medicine, Mayo 12-W, 200 First Street SW, Rochester, MN, 55905, USA. zendejas.benjamin@mayo.edu. 2. Department of Surgery, Mayo Clinic College of Medicine, Mayo 12-W, 200 First Street SW, Rochester, MN, 55905, USA. 3. Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, USA. 4. Multidisciplinary Simulation Center, Mayo Clinic College of Medicine, Rochester, MN, USA.
Abstract
BACKGROUND: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. METHODS: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). RESULTS: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert-novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. CONCLUSION: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.
BACKGROUND: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. METHODS: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). RESULTS: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert-novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. CONCLUSION: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.
Keywords:
Assessment; Fundamentals of Laparoscopic Surgery; Simulation; Validation
Authors: M C Vassiliou; G A Ghitulescu; L S Feldman; D Stanbridge; K Leffondré; H H Sigman; G M Fried Journal: Surg Endosc Date: 2006-02-27 Impact factor: 4.584
Authors: Breno Dauster; Andrew P Steinberg; Melina C Vassiliou; Simon Bergman; Donna D Stanbridge; Liane S Feldman; Gerald M Fried Journal: J Endourol Date: 2005-06 Impact factor: 2.942
Authors: Ganesh Sankaranarayanan; Henry Lin; Venkata S Arikatla; Maureen Mulcare; Likun Zhang; Alexandre Derevianko; Robert Lim; David Fobert; Caroline Cao; Steven D Schwaitzberg; Daniel B Jones; Suvranu De Journal: J Laparoendosc Adv Surg Tech A Date: 2010-03 Impact factor: 1.878
Authors: Emily D Dubina; Xuan-Binh D Pham; Alexander C Schwed; Hoover Wu; Imani McElroy; Amy H Kaji; Christian de Virgilio Journal: JAMA Surg Date: 2018-12-01 Impact factor: 14.766
Authors: Benjamin Zendejas; James W Jakub; Alicia M Terando; Amod Sarnaik; Charlotte E Ariyan; Mark B Faries; Sabino Zani; Heather B Neuman; Nabil Wasif; Jeffrey M Farma; Bruce J Averbook; Karl Y Bilimoria; Douglas Tyler; Mary Sue Brady; David R Farley Journal: Surg Endosc Date: 2016-12-07 Impact factor: 4.584
Authors: Chaitanya S Kulkarni; Shiyu Deng; Tianzi Wang; Jacob Hartman-Kenzler; Laura E Barnes; Sarah Henrickson Parker; Shawn D Safford; Nathan Lau Journal: Surg Endosc Date: 2022-09-19 Impact factor: 3.453
Authors: Adam D Shellito; Sonam Kapadia; Amy H Kaji; Cynthia M Tom; Christine Dauphine; Beverley A Petrie Journal: Surg Endosc Date: 2021-02-01 Impact factor: 4.584