Elif Bilgic1, Mohammed Al Mahroos1, Tara Landry2, Gerald M Fried1, Melina C Vassiliou1, Liane S Feldman3. 1. Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada. 2. Montreal General Hospital Medical Library, McGill University Health Centre, 1650, Cedar Avenue, L9. 309, Montréal, QC, H3G 1A4, Canada. 3. Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada. liane.feldman@mcgill.ca.
Abstract
BACKGROUND: Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument. METHODS: A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences). RESULTS: There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF. CONCLUSION: There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.
BACKGROUND: Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument. METHODS: A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences). RESULTS: There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF. CONCLUSION: There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.
Authors: Liane S Feldman; Sarah E Hagarty; Gabriela Ghitulescu; Donna Stanbridge; Gerald M Fried Journal: J Am Coll Surg Date: 2004-01 Impact factor: 6.113
Authors: Joris A J L Broeders; Werner A Draaisma; Hilda G Rijnhart-de Jong; André J P M Smout; Jan J B van Lanschot; Ivo A M J Broeders; Hein G Gooszen Journal: Arch Surg Date: 2011-03
Authors: Gunnar Ahlberg; Olli Kruuna; Carl-Eric Leijonmarck; Jari Ovaska; Arne Rosseland; Rune Sandbu; Cecilia Strömberg; Dag Arvidsson Journal: Am J Surg Date: 2005-02 Impact factor: 2.565
Authors: Iman Ghaderi; Farouq Manji; Yoon Soo Park; Dorthea Juul; Michael Ott; Ilene Harris; Timothy M Farrell Journal: Ann Surg Date: 2015-02 Impact factor: 12.969
Authors: Saba Balvardi; Anitha Kammili; Melissa Hanson; Carmen Mueller; Melina Vassiliou; Lawrence Lee; Kevin Schwartzman; Julio F Fiore; Liane S Feldman Journal: Surg Endosc Date: 2022-05-12 Impact factor: 4.584