Nicholas Smith1, John Harnett1, Andrew Furey1. 1. From the Department of Surgery (Smith), the Faculty of Medicine, Health Sciences Centre, (Harnett) and the Department of Orthopaedic Traumatology (Furey), Memorial University of Newfoundland, St. John's, NL.
Abstract
BACKGROUND: Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form. METHODS: A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach's α and Fleiss κ. RESULTS: Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach's α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval -0.071 to 0.364), demonstrating low interrater reliability. CONCLUSION: Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.
BACKGROUND: Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form. METHODS: A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach's α and Fleiss κ. RESULTS: Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach's α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval -0.071 to 0.364), demonstrating low interrater reliability. CONCLUSION: Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.
Authors: Barbara Goff; Lynn Mandel; Gretchen Lentz; Amy Vanblaricom; Anne-Marie Amies Oelschlager; David Lee; Andrew Galakatos; Matthew Davies; Peter Nielsen Journal: Am J Obstet Gynecol Date: 2005-04 Impact factor: 8.661
Authors: Peter C Ferguson; William Kraemer; Markku Nousiainen; Oleg Safir; Ranil Sonnadara; Benjamin Alman; Richard Reznick Journal: J Bone Joint Surg Am Date: 2013-11-06 Impact factor: 5.284
Authors: Kulsoom Laeeq; Nasir I Bhatti; John P Carey; Charles C Della Santina; Charles J Limb; John K Niparko; Lloyd B Minor; Howard W Francis Journal: Laryngoscope Date: 2009-12 Impact factor: 3.325
Authors: Cornelis J Hopmans; Pieter T den Hoed; Lijckle van der Laan; Erwin van der Harst; Maarten van der Elst; Guido H H Mannaerts; Imro Dawson; Reinier Timman; Bas P L Wijnhoven; Jan N M IJzermans Journal: Surgery Date: 2014-09-16 Impact factor: 3.982