Lyen C Huang1, Dante Conley2, Stu Lipsitz3, Christopher C Wright4, Thomas W Diller4, Lizabeth Edmondson5, William R Berry5, Sara J Singer6. 1. Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA Department of Surgery, Stanford University School of Medicine, Stanford, California, USA. 2. Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA Tanana Valley Clinic, Fairbanks, Alaska, USA. 3. Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA. 4. Greenville Health System, Greenville, South Carolina, USA. 5. Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA. 6. Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA Mongan Institute for Health Policy, Massachusetts General Hospital, Boston, Massachusetts, USA.
Abstract
OBJECTIVE: To assess the inter-rater reliability (IRR) of two novel observation tools for measuring surgical safety checklist performance and teamwork. SUMMARY BACKGROUND: Data surgical safety checklists can promote adherence to standards of care and improve teamwork in the operating room. Their use has been associated with reductions in mortality and other postoperative complications. However, checklist effectiveness depends on how well they are performed. METHODS: Authors from the Safe Surgery 2015 initiative developed a pair of novel observation tools through literature review, expert consultation and end-user testing. In one South Carolina hospital participating in the initiative, two observers jointly attended 50 surgical cases and independently rated surgical teams using both tools. We used descriptive statistics to measure checklist performance and teamwork at the hospital. We assessed IRR by measuring percent agreement, Cohen's κ, and weighted κ scores. RESULTS: The overall percent agreement and κ between the two observers was 93% and 0.74 (95% CI 0.66 to 0.79), respectively, for the Checklist Coaching Tool and 86% and 0.84 (95% CI 0.77 to 0.90) for the Surgical Teamwork Tool. Percent agreement for individual sections of both tools was 79% or higher. Additionally, κ scores for six of eight sections on the Checklist Coaching Tool and for two of five domains on the Surgical Teamwork Tool achieved the desired 0.7 threshold. However, teamwork scores were high and variation was limited. There were no significant changes in the percent agreement or κ scores between the first 10 and last 10 cases observed. CONCLUSIONS: Both tools demonstrated substantial IRR and required limited training to use. These instruments may be used to observe checklist performance and teamwork in the operating room. However, further refinement and calibration of observer expectations, particularly in rating teamwork, could improve the utility of the tools. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
OBJECTIVE: To assess the inter-rater reliability (IRR) of two novel observation tools for measuring surgical safety checklist performance and teamwork. SUMMARY BACKGROUND: Data surgical safety checklists can promote adherence to standards of care and improve teamwork in the operating room. Their use has been associated with reductions in mortality and other postoperative complications. However, checklist effectiveness depends on how well they are performed. METHODS: Authors from the Safe Surgery 2015 initiative developed a pair of novel observation tools through literature review, expert consultation and end-user testing. In one South Carolina hospital participating in the initiative, two observers jointly attended 50 surgical cases and independently rated surgical teams using both tools. We used descriptive statistics to measure checklist performance and teamwork at the hospital. We assessed IRR by measuring percent agreement, Cohen's κ, and weighted κ scores. RESULTS: The overall percent agreement and κ between the two observers was 93% and 0.74 (95% CI 0.66 to 0.79), respectively, for the Checklist Coaching Tool and 86% and 0.84 (95% CI 0.77 to 0.90) for the Surgical Teamwork Tool. Percent agreement for individual sections of both tools was 79% or higher. Additionally, κ scores for six of eight sections on the Checklist Coaching Tool and for two of five domains on the Surgical Teamwork Tool achieved the desired 0.7 threshold. However, teamwork scores were high and variation was limited. There were no significant changes in the percent agreement or κ scores between the first 10 and last 10 cases observed. CONCLUSIONS: Both tools demonstrated substantial IRR and required limited training to use. These instruments may be used to observe checklist performance and teamwork in the operating room. However, further refinement and calibration of observer expectations, particularly in rating teamwork, could improve the utility of the tools. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
Authors: Taylor Wurdeman; Steven J Staffa; Shehnaz Alidina; Ntuli A Kapologwe; David Barash; Ladislaus Buberwa; Eliudi Eliakimu; Erastus Maina; Sarah Maongezi; John G Meara; William Munyonyela; Rahma Mushi; Cheri Reynolds; Christopher Strader; John Varallo; Leonard Washington; David Zurakowski Journal: World J Surg Date: 2021-11-19 Impact factor: 3.352
Authors: Shehnaz Alidina; Gopal Menon; Steven J Staffa; Sakshie Alreja; David Barash; Erin Barringer; Monica Cainer; Isabelle Citron; Amanda DiMeo; Edwin Ernest; Laura Fitzgerald; Hiba Ghandour; Magdalena Gruendl; Audustino Hellar; Desmond T Jumbam; Adam Katoto; Lauren Kelly; Steve Kisakye; Salome Kuchukhidze; Tenzing Lama; William Lodge Ii; Erastus Maina; Fabian Massaga; Adelina Mazhiqi; John G Meara; Stella Mshana; Ian Nason; Chase Reynolds; Cheri Reynolds; Hannington Segirinya; Dorcas Simba; Victoria Smith; Christopher Strader; Meaghan Sydlowski; Leopold Tibyehabwa; Florian Tinuga; Alena Troxel; Mpoki Ulisubisya; John Varallo; Taylor Wurdeman; Noor Zanial; David Zurakowski; Ntuli Kapologwe; Sarah Maongezi Journal: Int J Qual Health Care Date: 2021-06-29 Impact factor: 2.038
Authors: Shehnaz Alidina; Pritha Chatterjee; Noor Zanial; Sakshie Sanjay Alreja; Rebecca Balira; David Barash; Edwin Ernest; Geofrey Charles Giiti; Erastus Maina; Adelina Mazhiqi; Rahma Mushi; Cheri Reynolds; Meaghan Sydlowski; Florian Tinuga; Sarah Maongezi; John G Meara; Ntuli A Kapologwe; Erin Barringer; Monica Cainer; Isabelle Citron; Amanda DiMeo; Laura Fitzgerald; Hiba Ghandour; Magdalena Gruendl; Augustino Hellar; Desmond T Jumbam; Adam Katoto; Lauren Kelly; Steve Kisakye; Salome Kuchukhidze; Tenzing N Lama; Gopal Menon; Stella Mshana; Chase Reynolds; Hannington Segirinya; Dorcas Simba; Victoria Smith; Steven J Staffa; Christopher Strader; Leopold Tibyehabwa; Alena Troxel; John Varallo; Taylor Wurdeman; David Zurakowski Journal: BMJ Qual Saf Date: 2021-02-05 Impact factor: 7.035