Scott C Watkins1, David A Roberts, John R Boulet, Matthew D McEvoy, Matthew B Weinger. 1. From the Department of Anesthesiology (S.CW., D.A.R., M.D.M., M.B.W.), Vanderbilt University School of Medicine; Center for Research and Innovation in Systems Safety (S.C.W., M.B.W.), Vanderbilt University Medical Center, Nashville, TN; and Foundation for Advancement of International Medical Education and Research (FAMER) (J.R.B), and the Educational Commission for Foreign Medical Graduates (ECFMG), Philadelphia, PA.
Abstract
INTRODUCTION: Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale. METHODS: Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall. RESULTS: The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74. CONCLUSIONS: Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.
INTRODUCTION: Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale. METHODS: Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall. RESULTS: The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74. CONCLUSIONS: Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.
Authors: Elizabeth D Rosenman; Mark J Bullard; Kerin A Jones; Laura Welsh; Sarah M Brolliar; Benjamin R Levine; James A Grand; Rosemarie Fernandez Journal: AEM Educ Train Date: 2019-02-19
Authors: Fabrizio Bracco; Gabriele de Tonetti; Michele Masini; Marcello Passarelli; Francesca Geretto; Danilo Celleno Journal: Int J Environ Res Public Health Date: 2018-03-03 Impact factor: 3.390
Authors: Jaycelyn R Holland; Donald H Arnold; Holly R Hanson; Barbara J Solomon; Nicholas E Jones; Tucker W Anderson; Wu Gong; Christopher J Lindsell; Travis W Crook; Daisy A Ciener Journal: Med Educ Online Date: 2022-12
Authors: Nico F Leenstra; Oliver C Jung; Fokie Cnossen; A Debbie C Jaarsma; Jaap E Tulleken Journal: Simul Healthc Date: 2021-02-01 Impact factor: 2.690