Andrew Koch Hall1, Jeffrey Damon Dagnone, Lauren Lacroix, William Pickett, Don Albert Klinger. 1. From the Department of Emergency Medicine (A.K.H., J.D.D.), Department of Community Health and Epidemiology (W.P.), and Faculty of Education (D.A.K.), Queen's University, Kingston; Department of Emergency Medicine (L.L.), University of Ottawa, Ottawa, Ontario, Canada.
Abstract
INTRODUCTION: The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees. METHODS: The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination. RESULTS: Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman ρ = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5). CONCLUSIONS: This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.
INTRODUCTION: The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees. METHODS: The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination. RESULTS: Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman ρ = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5). CONCLUSIONS: This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.
Authors: J Damon Dagnone; Andrew K Hall; Stefanie Sebok-Syer; Don Klinger; Karen Woolfrey; Colleen Davison; John Ross; Gordon McNeil; Sean Moore Journal: Can Med Educ J Date: 2016-03-31
Authors: Laura R Thompson; Cynthia G Leung; Brad Green; Jonathan Lipps; Troy Schaffernocker; Cynthia Ledford; John Davis; David P Way; Nicholas E Kman Journal: West J Emerg Med Date: 2016-12-05
Authors: Andrew Koch Hall; J Damon Dagnone; Sean Moore; Karen G H Woolfrey; John A Ross; Gordon McNeil; Carly Hagel; Colleen Davison; Stefanie S Sebok-Syer Journal: AEM Educ Train Date: 2017-09-19
Authors: Michael Jong; Nicole Elliott; Michael Nguyen; Terrence Goyke; Steven Johnson; Matthew Cook; Lisa Lindauer; Katie Best; Douglas Gernerd; Louis Morolla; Zachary Matuzsan; Bryan Kane Journal: West J Emerg Med Date: 2018-12-17