Andrew Koch Hall1, J Damon Dagnone1, Sean Moore2, Karen G H Woolfrey3, John A Ross4, Gordon McNeil5, Carly Hagel1, Colleen Davison1,6, Stefanie S Sebok-Syer7. 1. Department of Emergency Medicine Queen's University Kingston Ontario Canada. 2. Department of Emergency Medicine Northern Ontario School of Medicine Kenora Ontario Canada. 3. Department of Medicine Western University London Ontario Canada. 4. Department of Emergency Medicine Dalhousie University Halifax Nova Scotia Canada. 5. Department of Emergency Medicine University of Calgary Calgary Alberta Canada. 6. Department of Public Health Sciences Queen's University Kingston Ontario Canada. 7. Centre for Education Research & Innovation Schulich School of Medicine and Dentistry Western University London Ontario Canada.
Abstract
OBJECTIVE: Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment. METHODS: A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods. RESULTS: Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02). CONCLUSIONS: Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.
OBJECTIVE: Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment. METHODS: A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods. RESULTS: Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02). CONCLUSIONS: Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.
Authors: Jason R Frank; Linda S Snell; Olle Ten Cate; Eric S Holmboe; Carol Carraccio; Susan R Swing; Peter Harris; Nicholas J Glasgow; Craig Campbell; Deepak Dath; Ronald M Harden; William Iobst; Donlin M Long; Rani Mungroo; Denyse L Richardson; Jonathan Sherbino; Ivan Silver; Sarah Taber; Martin Talbot; Kenneth A Harris Journal: Med Teach Date: 2010 Impact factor: 3.650
Authors: Andrew Koch Hall; Jeffrey Damon Dagnone; Lauren Lacroix; William Pickett; Don Albert Klinger Journal: Simul Healthc Date: 2015-04 Impact factor: 1.929
Authors: J Damon Dagnone; Andrew K Hall; Stefanie Sebok-Syer; Don Klinger; Karen Woolfrey; Colleen Davison; John Ross; Gordon McNeil; Sean Moore Journal: Can Med Educ J Date: 2016-03-31
Authors: Andrew Koch Hall; J Damon Dagnone; Sean Moore; Karen G H Woolfrey; John A Ross; Gordon McNeil; Carly Hagel; Colleen Davison; Stefanie S Sebok-Syer Journal: AEM Educ Train Date: 2017-09-19
Authors: Andrew Koch Hall; J Damon Dagnone; Sean Moore; Karen G H Woolfrey; John A Ross; Gordon McNeil; Carly Hagel; Colleen Davison; Stefanie S Sebok-Syer Journal: AEM Educ Train Date: 2017-09-19