Samantha Halman1,2, Angel Yi Nam Fu2, Debra Pugh1,2,3. 1. Department of Medicine, The Ottawa Hospital, Ottawa, Ontario, Canada. 2. Faculty of Medicine, The University of Ottawa, Ottawa, Ontario, Canada. 3. Medical Council of Canada, Ottawa, Ontario, Canada.
Abstract
PURPOSE: Progress testing aligns well with competency-based medical education (CBME) frameworks, which stress the importance of continuous improvement. Entrustment is a useful assessment concept in CBME models. The purpose of this study was to explore the use of an entrustability rating scale within the context of an objective structured clinical examination (OSCE) Progress Test. METHODS: A 9-case OSCE Progress Test was administered to Internal Medicine residents (PGYs 1-4). Residents were assessed using a checklist (CL), global rating scale (GRS), training level rating scale (TLRS), and entrustability scale (ENT). Reliability was calculated using Cronbach's alpha. Differences in performance by training year were explored using ANOVA and effect sizes were calculated using partial eta-squared. Examiners completed a post-examination survey. RESULTS: Ninety one residents and forty two examiners participated in the OSCE. Inter-station reliability was high for all instruments. There was an overall effect of training level for all instruments (p < 0.001). Effect sizes were large. 88% of examiners completed the survey. Most (62%) indicated feeling comfortable in making entrustment decisions during the OSCE. CONCLUSIONS: An entrustability scale can be used in an OSCE Progress Test to generate highly reliable ratings that discriminate between learners at different levels of training.
PURPOSE: Progress testing aligns well with competency-based medical education (CBME) frameworks, which stress the importance of continuous improvement. Entrustment is a useful assessment concept in CBME models. The purpose of this study was to explore the use of an entrustability rating scale within the context of an objective structured clinical examination (OSCE) Progress Test. METHODS: A 9-case OSCE Progress Test was administered to Internal Medicine residents (PGYs 1-4). Residents were assessed using a checklist (CL), global rating scale (GRS), training level rating scale (TLRS), and entrustability scale (ENT). Reliability was calculated using Cronbach's alpha. Differences in performance by training year were explored using ANOVA and effect sizes were calculated using partial eta-squared. Examiners completed a post-examination survey. RESULTS: Ninety one residents and forty two examiners participated in the OSCE. Inter-station reliability was high for all instruments. There was an overall effect of training level for all instruments (p < 0.001). Effect sizes were large. 88% of examiners completed the survey. Most (62%) indicated feeling comfortable in making entrustment decisions during the OSCE. CONCLUSIONS: An entrustability scale can be used in an OSCE Progress Test to generate highly reliable ratings that discriminate between learners at different levels of training.
Authors: Pavlos Bobos; Dimitra V Pouliopoulou; Alexandra Harriss; Jackie Sadi; Alison Rushton; Joy C MacDermid Journal: PLoS One Date: 2021-08-03 Impact factor: 3.240