Angela D Blood1, Yoon Soo Park2, Rimas V Lukas2, James R Brorson2. 1. From Rush University Medical College (A.D.B.); Department of Medical Education (Y.S.P.), The University of Illinois at Chicago; and Department of Neurology (R.V.L., J.R.B.), The University of Chicago, IL. angela_blood@rush.edu. 2. From Rush University Medical College (A.D.B.); Department of Medical Education (Y.S.P.), The University of Illinois at Chicago; and Department of Neurology (R.V.L., J.R.B.), The University of Chicago, IL.
Abstract
OBJECTIVES: This study examines factors affecting reliability, or consistency of assessment scores, from an objective structured clinical examination (OSCE) in neurology through generalizability theory (G theory). METHODS: Data include assessments from a multistation OSCE taken by 194 medical students at the completion of a neurology clerkship. Facets evaluated in this study include cases, domains, and items. Domains refer to areas of skill (or constructs) that the OSCE measures. G theory is used to estimate variance components associated with each facet, derive reliability, and project the number of cases required to obtain a reliable (consistent, precise) score. RESULTS: Reliability using G theory is moderate (Φ coefficient = 0.61, G coefficient = 0.64). Performance is similar across cases but differs by the particular domain, such that the majority of variance is attributed to the domain. Projections in reliability estimates reveal that students need to participate in 3 OSCE cases in order to increase reliability beyond the 0.70 threshold. CONCLUSIONS: This novel use of G theory in evaluating an OSCE in neurology provides meaningful measurement characteristics of the assessment. Differing from prior work in other medical specialties, the cases students were randomly assigned did not influence their OSCE score; rather, scores varied in expected fashion by domain assessed.
RCT Entities:
OBJECTIVES: This study examines factors affecting reliability, or consistency of assessment scores, from an objective structured clinical examination (OSCE) in neurology through generalizability theory (G theory). METHODS: Data include assessments from a multistation OSCE taken by 194 medical students at the completion of a neurology clerkship. Facets evaluated in this study include cases, domains, and items. Domains refer to areas of skill (or constructs) that the OSCE measures. G theory is used to estimate variance components associated with each facet, derive reliability, and project the number of cases required to obtain a reliable (consistent, precise) score. RESULTS: Reliability using G theory is moderate (Φ coefficient = 0.61, G coefficient = 0.64). Performance is similar across cases but differs by the particular domain, such that the majority of variance is attributed to the domain. Projections in reliability estimates reveal that students need to participate in 3 OSCE cases in order to increase reliability beyond the 0.70 threshold. CONCLUSIONS: This novel use of G theory in evaluating an OSCE in neurology provides meaningful measurement characteristics of the assessment. Differing from prior work in other medical specialties, the cases students were randomly assigned did not influence their OSCE score; rather, scores varied in expected fashion by domain assessed.
Authors: Amanda K Lamarre; Katya Rascovsky; Alan Bostrom; Parnian Toofanian; Sarah Wilkins; Sharon J Sha; David C Perry; Zachary A Miller; Georges Naasan; Robert Laforce; Jayne Hagen; Leonel T Takada; Maria Carmela Tartaglia; Gail Kang; Douglas Galasko; David P Salmon; Sarah Tomaszewski Farias; Berneet Kaur; John M Olichney; Lovingly Quitania Park; Mario F Mendez; Po-Heng Tsai; Edmond Teng; Bradford Clark Dickerson; Kimiko Domoto-Reilly; Scott McGinnis; Bruce L Miller; Joel H Kramer Journal: Neurology Date: 2013-05-01 Impact factor: 9.910
Authors: Aaron M Spring; Daniel J Pittman; Yahya Aghakhani; Jeffrey Jirsch; Neelan Pillay; Luis E Bello-Espinosa; Colin Josephson; Paolo Federico Journal: Front Neurol Date: 2018-06-28 Impact factor: 4.003