David Richens1, Timothy R Graham2, John James3, Hettie Till4, Philip G Turner2, Cara Featherstone2. 1. Joint Committee on Intercollegiate Examinations, Edinburgh, United Kingdom. Electronic address: m.gray@jcie.org.uk. 2. Joint Committee on Intercollegiate Examinations, Edinburgh, United Kingdom. 3. The Sickle Cell Society, London, United Kingdom. 4. Assessment Support and Psychometric Services, Dundee, United Kingdom.
Abstract
INTRODUCTION: We explored effects of gender, ethnic origin, first language, and training status on scores in the Intercollegiate Specialty Board examinations in the UK and Ireland across the computer-marked written section and in the face-to-face oral and clinical section. METHODS: Demographic characteristics and examination results from 9987 attempts across 177 sittings from 2009 to 2013 were analyzed in an analysis of variance by training status, gender, ethnic origin, first language, and section (computer-marked multiple-choice examination vs face-to-face oral and clinical examination). RESULTS: We found increasing alignment between examiner and candidate characteristics during this period, with a 50% increase in examiners of Asian ethnic origin and a 60% increase in examiners whose first language is not English. The strongest factor in the analysis of variance was training status (F[2, 9818] = 27.67, p < 0.001), with candidates in training significantly outperforming others. Within "core candidates" (first attempt, in training), we found significant main effects for ethnic origin (F[5, 4809] = 2.36, p = 0.04), and first language (F[2, 4809] = 5.29, p = 0.003), but no interaction effects between these factors and section (both F < 1, p > 0.05). CONCLUSIONS: Training status was the most important factor in candidates' results. Although the analysis showed significant effects of ethnic origin and first language within "core candidates," these differences were statistically indistinguishable between the 2 sections of the examination, suggesting that the differential attainment by these factors cannot be attributed to examiner bias in a face-to-face examination.
INTRODUCTION: We explored effects of gender, ethnic origin, first language, and training status on scores in the Intercollegiate Specialty Board examinations in the UK and Ireland across the computer-marked written section and in the face-to-face oral and clinical section. METHODS: Demographic characteristics and examination results from 9987 attempts across 177 sittings from 2009 to 2013 were analyzed in an analysis of variance by training status, gender, ethnic origin, first language, and section (computer-marked multiple-choice examination vs face-to-face oral and clinical examination). RESULTS: We found increasing alignment between examiner and candidate characteristics during this period, with a 50% increase in examiners of Asian ethnic origin and a 60% increase in examiners whose first language is not English. The strongest factor in the analysis of variance was training status (F[2, 9818] = 27.67, p < 0.001), with candidates in training significantly outperforming others. Within "core candidates" (first attempt, in training), we found significant main effects for ethnic origin (F[5, 4809] = 2.36, p = 0.04), and first language (F[2, 4809] = 5.29, p = 0.003), but no interaction effects between these factors and section (both F < 1, p > 0.05). CONCLUSIONS: Training status was the most important factor in candidates' results. Although the analysis showed significant effects of ethnic origin and first language within "core candidates," these differences were statistically indistinguishable between the 2 sections of the examination, suggesting that the differential attainment by these factors cannot be attributed to examiner bias in a face-to-face examination.
Keywords:
Interpersonal and Communication Skills; Medical Knowledge; Patient Care; Systems-Based Practice; differential attainment; equality and diversity; surgical examinations
Authors: Ricky Ellis; Peter A Brennan; Amanda J Lee; Duncan Sg Scrimgeour; Jennifer Cleland Journal: J R Soc Med Date: 2022-02-16 Impact factor: 18.000