Peter A Brennan1, David T Croke2, Malcolm Reed3, Lee Smith4, Euan Munro4, John Foulkes4, Richard Arnett2. 1. Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom. Electronic address: Peter.brennan@porthosp.nhs.uk. 2. Department of Quality Enhancement, The Royal College of Surgeons in Ireland, Dublin, Ireland. 3. Dean, Brighton Medical School, Brighton, United Kingdom. 4. Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom.
Abstract
OBJECTIVE: Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates' scores compared with examiners who examined the same scenario all day. DESIGN, SETTING, AND PARTICIPANTS: An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach's alpha, and the comparative distribution of total candidates' scores for each day was evaluated using t-tests of unit-weighted z scores. RESULTS: A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching. CONCLUSIONS: This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.
OBJECTIVE: Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates' scores compared with examiners who examined the same scenario all day. DESIGN, SETTING, AND PARTICIPANTS: An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach's alpha, and the comparative distribution of total candidates' scores for each day was evaluated using t-tests of unit-weighted z scores. RESULTS: A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching. CONCLUSIONS: This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.
Keywords:
Interpersonal and Communication Skills; Medical Knowledge; Practice-Based Learning and Improvement; Professionalism; Systems-Based Practice; examiner; objective structured clinical examination; reliability; scenario; surgery
Authors: Dsg Scrimgeour; R Patel; N Patel; J Cleland; A J Lee; A J McKinley; Fct Smith; G Griffiths; P A Brennan Journal: Ann R Coll Surg Engl Date: 2019-02-18 Impact factor: 1.891
Authors: Boaz Shulruf; Arvin Damodaran; Phil Jones; Sean Kennedy; George Mangos; Anthony J O'Sullivan; Joel Rhee; Silas Taylor; Gary Velan; Peter Harris Journal: BMC Med Educ Date: 2018-01-06 Impact factor: 2.463
Authors: Peter Yeates; Alice Moult; Natalie Cope; Gareth McCray; Eleftheria Xilas; Tom Lovelock; Nicholas Vaughan; Dan Daw; Richard Fuller; Robert K Bob McKinley Journal: Acad Med Date: 2021-03-02 Impact factor: 7.840