Celia A Taylor1, Mark Gurnell2, Colin R Melville3, David C Kluth4, Neil Johnson3, Val Wass5. 1. Division of Health Sciences, Warwick Medical School, University of Warwick, Coventry, UK. 2. Department of Medicine, School of Clinical Medicine, University of Cambridge, Addenbrooke's Hospital, Cambridge, UK. 3. Lancaster Medical School, Faculty of Health and Medicine, Lancaster University, Lancaster, UK. 4. Medical Research Council Centre for Inflammation Research, University of Edinburgh, Edinburgh, UK. 5. Department of Medical Education, Faculty of Health, Keele University, Keele, UK.
Abstract
OBJECTIVES: Given the absence of a common passing standard for students at UK medical schools, this paper compares independently set standards for common 'one from five' single-best-answer (multiple-choice) items used in graduation-level applied knowledge examinations and explores potential reasons for any differences. METHODS: A repeated cross-sectional study was conducted. Participating schools were sent a common set of graduation-level items (55 in 2013-2014; 60 in 2014-2015). Items were selected against a blueprint and subjected to a quality review process. Each school employed its own standard-setting process for the common items. The primary outcome was the passing standard for the common items by each medical school set using the Angoff or Ebel methods. RESULTS: Of 31 invited medical schools, 22 participated in 2013-2014 (71%) and 30 (97%) in 2014-2015. Schools used a mean of 49 and 53 common items in 2013-2014 and 2014-2015, respectively, representing around one-third of the items in the examinations in which they were embedded. Data from 19 (61%) and 26 (84%) schools, respectively, met the inclusion criteria for comparison of standards. There were statistically significant differences in the passing standards set by schools in both years (effect sizes (f2 ): 0.041 in 2013-2014 and 0.218 in 2014-2015; both p < 0.001). The interquartile range of standards was 5.7 percentage points in 2013-2014 and 6.5 percentage points in 2014-2015. There was a positive correlation between the relative standards set by schools in the 2 years (Pearson's r = 0.57, n = 18, p = 0.014). Time allowed per item, method of standard setting and timing of examination in the curriculum did not have a statistically significant impact on standards. CONCLUSIONS: Independently set standards for common single-best-answer items used in graduation-level examinations vary across UK medical schools. Further work to examine standard-setting processes in more detail is needed to help explain this variability and develop methods to reduce it.
OBJECTIVES: Given the absence of a common passing standard for students at UK medical schools, this paper compares independently set standards for common 'one from five' single-best-answer (multiple-choice) items used in graduation-level applied knowledge examinations and explores potential reasons for any differences. METHODS: A repeated cross-sectional study was conducted. Participating schools were sent a common set of graduation-level items (55 in 2013-2014; 60 in 2014-2015). Items were selected against a blueprint and subjected to a quality review process. Each school employed its own standard-setting process for the common items. The primary outcome was the passing standard for the common items by each medical school set using the Angoff or Ebel methods. RESULTS: Of 31 invited medical schools, 22 participated in 2013-2014 (71%) and 30 (97%) in 2014-2015. Schools used a mean of 49 and 53 common items in 2013-2014 and 2014-2015, respectively, representing around one-third of the items in the examinations in which they were embedded. Data from 19 (61%) and 26 (84%) schools, respectively, met the inclusion criteria for comparison of standards. There were statistically significant differences in the passing standards set by schools in both years (effect sizes (f2 ): 0.041 in 2013-2014 and 0.218 in 2014-2015; both p < 0.001). The interquartile range of standards was 5.7 percentage points in 2013-2014 and 6.5 percentage points in 2014-2015. There was a positive correlation between the relative standards set by schools in the 2 years (Pearson's r = 0.57, n = 18, p = 0.014). Time allowed per item, method of standard setting and timing of examination in the curriculum did not have a statistically significant impact on standards. CONCLUSIONS: Independently set standards for common single-best-answer items used in graduation-level examinations vary across UK medical schools. Further work to examine standard-setting processes in more detail is needed to help explain this variability and develop methods to reduce it.
Authors: I C McManus; Andrew Christopher Harborne; Hugo Layard Horsfall; Tobin Joseph; Daniel T Smith; Tess Marshall-Andon; Ryan Samuels; Joshua William Kearsley; Nadine Abbas; Hassan Baig; Joseph Beecham; Natasha Benons; Charlie Caird; Ryan Clark; Thomas Cope; James Coultas; Luke Debenham; Sarah Douglas; Jack Eldridge; Thomas Hughes-Gooding; Agnieszka Jakubowska; Oliver Jones; Eve Lancaster; Calum MacMillan; Ross McAllister; Wassim Merzougui; Ben Phillips; Simon Phillips; Omar Risk; Adam Sage; Aisha Sooltangos; Robert Spencer; Roxanne Tajbakhsh; Oluseyi Adesalu; Ivan Aganin; Ammar Ahmed; Katherine Aiken; Alimatu-Sadia Akeredolu; Ibrahim Alam; Aamna Ali; Richard Anderson; Jia Jun Ang; Fady Sameh Anis; Sonam Aojula; Catherine Arthur; Alena Ashby; Ahmed Ashraf; Emma Aspinall; Mark Awad; Abdul-Muiz Azri Yahaya; Shreya Badhrinarayanan; Soham Bandyopadhyay; Sam Barnes; Daisy Bassey-Duke; Charlotte Boreham; Rebecca Braine; Joseph Brandreth; Zoe Carrington; Zoe Cashin; Shaunak Chatterjee; Mehar Chawla; Chung Shen Chean; Chris Clements; Richard Clough; Jessica Coulthurst; Liam Curry; Vinnie Christine Daniels; Simon Davies; Rebecca Davis; Hanelie De Waal; Nasreen Desai; Hannah Douglas; James Druce; Lady-Namera Ejamike; Meron Esere; Alex Eyre; Ibrahim Talal Fazmin; Sophia Fitzgerald-Smith; Verity Ford; Sarah Freeston; Katherine Garnett; Whitney General; Helen Gilbert; Zein Gowie; Ciaran Grafton-Clarke; Keshni Gudka; Leher Gumber; Rishi Gupta; Chris Harlow; Amy Harrington; Adele Heaney; Wing Hang Serene Ho; Lucy Holloway; Christina Hood; Eleanor Houghton; Saba Houshangi; Emma Howard; Benjamin Human; Harriet Hunter; Ifrah Hussain; Sami Hussain; Richard Thomas Jackson-Taylor; Bronwen Jacob-Ramsdale; Ryan Janjuha; Saleh Jawad; Muzzamil Jelani; David Johnston; Mike Jones; Sadhana Kalidindi; Savraj Kalsi; Asanish Kalyanasundaram; Anna Kane; Sahaj Kaur; Othman Khaled Al-Othman; Qaisar Khan; Sajan Khullar; Priscilla Kirkland; Hannah Lawrence-Smith; Charlotte Leeson; Julius Elisabeth Richard Lenaerts; Kerry Long; Simon Lubbock; Jamie Mac Donald Burrell; Rachel Maguire; Praveen Mahendran; Saad Majeed; Prabhjot Singh Malhotra; Vinay Mandagere; Angelos Mantelakis; Sophie McGovern; Anjola Mosuro; Adam Moxley; Sophie Mustoe; Sam Myers; Kiran Nadeem; Reza Nasseri; Tom Newman; Richard Nzewi; Rosalie Ogborne; Joyce Omatseye; Sophie Paddock; James Parkin; Mohit Patel; Sohini Pawar; Stuart Pearce; Samuel Penrice; Julian Purdy; Raisa Ramjan; Ratan Randhawa; Usman Rasul; Elliot Raymond-Taggert; Rebecca Razey; Carmel Razzaghi; Eimear Reel; Elliot John Revell; Joanna Rigbye; Oloruntobi Rotimi; Abdelrahman Said; Emma Sanders; Pranoy Sangal; Nora Sangvik Grandal; Aadam Shah; Rahul Atul Shah; Oliver Shotton; Daniel Sims; Katie Smart; Martha Amy Smith; Nick Smith; Aninditya Salma Sopian; Matthew South; Jessica Speller; Tom J Syer; Ngan Hong Ta; Daniel Tadross; Benjamin Thompson; Jess Trevett; Matthew Tyler; Roshan Ullah; Mrudula Utukuri; Shree Vadera; Harriet Van Den Tooren; Sara Venturini; Aradhya Vijayakumar; Melanie Vine; Zoe Wellbelove; Liora Wittner; Geoffrey Hong Kiat Yong; Farris Ziyada; Oliver Patrick Devine Journal: BMC Med Date: 2020-05-14 Impact factor: 8.775
Authors: Helena Ward; Neville Chiavaroli; James Fraser; Kylie Mansfield; Darren Starmer; Laura Surmon; Martin Veysey; Deborah O'Mara Journal: BMC Med Educ Date: 2018-04-23 Impact factor: 2.463
Authors: Peter Yeates; Alice Moult; Natalie Cope; Gareth McCray; Eleftheria Xilas; Tom Lovelock; Nicholas Vaughan; Dan Daw; Richard Fuller; Robert K Bob McKinley Journal: Acad Med Date: 2021-03-02 Impact factor: 7.840