John J Norcini1. 1. Foundation for Advancement of International Medical Education and Research (FAIMER), Philadelphia, Pennsylvania, USA. jnorcini@ecfmg.org
Abstract
OBJECTIVE: This instalment in the series on professional assessment provides an introduction to methods of setting standards. METHOD: A standard is a special score that serves as a boundary between those who perform well enough and those who do not. The practical steps in selecting it include: deciding on the type of standard; deciding the method for setting it; selecting the judges; holding the meeting; calculating the cutpoint, and deciding what to do afterwards. Four of the more popular methods are illustrated for both written and clinical examinations. RESULTS: The most important criteria for selecting a method for setting standards are whether it is consistent with the purpose of the test, based on expert judgement, informed by data, supported by research, transparent, and requires due diligence. The credibility of the standard will rely largely on the nature of the standard setters and the selection of a broadly representative and knowledgeable group is essential. After the standard has been set, it is important to ensure that stakeholders view the results as credible and that the pass rates have sensible relationships with other markers of competence. CONCLUSIONS: A standard is an expression of professional values in the context of a test's purpose and content, the ability of the examinees, and the wider social or educational setting. Because standards are an expression of values, methods for setting them are systematic ways of gathering value judgements, reaching consensus and expressing that consensus as a single score on a test.
OBJECTIVE: This instalment in the series on professional assessment provides an introduction to methods of setting standards. METHOD: A standard is a special score that serves as a boundary between those who perform well enough and those who do not. The practical steps in selecting it include: deciding on the type of standard; deciding the method for setting it; selecting the judges; holding the meeting; calculating the cutpoint, and deciding what to do afterwards. Four of the more popular methods are illustrated for both written and clinical examinations. RESULTS: The most important criteria for selecting a method for setting standards are whether it is consistent with the purpose of the test, based on expert judgement, informed by data, supported by research, transparent, and requires due diligence. The credibility of the standard will rely largely on the nature of the standard setters and the selection of a broadly representative and knowledgeable group is essential. After the standard has been set, it is important to ensure that stakeholders view the results as credible and that the pass rates have sensible relationships with other markers of competence. CONCLUSIONS: A standard is an expression of professional values in the context of a test's purpose and content, the ability of the examinees, and the wider social or educational setting. Because standards are an expression of values, methods for setting them are systematic ways of gathering value judgements, reaching consensus and expressing that consensus as a single score on a test.
Authors: Diana L Miglioretti; Laura Ichikawa; Robert A Smith; Lawrence W Bassett; Stephen A Feig; Barbara Monsees; Jay R Parikh; Robert D Rosenberg; Edward A Sickles; Patricia A Carney Journal: AJR Am J Roentgenol Date: 2015-04 Impact factor: 3.959
Authors: Patricia A Carney; Edward A Sickles; Barbara S Monsees; Lawrence W Bassett; R James Brenner; Stephen A Feig; Robert A Smith; Robert D Rosenberg; T Andrew Bogart; Sally Browning; Jane W Barry; Mary M Kelly; Khai A Tran; Diana L Miglioretti Journal: Radiology Date: 2010-05 Impact factor: 11.105