OBJECTIVE: To improve examination item quality by educating and involving course instructors in evidence-based item review and encouraging use of this process in future courses. METHODS: A peer-review process was implemented in a 2-course sequence (intervention) that involved training and review sessions before each examination and was compared to the previous year's courses (control). Instructors completed a presurvey and postsurvey on training, experience, self-confidence, and self-rated success in multiple-choice item writing. Statistics were calculated for all items in the control and intervention sequences and compared using independent t tests. Items also were classified into levels based on difficulty and discrimination, and distribution into these levels was compared between sequences with independent t tests. RESULTS: No significant difference was found between control and intervention sequence items with regard to mean difficulty (86.3% and 84.4%) or discrimination (0.23- and 0.25), respectively, although item classification distribution did appear to change between the control and intervention sequences' subjective feelings of confidence, and success in item writing increased between presurvey and postsurvey. Confidence in ability to peer-review test items and to implement a formal item evaluation process also increased. CONCLUSION: Item statistics did not change significantly, but reviewed and edited items distributed more favorably into item statistic-based categories. This method of review positively affected instructors' perceptions of their item-writing confidence and success and improved self-rated opinions of their ability to edit items and train others to do so.
OBJECTIVE: To improve examination item quality by educating and involving course instructors in evidence-based item review and encouraging use of this process in future courses. METHODS: A peer-review process was implemented in a 2-course sequence (intervention) that involved training and review sessions before each examination and was compared to the previous year's courses (control). Instructors completed a presurvey and postsurvey on training, experience, self-confidence, and self-rated success in multiple-choice item writing. Statistics were calculated for all items in the control and intervention sequences and compared using independent t tests. Items also were classified into levels based on difficulty and discrimination, and distribution into these levels was compared between sequences with independent t tests. RESULTS: No significant difference was found between control and intervention sequence items with regard to mean difficulty (86.3% and 84.4%) or discrimination (0.23- and 0.25), respectively, although item classification distribution did appear to change between the control and intervention sequences' subjective feelings of confidence, and success in item writing increased between presurvey and postsurvey. Confidence in ability to peer-review test items and to implement a formal item evaluation process also increased. CONCLUSION: Item statistics did not change significantly, but reviewed and edited items distributed more favorably into item statistic-based categories. This method of review positively affected instructors' perceptions of their item-writing confidence and success and improved self-rated opinions of their ability to edit items and train others to do so.
Keywords:
classroom assessment; item-writing; multiple-choice; peer review; test construction
Authors: Ralph F Jozefowicz; Bruce M Koeppen; Susan Case; Robert Galbraith; David Swanson; Robert H Glew Journal: Acad Med Date: 2002-02 Impact factor: 6.893