Dirk R Rutgers1, Fleur van Raamt2, Anouk van der Gijp3, Christian Mol4, Olle Ten Cate5. 1. Department of Radiology, University Medical Center, Heidelberglaan 100, Utrecht 3584 CX, The Netherlands; Radiological Society of the Netherlands, Vught 5260 CB, The Netherlands. Electronic address: d.rutgers@umcutrecht.nl. 2. Department of Radiology, Gelre Hospitals, Apeldoorn 7334 DZ, The Netherlands; Radiological Society of the Netherlands, Vught 5260 CB, The Netherlands. 3. Department of Radiology, University Medical Center, Heidelberglaan 100, Utrecht 3584 CX, The Netherlands. 4. Center for Image Sciences, University Medical Center, Utrecht 3584 CX, The Netherlands. 5. Center for Research and Development of Education, University Medical Center, Utrecht 3508 GA, The Netherlands.
Abstract
RATIONALE AND OBJECTIVES: The psychometric characteristics of image-based test items in radiological written examinations are not well known. In this study, we explored difficulty and discriminating power of these test items in postgraduate radiological digital examinations. MATERIALS AND METHODS: We reviewed test items of seven Dutch Radiology Progress Tests (DRPTs) that were taken from October 2013 to April 2017. The DRPT is a semiannual formative examination, required for all Dutch radiology residents. We assessed several stimulus and response characteristics of test items. The response format of test items included true or false, single right multiple choice with 2, 3, 4, or ≥5 answer options, pick-N multiple-choice, drag-and-drop, and long-list-menu formats. We calculated item P values and item-rest-correlation (Rir) values to assess difficulty and discriminating power. We performed linear regression analysis in image-based test items to investigate whether P and Rir values were significantly related to stimulus and response characteristics. Also, we compared psychometric indices between image-based test items and text-alone items. RESULTS: P and Rir values of image-based items (n = 369) were significantly related to the type of response format (P < .001), and not to which of the seven DRPTs the item was obtained from, radiological subspecialty domain, nonvolumetric or volumetric character of images, or context-rich or context-free character of the stimulus. When accounted for type of response format, difficulty and discriminating power of image-based items did not differ significantly from text-alone items (n = 881). Test items with a relatively large number of answer options were generally more difficult, and discriminated better among high- and low-performing candidates. CONCLUSION: In postgraduate radiological written examinations, difficulty and discriminating power of image-based test items are related to the type of response format and are comparable to those of text-alone items. We recommend a response format with a relatively large number of answer options to optimize psychometric indices of radiological image-based test items.
RATIONALE AND OBJECTIVES: The psychometric characteristics of image-based test items in radiological written examinations are not well known. In this study, we explored difficulty and discriminating power of these test items in postgraduate radiological digital examinations. MATERIALS AND METHODS: We reviewed test items of seven Dutch Radiology Progress Tests (DRPTs) that were taken from October 2013 to April 2017. The DRPT is a semiannual formative examination, required for all Dutch radiology residents. We assessed several stimulus and response characteristics of test items. The response format of test items included true or false, single right multiple choice with 2, 3, 4, or ≥5 answer options, pick-N multiple-choice, drag-and-drop, and long-list-menu formats. We calculated item P values and item-rest-correlation (Rir) values to assess difficulty and discriminating power. We performed linear regression analysis in image-based test items to investigate whether P and Rir values were significantly related to stimulus and response characteristics. Also, we compared psychometric indices between image-based test items and text-alone items. RESULTS: P and Rir values of image-based items (n = 369) were significantly related to the type of response format (P < .001), and not to which of the seven DRPTs the item was obtained from, radiological subspecialty domain, nonvolumetric or volumetric character of images, or context-rich or context-free character of the stimulus. When accounted for type of response format, difficulty and discriminating power of image-based items did not differ significantly from text-alone items (n = 881). Test items with a relatively large number of answer options were generally more difficult, and discriminated better among high- and low-performing candidates. CONCLUSION: In postgraduate radiological written examinations, difficulty and discriminating power of image-based test items are related to the type of response format and are comparable to those of text-alone items. We recommend a response format with a relatively large number of answer options to optimize psychometric indices of radiological image-based test items.
Authors: D R Rutgers; J P J van Schaik; C L J J Kruitwagen; C Haaring; W van Lankeren; A F van Raamt; O Ten Cate Journal: Med Sci Educ Date: 2020-05-13