Silvia Quadrelli1, Mohsen Davoudi2, Fernando Galíndez3, Henri G Colt4. 1. Instituto de Investigaciones Médicas Alfredo Lanari, Buenos Aires, Argentina. 2. Department of Pulmonary and Critical Care Medicine, University of California Irvine, UCI Medical Center, Orange, CA. 3. Hospital María Ferrer, Buenos Aires, Argentina. 4. Department of Pulmonary and Critical Care Medicine, University of California Irvine, UCI Medical Center, Orange, CA. Electronic address: hcolt@uci.edu.
Abstract
BACKGROUND: A need for improved patient safety, quality of care, and accountability has prompted the development of competency-based educational processes. Assessment tools related to bronchoscopy training, however, have not yet been developed or validated. PURPOSES: To determine whether 25 multiple-choice questions (MCQs) extracted from the free, Web-based Essential Bronchoscopist (EB) learning guide qualify in their original form as a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge. MATERIALS AND METHODS: Twenty-five randomly selected MCQs from among the top 70 question-answer sets of the EB were administered to 40 self-declared novice bronchoscopists (n = 13), experienced bronchoscopists (n = 21), and expert bronchoscopists (n = 6). A difficulty index and a discrimination index (DI) were calculated for each item. Internal consistency reliability was calculated using item-total correlation and Cronbach alpha. Content validity was determined by five independent experts. Ideal test items based on a difficulty index and item-total correlation were administered to a different group of 24 bronchoscopists to prospectively reassess internal consistency reliability. RESULTS: The mean (+/- SD) score for the 40 participants was 16.47 +/- 3.72 (median score, 17; score range, 7 to 22). The mean difficulty index was 0.65 +/- 0.22, and the mean DI was 0.52 +/- 0.28. Item total-correlations ranged from - 0.01 to + 0.71. Test content was unanimously validated. The Cronbach alpha was 0.69. There was no significant correlation between scores and the number of bronchoscopies performed or self-declared expertise. Eleven ideal test MCQs were identified. The internal consistency of these items remained satisfactory (Cronbach alpha = 0.75) when assessed prospectively in a different cohort. CONCLUSION: Reliable and valid MCQs were identified to initiate a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge.
BACKGROUND: A need for improved patient safety, quality of care, and accountability has prompted the development of competency-based educational processes. Assessment tools related to bronchoscopy training, however, have not yet been developed or validated. PURPOSES: To determine whether 25 multiple-choice questions (MCQs) extracted from the free, Web-based Essential Bronchoscopist (EB) learning guide qualify in their original form as a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge. MATERIALS AND METHODS: Twenty-five randomly selected MCQs from among the top 70 question-answer sets of the EB were administered to 40 self-declared novice bronchoscopists (n = 13), experienced bronchoscopists (n = 21), and expert bronchoscopists (n = 6). A difficulty index and a discrimination index (DI) were calculated for each item. Internal consistency reliability was calculated using item-total correlation and Cronbach alpha. Content validity was determined by five independent experts. Ideal test items based on a difficulty index and item-total correlation were administered to a different group of 24 bronchoscopists to prospectively reassess internal consistency reliability. RESULTS: The mean (+/- SD) score for the 40 participants was 16.47 +/- 3.72 (median score, 17; score range, 7 to 22). The mean difficulty index was 0.65 +/- 0.22, and the mean DI was 0.52 +/- 0.28. Item total-correlations ranged from - 0.01 to + 0.71. Test content was unanimously validated. The Cronbach alpha was 0.69. There was no significant correlation between scores and the number of bronchoscopies performed or self-declared expertise. Eleven ideal test MCQs were identified. The internal consistency of these items remained satisfactory (Cronbach alpha = 0.75) when assessed prospectively in a different cohort. CONCLUSION: Reliable and valid MCQs were identified to initiate a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge.
Authors: Paru Patrawalla; Lewis Ari Eisen; Ariel Shiloh; Brijen J Shah; Oleksandr Savenkov; Wendy Wise; Laura Evans; Paul Mayo; Demian Szyld Journal: J Grad Med Educ Date: 2015-12