Salah Eldin Kassab1, Shereen Hussain. 1. Royal College of Surgeons in Ireland, Medical University of Bahrain, Kingdom of Bahrain. skassab@rcsi-mub.com
Abstract
BACKGROUND: In the problem-based learning (PBL) medical curriculum at the Arabian Gulf University in Bahrain, students construct concept maps related to each case they study in PBL tutorials. AIM: To evaluate the interrater reliability and predictive validity of concept map scores using a structured assessment tool. METHODS: We examined concept maps of the same cohort of students at the beginning (year 2) and end (year 4) of the pre-clerkship phase, where PBL is the main method of instruction. Concept maps were independently evaluated by five raters based on valid selection of concepts, hierarchical arrangement of concepts, integration, relationship to the context of the problem, and degree of student creativity. A 5-point Likert scale was used to evaluate each criterion. Interrater reliability of the instrument was determined using the intraclass correlation coefficient (ICC) and predictive validity was measured by testing the correlations of concept map scores with summative examination scores. RESULTS: The ICC of the concept map scores in year 2 was 0.75 (95% CI, 0.67-0.81) and in year 4 was 0.69 (95% CI, 0.59-0.77). Overall concept maps scores of year 4 students were significantly higher compared with year 2 students (p < 0.001, effect size = 0.5). The relationship between the students' scores in concept maps and their scores in summative examination varied from no to mild correlation. CONCLUSION: The interrater reliability of concept map scores in this study is good to excellent. However, further studies are required to test the generalizability and validity of assessment using this tool.
BACKGROUND: In the problem-based learning (PBL) medical curriculum at the Arabian Gulf University in Bahrain, students construct concept maps related to each case they study in PBL tutorials. AIM: To evaluate the interrater reliability and predictive validity of concept map scores using a structured assessment tool. METHODS: We examined concept maps of the same cohort of students at the beginning (year 2) and end (year 4) of the pre-clerkship phase, where PBL is the main method of instruction. Concept maps were independently evaluated by five raters based on valid selection of concepts, hierarchical arrangement of concepts, integration, relationship to the context of the problem, and degree of student creativity. A 5-point Likert scale was used to evaluate each criterion. Interrater reliability of the instrument was determined using the intraclass correlation coefficient (ICC) and predictive validity was measured by testing the correlations of concept map scores with summative examination scores. RESULTS: The ICC of the concept map scores in year 2 was 0.75 (95% CI, 0.67-0.81) and in year 4 was 0.69 (95% CI, 0.59-0.77). Overall concept maps scores of year 4 students were significantly higher compared with year 2 students (p < 0.001, effect size = 0.5). The relationship between the students' scores in concept maps and their scores in summative examination varied from no to mild correlation. CONCLUSION: The interrater reliability of concept map scores in this study is good to excellent. However, further studies are required to test the generalizability and validity of assessment using this tool.
Authors: Sylvia Vink; Jan van Tartwijk; Nico Verloop; Manon Gosselink; Erik Driessen; Jan Bolk Journal: Adv Health Sci Educ Theory Pract Date: 2015-12-21 Impact factor: 3.853