BACKGROUND AND OBJECTIVES: Programs of assessment should reflect the multifaceted nature of medical competence. We experimented with new testing methods, ie, script concordance testing (SCT) and clinical reasoning problems (CRPs), combined with the habitual OSCE for an end of family medicine clerkship. Our aims were to compare students' scores with experts' scores, to determine whether the new tests detected learning over a 3-month period, and to examine whether the tests were redundant. METHODS: We conducted a longitudinal study on one cohort of family medicine clerks. Two formative testing sessions using both SCT and CRPs were held 3 months apart. Students' scores were compared to those of the panel of experts used to score the tests. We examined the difference in students' scores between the two testing sessions. Finally, we computed correlation coefficients between these scores and the summative OSCE. RESULTS: Panelists' scores were significantly higher than students' scores. SCT scores did not change significantly over 3 months whereas CRP scores improved (Wilcoxon z -3.058, effect size 0.461, P=.002). Correlations between the OSCE and the written tests were low or non-significant. There were low correlations between the first CRP and both SCTs (Spearman's rho 0.357 and 0.358) but not between the second CRP and any SCT. CONCLUSIONS: Written tests of clinical reasoning could provide relevant additional information to the evaluation of students' competence over the course of a family medicine clerkship. Further research is needed to determine the potential educational consequences of such programs of assessment.
BACKGROUND AND OBJECTIVES: Programs of assessment should reflect the multifaceted nature of medical competence. We experimented with new testing methods, ie, script concordance testing (SCT) and clinical reasoning problems (CRPs), combined with the habitual OSCE for an end of family medicine clerkship. Our aims were to compare students' scores with experts' scores, to determine whether the new tests detected learning over a 3-month period, and to examine whether the tests were redundant. METHODS: We conducted a longitudinal study on one cohort of family medicine clerks. Two formative testing sessions using both SCT and CRPs were held 3 months apart. Students' scores were compared to those of the panel of experts used to score the tests. We examined the difference in students' scores between the two testing sessions. Finally, we computed correlation coefficients between these scores and the summative OSCE. RESULTS: Panelists' scores were significantly higher than students' scores. SCT scores did not change significantly over 3 months whereas CRP scores improved (Wilcoxon z -3.058, effect size 0.461, P=.002). Correlations between the OSCE and the written tests were low or non-significant. There were low correlations between the first CRP and both SCTs (Spearman's rho 0.357 and 0.358) but not between the second CRP and any SCT. CONCLUSIONS: Written tests of clinical reasoning could provide relevant additional information to the evaluation of students' competence over the course of a family medicine clerkship. Further research is needed to determine the potential educational consequences of such programs of assessment.