R M Smith1. 1. University of Hawaii at Manoa, John A. Burns School of Medicine, Honolulu 96822.
Abstract
BACKGROUND: The three-step triple jump (TJ) examination aims to assess students' clinical problem-solving processes predominantly by means of subjective assessments administered by faculty. But training TJ administrators to ensure interrater reliability is both time- and cost-intensive, and difficult at best--hence the desire to test a more objective system of scoring students' TJ performances. METHOD: The sample was the 58 first-year students of the class of 1995, who in March 1992 were finishing the second 13-week unit in the problem-based curriculum at the University of Hawaii John A. Burns School of Medicine. To determine how well the school had succeeded in standardizing and objectifying its TJ examination (used for all units in the first two years), scores were correlated for various objective examinations independent of the TJ, the TJ administrators' subjective assessments, and the TJ objective assessments (regarding number of problem-based hypotheses generated and number of hypothesis-testing clinical database items elicited). The statistical methods used were linear regression, Student's unpaired t-test, chi-square, and the z-test. RESULTS: The TJ scores and independent objective examinations did not correlate significantly (suggesting that they assess different aspects of student achievement), but the TJ subjective and objective scores did correlate significantly. There were large standard deviations on the TJ objective scores, largely because one problem was significantly more difficult than the others (each student works on only one of several problems). However, the administrators' subjective scores for all problems were comparable. CONCLUSION: Because problems vary in difficulty, objective scores cannot be used across problems as a major component of all students' grades; but when a student has received an unsatisfactory score, an external reviewer can evaluate the appropriateness of the subjective score by comparing the student's objective performance with those of students who had the same problem and received higher subjective scores. That the administrators' subjective assessments for all problems were comparable not only suggests that the administrators were able to adjust for problem variability but also reinforces the appropriateness of using subjective assessments for the TJ examination.
BACKGROUND: The three-step triple jump (TJ) examination aims to assess students' clinical problem-solving processes predominantly by means of subjective assessments administered by faculty. But training TJ administrators to ensure interrater reliability is both time- and cost-intensive, and difficult at best--hence the desire to test a more objective system of scoring students' TJ performances. METHOD: The sample was the 58 first-year students of the class of 1995, who in March 1992 were finishing the second 13-week unit in the problem-based curriculum at the University of Hawaii John A. Burns School of Medicine. To determine how well the school had succeeded in standardizing and objectifying its TJ examination (used for all units in the first two years), scores were correlated for various objective examinations independent of the TJ, the TJ administrators' subjective assessments, and the TJ objective assessments (regarding number of problem-based hypotheses generated and number of hypothesis-testing clinical database items elicited). The statistical methods used were linear regression, Student's unpaired t-test, chi-square, and the z-test. RESULTS: The TJ scores and independent objective examinations did not correlate significantly (suggesting that they assess different aspects of student achievement), but the TJ subjective and objective scores did correlate significantly. There were large standard deviations on the TJ objective scores, largely because one problem was significantly more difficult than the others (each student works on only one of several problems). However, the administrators' subjective scores for all problems were comparable. CONCLUSION: Because problems vary in difficulty, objective scores cannot be used across problems as a major component of all students' grades; but when a student has received an unsatisfactory score, an external reviewer can evaluate the appropriateness of the subjective score by comparing the student's objective performance with those of students who had the same problem and received higher subjective scores. That the administrators' subjective assessments for all problems were comparable not only suggests that the administrators were able to adjust for problem variability but also reinforces the appropriateness of using subjective assessments for the TJ examination.
Authors: Jan Matthes; Alexander Look; Amina K Hahne; Ara Tekian; Stefan Herzig Journal: Naunyn Schmiedebergs Arch Pharmacol Date: 2008-01-11 Impact factor: 3.000