BACKGROUND: Exit examinations in medicine are 'high stakes' examinations and as such must satisfy a number of criteria including psychometric robustness, fairness and reliability in the face of legal or other challenges. AIMS: We have undertaken a critical review of the exit examination from the University of Adelaide focussing on the written components. This examination consisted of an objective structure clinical examination (OSCE), a multiple choice question (MCQ) paper and a modified essay question (MEQ) paper. METHODS: The two written papers were assessed for item writing flaws and taxonomic level using modified Bloom's criteria. Curriculum experts independently assessed adequacy of the examination for validity and fidelity. RESULTS: The overall examination had good fidelity and validity. The results of the MEQ and MCQ were strongly and positively correlated and there was a weak negative correlation between these papers and the OSCE. The MEQ had a higher proportion of questions focussed on recall of knowledge and the questions were more structurally flawed compared with the MCQs. The MEQ re-marking process resulted in lower scores than were awarded by the original, discipline-based expert markers. The MEQ paper failed to achieve its primary purpose of assessing higher cognitive skills. CONCLUSION: The University of Adelaide's MBBS programme has since dropped the MEQ paper from its exit examination and is evaluating in its place the Script Concordance test.
BACKGROUND: Exit examinations in medicine are 'high stakes' examinations and as such must satisfy a number of criteria including psychometric robustness, fairness and reliability in the face of legal or other challenges. AIMS: We have undertaken a critical review of the exit examination from the University of Adelaide focussing on the written components. This examination consisted of an objective structure clinical examination (OSCE), a multiple choice question (MCQ) paper and a modified essay question (MEQ) paper. METHODS: The two written papers were assessed for item writing flaws and taxonomic level using modified Bloom's criteria. Curriculum experts independently assessed adequacy of the examination for validity and fidelity. RESULTS: The overall examination had good fidelity and validity. The results of the MEQ and MCQ were strongly and positively correlated and there was a weak negative correlation between these papers and the OSCE. The MEQ had a higher proportion of questions focussed on recall of knowledge and the questions were more structurally flawed compared with the MCQs. The MEQ re-marking process resulted in lower scores than were awarded by the original, discipline-based expert markers. The MEQ paper failed to achieve its primary purpose of assessing higher cognitive skills. CONCLUSION: The University of Adelaide's MBBS programme has since dropped the MEQ paper from its exit examination and is evaluating in its place the Script Concordance test.