OBJECTIVE: To assess the practice effects from coaching on the Undergraduate Medicine and Health Sciences Admission Test (UMAT), and the effect of both coaching and repeat testing on the Multiple Mini Interview (MMI). DESIGN, SETTING AND PARTICIPANTS: Observational study based on a self-report survey of a cohort of 287 applicants for entry in 2008 to the new School of Medicine at the University of Western Sydney. Participants were asked about whether they had attended UMAT coaching or previous medical school interviews, and about their perceptions of the relative value of UMAT coaching, attending other interviews or having a "practice run" with an MMI question. UMAT and MMI results for participants were compared with respect to earlier attempts at the test, the degree of similarity between questions from one year to the next, and prior coaching. MAIN OUTCOME MEASURES: Effect of coaching on UMAT and MMI scores; effect of repeat testing on MMI scores; candidates' perceptions of the usefulness of coaching, previous interview experience and a practice run on the MMI. RESULTS: 51.4% of interviewees had attended coaching. Coached candidates had slightly higher UMAT scores on one of three sections of the test (non-verbal reasoning), but this difference was not significant after controlling for Universities Admission Index, sex and age. Coaching was ineffective in improving MMI scores, with coached candidates actually having a significantly lower score on one of the nine interview tasks ("stations"). Candidates who repeated the MMI in 2007 (having been unsuccessful at their 2006 entry attempt) did not improve their score on stations that had new content, but showed a small increase in scores on stations that were either the same as or similar to previous stations. CONCLUSION: A substantial number of Australian medical school applicants attend coaching before undertaking entry selection tests, but our study shows that coaching does not assist and may even hinder their performance on an MMI. Nevertheless, as practice on similar MMI tasks does improve scores, tasks should be rotated each year. Further research is required on the predictive validity of the UMAT, given that coaching appeared to have a small positive effect on the non-verbal reasoning component of the test.
OBJECTIVE: To assess the practice effects from coaching on the Undergraduate Medicine and Health Sciences Admission Test (UMAT), and the effect of both coaching and repeat testing on the Multiple Mini Interview (MMI). DESIGN, SETTING AND PARTICIPANTS: Observational study based on a self-report survey of a cohort of 287 applicants for entry in 2008 to the new School of Medicine at the University of Western Sydney. Participants were asked about whether they had attended UMAT coaching or previous medical school interviews, and about their perceptions of the relative value of UMAT coaching, attending other interviews or having a "practice run" with an MMI question. UMAT and MMI results for participants were compared with respect to earlier attempts at the test, the degree of similarity between questions from one year to the next, and prior coaching. MAIN OUTCOME MEASURES: Effect of coaching on UMAT and MMI scores; effect of repeat testing on MMI scores; candidates' perceptions of the usefulness of coaching, previous interview experience and a practice run on the MMI. RESULTS: 51.4% of interviewees had attended coaching. Coached candidates had slightly higher UMAT scores on one of three sections of the test (non-verbal reasoning), but this difference was not significant after controlling for Universities Admission Index, sex and age. Coaching was ineffective in improving MMI scores, with coached candidates actually having a significantly lower score on one of the nine interview tasks ("stations"). Candidates who repeated the MMI in 2007 (having been unsuccessful at their 2006 entry attempt) did not improve their score on stations that had new content, but showed a small increase in scores on stations that were either the same as or similar to previous stations. CONCLUSION: A substantial number of Australian medical school applicants attend coaching before undertaking entry selection tests, but our study shows that coaching does not assist and may even hinder their performance on an MMI. Nevertheless, as practice on similar MMI tasks does improve scores, tasks should be rotated each year. Further research is required on the predictive validity of the UMAT, given that coaching appeared to have a small positive effect on the non-verbal reasoning component of the test.
Authors: Caroline O Laurence; Ian T Zajac; Michelle Lorimer; Deborah A Turnbull; Karen E Sumner Journal: BMC Med Educ Date: 2013-12-01 Impact factor: 2.463