Barbara Griffin1, Piers Bayl-Smith1, Wendy Hu2. 1. Department of Psychology, Macquarie University, Sydney, New South Wales, Australia. 2. Western Sydney University School of Medicine, Sydney, New South Wales, Australia.
Abstract
CONTEXT: Evidence of predictive validity is essential for making robust selection decisions in high-stakes contexts such as medical student selection. Currently available evidence is limited to the prediction of academic performance at single points in time with little understanding of the factors that might undermine the predictive validity of tests of academic and non-academic qualities considered important for success. This study addressed these issues by predicting students' changing performance across a medical degree and assessing whether factors outside an institution's control (such as the uptake of commercial coaching) impact validity. METHODS: Three cohorts of students (n = 301) enrolled in an undergraduate medical degree from 2007-2013 were used to identify trajectories of student academic performance using growth mixture modelling. Multinomial logistic regression assessed whether past academic performance, a test of cognitive ability and a multiple mini-interview could predict a student's likely trajectory and whether this predictive validity was different for those who undertook commercial coaching compared with those who didn't. RESULTS: Among the medical students who successfully graduated (n = 268), four unique trajectories of academic performance were identified. In three trajectories, performance changed at the time when learning became more self-directed and focused on clinical specialties. Scores on all selection tests, with the exception of a test of abstract reasoning, significantly affected the odds of following a trajectory that was consistently below average. However, selection tests could not distinguish those whose performance improved across time from those whose performance declined after an average start. Commercial coaching increased the odds of being among the below-average performers, but did not alter the predictive validity of the selection tests. CONCLUSION: Identifying distinct groups of students has important implications for selection, but also for educating medical students. Commercial coaching may result in selecting students who are less suited for coping with the rigours of medical studies.
CONTEXT: Evidence of predictive validity is essential for making robust selection decisions in high-stakes contexts such as medical student selection. Currently available evidence is limited to the prediction of academic performance at single points in time with little understanding of the factors that might undermine the predictive validity of tests of academic and non-academic qualities considered important for success. This study addressed these issues by predicting students' changing performance across a medical degree and assessing whether factors outside an institution's control (such as the uptake of commercial coaching) impact validity. METHODS: Three cohorts of students (n = 301) enrolled in an undergraduate medical degree from 2007-2013 were used to identify trajectories of student academic performance using growth mixture modelling. Multinomial logistic regression assessed whether past academic performance, a test of cognitive ability and a multiple mini-interview could predict a student's likely trajectory and whether this predictive validity was different for those who undertook commercial coaching compared with those who didn't. RESULTS: Among the medical students who successfully graduated (n = 268), four unique trajectories of academic performance were identified. In three trajectories, performance changed at the time when learning became more self-directed and focused on clinical specialties. Scores on all selection tests, with the exception of a test of abstract reasoning, significantly affected the odds of following a trajectory that was consistently below average. However, selection tests could not distinguish those whose performance improved across time from those whose performance declined after an average start. Commercial coaching increased the odds of being among the below-average performers, but did not alter the predictive validity of the selection tests. CONCLUSION: Identifying distinct groups of students has important implications for selection, but also for educating medical students. Commercial coaching may result in selecting students who are less suited for coping with the rigours of medical studies.
Authors: Tal Baron; Robert I Grossman; Steven B Abramson; Martin V Pusic; Rafael Rivera; Marc M Triola; Itai Yanai Journal: PLoS One Date: 2020-01-15 Impact factor: 3.240
Authors: Giovanni Piumatti; Sissel Guttormsen; Barbara Zurbuchen; Milena Abbiati; Margaret W Gerbase; Anne Baroffio Journal: BMC Med Educ Date: 2021-07-07 Impact factor: 2.463