OBJECTIVES: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools. DESIGN, SETTING AND PARTICIPANTS: Data from four consecutive cohorts of students (3378 students, enrolled 2007-2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt. MAIN OUTCOME MEASURES: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes. RESULTS: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44-1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00-0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others. CONCLUSIONS: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A "sufficient evidence" selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.
OBJECTIVES: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools. DESIGN, SETTING AND PARTICIPANTS: Data from four consecutive cohorts of students (3378 students, enrolled 2007-2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt. MAIN OUTCOME MEASURES: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes. RESULTS: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44-1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00-0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others. CONCLUSIONS: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A "sufficient evidence" selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.