C Palermo1, E Volders1, S Gibson1, M Kennedy2, A Wray3, J Thomas3, M Hannan-Jones4, D Gallegos4, E Beck2. 1. Department of Nutrition and Dietetics, Monash University, Notting Hill, VIC, Australia. 2. School of Medicine, University of Wollongong, Wollongong, NSW, Australia. 3. Department of Nutrition & Dietetics, Flinders University, Bedford Park, SA, Australia. 4. School of Exercise and Nutrition Sciences, Queensland University of Technology, Brisbane, QLD, Australia.
Abstract
BACKGROUND: Assessment presents one of the greatest challenges to evaluating health professional trainee performance, as a result of the subjectivity of judgements and variability in assessor standards. The present study aimed to test a moderation procedure for assessment across four independent universities and explore approaches to assessment and the factors that influence assessment decisions. METHODS: Assessment tasks designed independently by each of the four universities to assess student readiness for placement were chosen for the present study. Each university provided four student performance recordings for moderation. Eight different academic assessors viewed the student performances and assessed them using the corresponding university assessment instrument. Assessment results were collated and presented back to the assessors, together with the original university assessment results. Results were discussed with assessors to explore variations. The discussion was recorded, transcribed, thematically analysed and presented back to all assessors to achieve consensus on the emerging major learnings. RESULTS: Although there were differences in absolute scores, there was consistency (12 out of 16 performances) in overall judgement decisions regarding placement readiness. Proficient communication skills were considered a key factor when determining placement readiness. The discussion revealed: (i) assessment instruments; (ii) assessor factors; and (iii) the subjectivity of judgement as the major factors influencing assessment. CONCLUSIONS: Assessment moderation is a useful method for improving the quality of assessment decisions by sharing understanding and aligning standards of performance.
BACKGROUND: Assessment presents one of the greatest challenges to evaluating health professional trainee performance, as a result of the subjectivity of judgements and variability in assessor standards. The present study aimed to test a moderation procedure for assessment across four independent universities and explore approaches to assessment and the factors that influence assessment decisions. METHODS: Assessment tasks designed independently by each of the four universities to assess student readiness for placement were chosen for the present study. Each university provided four student performance recordings for moderation. Eight different academic assessors viewed the student performances and assessed them using the corresponding university assessment instrument. Assessment results were collated and presented back to the assessors, together with the original university assessment results. Results were discussed with assessors to explore variations. The discussion was recorded, transcribed, thematically analysed and presented back to all assessors to achieve consensus on the emerging major learnings. RESULTS: Although there were differences in absolute scores, there was consistency (12 out of 16 performances) in overall judgement decisions regarding placement readiness. Proficient communication skills were considered a key factor when determining placement readiness. The discussion revealed: (i) assessment instruments; (ii) assessor factors; and (iii) the subjectivity of judgement as the major factors influencing assessment. CONCLUSIONS: Assessment moderation is a useful method for improving the quality of assessment decisions by sharing understanding and aligning standards of performance.