J M Weller1, M Misur2, S Nicolson2, J Morris3, S Ure4, J Crossley5, B Jolly6. 1. Centre for Medical and Health Sciences Education, Faculty of Medical and Health Sciences, University of Auckland, 2 Park Rd, Grafton, Auckland 1010, New Zealand Department of Anaesthesia, Auckland City Hospital, 2 Park Rd, Grafton, Auckland 1010, New Zealand j.weller@auckland.ac.nz. 2. Department of Anaesthesia, Auckland City Hospital, 2 Park Rd, Grafton, Auckland 1010, New Zealand. 3. Department of Anaesthesia, Royal Melbourne Hospital, Grattan Street, Parkville, VIC 3052, Australia. 4. Department of Anaesthesia, Wellington Hospital, Riddiford Street, Newtown, Wellington 6021, New Zealand. 5. Academic Unit of Medical Education, University of Sheffield, 85 Wilkinson Street, Sheffield S10 2GJ, UK. 6. University of Newcastle, University Drive, Callaghan, Newcastle, NSW 2308, Australia.
Abstract
BACKGROUND: The value of workplace-based assessments such as the mini-clinical evaluation exercise (mini-CEX), and clinicians' confidence and engagement in the process, has been constrained by low reliability and limited capacity to identify underperforming trainees. We proposed that changing the way supervisors make judgements about trainees would improve score reliability and identification of underperformers. Anaesthetists regularly make decisions about the level of trainee independence with a case, based on how closely they need to supervise them. We therefore used this as the basis for a new scoring system. METHODS: We analysed 338 mini-CEXs where supervisors scored trainees using the conventional system, and also scored trainee independence, based on the need for direct, or more distant, supervision. As supervisory requirements depend on case difficulty, we then compared the actual trainee independence score and the expected trainee independence score obtained externally. RESULTS: Compared with the conventional scoring system used in previous studies, reliability was very substantially improved using a system based on a trainee's level of independence with a case. Reliability improved further when this score was corrected for case difficulty. Furthermore, the new scoring system overcame the previously identified problem of assessor leniency and identified a number of trainees performing below expectations. CONCLUSIONS: Supervisors' judgements on trainee independence with a case, based on the need for direct or more distant supervision, can generate reliable scores of trainee ability without the need for an onerous number of assessments, identify trainees performing below expectations, and track trainee progress towards independent specialist practice.
BACKGROUND: The value of workplace-based assessments such as the mini-clinical evaluation exercise (mini-CEX), and clinicians' confidence and engagement in the process, has been constrained by low reliability and limited capacity to identify underperforming trainees. We proposed that changing the way supervisors make judgements about trainees would improve score reliability and identification of underperformers. Anaesthetists regularly make decisions about the level of trainee independence with a case, based on how closely they need to supervise them. We therefore used this as the basis for a new scoring system. METHODS: We analysed 338 mini-CEXs where supervisors scored trainees using the conventional system, and also scored trainee independence, based on the need for direct, or more distant, supervision. As supervisory requirements depend on case difficulty, we then compared the actual trainee independence score and the expected trainee independence score obtained externally. RESULTS: Compared with the conventional scoring system used in previous studies, reliability was very substantially improved using a system based on a trainee's level of independence with a case. Reliability improved further when this score was corrected for case difficulty. Furthermore, the new scoring system overcame the previously identified problem of assessor leniency and identified a number of trainees performing below expectations. CONCLUSIONS: Supervisors' judgements on trainee independence with a case, based on the need for direct or more distant supervision, can generate reliable scores of trainee ability without the need for an onerous number of assessments, identify trainees performing below expectations, and track trainee progress towards independent specialist practice.