Damian J Castanelli1,2, Joyce M W Moonen-van Loon3, Brian Jolly4, Jennifer M Weller5,6. 1. School of Clinical Sciences at Monash Health, Monash University, Clayton, VIC, Australia. damian.castanelli@monash.edu. 2. Department of Anaesthesia and Perioperative Medicine, Monash Health, Clayton, VIC, Australia. damian.castanelli@monash.edu. 3. Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, The Netherlands. 4. School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Newcastle, NSW, Australia. 5. Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland, Auckland, New Zealand. 6. Department of Anaesthesia, Auckland City Hospital, Auckland, New Zealand.
Abstract
PURPOSE: Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions. METHODS: We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments. RESULTS: We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively. CONCLUSIONS: Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.
PURPOSE: Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions. METHODS: We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments. RESULTS: We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively. CONCLUSIONS: Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.
Authors: C P M van der Vleuten; L W T Schuwirth; E W Driessen; J Dijkstra; D Tigelaar; L K J Baartman; J van Tartwijk Journal: Med Teach Date: 2012 Impact factor: 3.650
Authors: J M Weller; B Jolly; M P Misur; A F Merry; A Jones; J G M Crossley; K Pedersen; K Smith Journal: Br J Anaesth Date: 2009-03-31 Impact factor: 9.166
Authors: J M W Moonen-van Loon; K Overeem; H H L M Donkers; C P M van der Vleuten; E W Driessen Journal: Adv Health Sci Educ Theory Pract Date: 2013-03-15 Impact factor: 3.853
Authors: M J Watson; D M Wong; R Kluger; A Chuan; M D Herrick; I Ng; D J Castanelli; L Lin; A K Lansdown; M J Barrington Journal: Anaesthesia Date: 2014-04-18 Impact factor: 6.955
Authors: Harold G J Bok; Pim W Teunissen; Robert P Favier; Nancy J Rietbroek; Lars F H Theyse; Harold Brommer; Jan C M Haarhuis; Peter van Beukelen; Cees P M van der Vleuten; Debbie A D C Jaarsma Journal: BMC Med Educ Date: 2013-09-11 Impact factor: 2.463
Authors: Emma J Stodel; Anna Wyand; Simone Crooks; Stéphane Moffett; Michelle Chiu; Christopher C C Hudson Journal: Anesthesiol Res Pract Date: 2015-12-21