Leo Nunnink1, Carole Foot2, Bala Venkatesh3, Charlie Corke4, Manoj Saxena5, Mark Lucey2, Mark Jones3. 1. University of Queensland, Brisbane, QLD, Australia. l.nunnink@uq.edu.au. 2. University of Sydney, Sydney, NSW, Australia. 3. University of Queensland, Brisbane, QLD, Australia. 4. Geelong Hospital, Geelong, VIC, Australia. 5. University of New South Wales, Sydney, NSW, Australia.
Abstract
OBJECTIVE: To evaluate the use of high-fidelity simulation for summative high-stakes assessment of intensive care trainees, focusing on non-technical skills (NTS), testing feasibility and acceptability of simulation assessment, and the reliability of two NTS rating scales. DESIGN, SETTING AND PARTICIPANTS: Prospective observational study of senior intensive care trainees in a simulated specialist examination. METHODS: Participants undertook a simulated patient management scenario and were assessed using two rating scales: the Anaesthesia Non-technical Skills (ANTS) scale and the Ottawa Global Rating Scale (GRS). Assessors were trained, currently active, high-stakes examiners. Participants also completed a survey on simulation-based summative assessment. OUTCOME MEASURES: The inter-rater reliability of two rating scales for NTS assessment. We evaluated the feasibility of simulation-based assessment, and used survey results to assess acceptability to participants. RESULTS: Simulation assessment was feasible. Participants considered simulation-based high-stakes assessment to be acceptable and felt their scenario performance was reflective of real-world performance. Participants identified a need for debriefing following scenario-based assessment. Inter-rater reliability was fair for the ANTS and Ottawa GRS scores (intra-class correlation coefficient, 0.39 and 0.42, respectively). There was only fair agreement between raters for an NTS pass or fail (weighted kappa, 0.32) and for a technical skills pass or fail (weighted kappa, 0.36). CONCLUSIONS: Summative high-stakes assessment using a single simulated scenario was feasible and acceptable to senior intensive care trainees. The low inter-rater reliability for the ANTS and Ottawa GRS rating scales and for pass or fail discrimination may limit its incorporation into an existing examination format.
OBJECTIVE: To evaluate the use of high-fidelity simulation for summative high-stakes assessment of intensive care trainees, focusing on non-technical skills (NTS), testing feasibility and acceptability of simulation assessment, and the reliability of two NTS rating scales. DESIGN, SETTING AND PARTICIPANTS: Prospective observational study of senior intensive care trainees in a simulated specialist examination. METHODS:Participants undertook a simulated patient management scenario and were assessed using two rating scales: the Anaesthesia Non-technical Skills (ANTS) scale and the Ottawa Global Rating Scale (GRS). Assessors were trained, currently active, high-stakes examiners. Participants also completed a survey on simulation-based summative assessment. OUTCOME MEASURES: The inter-rater reliability of two rating scales for NTS assessment. We evaluated the feasibility of simulation-based assessment, and used survey results to assess acceptability to participants. RESULTS: Simulation assessment was feasible. Participants considered simulation-based high-stakes assessment to be acceptable and felt their scenario performance was reflective of real-world performance. Participants identified a need for debriefing following scenario-based assessment. Inter-rater reliability was fair for the ANTS and Ottawa GRS scores (intra-class correlation coefficient, 0.39 and 0.42, respectively). There was only fair agreement between raters for an NTS pass or fail (weighted kappa, 0.32) and for a technical skills pass or fail (weighted kappa, 0.36). CONCLUSIONS: Summative high-stakes assessment using a single simulated scenario was feasible and acceptable to senior intensive care trainees. The low inter-rater reliability for the ANTS and Ottawa GRS rating scales and for pass or fail discrimination may limit its incorporation into an existing examination format.
Authors: Benjamin Clarke; Samantha E Smith; Emma Claire Phillips; Ailsa Hamilton; Joanne Kerins; Victoria R Tallentire Journal: BMJ Simul Technol Enhanc Learn Date: 2020-09-29
Authors: Samuel Clarke; Timothy Horeczko; Matthew Carlisle; Joseph D Barton; Vivienne Ng; Sameerah Al-Somali; Aaron E Bair Journal: Med Educ Online Date: 2014-12-09