J Graham1, G Hocking, E Giles. 1. Department of Anaesthesia, Sir Charles Gairdner Hospital, Perth, WA 6009, Australia. jgraham@meddent.uwa.edu.au
Abstract
BACKGROUND: Performance assessment is becoming increasingly necessary in the medical workplace. Hospitals and patients expect safety, and under-performance by a doctor can compromise standards. By describing and quantifying performance, positive behaviour can be encouraged and unsafe behaviour remedied. Anaesthesia Non-Technical Skills (ANTS) is a behavioural marker system that can be used to assess non-technical skills in the workplace. METHODS: We determined whether specialist anaesthetists could be reliably trained to use ANTS at an assessor level in an 8 h programme. Unscripted videos of routine anaesthesia were produced for training and assessment purposes. Twenty-six participants attended rater training. Exercises in behaviour observation, rater error, frame of reference and performance dimension, and the use of ANTS were conducted throughout the day. Five videos were selected for formal assessment and data collected. Intra-class correlations (ICCs) were calculated for each element. RESULTS: The accepted value of r>0.7 was not reached. ICC calculated for each element was 0.11-0.62. Comparison of participants scores with those of expert raters showed poor agreement. CONCLUSIONS: Anaesthetists could not be trained to reliably use ANTS as a summative assessment tool using our 1 day programme. There was an inadequate correlation of scores between participants and experts. Two major problems contributed to the lack of agreement. Observed behaviours were often misclassified into the incorrect element and safety beliefs varied among anaesthetists. Other reasons for the failure to achieve success and potential future direction are discussed.
BACKGROUND: Performance assessment is becoming increasingly necessary in the medical workplace. Hospitals and patients expect safety, and under-performance by a doctor can compromise standards. By describing and quantifying performance, positive behaviour can be encouraged and unsafe behaviour remedied. Anaesthesia Non-Technical Skills (ANTS) is a behavioural marker system that can be used to assess non-technical skills in the workplace. METHODS: We determined whether specialist anaesthetists could be reliably trained to use ANTS at an assessor level in an 8 h programme. Unscripted videos of routine anaesthesia were produced for training and assessment purposes. Twenty-six participants attended rater training. Exercises in behaviour observation, rater error, frame of reference and performance dimension, and the use of ANTS were conducted throughout the day. Five videos were selected for formal assessment and data collected. Intra-class correlations (ICCs) were calculated for each element. RESULTS: The accepted value of r>0.7 was not reached. ICC calculated for each element was 0.11-0.62. Comparison of participants scores with those of expert raters showed poor agreement. CONCLUSIONS: Anaesthetists could not be trained to reliably use ANTS as a summative assessment tool using our 1 day programme. There was an inadequate correlation of scores between participants and experts. Two major problems contributed to the lack of agreement. Observed behaviours were often misclassified into the incorrect element and safety beliefs varied among anaesthetists. Other reasons for the failure to achieve success and potential future direction are discussed.
Authors: Benjamin Clarke; Samantha E Smith; Emma Claire Phillips; Ailsa Hamilton; Joanne Kerins; Victoria R Tallentire Journal: BMJ Simul Technol Enhanc Learn Date: 2020-09-29
Authors: Rachel D A Havyer; Majken T Wingo; Nneka I Comfere; Darlene R Nelson; Andrew J Halvorsen; Furman S McDonald; Darcy A Reed Journal: J Gen Intern Med Date: 2013-12-11 Impact factor: 5.128
Authors: Patricia Livingston; Lauren Zolpys; Christian Mukwesi; Theogene Twagirumugabe; Sara Whynot; Anna MacLeod Journal: Pan Afr Med J Date: 2014-09-26
Authors: Laura Zwaan; Lian Tjon Soei Len; Cordula Wagner; Dick van Groeningen; Mark Kolenbrander; Ralf Krage Journal: Adv Simul (Lond) Date: 2016-06-08
Authors: Nico F Leenstra; Oliver C Jung; Fokie Cnossen; A Debbie C Jaarsma; Jaap E Tulleken Journal: Simul Healthc Date: 2021-02-01 Impact factor: 2.690