Aaron S Dietz1, Peter J Pronovost2, Kari N Benson1, Pedro Alejandro Mendez-Tellez3, Cynthia Dwyer4, Rhonda Wyskiel1, Michael A Rosen2. 1. The Armstrong Institute for Patient Safety and Quality, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. 2. The Armstrong Institute for Patient Safety and Quality, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA Department of Anesthesiology and Critical Care Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. 3. Department of Anesthesiology and Critical Care Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. 4. Surginal Intensive Care Unit, Johns Hopkins Hospital, Baltimore, Maryland, USA.
Abstract
OBJECTIVE: Behavioural marker systems are advocated as a method for providing accurate assessments, directing feedback and determining the impact of teamwork improvement initiatives. The present article reports on the state of quality surrounding their use in healthcare and discusses the implications of these findings for future research, development and application. In doing so, this article provides a practical resource where marker systems can be selected and evaluated based on their strengths and limitations. METHODS: Four research questions framed this review: what are the attributes of behavioural marker systems? What evidence of reliability and validity exists? What skills and expertise are required for their use? How have they been applied to investigate the relationship between teamwork and other constructs? RESULTS: Behavioural markers systems are generally designed for specific work domains or tasks. They often cover similar content with inconsistent terminology, which complicates the comparison of research findings across clinical domains. Although several approaches were used to establish the reliability and validity of marker systems, the marker system literature, as a whole, requires more robust reliability and validity evidence. The impact of rater training on rater proficiency was mixed, but evidence suggests that improvements can be made over time. CONCLUSIONS: A consensus of definitions for teamwork constructs must be reached to ensure that the meaning behind behavioural measurement is understood across disciplines, work domains and task types. Future development efforts should focus on the cost effectiveness and feasibility of measurement tools including time spent training raters. Further, standards for the testing and reporting of psychometric evidence must be established. Last, a library of tools should be generated around whether the instrument measures general or domain-specific behaviours. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
OBJECTIVE: Behavioural marker systems are advocated as a method for providing accurate assessments, directing feedback and determining the impact of teamwork improvement initiatives. The present article reports on the state of quality surrounding their use in healthcare and discusses the implications of these findings for future research, development and application. In doing so, this article provides a practical resource where marker systems can be selected and evaluated based on their strengths and limitations. METHODS: Four research questions framed this review: what are the attributes of behavioural marker systems? What evidence of reliability and validity exists? What skills and expertise are required for their use? How have they been applied to investigate the relationship between teamwork and other constructs? RESULTS: Behavioural markers systems are generally designed for specific work domains or tasks. They often cover similar content with inconsistent terminology, which complicates the comparison of research findings across clinical domains. Although several approaches were used to establish the reliability and validity of marker systems, the marker system literature, as a whole, requires more robust reliability and validity evidence. The impact of rater training on rater proficiency was mixed, but evidence suggests that improvements can be made over time. CONCLUSIONS: A consensus of definitions for teamwork constructs must be reached to ensure that the meaning behind behavioural measurement is understood across disciplines, work domains and task types. Future development efforts should focus on the cost effectiveness and feasibility of measurement tools including time spent training raters. Further, standards for the testing and reporting of psychometric evidence must be established. Last, a library of tools should be generated around whether the instrument measures general or domain-specific behaviours. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
Authors: Michael A Rosen; Aaron S Dietz; Ting Yang; Carey E Priebe; Peter J Pronovost Journal: J Am Med Inform Assoc Date: 2014-07-22 Impact factor: 4.497
Authors: Emma Claire Phillips; Samantha Eve Smith; Benjamin Clarke; Ailsa Lauren Hamilton; Joanne Kerins; Johanna Hofer; Victoria Ruth Tallentire Journal: BMJ Simul Technol Enhanc Learn Date: 2020-04-16
Authors: Majken T Wingo; Rachel D A Havyer; Nneka I Comfere; Darlene R Nelson; Darcy A Reed Journal: BMC Med Educ Date: 2015-09-14 Impact factor: 2.463