Crew resource management training in healthcare: a systematic review of intervention design, training conditions and evaluation.

OBJECTIVES: Crew resource management (CRM) training formats have become a popular method to increase patient safety by consideration of the role that human factors play in healthcare delivery. The purposes of this review were to identify what is subsumed under the label of CRM in a healthcare context and to determine how such training is delivered and evaluated.
DESIGN: Systematic review of published literature. DATA SOURCES: PubMed, PsycINFO and ERIC were searched through 8 October 2018. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Individually constructed interventions for healthcare staff that were labelled as CRM training, or described as based on CRM principles or on aviation-derived human factors training. Only studies reporting both an intervention and results were included. DATA EXTRACTION AND SYNTHESIS: The studies were examined and coded for relevant passages. Characteristics regarding intervention design, training conditions and evaluation methods were analysed and summarised both qualitatively and quantitatively.
RESULTS: Sixty-one interventions were included. 48% did not explain any keyword of their CRM intervention to a reproducible detail. Operating room teams and surgery, emergency medicine, intensive care unit staff and anaesthesiology came in contact most with a majority of the CRM interventions delivered in a 1-day or half-day format. Trainer qualification is reported seldomly. Evaluation methods and levels display strong variation.
CONCLUSIONS: Critical topics were identified for the CRM training community and include the following: the need to agree on common terms and definitions for CRM in healthcare, standards of good practice for reporting CRM interventions and their effects, as well as the need for more research to establish non-educational criteria for success in the implementation of CRM in healthcare organisations. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The systematic review is based on a comprehensive search and includes a large number of studies from a broad spectrum of healthcare settings.

A publication bias can be assumed in the crew resource management (CRM) literature because most authors are evaluating a training programme developed and delivered by themselves. However, in this review, this bias is acceptable in that effect size evaluation is not within the scope of the manuscript.

Only publications were included that reported both the intervention and related effects. This excludes theoretical considerations of CRM training as well as studies reporting effects that were not specific about the intervention or applied a standardised format that might be described elsewhere.

A meta-analysis could not be performed due to variability in reporting quality and data availability.

Introduction

Human factors have been increasingly recognised as a source of medical error since the publication of ‘To Err Is Human’.1 Today, errors are acknowledged as a serious threat to patient safety with human factors a central issue, as described in the WHO Curriculum for Patient Safety.2

Both of these seminal publications make reference to human factors as an area of expertise of both engineers and cognitive psychologists. Human factors are defined as the systemic perspective on inter-relationships of environmental, organisational and job factors, in combination with human and individual characteristics which influence behaviour at work in a way that can affect health and safety.3 The science of human factors is about improving system performance and preventing accidental harm. For healthcare, this means supporting the cognitive and physical work of healthcare professionals and promoting high-quality, safe care for patients.4

‘Crew resource management’ (CRM) trainings were adapted from aviation for healthcare teams as an instrument to address human factors. Originally, the concept of CRM emerged in 1979 based on a National Aeronautics and Space Administration workshop that traced back the root cause of air traffic accidents to failures of interpersonal communications, decision-making and leadership.5 The aviation community acknowledges that technical failures are not the main threat to complex and potentially hazardous systems. Human factors, or a combination of technical, social and human factors, are the actual threats. Error would be “an inevitable result of the natural limitations of human performance and the function of complex systems. CRM is one of an array of tools that organisations can use to manage error”.5 Salas and colleagues defined CRM training as a “a family of instructional strategies designed to improve teamwork in the cockpit by applying well-tested training tools (eg, performance measures, exercises, feedback mechanisms) and appropriate training methods (eg, simulators, lectures, videos) targeted at specific content (ie, teamwork knowledge, skills, and attitudes)”.6 The purpose of CRM in high-risk organisations can be summarised as error countermeasures with three lines of defence: (1) avoidance of error, (2) trapping incipient errors before they are committed and (3) mitigating the consequences of those errors which occur and are not mitigated.5

First efforts to transfer CRM training from aviation to healthcare teams were initiated in the 1980s as ‘anaesthesia crisis resource management’.7 In recent years, there has been a steady increase in the number of healthcare team- and CRM training publications. Figure 1 demonstrates this increase in publications as a timeline based on a PubMed literature search.

Figure 1

Proportion of crew resource management (CRM) training publications listed in PubMed. Proportion is calculated as CRM publications per million total records in PubMed for a 3-year span. See the Literature search section for full search string; graph shows all search results before screening.

Despite the rising attention for the effects and benefits of CRM training in the medical domain, there is no defined content nor a consented syllabus for CRM training. Russ et al even argue that there would be some common misconceptions and fictions that may slow or hinder the integration of human factors into healthcare, like “Human factors consist of a limited set of principles that can be learnt during brief training”.4

Concerns about a lack of professionalism in designing and delivery of CRM programmes in general have also been raised in the past.8 9 Aviation has moved forward since then by standardising CRM training elements and making CRM training mandatory for flight crew, cabin crew and technical crew—not as one-time training but integrated into relevant parts of annual recurrent training.10

Today, there are few standardised curricula that address the impact of human factors on medical teams. One example is the systematic approach provided by the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) programme.11 It is a direct outcome of the Institute of Medicine ‘To Err Is Human’ report1 and defines a set of knowledge, skills and attitudes for teams, each with a definition, behavioural examples and supporting evidence as literature citations. In contrast, many other human factor team trainings are self-constructed interventions for individual hospitals or hospital units described as ‘CRM training’. Those are the focus of this review.

Systematic reviews of general healthcare team training were recently conducted by Weaver et al 12 and by Hughes et al.13 For CRM, a meta-analysis carried out by O’Dea and colleagues quantified the effects of CRM training14 and found support for the assumption that CRM training can positively impact teamwork in healthcare. But that analysis also revealed a need for greater precision in outcome assessment, improved standardisation of methods and measures, and more robust research design. In fields other than healthcare, there is also a limited number of publications with sufficient data to perform a meta-analysis for CRM-type interventions.15

The purpose of this review is to complement those approaches by focusing on team trainings in healthcare that explicitly claim to be delivered as a CRM format, or synonymous aviation-derived human factors training. We aim to investigate the content of trainings that are individually constructed but subsumed and labelled as ‘CRM’ interventions in healthcare. Further, this review seeks to identify what is ‘inside the box’ of CRM training in a healthcare context, to understand the conditions in which the trainings are delivered and to determine how such trainings are evaluated.

Publications were reviewed that reported CRM-like interventions or trainings (both terms used synonymously here) in healthcare environments. Given the history and ongoing evolution of the concept, studies using ‘Crew’ and ‘Crisis’ Resource Management were both included. Specific attention was paid to the eight properties described in box 1.

Table 1

Studies included in this review and target groups of the trainings

Publication	Target group
Armbruster et al (2014)53	Emergency medicine
Batchelder et al (2009)28	Emergency medicine
Blum et al (2004)54	Anaesthesia
Brock et al (2013)55	Students
Catchpole et al (2010)29	OR teams
Chan et al (2016a)56	Multiple
Chan et al (2016b)57	Multiple
Clay-Williams et al (2013)58	Multiple
Clay-Williams et al (2014)59	Multiple
Clay-Williams and Braithwaite (2015)60	Not specified
Coppens et al (2017)61	Nursing students
Duclos et al (2016)35	OR team
Fransen et al (2017)34	Obstetrics team
Grogan et al (2004)62	Multiple
Guerlain et al (2008)20	Surgery
Haerkens et al (2015)39	ICU team
Haerkens et al (2018)40	Emergency medicine
Haffner et al (2017)63	Medical students
Haller et al (2008a, 2008b)30 64	Obstetrics team
Hänsel et al (2012)65	Medical students
Hansen et al (2008)33	OR team
Hefner et al (2017)25	Multiple
Hicks et al (2012)66	Emergency medicine
Holzmann et al (1995)22	Anaesthesia
Hughes et al (2014)67	Emergency medicine
Jankouskas et al (2007)68	Paediatrics
Jones et al (2014)69	Surgery
Kemper et al (2014, 2016)41 48	ICU team
Kuy and Romero et al (2017)70	Surgery
Lehner et al (2017)71	Paediatrics
Marshall and Manus (2007)21	Surgery
Mason et al (2009)72	Surgery
Mayo et al (2011)73	ICU team
McCulloch et al (2009)74	OR team
Mitchell and Dale (2015)75	OR team
Moffatt-Bruce et al (2017)26	Multiple
Morey et al (2002)36	Emergency medicine
Morgan et al (2011)76	Anaesthesia
Morgan et al (2015)43	OR team
Müller et al (2007)77	Emergency medicine
Müller et al (2009)44	ICU team
Nielsen et al (2007)27	Obstetrics team
O’Connor et al (2013)78	Trainees
Pettker et al (2009, 2011)46 79	Obstetrics team
Reznek et al (2003)80	Emergency medicine
Ricci and Brumsted (2012)32	OR team
Robertson et al (2010)31	Students
Savage et al (2017)81	Paediatrics
Schmidt et al (2010)82	OR team
Shah et al (2013)83	Trainees
Shapiro et al (2004)24	Emergency medicine
Shea-Lewis (2009)47	Obstetrics team
Siems et al (2017)84	Critical care
St. Pierre et al (2004)85	Not specified
Sundararaman et al (2014)86	Radiation-oncology
Sweeney et al (2014)87	Emergency medicine
Truijens et al (2015)45	Obstetrics team
Tschannen et al (2018)88	Trainees
Verbeek-van Noord et al (2015)89	Emergency medicine
Westfelt et al (2013)90	Internal medicine
Zech et al (2017)49	Obstetrics team

ICU, intensive care unit; OR, operating room.

Description of intervention: What was the setting and/or format; which CRM content or theories were taught?

Duration of CRM training: How many contact-hours did the training intervention include?

Target group for training: What kind of attendees did the training address?

Number of participants: How many individuals attended each course?

Trainer qualifications: Which qualifications were reported for the CRM trainers?

Level of evaluation: At what level, according to Kirkpatrick,17 can the reported effect be categorised?

Method of evaluation: By what design and means was the evaluation carried out?

Reported outcome: What is the summary of the study’s findings?

A systematic review was conducted to analyse the content and practice of CRM training within healthcare settings. The results are reported in accordance with the PRISMA statement for reporting of systematic reviews and meta-analyses of studies that evaluate healthcare interventions.16

Eligibility criteria

Studies that reported individually constructed ‘CRM training’ or trainings based on CRM principles were evaluated within a healthcare context and included hospitals, their departments, emergency medical services or medical education. Only studies that both described an intervention and also reported its effects were considered.

Types of participants: healthcare staff.

Types of intervention: individually constructed training formats addressing CRM principles or aviation-derived human factors.

Types of outcome measures: studies reporting both the intervention and its effect.

Report characteristics: published in an academic journal, either in English or German.

Literature search

Studies were identified by searching electronic databases as well as an additional snowball search, for example, scanning reference lists of relevant articles.

Since CRM is a topic that impinges on several disciplines, the search strategy covered databases from different fields. PubMed was used for healthcare publications. PsycINFO was used for psychological publications. ERIC was searched for educational publications. General search terms were ‘crew resource’ or synonymously ‘crisis resource’ and ‘training’. The last search was completed on 8 October 2018.

For PubMed, the search was amended with ‘human factors’ to widen the filter for healthcare research that might be relevant but did not include CRM as a specific term in the title or abstract:

PubMed: ((((crew resource[Title/Abstract]) OR crisis resource[Title/Abstract]) OR human factor[Title/Abstract]) OR human factors[Title/Abstract]) AND training[Title/Abstract].

PsycINFO and ERIC: AB ("crew resource" OR "crisis resource") OR TI ("crew resource" OR "crisis resource") AND (AB training OR TI training).

Data collection process

All articles identified for this review were entered into MAXQDA, a software tool for qualitative and mixed-methods data analysis (MAXQDA, V.12; VERBI Software, Berlin, Germany).

The analysis was carried out in three phases with each article under review being read several times:

Semistructured explorative coding and coding framework development: At initial analysis, publications were coded for the following topics: (a) descriptive data about the intervention, (b) design of the intervention, (c) training conditions and (d) evaluation. This approach provided an understanding of the aspects of CRM trainings found within the literature and also provided a framework for more detailed coding (eg, the initial expectation was to find didactical concepts, educational objectives or specific reasons for the implementation of CRM trainings, but instead we found that few articles mentioned these details).

Focused coding: In the next stage, all articles were read again with the detailed coding structure derived from stage 1 wherein words, sentences or whole paragraphs related to a topic were coded in detail.

Compilation of results: Finally, a data table was assembled from the coded data, summarising each article in the review, with key data included according to the eight general properties of interest to this review.

The articles were read and co-coded by two reviewers (BG, LR). Disagreements were resolved immediately; when necessary, a third researcher (AZ) was consulted.

Method of analysis

Data items were coded and assembled in quantitative dimensions. Some data items required further processing. Keywords used in the intervention description and their respective level of detail were categorised as well as the description of trainer qualifications and the level of evaluation.

CRM content and theories

The keywords that related to training content (eg, ‘communication’, ‘situational awareness’) were assessed across all included publications. Each publication was then categorised for keywords and whether an explanation or definition was provided, which would allow for replication of the procedure.

Level of evaluation

Kirkpatrick’s four-level concept was adopted for this review in order to categorise the different evaluative approaches applied in the publications.17 The general validity of this sequential model for evaluation of training effects was recently supported by a meta-analysis.13 According to the Kirkpatrick model, the stages of evaluation were defined as:

Level 1 ‘reactions’: for example, questionnaires administered to participants directly after the training or in close temporal proximity, inquiring after their happiness, satisfaction, knowledge or perceived own performance. Such reaction data were gathered as direct feedback from the participants.

Level 2 ‘learning’: neutral proof that participants learnt something during the intervention. This could be observed and documented by facilitators or through an examination that participants completed after the training.

Level 3 ‘transfer’: changes in behaviour that occurred after approximately 3 months, in order to assess impact on daily routine. This could be measured by attitude questionnaires or observation. Kirkpatrick’s concept of this level suggests leaving a mid-term delay of 2 to 3 months before administering an attitudes questionnaire to provide adequate time for attitudinal changes to occur and to “give trainees time to get back to the job, consider the newly suggested behaviour, and try it out”.17

Level 4 ‘results’: effects of the intervention on visible outcomes within the whole organisation, which were supported by data. This was assumed to be the highest level of evaluation in that such data would relate invested effort in training to impact on the organisation, for example, a reduced rate of mistakes or adverse outcomes.

Patient and public involvement

Patients and/or public were not involved during any stage of this study.

Results

Study selection

The search of PubMed, PsycINFO and ERIC databases provided a result of 1037 publications. Through an additional manual search, 13 further publications could be identified that were included in the reviewing process. Each article’s eligibility was assessed by two reviewers (BG, LR) in an unblinded standardised manner.

First, the list of search results was screened for duplicates and obvious inclusion or exclusion criteria. Studies were excluded that described an intervention without report of evaluation data, reported evaluation data but without description of the intervention, did not describe a CRM-style intervention or did not take place in a healthcare context. Also, studies were excluded when reporting the application of a standardised training, like TeamSTEPPS, if not applied as an individualised version. A total of 877 publications were excluded for these reasons. Disagreements between the reviewers were resolved by discussion.

Nineteen publications were discarded because the full text of the study was not available. This resulted in 154 publications that contained full text. After further detailed examination, another 90 publications did not meet the inclusion criteria, for example, training that focused on a single communication aspect and did not encompass a more comprehensive teamwork approach for several aspects of human factors management. The process of study selection is depicted in figure 2.

Figure 2

Flow diagram of study selection.

For the 154 studies that entered the in-depth reading stage, 3265 passages (eg, training description) or data items (eg, number of participants) were coded with 91 different codes. Six hundred and seventy-three memos or summary texts were inserted by the reviewers.

Three pairs of publications were based on the same data (reporting different aspects of that data) and were treated as a single publication. In consequence, 64 publications were included in this review, but technically, 61 interventions were assessed. See table 1 for a full list of publications included in this review. See online supplemental file Digital Content 1 (SDC 1) for a comprehensive compilation of study characteristics.

CRM content and theories

Table 2 and figure 3 indicate the keyword count mentioned in the publications and whether a description of the CRM intervention content or foundational theory was provided. Approximately 23% of the mentioned keywords were explained with sufficient detail for reproducibility while 48% of the publications did not explain any keyword with sufficient detail for reproducibility. The criterion employed was whether an expert proficient in CRM research and practice could comprehend and reproduce the intervention based on the information given and the references provided.

Table 2

Topics used to describe the crew resource management (CRM) intervention

Topic	Keyword mentioned	Explanation sufficient for replication
Communication (general)	48	13
Situational awareness	37	9
Leadership	31	8
Teamwork	25	3
Decision-making	24	3
Briefing	23	15
Error management	23	6
Workload management	20	4
Closed loop communication	16	2
Acronyms (eg, SBAR)	14	7
Stress management	11	6
Re-evaluation	11	4
Speaking up	10	7
Red flags	8	2

SBAR, situation, background, assessment, recommendation.

Figure 3

Crew resource management (CRM) content in training descriptions: keywords versus provided reproducible explanation. SBAR, situation, background, assessment, recommendation.

Duration of CRM training

The majority (62%) of CRM interventions were delivered in a 1-day format with a duration of either a full day (6–10 hours: 38%) or a half day (<6 hours: 25%). Twenty-five per cent were multiple day interventions (>10 hours). Eight (13%) did not report the duration of training. As comparison, for civil aviation, the duration of an introductory CRM course is 2 or 3 days.10 15

Target group for training

The professional groups with the most CRM intervention training were operating room or surgery (21%), emergency medicine (20%), obstetrics and paediatrics (16%), intensive care units (ICUs) and anaesthesiology (13%). Students and learners were the target group for another 13% of the trainings.

Of the 61 studies included in this review, the majority (39%) originated from the USA based on the first author’s affiliation. Altogether, 62% of the studies took place in anglophone societies (USA, UK, Australia, Canada or Ireland). Overall, the publications in this review originated from 13 different countries—most of which are high ranking on the United Nations Human Development Index.18 The interconnection between culture and CRM programmes has previously been described at the national, organisational and professional levels.5 19

Number of participants per course

CRM training group size ranged mostly from 5 to 15 participants (see figure 4A). The smallest group size was composed of seven faculty surgeons who were individually trained via a 1-hour lecture.20 The largest reported group size was classroom-based CRM workshops at several sites with a maximum of 35 participants.21 Forty-four per cent of the publications did not report the number of participants per course.

Figure 4

(A) Participant group size for crew resource management interventions (n=34). (B) Boxplot of intervention programme sizes (n=53); three group sizes with relatively high numbers are not included in the boxplot calculation but denoted with an asterisk. The solid line indicates the median. Whiskers indicate the maximum or minimum scores that are not outliers.

Setting

Approximately 43% of the interventions were classroom only or did not report the setting (7%), with the remainder using a simulation environment for CRM training (50%). For example, Holzman and colleagues22 used a simulator mannequin as a patient and had professional nurses and surgeons as role players following a scripted scenario. The simulations took place either in a regular hospital ward23 or in an environment that completely simulated patient care situations.24

Intervention programme sizes

Of all studies, 53 (87%) reported the total number of CRM intervention participants. Together, these studies represent CRM programmes with about 19 500 participants. The large number of participants originates from only two studies accounting for 53% of the sum of training participants.25 26 Both publications refer to different aspects and timeframes of the same large-scale CRM implementation so that an overlap in participants must be assumed as bias in this calculation. Another outstandingly large intervention with 1300 trained participants stems from a multiple-hospital intervention.27 Figure 4B displays intervention programme sizes as boxplot.

Trainer qualifications

The qualifications of the trainers were coded in four categories based on educational background:

Not available (n/a) or not specified (n=13), for example, ‘trained facilitator’ or ‘a medical education expert’ (n=8).

Description of a (medical) professional or someone with technical qualifications, for example, ‘four instructors (two emergency physicians and two anaesthesiologists)’ (n=11).

Non-medical qualifications that indicate didactic or psychological competence or explicitly state external consultation with professionals, for example, psychologists, aviation trainers, consulting company (n=19).

Specific education or preparation of trainers is explained (at least in duration and content) (n=10).

Overall, 34% (n=21) of the studies did not report or specify staff qualifications for the training intervention. Eighteen per cent reported qualifications that were medical or technical but did not indicate expertise in CRM theory or practice or in any didactic competence. Thirty-one per cent (n=19) mentioned psychological or didactical qualifications of the trainer team or stated that external expertise assisted with the training. Only 16% (n=10) of the studies specifically explained how staff were prepared and trained as CRM trainers.

Level of evaluation

The majority of studies, 34 of 61 (56%), reported data for Kirkpatrick level 1, with 13 publications (21%) built solely on level 1 data. Fifteen studies (25%) contained level 2 data, 31 (51%) level 3 data, and 20 (33%) reported effects for level 4.

Figure 5 summarises the evaluation levels for these studies. The vertical axis distinguishes between evaluation levels with the studies indicated on the horizontal axis. A light grey marker indicates deviations from Kirkpatrick’s concept, for example, when safety attitudes questionnaires were administered immediately following a training intervention instead of allowing participants a delay of several weeks for attitudinal changes in daily practice. The grey marker also indicates reports of qualitative but no quantitative data.

Figure 5

Studies and their level of evaluation (1–4). The figure demonstrates the combination of levels chosen for evaluation of the crew resource management intervention effect, starting with only level 1 on the right and progressing to multiple levels, including level 4, on the left. Studies marked with a light grey colour reported qualitative data for a level or differed methodologically to the Kirkpatrick concept.

Methods of evaluation

Evaluations for level 1 were elaborate and carried out by self-constructed questionnaires designed to measure a participant’s happiness or satisfaction with the training format and content. No patterns were discerned at this level during this review. For level 2, the learning effect was mostly evaluated using behavioural criteria during simulations, which were based on videotape simulations28 or by direct observation during a simulation.29 Few studies evaluated the learning effect with questionnaires post course.30 31 For the 31 studies reporting data for level 3, 55% (n=17) applied standardised and validated versions of questionnaires assessing safety and teamwork related attitudes like the Safety Attitudes Questionnaire (n=7), Hospital Survey on Patient Safety Culture (n=5), Human Factors Attitude Survey (n=4) or Teamwork Attitudes Questionnaire (n=1). The 20 evaluations for level 4 were mainly based on error rates or adverse outcomes, or reduced malpractice expenses as defined by pay-outs and legal fees after the intervention.32 Other authors reported qualitative data supporting the positive effect of CRM training interventions. Hansen and colleagues performed a phone survey 6 months after the training and identified 12 lifesaving cases that were attributed to the effect of the training.33

Reported outcome

Nearly all publications reported a positive impact of the CRM intervention for one or more levels of their evaluation. However, some of the large multicentre studies that investigated effects specifically for outcome parameters like adverse events or complication rates did not find significant effects. Nielsen et al compared adverse outcomes before and after training for delivery room personnel, but no statistically significant differences were observed between intervention and control groups.27 Similar results are reported by Fransen et al for obstetric complications.34 Duclos et al observed a one-third reduction in major adverse events after team training implementation, but with a similar reduction in the control arm of the study, so the effect cannot be attributed exclusively to the intervention.35

Discussion

The purpose of this systematic review was to analyse the published content and practice of CRM interventions within a healthcare context. We wanted to know what composition of theories and skills were considered under this training type and how their impact was evaluated. A set of 61 studies (64 publications) met inclusion criteria and were analysed and summarised both qualitatively and quantitatively.

There is no easy answer to the question ‘what is CRM in healthcare’? Diverse concepts and methods of communication and collaboration were found from which CRM trainings were composed. Training formats and evaluation approaches varied such that CRM appears currently to be less a tangible training intervention but rather an umbrella under which healthcare organisations can manage human factors and safety training. An unexpectedly large number of publications in this review did not provide a description of the intervention, which would have permitted future study replication.

The increasing number of publications indicates a rising interest in CRM training interventions by many practitioners and scientists within medical contexts. CRM training for aviation evolved over several conceptual generations.5 Presuming a similar process for the healthcare domain, this investigation provides a snapshot of the state of CRM training in healthcare. It also provides transparency and inspiration for those who want to design and implement future CRM formats. Also, it may be useful to fellow researchers who intend to evaluate an intervention’s effects.

Intervention design

A particular focus of this review was to examine what healthcare CRM training consists of and to identify the conceptual core from which such training was derived. The results provide only a fuzzy image of how CRM is practised in healthcare contexts.

Figure 6 identifies the most frequently used keywords in CRM descriptions. It is provided as a Word Cloud where font size is determined by the occurrence of a term (compiled from table 2). The pattern seems representative for the scope of CRM trainings inherent in the literature: CRM was found to be attributed to communication in general as well as to specific communication techniques (such as briefings or closed loop communications) in particular; it also draws from psychological concepts such as situational awareness or decision-making, and it relates to management aspects like leadership, teamwork and workload management. However, the reviewed studies lacked a shared definition of CRM—like a framework for all those terms. They also indicate the need for a structural model that distinguishes between theoretical, conceptual and technical levels of CRM.

Figure 6

Word Cloud for topics considered by crew resource management (CRM) interventions.

Forty-eight per cent of the studies did not provide adequate detail or keywords that would allow the CRM intervention to be replicated. It was common for a term or concept to be used as part of a CRM intervention but without a definition of the term or concept. In 48 articles, ‘communication’ was a content keyword, but only 13 further specified this topic (eg, a description of what it exactly means to train participants in communication). Similarly, 24 articles identified ‘decision-making’, but only three delivered a reproducible explanation of how this was implemented in the training. Few articles included an extensive description of their curriculum. In contrast, Morey et al 36 provided a good practice example for a structured description of course components while others referred to a Wiki-website as reference for CRM. Although rationale may exist in specific cases, inaccuracy has the potential to weaken the validity of CRM research and limits possible insight to be derived from future meta-analysis of this field.

An important step for healthcare CRM application and research is an established definition for CRM in healthcare with a glossary of concepts, terms and techniques. This would provide opportunities for standardisation, comparisons among approaches and consistency in the evaluation of results. The TeamSTEPPS programme11 could be an example of how to generate a framework of definitions for CRM training that results in a set of intervention building blocks, with each block defined, repeatable and embedded in a larger context. Also, the Non-Technical Skills for Surgeons (NOTSS) educational system for assessment and training of non-technical skills in surgery could be a valuable starting point for further developments.37 38

Impact of change management programmes

A number of the studies in this review mentioned additional non-educational measures for implementing CRM that went beyond the scope of training formats. For instance, Moffatt-Bruce et al involved executive and senior managers during an early stage of the intervention to make those stakeholders part of a culture transformation.26 Morey and colleagues36 described a 6-month phase of coaching and mentoring after CRM training for all staff during normal shifts. Haerkens and colleagues39 40 had training groups create a shortlist of practical ‘action points’ to be used during an implementation year. Marshall and Manus described an implementation phase for their CRM programme as well as post-training sustainability strategies and techniques for different hospitals.21 Kemper et al specifically investigated barriers and enablers for action after CRM training.41

Those studies indicated that the CRM intervention was not only delivered as training or coaching but was also embedded in an organisational change process or was at least accompanied by organisational change-management measures. Unfortunately, of the publications identified in this subgroup, only few reported means, SD and sample size, which are data necessary for effect size calculation. Hence, we were unable to perform a meta-analysis.

Our observations can only provide a direction for future research. The additional non-educational factors beneficial to CRM training are largely unknown. However, the essential meaning of organisational support for the success of CRM and positive sustained effects of CRM trainings has been emphasised.8 If training alone will not provide a noticeable change in behaviour, more research is needed to understand the prerequisites and necessary measures at the organisational level for successful implementation of CRM in hospitals.

Training conditions

Duration

CRM training interventions are constrained by time in healthcare settings. The majority were delivered in a day or less, nine interventions took 12 to 16 hours, and only six of the programmes exceeded 16 hours. Such short training periods can hardly provide adequate time to cover the many topics encompassed by CRM keywords found in the studies. It is more realistic for CRM-type interventions to focus on particular themes of CRM and place emphasis on a subset of topics during the training intervention. However, this limits the comparison of CRM interventions as well as the analysis of their effectiveness.

Target group

In this review, more than 80% of CRM training was delivered to professionals who work as teams in operating rooms, emergency medicine, ICUs, anaesthesia or obstetrics. These teams will often fulfil characteristics outlined by Hughes et al.: low temporal stability, short team life span, functional role structure, high skill differentiation, rotating leadership structure, high authority differentiation and high interdependence.13 These healthcare teams are confronted with critical situations on a regular basis and therefore likely acknowledge the benefits of CRM. This may explain the dominant representation of these kinds of teams among the study populations. However, it is reasonable to assume that CRM would benefit other teams involved in patient care as well.

Number of participants and use of simulators

The number of training participants per group typically ranged from 5 to 15, indicating a high degree of dense and interactive training. Further, there was no reason to assume that CRM training necessarily involved high-fidelity simulators or that CRM and simulation training were inseparably related. In fact, 43% of the training outlined in the assessed studies was classroom-based. This proportion may be lower due to the varying availability of comprehensive curriculum descriptions. It still seems plausible as Cook et al compared the effectiveness of technology-enhanced simulations versus other instructional methods.42 They found only small to moderate positive effects, but concluded that the merits of simulation likely vary for different educational objectives. CRM training seems to reflect this variability in that the use of simulators is the choice of the educator—but not a necessity.

Trainer qualification

Since the contents and theories contributing to the field of human factors are derived from various scientific areas, it would be expected that the selection of trainers would reflect various professional backgrounds. It would also be reasonable to assume that trainer preparation prior to the CRM intervention would be a key consideration.

However, only a small number of articles explained trainer education. Approximately one-third of the articles mentioned trainer backgrounds in psychology, educational science, or that external expertise was employed to provide the intervention. Regarding the latter, reporting should improve: being employed from an external supplier does not necessarily qualify a trainer nor does it indicate that such a trainer is a human factors professional.

In contrast, many articles were very precise in describing the technical setting of the intervention, including the manufacturer and type of simulation equipment used. However, more than half of the studies did not report the qualifications of the faculty nor their professional background or preparation for the challenges of interactive training formats with a small number of participants.

Evaluation

Levels and methods

Approaches and measures to evaluate CRM interventions are multiple and some display a high degree of creativity. Morgan et al 43 observed a ‘glitch rate’ per hour as the number of deviations from established surgical procedures. Müller et al analysed the salivary cortisol levels of participants.44 Truijens et al interviewed the patients of CRM participants and assessed the patient’s perceived quality of care, using a validated psychometric questionnaire.45

Standardised tools in the form of attitude questionnaires were used to assess level 3 outcomes. Some studies calculated adverse outcome indices to describe level 4 effects.27 34 35 39 46 47 For future research, these indices will allow for comparison of CRM effects and organisational outcomes. However, the challenge will be to separate the impact of the CRM intervention from other factors influenced by the organisation.

Further, few studies addressed Kirkpatrick’s scheme for levels 2 and 4: the impact on personal learning and organisational outcomes. Only 15 studies evaluated level 2 ‘did participants learn something?’ and another 20 evaluated level 4 ‘is there an effect for the organisation?’. It is worth noting that only one of the studies evaluated all levels simultaneously.41 48

Sample group size

Although some studies evaluated large-scale CRM programmes involving different hospitals,21 25 27 34 35 48–50 most refer to a very specific training format at a single site. Few studies reported the number of participants of their intervention in comparison with the number of staff employed in a unit or hospital. The investigation of the ‘critical mass’ needed to be trained for an organisation-wide impact on safety culture might be worth further research.

Most reported intervention programme sizes ranged from some dozens to a few hundred participants. This indicates that data stems from evaluations of new initiatives rather than long-running and broadly implemented programmes. In the future, it will be important to investigate the long-term effects of CRM programmes at the organisational level as well as reporting long-term outcomes and adjustments due to such programmes.

Efficiency of training concepts and didactics

Training formats and related didactic concepts were investigated in only a few studies. Most studies asked ‘Does what we do work at all?’ but not ‘Are we doing it in the best possible way?’. Both the design and application of CRM have been previously critiqued.51 Fifteen years ago, Salas et al 8 raised concerns about trainer qualifications and the role of subject matter experts in training design and in didactic concepts. Even if those remarks referred to CRM in aviation, this review supports those concerns for healthcare as well and recommends more attention to trainer selection and qualifications as well as didactic concepts. Stronger consideration of such factors could narrow down variance in training efficiency and, as Duclos et al put it, the question of “whether the intervention was inherently ineffective, inadequately applied or applied in an inappropriate context”.35

Studies: completeness, reproducibility and comparability

Although the original plan, this systematic review was not completed as a comprehensive meta-analysis because the quality of the data and the nature of the CRM reporting practices did not permit such an analysis. Therefore, we strongly recommend the development of evaluation and reporting standards in order to foster completeness, reproducibility and comparability of CRM intervention studies. A set of facts and figures that were searched for during this review—and often missing—is compiled in table 3. It suggests basic reporting requirements for future CRM training evaluations in addition to the reporting guidelines for healthcare simulation research as recommended by Cheng et al.52

Table 3

Suggested minimum requirements for future crew resource management (CRM) evaluations

Topic	Explanation
Intervention design Aims Methods Contents	Aims, conceptual and theoretical foundation of the intervention; Methods applied (eg, training, workshops, organisational change management); Reproducible description of intervention contents.
Training conditions Duration of training Target group No of participants Trainer qualification	Duration in hours and days; Target group of participants; No of participants per group; General qualification of training faculty (eg, physician, psychologist) as well as any special qualification to deliver CRM training.
Evaluation Method of evaluation Sample group size Statistical data Outcomes	Method and levels of evaluation (eg, questionnaires, data sources); Sample group size and sufficient statistical data (eg, means and effect sizes); Outcomes observed, preferable including the organisational level.

CRM is often advocated as essential to patient safety. Hence, investigations need to be more precise in describing CRM intervention ingredients, who should receive it, who actually receives it, in what dosage, at what time, in combination with what and to what effect. To reach an evidence-based level for CRM interventions in medicine, it is of utmost importance to develop common concepts, wording and publication standards.

The word limit of some academic journals will not allow a comprehensive and reproducible description of a CRM intervention to be published together with a sound evaluation. Common standards and more extensive use of study protocols or digital supplements may be an option in such cases.

Limitations

Based on the data herein, CRM is an important tool that improves teamwork and patient safety. Nearly all studies in this review reported positive effects. However, there is reason to assume a significant publication bias for authors evaluating a training programme mostly developed and delivered by themselves. This bias is acceptable in that the focus of this review was not to evaluate outcomes but rather to analyse the concepts of CRM training in the broadest sense.

Only studies were included that reported both the intervention and related effects. This excludes theoretical considerations of CRM training as well as studies reporting effects that are not specific about the intervention. A formal quality assessment of the studies that met the inclusion criteria was not applied despite the requirement to be published in an academic journal. The quality of the articles in this review varies and is a potential source of bias. This is mitigated by the focus of this review on compiling an overview of approaches for design, delivery and evaluation of CRM interventions. The efficiency of those CRM trainings was not assessed. However, from the set of studies included in this review, only a portion would qualify for a meta-analysis.

Conclusion

This review provides an overview of topics covered in CRM interventions, the design and duration of training, and the evaluation methods. It also demonstrates that CRM training is much more than merely simulation training. Actually, the term ‘CRM’ appears to be loaded with such a plurality of skill areas, topics, multiple settings for training, and evaluation procedures, that a process by which to define common core values and standards is necessary.

After this systematic review of healthcare CRM publications, the critical needs identified by O’Connor and colleagues15 are confirmed: We hope that this article can again “serve to raise awareness (…) of the importance of relying on the science of training to design, deliver, and evaluate it”.8 Specifically, we see three urgent appeals for the future development of CRM in healthcare:

Practitioners and researchers need to agree on common terms and definitions regarding the meaning of healthcare CRM.

Researchers should consider good practice for reporting intervention design and data evaluation.

More research is needed to establish criteria for success in implementing CRM in healthcare organisations. Attention should be paid to both the intervention itself as well as the conditions of the surrounding organisational structure.