Generic analysis method to learn from serious adverse events in Dutch hospitals: a human factors perspective.

BACKGROUND: Hospitals in various countries such as the Netherlands investigate and analyse serious adverse events (SAEs) to learn from previous events and attempt to prevent recurrence. However, current methods for SAE analysis do not address the complexity of healthcare and investigations typically focus on single events on the hospital level. This hampers hospitals in their ambition to learn from SAEs. Integrating human factors thinking and using a holistic and more consistent method could improve learning from SAEs. AIM: This study aims to develop a novel generic analysis method (GAM) to: (1) facilitate a holistic event analysis using a human factors perspective and (2) ease aggregate analysis of events across hospitals.
METHODS: Multiple steps of carefully evaluating, testing and continuously refining prototypes of the method were performed. Various Dutch stakeholders in the field of patient safety were involved in each step. Theoretical experts were consulted, and the prototype was pretested using information-rich SAE reports from Dutch hospitals. Expert panels, engaging quality and safety experts and medical specialists from various hospitals were consulted for face and content validity evaluation. User test sessions concluded the development of the method.
RESULTS: The final version of the GAM consists of a framework and affiliated questionnaire. GAM combines elements of three methods for SAE analysis currently practised by Dutch hospitals. It is structured according to the Systems Engineering Initiative for Patient Safety model, which incorporates a human factors perspective into the analysis. These eases aggregated analysis of SAEs across hospitals and helps to consider the complexity of healthcare work systems.
CONCLUSION: The GAM is a valuable new tool for hospitals to learn from SAEs. The method can facilitate a holistic aggregate analysis of SAEs across hospitals using a human factors perspective, and is now ready for further extensive testing. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Adverse events in healthcare represent a major source of morbidity and mortality and result in substantial societal costs.1–4 Hospitals in numerous countries, such as the Netherlands, the USA, Canada, Australia and the UK, aim to learn from previous events by thoroughly investigating these. In particular, hospitals invest in studying the events that have caused temporary or permanent disability, death or prolonged hospital stay, also known as sentinel events or serious adverse events (SAEs).5–10 After hospitals investigate a SAE, they write a detailed report. This report starts with a comprehensive reconstruction of the event and concludes with a root cause analysis (RCA), in which hospitals search for root causes and formulate recommendations to prevent recurrence.11 SAE investigations and the associated reports offer great potential for hospitals to learn from each other and improve patient safety. However, SAE investigations and analyses are being criticised for oversimplification,12 resulting in weak solutions,10 11 13 and overall ineffectiveness.11 13–15 Two important shortcomings impede hospitals in particular in their ambition to learn from SAEs.

First, SAE investigations often neglect the complexity of healthcare.13 15 Current methods applied for SAE analysis, focus on finding one linear root cause, even though SAEs are more likely to arise from interactions and combinations of causes and contributing factors in the complex healthcare system.10 13 14 SAE analysis thus needs a shift from focusing on finding the ‘one’ linear root cause towards searching for the combination of interrelated contributing factors and causes. The incorporation of human factors thinking in SAE analysis is assumed to induce this shift.16–20 Human factors is concerned with the understanding of interactions and interdependencies between humans and other elements of a work system.21–23 It searches for opportunities to design healthcare systems that have a greater tolerance of faults and thus might improve the resilience of the system.24–26 One model that integrated principles of human factors into the healthcare domain is the Systems Engineering Initiative for Patient Safety (SEIPS).27–32 This theoretical model is used to study patient safety hazards and adverse events in various healthcare settings.16 30–35 SEIPS is based on the well-known structure–process–outcome model for healthcare quality.36 It therefore is assumed to be familiar to audiences working in healthcare quality and patient safety.31 Although the potential of SEIPS to improve patient safety is proven,31 adoption of the model in retrospective SAE analysis is still limited and should be refined and accelerated.16 30

A second shortcoming of current SAE investigations is that hospitals typically focus on single events within their own organisation,13 14 even though attempts to learn from aggregate analysis of multiple events across hospitals, as already performed in Australia37 and the USA,38 39 are believed to improve learning.38–41 It helps to discover combinations of recurring, underlying, patterns of causes and contributing factors42 and may improve the formulation of more effective, system-aimed recommendations.14 38 However, methods for analysing SAEs vary substantially,10 which complicates aggregate cross-hospitals analysis of SAEs. For example, in the Netherlands, where aggregate analysis of SAEs is scarce, hospitals use three differing methods for analysing SAEs41 43 44: Prevention and Recovery Information System for Monitoring and Analysis (PRISMA-medical),45 Tripod Beta46 and Systemic Incident Reconstruction and Evaluation (SIRE).47 These methods are useful for a structured analysis of SAEs, yet based on their description in scientific articles,41 44 48 49 anecdotal evidence43 46 47 and experience, each method has its own approach, focus and limitations. PRISMA-medical, for example, is highly analytical and categorises latent (technical and organisational) as well as active (human) failures.45 However, the absence of organisational or technical barriers that could prevent SAEs are not considered a root cause in this method.50 Tripod Beta, however, thoroughly investigates system failures and organisational barriers that could prevent SAEs. However, human errors are also explained as organisational or system failures,46 49 even though evidence indicates that flaws in the system and cognitive factors play an important role in the emergence of human error.51–53 SIRE, in turn, emphasises particularly on the primary process and provides a comprehensive narrative description of the event.47 54 To ease aggregate cross-hospital analysis of SAEs, Smits et al43 made a primary attempt to integrate the structures and foci of PRISMA-medical, SIRE and Tripod Beta into a generic framework. Anecdotal evidence of the benefits of this framework to perform an aggregate cross-hospital analysis was presented, but the study emphasised the importance of further development and evaluation.43

Addressing the complexity of healthcare13 and using a consistent method10 41 could thus improve learning from SAEs. This could help to formulate more effective recommendations to enhance patient safety. An alternative approach to study SAEs is therefore necessary. This study aims to develop a novel generic analysis method (GAM) that integrates the SEIPS model31 32 and the PRISMA-medical, SIRE and Tripod Beta framework43 to: (1) facilitate a more holistic analysis using a human factors perspective and (2) ease aggregate analysis across hospitals.

Methods

We followed a structured process (figure 1) of developing, evaluating, testing and refining prototypes of a GAM.

Figure 1

Overview of the steps taken in developing the Generic Analysis Method. PRISMA, Prevention and Recovery Information System for Monitoring and Analysis; SIRE, Systemic Incident Reconstruction and Evaluation; SEIPS, Systems Engineering Initiative for Patient Safety.

Description of the prototype

A first prototype of the GAM was developed based on the theoretical SEIPS model31 32 and the framework43 combining PRISMA-medical, SIRE and Tripod Beta. This prototype included three main pillars. The first pillar addresses the gathering of basic SAE information. Second, each sociotechnical domain of the SEIPS model is evaluated. All relevant characteristics should be collected about the: (1) persons involved, (2) tasks performed, (3) technologies used, (4) organisational factors affecting the work, (5) the physical environment, and (6) the external environment. In this evaluation, we also adopted some important elements of PRISMA-medical, SIRE and Tripod Beta as accumulated by Smits et al43 related to these six domains. The third pillar focuses on the outcomes of the SAE for the involved patients and/or their family, healthcare professionals and the organisation. This part also includes the evaluation of the recommendations formulated in response to the event.

Evaluating and testing the prototype

We started with consulting two theoretical experts (A), both academics in the field of patient safety, to evaluate the first version of the prototype and ask for potential additional theoretical concepts that should be embedded.

We then used the prototype for a pretest session (B), in which the author (MCB) analysed six SAE reports covering the variation of RCA methods used by Dutch hospitals (PRISMA-medical, SIRE and Tripod Beta). These reports were based on thorough SAE investigations performed by independent committees of various hospitals in 2018 or 2019. The multidisciplinary hospital committees investigating the event, typically consisted of a combination of clinicians and quality and safety officers. Committees combined multiple sources for studying the event. The most important source being in-depth interviews with involved healthcare personnel and patients and/or family. Additionally, the committees reviewed all relevant information such as patient records, test results (eg, medical images, laboratory findings, etc) and documentation (eg, guidelines, work instructions, etc). The combination of interviews with the persons involved and reviewing all relevant documentation resulted in an extensive and detailed reconstruction of the event. After the event description, the hospital committees performed an RCA and formulated recommendations to prevent recurrence. The comprehensive data in the SAE reports was suited for a retrospective (re)analysis of the event and, hence, a first test of the prototype’s content and face validity. By including a sample covering all three RCA methods, we could test whether our prototype was applicable irrespective of the initial RCA method used. This step contributed to improving the usability of the prototype, for example, the prototype could be aligned with the working methods of the hospitals.

Next, a panel with experts from practice (C) was consulted for a thorough evaluation of the face and content validity of the prototype during a face-to-face meeting. This panel consisted of a mix of quality and safety officers (n=8) and medical specialists with direct experience in patient safety (n=5). All of them were currently practising in various Dutch general hospitals (n=12). Minutes were taken during the meeting, and the transcribed notes were sent for verification to the participants afterwards.

Subsequently, two potential users, both quality and safety officers, tested the prototype (D). Just as in the pretest session, information-rich SAE reports were used. The users analysed two distinct reports, written by two general hospitals in 2018. This test session was meant to further evaluate content validity of the method as experienced by potential users. It also helped to find flaws and practical barriers and provided the possibility to observe users’ interactions with the prototype. Commentaries from the respondents were used to refine the usability. In addition, the author (MCB) analysed the same two reports. This allowed a rough assessment of the agreement in analyses between the potential users and the researcher. As the most important questions were open ended, a qualitative approach on assessing the agreement between raters was performed. Overall agreement on all questions was reviewed by eye assessment, and differences between the interpretation of raters for the important questions were reflected on (eg, by considering: did raters give different answers? Why did raters not answer that question the same way? Were there random outliers, or did it show a systematic difference in interpretation?).

Patient and public involvement

Since the method is specifically intended for use by healthcare professionals to learn from SAEs across hospitals, professionals (potential users) were consulted at multiple stages during the study. Patients were not involved in developing the method as they currently have no formal role in performing (aggregate) analysis of SAEs. Nevertheless, it is important to involve patients and their family in the SAE investigation by including their vision on the event through interviews and giving them insight into the findings.

Results

Evaluation and test results

In the first step of evaluation, theoretical experts (A) stressed the importance to assess the element of human error in SAEs and therefore referred to the model of unsafe acts.52 This model helps to differentiate between various types of human errors (eg, violations, mistakes, slips and lapses). This could provide a more in-depth understanding of these errors. Another concept the experts recommended to include were the efficiency–thoroughness trade-off principles. Evaluation of the trade-offs between working efficiently and working thoroughly, in this context safely, might enhance our understanding of safety critical decisions being made in complex healthcare situations.55 As these trade-offs reflect situations in which there is a conflict between, for example, production pressure and providing safe and high quality care, they may raise awareness to more system-related issues.

After pretesting the prototype (B), questions and answer categories were reconsidered and rewritten in order to make it more compatible with the information gathered in SAE investigations. Test cases also gave a useful overview of the SAEs and helped to identify relevant characteristics, contributing factors and root causes related to each of the sociotechnical work system domains. Examples of these findings are listed in table 1.

Table 1

Description of the six interacting domains of sociotechnical work system, adopted from Holden et al31 and Carayon et al32

Sociotechnical element	Description	Example of influence on SAE
Person(s)	The central component of the work systems are the persons involved. This component is not limited to the healthcare professional(s), but also considers the patient and their family, and professionals of supporting services. The characteristics of the professionals and teams can be analysed, focusing on the level of knowledge, level of experience and the perceived workload. Additionally, the teamwork and collaborations are considered. Various patient characteristics (eg, physical, psychological or social) can contribute to a SAE and are therefore be included in the analysis.	Patient/family A patient indicates atypical complaints when suffering from a ruptured aneurysm of the abdominal aorta. The family of a patient disputes the treatment chosen and frustrates the care provided.
		Healthcare professional(s) A healthcare professional chooses to work after two nights of bad sleep due to personal problems, makes a slip and incorrect medication is being prescribed. An interim surgeon uses an instrument (s)he is unfamiliar with, which results in misapplication.
		Other professionals A technician turns off the alarm modus of a patient monitoring system during maintenance by mistake, as a result of insufficient product knowledge.
Tasks	The tasks element evaluates characteristics of the tasks of the persons involved. These can be considered by looking at the complexity, variety and ambiguity of the tasks, and observing if they coincide with other tasks. Furthermore, eventual efficiency–thoroughness trade-offs made while performing these tasks must be evaluated.	Tasks A radiologist swaps two patients with similar names and conditions and fills in his assessment of a magnetic resonance image in the record for the wrong patient.
		Tasks-related efficiency–thoroughness trade-offs It will be checked/done by someone else* – a patient is transferred from a busy emergency department to another ward without performing an ECG, because time is scarce and the personnel expect the nursing ward receiving the patient will take care of this. This way it is much quicker* – instead of following a procedure in which medicine is allocated on the patient ward, accompanied by the patient, a nurse prepares the medicines of all patients together at the nurses’ station bearing a greater risk of making mistakes in swapping medicines.
Technologies	Technologies used by the person(s) involved must be evaluated. Important features of the technologies are, for example, how easy they are to use, their accessibility, level of automation, functionality and how easily the technologies can be transported and adopted to other settings. This evaluation is performed preferably by both healthcare professionals and technical experts.	A monitor in the operating theatre crashes during surgery resulting in limited visual information for the surgeon. A ventilator on the intensive care unit does not provide an alarm when the ventilated air was not heated and moisturised because the apparatus had become loose.
Organisation	Organisational factors that contribute to the SAE and/or may influence other elements. This includes an evaluation of material factors, such as financial incentives and the accessibility and availability of resources (eg, time, money, goods and services) as well as social factors (eg, management style, culture, hierarchy, social norms and values).Since the availability of resources and social factors are analysed in this component, an assessment of the eventual efficiency–thoroughness trade-offs related to social factors and resources is incorporated.	The staffing of nurses is so low that during the holiday season the minimum level of experience is not guaranteed. A hierarchical relationship between a resident and his supervisor forms a barrier to call during a night shift. Instead, the resident saves all his questions until the next day.
		Efficiency–thoroughness trade-offs related to the organisation We always do it in this way here* – an efficient, but unsafe, workaround—a way of temporarily addressing workflow problems—is taught to new nurses. It is not my/our responsibility* – after transferring a cardiac patient from the cardiology ward to the intensive care unit, the cardiologist does not feel responsible for this patient anymore, while the patient is, among other things, still suffering from cardiac instability.
Physical environment	Factors that define the physical environment such as lighting, noise, vibrations, temperature, the physical arrangement of the room(s) and the available space and air quality.	Noise on an emergency department disturbs the communication between a nurse and a physician leading to a miscommunication about the volume of medicine. The physical arrangement of a door, bed and other furniture in an emergency department room impedes the crashcar from being positioned inside the room, which hinders the rapid response team in their interventions.
External environment	Factors on a macrolevel that might affect decisions on a microlevel in the sociotechnical work systems and procedures.	Shortages in the labour market may lead to a deficiency of qualified nurses. A budget cut and sustainability policy, aimed at reducing water waste, might lead to a faulty legionella bacteria prevention protocol.

*The efficiency–thoroughness trade-off descriptions are adopted from Hollnagel.55

The discussion with experts from practice (C) resulted in a few additional topics that were particularly relevant and therefore needed to be explicitly adopted (eg, the identification and classification of fragility among elderly patients). Also, more practical issues were put forward such as the description and categorisation of clinical areas and age groups. The boundaries of external factors, such as financial restrictions or labour shortages, were argued and established as well.

The final test phase by potential users (D) brought up some important modifications of the questionnaire (eg, textual revision of questions, adjustment or addition of answer categories and merging some of the questions). This test session also presented important misinterpretations of a few questions and answer categories. Overall agreement between the potential users and researcher was acceptable, with exemption of the misinterpreted questions. We decided better instructions were needed and composed an explanation file and example case in which the ambiguities were explained to help future users in interpreting the questions and answer categories as intended.

Analysis of the event reconstructions using the prototype helped in systematically considering all aspects of the sociotechnical work system. This resulted in a holistic analysis of the event. Mapping contributing factors and causes according to the prototype assisted in evaluating interactions and interdependencies between factors of various domains, addressing the complexity.

Description of the final GAM

The described process of developing, evaluating, testing and refining prototypes resulted in a final version of the GAM, consisting of a framework (figure 2) and an affiliated questionnaire (online supplemental appendix 1). Both follow a basic structure for analysing SAEs.

Figure 2

GAM framework, based on the work of Smits, Langelaan, and De Groot43 and the SEIPS models31 32

The first step of the model only intents to gather basic information on what happened. A suggestion of the most important elements that could be included is presented in the upper part of figure 2.

The method then evaluates the six domains of the work systems according to the SEIPS model. Each domain includes a description of relevant characteristics and potential contributory factors. Interactions and interdependencies between domains and factors can be mapped to address and visualise complexity. Online supplemental appendix 2 provides an example of a case for which the work system was analysed using the GAM. In the tasks element, we included a question to get insight in unsafe human acts52 contributing to the event. For the tasks and organisational elements, trade-offs are evaluated between efficiency and thoroughness. Table 1 specifies the six domains of the work system with a more detailed description and provides some examples, based on the test sessions and previous case studies.

After analysing the work systems and processes, the consequences of the SAE must be considered. It is important to assess the effects on patients and family, as well as caregivers, other professionals and the healthcare facility. Outcomes of SAEs can be multidimensional and may affect the physical, psychological, juridical, social and financial situation of all those involved. When evaluating the impact of SAEs, both immediate as well as distant outcomes must be considered. Formulated recommendations are incorporated in this part as well, so evaluation of the type(s) of recommendations as suggested by the hospital in response to the SAE is possible.

Discussion

Previous studies highlighted issues that hamper hospitals in their ambition to learn from SAEs. Methods used for analysing events, for example, have a strong focus on finding a linear root cause, thereby neglecting the complexity of healthcare.13 14 Hospitals also often study single events within their organisation, instead of analysing multiple events across organisations.13 Substantial variety in methods applied might complicate attempts to perform such aggregate cross-hospital analysis. Although potential solutions such as integrating human factors thinking in SAE analysis and using a more consistent analysis method have been proposed,10 18 20 41 there remains a need for methods embracing these possible improvements. This study aims to develop a novel GAM that stimulates a holistic analysis using a human factors perspective and eases aggregate analysis across hospitals.

A GAM prototype was developed based on a human factors model and integrating important elements of three currently practised methods for analysing SAEs. The paper describes the multiple steps of carefully evaluating, testing and refining prototypes. Suggestions of theoretical experts to complement the prototype were incorporated. Pretesting, evaluation by experts from practice and user test sessions, helped to improve the usability and face and content validity. User test sessions also allowed us to roughly assess inter-rater agreement and think of manners to improve future reliability. These steps resulted in a final version of the GAM, consisting of a framework and affiliated questionnaire. These assist in gathering structured information about the event, evaluate all domains of the sociotechnical work system, map interactions between factors from various domains and assess outcomes of the SAE.

The SEIPS model on which the GAM is based helps to focus beyond hierarchical, linear and causal relations and stimulates evaluation of complexity. To successfully unravel the complexity of healthcare work systems, it is essential to understand the interactions and interdependencies between the persons involved and the other domains of the sociotechnical work system (eg, technology, tasks, organisation and environment). As described by Holden et al,31 any analysis of the work systems should, therefore, include three important concepts: (1) the sociotechnical work systems must be viewed as dynamic, hierarchical and interactive, (2) all individuals engaged in a process or work system should be mapped systematically to reveal social structures and (3) analysis must depict how dynamic systems evolve over time in intended and unintended ways. As exemplified in online supplemental appendix 2, the GAM helps to evaluate the dynamics and interactions within a work system. By mapping all domains of the work system including the persons involved in a structured manner, it might be possible to depict the complex properties of SAEs. Whenever more SAEs are systematically mapped this way, underlying patterns of recurring interactions and structures might emerge. The inclusion of the trade-offs between efficiency and thoroughness,55 as adopted in the organisational and task elements of the GAM, could provide an opening to compare actual (work as done) and expected work practices (work as imagined). This could help to understand how work systems have evolved over time in intended and unintended ways.

In addition to incorporating human factors thinking, GAM combines the narrative approach of SIRE47 with the more analytical approach of PRISMA-medical45 and Tripod beta46 on both organisational barriers and human-related issues. Integrating important elements of these methods helps to perform a holistic analysis. At the same time, the combination of existing methods in one generic method might ease aggregate analysis of SAEs across hospitals.

As GAM is structured according to the widely used SEIPS model30 31 56 and built on methods that are currently applied by hospitals, we believe GAM may be easily adopted. Other promising methods that aim to improve learning from RCA such as the Human Factors Analysis and Classification System,57 AcciMap58 and Functional Resonance Analysis Method59 were not specifically build on the well-known SEIPS model and existing methods. Other methods may also need a specific practice to investigate SAEs or tend to focus specifically on a small and delineated process. This might make them less easy to adopt for SAE analysis. Existing initiatives for aggregate analysis, such as those performed in the USA and Australia37–39 and the preliminary attempts in the Netherlands,41 43 lack the important human factors perspective to guide analysis. Aggregate analysis using the GAM might therefore lead to better understanding of SAEs and could result in more effective system-aimed solutions. Besides retrospective analysis of patient safety events, methods such as the Health Care Failure Mode and Effect Analysis60 61 and Tripod Delta HC62 intent to prospectively identify (latent) risks for patient safety. These methods remain important and can be used complementary to retrospective aggregate analysis of SAEs using the GAM. Outcomes of retrospective GAM analysis might help to identify risks for such prospective analysis.

In its most basic application, GAM can help to perform a structured, holistic analysis of SAE reports from various hospitals, irrespective of the initial RCA method used. The method is thus suited to study the existing ‘treasure’ of SAE reports that is stored in the archives of hospitals and unleash its potential to jointly learn from SAEs. Considering GAM was developed in the Netherlands and integrates methods Dutch hospitals use for SAE analysis, the generic method is particularly suited for application in the Dutch context. However, PRISMA-medical, SIRE and Tripod are not solely used by Dutch hospitals. In particular PRISMA-medical is applied by hospitals in various other countries as it has been accepted by the World Alliance for Patient Safety of the WHO.63–65 At the same time, we believe GAM is also appropriate to stimulate and improve aggregate analysis of SAEs for hospitals currently using other methods, such as cause and effect diagrams, five whys and fishbone diagrams. These methods are after all also based on the principle of investigating single SAEs and a search for a root cause. Application of the GAM might in this case be more time consuming though.

Limitations

A limitation of the study was the modest user test session, as it was limited to two cases. The process of testing was highly time consuming for potential users. Although we performed other testing and evaluation steps, including a pilot phase in which six more reports were analysed by the author, the number of cases in the user test session can be considered as a weakness. More extensive testing is therefore needed to further assess reliability and validity. Further testing might also provide leads to lean the method and improve its usability.

Another issue we should address is that we tested the GAM by applying it to reanalyse information gathered in SAE reports. Although the reports provided a thorough description of the event, the investigation did not specifically study the event according to the GAM. This might have made it difficult to subtract new or other relevant factors from the reports than those already found in the initial RCA. The test sessions, however, indicated that adopting the GAM perspective to reanalyse information as included in the event reconstruction of the SAE reports can still lead to new insights. Directly studying an event using the GAM enables investigators to ask specific additional questions during the interviews with those involved on subjects that are only limitedly covered in traditional SAE investigations and analyses. Based on this study, we cannot draw strong conclusions on the usability of GAM for directly studying a SAE, but given its value in the analysis of SAE reports, the method seems promising for this purpose as well.

Conclusion

A novel GAM was carefully developed. On balance, the method is a promising tool for taking a next step in learning from SAEs. It facilitates a holistic analysis of the SAE and addresses the complex sociotechnical work system in which SAEs in healthcare originate. GAM could also ease aggregate analysis of SAE across hospitals. Using GAM might empower hospitals in their ambition to learn from SAEs and could help to formulate more effective system-aimed recommendations to prevent recurrence of events. The method can now be used to analyse a large sample of events to further explore its potential to improve patient safety.