Algorithmic prediction of failure modes in healthcare.

BACKGROUND: Preventing medical errors is crucial, especially during crises like the COVID-19 pandemic. Failure Modes and Effects Analysis (FMEA) is the most widely used prospective hazard analysis in healthcare. FMEA relies on brainstorming by multi-disciplinary teams to identify hazards. This approach has two major weaknesses: significant time and human resource investments, and lack of complete and error-free results.
OBJECTIVES: To introduce the algorithmic prediction of failure modes in healthcare (APFMH) and to examine whether APFMH is leaner in resource allocation in comparison to the traditional FMEA and whether it ensures the complete identification of hazards.
METHODS: The patient identification during imaging process at the emergency department of Sheba Medical Center was analyzed by FMEA and APFMH, independently and separately. We compared between the hazards predicted by APFMH method and the hazards predicted by FMEA method; the total participants' working hours invested in each process and the adverse events, categorized as 'patient identification', before and after the recommendations resulted from the above processes were implemented.
RESULTS: APFMH is more effective in identifying hazards (P < 0.0001) and is leaner in resources than the traditional FMEA: the former used 21 h whereas the latter required 63 h. Following the implementation of the recommendations, the adverse events decreased by 44% annually (P = 0.0026). Most adverse events were preventable, had all recommendations been fully implemented.
CONCLUSION: In light of our initial and limited-size study, APFMH is more effective in identifying hazards (P < 0.0001) and is leaner in resources than the traditional FMEA. APFMH is suggested as an alternative to FMEA since it is leaner in time and human resources, ensures more complete hazard identification and is especially valuable during crisis time, when new protocols are often adopted, such as in the current days of the COVID-19 pandemic.

Background

Medical errors are not rare, but are potentially preventable [1-6]. About 10% of all patients admitted to hospitals are affected by an adverse event, most of which can be prevented [7] or minimized [8, 9]. Crisis like the COVID-19 pandemic creates an even acuter focus on the prevention of errors. Institutional initiatives have made prospective hazard analysis (PHA) an integral part of medical practice, but the methods used for it are tedious and long [10].

Failure Modes and Effects Analysis (FMEA) is a widely used method for PHA in healthcare [11, 12]. FMEA relies on brainstorming methodology by multi-disciplinary teams to identify hazards, which results in two points of concern. The first point of FMEA is that it requires a significant investment of time and human resources [3, 18–22]. In the complex reality of most healthcare organizations, especially public systems, this is a critical limitation. The current COVID-19 pandemic made it even acuter, as new or revised protocols are often adopted without experience to support an FMEA procedure. Previous studies have tried to overcome the limitations of significant investments of time and human resources by omitting selected team members, by using simplified scoring methods or by foregoing creating a process map [23]. Although both omissions may result in time and human resource reduction, they might do so at the expense of the quality of the PHA [23]. Also, FMEA results depend on the skills and previous experience of the practitioners [13-17] and thus might turn out incomplete, less valid and not error free [17]. The second possible shortcoming of FMEA is that its validity, as well as that of most PHA methods, has not yet been fully proven [21, 24–28]. This may lead to poor correlation between hazards, identified by FMEA teams and those reported on the incident database [29]. This incomplete validity occurs because some hazards are not identified [16, 17, 30]. Thus, different teams performing the same PHA on the same process may identify different hazards and may assign them different weights [27, 28, 31], since they might score hazards subjectively [24, 32, 33, 36].

We herein present the algorithmic prediction of failure modes in healthcare (APFMH), a process that uses a structured, two-part questionnaire that replaces the resources-draining brainstorming that is used in the FMEA for hazard identification and ranking. The first part of the APFMH questionnaire is based on methodical hazard identification checklist (MHIC) [35], a novel methodology that uses four prototypical categories of hazards for the structured identification of possible hazards in each step of a given medical or administrative process. Compared to traditional brainstorming, MHIC is much better for prospective identification of hazards in healthcare systems [35]. This in turn makes it suitable for analysis of new or significantly revised processes. The second part of the APFMH is hazard prioritization, which is done according to a decision tree.

The main objective of the present study was to introduce the APFMH, an alternative method for prospective hazard analysis that is lean in time and human resources and ensures more complete identification of hazards, especially during times of crisis. Specifically, the objective is to analyze whether the APFMH is leaner in time and human resources and ensures the complete identification of hazards in comparison to the regular FMEA.

Methods

Setting

The study took place at the dedicated imaging unit of the emergency department (ED) at Sheba Medical Center, a 1900-bed academic medical center in Israel. The study focused on possible failures in the process of patient identification. The imaging unit provides general X-ray services, CT with and without contrast agent, ultrasound services and invasive X-ray diagnostics. MRI and non-urgent ultrasound tests are performed at the Sheba Medical Center central imaging department. The study focused on improving the patient identification for X-ray and CT imaging processes. This location was chosen because the ED has the most dynamic environment in the hospital, therefore has processes that are most prone to patient identification errors.

Participants

The FMEA team included seven members who participated in the brainstorming phase of the FMEA process. Two of them are quality and risk management experts, three of them are X-ray technicians and two are physicians from the imaging department. All members are employees of the medical center. The APFMH facilitator included a quality engineer (PhD), with theoretical knowledge and practical experience in both APFMH and FMEA in healthcare systems. The facilitator who performed the APFMH was not involved in the FMEA and was not exposed to its products. The APFMH validation team included two quality and risk management experts who were part of the FMEA team. They validated the product of the MHIC-derived list, approved that all hazards so described were indeed possible in real life and pointed out the hazards missing from the list.

Outcome measures

As outcome measures, we have considered the following: the total number of hazards predicted by APFMH and FMEA methods, stratified according to high and low ranked hazards; the total participants’ working hours (teams and the facilitator) invested in each process; the total number of hazards predicted and ranked by APFMH which were approved by the validation team. We also compared the total number of adverse events in patient identification before and after the implementation of the recommendations.

Study design

The flow chart of the intervention is depicted in Figure. 1. It contains six steps.

Figure 1

Flow chart describing the intervention.

The first step is process analysis by FMEA. The FMEA was done by the seven team members, as described above, based on the flow chart of the process. The analysis was conducted following hazard identification, ranking, prioritization according to the risk priority number and a recommendation brainstorming session.

The second step is process analysis by APFMH which included two parts: first, creating a flow chart of the process by the facilitator, and, second, prediction of hazards by the facilitator, using MHIC [35]. MHIC is a structured, focused task checklist to break down the task of hazard prediction into subcategories. The four prototypical categories include four types of hazards [35]: over-doing (OD), under-doing (UD), mistake (M) and concept (C). OD hazard is the hazard of adding unnecessary elements to the process, such as typing the patient’s identifiers twice. UD hazard is the omission of necessary elements from a process, for example, omitting elements from the registry of the patient. M hazard is wrong implementation of a correct concept. For example, a patient undergoes an X-ray exam although a CT was ordered. C hazard is an undefined, missing, unfit or otherwise incorrect concept, for example, acting under a wrong diagnosis or implementing an incorrect diagnostic plan.

For each step of the process, the checklist asks whether OD/UD/M/C hazards can occur. The first part of the APFMH questionnaire uses a five-column table that includes on the first column the step in the process to be examined, and the other four are OD, UD, M and C hazards. The user is asked to predict all possible hazards that make sense for each type of hazard for each step in the process.

The third step is hazard ranking and prioritization. In APFMH, the ranking and prioritization of hazards is done by the facilitator, following a decision tree algorithm (see Figure. 2) that contains two parts: one for the severity of the impact, which is similar to the severity ranking in FMEA. The second part relates to the existence of measure(s) that can prevent the hazard or its damage within the process. This part is unique to the process of APFMH.

Figure 2

APFMH decision tree.

In the first part, the facilitator is ranking the severity of the impact of the hazard as: I—catastrophic event, II—major event, III—moderate event and IV—minor event. If the ranking is I/II, the hazard continues to the second part of the diagram.

The second part of the diagram relates to the existence of measure(s) that can prevent the hazard or its damage within the process. This new factor is an objective factor that replaces the ‘probability’ factor that is used in FMEA and has not been validated yet. The ranking scale of the measures that can prevent the hazards or their damage is: A—None; B—the preventive measure would probably not prevent the hazard or its damage; C—the preventive measure would probably prevent the hazard or its damage and D—the preventive measure will eliminate the hazard or its damage. The hazard analysis is determined by the facilitator and verified by the validation team members. A measure that can prevent the hazard or its damage is based on the flow chart of the process. The ranking is applied to measures that are part of the process and can prevent the hazard or its damage.

In the second part, if the ranking is A/B, then the hazard gets the highest priority.

The fourth step is APFMH validation by FMEA team. The team members were given the lists of hazards predicted by APFMH and their ranking and prioritization. They were asked if the hazards identified in the process make sense. In case of a negative response, the team members were asked to delete hazards that do not make sense to them. In addition, the team was asked to approve/correct the ranking and prioritization of the hazards.

The fifth step is recommendations. The validation team discussed the recommendations for new preventive measures that can be applied in order to tackle the hazards that were ranked as high priority.

The sixth step is the implementation of the recommendations, which was done at the end of 2017 (October -December 2017).

It is important to mention that after each step performed by the facilitator, the outputs of the steps were submitted for approval or comments by the validation team. These people are the most superior in the system under study, projected to be the most experienced, and have the best perspective to evaluate the type of hazards and their likelihood of occurring and their possible damages.

Data collection

The following data were collected by the two quality and risk management experts and analyzed by our statistical expert: (1) The total number of hazards predicted or identified by APFMH and FMEA methods, respectively, stratified according to high and low ranked hazards; (2) The total number of hazards that were not approved by the validation team; (3) The total participants’ working hours (teams and the facilitator) invested in each of the two processes. This included the hours the process analysis took, from start to conclusion.

All the adverse events that were reported in the context of patient identification, before and after the recommendations implementation, were collected from the hospital computerized reporting system.

Statistical analysis

The proportion of success for each method (FMEA/APFMH) and severity (low/high) combinations were calculated. For each of the four combinations, Wilson continuity-corrected confidence intervals were calculated using SAS9.4 FREQ procedure with the BINOMIAL option. The Wilson interval has been shown to have better performance than both the Wald interval and the exact (Clopper–Pearson) interval [29, 30]. Due to the matched data structure, a stratified, conditional logistic regression model was applied to infer the differences between the methods for both high and low hazard cases. Exact Logistic procedure of SAS 9.4 was applied due to the sparsity (‘zero cells’) of the data.

Results

Predicted and identified hazards

APFMH predicted 32 hazards, beyond which FMEA did not add. Of these, 12 (37.5%) were ranked as entailing ‘high risk’ and 20 were ranked as ‘low risk’. FMEA identified only 15 hazards, of which 7 (47%) were ‘high risk’. FMEA did not identify 17 hazards, of which 5 (29.4%) were ‘high risk’. According to the validation team members, all hazards that were predicted by APFMH were found to be real and sensible. The total number of the predicted and identified hazards by both methods, the number of ‘high ranking’ hazards by both methods and examples of hazards that were predicted by APFMH and were not identified by FMEA can be seen in Table 1. The comparison of hazards indicated by APFMH and FMEA according to hazard ranking can be seen in Table 2. All hazards that were identified by FMEA were identified by APFMH.

Table 1

The total number of identified hazards in both methods, the number of ‘high ranking’ hazards in both methods and examples of hazards that were identified by APFMH and not by FMEA

Process section	Total hazards identified by APFMH/FMEA		‘High ranking’ hazards by APFMH/FMEA
	APFMH	FMEA	APFMH	FMEA	Example of hazard that was identified by APFMH and not identified by FMEA
Open patient details at RISS system	3	1	-	-	Open multiple patient information at the same time
Calling the right patient	3	1	-	-	Calling to multiple patients
Patient enters the examination room	4	1	-	-	Multiple patients entering the examination room
Identifying the patient by name and ID number	9	2	5	1	Questioning a disoriented patient about their identification information
Comparison of patient referral details and patient information in the RISS system	6	4	3	3	An incomplete comparison is done
Correlation between patient details in the MOD vs. RISS systems	2	2	2	2	____
Documentation of conducting an image	3	2	2	1	More than one patient opens
Confirmation of an image execution	2	2	-	-
Total	32	15	12	7

*The system RISS is a computer system that is used as a patient record in imaging. The system MOD is a computer system that is used as software for performing the images.

Table 2

Comparison of hazards identified by APFMH and FMEA, according to hazard ranking—high and low

			FMEA
All hazards			Identified by FMEA	Not identified by FMEA	Total hazards
APFMH	Identified by APFMH	High	7	5	12
		Low	8	12	20
		Total	15	17	32
	Not identified by APFMH	High	0	0	0
		Low	0	0	0
		Total	0	0	0
Total hazards			15	17	32

The success rate (proportion of identified hazards by a method, compared to the grand total that were identified by both methods together) by FMEA for high-risk hazards was 0.583 (95% CI: 0.286–0.835), while the success rate by APFMH for high-risk hazards was 1.00 (0.699–1.00). The success rate by FMEA for low-risk hazards was 0.4 (0.200–0.636), while the success rate by APFMH for low-risk hazards was 1.00 (0.799–1.00).

The success rate (proportion of identified hazards by a method, compared to the grand total that were identified) by FMEA for high-risk hazards was 0.583 (95% CI: 0.286–0.835), while the success rate by APFMH for high-risk hazards was 1.00 (0.699–1.00). The success rate by FMEA for low-risk hazards was 0.4 (0.200–0.636), while the success rate by APFMH for low-risk hazards was 1.00 (0.799–1.00).

The results of the logistic regression show that there is a highly significant difference (P < 0.0001) between the two methods, with odds ratio (24.03 and 95% CI: 5.19 to infinity) indicating that the odds of a correct prediction or identification were significantly higher using APFMH instead of FMEA.

Number of working hours invested in each method

The working hours invested in each method (FMEA vs. APFMH) can be seen in Table 3. FMEA required 63 h, while APFMH required 21 h.

Table 3

Participants’ working hours in each method (FMEA vs. APFMH)

		No. of participants’ working hours
Method	Phase	Participants	Hours	Total
FMEA	Process modeling (flow chart)	1 (facilitator) + 7 (team for approving)	1 + 0.5×7	4.5
	Hazard identification and analysis	7	6	42
	Recommendations	7	2	14
	Administration and preparations	2	3	6
	Total: 63 h
APFMH	Process modeling (flow chart)	1 (facilitator) + 7 (team for approving)	1 + 0.5×7	4.5
	Hazard identification	1 (facilitator)	1	1
	Hazard ranking and prioritization	1 (facilitator)	1.5	1.5
	Team validation	7	1	7
	Recommendations	7	1	7
	Total: 21 h

The number and ranking of hazards approved by the validation team (APFMH)

The validation team approved all the rankings that resulted from the APFMH process.

Implementation of the recommendations

The recommendations were implemented during December 2017. Before the implementation, during 2017, 36 adverse events related to patient identification were reported out of 159 665 procedures. After the implementation, during 2018, only 20 adverse events related to patient identification were reported out of 168 122 procedures (P = 0.0026) (Table 4).

Table 4

Number of procedures, number of adverse events related to patient identification and rate of adverse events. Number of procedures before and after the recommendation’s implementation

	2017(Before implementa-tion)	2018(After implementa-tion)
No. of procedures	159 665	168 122
No. of adverse events	36	20
Rate of adverse events	0.02%	0.01%

Searching the root causes for the adverse events that occurred in 2018 (after the implementation of the recommendations) indicated that most of them were potentially preventable, had all recommendations been fully implemented.

Discussion

In this paper we introduce the APFMH, an alternative method for the traditional FMEA. The results suggest that our novel version of APFMH is more effective in identifying hazards (P < 0.0001) and is leaner in terms of resources invested than the traditional FMEA method: FMEA required 63 h, while APFMH required 21 h. In addition, after the implementation of the recommendations, the adverse events decreased by 44% annually (P = 0.0026), while most of them were likely to be prevented had all recommendations been fully implemented. This new method is especially suitable in recent pandemic times, when time and resources are limited and, therefore, a complete and accurate FMEA is not achievable.

The FMEA method was first developed in 1950 by reliability engineers to detect and assess problems arising from weapons system malfunction. In the 1960s, it was adopted by NASA for forecasting failures; in the 1990s, it was adopted by the U.S. manufacturing industry. The method was adopted in the world of patient safety improvement by the National Center for Patient Safety (NCPS), founded in 1998 by the Veteran Health Administration (VA), and had become the most widely used technique in healthcare to identify and eliminate known and/or potential hazards before they actually occur. Although FMEA is recommended by the VA National Center for Patient Safety and The Joint Commission, it has not become a routine, daily working tool. Rather, it is adopted very slowly [21] and usually implemented merely as required by regulatory authorities and mostly just prior to audits.

Our APFMH enables medical organizations to comply with regulatory requirements and enhance improvement processes by implementing a novel form of FMEA that comes with a significantly lower, organizational and financial price. Moreover, it enables medical institutes to cope with crisis-days’ frequently revised protocols and high likelihood of errors, as in the case of the current pandemic. In this respect it is important to note that the time in terms of manpower taken for APFMH to achieve the analysis and compose the recommendations was a third of that taken for the traditional FMEA. Moreover, a lesser amount of people was involved in APFMH, which saves the need to coordinate meetings with people from various parts of the institute—an element that was not counted in the total invested time. When this is factored in together with APFMH’s superior performance that is reflected in a higher number of identified failure modes, especially highly ranked ones, it calls for a considerably more frequent application of this analysis for an ever-growing variety of medical processes.

It is important to mention that the differences between APFMH and FMEA are both in the identification of the failure cycle and the criteria for rating the failure cycle. Identifying the nature of the failure by the MHIC method within a standard FMEA will probably result in full identification in less time, even if the failure cycle is subsequently rated according to the criteria of the regular FMEA.

The primary limitation of our study is mainly the pioneering aspect; namely, it is first and small. The validation of its applicability awaits further research, part of which is currently in process under our research group. An additional limitation, to some degree, is it being based on logical deduction by a person who is external to the professions being dealt with. We have come across hesitation to adopt a process that was invented by individuals who are external to the profession. The main advantage of this method is its brevity, which increases the chances that teams all over the organization will adopt it joyfully as a substitute to its predecessor.

One might also claim that since FMEA ‘lives’ on the interaction between the participants, who stimulate each other and facilitate the process, using a single risk manager in doing the APFMH outline is risky. Yet, the use of MHIC, a methodical and robust process, obviates this concern, as we found it to result in the ‘discovery’ of risks not predicted by brainstorming, yet approved by the validation team as true possibilities. It is also possible that the sharing of two participants in both study arms might have interfered with the study. Yet, since the FMEA was done first, the more methodical way of prospective prediction could not be influenced. Because it essentially does not require previous experience on the part of the analyzar.

In conclusion, our results suggest that our novel version of APFMH is more effective in identifying prospective hazards, leaner in terms of resource allocation and better applied during crisis time when new protocols are frequently presented, than the traditional FMEA.

Using this tool may influence the culture and nature of healthcare services and will potentially enable smooth continuation of quality and safety control even during crises. Moreover, since this methodology can be used in both administrative and medical processes, it can probably be used in other industries in which hazards identification is critical. In the current COVID-19 atmosphere, where all resources are short and limited, using our lean method makes it possible to run it online with new protocols introduced frequently, which cannot be done with the more resource-draining FMEA. Further studies are needed to prove this claim. We envision the process of prospective identification of failure modes, applied with the lean method of APFMH, becoming an everyday routine, whenever it is deemed necessary, i.e. for either newly adopted routines or existing routines that have yielded too many failures.