Failure mode and effects analysis drastically reduced potential risks in clinical trial conduct.

BACKGROUND: Failure mode and effects analysis (FMEA) is a risk management tool to proactively identify and assess the causes and effects of potential failures in a system, thereby preventing them from happening. The objective of this study was to evaluate effectiveness of FMEA applied to an academic clinical trial center in a tertiary care setting.
METHODS: A multidisciplinary FMEA focus group at the Seoul National University Hospital Clinical Trials Center selected 6 core clinical trial processes, for which potential failure modes were identified and their risk priority number (RPN) was assessed. Remedial action plans for high-risk failure modes (RPN >160) were devised and a follow-up RPN scoring was conducted a year later.
RESULTS: A total of 114 failure modes were identified with an RPN score ranging 3-378, which was mainly driven by the severity score. Fourteen failure modes were of high risk, 11 of which were addressed by remedial actions. Rescoring showed a dramatic improvement attributed to reduction in the occurrence and detection scores by >3 and >2 points, respectively.
CONCLUSIONS: FMEA is a powerful tool to improve quality in clinical trials. The Seoul National University Hospital Clinical Trials Center is expanding its FMEA capability to other core clinical trial processes.

Introduction

Quality of clinical trials depends on the way that they are conducted, which is indispensable for ensuring the credibility of the results to determine the efficacy and safety of a diagnostic, prognostic, or therapeutic product under testing.1 Furthermore, ethical conduct of clinical trials is critical to protect the rights and safety of human research subjects, which contributes to the quality of clinical trials. However, many quality risks can happen during the entire course of clinical trials, undermining the validity of the clinical trial results and threatening the rights and safety of human research subjects. These quality risks include, but are not limited to, protocol deviations, poor conduct, inadvertent errors, inadequate record keeping, unethical treatment of clinical trial participants and so on.1,2 Recognizing those quality risks in clinical trials, the regulatory agencies such as the US Food and Drug Administration and European Medicines Agency have recently published guidelines that emphasize the importance of maintaining high quality throughout clinical trials.3,4 These initiatives by the regulatory agencies have made clinical trial investigators and site managers realize the importance of a suitable risk management tool applied in a pre-emptive manner not only to improve the quality of the clinical trials, but also to avoid serious official actions such as the disqualified/restricted/assurance list for clinical investigators5 when they fail to meet the regulatory standards.

Several quality risk management tools have been proposed for industry and regulators such as basic risk management facilitation methods (ie, flowcharts and check sheets), failure mode and effects analysis (FMEA), fault tree analysis, hazard analysis and critical control points, hazard operability analysis, and preliminary hazard analysis.6 Of these tools, FMEA2 is a systematic, proactive and teamwork-based method aimed to identify and assess the causes and effects of potential failures in a system, thereby preventing them from happening beforehand.7,8 Typically, an FMEA is performed by completing the following 5 steps: selection of processes to be assessed, construction of a multidisciplinary team, collection and classification of risk scores from each process, conduct of a risk analysis, and implementation of remedial actions and reanalysis to see if those actions are effective.9 The multidisciplinary team performs brainstorming to assign a risk priority number (RPN) to each process of potential failure. RPN is the product of scores representing the 3 aspects of a risk: severity (ie, how severe is the consequence if the process fails?), occurrence (ie, how frequently can a process fail?), and detection (ie, how easily can a failure be detected if the process fails?).7

FMEA has been employed in the health care industry since 1990s, including a Health Failure Mode and Effect Analysis suggested by The Veterans Affairs National Center for Patient Safety.8 Moreover, the Joint Commission on Accreditation of Healthcare Organization recommended that FMEA be conducted for risk assessment at least once a year.9 Since then, many clinical departments in the hospital such as radiology,10 cardiology,11 dialysis unit,12 and intensive care unit8 have actively accepted FMEA as a critical quality improvement tool. For example, FMEA enabled Mercy Hospital Springfield to reduce the incidence of surgery site infections, resulting in a huge cost saving, shortened length of stay, and fewer readmissions.13 Likewise, FMEA reduced the median door-to-balloon time from 146 to 32 min in patients with ST-segment elevated myocardial infarction, which contributed to a lower risk of reinfarction and mortality.11

Although FEMA has been successfully used to increase the quality of various clinical procedures and processes in the hospital system and the utility of FEMA in mitigating risks for human subjects in clinical research was suggested,14 its application to the clinical trials system has been infrequent. Based on this understanding, the objectives of the present study were 1) to identify, assess and prioritize quality risks in clinical trials using FMEA in a tertiary care university hospital setting and 2) to evaluate the impact of remedial actions following the FMEA assessment.

Methods

Setting

Seoul National University Hospital (SNUH) Clinical Trials Center (CTC) was established in the year 1995 as the first fully-dedicated university hospital-affiliated clinical trial support unit in Korea. Since its establishment, SNUH CTC has developed 254 standard operating procedures (SOPs) at the time of preparing this manuscript that covered various activities in clinical trials in accordance with the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) – Good Clinical Practice and other international guidelines. These SOPs have been regularly reviewed and updated to correctly reflect the current best practices. SNUH CTC has also constantly upgraded the quality of clinical trials through diverse in-house online and offline education and training programs for clinical research personnel. Thanks to the state-of-the-art facility and quality operation system along with the advanced infrastructure, the number of clinical trials conducted at SNUH CTC has rapidly increased for the past 20 years. For example, SNUH CTC manages >253 active clinical trial protocols as of December 2015. Furthermore, SNUH CTC has closely worked with other research institutes and pharmaceutical companies both inside and outside of Korea to develop new medical goods. The present study was exempt from the Seoul National University Hospital Institutional Review Board review because its research activities did not constitute any human subject and it involved only the collection of existing data, documents, or records, available such that subjects could not be identified, directly or through personal identifiers linked to the subject.

Pfizer’s investigator networks, site partnerships and infrastructure for research excellence (INSPIRE) project

From November 2012 to September 2014, the Investigator Site Development Lead Group of Pfizer introduced FMEA methodology to their INSPIRE sites, one of which was SNUH CTC, to contribute to solidifying the clinical research capabilities and enhancing the quality and competitiveness in clinical trial operations of the sites. INSPIRE sites were highly interested in quality improvement, had resources and system for quality management, and were willing to participate in this FMEA project. It was anticipated that FMEA would help the sites proactively identify the most serious risks that could fail in clinical trial operations, thereby preventing them from happening in the first place. Furthermore, FMEA was expected to assist the sites to become sensible about possible concerns that were not, in reality, serious or likely enough to warrant special attention.

Baseline FMEA

To assess the baseline risk profile, an FMEA was performed from January 2013 to June 2013 by completing the following steps.

Creation of an FMEA focus group

An FMEA focus group was formed, which consisted of 6 people collectively representing the quality improvement group, clinical research coordinators (CRCs), and clinical investigators of SNUH CTC. Additionally, an investigator Site Development Lead from Pfizer also joined the SNUH FMEA focus group to guide and assist the onsite FMEA processes. The focus group was able to consult with other clinical trial experts such as Phase I unit staff to complement its expertise.

Development of process maps and identification of potential failure modes

The FMEA focus group first enumerated the overall clinical trial processes of SNUH CTC. After several rounds of internal discussion and process evaluations, the focus group selected 6 core processes that were deemed most prone to risk failure based on the previous audits and regulatory inspections at SNUH CTC. Then, the focus group identified a total of 114 potential failure modes based on brainstorming followed by iterative reviews. The 6 selected core processes (number of potential failure modes in each core process) included: obtaining informed consent (32), screening and patient identification/enrollment (16), site personnel training (8), document management, including source documents and essential documents (23), safety management such as reporting serious adverse event and suspected unexpected serious adverse reaction (25), and inspection readiness (10) (Table 1). The final list was reviewed and concurred by those who operated or were in charge of the selected clinical trial processes. Then, the FMEA focus group reviewed all those 6 core processes and 114 potential failure modes in detail to create a process map for each process as shown in Figure 1 for obtaining informed consent as an example (Table S1). Furthermore, potential causes for each failure mode were listed and the impact of the failure modes was described in the process worksheet.

Table 1

Summary of the severity, occurrence, and detection scores and RPN by six selected core processes

Core process	No of potential failure modes	Score			RPN
		Severity	Occurrence	Detection
Obtaining informed consent	32	7.11±2.34	2.71±1.85	3.48±2.05	81.24±88.89
Screening and patient identification/enrolment	16	6.42±1.98	2.39±0.95	3.10±1.40	51.48±31.46
Site personnel training	8	7.80±1.47	4.80±2.24	5.40±1.35	195.25±82.42
Document management including source documents and essential documents	23	7.40±1.27	3.72±2.16	5.09±1.65	127.29±70.48
Safety management such as reporting SAE and SUSARs	25	6.76±2.27	2.43±1.17	4.63±0.95	69.04±28.38
Inspection readiness	10	5.63±1.89	1.95±1.18	3.53±2.06	37.53±25.46
Total	114	6.92±2.08	2.88±1.81	4.16±1.83	86.07±75.12

Note: Mean ± standard deviation is displayed.

Abbreviations: RPN, risk priority number; SAE, serious adverse event; SUSAR, suspected unexpected serious adverse reaction.

Figure 1

Example of a process map for FMEA (obtaining the initial informed consent).

Abbreviation: FMEA, failure mode and effects analysis.

Scoring with RPNs

The FMEA focus group assigned a score to each identified failure mode in terms of its severity (S), occurrence (O), and detection (D). These scores ranged from 1 to 10 based on the degree of impact, frequency, and the probability of detection of each failure mode (Table 2). Any number between the 2 adjacent scores in Table 2 was assigned if the FMEA focus group assessed the score somewhere in the middle of them. The Phase I unit staff independently cross-checked the assigned S, O, and D scores for concurrence, and any differences between them and the focus group were discussed until an agreement was reached. Then, the RPN was defined as the product of the S, O, and D scores. An RPN of <40, 40–160, and >160 was considered low, moderate, and high risk, respectively.

Table 2

Rating scale of the severity occurrence, and detection scores for FMEA

Score	Description
Severity
1	No impact to patient safety
	Minor impact to data quality/integrity or protocol compliance
	Significant impact to operational feasibility
4	Minor impact to patient safety
	Significant impact to data quality/integrity or protocol compliance
	Major impact to operational feasibility
7	Significant impact to patient safety
	Major impact to data quality/integrity or protocol compliance
	Severe impact to operational feasibility
10	Major impact to patient safety
	Major impact to both data quality/integrity and protocol compliance
Occurrence
1	Remote: failure is unlikely
4	Moderate: occasional failures
7	High: repeated failures
10	Very high: failure is almost inevitable
Detection
1	Failure is detected immediately
3	Failure is detected with quality control checkpoint
5	Failure is detected with subsequent check
7	Failure is detected with random internal audit
9	Failure is detected with external audit inspection
10	Failure will not be detected at all

Abbreviation: FMEA, failure mode and effects analysis.

Follow-up FMEA

Based on the results of the baseline FMEA, the focus group developed a list of action plans to preempt high-risk (ie, RPN >160) failure modes by reducing any of the S, O, and D scores. These action plans also included further steps, responsibility, and timeline to address the identified potential high-risk failure modes that were approved by SNUH CTC Director. SNUH CTC then began a center-wide quality improvement initiative based on the action plans. To assess the effectiveness of this initiative ~1 year later (ie, September 2014), the FMEA focus group conducted a follow-up RPN scoring for the high-risk failure modes identified during the baseline FMEA.

Results

Baseline FMEA and RPN Scoring

RPNs for 114 potential failure modes identified by the FMEA focus team varied widely, ranging from 3 to 315 with a median value of 66.5 (Figure 2). Although most of the 114 potential failure modes had an RPN of low (n=28, 24.6%) or moderate risk (n=72, 63.2%), 14 failure modes (12.3%) were considered high risk (ie, RPN >160) (Figure 2). These 14 high-risk failure modes belonged to the following 3 core processes: site personnel training (n=5), document management, including source documents and essential documents (n=4), and obtaining informed consent (n=5) (Table 3 for more details). The mean RPNs for these high-risk core processes were also greater than those for other core processes (81.24–195.25 vs 37.52–69.04, Table 1). The highest score was always assessed in severity, followed by detection and occurrence (6.92 vs 4.16 vs 2.88, Table 1).

Figure 2

Histogram of the distribution of the RPNs for 114 potential failure modes (baseline FMEA).

Abbreviations: FMEA, failure mode and effects analysis; RPN, risk priority number.

Table 3

Comparison of RPNs for14 high-risk failure modes before and after the quality improvement initiative was implemented

Core process	Failure mode	S	O	D	Baseline RPN	Remedial action plan	S	O	D	Reassessed RPN	% change*
Obtaining informed consent	CRCs do not check whether the IRB approval date on the ICF is most up-to-date. Instead, they use an outdated version of subject information sheet and the ICF.	9	7	6	378	To grant CRCs an authority to get access to the clinical trials management system so that they can directly check whether the ICF is most up-to-date and IRB-approved.	7	1	2	14	−96.3
	Investigators do not ask if a clinical trial participant had any additional questions about the study before they sign the ICF at the first time. The investigator do not document issues discussed during the informed consent obtaining session.Subjects cannot or do not ask a question.	7	7	4	196	To develop a system by which clinical trial participants can consent and sign the ICF on an individual basis such as making a room available for them where they can ask any clinical trial-related question for investigators and get those answered.	7	7	4	196	0.0
	Clinical trial participants do not take sufficient time to decide whether to participate in the study before they sign the ICF.	7	7	4	196	To develop a system by which clinical trial participants can consent and sign the ICF on an individual basis such as making a room available for them where they can ask any clinical trial-related question for investigators and get those answered.	7	7	4	196	0.0
	Because CRCs use an outdated version of the ICF, clinical trial participants do not get the most up-to-date information about the study and study drug in a timely manner.	9	5	6	270	To grant CRCs an authority to get access to the clinical trials management system so that they can directly check whether the clinical protocol and study-related information is most up-to-date and IRB-approved.	9	2	2	38	−85.9
	Investigators do not ask if a clinical trial participant had any additional questions about the study before they sign the ICF at re-consenting. The investigator do not document issues discussed during the informed consent obtaining session. Subjects cannot or do not ask a question.	7	7	4	196	To develop a system by which clinical trial participants can consent and sign the ICF on an individual basis such as making a room available for them where they can ask any clinical trial-related question for investigators and get those answered.	7	7	4	196	0.0
Document management, including source documents and essential documents	Required essential documents are not filed appropriately. Filing is done without standards or a proper archiving system.	8	7	5	280	To grant CRCs an authority to get access to the clinical trials management system so that they can directly check whether the clinical protocol is most up-to-date and IRB-approved.	4	2	3	24	−91.4
	Missing, incomplete, or outdated essential documents in the SMF. Essential documents are not separately filed by their version.	7	7	5	245	To grant CRCs an authority to get access to the clinical trials management system so that they can directly check whether the clinical protocol is most up-to-date and IRB-approved.	4	2	3	24	−90.2
	Essential documents are not checked for completeness.	7	7	6	245	Investigators and CRCs to review all of the study-related documents after study completion.	4	2	3	24	−90.2
	Source documents are not reviewed or signed in a timely manner.	7	7	5	245	CRCs to notify investigators to sign off all of the study-related documents within 3 weeks after the completion of the clinical trial.	5	1	3	15	−93.9
Sitepersonneltraining	Lists of required qualification/training/experience of site personnel are missing.	7	4	7	196	To develop a clinical trial center-wide tracking system for individual history of training and education.	7	3	3	63	−67.9
	New staff do not complete the assigned training and education within the pre-specified timeline. Training for new staff are inconsistent or inadequate.	9	4	7	252	The clinical trials center director to have the QI team to double check for a new employee to complete the assigned education and training within 4 weeks of employment.	7	2	3	42	−83.3
	Incomplete or inadequate study-specific training.	9	5	5	225	To develop a new online or paper-based tracking system and log for continuous education and training.	7	2	3	42	−81.3
	The training folder in the SMF is not updated in a timely manner. Incomplete or inadequate training records are found in the SMF.	5	9	7	315	The QI team to request the education and training completion logs every month.	5	2	3	30	−90.5
	The annual training plan is not evaluated regularly for its appropriateness. Additional training plans do not incorporate the results of the evaluation.	7	7	5	245	The QI team to conduct a bi-annual status assessment of education and training.	5	4	4	80	−67.3
Mean (SD)		7.50 (1.16)	6.43 (1.40)	5.43 (1.09)	257.67 (48.60)		6.07 (1.54)	3.14 (2.21)	3.14 (0.66)	49.33 (20.33)	−90.5

Notes:

Percent change in RPN from baseline. (−) denotes reduction in RPN.

Abbreviations: CRC, clinical research coordinator; D, detection score; ICF, informed consent form; IRB, institutional review board; O, occurrence score; QI, quality improvement; RPN, risk priority number; S, severity score; SMF, site master file.

The failure modes with the 2 highest RPNs were 1) using outdated informed consent form (ICF) (RPN =378) and 2) not updating training records (RPN =315) (Table 3). The 14 high-risk failure modes (ie, RPN >160) had an S score of >7 except for one failure mode (Table 3). Furthermore, the RPN of the 14 high-risk failure modes was mainly driven by the S score (mean ± SD, 7.50±1.16), which was significantly or marginally higher than that of the D score (5.43±1.09, p=0.002, analysis of variance [ANOVA]) or the O score (6.43±1.40, p=0.0651, ANOVA) (Table 3).

Remedial actions for the high-risk failure modes

For each high-risk failure mode, a remedial action plan was devised such as granting CRCs an authority to get access to the clinical trials management system in case “CRCs use an outdated version of patient information and the informed consent form” (Table 3). Of these 14 remedial action plans, “developing a system by which clinical trial participants can consent and sign the ICF on an individual basis such as making a room available for them where they can ask any clinical trial-related question for investigators and get those answered” was thought to be able to simultaneously address 3 high-risk failure modes (Table 3). However, this remedial action was not implemented due to space limitation at the time of devising remedial action plans. Instead, it was decided to put this action plan into practice during the next round of CTC renovation and expansion, which was scheduled in the first half of 2017.

Implementing the remedial actions for the remaining 11 high-risk failure modes dramatically reduced the mean RPN from 257.67±48.60 to 49.33±20.33 (90.5% reduction from baseline, Table 3). Furthermore, 9 and 2 high-risk failure modes were assigned an RPN of low and moderate risks, respectively (Table 3). Interestingly, those decreases in RPN were mainly driven by reductions in the O and D scores (6.43±1.40 to 3.14±2.21 in O score and 5.43±1.09 to 3.14±0.66 in D score).

Discussion

We demonstrated that FMEA is useful to systematically and proactively address quality risks in clinical trials, as previously suggested that integrating FMEA techniques with research governance could reduce human risks in clinical research.14 After a baseline FMEA, we identified 114 potential failure modes in 6 core clinical trial processes, 14 of which were high risk (Tables 1 and 3). Furthermore, FMEA was effective in improving the quality of clinical trial performance. For example, we devised a set of remedial action plans based on the results of the baseline FMEA, which led to a significant decrease in RPN by 80% (Table 3). To the best of our knowledge, the present study showed for the first time that FMEA can identify vulnerable risk areas in clinical trials and contribute to quality improvements in clinical trial process, similar to those reported previously in other areas.8,10–13,15

We selected 14 high-risk failure modes with an RPN >160 and developed action plans to rectify them. Although the S score was significantly or marginally higher than the O or D scores in high-risk failure modes, the reduction in RPN after remedial actions was mainly driven by the decreases in the O and D scores (Table 3). Previous studies reported similar findings; remedial actions mostly reduced the O and D scores rather than the S score.12,16,17 Because the S score is associated with the nature of a failure mode, it is understandable that there is hardly an action plan to directly affect the S score. However, FMEA is still effective in mitigating risks as clearly shown in our results. This indicates that preemption of risk occurrence based on FMEA, along with increased detectability, can successfully address a potential failure mode, no matter how severe it can be when it happens.

Our FMEA focus team newly developed a double-checking procedure for several high-risk failure modes. This remedial action was effective to prevent occurrence or to increase the likelihood to detect it as we showed in the present study. However, the remedial action could have been more effective if it was strengthened by a formal checklist. For example, a previous FMEA for gynecologic high-dose-rate brachytherapy used a checklist to improve quality that listed a timestamp for entire brachytherapy to keep a steady and non-missing workflow.18 Checklist has been commonly used to improve the perception of teamwork and safety climate19 or to assure quality.18 Because clinical trials are a very complex process involving many parties and workforces with diverse credential and training backgrounds, team communication is of utmost importance. The World Health Organization (WHO) also developed a WHO surgical safety checklist to decrease errors and adverse events during surgery through improved teamwork and team communication. Thus, checklists would have been additionally beneficial to improve quality of clinical trial, which can be tested in further FMEA.

Scoring for failure modes is subjective in any FMEA.20 To minimize subjectivity in risk scoring, we created a multidisciplinary FMEA focus group consisting of collective expertise in various areas of clinical trials and ability to identify potential problems that could arise in the conduct of a clinical trial. Furthermore, we developed an objective scoring system to reduce bias while increasing consistency (Table 2). Likewise, the final scores for potential failure modes were independently assessed by another team for concurrence and any difference was discussed until a consensus was reached.21,22 Therefore, risk scoring in our results was of less bias and better reliability.

Meaning of RPN implicated by itself in FMEA is controversial. However, the decreased RPN supports the feasibility of the FMEA approach and its utility for aiding the adoption of technologies because scoring process before and after remedial actions in our results is consistent as the same FMEA focus group scored RPN for high-risk failure modes. Therefore, the decreased RPN by remedial actions offers probability that FMEA can be applied to understand processes and probably also for identifying risks worth addressing in clinical trial center as a useful quality control tool.

Limitations

The present study had 3 major limitations. First, only 6 core processes were selected for FMEA in this study, which could not represent the whole aspect of a clinical trial. However, quality in 1 area of clinical trials is closely linked with that in other areas given the complex and concerted nature of the clinical trial. Those 6 core processes were deemed most risky based on the previous audits and regulatory inspections that SNUH CTC had undergone. Therefore, the ability and utility of FMEA to improve the quality in clinical trials, particularly for areas most prone to risk failure as we showed in the present study, can be readily applied to other less risky areas. Second, our results may not be directly applicable to smaller clinical trial centers or private investigational sites where resources for FMEA are likely to be limited. Theoretically, individual clinical scientists can perform an FMEA due to its simple nature,14 but we do not see this as a reality. Based on our experience, it is assumed that at least 1 full-time quality person is required to prepare, initiate, manage, analyze, and follow-up an FMEA project. In fact, considering the resource required to perform an FMEA, it is recommended to conduct it sequentially around several important SOPs every year or so even for large health care organizations.9 Finally, like other FMEAs, our analysis could be subjective, and depended on the experience and expertise of the FMEA focus group. Therefore, future studies are warranted to quantitatively assess the effect of the remedial actions, particularly in clinical trial settings.

Conclusion

FMEA is a powerful tool to improve quality in clinical trials by proactively and systematically identifying potential quality risks. SNUH CTC is expanding its FMEA capability to other core clinical trial processes.