Identifying risk characteristics using failure mode and effect analysis for risk management in online magnetic resonance-guided adaptive radiation therapy.

Background and purpose: Online magnetic resonance-guided adaptive radiotherapy (MRgART) is a new technology of radiotherapy and requires a new quality control program in many aspects. This study aimed to gain a deeper understanding of risks in online MRgART through the application of failure mode and effect analysis (FMEA) for more enhanced and effective quality assurance (QA) programs. Materials and methods: We present an FMEA conducted by a multidisciplinary team with more than two years of experience. A process map describing the whole process of online MRgART was developed and potential failure modes were identified. High-risk failure modes and their potential causes and corrective measures were also identified. Failure modes were classified into three categories, MRgRT, online ART, and conventional RT, to investigate their features. A comparison with previous studies was also conducted to gain a general perspective.
Results: In total, 153 failure modes and 49 high risks were identified. Among all failure modes, 51, 63, and 66 were related to MRgRT, online ART, and conventional RT, respectively. The hazardous processes were structure segmentation, treatment planning, and treatment beam delivery. Lists of failure modes identified in this study and previous studies were presented. Based on the results, characteristics and general aspects of the risks were discussed.
Conclusion: Exploring the results of the FMEA enhanced our understanding of risk characteristics to improve QA program of online MRgART.

Introduction

Magnetic resonance-guided radiation therapy (MRgRT) systems that enable online adaptive radiation therapy (online ART) have recently become commercially available and are widely used [1], [2]. In this system, the intrafractional motion of the tumor and organs can also be managed by acquiring MR cine images during treatment. By this technology, we can monitor the motion of the tumor and risk organs and use the anatomical information as a surrogate for gating [3], [4], [5], [6].

Online MRgART is mainly based on two new innovations, “online ART” and “MRgRT.” The entire treatment process, from treatment preparation to completion of all treatment fractions, has changed from the conventional process where a treatment plan is created before treatment based on the image acquired by computed tomography (CT) and use the plan for multiple fractions until the end of treatment or until the plan needs to be changed. Radiotherapy following the conventional process is referred to as “conventional RT” in this paper, in contrast to online MRgART. This new treatment introduces new types of risks and may also change the prioritization of risks known in conventional RT; therefore, a new quality assurance (QA) program should be developed [3], [4], [5], [7], [8].

MRgRT involves new types of risks, such as risks associated with the integration of MR imagers into radiotherapy equipment, risks caused by the fact that the “primary” planning image is an MR image rather than a CT image, and tumor monitoring during treatment to ensure accurate beam delivery to the tumor. The “primary” image here means the image used to reference anatomical information in planning, including re-planning in online ART. The risks involved in online ART are mainly due to daily treatment preparation: image acquisition, re-contouring, reoptimization, plan checking, and patient-specific QA ought to be completed while the patient is in the treatment position before beam delivery. Additionally, some steps in pretreatment planning, which are conducted a few days before the treatment starts, need to be conducted considering the online ART to be performed later. Moreover, there would be risks due to the design of hardware and software of online MRgART systems and supporting tools; therefore, it is important to comprehensively identify inherent critical risks and build an effective and efficient QA program that fits online MRgART procedure.

One of the tools to establish a QA program is failure mode and effect analysis (FMEA) with process mapping, a widely accepted method in high reliability industries; the American Association of Physics in Medicine (AAPM) Task Group (TG) 100 recommends the use of this approach in establishing a quality management program [9]. Process map is a visual description of a workflow and helps identify the risks and weakness in the process. Collaborating with many radiotherapy professionals is recommended to comprehensively involve all the processes and enhance our understanding of the overall process flow. FMEA is a risk-based prospective approach for safety that identifies failure modes, evaluates their risks, and identifies potential causes and corrective measures. FMEA can be a tool to establish a quality management program for any new treatment method in an individual clinic. Since risks are quantitatively evaluated, the established quality management program is effective and efficient. Accomplishing FMEA by a multidisciplinary team brings them not only a comprehensive quality management program with a wider perspective but also a deeper and common understanding of the treatment and higher consciousness of safety. The applications and necessities of FMEA have been reported for radiation therapy [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], MRI-based treatment planning [26], and online ART [7], [27].

The aim of this study was to gain a deeper understanding of risks in online MRgART through FMEA for more enhanced and effective QA programs. An FMEA for online MRgART was performed by experienced multidisciplinary team and the results were analyzed to understand risk characteristics. Comparison with previous studies were also made to gain general perspective.

Methods

An application of FMEA

For better understand the risks in the MRgART process, here we present an dedicated example of FMEA conducted by a multidisciplinary team with more than two years of experience at the National Cancer Center Hospital (NCCH) Japan.

The NCCH began online MRgART with the MRIdian system (ViewRay Inc., Cleveland, OH, USA) in March 2018. As of July 2020, 237 patients have been treated, and 52 % of them have undergone at least one online ART during treatment. QA program for online MRgART was originated with the vendor’s advice and experience from conventional RT. This program has been gradually developed by adding corrective measures for the errors and near-miss events occurring in the actual practice. More detailed information on MRgART at the NCCH as well as checklists for pretreatment planning and on-line ART planning are described in Supplementary Material A.

Process map and FMEA-collecting failure modes

We formed a multidisciplinary team comprising two radiation oncologists, five medical physicists, five radiation technologists, and two medical physics students familiar with online ART. Process maps describing workflows for the entire treatment of MRgRT and for the day of the treatment of online ART were created. Based on these process maps, each member of the team identified failure modes through brainstorming and subprocesses to which the failure modes belong.

Process map and FMEA-analysis

Three attributes scored each failure mode: the likelihood of occurrence (O), severity if the failure occurs (S), and likelihood of the failure being lack of detectability (D), each ranging from 1 to 10. The larger the value, the higher the risk, and then the risk priority number (RPN), RPN = O S D, was determined to evaluate the risk of each failure mode. Before starting the scoring, we initially held a meeting to discuss and assign scores based on Table II in AAPM TG 100 [9] to 20 randomly selected failure modes, to avoid large inter-observer errors in O, S, and D values due to subjectivity and to standardize values among members. We then made reference tables for scoring based on the results and observations from the meeting (Supplementary Table S2a, b, and c). Each member individually assigned scores for each of the three factors for each failure mode, referring to Supplementary Table S2 and Table II in AAPM TG 100. The collected scores for each failure mode were averaged and assigned as the final scores of O, S, and D. For significant differences > 2.5 between some scores given by the members and median values, the final value was determined through discussion. The RPN value was obtained by multiplying the final values of each factor.

Based on its definition, the RPN value is a risk indicator, and the larger the value, the higher the risk. Additionally, S itself is an important variable as it corresponds to the severity of the failure mode when it occurs; therefore, we classified failure modes that ranked in the top 20% of RPN values or severity of 8 or higher as high-risk failure modes.

For high-risk failure modes, possible causes and potential corrective measures were identified by the members, and then meetings were held to review them. Possible causes and corrective measures were classified into six and seven categories, respectively.

Categorizing failure modes into different aspects of online MRgART

Failure modes were classified into three categories to investigate their features in online MRgART; each category represented different aspects of online MRgART: failure modes related to online ART, MRgRT, and those typical in conventional RT. A single-failure mode could belong to more than one category. For example, the failure mode “planning policies and treatment instructions are not properly documented,” was related to “conventional RT” and “online ART,” because the information is always necessary for later reference in radiation therapy and necessary for online ART planning.

Comparison with previous studies

Identified failure modes and high-risk failure modes were compared with two published papers, by Noel et al. [7] and Klüter et al. [27], dealing with FMEA for online ART. A comparison was made by sorting failure modes of each study into categories and subcategories representing a generalized process of online MRgART.

Results

Fig. 1 shows process maps describing the workflows for the entire treatment of MRgRT and for the daily treatment of online ART. Based on the process maps, 153 failure modes were identified (Supplementary Table S5), with RPN scores ranging from 64 to 535. Of those, 51 failure modes were related to MRgRT, 63 were related to online ART and 66 were also typical in conventional RT (Table 1). Forty-nine high-risk failure modes were identified, 31 of which were selected due to RPN, 28 due to S, and ten due to both RPN and S (Table 1). Numbers of failure modes and high-risk failure modes belonging to each subprocess are presented in Table 2. Some failure modes could occur in multiple subprocess; in that case, the failure mode was counted as one in each subprocess. High-risk failure modes are listed in Supplementary Table S3. Possible causes and corrective measures for high-risk failure modes are summarized in Fig. 2.

Fig. 1

Process maps of whole MRgRT (upper) and daily online ART (lower). The contents in the orange dotted line in the process maps of whole MRgRT describe the steps in the non-online ART workflow, which is replaced with the daily online ART process if online ART is performed. Blue star, yellow star, and the combined star represent a high-risk failure mode with an RPN score ranked in the top 20%, a high-risk failure mode with S 8, and their combination, respectively. Abbreviations: TPS, treatment planning system; ROI, region of interest; PACS, picture archiving and communication system; RIS, radiotherapy information system; HIS, hospital information system; MF, magnetic field; ED, electron density; Rx, prescription; SI superior-inferior; DVH, dose volume histogram.

Table 1

Number of high-risk (with RPN top 20% or S 8) and very high-risk failure modes (with RPN top 5 % or S 9) classified to each type of feature of online MRgART.

	Conventional RT	Online ART	MR-guided RT	Total
All	66	63	51	153
High risk	29	23	9	49
RPN top 20%	20	18	4	31
S≥ 8	19	5	7	28
Very high risk	8	5	5	14
RPN top 5%	4	5	1	8
S≥ 9	5	0	4	7

Table 2

Number of failure modes, failure modes with RPN in the top 20%, failure modes with S 8, and high-risk failure modes that belong to each subprocess.

Subprocesses	FM	RPN top20%	S≥8	High risk FM
1. Treatment prescription	6	1	2	2
2. Immobilization	4	0	0	0
3. CT simulation	7	1	1	1
4. Transfer images	2	0	0	0
5. Import images	5	0	3	3
6. MR simulation	19	1	3	3
7. Structure segmentation	30	7	5	8
8. Treatment planning	29	9	6	12
9. Plan approval	8	4	5	5
10. Plan check	13	1	2	3
11. Patient-specific QA	4	0	0	0
12. Plan check	9	1	2	3
13. Initial treatment	11	5	4	8
14. Chart rounds	1	1	0	1
15. Subsequent treatments	11	5	4	8
A1. MR imaging & positioning	15	2	3	5
A2. Structure segmentation	28	9	3	12
A3. Predict dose	12	4	2	6
A4. On-line ART plan check	9	3	2	5
A5. On-line ART reoptimize	22	10	2	12
A6. On-line ART pre-treatment QA	16	4	2	6
A7. MR cine imaging	11	3	2	5
A8. Treatment beam delivery	18	6	7	12
A9. On-line ART post-treatment QA	10	1	3	4

FM, failure mode; CT, computed tomography; MR, magnetic resonance; ART adaptive radiotherapy; QA, quality assurance.

Fig. 2

Distribution of potential causes (a) and corrective measures (b) for high-risk failure modes.

Numbers of failure modes and high-risk failure modes identified in this study and previous reportswere classified into category and subcategory representing generalized process of MRgART, and are listed in Table 3 and Supplementary Table S4 respectively. There were some differences among studies. Previous studies conducted FMEA prior to online ART implementation and did not include risk mitigation processes. Conversely, our analysis was conducted after two years of experience and included risk mitigation procedures used in actual practice. The method used to distinguish high-risk failure modes in Klüter et al.’s report was different from that of other reports. In the report by Noel et al., individual failure modes were not described but only a summary was given; therefore, each item deciphered from the summary was counted as one but may include more than one failure mode. Our sorting for failure modes identified by other authors may differ from their intentions because online ARTs dealt with other reports have different systems and workflows from ours. A full list of failure modes identified is also shown in Supplementary Table S5.

Table 3

Number of failure modes and high-risk failure modes classified into each category.

	Category	Number of FMs			Number of high-risk FMs
		NCCH	Noel et al.	Kluter et al.	NCCH	Noel et al.	Kluter et al.
Pretreatment preperation	Planning/treatment directives	3	11	3	3	4	2
	Immobilization	4	–	–	–	–	–
	CT simulation	8	–	–	1	–	–
	MR simulation	11	–	4	1	–	2
	Image data transfer and registration	9	–	–	3	–	–
	Structure segmentation(TP anatomy)	16	–	2	4	–	2
	Treatment planning	25	–	–	13	–	–
	Plan approval	7	–	2	3	–	2
	Patient-specific QA	4	–	–	–	–	–
	Real-time tracking preparation	6	–	–	–	–	–
Online ART	Initiation of online ART	1	3	9	1	1	7
	Positioning and imaging	8	7	18	1	–	4
	Image data transfer and registration	3	5	3	–	1	–
	Structure segmentation(TP anatomy)	9	8	13	3	5	10
	Check dose from original plan	1	–	3	–	–	3
	Reoptimize	15	13	13	5	6	10
	Plan approval	4	2	2	1	2	2
	Pretreatment QA	5	–	5	1	–	3
	Treatment beam delivery	5	2	8	4	1	7
	Real-time tracking	6	–	2	1	–	2
	Post-treatment QA	1	–	–	–	–	–
	Others (ART)	3	–	–	1	–	–
	Device malfunctions	10	–	1	7	–	1
Others	Others (general)	5	1	1	–	–	1

Discussion

High-risk subprocesses in the whole MRgRT process were import images, structure segmentation, treatment planning, plan approval, and initial and subsequent treatment, since those subprocess involved a relatively large number of high-risk failure modes (8 or more), or more than half of the identified failure modes were high-risk (Table 2). This result was the same as that in FMEA for IMRT performed in AAPM TG 100 [9]. High-risk subprocesses in daily online ART were structure segmentation, plan check, reoptimization, and treatment beam delivery (Table 2). We observed that in pretreatment preparation and daily online ART, the subprocesses involving a relatively large number of high-risk failure modes (8 or more) were the same: structure segmentation and treatment planning (reoptimization in daily online ART). The treatment beam delivery process (initial and subsequent treatments in whole treatment process) was also risky, regardless of whether the treatment was daily online or non-online ART.

The characteristics of the risks associated with the categories of online MRgART features (online ART, MRgRT, and conventional RT) can be explained in Table 1. The feature related to conventional RT was the most hazardous in terms of numbers of high- and very high-risk failure modes, followed by features related to online ART. Even though MRgRT had the lowest number of high-risk failure modes, it involved some of the most severe failure modes: the two most severe failure modes with S = 10 were related to MRgRT, and the number of failure modes with S 9 was as high as the other categories. Conversely, online ART did not involve failure modes with S  9. This is because even if a failure occurs in daily online ART, the plan is rechecked and recreated in the next day of treatment, which causes less propagation of the failure; therefore, there were no failure modes with a high severity. The detailed individual features responsible for the high-risk failure modes categorized as online ART and MRgRT were identified and summarized in Supplementary Material C and D.

Possible causes for high-risk failure modes were classified and are illustrated in Fig. 2(a). The most common cause was human failure, which is reasonable based on the results of FMEA for IMRT on TG 100 [9] and experience in clinical practice. The second common cause was hardware/software failure. Contrary to human failure, hardware/software failure was not very common in TG100. Hardware/software failure was classified into two types: due to the design or malfunction. The relatively high incidence of hardware/software failure was attributed to the recently developed treatment technology. The third common cause for high-risk failure modes was inadequate communication. Since the plan is recreated and rechecked daily in online ART by the staff who may be different from that in charge of the pretreatment plan, sharing clear and detailed planning directives is important along with the communication in conventional treatments.

Potential corrective measures for high-risk failure modes were classified and are illustrated in Fig. 2(b). Multiple measures were individually identified for almost all high-risk failure modes, most of them typical of those that could be identified in conventional RT, such as IMRT. However, selecting or constructing a measure for the actual clinical practice of online MRgART from those potential measures requires further consideration based on their effectiveness and limitations due to time and effort. The most common potential corrective measure was a stricter confirmation. This is the simplest and most straightforward method to deal with the problem; however, especially for online ART, adding check items should be carefully done due to time constraints. The second most common corrective measure was improvement of hardware/software and automated operations, which may exclude errors more effectively and efficiently than other measures. This is discussed in more detail in the next paragraph. The third most common corrective measure was education. Since online MRgART is a new technology and part of the treatment procedure differs from the conventional one, the knowledge of conventional treatment alone is insufficient and requires adequate education before treatment implementation. Performing FMEA would be an effective way of education.

The second most common corrective measure, improvement of hardware/software and automated operations, were identified for approximately half of the high-risk failure modes and most of them were improvement of the treatment and planning devices themselves to make them less prone to failure. While users and dedicated software should deal with specific and specialized failure modes, vendors of the treatment system should deal with more fundamental and common failure modes, to improve the system’s resistance and tolerance to failures. In Supplementary Material E, the necessary elements for hardware/software design and functions to reduce risks, revealed from identified corrective measures, are listed. They were roughly categorized into three: improve error-prone functions, reduce failures through technological innovation, and consistently manage treatment through integration with other software. As evident in online ART, high-precision treatment techniques require more time, effort and human resources for quality control, hindering the possibility of advanced radiotherapy for more patients. Therefore, innovations in safety management technology for the treatment devices themselves will gain importance.

The comparison with previous studies revealed a unique aspect of our report, which is inclusion of pretreatment preparation process in FMEA. Noel et al. did not include them in their FMEA, and Klüter et al. did not seem to distinguish them from failure modes in online ART, except for the four identified in the process before the first treatment. However, 100 failure modes in pretreatment process were identified in this study, and the number of high-risk failure modes in the pretreatment process was comparable to those in the daily online ART process. Severities of high-risk failure modes in the pretreatment process tended to be slightly higher than those in the daily online ART process (Fig. 3). This is because planning directives and some parameters, including dose constraints, dose prescription, sequences for automatic segmentation, and planning parameters, are transferred from the original plan to online ART reoptimization as initial settings that are unchanged unless necessary or as unchangeable parameters; those failure modes could be propagated to the subsequent treatments. We believe that these characteristics are not only unique in our system but also true for other systems; some of the settings in the original plan should be carried over into the adapted plan to shorten the preparation time before the beam delivery of online ART. Meanwhile, in online ART process, all reports consistently identified a relatively large number of high-risk failure modes in structure segmentation, reoptimize, plan approval, and treatment beam delivery processes. The planning/treatment directive process also contained a large number of high-risk failure modes (Table 3). This result was consistent with that in an FMEA for IMRT performed in AAPM TG 100 [9]. The comparison also revealed the limitation of an FMEA, i.e., the difficulty of identifying complete failure modes. About half of the undetected subcategories (Supplementary Table S4) by NCCH were those that we could not find, while the other half were not relevant to our system. Therefore, failure modes identified by other groups should be considered when conducting FMEA. Supplementary Tables S4 and S5 are a good reference for this purpose.

Fig. 3

Box plot of RPN and severity of failure modes in the pretreatment preparation and online ART processes.

Failure modes and FMEA results conducted in this study are specific to the online MRgART at the NCCH. However, some failure modes can aid other facilities for performing FMEA. Failure modes identified in this study along with those identified in previous studies [7], [27] are summarized in Supplementary Material F. The risk level for each failure mode is also site-specific, specifically O depends on how the treatment is prepared and delivered and D on the QA program. However, differences in S among facilities would not significant in principle. Failure modes with a particularly high severity in these FMEA results will also be informative for QA programs in other facilities.

In conclusion, we have revealed risk characteristics of online MRgART by exploring the results of FMEA. We investigated hazardous processes, risk characteristics associated with three different features of MRgART: MRgRT, online ART and conventional RT. Potential causes, corrective measures and importance of innovations in safety management technology for the treatment devices were also discussed. Failure modes were compared with other studies and obtained general understanding of the risks. All analysis in Discussion was devoted to understand risk characteristics and general aspects rather than focusing on individual failure modes, which are novelties of this report.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.