A prioritization framework for the analysis of near misses in radiation oncology.

INTRODUCTION: The term near miss implies the aversion of a harm event but often there is a lack of evidence when establishing a link between a failure in process and potential harm. The focus of this study was to use reported incident data to inform a prioritization framework for the triage of near miss events in a radiation therapy program.
MATERIALS AND METHODS: Actual and near miss events during the study period were categorized using thematic analysis based on incident types. Near miss were characterized based upon their potential to result in harm to the patient using the concepts of failure modes and Analytic Hierarchy Process (AHP) theory. Near miss events were assessed for occurrence, detection and the potential impact and then assigned a summative normalized score reflecting prioritization recommendations, the normalized 10 point score (NTPS).
RESULTS: 107 events were reported within the study timeframe. 65% of event type categories (n = 20) were attributed to near misses. 107 total events we analyzed using the framework with a maximum NTPS of 4 achieved across all event types. Of the 47 actual events 100% received a NTPS of 3 or greater. Of the 60 near miss invents 47% received an NTPS less than or equal to 1. Finally 15% of near miss events received a NTPS of 3 or greater.
CONCLUSIONS: Near miss events provide a unique opportunity for learning however, can yield a great deal of data potentially limiting the resources for effective incident learning. A FMEA and AHP based prioritization framework for the triage of near miss events, including the likelihood of occurrence, probability of the event to go undetected and the potential impact if the incident did occur, allows for the optimal focus of programmatic resources in the analysis of these events.

Introduction

Failure modes effect analysis (FMEA) is a prospective methodology to assess the robustness of a system. The methodology maps the process and identifies steps at which potential failures can occur [1]. These failure modes are characterized by several features including; what is the likelihood that this failure would occur, if it did occur what would be the potential impact and the probability that the failure would go undetected [2]. Each characterization is assigned a measure which collectively provides insight into the processes most vulnerable to failure and opportunities for improvement [3]. To evaluate multiple criteria described in this methodology a systematic process to describe the contribution of each element is necessary. Analytic Hierarchy Process (AHP) theory, described by Thomas L. Saaty is a methodology to develop priority scales based on expert input [4]. AHP is a tool used to break down the measures that contribute to a decision into a number of easily comprehended sub-criteria, using pairwise comparison of measured or subjective inputs [5].

Incident learning is a retrospective process to building a robust system. Reporting and recognizing the opportunities for programmatic improvement through the iterative learning from adverse events and near misses is key to a successful incident learning program [5]. A near miss, as defined by Kaplan, is a condition in which “the potential for harm may have been present, but unwanted consequences were prevented because some recovery action was taken” [6]. Variations on this definition exist within all major national and international incident reporting systems (European Radiation Oncology Safety Education Information System: ROSEIS, American Radiation Oncology Incident Learning System: RO-ILS and Canadian National System for Incident Reporting in Radiation Therapy: NSIR-RT). In 1931 Heinrich was the first author to correlate the pathways of near misses and actual incidents [7]. The term near miss implies the prevention of a harm event but often there is a lack of evidence establishing a link between a failure in process and potential harm. The understanding that mitigating the cause of near misses will in turn prevent actual events, in the health care domain, is based primarily on anecdotal evidence [8]. In addition, near-miss events are much more common than adverse events, evidenced in a magnitude of 7–100 times more frequent [9]. Therefore, near miss reporting can yield a great deal of data, potentially limiting programmatic resources available for incident learning. To effectively analyze the data in a manner to inform the priorities of the program a methodology is required to identify near miss events, which if mitigated may in turn prevent potential harm in the future.

Near miss events share similar characteristics with those used in the evaluation of FMEA. The likelihood that the failure would occur, if it did occur what would be the potential impact and what are the odds that the failure would go undetected can be used to describe a near miss event. The focus of this paper is to describe an incident prioritization framework for the triage of radiation therapy incidents, leveraging the concepts of FMEA and AHP. By evaluating the framework through its application of a radiation therapy program’s reported incident dataset. The prioritization framework will identify events of greatest risk for the program to focus resources for analysis and mitigation.

Methods

Analytic Hierarchy Process (AHP) theory was used to characterize the failure modes (criteria) and sub-criteria which contribute the FMEA decision making process, the likelihood that this failure would occur, if it did occur what would be the potential impact and the probability that the failure would go undetected. Table 1. Describes the sub-criteria used to characterize each failure mode and their related contributing factors.

Table 1

Criteria and sub-criteria for analytic hierarchy process prioritization.

Criteria	Failure May Occur	Potential Impact	Failure Would Go Undetected
Sub-Criteria 1	0–15% of program incidents	No harm patient is asymptomatic and no treatment is required.	Event was detected because it was undeliverable
Sub-Criteria 2	16–30% of program incidents	Mild symptoms, if present, are mild; no or minimal intervention (observation, investigation, minor treatment) is required; harm or loss of function is minimal or intermediate but short term.	Event was detected at 1st QA
Sub-Criteria 3	31–45% of program incidents	Moderate patient is symptomatic, requiring intervention (additional treatment or operative procedure) or a prolonged hospital stay; long-term or permanent harm or loss of function.	Event was detected at pre-treatment QA
Sub-Criteria 4	46–60% of program incidents	Severe patient is symptomatic, requiring life-saving intervention or a major surgical/medical intervention; shortened life expectancy or major long-term or permanent harm or loss of function.	Event was detected at on-treatment QA
Sub-Criteria 5	61–75% of program incidents	Death on the balance of probabilities, death was caused or brought forward in the short term by the incident.	Treatment delivery
Sub-Criteria 6	76–90% of program incidents		Event was detected at sporadic QA
Sub-Criteria 7	91–100% of program incidents

When describing an event, the likelihood that this failure would occur was described as the percentage of program incidents. The potential impact was described using the acute harm scale as defined by the NSIR-RT minimum dataset. [10]. The probability that the failure would go undetected was described using the points throughout the planning-treatment process at which an event could be detected.

Expert opinion was used to describe the relative importance of each criteria with respect to one another. For example the potential impact was identified as six times more important than probability of failure and four times more important than probability of detection. The same process was completed for each set of criteria and set of sub-criteria described in Appendix I. Using Eigen vector and Eigen value calculation, described in Appendix II, relative weights for each criteria and sub-criteria were derived [4].

Reported incident events were reviewed during a one-year period from a large metropolitan cancer program. Actual and near miss events during the study period were categorized using thematic analysis based on incident types. Near miss events were extracted for further review and were characterized based upon their potential to result in harm to the patient using the concepts of failure modes. Near miss events were assessed for their likelihood of occurrence, probability to go undetected and the potential impact if the incident did occur. Likelihood of occurrence was assessed using descriptive statistics of near miss event types during the study period. To assess the probability of the failure going undetected near miss events were characterized based on the barrier step at which the event was detected. Finally, the potential impact if the incident did occur was described through a comparison of actual events extracted for the study period. Near miss events that shared incident type classifications with actual incidents were assigned the same acute toxicity outcomes with respect to the potential for impact to the patient.

Based on the criteria above each incident was assigned a percentage of program incidents, acute harm scale value and a point along the planning-treatment process at which an event could be detected. Applying the relative weights for each sub-criteria as determined by the AHP analysis a score for each criteria (likelihood of occurrence, probability to go undetected and potential impact) was derived. The summative normalized score were used to develop prioritization recommendations with respect to the incident classification types best suited for further investigation.

Results

Incidence

107 events were reported within the study timeframe of which 60 near miss events were identified. A summary of the near miss and actual incident event thematic analysis classification is shown in Fig. 1.. 65% of event type categories were attributed to near misses. The most common near miss event was “referencing” referring to referencing to tattoos in documentation (i.e. setup point is 0.5 cm superior to tattoo). The most common actual incident “bolus omitted” referred to the omission of the accessory during treatment.

Fig. 1

Thematic analysis: Near miss and actual event classification.

Detection

The barrier step at which the event was detected was described using six sub-criteria. The number of near miss events stratified by barrier step is described in Table 2. 20 events were deemed undeliverable and therefore could not result in patient harm. The remaining 40 events were detected along the planning-treatment trajectory. No near miss events were identified via sporadic Quality Assurance (QA), an event not part of a defined QA process.

Table 2

Near miss events by barrier step.

Barrier Step	# of Events
Undeliverable	20
1st QA	9
Pre-Treatment QA	16
On-Treatment QA	11
Treatment Delivery	4
Sporadic QA	0

Potential impact

Of the 60 near miss events identified 29 were identified as having a no potential impact with respect to acute medical harm, 30 were identified as the potential to cause mild harm and one was identified as having the potential for moderate harm.

Analytic Hierarchy process analysis

All 107 event types, near miss and actual incidents were evaluated using the AHP tool. The normalized 10 point score (NTPS) represents the score given to an incident relative to the worst case scenario (the most common incident, with a potential impact of death and caught via an undefined sporadic QA process). Table 3 describes the events with the highest and lowest NTPS. These represent event types with the greatest and least programmatic risk. Under the event type category “Wrong Setup” which is one of the least common event type an NTPS scored of 4 was assigned. The near miss was not detected until treatment delivery and similar “Wrong Setup” treatment incidents have resulted in moderate patient harm. Conversely, the event type category “Mislabeled” was assigned the score of 4. This event type was the second most common, however, has not been associated with patient harm and was identified within the first QA barrier.

Table 3

Event classification by criteria and normalized 10 point score.

Event #	Event Type	Failure May Occur	Potential Impact	Failure Would Go Undetected	Normalized 10 Point Score
Actual	Fractionation Issues	0–15%	Mild	Sporadic QA	4
Actual	Accessory Issue	0–15%	Mild	Sporadic QA	4
Actual	Protocol Not Followed	0–15%	Mild	Sporadic QA	4
Actual	Tx Technique Not Completed	0–15%	Mild	Sporadic QA	4
Near Miss	Wrong Setup	0–15%	Moderate	Treatment Delivery	4
Near Miss	F/S Error	0–15%	Mild	1st QA	1
Near Miss	Referencing	46–60%	Mild	Undeliverable	1
Near Miss	Mislabeled	16–30%	No Harm	Pre-Treatment	1
Near Miss	Accessory	0–15%	No Harm	Pre-Treatment	1
Near Miss	Patient Related Circumstances	0–15%	Mild	Undeliverable	1
Near Miss	Mislabeled	16–30%	No Harm	1st QA	1
Near Miss	Mislabeled	16–30%	No Harm	Undeliverable	1
Near Miss	Incorrect Iso Slice Marked	0–15%	No Harm	Undeliverable	1
Near Miss	Sites Omitted	0–15%	No Harm	Undeliverable	1
Near Miss	Accessory	0–15%	No Harm	Undeliverable	1
Near Miss	Undeliverable	0–15%	No Harm	Undeliverable	1
Near Miss	Tattoo Cannot be Found	0–15%	No Harm	Undeliverable	1
Near Miss	Reference Image Omitted	0–15%	No Harm	Undeliverable	1
Near Miss	F/S Error	0–15%	Mild	1st QA	1
Near Miss	Referencing	46–60%	Mild	Undeliverable	1
Near Miss	Mislabeled	16–30%	No Harm	Pre-Treatment	1
Near Miss	Accessory	0–15%	No Harm	Pre-Treatment	1

Analytic hierarchy process analysis

107 total events we analyzed using the framework of which a maximum normalized 10 point score of 4 was assigned. Of the 47 actual events 100% received a normalized 10 point score of 3 or greater. Of the 60 near miss invents 47% received an NTPS less than or equal to 1. Finally 15% of near miss events received a NTPS of 3 or greater.

Discussion

Van der Schaff et al. (1991) sought to distinguish the goals of near miss learning from traditional incident analysis. Identifying three goals he described a qualitative, quantitative and intangible aspect to the reporting. Qualitatively near miss reporting provides insight into how errors develop and might potentially lead to adverse events. Quantitatively near miss reporting allows a program to build a database of root causes of near misses. Trending this database provides a way to target the most prominent factors as possible targets for error-reduction. Finally, near miss reporting serves as a reminder that safety risks continue to exist and the continued need for staff vigilance [9]. It is evident that near miss events provide a unique opportunity for learning and should be leveraged for programmatic gain. Lam et al. examined radiation therapy near miss and actual incidents based on incident type and stage of origin. They found near miss events shared different characteristics than actuals concluding the traditional practice of analyzing and managing these events in a similar manner is not the optimal approach to managing risk in radiotherapy. Furthermore, the authors noted near miss events reflected different failure modes than actual incidents and utilizing different approaches for analysis could glean valuable information for learning. [11]. Another study conducted by Bates et al. examined near miss and actual events in the medication setting. It was evidenced that the characteristics of near misses were different than those of actual events. Although both incident types had the same underlying cause, near misses involving a modest overdose were more likely to result in harm than errors involving massive overdoses, since the near miss actions were more likely to be carried out [12]. Wright et al. reviewed 1351 reported events within a radiation oncology program, classifying the events as either workflow or near miss events. In addition, the authors assigned a Risk Priority Number (RPN), numeric assessment of risk assigned to a process as part of Failure Modes and Effects Analysis (FMEA) to near miss events. It was found that events that had originated and were detected in the treatment delivery stage had the greatest mean overall RPN and were therefore associated with the greatest risk [13]. Although near miss events provide a unique opportunity for learning how these events should be used and best analyzed continues to be explored. In the study conducted by Wright et al. of the 1351 reported events 51 (3.8%) were categorized as proper near miss events, the remainder being work flow related events [13]. As previously mentioned near misses have a frequency of 7 to 100 times the incidence of actual incidents [9]. To fulfill the goals described by Van der Schaff et al., near miss events should be captured in the incident learning system (available for analysis and learning). How the near misses are used to support program decision regarding how and where to invest quality and safety improvements requires consideration. Near miss events should be linked to the causal continuum, providing an association with the event and evidenced potential for harm. The application of the framework described in this paper provides a mechanism to objectively prioritize events including near misses with an evidenced potential for harm, risk of incidence and risk of occurrence.

Conclusion

Near miss events provide a unique opportunity for learning and should be leveraged for programmatic gain. Near miss reporting, however, can yield a great deal of data potentially limiting the resources available to effectively analyze the data in a manner to inform programmatic priorities. Failure modes effect analysis (FMEA) identifies steps at which potential failures can occur. Analytic Hierarchy Process (AHP) theory synthesizes multiple objective and subjective inputs to develop evaluation scales. Together these tools create a prioritization framework capable of integrating the number of factors associated with incident reporting. As described this framework objectively evaluates events using the criteria of occurrence, probability of detection and potential impact to assign events a relative priority. This priority can be further evaluated by programs to assess the allocation of resources for incident learning and mitigation resources.

Declaration of Competing Interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Item	Failure may occur	Potential Impact	Failure would go undetected
Failure may occur	1	0.166666667	0.25
Potential Impact	6	1	2
Failure would go undetected	4	0.5	1
SUM	11	1.666666667	3.25

Item	Failure may occur	Potential Impact	Failure would go undetected	Weight
Failure may occur	0.090909091	0.1	0.076923077	8.93%
Potential Impact	0.545454545	0.6	0.615384615	58.69%
Failure would go undetected	0.363636364	0.3	0.307692308	32.38%

Item	0–15%	16–30%	31–45%	46–60%	61–75%	76–90%	91–100%
0–15%	1	0.25	0.2	0.166667	0.142857	0.125	0.111111
16–30%	4	1	0.25	0.2	0.166667	0.142857	0.125
31–45%	5	4	1	0.25	0.2	0.166667	0.142857
46–60%	6	5	4	1	0.25	0.2	0.166667
61–75%	7	6	5	4	1	0.25	0.2
76–90%	8	7	6	5	4	1	0.25
91–100%	9	8	7	6	5	4	1
SUM	40	31.25	23.45	16.61667	10.75952	5.884524	1.995635

Item	0–15%	16–30%	31–45%	46–60%	61–75%	76–90%	91–100%	Weight
0–15%	0.025	0.01	0.00853	0.01003	0.01327	0.02124	0.05568	2.03%
16–30%	0.1	0.03	0.01066	0.01203	0.01549	0.02428	0.06263	3.67%
31–45%	0.125	0.12	0.04264	0.01505	0.01859	0.02832	0.07159	6.13%
46–60%	0.15	0.16	0.17057	0.06018	0.02324	0.03399	0.08351	9.74%
61–75%	0.175	0.19	0.2132	0.24072	0.09294	0.04248	0.10022	15.09%
76–90%	0.2	0.22	0.25586	0.30090	0.37176	0.16993	0.12527	23.54%
91–100%	0.225	0.25	0.29850	0.36108	0.46470	0.67975	0.50109	39.80%

Item	No Harm	Mild	Moderate	Severe	Death
No Harm	1	0.25	0.2	0.166667	0.142857
Mild	4	1	0.25	0.2	0.166667
Moderate	5	4	1	0.25	0.2
Severe	6	5	4	1	0.25
Death	7	6	5	4	1
SUM	23	16.25	10.45	5.616667	1.759524

Item	No Harm	Mild	Moderate	Severe	Death	Weight
No Harm	0.043478	0.015385	0.019139	0.029674	0.081191	3.78%
Mild	0.173913	0.061538	0.023923	0.035608	0.094723	7.79%
Moderate	0.217391	0.246154	0.095694	0.04451	0.113667	14.35%
Severe	0.26087	0.307692	0.382775	0.178042	0.142084	25.43%
Death	0.304348	0.369231	0.478469	0.712166	0.568336	48.65%

Item	Undeliverable	1st QA	Pre-Treatment QA	On-Treatment QA	Treatment Delivery	Sporadic QA
Undeliverable	1	0.25	0.2	0.166667	0.2	0.166667
1st QA	4	1	0.25	0.2	0.166667	0.142857
Pre-Treatment QA	5	4	1	0.25	0.333333	0.25
On-Treatment QA	6	5	4	1	0.5	0.333333
Treatment Delivery	5	6	3	2	1	0.5
Sporadic QA	6	7	4	3	2	1
SUM	27	23.25	12.45	6.616667	4.2	2.392857

Item	Undeliverable	1st QA	Pre-Treatment	On-Treatment QA	Treatment Delivery	Sporadic QA	Weight
Undeliverable	0.03703	0.01075	0.01606	0.02519	0.04762	0.06965	3.44%
1st QA	0.14815	0.043011	0.02008	0.03022	0.03968	0.05970	5.68%
Pre-Treatment	0.18519	0.172043	0.08032	0.03778	0.07937	0.10448	11%
On-Treatment QA	0.22222	0.21505	0.32129	0.15113	0.11904	0.1393	19.5%
Treatment Delivery	0.18519	0.25807	0.24096	0.30227	0.2381	0.20896	23.9%
Sporadic QA	0.22222	0.30108	0.32129	0.4534	0.4762	0.4179	36.5%

Sum of columns Normalize and calculate first normalized principal Elgen vector x1	N=1/sumcol1a/sumcol2b/sumcol31/a/sumcol11/sumcol2c/sumcol31/b/sumcol11/c/sumcol21/sumcol3,x1=∑nrow1n∑nrow2n∑nrow3n
Calculate the largest Elgen value λ	λ=sumCol1∗x1+sumCol2∗x2+sumCol3∗x3
Calculate Consistency Index	cl=λ-nn-1
Verify Consistency Ratio <10%	CR=CIRI

n	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
RI	0	0	0.58	0.9	1.12	1.24	1.32	1.41	1.45	1.49	1.51	1.54	1.56	1.57	1.59