Use of the AAPM Safety Profile Assessment Tool to Evaluate the Change in Safety Culture After Implementing the RABBIT Prospective Risk Management System.

PURPOSE: Hospitals traditionally focus on reactive risk management such as incident reporting, but prospective risk management systems such as failure modes and effects analysis are also important tools to reduce risks and improve the safety culture. In 2015, the St George Cancer Care Centre (STGCCC) developed a multidisciplinary risk-based system for the safe and effective implementation of new technologies and techniques, using risk and benefit balance impact templates (RABBIT) developed in-house. The purpose of this study was to determine whether risk management and the safety culture in radiation oncology were perceived to have improved since the introduction of the RABBIT system. METHODS AND MATERIALS: In 2017, radiation oncologists, radiation therapists, and medical physicists were asked to rate the department before and after the introduction of the RABBIT using questions from the American Association of Physicists in Medicine Safety Profile Assessment (SPA) tool. Answers relating to the implementation of new technology/techniques are presented.
RESULTS: STGCCC staff confirmed that the RABBIT system has improved the implementation of new technology/techniques, with an average SPA question score improvement from 3.9 to 4.4 (of 5.0). This compares favorably with the SPA world average of 3.5 (October 2017). The improvement is attributed to risks being formally identified and managed and adequate staff training being mandatory and systematic. There were also perceived improvements in teamwork, probably because the introduction of structured multidisciplinary teams resulted in each group having a better understanding of the workflows and priorities of the other groups.
CONCLUSIONS: This study shows that prospective risk management at STGCCC has improved the perceived quality of the implementation of new technology/techniques. The RABBIT is a simple and effective method for achieving this improvement in safety culture. The American Association of Physicists in Medicine SPA is a valuable tool for assessing the success of quality initiatives and identifying opportunities for further improvement.

New medical technology is often clinically implemented without a formal risk assessment. We developed a multidisciplinary risk-based system using in-house risk and benefit balance impact templates. These templates incorporate the best practice principles outlined by Task Group 100 and many other evidence-based quality management recommendations. The American Association of Physicists in Medicine Safety Profile Assessment tool was used to survey radiation oncologists, radiation therapy technologists, and physicists, and the results showed that use of the risk and benefit balance impact templates improved the quality of the implementation process.

Introduction

Radiation oncology technology continues to evolve rapidly both in complexity and scope. Early adoption of new technology is generally beneficial for patients with cancer, but it also entails significant risks. Conversely, slow uptake of technology avoids introducing new risks, but it may result in continued suboptimal clinical outcomes. Even though state-of-the-art equipment may be capable of more accurate delivery of radiation therapy, if the environment in which it is used is suboptimal (eg, low staffing levels, inadequate staff training, or poor safety culture), there is potential for serious harm. Marks et al showed that the treatment deviation rate decreased slowly over time after the introduction of a new technology, which they attributed to the learning curve of staff using the new technology.

The relentless pace of change places a high burden on the radiation oncology staff members who are tasked with commissioning and releasing new technology and techniques. They often have insufficient time and knowledge to use best-practice project management methods, which results in the technology or technique being released for clinical use when the implementation team reaches a certain ill-defined comfort level, possibly as a result of undue pressure from other parties. Ideally, an implementation project will include the prospective identification and management of associated risks. Quantitative failure modes and effects analysis (FMEA), as described by the American Association of Physicists in Medicine (AAPM) Task Group 100, is an excellent method for assessing the risks of current clinical practice. However, FMEA may be challenging if project team members have no experience with the new technology or technique being implemented, have little support from senior management, and are not given sufficient time to perform the analysis.

In 2015, to help streamline and improve the implementation process, the St George Cancer Care Centre (STGCCC) developed a multidisciplinary risk-based system called risk and benefit balance impact templates (RABBIT) for the safe and effective implementation of new technology and techniques. Prior to the use of the RABBIT, there was no formalized prospective risk analysis as part of the implementation of new medical technology or techniques at STGCCC. There was also no formal way to terminate or pause a project, resulting in the potential for rushed clinical implementation. Conversely, sometimes there was delayed adoption of beneficial technology because of risk-averse staff attitudes. The pre-RABBIT implementation of new technology tended to be a serial process, in which one professional group did its part before handing over to the next group, with people working in silos.

From 2015 to 2018, STGCCC had 35 implementation projects that were managed by multidisciplinary teams (MDTs) using RABBITs, identifying more than 200 risks in total. For all but 2 of these projects, the residual risks were outweighed by the benefits, with the projects cleared for clinical use. The 2 projects for which this was not the case were aborted. Any risks that still scored highly at the time of clinical release of the approved projects were actively managed by reviewing the RABBIT for that project on a periodic basis after implementation to ensure that effective corrective actions were taken. For each of these risks, a staff member was identified on the RABBIT as the owner of that risk and assumed responsibility for this follow-up.

The implementation projects ranged from large equipment, such as a linear accelerator, through medium-sized projects, such as the introduction of the deep-inspiration breath-hold technique, down to small projects, such as the urgent commissioning of a new palliative treatment technique for a single patient.

The purpose of this study was to test the hypothesis that the safety culture and risk management in radiation oncology were perceived to have improved 2 years after the introduction of the RABBIT system. To do this, radiation oncologists, radiation therapists, and medical physicists were surveyed in 2017 using a limited number of questions from the online AAPM Safety Profile Assessment (SPA) tool and were asked to retrospectively rate the department before and after the introduction of the RABBIT. This study reports only on the SPA questions relatied to the implementation of new technology and techniques.

Rahn et al also used the SPA tool to assess safety culture as part of a program to introduce a new incident learning system in a radiation oncology department. In contrast with this study, they scored their department on all 92 questions; however, they did not survey multiple staff members and did not report on any change in SPA scores as a result of their new incident learning system.

Methods and materials

RABBIT implementation tool

The RABBIT for the implementation of new radiation therapy technologies and techniques is a project management system that includes a risk/benefit analysis and a list of outstanding and completed action items. The RABBIT is a Microsoft Word template created using Developer Mode to provide dropdown menus and tool tips. Because it is Word file, no extra costs or training are required for users who already have Microsoft Office, and the files can be easily customized to match local policies. The completed and signed document fulfills the functions of both a project report (with links to supporting documentation) and a clinical release note.

Each RABBIT has 4 sections, as shown in Figure 1: project scope definition, project preparation level, risk and benefits balance review, and an MDT decision on the project status. The template guides an MDT through the 4 steps by presenting dropdown options at each step. The RABBIT improves efficiency and safety by allowing the team to justify the reduction of resources used to manage low-rated risks and to increase resources for high-rated risks. Approval and sign off by a representative of each profession on the final RABBIT document adds to legal evidence of due diligence in risk management and prompts the multidisciplinary implementation team to notify senior management of significant risks, in accordance with local policies.

Figure 1

Risk and benefit balance impact template methodology for implementing new medical technologies and techniques.

To illustrate the benefits of the RABBIT methodology, an example is provided of the clinical implementation of a deep-inspiration breath-hold technique for reducing the dose to the heart during breast irradiation. Unlike a complete FMEA process, which quantifies the risks associated with every step in the complete process map, the RABBIT system can be used to consider only the new risks that are introduced by the change in technique or technology compared with the current method. In this case, the main new risks (additional to those associated with free-breathing breast treatment) were identified to be longer treatment times, patient anxiety, and the possibility of geometric miss because of inadequate training of radiation therapists or patients' inability to stay in the breath-hold position. Risk mitigation strategies included the introduction of intrafraction portal imaging and patient coaching to increase breath-hold compliance and screen out those who were not suitable candidates for the deep-inspiration breath-hold technique. Subsequently, the MDT reached a consensus that the clinical benefits of reducing the heart dose outweighed the mitigated risks, and the technique was released for clinical use.

AAPM SPA tool

SPA is an open-access online tool with 92 questions on key aspects of safety and quality in radiation therapy departments. It was created by AAPM's Working Group on the Prevention of Errors. Although the AAPM is a physicist organization, the survey covers the performance of the entire department and was designed with significant input from colleagues from relevant organizations such as the American Society for Radiation Oncology. Ideally the survey would be completed by an MDT to get an accurate and balanced picture of the current level of safety and quality. After completing the survey, users can see how their results compare with the average of all other participants. The survey can be repeated on a regular basis to show any improvements.

A subset of 10 SPA questions (Table 1) was chosen because these questions were the most relevant for assessing the safety culture related to implementation of new technologies and techniques. To be included in the survey, staff must have had experience with the RABBIT and have been working at STGCCC before the introduction of the RABBIT. The developers of the RABBIT system, including the authors of this study, were excluded from the survey. Five medical physicists, 5 radiation therapists, and 3 radiation oncologists were asked to complete the survey, and there was a 100% response rate. The survey responses were anonymous. Limitations of this study include the small sample size and the retrospective nature of the survey, which could increase uncertainty as a result of recall bias.

Table 1

AAPM SPA questions used in the local survey, world average scores (October 2017), local pre- and post-RABBIT scores, and change due to RABBIT

SPA #	1 = Never/strongly disagree2 = Rarely/disagree3 = Sometimes/neutral4 = Most of time/agree5 = Always/strongly agree	World average (October 2017)	Local pre-RABBIT average	Local post-RABBIT average	Change due to RABBIT	P-value
10	The clinical staff from different disciplines work well together as a team.	4	3.5	3.9	+0.4	.012
26a	Pre-clinical commissioning is performed for the following systems: Treatment planning systems.	4	4.4	4.8	+0.4	.008
26b	Pre-clinical commissioning is performed for the following systems: Treatment delivery systems.	4	4.8	4.9	+0.1	.165
26c	Pre-clinical commissioning is performed for the following systems: Treatment management system.	4	4.5	4.7	+0.2	.082
34	When implementing major new equipment or process changes, staff are provided with sufficient training to ensure competency prior to clinical implementation.	3	3.6	4.2	+0.6	.003
36	Commissioning results are documented prior to clinical implementation.	4	3.9	4.4	+0.5	.008
37	Treatment processes are documented prior to clinical implementation.	3	3.5	4.1	+0.6	.003
38	An independent review of commissioning results is performed prior to implementation of new clinical systems and processes.	3	3.2	3.8	+0.6	.002
39	Potential risks associated with the introduction of new clinical systems and processes are assessed prior to implementation.	3	3.5	4.6	+1.1	<.001
41	An end-to-end test is performed prior to implementation of new or upgraded clinical systems and processes.	3	3.9	4.6	+0.7	<.001
	Average scores across all 10 questions:	3.5	3.9	4.4	+0.5

Abbreviations: AAPM = American Association of Physicists in Medicine; RABBIT = Risk and Benefit Balance Impact Template; SPA = Safety Profile Assessment.

Each question in the SPA tool is in the form of a statement with 6 possible responses: always/strongly agree, most of the time/agree, sometimes/neutral, rarely/disagree, never/strongly disagree, and don't know/not applicable. The first 5 options are assigned numerical values of 5, 4, 3, 2, and 1, respectively. The don't know/not-applicable answer was not assigned a number and was not included in the numerical analysis. The SPA website provides the average result for all institutions worldwide that have participated in the SPA survey to date. The worldwide average values for the 10 questions used in this project were obtained at the time the STGCCC survey was conducted and were compared with the STGCCC results.

Results

The results of the STGCCC responses to the SPA questions relating to the implementation of new techniques and technologies are shown in Figure 2. The arrow indicates the average result for all institutions that have participated in the SPA survey. The shift in the STGCCC responses to the right (higher numbers) showed that staff members believe that the introduction of the RABBIT system has improved risk management. Both pre-RABBIT and post-RABBIT results are better than the SPA average for all questions, other than teamwork.

Figure 2

Pre- and post-risk and benefit balance impact template survey results for the 10 Safety Profile Assessment questions used in the survey. The arrow indicates the average result for all institutions that have participated in the survey as of October 2017.

The significance of the improvement from the pre-RABBIT scores to the post-RABBIT scores was investigated by calculating the P-value using a paired sample t test with a null hypothesis that there was no change in scores before and after the use of the RABBIT. The P-value for 7 questions was < .05. Of the other 3questions, 2 (26b and 26c) already scored highly before the RABBIT, so no great improvement was possible. For the teamwork question (question 10), the pre-RABBIT result was lower than the SPA average, but the post-RABBIT response nearly matched the SPA average, indicating a perceived improvement, although this had low statistical significance (P = .012).

Discussion

The implementation of new techniques and technology in a safe and efficient manner is an ongoing challenge in the high-tech area of radiation oncology. Utilization of an implementation management tool such as the RABBIT, combined with outcome measures from the AAPM SPA tool, provides a simple and cost-effective framework for project work and ongoing quality improvement.

The perception of the STGCCC staff surveyed was that, before the introduction of the RABBIT, some areas in the department only sometimes met the SPA criteria. The lowest scoring question was “An independent review of commissioning results is performed prior to implementation of new clinical systems and processes.” The survey was unable to identify whether this is because independent reviews did not take place or because most of the respondents were unaware that they did take place.

All responses for the post-RABBIT environment showed improvement. The largest improvement (1.1 of 5) was for the question “Potential risks associated with the introduction of new clinical systems and processes are assessed prior to implementation,” which is not surprising given that the RABBIT requires the implementation team to list and rate associated risks, something that was not previously done in a formal manner. The smallest improvement (0.1) was for the question “Pre-clinical commissioning is performed for the following systems: Treatment delivery systems,” but because the average score for this response increased from 4.8 to 4.9 (maximum possible score 5.0), an improvement of >0.2 would not have been possible.

The impact of RABBITs on the safety-critical performance of a department will depend on the existing performance relative to optimal and world-average levels (ie, there may be limited measurable benefit to implementing RABBITs for sites already at optimal levels). This study has shown significant improvement for a site that performs at levels close to those of the world average, and the RABBIT would likely also be helpful for departments with suboptimal levels.

The improvement in the quality of the implementation of new technologies and techniques with the RABBIT system at STGCCC is attributed to 2 major changes. First, risks are formally identified and managed before the implementation of a clinical release. Second, adequate staff training is now mandatory and systematic.

There was a small increase in the average score of the perceived quality of teamwork, which is likely due to the introduction of formalized multidisciplinary teams, resulting in each professional group having a better understanding of the priorities and work of the other groups. However, the post-RABBIT score is still slightly lower than the SPA average, indicating that further work is warranted on improving teamwork at STGCCC.

Conclusions

This study has shown that prospective risk management improves the perceived safety of implementing new radiation oncology technologies and techniques. The RABBIT has proven to be a simple and efficient method for achieving this. The AAPM SPA is an effective tool to assess the success of quality initiatives, such as the RABBIT, and to identify opportunities for further quality improvement. This study was retrospective and therefore may have been subject to recall bias. Future surveys on the change in safety culture at other clinics planning to use the RABBIT will be prospective.