The RABBIT risk-based approach to clinical implementation of new technology: SRS as a case study.

Radiation oncology technology continues to evolve rapidly, resulting in advanced versions frequently being brought to market. Before a new product is used standard tests are carried out to reduce the risks associated with failure of the equipment to comply with well-established technical specifications. It is much harder to identify and reduce the risks associated with how the new technology is used clinically, such as those related to poor communication and high workload. To ensure that new technology and techniques are used safely and appropriately the implementation project should be managed by a multidisciplinary team (MDT) made up of representatives from all the relevant professions. The MDT's role is to agree on the project scope, identify and rank all risks and benefits, and direct resources towards mitigating the highest risks. Before clinical release there should be consensus from the MDT that the benefits of the new technology outweigh the residual risks. The introduction of initiatives to optimise current practice may involve major changes which can be met with barriers such as limited support from management, insufficient time for MDT meetings, and staff fearful of being shown to have poor practices. To help overcome these challenges our team at St George Hospital Cancer Care Centre has developed a Risk and Benefit Balance Impact Template (RABBIT), which guides an MDT through the rapid implementation and safe use of new technology and techniques with an easy to follow Microsoft Word document. The implementation of stereotactic radiosurgery is used as a case study to illustrate the RABBIT methodology. The RABBIT is a user-friendly method for a busy radiotherapy clinic to transition to a risk-based MDT approach for the implementation of new technologies and techniques. When staff from all disciplines feel empowered to raise concerns about risks the workplace become inherently safer for patients and staff alike. Crown

Introduction

All branches of medicine use advanced technology to some extent, but few are entirely reliant on it to the degree that radiation oncology is. The techniques used to improve the dose distribution within the patient have changed enormously, particularly in imaging, dynamic dose delivery and the management of patient motion/changed patient anatomy [1], [2], [3], [4].

The desire for improved clinical outcomes results in the rapid uptake of new radiotherapy products, often before their efficacy is proven [5], [6]. With new technology there is little data on the clinical risks and benefits, so early adopters have to estimate these with a high degree of uncertainty. Once the financial expenditure has been made there is necessarily a pressure to use the new technology so as not to have wasted the investment.

The traditional implementation process is a serial one (Fig. 1): a decision is made to purchase new equipment based on product marketing and reports of experience from other centres; physicists commission the new technology and make sure it meets the required technical specifications; radiation therapists (RTs) are trained how to use it and develop protocols for its use, at which point it is put into clinical use.

Fig. 1

The traditional process for implementing new radiation oncology technology.

A key limitation of the traditional implementation process is that it may only consider if the equipment is functioning within tolerances, and may not include a formal assessment of whether its use is safe or appropriate for the local clinical environment with its unique combination of staffing levels, expertise, patient cohorts and level of institutional support [6], [7], [8]. The potential for patient harm is further increased if there is no formal periodic review process including multidisciplinary consultation once the equipment is in use, or a robust incident learning system [9]. Our experience shows that the traditional approach can also result in departmental resources being wasted through the use of the new technology for patients who will not benefit from it, and operational inefficiency and workforce stress due to lack of training, support or required infrastructure.

Failure modes and effects analysis (FMEA) is a useful tool for comprehensively analysing the risks associated with each process in the clinical workflow for a cancer patient, including imaging and diagnosis, CT simulation, treatment planning and treatment delivery [8]. The key steps in an FMEA are to create a process map, evaluate what failure modes are possible for each process, and rank each failure mode on the probability of its occurrence, the severity of its impact and its detectability [10]. A recent literature review of FMEA in healthcare [11] showed that 30 of the selected 153 publications were in radiation oncology, compared to none or one for most areas of healthcare, and 22 of the 30 reports had been published since 2014. This demonstrates that there is growing recognition within the radiation oncology community of the value of formal risk management tools.

An FMEA of intensity modulated radiotherapy carried out by AAPM Task Group 100 [8] shows that only 11% of risks were classified as being related to failures of software or hardware; the other 89% were related to non-equipment failures, such as human failures (35%), lack of standardized procedures (15%), inadequate training (15%), inadequate communication (10%) and lack of staff (5%). The percentage values will vary from clinic to clinic, but it is likely that this is a common trend, which is at odds with the typically high allocation of resources to prevent equipment failures and the low allocation for the reduction of the non-equipment failures [8].

In order to direct resources more appropriately the implementation project should be managed by a multidisciplinary team (MDT) with representatives from all the relevant staff groups, in particular radiation oncologists, radiation therapists (RTs), physicists and nursing staff, where appropriate. Fig. 2 shows that a key part of the process is a decision point where the MDT reaches consensus on whether the benefits of clinical use of the new technology or technique outweigh the residual risks. Formally including this step reduces the chance of the technology being used inappropriately.

Fig. 2

The risk/benefit-based process for a multidisciplinary team to implement new technology and techniques.

In an ideal world radiation oncology departments would carry out a well-informed FMEA prior to the introduction of all new technologies and techniques, but in our experience many staff members find this process too daunting, too time consuming and relatively inaccurate considering that they have no prior local experience or incident learning data to base their risk ratings on. Published FMEAs from other centres are a good resource but the reported risk ratings may not be relevant to other clinics. If published results are used without being modified for the local environment there could be a high number of understated or overstated risks or benefits, potentially resulting in inefficient or unsafe clinical use of a new technology.

The RABBIT

In order to provide a solution that would be readily adopted by all staff members at our centre, in 2015 we developed a simple risk-based project management system for releasing new technologies and techniques, called the Risk and Benefit Balance Impact Template (RABBIT) [12]. The main aims of introducing the RABBIT were to facilitate the rapid implementation and the safe and appropriate use of new radiation oncology technologies and techniques; to have an easy and effective review process of the new technologies and techniques after a period of clinical use; and to improve the teamwork and safety culture within our clinic.

The RABBIT template is a Microsoft Word document which means there are no extra costs or training required for users who already have Microsoft Office, and the RABBIT file can be easily customised to match local requirements.

To be effective the RABBIT process should be carried out by an MDT with representatives from all the main staffing groups who will be involved with the new technique. Team members are trained in the general principles of risk management and in the use of the RABBIT. Every time the MDT members meet they update the risk/benefit analysis and the list of outstanding and completed action items, and then save a new version with the date in the file name. This document serves as a record of progress and separate meeting minutes may therefore not be required. When the MDT reaches consensus that the benefits outweigh the risks and that the technology should be released for clinical use, the completed RABBIT template is signed by a representative of each of the professions on the MDT. The final document fulfils the functions of both a project report (with links to supporting documentation) and a clinical release note.

The RABBIT has four steps to guide the multidisciplinary team members through the process described previously [12]:

Step 1: Scope definition

Team members learn about the new technology or technique from the literature or other users; they agree on which patients are likely to benefit most and how the success of the project will be measured; they identify the resources required, the timeline of the major milestones, and any proposed restrictions of use (eg limited patient cohort).

Step 2: Project preparation

Equipment commissioning is commenced and there is a review of compliance of the project with local, national and international requirements. In parallel to this all staffing groups commence training and writing standard operating procedures for clinical use. End-to-end testing is carried out, with each step being performed by the staff members who will do this task as part of the real clinical workflow.

Step 3: Risk-benefit review

Throughout the commissioning period the MDT members meet periodically and use their newly gained knowledge and experience to identify and reassess the associated risks and benefits and rate these. The rating system is flexible and should suit the local users: at its most complex it can be a quantitative risk priority number as used in FMEA through to the simplest method of using the terms Low, Medium and High. As the project progresses the risks which have been mitigated can be downgraded, for example after equipment testing, staff training, the use of additional safety measures or by limiting the project scope. The team then focusses on the remaining significant risks and any new ones that have been identified. The time spent on each implementation task should be proportional to the rating of the risk that would exist if the task was not carried out. The analysis should not be confined to clinical care issues, and should consider financial, workforce, legal and reputational risks and benefits.

Step 4: Multi-disciplinary team decision

Once the commissioning tasks have been completed, staff have been trained and the standard operating procedures and other documentation are finalised, the MDT meets to decide whether the benefits of the new technology outweigh the risks as it will be used in their centre. If yes, it can be released for clinical use. If not, then more risk mitigation actions must be identified or further restrictions placed on its use. After the initial use of the new technology or technique the MDT meets again to review how well the measures of success have been achieved and identify any changes to the risk/benefit balance.

The RABBIT was developed as an aid for bringing new technology into clinical use but it has also proven to be a valuable tool for reviewing the success of the project in the months or years after clinical release. An incident learning system provides valuable data to the review team, but in our experience the RABBIT review meetings capture additional information about the unreported near misses, frustrations, inefficiencies and unanticipated benefits that users of the technology have experienced. The scheduling of these reviews should be dependent on the level of risk. It can be time based, being more frequent after clinical release then becoming less frequent (e.g. starting monthly, then going to yearly), or it can be based on number of patients (e.g. review after the first 10 patients, then again after the first 100 patients).

The RABBIT can be used for large projects (e.g. installation of a new linear accelerator) and for small projects (e.g. a minor change in a treatment technique or a new item of dosimetry equipment). Large projects require three or four MDT meetings of around one hour per meeting, with approximately the same time required for the project lead to pre-fill the RABBIT template and provide links to documentation. The smallest projects require approximately one hour to pre-fill the RABBIT template and two 30 minute MDT meetings, the first one to establish the scope and required actions, and the last one to sign off on the clinical release.

In our clinic 48 RABBITs were completed between 2015 and 2020, and all but one of these resulted in successful clinical implementation of a new technique or technology. In one case clinical implementation did not proceed because after completing the RABBIT the MDT group agreed that the benefits did not outweigh the risks and so the project was terminated.

In 2017 a survey was carried out of the radiation oncologists, RTs and physicists who had most experience of using the RABBIT system [12]. The survey questions were taken from the AAPM Safety Profile Assessment tool which is a free on-line tool for evaluating the safety culture of a radiotherapy department [13], [14]. The results showed that there had been a perceived improvement in the process of implementing new technology since the introduction of the RABBIT, and in how well the clinical staff from different disciplines work together as a team.

Case study: Stereotactic radiosurgery

The RABBIT was used for introducing stereotactic radiosurgery (SRS) for treating multiple brain metastases in our clinic, and the document at the time of clinical release is shown in Appendix A. An MDT was formed comprised of physicists, radiation therapists and radiation oncologists. Only two members of the MDT had prior SRS experience, so the other group members had to base their risk assessment on their experience of similar treatments (eg stereotactic body radiotherapy), reviewing the literature and visiting other centres already using the same equipment. There are several publications on the application of FMEA to SRS [4], [15], [16], [17], [18], [19] which are an excellent resource for groups preparing to implement this.

The SRS MDT identified the risk and benefits and rated them as Low, Medium or High. This rating system was taken from the NSW Health Risk Matrix [20] which is used by our local health district. This matrix gives clear examples for each likelihood rating (rare through to almost certain) and consequence rating (minimal through to catastrophic), which the MDT found helpful to reach consensus on the rating of each risk.

The MDT identified 11 clinical risks, one workforce risk, one financial risk and one relating to community expectations. At the time of clinical release four risks were rated to be medium: suboptimal image registration, suboptimal GTV, PTV or OAR contours, inaccurate patient position, and staff stress due to tight timelines, lack of support or lack of training. The remaining seven risks were rated to be low.

Two clinical benefits were identified, being lower brain toxicity for SRS patients compared to whole brain radiotherapy, and fewer patients being referred to other clinics for SRS treatment. There were also the benefits of meeting community expectations to offer an SRS service, and of providing an opportunity for professional development of staff.

Post-implementation MDT meetings were held to review the metrics associated with each risk, such as how many plans failed to meet the dose objectives or how many patients had local recurrence or unexpected side effects. No risks were upgraded or added to the RABBIT and it can be concluded that SRS implementation at our clinic has been successful and safe.

Discussion

Introducing a risk-based approach to a busy clinic which operates on the traditional implementation model is not an easy task. Barriers [21], [22] include staff

being defensive when acknowledging the risks associated with their processes

having conflicts of interest associated with either keeping or replacing the old technology

having a fear of displaying their ignorance about new practices

having a fear of being shown to have poor current practices

lacking the confidence or time to learn about risk analysis

not having time to attend the MDT meetings.

In the traditional implementation model there is an implicit reward system for senior staff who give a project the green light too early, because they are praised by management for releasing the technology ahead of schedule. The junior staff who have to deal with the consequences of the premature release generally have no forum to voice their concerns, and resulting adverse events may be under-reported. Staff requesting more time before clinical release can be labelled obstructive or offensive if they have highlighted risks associated with processes managed by other staff members, potentially resulting in demarcation disputes and interpersonal conflict. Conversely a risk-averse management team may unduly delay the introduction of beneficial technology due to “paralysis by analysis” [23].

While the identification of risks can be a potentially negative task, the RABBIT encourages a positive experience by focusing on solutions. Many MDT members become enthusiastic about trying to identify all the associated risks and contributing factors, and sharing ideas on risk reduction strategies. For example one risk description could be “Patient harm due to correct radiation dose being delivered to the wrong location” for which staff might identify contributing factors such as the use of the wrong planning images, incorrect contouring, imaging equipment faults, patient movement, changes in patient anatomy, or user error; another risk could be “Patient harm due to wrong dose delivered to the correct location” with contributing factors including errors in the treatment planning system, beam delivery system faults, errors in the dose calibration system or user error. Each set of contributing factors involves more than one process and more than one professional group.

This method avoids the blame game of focussing solely on the staff practices around the highest risk processes, and instead fosters a team approach where the staff from different professional groups willingly identify contributing factors that they are aware of from their own work practices once they see that the other groups are doing the same. Members of an implementation team who have participated in previous risk-based projects become progressively less defensive and more pro-active in identifying risks associated with their own practices.

All members of the MDT are empowered to voice their concerns prior to the clinical release of the technology in question, regardless of their seniority or their relative traditional decision making position. The MDT sign-off ensures that the technique or technology cannot be used clinically if one professional group believes that the benefits do not outweigh the risks. In our experience the radiation oncologists appreciate having a better understanding of the risks because they are ultimately responsible for the medical interventions that they have prescribed. It is also reassuring for the group to have the radiation oncologist participate in the final sign off and share the responsibility for the decision to go clinical.

Where there is an acute lack of resources or a pressure to release technology with limited time, one strategy is to use one of the many published FMEAs for the same or similar technology as starting point to guide the RABBIT risk analysis, tailoring the risk ratings to the local clinic’s resources and requirements. One MDT member can be assigned to pre-fill the RABBIT so the first meeting is spent fine-tuning the document instead of filling in a blank template. This saves time for the other team members, and listing risks that have already been published may be less confronting than basing them on local experience.

It would be highly ineffective to spend equal time mitigating all risks, both large and small, so the team should focus on the bigger risks. However there is a benefit to initially documenting every risk (however small) that has been flagged by staff members, because this acknowledges their concerns as being valid. Staff who have been listened to at this point feel empowered to raise safety concerns in the future and encourage others to do so, thereby improving the general safety culture. Listing the small risks does not necessarily slow the project down because there are usually good controls already in place, and it can be documented that these risks are accepted with no further action required. If one of the risks initially identified as being small turns out to be a major problem the implementation team has evidence to demonstrate that they were aware of the risk and ranked it based on their professional judgement with the reasoning recorded in the RABBIT, which is clearly a more responsible approach than not documenting it in the first place.

The chance of successfully transitioning to a risk-based implementation system is highest if there are one or more dedicated champions willing to convince the other staff members of the benefits. Explaining that most risks are associated with non-equipment failures helps them to understand that even if the equipment is functioning correctly there are many ways a patient can be harmed once the technique is released for clinical use. Acceptance of the benefits of the RABBIT in our department occurred gradually when the staff observed that a standard implementation process resulted in a track record of safe and efficient introduction of clinical improvements.

On-line quality improvement tools and publications on FMEAs and RABBIT-style risk assessments can help radiotherapy centres around the world provide better outcomes for their patients. The AAPM SPA survey [14] and the IAEA’s free on-line course Safety and Quality in Radiotherapy [24] are both excellent resources for educating staff on the need for a unified approach to improving the safety culture.

Conclusion

When introducing new technology or new techniques it is recommended that a multidisciplinary team completes a formal risk assessment, and focusses their resources on mitigating the highest risks. Moving from an implementation approach which is focussed on physics equipment commissioning to an MDT risk-based one which listens and responds to the concerns of all relevant staffing groups can be a challenging process.

FMEA and the RABBIT are both useful tools but different in their approach. FMEA is quantitative and most effective when there is clinical experience to guide the risk ranking. The RABBIT is qualitative and appropriate to use at the beginning of a project where there is little or no clinical experience of the new technology or technique. The RABBIT template guides the MDT through the four steps of scope definition, project preparation, risk–benefit review and the final decision on whether to release the technology for clinical use or not.

In this paper we have illustrated the value of the RABBIT for the rapid and effective clinical introduction of intracranial SRS for multiple metastases in our centre. During the implementation process we referred to publications from other centres on their FMEA for SRS, but we found that the most value came from bringing together a wide range of our staff members to discuss the local challenges and encouraging them to propose methods of overcoming these. At the post-implementation RABBIT review meetings staff members were comfortable raising concerns that would likely not have been reported through our incident learning system or other quality management avenues.

In summary the use of the RABBIT at our clinic has helped us achieve our aims of implementing new technology and techniques rapidly and safely, and improving team work and the safety culture. Regardless of the implementation system used, any forum which encourages the open discussion of risks in a sensitive and constructive manner will result in benefits for patients and staff alike.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.