Quality management in radiation therapy: A 15 year review of incident reporting in two integrated cancer centres.

Fifteen years of reported incidents were reviewed to provide insight into the effectiveness of an Incident Learning System (ISL). The actual error rate over the 15 years was 1.3 reported errors per 1000 treatment attendances. Incidents were reviewed using a regression model. The average number of incidents per year and the number of incidents per thousand attendances declined over time. Two seven-year periods were considered for analysis and the average for the first period (2005-2011) was 6 reported incidents per 1000 attendances compared to 2 incidents for the later period (2012-2018), p < 0.05. SAC 1 and SAC 2 errors have reduced over time and the reduction could be attributed to the quality assurance aspect of IGRT where the incident is identified prior to treatment delivery rather than after, reducing the severity of any potential incidents. The reasoning behind overall reduction in incident reporting over time is unclear but may be associated to quality and technology initiatives, issues with the ISL itself or a change in the staff reporting culture. Crown

Introduction

Radiation Oncology (RO) treatment planning and delivery is continually evolving and developing. Over the past ten year’s new technologies, such as image guided radiation therapy (IGRT), automatic planning and, volumetric arc therapy (VMAT), have led the way for highly conformal treatment planning and delivery, that was not previously achievable. These technologies have been adopted quickly, as they are promoted with the promise of increased efficiencies, and the potential to treat more patients [1]. Although new technologies are promoted as being more efficient, they also bring far more complexity to RO processes, consequently resulting in the need to develop quality assurance (QA) processes that keep up to date with the technology, and minimise the risk to patients [2]. The Institute of Medicine (IOM) report “To Err is Human: building a Safer Health System” proposed the use of Incident Learning Systems (ILS) as a method to learn from incidents [3].

ILS in RO have been introduced in many RO departments, as they aid in providing a feedback loop, in which near miss or actual errors can be analysed for root causes and contributing factors, as well as providing insight for potential process or QA changes, and staff education, which is essential in an ever changing and developing work environment [4]. ILS have been shown to be an important means of monitoring, investigating and learning from RO errors and near misses [5], [6]. A review of the literature by Bissonnette et al. [7] found published incidents rates in RO range from 1.5 to 8.0 incidents per 100 courses of treatment, with the majority of incidents being a near miss.

ILS have been implemented in many organisations globally to improve safety in RO delivery. The development and use of ILS is supported by professional society recommendations, industry regulations, accreditation and objective evidence [8]. These include ASTRO’s “Target Safely” campaign [9], The European society for radiotherapy and oncology’s (ESTRO) Radiation Oncology Safety Education Information System (ROSEIS) [10], a voluntary web based reporting system designed for individual clinic use as well as facilitating information exchange between clinics. Safety in Radiation Oncology (SAFRON) [11] which is a voluntary reporting and learning system of radiotherapy incidents developed by the International Atomic Energy Agency, as well as the Canadian National System for Incident Reporting in radiation therapy [12], [13]. Incident reporting and ILS have been proposed as key factors in the safety management, not just in RO, but also in a variety of other safety critical industries, such as aviation [14] as it facilitates improvements in practice that enhance safety [4]. With RO, there is evidence that the use of ILS decreases incidents overtime, due to actions arising from incident analysis. It has also been demonstrated that the effective use of and ILS also promotes an active safety culture and strongly encourages the reporting of incidents [15], [16], [4].

A department-based ILS was introduced into our departments in 2004. The ILS performance was analysed over an initial three year study period (2004–2007) and the results indicated a reduction in error rate, both near miss and actual error [17]. However, determining whether the system specifically improves safety remains uncertain since reporting rates are influenced by other variables such as the introduction of new technology and changes in reporting culture [8].

Materials and methods

Radiotherapy departments

RO services were offered over two geographical sites in metropolitan South-Western Sydney, Liverpool and Campbelltown Hospitals. These departments have a shared organisational structure and quality governance framework with a common incident reporting process and electronic medical records. The department saw rapid development with services across both sites expanding during the reporting period. New equipment was introduced during the study period including a magnetic resonance imaging (MRI) simulator, Tomotherapy treatment unit, a new linear accelerator vendor, installation of a new orthovoltage treatment unit and treatment planning system (TPS). Treatment techniques have also developed over time with the majority of treatments evolving from 3D conformal to intensity modulated radiation therapy (IMRT) and volumetric modulated arc therapy (VMAT) with flattened and flattening filter free beams (FFF). See Fig. 1 for key milestones in the department’s development and expansion.

Fig. 1

Key milestones in the department’s development and expansion from 2004 to 2019.

Current ILS

New South Wales (NSW) Health, including our health service, manages all reportable incidents utilising the Incident Information Management System (IIMS). IIMS is a generic system, used to notify and manage incidents across NSW public health facilities. IIMS lacks RO-specific incident classifications, therefore our health service designed its own in-house ILS that is used in conjunction with IIMS. Systematic RO-specific incident reporting was introduced into our department in May 2004, initially in paper format, which was then manually transcribed into an electronic database. A dedicated electronic ILS was introduced in December 2011. The definitions and classifications used to categorise incidents in both reporting systems were obtained from Towards Safer Radiotherapy [9]. In both reporting systems, incident severity was classified by the Severity Assessment Code (SAC) matrix. SAC score grades the severity of incidents into four severity levels, 1 being the highest and 4 being the lowest severity (Fig. 2). SAC score is determined by assessing the actual outcome of an incident and the potential consequence, which is the worst-case scenario for the incident being assessed. Both the actual and potential outcomes are then scored by assessing the incident against the SAC matrix to determine the appropriate severity level.

Fig. 2

Severity assessment code matrix.

In the database all identified incidents are classified into four risk categories, ranging from highest risk (1) to lowest risk (4) these risk categories relate to SAC Score (Fig. 2). Level 1 incidents are extreme risk incidents and are reported to the Environmental Protection Agency (EPA), the EPA administers the Radiation control Act 1990 and Radiation control Regulation 2013. A Root Cause Analysis (RCA) is conducted by senior management. Level 1 incidents are incidents where there is a variation in prescribed dose of greater than 10%. Level 2 incidents are similar to level 1 but do not need to be reported to the EPA as the variation to prescribed dose is less than 10%. Level 3 and 4 risk are near miss incidents, with level 3 incidents being those classified with a higher potential risk than level 4. These levels were derived from the incident reporting standards published by RANZCR in 2012 [18]. The ILS has drop-down menus for risk categories of reporting and allows for a narrative within the individual entry form. There were thirteen risk categories defined in 2014 that all incidents are currently associated under (Table 1). Once the incident has been entered into the system, the reporter notifies any staff involved to review and add any additional comments before the incident is submitted for management. The management team is responsible for determining the SAC score as defined by NSW Health Policy on incident reporting, [19] using the SAC matrix from Fig. 2. After this assessment, the managers make recommendations on preventative actions. The management team is responsible for data interrogation and reporting and to define the root causes of the incident [20]. This information is fed back to all staff at the monthly multidisciplinary RO incident review meeting to improve knowledge and educate staff. Since the introduction of an ILS this feedback loop to staff has remained consistent although the incident review meeting was expanded in 2017 to include RO’s and medical physics.

Table 1

Incident categories for classification in the ILS.

Categories
Other	Radiation oncologist
Treatment	Bolus
Imaging	Prescription
Pre-treatment	Shielding
Simulation	Medical physics
Computing	Immobilisation device
Documentation

Data collection and analysis

Data was collected over fifteen years from 26th May 2004 to 30th June 2019. Actual errors were defined as patients receiving an error in delivered treatment while near-miss errors were those determined prior to treatment being delivered. The number of total errors, subdivided into actual errors and near-miss errors were calculated, and reported per 1000 treatment attendances. The number of incidents per incident category was also evaluated.

Incidents drawn from the two independent samples of patients, treated during the two time periods, 2005 to 2011 and 2012 to 2018, were compared for any difference. A non-parametric method of comparison deemed more appropriate and the median number of incidents between 2 periods were compared (Mann-Whitney U test). A p-value of <0.05 was considered as significant. The incidents during 2004 and 2019 were excluded for further comparison as the data were not available for the whole year.

Results

In total 1727 reports were submitted into the ILS (Fig. 3). Reporting peaked in 2006 with 251 incidents reported and decreased to 43 in 2013. In the same time period there were 446,950 treatment attendances and 26,946 courses equating to 3.9 incidents being reported per 1000 patient attendances and 64.1 per 1000 treatment courses. Reports were submitted by radiation therapists (RT) (90%), medical physicists (MP) (6%) and ROs (4%).

Fig. 3

Total number of incidents reported per year.

Of all incidents, 1166 (67.5%) of the incidents were classified as near miss and 561 (32.5%) were classified as actual errors. The actual error rate over the fifteen year period is 1.3 reported errors per 1000 treatment attendances and 2.6 reported incidents per 1000 treatment courses (Fig. 4), similar rates to that mentioned by Bissonnette [7]. The overall number of reported incidents including the actual errors and near misses during the years from 2004 to 2019 was 1724. The number of incidents between two seven year periods (2005–2011 and 2012–2018) (N = 1496) have reduced by about 50%, the medians significantly reduced from 129 in the earlier period to 65 in the later period (p < 0.05).

Fig. 4

Actual errors reported (green) and near miss errors reported (red) per year.

Of the total number of reported incidents, 0.2% were rated at SAC 1, 0.5% were rated at SAC 2, 9.0% were rated at SAC 3 and 90.3% were rated at SAC 4 (Fig. 5). The potential SAC scores, that being the score assigned to the consequence of the incident if it had occurred, was 3.4% at SAC 1, 19.1% at SAC 2, 44.1% at SAC 3 and 33.4% at SAC 4.

Fig. 5

Actual Severity Assessment Code (SAC) score (NSW Health) per year against attendances.

The severity scale was measured as a ratio of reported incidents per 1000 treatment attendances. Fig. 5 indicates that of all incidents reported. SAC 1 and 2 severity scale was 0.03. SAC 3 was 0.35 and SAC 4 was 3.49. The comparison of overall SAC 3 and SAC 4 incidents during each seven year period showed a significant (p < 0.05) decline in median numbers by 65. Fig. 6 highlights the decrease in SAC 1 and SAC 2 over time.

Fig. 6

Actual Severity Assessment Code (SAC) score (NSW Health) 1 and 2 per year against attendances.

The most common incident category by classification of incidents was “Other” with 308 incidents, followed by “treatment” with 270 incidents. The least common category was “Immobilisation” with only 18 reported incidents entered into this category (Fig. 7). The “Other” category is used to assign an incident that cannot be categorised.

Fig. 7

Total number of incidents per category.

Discussion

The aim of this paper was to review the fifteen years of incident reporting in our department to provide insight into the success and failures of the ILS as well as determine the future direction of incident reporting in our department. The results suggest a reduction in the incident reporting rate over time within the department. The average incident reporting rate for the first seven years period (2005–2011) was 6 reported incidents per 1000 attendances compared to 2 incidents reported per 1000 treatment attendances for the second period (2012–2018). In addition, the severity of incidents reported has also decreased, with no SAC 1 incidents reported since 2011.

Our rate of RO incidents reported are similar to other studies published [19], [6]. Mitchell et al. analysed five studies and found that the rate of reported incidents per 1000 courses of treatment varied from 34 to 67 [18]. Our analysis identified a rate of reporting an incident of 64 incidents per 1000 courses over the fifteen year period, although this has now dropped to 14 incidents for the last five years of reporting. Interestingly the rate of reporting decreased with the introduction of the electronic ILS in late 2011 although patient numbers have continually increased at a rate of approximately 5% each year. Hoopes et al. [15] noted a similar decline in reporting rates in the literature which they attributed to changes in QA processes from robust incident analysis which is reflected in our incident rate.

As seen in Fig. 1 our department has undergone significant development since 2004. It could be hypothesized that the introduction of new technologies and the changes in quality assurance have impacted the incident reporting rates. A decrease in the severity of incidents reported was seen especially in 2013. That year saw the implementation of daily image guided radiation therapy (IGRT) prior to treatment delivery. The impact of IGRT in reducing the severity of errors is reported by Greenham et al. [21], as this technology identifies errors prior to treatment delivery rather than after the treatment is delivered, reducing the potential severity of the incident to the patient. This is reflected a decrease in reported SAC scores since 2013.

A noticeable increase occurred in 2017 as the number of reported incidents increased to 96 in 2017 from an average of 77 per year in the preceding few years. The reporting year also saw the introduction of a department-wide incident review meeting from an RT only based meeting. The inclusion of the entire MDT and the promotion of a positive safety reporting culture is noted [22] as a contributing factoring in incident reporting and may therefore be attributed to the spike in incident reporting seen this year. Reflecting on developments in our practice (Fig. 1), both the introduction of daily imaging for breast radiotherapy treatment, as well as the move to an electronic based treatment chart (ETC) were also two major practice changes in 2017. Review of the incidents entered in 2017 found 10.4% of reported incidents were related to the new breast imaging technique as well as 5.2% of incidents were related to ETC. All errors were identified with imaging prior to treatment delivery, supporting the theory that IGRT aids in the detection of errors but with a resultant decrease in the severity of the incident as it is identified at the pre-treatment stage. Huang et al. [1], had similar findings and concluded that although new technologies are promoted as being more efficient, they can also increase the incident rates as the complexity of the technique increases, the chance of error is also greater.

The culture of incident reporting in RO across all disciplines has also been reviewed. Our data indicated that 90% of incidents reported by RTs and only 4% from ROs and only 6% from MPs. Our reporting rates across the disciplines similar to that reported by Greenham et al., with ROs and MP rarely submitting adverse event reports and the majority of reports are submitted by RT staff. It would be logical to expect that RTs report the most incidents in a radiation-specific reporting system due to their involvement throughout the dose planning and treatment phases; therefore, they would most commonly detect and report the errors. However, the very low rates of ROs and MP reports suggests room for improvement with respect to the engagement of these disciplines in the reporting process.

There is an inherent need for accurate taxonomy within any ILS to support quality improvement as well as effectively capture incident trends in RO [7], [18]. As 25% of our incidents were classified as “Other”, it is hard to clearly identify trends to initiate quality assurance activities to reduce the frequency of these errors. The overuse of the “Other” category may identify a lack of knowledge of how to categorise incidents and time constraints when reporting, but also indicates that refinement of the categories is required [13], [5]. It is recommended that for future improvements, ongoing refinements in ILS taxonomy occur to accommodate new error pathways, with the emergence of new technology and techniques as well as helping to identify incident trends accurately [13], [20].

Conclusion

The analysis of fifteen years of incident data has been useful in reviewing the effectiveness of our ILS. The data correlated well with literature in that the development of new technologies, especially IGRT, has impacted the severity of incidents reported, with a reduction in SAC 1 and 2 errors. This can be attributed to the quality assurance aspect of IGRT where the incident is identified prior to treatment delivery rather than after, reducing the severity of any potential incidents. The introduction of an MDT meeting may be a contributing factor for increased reporting in 2017 as it potentially encouraged a positive reporting culture and renewed awareness of the reporting process. It should be noted that in 2017 the implementation of new techniques led to an increase in the number of incidents reported. This review is limited by the variations in incident classification in the ILS, with a large proportion of incidents classified as “Other”. Review of the ILS has shown that due to process changes the severity of errors has decreased, but it has also highlighted that taxonomy improvements are essential for better categorisation of incidents in the future.

Declaration of Competing Interest

None.