Errors detected during physics plan review for external beam radiotherapy.

Background and purpose: Risk management in radiotherapy is of high importance. There is not much data published on errors occurring in the treatment planning process of external beam techniques. The aim of this study was to investigate errors occurring during physics plan review in external beam radiotherapy. Materials and methods: Over a period of 14 months errors observed during the physical review process are reported. The errors were grouped and evaluated regarding treatment machine, technique, and treatment site. In addition, a correlation between frequency of errors and staff shortage was analyzed.
Results: Subgroups of grave errors (g-errors) and slight errors (s-errors) were defined to consider the different impact on the patient and clinical workflow of the errors. In 1056 plans reviewed, 110 errors (41 g-errors, 69 s-errors) were detected. The most common g-errors and s-errors were "Wrong gantry angle at setup field" (n = 19) and "Wrong field label" (n = 24), respectively. A correlation of number of errors and treatment machine, technique, or anatomical site could not be found. No correlation between staff shortage and number of errors was observed. Conclusions: The process of reviewing treatment plans is a relevant topic to consider in risk analysis of the radiotherapy workflow. The review process could be improved by enhancements in the treatment planning systems, use of digital dose prescription, and treatment planning templates.

Introduction

Radiotherapy is a very successful method with increasing complexity in treating cancer. The treatment planning system (TPS) holds a central position in the process of radiotherapy treatment. In a TPS the dose distribution is calculated, parameters for the treatment machines are defined, and stored in a data base for subsequent treatment. Many efforts were made to achieve safe treatment and minimize potential errors in the last years. European guidelines for risk management in radiotherapy were published [1] to support Council Directive 97/43/EURATOM. In 2016 the American Association of Physicists in Medicine (AAPM) presented the report of the Task Group (TG) 100 on application of risk analysis methods to radiation therapy quality management [2]. More recently Ford et al. published the TG 275 report of the AAPM [3]. In this report the importance of physics chart review is highlighted and potential errors are described in detail including their severity, occurrence, and detectability. In another study based on a collection of 4407 incidents, Ford et al. highlighted that the most effective preventive measure is plan review [4]. In several publications occurring or potential errors are discussed. According to TG 100 [2] errors are “failures consisting of acts, either of commission (doing something that should not have been done) or omission (not doing something that should have been done), that incorrectly execute the intended action required by the process.” Errors might have an impact on the treatment and thus are a central element for risk analysis, e.g. by Failure Mode and Effects Analysis (FMEA). According to the European ACCIRAD report [1], [5] an adverse error-event results in unintended harm, either minor or serious, to the patient. However, Ford et al. [6] define “error” as “failure to complete a planned action as intended or the use of an incorrect plan of action to achieve a given aim.” Following this reading, an error can either have a direct impact on the irradiation quality, but can also delay the treatment process. In this study, we will define the term “error” narrowly, following the publication by Ford et al. Covington et al. [7] presented a similar investigation on errors in external beam therapy over a period of one year using a plan checker software.

The aim of our study was to detect and investigate errors occurring in physics plan review in external beam radiotherapy treatment plans. The manuscript at hand presents original data recorded over a 14 months period in a university clinic showing errors in treatment plans for photon external beam radiotherapy (EBRT) detected during the physical review process.

Material and methods

Treatment planning process

Treatment planning in this analysis was conducted using TPS Eclipse v13.6 (Varian a Siemens Healthineers Company, Palo Alto, CA). In the university clinic of radiotherapy, a team of six medical physicists and one trainee, with one to 14 years of experience, performed the EBRT planning service in a weekly changing rotation principle, i.e. two physicists had treatment planning service per week. The individual experiences of the physicists (planners) and the composition of the two-person teams were not considered in our analysis. As typical for radiotherapy clinics the treatment planning request was issued by radiotherapists. At this stage the required imaging for the patient treatment (e.g. computed tomography, magnetic resonance imaging, or positron emission tomography) have already been acquired and stored in the ARIA oncology information system (Varian) including the TPS Eclipse. Contouring of organs at risk (OARs) and clinical target volumes (CTVs) was already conducted as well as the necessary image registrations. Together with the treatment planning request a radiotherapy treatment concept including dose prescription signed by a senior radiotherapist was passed in paper form to the medical physics department planning service. Here, the treatment planning took place by the aforementioned medical physicists. Typically two to four days were allowed until a treatment plan was finalized and ready for treatment. After creating and optimizing the treatment plan by the physicist, it was discussed between the planner and a senior radiotherapist on a computer screen. When both decided that the treatment plan was acceptable, a hardcopy was printed and the plan was ready for review. In the review process a senior physicist (not the planner of this treatment plan) had to check this treatment plan in the TPS as well as the plan printouts. Once the review process was complete, the printouts were signed and the approval status in ARIA for this plan was digitally set by password to “Planning approved”. Any errors in the treatment plan that were detected at this step were recorded for this study and are presented in this manuscript. After the physics approval the signed treatment plan was checked again by a member of another professional group, a senior radiotherapist, and the approval status in ARIA was changed to “Treatment approved”. When this was finalized, the treatment plan was cleared for treatment and treatment parameters were automatically locked in the database. The treatment planning process from prescription to final plan approval just before treatment is shown in Fig. 1. Treatment plans using intensity modulated radiotherapy (IMRT), volumetric modulated arc therapy (VMAT), or stereotactic techniques underwent an additional patient specific quality assurance before the first dose fraction delivery, but this is not considered in this study. In this manuscript all errors occurring during a 14-month period (from 31.01.2019 to 08.04.2020) detected by a single physics reviewer are presented.

Fig. 1

Illustration of the treatment planning and review process in ARIA. The approval status of the treatment plans is also included as well as the person who is carrying out the approval. The dashed rectangle shows the portion of the plan review that is addressed in this study.

Error classification

The occurred errors were documented in an Excel (Microsoft Office 2010, Microsoft Corporation, Seattle, WA) spreadsheet including the following data: date, short description of the error, anatomical treatment site, treatment technique, and type of treatment machine. After short feedback discussion with the planner errors were corrected and the treatment plan was run through the review process for a second time. These second review processes were not considered in this investigation but no errors/deviations were observed in this step. The number of reviews per month varied due to alterations in patient load and holiday periods. The investigated physics review herein is only a measure within the risk management strategy in our clinic. Other errors might occur earlier in the process, as in contouring, or later e.g. during patient set-up.

We investigated the kind of errors that occurred and analyzed if the occurrence correlated with type of treatment machine, radiation technique, or anatomical treatment site. Also, the frequency of errors was analyzed in regard to shortage of staff, e.g. during holiday periods and, furthermore, what kind of errors could be trapped in the TPS.

Errors were divided into two groups: g-errors and s-errors. g-errors have the potential to harm the patient, s-errors have not, but they can delay the treatment process or can result in inconsistencies in treatment reporting. This distinction between g- and s-errors was made in an interdisciplinary discussion in our clinic, in which all errors that occurred were considered. Ford et al. [4] followed a similar method. They established baselines for high and low severity incidents in a face-to-face meeting using the ANS score [8]. Also, Siochi et al. [9] made a distinction of error severity levels. It should be emphasized that no treatment failure or incidence resulted from the errors in this investigation or occurred during the time of this study in our clinic. Statistical comparisons were performed using Chi-square tests in Excel. Statistical significance was assumed with p < 0.05. All patients in this study gave informed consent for data processing for scientific purposes for our clinic.

Results

Altogether 110 errors of 29 different types were detected in the physical review process of treatment plans. In particular we found 41 g-errors (3.9 % of all plans) and 69 s-errors (6.5 %). Table 1 shows all errors occurring in this study more than once including a short description. The complete list of errors can be found in the supplementary material of this publication. In this table the frequency of the errors is also listed. For some errors a technical barrier was already implemented in the Eclipse/ARIA system. That means, the system denied final (treatment) approval, until a correction was performed. These errors were marked by a “T” in the last column of Table 1. In addition, errors that could be avoided by the TPS are marked by a “(T)” in Table 1, but this had not been implemented in our clinic at the time of this study.

Table 1

Short description of errors (g-errors and s-errors) observed in this study during the review process. In addition, the frequency (number of occurrences) of the errors are given for all errors with frequency > 1. In the last column it is marked by a “T” whether a technical barrier in the TPS for the error is already implemented. “(T)” indicates the errors that could be trapped by the TPS, but this had not been implemented at the time of this study.

	Description	Frequency	Barrier in TPS
G-Errors
Wrong gantry angle at setup field	Two setup fields are defined at 0° and 90°/270°. If a gantry angle is not correct, the setup field will be modified prior acquisition.	19	(T)
Wrong dose/number of fractions	A wrong dose or number of fractions compared to prescription was entered in the TPS. This can result in over or under dose to the target volume and over dose to healthy tissues.	5	(T)
Wrong gantry angle	Wrong gantry angle means the gantry angle is erroneously deviating from a predefined planning technique like a box; e.g. 95° gantry angle instead of 90°.	5	(T)
MLC not conformal to planning target volume (PTV)	The MLC was not properly adjusted in regard to the PTV.	2
Isocenter not defined or at false position	If the isocenter is not or falsely defined the TPS uses the Dicom center as isocenter. This means after patient setup using simulation markers, a wrong couch shift can be applied for the setup images. A wrong delivery of a treatment field will most likely not occur as the correct field position is checked by a setup field.	2
S-Errors
Wrong field label	Treatment fields are labelled according to the gantry rotation. Even if this is incorrect the correct treatment will be applied.	24
Wrong machine	The treatment plan is calculated for a wrong treatment machine. This can mean a swap between the two Artiste linacs (which are identical in regard of dosimetry) or a swap between Artiste and TrueBeamSTx. Incorrect treatment is not possible. This issue can result in a new treatment planning or affect the patient logistic.	7
Setup field with MLC	In the considered clinic open setup fields without MLCs are created; in other clinics the MLC might be included in the setup field. To include an MLC in the setup field is a deviation from internal clinical rules and no g-error as the acquisition of the setup field with MLC would not be repeated with a setup field without MLC.	7	(T)
Isocenter position not rounded	Typically, in the considered workflow isocenter positions are rounded to half of a centimeter. To omit this is a violation of internal clinical rules but cannot result in wrong treatment.	4
Primary reference point for IMRT/VMAT with location	In the considered treatment plans, normally a primary reference point with location and plan normalization (100 % dose) is used. For VMAT or IMRT and some 3D technique, the normalization is done differently (e.g. 100 % at target mean). In these cases, a location of the primary reference point might result in an incorrect dose reporting.	4
No DRR for setup field created	DRRs (digital reconstructed radiographs) are created in the TPS and are linked to the setup field. If a DRR is not created this will be done at the console of the linac.	3
Fields with zero dose rate	If a treatment field is defined without a dose-rate definition, a plan approval is not possible.	3	T
MLC exceeds physical constraints	Leaf positions and movements are restricted by physical constraints as maximum leaf overtravel and leaf span. In some cases, e.g. after manual adjustment of leaf positions, it can occur that these are not fully met after dose calculation. When such plans are approved an error message appears. A verification of MLC positions in the TPS is needed followed by new dose calculation.	3	T
Jaws not collimating the MLC field	For MLC fields the collimation by the jaws must match the MLC aperture to minimize dose leakage within the range of the leaf width. This is not the case in clinics where back-up jaws are set to the maximum aperture of all field segments.	2
Old plan version printed	Treatment plans are still printed in the considered clinic for archiving and handout at the linac. If a wrong version was printed this cannot result in a wrong treatment, because the approved correct digital treatment plan is transferred to the linac and differences with the printed plan will attract attention.	2
Bad printout quality	By bad printout quality of a treatment plan, like improper zoom settings in isodose representation, no wrong treatment can emerge as the correct plan parameters are used in ARIA.	2
Not enough dose from field at primary reference point	The primary reference point receives a dose contribution from every field in the treatment plan. If the reference point is located far away from one or more treatment fields their dose contribution will be too low to be considered. A plan approval is not possible. In such cases either the location of the primary reference point has to be adjusted or the reference point is changed to a reference point without location.	2	T
Primary reference point undefined	Without a primary reference point plan approval is not possible.	2	T
Wrong energy for setup field	If a wrong energy of a setup field is set, the treatment plan can be approved, but the setup field cannot be acquired.	2	T

The errors most commonly detected by far are “Wrong field label” (n = 24, s-error) and “Wrong gantry angle at setup field” (n = 19, g-error). Other frequently occurring errors were “Wrong machine” (n = 7, s-error), Setup field with MLC” (n = 7, s-error), “Wrong dose/number of fractions” (n = 5, g-error), and “Wrong gantry angle” (n = 5, g-error).

A total of 689 treatment plans were planned for the two Artiste (Siemens Healthineers, Erlangen, Germany) linacs and 367 for the TrueBeamSTx (Varian) linac. Errors occurred as follows: 67 for the Artiste linacs and 43 for the TrueBeamSTx (see Table 2). Chi-square analyses showed no significant difference in error occurrence between the machine types (p≫0.05). The 1056 treatment plans contained 767 conformal 3D plans, 18 IMRT plans, 220 VMAT plans, and 51 plans using aperture modulated arc therapy (AMAT). AMAT and VMAT were available at the TrueBeamSTx only. The represented anatomical treatment sites were: brain (n = 116), head and neck (n = 100), breast (n = 210), extremities (n = 43), chest (n = 274), pelvis (excluding prostate) (n = 195), and prostate (n = 118). The results in Table 2 illustrate that most errors occurred in 3D treatment plans (n = 84). But, the percentage of plans with errors was more balanced with 11 % for 3D plans, 9 % for VMAT, 10 % AMAT, and for 6 % IMRT with the smallest fraction. Furthermore, anatomical treatment sites with the highest percentage of plans with errors were pelvis (14 %), breast (14 %), and head and neck (H&N) (12 %).

Table 2

Number of treatment plans and errors are listed with respect to treatment machines, treatment technique, and anatomical site. Errors are given in absolute numbers and in percent.

	Number of plans	Percentage of plans	Number of errors	Percentage of plans with errors
Treatment machine
2x Artiste	689	65 %	67	10 %
1x TrueBeamSTx	367	35 %	43	12 %
Treatment technique
3D	767	73 %	84	11 %
IMRT	18	2 %	1	6 %
VMAT	220	21 %	20	9 %
AMAT	51	5 %	5	10 %
Anatomical site
Brain	116	11 %	8	7 %
H&N	100	10 %	12	12 %
Breast	210	20 %	29	14 %
Extremities	43	4 %	3	7 %
Chest	274	26 %	22	8 %
Pelvis (no prostate)	195	19 %	28	14 %
Prostate	118	11 %	8	7 %

Fig. 2 depicts the probability of treatment plan errors per month. Chi-square test revealed no significant increase in error rate during periods of staff shortage (p≫0.05).

Fig. 2

Percentage of errors in treatment plans per month. Bars of months in which more than 25% of the physics staff was not in the clinic (e.g. holiday period, sickness leave) are marked in dark grey.

Discussion

It was demonstrated, that in the review process of radiotherapy treatment plans errors could be filtered out effectively. Thus, the physical review process is a central element in risk management. With a percentage of 3.9 % g-errors and 6.5 % s-errors it is indispensable to have a well working review mechanism for radiotherapy treatment plans.

The most common s-error was “Wrong field label”. This means that the label (name) was not correctly entered in Eclipse. In our clinic it was custom to name the field labels according to the value of the gantry angle, e.g. a field with gantry angle 90° was labelled “90” manually in Eclipse. The stored data of the treatment fields were loaded from the ARIA database to the linear accelerator (linac) console. Even if the label of a treatment field was wrong (e.g. “100” instead of the correct “90”) the correct gantry angle of 90°would have been used by the linac system and displayed on the console and in-room monitor, because the gantry angle is stored in another entry of the data set that cannot be changed by the radiographer at the linac. Because this error has no potential to harm the patient, we categorized it as s-error. The setup fields could be manually created in Eclipse during the planning process. Typically, two setup fields with gantry angles of 0° and 90° or 270° were defined. These were used for isocentric position verification of the patient at treatment. When these field angles were not set correctly, they had to be adapted at the first day of treatment for a proper setup field. If a wrong gantry angle was entered in the TPS this would have been corrected by the radiographer. However, this may be overlooked and the setup-field may need to be repeated with a correct gantry angle. This would result in unnecessary dose exposure to the patient due to the corrected setup-field. Setup fields could be created in the TPS on base of an existing treatment field. They could also contain the multileaf collimator (MLC) positions from this treatment field. In the workflow of our clinic the MLC should not be included in the setup fields and must be deleted. This was wrongly set seven times (n = 7) during this study. Nevertheless, setup fields would not be repeated without MLC, so we defined this error as s-error. Another s-error that also occurred with high frequency (n = 7) was a falsely selected treatment planning machine. Here, in particular the two Artiste machines were sometimes swapped in the treatment plan. Both were matched in terms of dosimetry, but the personnel must manually override each time the patient was treated if the wrong Artiste linac was selected.

In this investigation it was also shown that technical barriers in the TPS can detect some s-errors. The TPS was able to detect 14 of 95 s-errors.

Reviewing the listing in Table 2, it is clear that TPSs have the potential to capture more treatment planning errors. This requirement is in line with the recent TG 275 report [3], where the authors suggested some improvements to vendors of TPSs in order to automate the physics plan review. An example for this is the field label: in some clinics treatment fields in the TPS are labelled according to the gantry angle, e.g. a field with gantry angle of 90° is labelled “90”. If this labeling was not correct, e.g. if the planner forgot to update the label after manually optimizing a gantry angle in 3D conformal treatment plan, the field will be applied correctly at the linac, despite the labelling does not comply with internal clinical rules. An automatic field-labelling could be implemented in the TPS as option for users. Another example would be the implementation of a prompt in the TPS if the isocenter was not set. This would avoid an error when the isocenter was not set by the user. The s-error “Primary reference point for IMRT/VMAT with location” could as well be trapped if this parameter was automatically checked by the TPS.

Other error occurrences might be reduced when using treatment planning templates. In such templates many parameters are already set (e.g. setup fields discussed above), so that manual (error-prone) interaction by the user is limited. In future, the use of artificial intelligence (AI) could have the potential to check treatment plans for quality and inconsistencies [10]. It is conceivable that errors such as “Wrong dose/number of fractions” or “Wrong gantry angle” are detected by the AI.

At the time of data collection for this investigation in our clinic, the planning request was in a printout from. The errors found in “Wrong dose/number of fractions” were due to the transfer from the printout to the TPS by the physicist sometimes in addition with unclear dose prescription. If the correct dose prescription has already been digitally entered and approved into the TPS by the radiotherapist via “Prescribe treatment” in ARIA, this error cannot occur, as the treatment plan is linked to the prescription of the treatment plan.

In this study we show the workflow and errors occurring in our university clinic. Workflows vary across radiotherapy clinics, as does the assessment of errors. Our results demonstrate where improvements in the planning process to minimize the errors in external beam treatment planning are possible. In particular, the use of digital dose prescribing and treatment planning templates are beneficial and have subsequently been implemented in our clinic. For ARIA the CarePath management system is available. This is a software module in ARIA to assist the treatment workflow. It could further reduce the occurrence of errors and save time. We plan to implement this system in our clinic in the near future.

Workflow and review processes are clinic specific. Our results obtained in the study underline the importance of performing a risk analysis which includes the review step. The data is a good base for analyzing occurrences of errors in the treatment planning process. Consequently, the results can be used for further risk analysis in the external treatment planning process in our clinic, e.g. by FMEA [2], [11], [12].

Covington et al. evaluated 2830 external beam plans using a plan-checker tool and found 182 errors (6.4 %) over a period of one year [7]. They concluded that using automated plan checking did not reduce number of errors but decreased patient treatment process delays. Simulation of errors in radiotherapy was presented in a manuscript by Gopan et al. [13]. In their study they created “mock” errors based on an internal incident learning system and the SAFRON database in treatment plans and found a detection rate of 67 % in the physics plan review. Using Bayesian networks Luk et al. [14] could demonstrate that such networks have the ability to detect classes of errors in radiotherapy. Also, Lack et al. applied error detection using an external software for trapping transfer and TPS errors [15]. Using their software tool the number of errors decreased from 84 (in 2016) down to 44 in 2017. This shows clearly that automation can help in error detection.

The physics review process should be embedded in a more global review strategy of the clinic. In the TG 275 report needs and strategies were demonstrated for review processes in radiotherapy. These reviews can be seen as a team process as recently published by Kutuk et al. [16]. In a prospective multi-disciplinary trial they identified needs for changes in 19 % of procedures. In a similar approach Vijayakumar et al. showed that 8 % of treatment plans required re-planning and 23 % required minor changes [17].

In our investigation of 1056 treatment plans over a 14 months period we identified 110 errors in the physical review process. The distinction between grave and slight errors makes sense and consensual or normalized definitions on error classes are needed. g-errors can impair patients, while s-errors cannot harm the patient but delay the treatment process. This classification may simplify further risk analysis. The necessity of physical planning review in the clinic was clearly demonstrated by presenting original data. It should be noted that each clinic must develop its own quality assurance system that fits its individual processes and individually assesses the distinction between g- and s-errors.

We further suggested improvements in the physical review process by enhancements in the TPSs, use of digital dose prescription, and treatment planning templates. No effect of staff shortage on the percentage of plans with errors was found. There was no evidence of correlation between occurrence of errors and treatment machine, treatment technique, or anatomical site in this study.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.