Automation of DVH Constraint Checks and Physics Quality Control Review Improves Patient Safety in Radiotherapy.

This study investigates whether patient safety can be enhanced by the implementation of an automated electronic checklist (PlanCheck) for physics quality control review (QCR) of radiotherapy photon plans. PlanCheck evaluates both technical aspects and DVH constraints. Three hundred and thirty-one consecutively approved radiotherapy plans previously reviewed with manual QCR were retrospectively checked with PlanCheck. Four hundred and thirty-three (3.4%) of the 12783 automated technical checks executed in the 331 plans yielded an error. All errors were scored using the severity rating from the American Association of Physicists in Medicine TG-100 report. Nineteen of these errors (4%) either could have affected or affected target dose (severity 5+) implicating a maximum dose difference to the target or a critical organ at risk of 0.5% to 10% and 3 errors could have resulted in stereotactic brain treatments being delivered to the wrong location (severity 10). Forty-seven breast cancer plans were retrospectively subjected to automated DVH check, 10 undocumented dose constraint violations were found. PlanCheck has been shown to reduce errors in manually reviewed radiotherapy plans and thus to enhance patient safety. Copyright:

INTRODUCTION

The physics quality control review (QCR) of radiotherapy treatment plans has been proven the most effective check for preventing accidents in radiation oncology.[1] However, the increased complexity of treatment plans, time pressure, and shortage of qualified medical physicists (QMPs) have made QCR more challenging. As an aid in the QCR process, checklists are now recommended by the American Society for Radiation Oncology[2] and in 2015 American Association of Physicists in Medicine (AAPM) published guidelines for the implementation and maintenance of checklists.[3] However, QCR checklists constantly need to be adapted to new treatment techniques and updated with new departmental guidelines and new national laws and regulations. The difficulty of ensuring that all QMPs use the same physical paper checklists has been obvious at our institution and has been a threat to patient safety. As an alternative to paper checklists, electronic checklists have proven successful both with regard to standardization, reduction of plan rejection rates, reducing patient delays, reducing QCR time, and for enhancing patient safety.[45678] However, the implementation of an electronic checklist in a radiotherapy clinic can be both time-consuming and require extensive programming skills. PlanCheck is a semi-automated electronic checklist containing 39 automated checks of technical radiotherapy plan aspects. As opposed to previously published electronic checklists,[4567891011] PlanCheck not only checks technical plan aspects but also contains automated dose-volume histograms (DVHs) constraints checks for all local dose constraints for all plans treated with photons at our institution. In this study, we assess the impact on patient safety of both the technical checks and the DVH constraint checks. PlanCheck is created with minimum effort using the Eclipse Scripting Application Programming Interface (ESAPI)[12] (Varian Medical Systems, Inc, Palo Alto, CA). Developing a checklist as a script greatly simplifies and shortens the implementation of an automated checklist compared to stand-alone programs,[81011] thus making plan QCR automation more readily available for clinics with sparse programming experience and time.

MATERIALS AND METHODS

The PlanCheck script is a C# in-house-developed electronic checklist created using ESAPI, partly based on a previous scripting project,[13] and interacting with a local database created in MySQL[14] (Oracle Corporation Redwood City, CA) containing all structure names and DVH constraints.

Our institution has made use of a physical manual checklist for QCR of treatment plans for many years. The purpose of PlanCheck was to automate as many of the checks in this physical checklist as technically possible. PlanCheck currently automates 39 checks [Appendix Table 1] and checks 856 dose constraints for all 224 diagnoses treated with photons at our institution. PlanCheck is executed on a plan-by-plan basis and generates a report showing both the values expected by the script (fetched from the database) and the values extracted from the plan and the DVHs through ESAPI. Checks producing no errors are showed in green and checks that are out of tolerance are showed in red. Furthermore, the checks are written in a traffic light system, where the color of the traffic light indicates the importance of the check, red traffic light indicating the highest level of importance, yellow being the intermediate importance level, and green being the lowest level of importance. In order to standardize the physics QCR procedure, the checks that are not automated are printed in the report in the form of a manual checklist.

PlanCheck is dynamic in the sense that only the checks that are relevant for the plan in question will be activated [see what checks are activated for what plans in Appendix Table 1]. Adherence to all checks activated for a specific plan ensures a complete physics QCR of that plan at our institution and is followed by a checkpoint to be signed off electronically. In order to save time in the clinical workflow, PlanCheck is not only executed by the QMP as part of the physics QCR but the automated part of the script is also executed by the treatment planner at the planning stage before reaching plan approval and physics QCR, thus avoiding unnecessary plan iterations due to errors caught late in the planning process.

This study investigates whether automated physics QCR with PlanCheck reduces the number of errors in plans previously clinically approved with manual QCR, i.e., using a paper checklist. To assess the impact on patient safety of the 39 technical checks, a retrospective study was conducted. Thus, 331 consecutively approved plans, approved for treatment with manual QCR between July 1 and August 31, 2017 (before the implementation of PlanCheck), were subjected to automated QCR with the script. All errors were automatically saved in a database and the error categories (a combination of the type of check and the type of the plan) where given a severity score using the recommendations from the AAPM TG-100 report.[15] Errors with a possible severity score of 5 or higher (5+) where reviewed and the errors where scored and evaluated individually. The dosimetric impact of errors with severity of 5+ were assessed by subtracting the treatment approved dose distribution from the intended dose distribution in the Eclipse treatment planning system (TPS) (Varian Medical Systems, Palo Alto, CA). The largest dose difference (DD) in the target or a critical organ at risk (OAR) thus found was recorded.

The assessment of the impact of the automated DVH checks on patient safety is made difficult since these results are not saved in the database. However, our local dose constraints for breast cancer patients were revised in December 2020 while PlanCheck was not updated until January 2021. Thus, in December 2020, 47 consecutive breast cancer radiotherapy plans were approved using manual DVH checks only. This gave us the possibility to in February 2021 retrospectively investigate whether PlanCheck could catch DVH violations that were overlooked in the manual QCR in these breast cancer plans. The DVH constraint violations thus found were recorded and assessed dosimetrically. Dose constraint violations detected by PlanCheck but documented in the patient journal were excluded from the analysis.

RESULTS

A total of 12783 automated technical plan checks were executed in 331 consecutively approved plans, resulting in 433 potential errors detected (3.4% of the checks resulted in an error). The distribution of the detected errors between the checks is shown in Figure 1 [Descriptions of checks in Appendix Table 1].

Figure 1

Histogram showing the per-check distribution of the 433 errors caught by the automated checks in PlanCheck sorted in pareto order. The dashed line showing the cumulative percentage of each error to the total amount of detected errors

In Table 1, the severity distribution of the detected errors, according to the AAPM TG-100 report is shown. Eighty-four percent of the errors (362) had no impact while 11% (48 errors) were assessed to potentially cause inconvenience, either to the staff or to the patient (severity 1–3). Four plans had errors that could have led to suboptimal dose deliveries (severity 4). Of the 14 plans scored with severity 5, six plans had a wrong dose normalization method (DD 0.5%–2%), two breast cancer plans did not include the couch in the dose calculation (DD 0.5% and 2%), 5 plans had an incorrect mean dose to the target compared to the prescription dose (DD 2%–2.5%), and one plan had an incorrect dose resolution (DD 5%). Two plans with errors in the dose resolution were assessed to have a severity score of 6 (DD 8% and 10%). Three plans had errors that could have led to stereotactic brain treatments being delivered to a very wrong location (severity 10).

Table 1

Overview of the distribution of severities among the technical plan errors detected by the automated checks

Severity score	Number of errors
No	362
1	44
2	2
3	2
4	4
5	14
6	2
10	3
Severity ≥1	71
Sum	433

In the assessment of the automated DVH checks on the 47 breast cancer plans subjected to physics QCR with PlanCheck, 10 errors were found. Three of these plans delivered 5-14 Gy too high max dose to the spinal cord while four plans delivered 0.4-0.9 Gy% too high mean dose to the heart. Two plans showed 0.3 and 1.4 Gy increased D35% to the ipsilateral lung, a single plan was found to have excessive dose outside the CTV by 0.2 Gy.

DISCUSSION

In agreement with previously published work,[716] our results show that the automated electronic checklist can detect errors overlooked in manual QCR.

In addition to the obvious enhancement in patient safety gained by reducing errors of geographical or dosimetric impact (severity 5+), reducing also clerical errors saves time in the clinic and has a positive impact on patient safety since corrective measures by dosimetrists and physicists are often made under time pressure. Furthermore, our previous study[16] showed that PlanCheck reduces the mean time spent per plan QCR from 16:20 minutes ±8:50 to 12:00 minutes ±9:20 (P = 0.009).

PlanCheck is continuously updated to ensure that the checks included are catching the errors seen in incidents or in clinical practice and to ensure that the script is keeping up with the technical advances in the clinic. Furthermore, the amount of checks being automated is continuously increasing, thus easing the time pressure on the physicist performing the QCR. Some parts of the check, for example, the shape and position of the gross tumor volume and clinical target volume or the dose distribution outside of the target and OARs, will be manual for still quite some time. However, due to the new technical possibilities, the size of the planning target volume is something that will be checked automatically in a future version of the software. Occasionally, we have seen that errors that could have been caught by PlanCheck have remained uncorrected in the approved treatment plan. To deal with this issue, we have recently implemented check points to be signed off whenever PlanCheck should be run, either at the planning stage or during QCR.

Since all errors were detected retrospectively some patients in this study were treated with defective treatment plans. The 19 errors with a severity of 5 + were reviewed manually, and no error was assessed to have impacted the clinical outcome to the patient, however all 19 errors were reported in accordance with national guidelines. Specifically, the three plans with severity 10 were found to not have resulted in mistreatment, although they could have resulted in a complete geographic miss. This study shows that PlanCheck contributes to improving patient safety. PlanCheck is currently used routinely for all diagnoses treated with photons at our institution. Since all structure names and dose constraints are held in a database, PlanCheck is both easy to maintain and easily implementable in other institutions using the Eclipse TPS. Versions of PlanCheck are currently used at Rigshospitalet (Copenhagen, Denmark) and at Zealand University Hospital (Næstved, Denmark) and another version is being implemented at Herlev Hospital (Herlev, Denmark). To aid in the implementation of PlanCheck in other institutions we have recently made PlanCheck publicly available on GitHub along with an SQL file containing the full constraint database.[17] As one of the authors performed the implementation at Zealand University Hospital, we have no data of how long a full implementation of the script would take for institutions unfamiliar with the script. However, the documentation on GitHub not only contains information about the content of the files and the functionality of the methods but also on what methods need editing upon implementation of PlanCheck in a new institution.

CONCLUSION

PlanCheck reduces the number of undetected technical errors and DVH constraint violations in treatment plans compared with manual QCR at our institution and thus enhances patient safety. Furthermore, PlanCheck is both easy to maintain and easily implementable in other institutions that are using the Eclipse TPS. It has proven its ability to catch rare errors with high potential severity for the patients, i.e., errors easily missed in a manual QCR.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

A short description of the checks in PlanCheck, including information about what types of plans the checks are activated for and whether the checks are executed automatically or performed manually by the physicist

Type of check	Name of automated check	Description of check	When executed?
Automated		Dose constraints	All plans
Automated	Plan_name_id	That the plan ID equals the plan name. Important for certain imaging devices	All plans
Automated	Course_diagnose	Diagnose attached to course	All plans
Automated	Number_of_fractions	Number of fractions	All plans
Automated	Fraction_dose	Dose per fraction	All plans
Automated	Ref_point_number	Reference point number	All plans
Automated	Scan_name	Scan name	All plans
Automated	Use_gated	The plan can be used gated	All plans
Automated	Scan_date	Scan date	All plans
Automated	User_origin_in_body	Is the user origin inside the body structure?	All plans
Automated	Couch_type	Has the right couch top been added?	All plans
Automated	Couch_HU_int	Is HU of internal couch correct?	Only plans with couch
Automated	Couch_HU_ext	Is HU of external couch correct?	Only plans with couch
Automated	Virtual_bolus	Are virtual boluses attached to all fields?	All plans
Automated	Same_isocenter	Do all fields have the same isocenter?	All plans
Automated	Treat_name	Names of treatment fields and setup fields. Also checks whether there is an image field for partial breast irradiations	All plans
Automated	DIBH_wedge	Wedge on DIBH field?	Only static DIBH fields
Automated	RA_collimator	Is any collimator placed at a cardinal angle and are there any identical collimator angles?	VMAT only
Automated	Arc_x_coll	Are the X field sizes below departmental limits?	VMAT only
Automated	y_coll	Are Y collimators below departmental limits?	VMAT only
Automated	Setup_coll	Setup field size (CBCT and OBI)	All plans
Automated	cbct_bones	Are the bones delineated?	Only CBCT as setup field
Automated	Dose_algorithm	Algorithm used for dose calculation	All plans
Automated	Dose_resolution	Dose calculation resolution	All plans
Automated	Dmean_target	Mean dose to target	All plans
Automated	Refpoint_target	Is physical reference point inside the target structure?	Only for physical reference points
Automated	Total_referencepoint	Reference point total dose limit	All plans
Automated	Daily_referencepoint	Reference point daily dose limit	All plans
Automated	Session_referencepoint	Reference point session dose limit	All plans
Automated	mu_gy	Number of MU per Gy. Fails if≥300 MU/Gy	All plans
Automated	Dose_2_refpoint	Dose to reference point	All plans
Automated	Lower_objective_oar	Lower objective on OAR?	All plans
Automated	Dmax_in_target	Is the maximum dose inside the target structure?	All plans
Automated	mlc_at_max	Are any MLC’s at maximum extension?	All plans
Automated	Normalization	Normalization method	All plans
Automated	obi_angle	OBI angles	OBI setup fields only
Automated	Normal_tissue_objective	Has a ring or normal tissue objective been used?	VMAT only
Automated	Plan_norm_value	Plan normalization value	All plans
Automated	Virtual_refpoint	Virtual reference point used?	All plans
Automated	Treatment_time	Treatment time	All plans
Manual	N/A	Is the plan ID the same as in the patient journal?	All plans
Manual	N/A	Correct accelerator?	All plans
Manual	N/A	Placement of user origin	All plans
Manual	N/A	Pacemaker or ICD accounted for?	All plans
Manual	N/A	Metal artefacts, air gaps and contrast agents accounted for?	All plans
Manual	N/A	Is the target delineated according to the patient journal?	All plans
Manual	N/A	Is the target structure properly delineated?	All plans
Manual	N/A	Placement of couch on CT scan	All plans
Manual	N/A	Accuracy of body contour	All plans
Manual	N/A	Target structures cropped from body structure?	All plans
Manual	N/A	PTV margin	All plans
Manual	N/A	Should there be a bolus?	All plans
Manual	N/A	Correct treatment technique used?	All plans
Manual	N/A	Isocenter position	All plans
Manual	N/A	MLC movements	VMAT only
Manual	N/A	Should arms be delineated and used as objectives?	VMAT only
Manual	N/A	Number of arcs	VMAT only
Manual	N/A	Matching strategy	All plans
Manual	N/A	Positioning of calculation box	All plans
Manual	N/A	Dose distribution outside target and OARs	All plans
Manual	N/A	Does the dose overlap with previous treatments?	All plans
Manual	N/A	The value of the maximum dose according to position	All plans
Manual	N/A	Planning CT and PTV margins	Lung plans with DIBH
Manual	N/A	Should DIBH be used?	Breast cancer and lung cancer
Manual	N/A	The appearance of the DIBH curve	DIBH plans
Manual	N/A	Placement and thickness of bolus	If bolus used
Manual	N/A	MU of fields connecting supraclavicular fields with breast fields	Breast cancer where supraclavicular lymph nodes are treated

DIBH: Deep inspiration breath-hold, CT: Computerized tomography, PTV: Planning target volume, VMAT: Volumetric modulated arc therapy, OAR: Organ at risk, HU: Hounsfield unit, RA: RapidArc, CBCT: Cone beam computed tomography, OBI: On-board imaging, MLC: Multileaf collimator, ICD: Implantable cardioverter defibrillator, MU: Monitor unit