Offline Quality Assurance for Intensity Modulated Radiation Therapy Treatment Plans for NRG-HN001 Head and Neck Clinical Trial Using Knowledge-Based Planning.

PURPOSE: This study aimed to investigate whether a disease site-specific, multi-institutional knowledge based-planning (KBP) model can improve the quality of intensity modulated radiation therapy treatment planning for patients enrolled in the head and neck NRG-HN001clinical trial and to establish a threshold of improvements of treatment plans submitted to the clinical trial. METHODS AND MATERIALS: Fifty treatment plans for patients enrolled in the NRG-HN001 clinical trial were used to build a KBP model; the model was then used to reoptimize 50 other plans. We compared the dosimetric parameters of the submitted and KBP reoptimized plans. We compared differences between KBP and submitted plans for single- and multi-institutional treatment plans.
RESULTS: Mean values for the dose received by 95% of the planning target volume (PTV_6996) and for the maximum dose (D0.03cc) of PTV_6996 were 0.5 Gy and 2.1 Gy higher in KBP plans than in the submitted plans, respectively. Mean values for D0.03cc to the brain stem, spinal cord, optic nerve_R, optic nerve_L, and chiasm were 2.5 Gy, 1.9 Gy, 6.4 Gy, 6.6 Gy, and 5.7 Gy lower in the KBP plans than in the submitted plans. Mean values for Dmean to parotid_R and parotid_L glands were 2.2 Gy and 3.8 Gy lower in KBP plans, respectively. In 33 out of 50 KBP plans, we observed improvements in sparing of at least 7 organs at risk (OARs) (brain stem, spinal cord, optic nerves (R & L), chiasm, and parotid glands [R & L]). A threshold of improvement of OARs sparing of 5% of the prescription dose was established for providing the quality assurance results back to the treating institution.
CONCLUSIONS: A disease site-specific, multi-institutional, clinical trial-based KBP model improved sparing of OARs in a large number of reoptimized plans submitted to the NRG-HN001 clinical trial, and the model is being used as an offline quality assurance tool.

Introduction

The NRG-HN001 is a phase II/III multi-institutional clinical trial involving patients who were diagnosed with nasopharyngeal carcinoma based on the presence of Epstein-Barr virus DNA. The trial mandates that all patients are treated using intensity modulated radiation therapy (IMRT). Head and neck IMRT treatment planning is challenging and complex owing to the involvement of several planning target volumes (PTVs), simultaneous integrated boost, and attempts at sparing several organs at risk (OARs) (eg, spinal cord, brain stem, optic structures, and several glands). Inverse treatment planning, used to develop IMRT treatment plans, relies on an iterative approach for optimization to achieve the goals of the treatment planning process., The time required to develop a good plan and the quality of the plan depend on the experience of the planner; therefore, variation is always observed in the quality of plans and the time needed for planning (efficiency).2, 3, 4, 5, 6 Many attempts have been made to improve the quality and efficiency of IMRT planning: knowledge based-planning (KBP),3, 4, 5, 6, 7, 8, 9, 10 auto-planning,,, iCycle (multicriterial beam profile optimization and beam angle selection),14, 15, 16, 17 and multicriteria optimization.18, 19, 20, 21, 22

In addition to their use in treatment planning, KBP models have been used as quality assurance (QA) tools in clinical trials.23, 24, 25, 26 The Imaging and Radiation Oncology Core (IROC) radiation therapy quality assurance center (RTQA), which reviews all treatment plans of patients enrolled in the NRG Oncology clinical trials, started using the available KBP tool for QA of patients’ treatment plans. All previous KBP-related studies mentioned previously used institutional treatment plans to train the KBP models and used these models as retrospective QA tools. To our knowledge, this is the first study that aims to investigate whether a disease site–specific KBP model trained using treatment plans from a multi-institutional clinical trial (NRG-HN001) can improve the quality of treatment plans of patients enrolled in the same clinical trial.

Methods and Materials

Patients enrolled in the NRG-HN001 clinical trial were randomized and selected according to the patient’s selection criteria, which are described in the protocol. The prescription dose is 69.96 Gy/33 fractions. The trial offers the option of prescribing an intermediate dose of 62.7 Gy to small volume lymph nodes. If the treating physician decides to treat all sites (primary, upper, and lower neck) with a single simultaneous integrated boost IMRT plan, the lower neck should receive 54.12 Gy; if gross nodes are present in the lower neck, the surrounding subclinical region must receive 59.4 Gy. The dosimetric compliance criteria of the NRG-HN001 clinical trial can be found elsewhere.

Fifty treatment plans (submitted by different participating institutions) for patients enrolled in the NRG-HN001 clinical trial were selected (randomly and passed the IROC contouring and dosimetric QA process) to build a KBP model using the commercial RapidPlan algorithm available in the Eclipse Treatment Planning System (TPS) version 13.6.15 (Varian Medical Systems, Palo Alto, CA) in the NRG/IROC cloud environment. The model included all structures that were required to be contoured and used structure names for target volumes and normal structures, as defined by the contouring guidelines of the clinical trial. Most cases submitted under the NRG-HN001 trial from different institutions used volumetric modulated arc therapy and some used static IMRT; treatment plans involved in this study were planned (by the submitting institutions) using different versions of Eclipse TPS and Pinnacle TPS and were delivered using Varian (iX, Trilogy, and TrueBeam) and Elekta (Synergy, Agility, and Versa) machines. Therefore, treatment plans of different centers participating in the clinical trial, used to build the KBP model, represented a wide spectrum of TPSs, planning experience, planning priorities, beam models, and delivery methods. The model was trained and tested for its quality using statistical presentation of the training set. We tested the effect of removing dosimetric outliers on the performance of several models using multiple processes for outlier removal from the original model (which contained outliers); we observed variations in the sparing of OARs in some treatment plans, whereas there were no variations in other treatment plans. This observation supports the observation made by Delaney et al, that outlier removal has minimal effects on the sparing of OARs. Therefore, we decided to use the original KBP model for reoptimization of all treatment plans. We used line objectives (ie, placing lines of optimization objectives along the inferior boundary of the predicted dose-volume histogram [DVH] range) to optimize mean doses to the parotid glands; we also used line objectives and maximum dose point objectives to optimize the maximum doses of different structures. The model was validated using 10 treatment plans and was later used to reoptimize treatment plans of 40 other cases (all 50 plans were not included in the training model and are henceforth referred to as “KBP plans”; “submitted plans” refer to those that were originally submitted by the participating institutions). The fact that the model was only used to predict DVHs and to generate priorities for guiding reoptimization in a single process needs to be emphasized here. Field geometry or delivery methods were not altered during KBP-related reoptimization of treatment plans. The reoptimization process was performed blindly (without looking at the submitted plan) by 1 user. All KBP plans were calculated using the Varian 21 EX machine that is configured in our Eclipse TPS and using a calculation grid of 0.25 cm. All KBP reoptimized plans were normalized so that 95% of target volume receives 100% of prescription dose to ensure consistency in target coverage of KBP plans. DVHs for the aforementioned 50 KBP plans were compared with those of the submitted plans using MIM Software (MIM Vista Corporation). Dose parameters were compared based on the dosimetric compliance criteria of the clinical trial using the nonparametric Wilcoxon signed-rank test. Differences were considered significant at P < .05 (2-sided). A comparison of dose parameters in KBP and treatment plans submitted by a single institution and multi institutions was also performed. The model has been used as a QA tool for treatment plans submitted to this clinical trial.

Results

Figure 1 shows an example of mean DVHs for PTVs (PTV_6996) in the KBP and submitted plans. Table 1 lists the dosimetric parameters of the PTVs and OARs. All PTV dose constraints were met by the KBP and submitted plans. Mean values for dose received by 95% of PTV_6996 were 69.6 ± 1.2 Gy and 69.1 ± 1.7 Gy in the KBP and submitted plans, respectively. The maximum dose of PTV_6996 was found to be 3% higher in the KBP plans than in the submitted plans; however, all plans met the maximum dose dosimetric constraint (<84 Gy). Figure 2 shows the mean DVHs (of all 50 KBP plans and of submitted plans) for the brain stem, spinal cord, optic chiasm, optic nerves (right and left), and parotid glands (right and left). Figure 3 shows a case-by-case comparison of the relative percentage difference for D95% (Gy) of PTV_6996; D0.03cc (Gy) of brain stem, spinal cord, and optic structures; and mean dose to parotid glands in KBP and submitted plans. As observed in Figure 2, mean DVHs for all OARs were lower (fewer doses to certain percent volumes) in the KBP plans than in the submitted plans. This result can also be observed in Table 1; the maximum dose (D0.03cc [Gy]) to the brain stem and spinal cord was lower in 33 of 50 (66%) KBP plans. In 40 of 50 (80%) KBP plans, D0.03cc to the optic structures (chiasm, right and left optic nerves) was lower in the KBP plans than in the submitted plans. The differences in D0.03cc between the KBP and submitted plans were statistically significant for the aforementioned OARs. The differences in mean dose between the KBP and submitted plans were also significant for the right and left parotid glands. Figure 3 indicates that OAR sparing was improved in most cases despite having similar PTV coverage (and in some cases improved PTV coverage). The mean values for D0.03cc to the right and left temporomandibular joints (tm joint_R and tm_joint_L) were also lower in the KBP plans than in the original plans; however, the mean values for D0.03cc to the mandible, brachial plexus, and right and left temporal lobes (temporal lobe_R and temporal lobe_L) were almost similar in the KBP and submitted plans.

Figure 1

Mean dose volume histograms (DVHs) for planning target volume (PTV)_6996 in the submitted and knowledge-based plans.

Table 1

Mean ± SD for PTVs and organs at risk dosimetric parameters in the submitted and KBPs

Structure/dosimetric parameter	Mean ± SD (minimum, maximum)
	Submitted plans	KBP reoptimized
PTV_6996 RT dose	(94 ± 3)%	(95 ± 1)%
PTV_6996 D0.03cc (Gy)	77 ± 2	79 ± 2
PTV_5940 RT dose	(96 ± 2)%	(95 ± 1)%
PTV_5412 RT dose	(96 ± 3)%	(95 ± 2)%
Brain stem D0.03cc (Gy)	52 ± 4 (43, 64)	50 ± 5 (36, 57)
Spinal cord D0.03cc (Gy)	42 ± 4 (31, 45)	40 ± 4 (30, 51)
Optic chiasm D0.03cc (Gy)	35 ± 17 (4, 69)	29 ± 18 (5, 63)
Optic nerve_R D0.03cc (Gy)	37 ± 15 (5, 55)	31 ± 18 (5, 65)
Optic nerve_L D0.03cc (Gy)	38 ± 15 (4, 56)	32 ± 18 (5, 60)
Paratoid_R mean dose (Gy)	32 ± 9 (13, 64)	30 ± 7 (21, 64)
Paratoid_L mean dose (Gy)	36 ± 12 (20, 63)	32 ± 9 (20, 59)
Mandible D0.03cc (Gy)	71 ± 4 (59, 75)	71 ± 4 (59, 79)
Brachial plexus D0.03cc (Gy)	66 ± 4 (57, 77)	66 ± 4 (59, 75)
TM joint_R D0.03cc (Gy)	58 ± 12 (30, 73)	55 ± 13 (25, 76)
TM joint_L D0.03cc (Gy)	60 ± 11 (30, 75)	57 ± 12 (36, 74)
Temporal Lobe_R D0.03cc (Gy)	66 ± 7 (37, 77)	66 ± 9 (38, 84)
Temporal Lobe_L D0.03cc (Gy)	66 ± 5 (55, 76)	65 ± 9 (31, 84)

Abbreviations: KBP = knowledge-based planning; PTV = planning target volume; RT = radiation therapy; SD = standard deviation.

D0.03cc (Gy) is the maximum dose as defined by NRG Oncology clinical trials. Numbers in parentheses are the mnimum and maximum values of the parameter.

Figure 2

Mean dose volume histograms for the brain stem, spinal cord, optic nerve, optic nerve_L, optic chiasm, parotid_R, and left parotid_L in the submitted and knowledge-based planning (KBP) plans.

Figure 3

Percentage relative dose difference for D95% (Gy) of planning target volume (PTV)_6996, D0.03cc (Gy) for brain stem, spinal cord, and optic structures, and mean dose for parotid glands in knowledge-based planning and submitted plans.

On calculating the median absolute and relative percent differences in D0.03cc between the KBP and submitted plans, we found that D0.03cc was lower in the KBP plans than in the submitted plans. The median absolute and relative percentage differences in D0.03cc were as follows: brain stem (2.9 Gy, 5%), spinal cord (2.3 Gy, 5%), optic nerve_R (5.5 Gy, 16%), optic nerve_L (5.2 Gy, 16%), and chiasm (5.1 Gy, 18%). We also calculated the median absolute and relative percentage differences in mean dose to the parotid glands between the KBP and submitted plans and found that the mean doses were lower in the KBP plans than in the submitted plans. The median absolute and relative percentage differences in mean dose to the parotid glands were as follows: parotid_R (3.0 Gy, 10%) and parotid_L (3.2 Gy, 10 %). These results suggest that a relative improvement of at least 5% can be achieved in the dose parameters of different OARs.

The multi-institutional KBP model was also used to perform a test on quality analysis of treatment plans submitted by individual institutions and multi institutions. Table 2 lists average differences between KBP and original plans for various dosimetric parameters in 2 cohorts of patients (17 patients submitted by 1 specific institution and 17 patients submitted from multiple institutions). The 17 patients from multiple institutions were randomly selected to represent the general treatment quality of all the plans submitted to this trial. Differences in dose parameters can be seen between the 2 cohorts; in particular, significant differences were observed in the dose parameters of spinal cord, parotid glands, cochleae, and glottic and supraglottic larynx. These differences indicate specific planning patterns of the single institution.

Table 2

Comparison of average differences of dosimetric parameters in KBP and submitted plans of multi- and single-institution cohorts

Structure	Dosimetric parameter	Multi-institutional plans	Single institution plans	P value
PTV_6996	V69.96 Gy (%)	0.02 ± 0.04	0.00 ± 0.02	.06
Brain stem	D0.03cc (Gy)	–4.38 ± 4.90	–3.83 ± 2.60	.35
Spinal cord	D0.03cc (Gy)	–4.07 ± 4.08	0.23 ± 2.09	.01
Optic nerves	D0.03cc (Gy)	–5.66 ± 8.59	–8.86 ± 6.94	.10
Optic chiasm	D0.03cc (Gy)	–4.15 ± 6.67	–6.18 ± 6.41	.38
Temporal lobes	D0.03cc (Gy)	–2.41 ± 4.75	–4.07 ± 4.51	.11
Parotid glands	Dmean (Gy)	0.25 ± 6.17	–11.33 ± 4.89	<.00001
Cochleas	Dmean (Gy)	–4.11 ± 11.95	–17.04 ± 6.25	<.00001
Larynx GSL	Dmean (Gy)	0.48 ± 9.39	–6.5 ± 3.96	.02

Abbreviations: KBP = knowledge-based planning; GSL = glottic and supraglottic larynx; PTV = planning target volume.

The IROC radiation therapy quality assurance center has developed a work flow that is to be implemented in this clinical trial. According to this, the KBP model will be used to generate predictive DVHs of normal structures in reviewed submitted plans with scores 2 (variation acceptable) or 3 (deviation unacceptable). The predicted DVHs will be compared with the DVHs of the submitted plan obtained from the treating institution. The work flow has been used to review an additional 34 treatment plans (other than the 50 plans used to train the model and the 50 plans used in the comparisons between KBP and submitted plans mentioned previously) so far. Table 3 lists comparisons in terms of the number of treatment plans (the 34 plans evaluated using the KBP- based work flow evaluation) with different scores before and after KBP optimization.

Table 3

Comparison of QA scores before and after KBP model reoptimization

Structure	Dose parameter	Submitted plans			KBP reoptimized plans
		Score 1	Score 2	Score 3	Score 1	Score 2	Score 3
PTV_6996	V69.96 Gy (%)	26	6	2	29	5	0
PTV_6996	D0.03cc (Gy)	32	1	1	34	0	0
PTV_5940	V59.4 Gy (%)	26	8	0	30	4	0
PTV_5412	V54.12 Gy (%)	34	0	0	34	0	0
Brain stem	D0.03cc (Gy)	11	22	1	34	0	0
Spinal cord	D0.03cc (Gy)	31	3	0	34	0	0
Optic nerve_R	D0.03cc (Gy)	33	1	0	34	0	0
Optic nerve_L	D0.03cc (Gy)	32	1	1	34	0	0
Optic chiasm	D0.03cc (Gy)	30	3	1	34	0	0
Mandible	D0.03cc (Gy)	14	18	2	28	6	0
TMjoint_R	D0.03cc (Gy)	34	0	0	34	0	0
TMjoint_L	D0.03cc (Gy)	34	0	0	34	0	0
Temporal Lobe_R	D0.03cc (Gy)	25	4	5	33	0	1
Temporal Lobe_L	D0.03cc (Gy)	27	3	4	31	3	0
Parotid_R	Dmean (Gy)	4	10	20	8	23	3
Parotid_L	Dmean (Gy)	6	2	26	9	20	5

Abbreviations: KBP = knowledge-based planning; PTV = planning target volume; QA = quality assurance.

Discussion

Performance of the NRG-HN001 KBP model was proven to be satisfactory, as improved sparing of many OARs was achieved without sacrificing the dose parameters of the PTVs. It should be noted that the mean values for D0.03cc of PTV_6996 were 3% higher in the KBP plans than in the submitted plans. However, improved sparing of OARs was achieved in some KBP plans, with D0.03cc of PTV_6996 values lower than those in the submitted plans. Our results are based on a single KBP reoptimization of the submitted plans; we did not attempt to reduce the D0.03cc to the PTV through repeated optimization, as is routinely done in clinics. The model proved that a relative improvement of at least 5% can be achieved in the dose parameters of many OARs; therefore, it can be a very helpful tool for improving the quality and efficiency of treatment planning in patients enrolled in clinical trials.

The disease site-specific, multi-institutional KBP model was trained using treatment plans from the clinical trial itself. Therefore, this model might be advantageous over single-institutional models, as extrapolations based on contouring differences that would possibly result in poor DVH predictions could be avoided in multi-institutional models. The results listed in Table 2 support this argument, as bias of data submitted from single institutions was clearly identified. Using models built with data submitted from multi-institutions for QA purposes also provides a more realistic peer review and averages out specific opinions and planning patterns of individual institutions. We do not believe that the improvement in sparing of OARs is related to differences in treatment planning systems or accelerator models. Multileaf collimators (MLC) parameters, especially leaf transmission, are expected to affect the quality of head and neck IMRT treatment plans., However, because some of the submitted plans used similar MLC and linear accelerator (LINAC) models, as the standard machine used in the KBP model and other plans used slightly different MLC/LINAC, the effect of the MLC/LINAC difference in the optimization should be minimal. This is merely speculation and should be investigated in a systematical manner in an independent study. Chang et al reported better sparing of parotid glands, comparable target coverage, and inferior performance in achieving spinal cord and optic chiasm priorities with RapidPlan than with manual planning. Improvement in sparing of OARs in head and neck IMRT plans was also achieved using other methods, such as auto-planning, iCycle,, and multicriteria optimization. In our study, improvement in sparing the brain stem and spinal cord was observed in 66% of KBP plans, whereas improvement in sparing the optic structures was observed in 80% of KBP plans.

The results of this study indicated that the OAR sparing achieved can vary among patients, probably because of the variation in the overlap of normal structures with target volumes and the distance between target volumes and normal structures. Variation in the improvement of OAR sparing can also be attributed to the differences in planning experience at different institutions, suggesting that the quality and consistency of treatment planning can be improved using KBP models. Tol et al suggested the use of the RapidPlan model as an online QA tool in clinical trials for determining whether a particular treatment plan can be accepted for a trial. They suggested evaluation of the mean dose to the contralateral parotid gland, upper larynx, and oral cavity; if the plan passes, it can then be accepted for the trial. Tol et al also used a KBP model as a prospective patient-specific treatment plan QA tool for treatment plans in the European Organisation for Research and Treatment of Cancer (EORTC) EORTC-1219-DAHANCA-29 multi-institutional trial. The EORTC study is different from our study in 2 aspects: it used treatment plans from a single institution to develop the KBP model and it used 2-arc volumetric modulated arc therapy for replanning the submitted cases, although some of the original plans were planned using static IMRT.

The results of this study and previous studies show that, in most cases, the achieved dose parameters can be improved further, even if the dose constraints (mandated by the clinical trial) have been originally achieved. The clinical relevance of the gain in OAR sparing versus PTV dosimetric parameters is a clinical decision, and it is up to the treating physician. However, we believe that further investigations are needed to determine the influence of these improvements on tumor control probability and normal tissue complication probability, and these investigations are beyond the scope of this study.

Our study indicated that a relative improvement of at least 5% can be achieved in the maximum dose to the brain stem, spinal cord, and optic structures and in the mean dose to any parotid gland. Therefore, we believe that the value of 5% (3.5 Gy) of the prescription dose (69.96 Gy) should be used as a threshold for providing the QA result back to the submitting institutions for quality improvement. This threshold level of 5%, which is the basis of calibration dosimetry, is deemed to be clinically relevant.32, 33, 34, 35, 36, 37 Results of the QA process would be provided to the participating institution if improvement is observed in sparing of at least 5 structures, including the brain stem, spinal cord, optic structures, and parotid glands. The model would be available on the NRG Oncology website, so that it can be directly used in treatment planning for patients enrolled in this clinical trial, and this may help improve the efficiency, quality, and consistency of treatment planning and hopefully would improve outcome. Institutions that do not have access to KBP tools may use any of the other aforementioned advanced treatment planning tools to improve the quality and efficiency of treatment planning. Table 3 indicates improvement in QA scores for brain stem, mandible, temporal lobes, and parotid glands.

Conclusions

A disease site–specific, multi-institutional KBP model was built for the NRG-HN001 clinical trial, and it was proved to improve the sparing of at least 7 OARs (brain stem, spinal cord, optic nerves R &L, chiasm, and parotid glands R &L) in 33 of 50 (66%) cases. This model will be used as a prospective offline QA tool and will be made available on the NRG Oncology website.