A multinational audit of small field output factors calculated by treatment planning systems used in radiotherapy.

BACKGROUND AND
PURPOSE: An audit methodology for verifying the implementation of output factors (OFs) of small fields in treatment planning systems (TPSs) used in radiotherapy was developed and tested through a multinational research group and performed on a national level in five different countries.
MATERIALS AND METHODS: Centres participating in this study were asked to provide OFs calculated by their TPSs for 10 × 10 cm2, 6 × 6 cm2, 4 × 4 cm2, 3 × 3 cm2 and 2 × 2 cm2 field sizes using an SSD of 100 cm. The ratio of these calculated OFs to reference OFs was analysed. The action limit was ±3% for the 2 × 2 cm2 field and ±2% for all other fields.
RESULTS: OFs for more than 200 different beams were collected in total. On average, the OFs for small fields calculated by TPSs were generally larger than measured reference data. These deviations increased with decreasing field size. On a national level, 30% and 31% of the calculated OFs of the 2 × 2 cm2 field exceeded the action limit of 3% for nominal beam energies of 6 MV and for nominal beam energies higher than 6 MV, respectively.
CONCLUSION: Modern TPS beam models generally overestimate the OFs for small fields. The verification of calculated small field OFs is a vital step and should be included when commissioning a TPS. The methodology outlined in this study can be used to identify potential discrepancies in clinical beam models.

Introduction

The challenges of small field dosimetry in photon beams have been investigated for more than two decades [1], [2] and gained importance with the implementation of advanced treatment techniques such as intensity modulated radiotherapy (IMRT), stereotactic (body) radiotherapy (SBRT) and stereotactic radiosurgery (SRS). The main issues in accurate determination of field OFs in these beams are the loss of lateral charged particle equilibrium and source occlusion, as well as detector related effects such as volume averaging and the difference between the density of the detector material and water [3], [4], [5], [6], [7], [8], [9]. In 2008, Alfonso et al. introduced a formalism on the determination of small field OFs using detector and field size specific correction factors [10]. Several research groups have investigated these factors for a wide range of passive and active detectors [7], [11], [12], [13], [14], [15]. Recently, a new code of practice on small static fields used in external beam radiotherapy has been published [16].

Accurate experimental determination of OFs in small fields is only one component of the TPS to calculate the dose correctly. The other part is the implementation or modelling of these OFs in a TPS, which is critical for the dose calculation accuracy. Dose calculation algorithms have evolved from simple factor based dose calculation as found in [17], [18], to model-based algorithms [3], [19], [20], [21], [22], stochastic linear Boltzmann transport equation solvers such as Monte Carlo algorithms [23], and deterministic linear Boltzmann transport equation solvers [24], [25]. These model-based algorithms rely on an accurate source model describing the energy fluence entering the patient for a given aperture. Source models are usually optimized based on basic beam data provided by the user. For the optimization of small field OFs, e.g. for an accurate modelling of the penumbra, the aforementioned effects in small fields have to be considered. It is expected that the accuracy of source models for small fields will increase with improvements in the measurement accuracy of small fields accounting for ion chamber and fluence corrections [3], provided that manufacturers are willing to optimize their source models for small fields. However, small field calculations for TPSs are usually not the focus of standard commissioning procedures and therefore may be prone to errors.

National and international organizations have provided recommendations on acceptance testing, commissioning and quality assurance of medical TPSs [26], [27], [28]. These documents outline the individual steps in validating the general functionality of the TPS and especially dose calculation accuracy. All of these documents recommend the verification of OFs by recalculation and comparison against measurements as OFs have a direct impact on the number of monitor units necessary to deliver the prescribed dose. E.g. TECDOC-1583 suggests the comparison of calculated and measured OFs for field sizes ranging from 3 × 3 cm2 to 40 × 40 cm2 using a tolerance of ±2% [27]. A comprehensive data set on small field OFs of various treatment machines produced by different vendors has been determined by the Imaging and Radiation Oncology Core Houston QA Centre (IROC-Houston QA Centre, formerly the Radiological Physics Centre). OFs were measured for field sizes down to 2 × 2 cm2 at a depth of 10 cm in water at 100 cm SSD on more than 150 linear accelerators as a part of the on-site visits. These measurements were made using a cylindrical ionization chamber. The measured OFs were grouped according to energy and linear accelerator manufacturer. Even when grouped across multiple accelerator models, the measured OFs were highly consistent for a given energy and manufacturer. The average standard deviation of the output factor for a given manufacturer and energy was less than 0.5% except for the 2 × 2 cm2 fields which was 0.7%, indicating that the reference OFs were highly consistent and descriptive of the linacs. Besides that, average differences between calculated and measured output factors for the 2 × 2 cm2 field ranging from 1.3% to 5.8% depending on the linac vendor and beam energy were observed [29], [30].

A coordinated research project was launched to develop audit methodologies for testing the implementation of treatment techniques with different complexities. The aim of this project was to make these methodologies available to national external audit groups and assist them with the local development of these audits. In particular, dosimetry audit of small fields was of interest because of the prevalence of difficulties both in conducting small field dose measurements as well as in computing them. One contributing factor to observed errors with small fields is the agreement between calculated and measured lateral small beam profiles. Discrepancies of more than 3 mm have been observed which could potentially lead to an unsatisfactory accuracy in dose calculation of advanced treatment techniques [31]. Another aspect is the accuracy of calculated small field output factors (OFs) using treatment planning systems (TPSs) employed in clinical practice, which is focus of this work.

The results of this audit, which was designed within a multinational coordinated research project, tested with national audit groups in a multi-centre setting and implemented on a national level in a few countries, are given.

Materials and methods

Audit development

A simple dose calculation exercise was designed by a group of IAEA consultants to assess the TPS model accuracy of small field OFs. A multicentre study was initiated to validate the audit procedure, clarity of instructions and completeness of the reporting form. The exercise was performed among all of the centres participating in the IAEA coordinated research project. They tested the methodology in their institutions in order to demonstrate the feasibility of implementing this audit on a national level within their countries. A total of 17 institutions in 14 countries (Algeria, Brazil, China, Cuba, Czech Republic, India, Poland, Thailand, Austria, Belgium, Finland, Sweden, UK, and USA) participated together in the multicentre phase of this audit. The participating institutions were considered as centres of excellence in this field. Finally, this audit was performed on a national level in Brazil, China, Czech Republic, India and Poland obtaining results from a total of 103 institutions. A summary of the treatment machine manufacturers and models involved in this project is provided in Table 1.

Table 1

Summary of treatment machines grouped by manufacturer and model.

Audit run	Linac Manufacturer	Linac Model	Number of Linacs	Nominal beam energies[MV]
Multicentre run
	Varian	Clinac	8	6, 15, 18
		TrueBeam	2	6
		Trilogy	1	6
		Novalis STx	2	6, 15
		TrueBeam STx	3	6, 15
	Elekta	Synergy	5	6, 10
		Precise	1	6
	Siemens	Primus	1	6
National runs
	Varian	Clinac	59	6, 10, 15, 18, 20
		TrueBeam	16	6, 10, 15, 20
		Trilogy	11	6, 10
		Novalis STx	4	6, 15
		TrueBeam STx	4	6, 10, 15
		Unique	6	6
	Elekta	Synergy	47	6, 10, 15, 18
		Precise	7	6
		Axesse	2	6
		Versa HD	1	6
	Siemens	Artiste	17	6, 15
		Primus	5	6
		Oncor	4	6

Audit methodology

The participants were asked to calculate the monitor units (MUs) necessary to deliver 10 Gy on the central axis at 10 cm depth, 100 cm SSD in water for 5 MLC shaped fields (10 × 10 cm2, 6 × 6 cm2, 4 × 4 cm2, 3 × 3 cm2 and 2 × 2 cm2) using their TPSs. For Varian linacs with a tertiary MLC, the field size was defined by the MLC while the secondary jaws remained at a 10 × 10 cm2 field size. The dose per MU for each field was calculated and normalized to the 10 × 10 cm2 field. These calculated OFs were compared to reference output factors published by IROC-Houston QA Centre for the same beam energy and linac manufacturer [29], [30]. References to these publications were included in the instructions for the participating centres. For analysis, the ratio of each institution’s TPS calculated OFs to the reference OFs was determined for each field size and nominal beam energy. An action limit of ± 3% for the 2 × 2 cm2 field and ± 2% for fields larger than 2 × 2 cm2 was defined. These action limits were derived using four times the average standard deviation of the reference data. The results were grouped by nominal beam energy (≤6 MV and >6 MV) and by TPS – treatment machine combination. The multicentre run was performed at the end of 2013 and the national runs were initiated in 2014. Data collection was completed in 2016.

Statistical analysis

For the 2 × 2 cm2 and 3 × 3 cm2 fields, Linac – TPSs combinations with more than four data points were statistically analysed using the Shapiro-Wilk test for normality. The results of this test indicated that normal distribution could not be assumed for all investigated Linac – TPS combinations. Therefore, one-sample Wilcoxon signed-rank test was used to investigate whether the data differed significantly from unity. A significant difference would indicate that the TPS, for specific linac-TPS combinations, produced OFs that were, on average, systematically different and biased as compared to reference OFs. The p-values of the Wilcoxon test were adjusted for multiple testing using a Bonferroni correction. A p-value of less than 0.05 was considered to be significant. The statistical computation was performed in R (version 3.3.3, The R Foundation for Statistical Computing, Vienna, Austria).

Results

In total, 856 OFs calculated by various TPSs were collected. On average the TPSs tended to overestimate the OFs compared to the reference OFs, with increasing deviations as the field size decreased. Seven beams had a deviation of calculated OFs compared to the reference OFs larger than 10% for the 2 × 2 cm2 field. The entire set of ratio data are shown in Fig. 1 and Table S1 in the supplementary material. Of significance was the large increase in the spread of the data as the field size decreases indicating greater variability in TPS calculated OFs.

Fig. 1

A plot of the ratio of TPS calculated to reference OFs (N = 856) as a function of field size for all investigated beams (N = 215). The red lines are the action limits of ±3% for the 2 × 2 cm2 field and ±2% for fields larger than 2 × 2 cm2. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

IROC-Houston QA Centre’s reference data were verified by experimental determination of OFs for a subset of treatment machines which were investigated in the multicentre run within this coordinated research project using different detectors. The largest deviations compared to the IROC data were found for the 2 × 2 cm2 field. For this field, on average, the OFs where higher by 0.7% (max. 2%) and 1.2% (max. 2.4%) for Varian and Elekta treatment units, respectively.

Audit results by nominal beam energy

The majority of the data was generated using photon beams with a nominal energy of 6 MV. As presented in Table 2, a clear trend of increasing deviation of OFs calculated by the institutions to reference OFs towards smaller field size was observed. Consequently, the percentage of data points exceeding the action limit increased accordingly. For the 6 MV beam 2 × 2 cm2 fields, the mean ratio of calculated OFs to reference OFs was 1.027 ± 0.004 and 1.020 ± 0.002 for the multicentre and national runs, respectively. For this field size and beam energy, 30% of the data points were exceeding the action limit of ±3% for the national runs, while for the multicentre run 35% deviations were greater than ±3%. For the 2 × 2 cm2 fields with nominal beam energies higher than 6 MV were characterized by a lower mean ratio of calculated OFs compared to reference data, with values of 1.014 ± 0.007 and 1.017 ± 0.006 for the multicentre and national runs, respectively.

Table 2

Summary of results of the ratio of calculated OFs with respect to reference OFs for the audit performed for multicentre level and for national level. The data are grouped by nominal beam energy and field size. The mean value (mean) as well as the standard deviation (sd), the standard deviation of the mean (sdm), the number of data points (N) and the percentage of data points exceeding the action limit are provided. The action limits were ± 3% for the 2 × 2 cm2 field and ± 2% for fields larger than 2 × 2 cm2.

	Field size (cm × cm)
	2 × 2	3 × 3	4 × 4	6 × 6
Nominal beam energy 6 MV
Multicentre run
mean	1.027	1.018	1.010	1.002
sd	0.017	0.008	0.007	0.006
sdm	0.004	0.002	0.002	0.001
N	20	20	20	20
% exceeding the action limit	35%	35%	10%	0%
National runs
mean	1.020	1.012	1.006	1.001
sd	0.028	0.017	0.014	0.009
sdm	0.002	0.001	0.001	0.001
N	133	137	137	137
% exceeding the action limit	30%	31%	12%	4%
Nominal beam energy >6 MV
Multicentre run
mean	1.014	1.017	1.007	1.005
sd	0.021	0.014	0.014	0.005
sdm	0.007	0.005	0.005	0.002
N	9	9	9	9
% exceeding the action limit	33%	33%	22%	0%
National runs
mean	1.017	1.008	1.003	1.001
sd	0.045	0.017	0.008	0.006
sdm	0.006	0.002	0.001	0.001
N	49	49	49	49
% exceeding the action limit	31%	20%	4%	2%

Audit results by combination treatment unit and TPS

The results grouped by treatment unit and TPS are presented in Table S2 in the supplementary material. For the 2 × 2 cm2 field in the national runs, mean values of the ratio of calculated and reference OFs which were significantly different from unity i.e. for Varian linacs – Eclipse TPS (1.030 ± 0.003, p-value < 0.01), Elekta linacs – Monaco (1.013 ± 0.003, p-value < 0.01), and Siemens linacs – Oncentra (1.033 ± 0.006, p-value = 0.016). Examples of two most common combinations of linac manufacturer and TPS can be seen in Fig. 2.

Fig. 2

A graphical representation of the ratio of calculated to reference OFs. Each data point represents a different beam. The combination Varian linac – Eclipse is depicted on the left and the combination Elekta linac – Monaco is depicted on the right. The red lines are the action limits of ±3% for the 2 × 2 cm2 field and ±2% for fields larger than 2 × 2 cm2. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Discussion

This simple audit methodology allows a quick assessment of the accuracy of the employed dose calculation algorithm and beam model in small photon fields. Particularly for advanced radiotherapy such as IMRT, SBRT and SRS, the use of small fields is common, and so the accuracy of small field OFs directly impacts the accuracy of patient dose calculation. Naturally, these are not the only contributing factors to the dose calculation accuracy of TPSs for treatments, other factors, such as tissue heterogeneity, have a substantial impact as well. Nevertheless, small field OFs must be verified as an integral part of the TPS commissioning process. Such testing ensures accurate values, and also helps to identify the limits of the employed beam model in terms of minimum field size. Most TPSs allow the specification of parameters, e.g. minimum segment area or minimum leaf gap, which prevent the use of field sizes too small for accurate dose delivery.

Separating the results by nominal beam energy revealed a rate of results exceeding the action limit of 30% for the 2 × 2 cm2 field for beams with nominal beam energy of 6 MV, compared to 31% for beam energies higher than 6 MV for the audit performed on national levels, which is consistent with observations during the multicentre run. The majority of data was provided for the combination Varian linac – Eclipse for a nominal Energy of 6 MV. This particular combination was characterized by a mean 2 × 2 cm2 field calculated to reference OFs ratio of 1.030 and a standard deviation of the mean of 0.003 for the national run. Kerns et al. also reported measured OFs being consistently lower than TPS calculated OFs, on average 1.6% over field sizes ranging from 2 × 2 cm2 to 6 × 6 cm2 with 64% showing a discrepancy larger than 1% [32]. For the 2 × 2 cm2 field, standard deviations of up to 7.9% for OFs calculated by institutions have been reported by Followill et al. [29], [30].

Eclipse allows the user to perform beam modelling autonomously without interacting with the TPS manufacturer, which could be the reason for the larger offset and spread of data. In order to model small fields adequately in Eclipse, head scatter parameters need to be adjusted and after that, the dose calculation accuracy needs to be verified for all field sizes [30], [33]. For other TPSs, e.g. Monaco or Oncentra for Elekta machines, for which the beam modelling process is performed by the manufacturers based on basic beam data provided by the users the mean ratio is more accurate compared to Eclipse. This should not mean that accurate beam modelling is not possible using Eclipse, but that generating beam models for small beams is a task which needs to be performed carefully and that assistance by the manufacturers can improve the results. There are examples in the literature [34] and also in this data set (see Fig. 2) where good agreement between calculated and published OFs was achieved for the combination Varian linac and Eclipse TPS. For Monaco the mean ratio of the 2 × 2 cm2 field was significantly higher than unity, indicating that there is also room for improvement. In principle, this dataset might allow institutions to understand potential limitations in beam modelling of their treatment machines, and refinements of the investigated dose calculation algorithms in general, but this would go beyond the scope of this work.

Another reason for the increasing deviation of calculated OFs compared to reference OFs can be that the input data used to determine the beam model parameters did not extend to small fields. Some TPSs rely solely on measurements of relatively larger fields (e.g., only as small as a 4 × 4 cm2 field) and then extrapolate for smaller field sizes. A similar situation can appear when a TPS was initially commissioned for large fields only and the recommissioning of the system was not performed for the introduction of treatment techniques using small fields. Moreover, the input data could have deviated substantially from reference data either due to actual machine characteristics or inaccurate data acquisition. As part of the follow-up for beams exceeding the action limit, the institutions were asked to provide measured OFs for small beams. Only eight follow-up results were available at the time when this paper was written, therefore a final conclusion cannot be drawn. However, the majority of follow-up results showed an agreement of measured OFs by the institutions with reference OFs within the expected uncertainty, which was also found in the multicentre run. For one institution where a substantial deviation from reference OFs was found, the calculated OFs were higher and measured OFs were lower compared to reference data. It was determined that the institution had used a detector which was not suitable for the field size and the experimental data were either implemented incorrectly or not considered at all. Results thus far have consistently shown that the reference values have been accurate representations of the linacs examined, within the expected uncertainties of the reference set.

On average, the ratio of calculated OFs to reference OFs are higher than unity and increase with decreasing field size. This suggests that the dose in small fields calculated by TPSs is frequently overestimated. Effects on the dosimetric quality of the patients’ treatment plans depend on the applied treatment technique. For example, for stereotactic treatments the deviations of calculated OFs compared to actual OFs can add up to unacceptable levels as the treatment plans consists mainly of small fields. This may or may not be the case for static and dynamic IMRT techniques since these treatment techniques consist of a combination of large and small fields. IROC-Houston QA Centre recently commissioned a recalculation system based on the data collected during on-site visits. This system was used to recalculate more than 200 treatment plans for their head and neck IMRT phantom audit and revealed that 17% of the investigated treatment plans had considerable calculation errors. Considering only those institutions who failed IROC-Houston’s acceptance criteria for irradiation of the head and neck phantom, 68% had calculation errors [35]. While the underlying cause of these computational errors has yet to be elucidated, a contributing factor is potentially the high frequency of small field output factor errors.

Looking at the data from a different perspective raises the question whether discrepancies between calculations and measurements can be explained by systematic errors in the reference data. Recently published correction factors for the A16 ion chamber [16] indicate this chamber is reliable for the investigated field sizes. Other publications using the same reference data have shown that the difference between calculated and measured output factors is more pronounced for fields limited by the MLC with the jaws open to a larger field size which is the more challenging situation compared to fields limited by jaws and MLC [32]. Therefore, the discrepancies are more likely to be caused by inaccurate dose calculation. This work was restricted to a minimum field sizes of 2 × 2 cm2 due the available reference data in [29,30]. Even larger deviations of calculated OFs compared to measured OFs can be expected for smaller field sizes, e.g. 1 × 1 cm2, necessitating even more careful commissioning of TPSs.

In conclusion, it has been shown that OFs of small photon fields generated by TPSs often differ substantially from measured reference OFs. The correct implementation of OFs of small fields in TPSs needs to be carefully validated by the user in light of the high frequency of this error. The methodology proposed in this work can be used for this purpose. Special care must be taken if the users alone are responsible for creating their beam model without the support of the TPS manufacturer. In any case, the final responsibility concerning the accuracy and utilization of the beam model lies with the clinical medical physicist.