The current status and shortcomings of stereotactic radiosurgery.

Background: Stereotactic radiosurgery (SRS) is a common treatment for intracranial lesions. This work explores the state of SRS treatment delivery to characterize current treatment accuracy based on treatment parameters.
Methods: NCI clinical trials involving SRS rely on an end-to-end treatment delivery on a patient surrogate (credentialing phantom) from the Imaging and Radiation Oncology Core (IROC) to test their treatment accuracy. The results of 1072 SRS phantom irradiations between 2012 and 2020 were retrospectively analyzed. Univariate analysis and random forest models were used to associate irradiation conditions with phantom performance. The following categories were evaluated in terms of how they predicted outcomes: year of irradiation, TPS algorithm, machine model, energy, and delivered field size.
Results: Overall, only 84.6% of irradiations have met the IROC/NCI acceptability criteria. Pass rate has remained constant over time, while dose calculation accuracy has slightly improved. Dose calculation algorithm (P < .001), collimator (P = .024), and field size (P < .001) were statistically significant predictors of pass/fail. Specifically, pencil beam algorithms and cone collimators were more likely to be associated with failing phantom results. Random forest modeling identified the size of the field as the most important factor for passing or failing followed by algorithm.
Conclusion: Constant throughout this retrospective study, approximately 15% of institutions fail to meet IROC/NCI standards for SRS treatment. In current clinical practice, this is particularly associated with smaller fields that yielded less accurate results. There is ongoing need to improve small field dosimetry, beam modeling, and QA to ensure high treatment quality, patient safety, and optimal clinical trials.

A substantial number of institutions fail to meet IROC/NCI standards for SRS treatment delivery.

Smaller fields in SRS treatment yield less accurate results in phantoms.

Pencil beam algorithms were more associated with failing phantom results.

With the increased prevalence of stereotactic radiosurgery (SRS) as standard of care for brain metastasis, understanding the state of this technique and particularly the impact of treatment options that influence delivery accuracy is vital. While challenges to SRS treatments are known, eg, small field dosimetry and patient setup, the impact of these challenges, and prevalence of associated treatment shortcomings are not well studied or understood. This study highlights the state of accuracy of SRS across the radiotherapy community and identifies important treatment parameters that predict treatment accuracy. A substantial portion (15%) of the radiotherapy community fails to meet treatment delivery standards; while the field has progressed in both general knowledge and technology, challenges continue to limit the improvement of this treatment modality. Several key features are highlighted in predicting inaccurate treatment deliveries, specifically pencil beam algorithms and small fields, which should be approached with care when used for SRS treatments.

Stereotactic radiosurgery (SRS) is a standard-of-care treatment for indications such as metastatic brain lesions that is supported by extensive clinical trial data.[1-5] SRS treatments have many technical differences from traditional radiotherapy because of the small tumor volumes targeted and the need for precise target localization. These special requirements have resulted in the development of unique treatment systems for SRS and, as a result, it is delivered with a wide range of different treatment delivery platforms and collimators, and the treatment dose is calculated using a wide range of treatment planning systems and dose calculation algorithms.

Regardless of the delivery platform and dose calculation method, SRS involves many technical challenges, including difficulties with small field dosimetry, dose calculation accuracy, and precise patient setup. Ensuring that the correct dose is delivered to the right location in SRS is of particular importantance[6-9] (and many tools have been developed to aid in this[13-17,19]) not only because of the unique challenges of this treatment, but also because of the inability to correct any error later in treatment. Despite this, radiotherapy institutions have been found to be routinely lagging behind current quality assurance recommendations,[10] both in the criteria used for assessing SRS plans and in dealing with identified plan inaccuracies.[11,12]

Particularly robust verification of plan accuracy is achieved with an independent end-to-end evaluation.[18,20-22] The largest program that supports such a test is the Imaging and Radiation Oncology Core (IROC), which supports clinical trials involving radiotherapy from the National Cancer Institute (NCI). IROC provides a standardized end-to-end SRS patient surrogate (phantom; Figure 1) as a credentialing test that serves as a prerequisite for institutions to participate in NCI-sponsored clinical trials using SRS. A phantom is sent to an institution where it is treated like a patient, undergoing radiotherapy planning and delivery according to the clinic’s technique. After irradiation, it is returned to IROC and evaluated to see how the delivered and planning dose compare.

Figure 1.

IROC Houston’s two SRS head phantoms: (a) water-filled with a dosimetric and imaging insert and (b) single insert that is constructed with high-impact poly. The phantom contains a 1.9 cm spherical target made of solid water where dose is measured with two single-loaded TLDs and two planes of GAFchromic film. Abbreviations: IROC, Imaging and Radiation Oncology Core; SRS, stereotactic radiosurgery; TLD, thermoluminescent dosimeters.

The results of these credentialing tests can elucidate the pervasiveness and degree to which the challenges in SRS, particularly how the various delivery and dose calculation platforms, affect current clinical practice. In this study, we reviewed the most recent 1072 SRS phantom irradiations using univariate analysis and random forest models[23-26] to explore the state of SRS treatment delivery as well as determine underlying trends and features that lead to suboptimal delivery.

Materials and Methods

IROC Phantom

IROC’s SRS head phantom contains a 1.9 cm spherical target made of solid water. The accuracy of the delivered dose in the target is measured with two single-loaded thermoluminescent dosimeters (TLDs) and two planes (coronal and sagittal) of GAFchromic film. The TLD provide a highly precise[27] assessment of dose accuracy at two points in the target; the film provides planar assessment of the dose distribution.

Institutions are instructed to deliver a treatment consistent with their clinical practice with a maximum target dose of ~30 Gy. The process includes immobilization, treatment planning, localization, and delivery. To pass, the measured point dose (via TLD) must agree with the treatment planning system (TPS) calculation within 5%, and ≥85% of pixels must pass a 5%/3 mm gamma analysis on each of the two film planes. Figure 1 shows the two types of SRS head phantoms available: the original SRS head phantom is water-filled with a dosimetric and imaging insert that must be interchanged between simulation and treatment, while the newer design (available post-2017) has a single insert for both imaging and dosimetry that is constructed primarily from high-impact poly. Both phantom types are used interchangeably and have identical targets and dosimetry.

Dataset

SRS phantom data were collected from 2012 to October 31, 2020. Nine values were extracted for each irradiation: overall phantom performance (pass/fail) status, average TLD ratio (measured:calculated), average percentage of pixels passing gamma for both planes (gamma analysis was implemented after 2013), as well as irradiation date, TPS algorithm,[28] treatment machine, beam energy, collimator, and the delivered field size (while the phantom has a single target size, different institutions treated larger or smaller volumes; this was acceptable as long as the delivered dose matched the intended dose).

Irradiation date was defined only based on the year and considered as a categorical variable. Treatment machine was defined based on machine class from the dosimetric characteristics in IROC’s reference data.[29,30] Collimators were divided into three categories: cone of any size (including variable), high-definition MLC (leaf width <0.5 cm; HD120, micro MLC models, Incise2), and low-definition MLC (leaf width ≥0.5 cm; Millennium 120, Elekta Agility, Elekta 80, Tomotherapy binary MLC). The delivered field size was calculated from the average extent of the 50% isodose line based on the film measurements and averaged over all three-profile directions (superior-inferior, right-left, and anterior-posterior). Specifically, the 50% isodose line was calculated based on the average dose across 80% of the tumor extent. Demographic data are shown in Table 1.

Table 1.

Pass Rate of Treatment Parameters

Category	Constituents	N (% of Total Samples)	Pass Rate (%)
	Truebeam	419 (39.1)	84.5
	Trilogy	162 (15.1)	82.7
	Gammaknife	130 (12.1)	89.2
Treatment machine class	Varian Base	121 (11.3)	81.0
	Cyberknife	98 (9.1)	87.8
	Elekta Agility	60 (5.6)	83.3
	6	526 (49.1)	85.0
	6 SRS	60 (5.6)	81.7
Beam energy	6 FFF	248 (23.1)	84.3
	Co-60	129 (12.0)	89.2
	10 FFF	61 (5.7)	78.7
	Pencil Beam	231 (21.6)	75.8
	Monte Carlo	76 (7.1)	90.8
TPS algorithm	AAA	285 (26.6)	86.0
	S/C	112 (10.5)	85.7
	GBBS	66 (6.2)	87.9
	Measured	172 (16.0)	90.7
	Cone	231 (21.6)	78.8
Collimator group	High definition	205 (19.1)	87.3
	Low definition	374 (34.9)	85.8

Abbreviations: SRS, stereotactic radiosurgery; TPS, treatment planning system; FFF, flattening filter-free; GBBS, grid-based Boltzmann solver.

Analysis

We first evaluated the relationship between treatment conditions and the performance (pass/fail rate) of the phantom using univariate analysis (Pearson chi-squared test with Tukey post hoc analysis; IBM SPSS 24). Follow-up analysis was done using random forest methods to predict pass/fail (classification), as well as average TLD ratio and average percentage of pixels passing gamma (regression), through the above categories. For all random forest analyses, missing data were imputed using 100 trees for all random forest models and then upsampled to ensure a balanced dataset between pass and fail. In addition, four hyper-parameters (number of trees, number of variables split to randomly sample for each descending node, percentage of samples to train on, and minimum number of samples within the terminal nodes) were tuned to minimize out-of-bag (OOB) error. These hyperparameters were tuned over the following hyperspace: number of trees (200-1000), number of variables split to randomly sample for each descending node (2-6), percentage of samples to train on (55%, 63.2%, 70%, 80%), and minimum number of samples within the terminal nodes (1-20). Since the random forest algorithm is self-validating through bootstrap aggregating, we split our dataset into a training (70%) and a testing set (30%). The implementation of this random forest classification was run three times to assess fluctuations within the data and results. Random forest was performed using R (4.0.2) via the ranger and missRanger packages.[31-33]

Results

For the 1072 irradiations of the IROC SRS phantom between 2012 and 2020, the average pass rate was 84.6%, indicating that more than 15% of institutions failed to deliver their intended dose accurately. Of the 164 failures, 56.4% failed due to point dose assessment only (TLD; typically indicating the wrong dose was delivered to the right location), 30.3% failed due to planar assessment only (typically indicating that the right dose was delivered to the wrong location), and 13.1% failed due to both point and planar assessment (indicating multiple or substantial issues). The average dose ratio in the target (measured/predicted) and planar agreement (percentage of pixels passing gamma) were 0.982 ± 0.035 and 95.6 ± 6.8%, respectively, indicating that the dose was, on average, underestimated by 1.8%. Figure 2 shows the evolution of the pass rate, average TLD ratio, and number of irradiations per year. The passing rate and percentage of pixels passing gamma have not significantly changed (regression analysis, P = .975 and P = .415, respectively), while the average TLD ratio (point dose) has steadily improved, moving closer to 1.000, at a rate of 0.002 per year (significantly positive sloping trend line, P = .005). The relationship between each of the different treatment parameters and TLD ratio and gamma results is shown in Figure 3.

Figure 2.

a) The average measured to delivered point dose (average TLD ratio) value for the SRS phantom through the years b) Pass rates for the SRS head phantom through the years with an overall average of 84.6% (solid line) combined with the number of irradiations performed per year (dashed line). The error bars presented in graph a represent ± one standard deviation. There has been an overall increase in point dose performance relative to a site’s dose calculation but passing rates have remained constant over the span of our data set.

Figure 3.

Inter-comparison of irradiation year, average TLD ratio, average percentage of pixels passing gamma, and field size. The lines represent the passing criteria for the SRS head phantom. Noteworthy are that as the irradiation year progresses, average TLD ratio begins to tighten around 1.0 (panel a). In addition, failures in the average TLD ratio tend to be more prevalent for smaller field sizes (≤ 3.0 cm; mid-bottom panel).

Univariate analysis showed that the institutions’ TPS algorithm (P < .001), collimator (P = .024), and field size (P < .001) were significantly associated with phantom performance, while other variables, including energy (P = .531), machine (P = .832), and year of irradiation (P = .118) were not statistically significant (Table 2). In particular, calculations based on a pencil beam algorithm had a 10%-15% lower pass rates than other algorithms (AAA: P = .01, GBBS: P = .015, Monte Carlo: P = .001, Measured: P < .001, and Superposition/Convolution: P = .015); no other differences existed between TPS algorithms. Irradiations delivered with cone collimators had an 8.5% lower pass rates than high definition MLC collimators (p = 0.035), and 7% lower pass rates than low-definition collimators (although not significant; p = 0.055); low definition and high definition MLCs showed no difference. Finally, treatments that used field sizes <3 cm were significantly less likely to pass (15.3% reduction in pass rate) than irradiations done with larger fields.

Table 2.

Detailed Output Parameters of Algorithm and Collimator

TPS Algorithm	Pencil Beama	Monte Carlo	AAA	S/C	GBBS	Measurement-based
N	231	76	285	112	66	172
% (N)	21.6%	7.1%	26.6%	10.5%	6.16%	16.0%
Average TLD ratio	0.967	0.974	0.992	0.981	0.999	0.987
Average % pixels passing	94.8%	96.6%	95.1%	94.0%	95.8%	97.8%
Pass rate	75.8%	90.8%	86.0%	85.7%	87.9%	90.7%
Collimator	Conea		HD		LD
N	231		205		374
% (N)	21.6%		19.1%		34.9%
Average TLD ratio	0.973		0.985		0.987
Average % pixels passing	96.8%		95.2%		94.5%
Pass rate	78.8%		87.3%		85.8%

Abbreviations: AAA, Anisotropic Analytical Algorithm; HD, high definition; LD, GBBS, grid-based Boltzmann solver; low definition; TLD, thermoluminescent dosimeters; TPS, treatment planning system.

aThe categories that showed further statistical significance utilizing the Tukey post hoc test. Specifically, pencil beam compared to all other algorithms and cone collimators compared to HD were statistically significant.

Our random forest results reinforced the findings of the univariate analysis and provided the additional depth of information. The variables evaluated were found to account for the vast majority of the variations seen by different institutions: particularly for classification, which had an accuracy of 90.9 ± 0.3% (regression accuracy: 83.1 ± 2.5%). For overall pass/fail, average TLD ratio, and average percentage of pixels passing, the field size was the most important predictor. The next three most important variables were year of irradiation, machine, and algorithm; however, the specific order of importance varied depending on which outcome was of interest. For example, for average TLD ratio regression, the TPS algorithm was the second most important variable, while for percentage of pixels passing gamma, TPS algorithm was the fourth most important. Interestingly, and different from the univariate analysis, collimator was found to be among the least important parameters in predicting phantom performance. Figure 4 shows the ranking of the variables of importance for classification and regression random forest modeling.

Figure 4.

The variables of importance from our (a) performance classification, (b) regression average TLD ratio, and (c) regression percentage of pixels passing gamma random forest models. Error bars represent 1 SD. The level of importance is in arbitrary units (AU) and can only be compared to other variables on a single graph. Abbreviation: TLD, thermoluminescent dosimeters.

Discussion

The state of clinical SRS practice was elucidated in this study. The pass rate of the SRS phantom has remained steady over the past 8 years at approximately 85%, despite an increase in the average TLD ratio. This implies that clinical practice still struggles to accurately align and deliver dose to targets in SRS treatments, despite increases in utilization and advancements in this treatment modality. Specifically, our data show that the interplay of three categories contributed to the majority of failing phantom results: TPS algorithm (pencil beam), collimator (cone), and small treatment fields.

Pencil beam algorithms have been well known to poorly model the true dose distribution, particularly in heterogeneous anatomy, and including in other IROC phantoms (non-SRS).[34,35] However, the SRS head phantom, which has homogenous anatomy, indicates that the failure of the pencil beam algorithm may extend beyond poor modeling in heterogeneous media. Interestingly, the pencil beam algorithm not only underperformed compared to more advanced algorithms but also simpler measurement-based algorithms, highlighting a pronounced deficiency.

Collimator and field size need to be examined together to understand their effect on SRS delivery performance. First, we found that cone collimators were used for the majority of small treatment fields (field sizes ≤3 cm). Second, all random forest models indicate that field size was the most important variable while collimator was the least important variable. Combining these two analyses indicates that field size is the driving force behind failures, and cone collimators are simply correlated with field size. There are two reasons why small fields are likely to predict phantom failure. From an institution’s perspective, smaller fields are harder to characterize dosimetrically and contain larger dose calculation uncertainties compared to conventional radiotherapy.[18] On IROC’s side, the dose measurement is also more challenging: in particular, volume averaging effects over the TLD are more pronounced for smaller fields, especially as dose uniformity across the target in SRS treatments is often not a priority (while the measured TLD dose is compared to a corresponding volume in the treatment planning system, this is nevertheless more sensitive to any positioning uncertainty).

The suitability of the criteria set forth by IROC in conjunction with the NCI can be evaluated, and separated into two categories: (1) 5% dose difference for TLD and gamma analysis and (2) 3 mm distance to agreement for gamma analysis. Through IROC’s head and neck phantom, Carson et al justified the 5% dose difference as a reasonable threshold to evaluate phantom performance with a probability of measurement-noise induced failure being less than 0.5%.[21] However, due to the ablative nature of SRS therapy, such dose errors may be less clinically important. Nevertheless, medical physics practice standards are clear that 5% should be obtainable in terms of dosimetric accuracy, despite the additional challenges associated with small field dosimetry.[18,36] The registration error associated with IROC’s distance to agreement is approximately 1 mm, which is consistent with medical physics tolerances standards for SRS.[9,37] Given the nature and goal of SRS treatment, 3 mm is a generous spatial margin to assess performance within the phantom.

It is of note that the two analysis techniques utilized in this paper differ in their goals and interpretation. Univariate analysis directly compared two variables, while the random forest models analyzed the larger picture, modeling the interplay of all the variables. Both techniques allow for different viewpoints of the data, and through their combination, we can understand the complex nature of our data and the true interplay of treatment parameters on performance. In addition to elucidating the interplay between collimator and field size as described above, a deeper understanding is also available in terms of the TPS algorithm’s impact. Figure 4 shows TPS algorithm’s ranking in importance at predicting outcome. While it is the second most important factor for the TLD dose accuracy, it is only the fourth most important variable for gamma agreement or overall pass/fail. This information, combined with our univariate analysis, allows for the reasonable conclusion that pencil beam algorithms struggle to accurately predict dose compared to other algorithms (affecting the TLD ratio) but does not affect alignment (ie, the gamma pass rate).

Through the use of random forest and univariate analysis, three main categories of failure were found that continued to suppress the pass rate for SRS treatment. To avoid continuing this trend into the future, additional steps can be taken by institutions. First, a careful understanding of the difficulty and methods for accurate characterization of small fields is essential. The opportunity for improvement has been aided greatly by recent guidance from the IAEA,[36] and it is essential that such guidance be followed by any SRS program. Second, independent evaluations of an SRS program (such as using IROC’s end-to-end QA phantom) should be used. Such evaluations provide an opportunity for review and testing of the program and are strongest when based on an independent test using independent dosimeters and analysis methods. Any unexpected or undesirable results should be thoroughly reviewed, for example with help from IROC’s team, to determine likely reasons for the problems. Once properly investigated, diagnosed, and rectified, the independent testing should be repeated.

Conclusions

A collection of 1072 SRS head phantoms irradiated from 2012 to 2020 were analyzed to understand the performance and state of the radiotherapy community and shortcomings in different treatment techniques. Despite an increase in the dosimetric accuracy, the pass rate has remained steady at approximately 85%. Field size had the largest impact in determining performance (pass/fail) and, specifically, small fields (≤3 cm) were associated with more failures across the radiation oncology community. In addition, pencil beam algorithms lagged in performance compared to other dose calculation algorithms in terms of accuracy.

These results broadly indicate a need for improvement in SRS treatments across the community. Treatment parameters that need particular attention to ensure accurate treatment delivery include, smaller field sizes and pencil beam algorithms (which should be used with caution).