Validation of a Software Upgrade in a Monte Carlo Treatment Planning System by Comparison of Plans in Different Versions.

PURPOSE: Validation of a new software version of a Monte Carlo treatment planning system through comparing plans generated by two software versions in volumetric-modulated arc therapy (VMAT) for lung cancer.
MATERIALS AND METHODS: Three patients who were treated with 60 Gy/30 fractions in Elekta Synergy™ linear accelerator by VMAT technique with 2% statistical uncertainty (SU) were chosen for the study. Multiple VMAT plans were generated using two different software versions of Monaco treatment planning system TPS (V5.10.02 and V5.11). By keeping all other parameters constant, originally accepted plans were recalculated for the SUs of 0.5%, 1%, 2%, 3%, 4%, and 5%. For plan evaluation, the metrics compared were conformity Index (CI), homogeneity Index (HI), dose coverage to planning target volume (PTV), organ at risk (OAR) doses to spinal cord, pericardium, bilateral lungs-PTV, esophagus, liver, normal tissue integral dose (NTID), volumes receiving dose >5 and >10 Gy, calculation time (tCT), and gamma pass rates.
RESULTS: In both versions, CI and HI improved as the SU increased from 0.5% to 5%. No significant dose difference was observed in Dmean to PTV, bilateral lungs-PTV, pericardium, esophagus, liver, normal tissue volume receiving >5, and >10 Gy and NTID. It was observed that while the tCT and gamma pass rates decreased, the maximum dose to PTV increased as the SU increased. No other significant dose differences were observed between the two MC versions compared.
CONCLUSION: For lung VMAT plans, in both versions, SU could be accepted up to 3% per plan with reduced tCT without compromising plan quality and deliverability by accepting variations in point dose and an inhomogeneous dose within the target. The plan quality of Monaco™V5.10.02 was similar to Monaco™TPS-V5.11 except for tCT.

INTRODUCTION

The volumetric-modulated arc therapy (VMAT) is an advanced radiation technique, which can achieve highly conformal dose distributions with more efficient treatment delivery than conventional static field intensity-modulated radiotherapy (IMRT). In addition, it has potential benefits compared to IMRT regarding the tumor control probability and reducing dose to normal structures.[12] Several of the commercially available treatment planning systems (TPSs) are capable of doing VMAT plans employing different dose calculation algorithms.[3] Among all existing algorithms, the Monte Carlo (MC) algorithm is considered as the most accurate one. Besides, it has been used for the generation of benchmark dose distribution and to evaluate other dose calculation algorithms.[4] Fotina et al. have reported that the MC dose calculation accuracy was slightly higher in low-density material compared to advanced kernel methods.[5] However, its inherent statistical uncertainty (SU) will determine the dose calculation accuracy and calculation time (tCT). The International Commission on Radiation Units and Measurements recommends that the overall dose accuracy should be kept within 5%.[6] As per American Association of Physicists in Medicine (AAPM) Task Group (TG) 65 report, uncertainty in dose calculations ranged from 1% to 5% when tissue heterogeneities are present.[7]

Traberg Hansen et al. reported that the choice of dose calculation algorithm had a large influence on a treatment plan for lung cancers.[8] Moreover, various other studies have revealed that MC dose calculations were potentially more accurate than the other commercial algorithms for such situations.[910] Similarly, Woon et al. reported in their study that the MC algorithm was found to have better accuracy regarding inhomogeneity correction.[11] Jiang et al. reported that large statistical uncertainties were expected to “blur” the dose-volume histogram (DVH) curves, and the resultant isodose distribution might become unreliable.[12]

The desirable features of any dose calculation algorithm are that it should be fast and at the same time accurate enough. In MC based systems, a decrease in tCT results in increased SU and hence accuracy in dose calculation has to be determined.[13] Among all sites, lung cancer has the most complex heterogeneities, organ structures, besides the presence of many interfaces of air-tumor and bone-tumor as compared to other sites. Hence, plans for target volumes in lung, considered conventionally as the most suitable case to evaluate the dose calculation accuracy in MC systems, were selected in present study to study the plan quality and calculation efficiency.

Installation of new updates such as fixing of old bugs, addition of new tools and modification of dose calculation algorithms are regular features in every TPS. Recently, the TPS Monaco™ software V5.10.02 (Elekta Ltd, Missouri, USA) was upgraded to version V5.11 in our center. The new version includes refactoring changes as compared to its previous versions in the MC calculation code which needs to be validated by clinical physicists as recommended by AAPM TG-119, TG53, and TG105.[141516] To the best of our knowledge, very little or no precise data are available regarding the comparison of plan metrics between these two software versions.

The aim of this study was to validate the new software version V5.11 through comparison of plan quality metrics for lung VMAT plans generated using these two software versions.

MATERIALS AND METHODS

Simulation, contouring, and prescription:

For the present study, a total of three patients were selected who were planned with 60 Gy in 30 fractions using VMAT technique. The patients were immobilized using thermoplastic mold and simulation was performed with a carbon fiber tabletop on 16 slice positron emission tomography-computed tomography (CT) simulator (Simens® Biograph Truepoint® HD, Siemens AG, Medical solution, Erlagen, Germany). CT images of 3-mm slice thickness were used for VMAT treatment planning. All tumor volumes such as gross tumor volume, clinical target volume, planning target volume (PTV), and organ at risk (OAR) volumes were contoured by an experienced radiation oncologist with radiologist's support as per multidisciplinary protocol of the institution.[17]

The dosimetric parameter used for volumetric-modulated arc therapy planning

The VMAT plans were generated for Elekta Synergy™ linear accelerator (Elekta Ltd, Crawley, UK) with 1 cm leaf width at isocenter. A fluence width of 0.3 cm, medium fluence smoothing, 0.3 cm grid size, 0.8 cm segment width, 20° gantry interval, and different partial arcs were used for generating VMAT plans. In MC dose calculation, SU 2% per plan was used initially for all the plans as this was recommended by the vendor. After a clinically acceptable plan was generated, plans were recalculated for SU values of 0.5%, 1%, 3%, 4%, and 5% keeping all other parameters constant. Thus, for three patients, a total of 18 plans were generated at six different SU levels and for both versions combined, a total of 36 plans were generated. Each plan was evaluated using DVH generated by the planning software. The plan quality was analyzed using different dosimetry indices as mentioned below. In addition, dose coverage to PTV, OAR doses, tCT, and plan deliverability were also analyzed.

Dosimetric indices used for volumetric-modulated arc therapy plan evaluation

Conformity Index

It is defined as the ratio of the volume receiving the prescribed dose (V100%) and volume of PTV[18]

Conformity Index (CI) = V100%/PTV.

Homogeneity Index

It is defined as the ratio evaluating the dose homogeneity in PTV where D2%, D98%, and D50% are the doses received by 2%, 98%, and 50% volume of the PTV, respectively.[19]

Homogeneity Index (HI) = (D2%− D95%)/D50%.

Normal tissue integral dose

It is defined as the product of mean dose (Dmean) to the body (body-PTV) and volume of PTV.[20]

Normal tissue integral dose NTID (Gy. L) = D (Gy) × V (L) where D (Gy) is the Dmean delivered to volume V (L) (where L-liter)

The volume of normal tissue receiving ≥5 and ≥10 Gy was also analyzed.

Calculation time

The total MC (tCT) for VMAT plans are reported in Monaco™ console window. The calculation speed is based on algorithm and computer hardware configuration. For this study, HP Z820 workstations, 32GB RAM, Intel® CPU E5-26700 @ 2.60 GHz (2Processor), the 64-bit operating system was used.

tCT (mins) = Start time (mins) − End time (mins)

Planning target volume dose coverage and dose to the organ at risk volumes

The dose coverage to PTV was analyzed as D98%, D95%, D50%, and D2% where D were the doses received by 98%, 95%, 50%, and 2% of the volume of the PTV. In addition, maximum dose (Dmax) and Dmean to PTV volume were analyzed. The Dmean, Dmax, and other dose-volume parameters were analyzed for spinal cord, pericardium, esophagus, bilateral lungs-PTV, and liver.

Plan delivery results

The TPS plan accuracy was verified using PTW 729 ion chamber array with OCTAVIUS™ Phantom (PTW–Freiburg, Germany) based on two-dimensional planar dose verification method. The gamma pass rates were estimated for the 3% - 3mm and 3% - 2 mm criteria comparing the TPS calculated dose and the measured dose.[21]

Statistical analysis

The plan quality metrics were compared for the plans generated by both software versions at different statistical uncertainties and determining their P values. For data analysis, SPSS (SPSS V.16, IBM, USA) software was used.

Validation of statistical uncertainty on Monaco treatment planning system Monte Carlo calculation

Monaco™ (Elekta Ltd, Missouri, USA) TPS uses MC algorithm for dose calculation in VMAT plans. It has an option to choose SU per plan ranging from 0.5% to 5% during dose calculation, with decreasing SU leading to improved dose calculation accuracy. Nevertheless, decreasing the SU could significantly increase the dose tCT.[2223] The planning time in any clinic is always an important factor which should not compromise with plan quality and deliverability. Therefore, the compromise between SU and dose tCT should be studied properly for various treatment sites to generate clinically acceptable plans in the smallest calculation time possible.

Modification in the Monte Carlo code used in upgraded version Monaco™ V5.11:

The vendor of Monaco™ V5.11 has mentioned in the release notes that in the older version of the software, dose optimization and sequencings were clinically inefficient and took much time to generate VMAT plans. The new MC code was refactored to enhance its clinical efficiency.[24] The new version includes refactoring changes which introduces a change to the random number sequence in the MC calculation code. Besides, this change in the random number sequence causes changes in the dose calculation in the new Monaco™ version which needs to be validated by a clinical physicist for QA. The notes also mention that the dose difference will be observed only when the identical calculation is made between Monaco™ V5.10.02 and Monaco™ V5.11. This type of identical dose calculation is used for upgrading the validation testing.

It is also stated by the vendor that the natures of dose calculation differences were expected to be limited to statistical/MC uncertainty. Since the various mathematical routines are performed using MC code, the refactoring changes brought about minor changes in the numerical precision. As a result of this change, dose differences in isolated voxels near the surface of the external contour may be observed in Monaco™V5.11 as compared to Monaco™ 5.10.02.

In the newer version, calculation involving segment shape optimization also uses the random number sequence during optimization. The optimized plans produce slightly different segment shapes, as compared to the older version. However, plan quality is maintained. The user is reminded that the segment shapes vary from the older version to the new version since the optimized plans are not identical between releases.

The release notes also stated that an intentional change was made in the threading routine used for particle simulations which returned functionality found in earlier versions of the optimization engine. The threading changes remove the variability which was expected to improve maintainability, ease of troubleshooting, and customer TPS commissioning efforts for version upgrades.

RESULTS

Plan quality metrics were calculated using DVH and compared for the plans generated using two different software versions. There were some similarities and dissimilarities due to the influence of SU. For analysis, raw data were arranged in a master sheet, and analysis of data was done by the objectives of the study. Data were analyzed using descriptive and inferential statistics and the results are presented with the help of tables, figures, and charts.

Target volume and organ at risk

Table 1 depicts that the CI and HI values improved as SU increased from 0.5% to 5% in both the versions. However, no significant differences were observed between different SU% levels and between the dose calculation versions (P > 0.05).

Table 1

Comparison of quality metrics of volumetric modulated arc therapy plans done on two versions of a Monte Carlo based treatment planning system software (n=3)

As shown in Tables 2 to 4, the PTV dose coverage (PTVmean, V95%), mean dose to volumes bilateral lungs-PTV (V30%), ipsilateral lung (V20), pericardium (V33% and V67%), esophagus, and liver decreased marginally as SU increased from 0.5% to 5%. However, no significant dose difference was found (P > 0.05). No significant dose difference was observed between two MC calculation versions (P > 0.05) either. The Dmax to PTV increased as SU increased from 0.5% to 5% with significant dose difference (P < 0.05), but there was no significant difference between two versions (P > 0.05).

Table 2

Comparison of dose coverage to planning target volume and maximum dose to spine (n=3)

Table 4

Comparison of doses to organs at risk (n=3)

For the spinal cord, it was observed that when the SU increased from 0.5% to 5%, the Dmax value increased by 2.4% in the older version and by 1.1% in the newer version. However, at each SU level there was no significant dose difference observed between the two different versions (P > 0.05).

Normal tissue dose

As shown in Tables 1 and 5, for the normal tissue volumes receiving ≥5, ≥10 Gy and NTID, only small dose differences, which were not significant (P > 0.05), were observed when the SU was increased from 0.5% to 5%. No significant dose differences between two MC dose calculation versions were observed (P > 0.05).

Table 5

Comparison of doses to normal structures and gamma pass rates for different criteria (n=3)

Calculation time

Table 1 shows the effect of SU on MC dose tCT. The tCT decreased as SU increased from 0.5% to 5% and showed a significant difference (P < 0.05). On comparison of both versions, it was observed that there were no significant differences (P > 0.05). However, newer version was faster in dose calculation as compared to older version.

Plan delivery efficiency

Table 5 shows the results of the planar dose verification tests for the gamma criteria of 3%/3 mm and 3%/2 mm at various SU. The gamma pass rates decreased as the SU increased which was found to be statistically significant (P < 0.05). However no significant differences in gamma pass rates were observed between two the MC dose calculation versions (P > 0.05) for both the criteria.

Comparison of isodose lines

Figure 1 shows the comparison of three different (57, 30, 5 Gy) isodose lines for the different SUs in the coronal, sagittal and axial isocentric planes. The isodose lines at each dose level superimpose on each other for the different SUs. Figure 2 also shows that the isodose lines are almost identical with practically no difference between the two software versions. Figure 3 represents the DVH comparison for the two versions for the PTV and for lung. The DVH lines are once again overlapping on each other depicting there is no dose difference between the two versions.

Figure 1

The effect of statistical uncertainty on Monte Carlo dose calculation from statistical uncertainty 0.5%–5% on 57, 30 and 5 Gy isodose lines for lung volumetric-modulated arc therapy plan

Figure 2

Comparison of isodose lines (57, 30, and 10Gy) on axial, coronal and sagittal for the lung planes for the lung volumetric-modulated arc therapy plan

Figure 3

Comparison of dose volume histogram curves of two different Monte Carlo dose calculation versions in Monaco™ treatment planning system for lung volumetric-modulated arc therapy plan

DISCUSSION

The results of the present study are supported by Mohan et al. and Keall et al. who reported that the inherent uncertainy associated with the MC calculations was inversely proportional to the square root of computation time.[413] It was a compromise between dose calculation accuracy and acceptable SU. The inherent SU determined the accuracy of the dose calculation and tCT.[3] In clinical treatment planning, variation in SU can result in significant changes in the VMAT plan quality.

The results reveal that in both the software versions of MC dose calculation engine tested in this study, increase of SU from 0.5% to 5% did not make any significant difference in CI, HI, NTID, dose received by 95% of PTV, OAR doses, normal tissue volume receiving ≥5 and ≥10 Gy. However it The it itled to an increase in Dmax to PTV, and decreases in tCT and gamma pass rates all of which were significant. These results can help in assisting the clinical physicist to decide on the optimal level of SU% to be accepted for treatment planning.

Fogliata et al. compared Eclipse treatment planning system's (Varian Medical System, Palo Alto, CA) Acuros® XB (AXB) calculations against measurements and calculations performed with a previously validated dose calculation algorithm, the anisotropic analytical algorithm (AAA). They reported that AXB accurately reproduced measured and calculated data and only small deviations were observed in all the investigated quantities.[25]

There are very few studies reporting the importance of implementing an upgrade software version in a treatment planning system after comparing it with the old version. Ojala and Kapanen conducted a study that quantified the effect of modification implemented in the new algorithm version AXB11. The plans were first created with the AXB10 and then recalculated with AXB11 for ten IMRT and VMAT plans. They reported that no large deviations were present in DVH analysis results between the two versions of AXB algorithms. A clear improvement with the AXB11 over the AXB10 was the dose calculation accuracy in air cavities.[26] The study findings were supported by Krishna et al. who performed a comparative study of old Eclipse™ V8.8 with new Eclipse™ V13.6 on IMRT plans for different clinical sites. Their results had shown that the plan quality of new Eclipse™ V13.6 was maintained with almost same doses as compared to old version 8.8.[27]

The results of the present study show that there are no significant variations between the two software versions of the MC dose calculation engine, tested at various SU levels. Although the Monaco™ V5.11 release came with refactoring changes in MC code, the plan quality metrics of Monaco™ V5.11 were similar to Monaco™ V5.10.02. The dose tCT was faster in Monaco™ V5.11 than Monaco™ V5.10.02 which was an advantage in VMAT planning. The gamma pass rates for the 3%/3 mm and 3%/2 mm criteria did not show any significant difference between the two versions. Similarly the results of isodose line comparison also showed no significant differences.

As the SU was changed from 0.5% to 5% in both versions, it did not result in any significant differences in the plan quality metrics except in tCT, Dmax to PTV and gamma pass rate. Moreover, for both versions, the compared plan quality metrics were similar to each other except for dose calculation speed which was faster in Monaco™ V5.11 than Monaco™ V5.10.02.

CONCLUSION

In both MC versions, for lung VMAT plans, SU could be accepted up to 3% per plan with reduced tCT without compromising target coverage, OAR doses, and plan delivery by accepting variations in point dose and an inhomogeneous dose within the target. The choice of optimal acceptance of SU% in Monaco™ V5.11 could decrease planning time which can be an advantage in a busy clinic.

Although there was a change in the MC code of upgraded Monaco™ V5.11, it was found to maintain the same plan quality and deliverability as compared to Monaco™ V5.10.02 for lung VMAT plans. It was observed that Monaco™ V5.11 was faster regarding calculation speed than Monaco™ V5.10.02.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

Table 3

Comparison of doses to organs at risk (n=3)