Exploration of clinical preferences in treatment planning of radiotherapy for prostate cancer using Pareto fronts and clinical grading analysis.

INTRODUCTION: Radiotherapy treatment planning is a multi-criteria problem. Any optimization of the process produces a set of mathematically optimal solutions. These optimal plans are considered mathematically equal, but they differ in terms of the trade-offs involved. Since the various objectives are conflicting, the choice of the best plan for treatment is dependent on the preferences of the radiation oncologists or the medical physicists (decision makers).We defined a clinically relevant area on a prostate Pareto front which better represented clinical preferences and determined if there were differences among radiation oncologists and medical physicists. METHODS AND MATERIALS: Pareto fronts of five localized prostate cancer patients were used to analyze and visualize the trade-off between the rectum sparing and the PTV under-dosage. Clinical preferences were evaluated with Clinical Grading Analysis by asking nine radiation oncologists and ten medical physicists to rate pairs of plans presented side by side. A choice of the optimal plan on the Pareto front was made by all decision makers.
RESULTS: The plans in the central region of the Pareto front (1-4% PTV under-dosage) received the best evaluations. Radiation oncologists preferred the organ at risk (OAR) sparing region (2.5-4% PTV under-dosage) while medical physicists preferred better PTV coverage (1-2.5% PTV under-dosage). When the Pareto fronts were additionally presented to the decisions makers they systematically chose the plan in the trade-off region (0.5-1% PTV under-dosage).
CONCLUSION: We determined a specific region on the Pareto front preferred by the radiation oncologists and medical physicists and found a difference between them.

Introduction

In radiation therapy, the conventional treatment planning optimization process is a trial and error loop that leads to a single plan. Most optimization systems need a set of goals formulated as dose-volume objectives, each assigned with a ‘weight’ or ‘importance’. The choice of appropriate objective weights is not obvious to the planner as they are not directly related to a clinical objective [1]. Furthermore, the quality of the entire 3D dose distribution is assessed through a single cost function value that combines all the objectives used to formulate the treatment planning problem [2].

Another approach used to attain a high-quality treatment plan is auto-planning (see [3], [4] for extensive reviews). In this system, the plan is created from a database where previous plans are stored. The new plan is obtained from cases stored in the database, usually by “optimization” of dose volume histogram (DVH) data. This method has demonstrated that it improves upon the final plan compared to the manual technique of conventional optimization methods. Note, however, that the goal motivating this technique is not to obtain an optimal plan but to automatize the treatment planning process [5], [6], [7].

At the end of the optimization run, either with the manual optimization method or with auto-planning, the decision maker – a radiation oncologist (RO) or medical physicist (MP) – is not given information to conclude whether a more clinically optimal solution could be reached. Due to the long optimization process, the decision maker usually evaluates one or a few available solutions out of the entire continuum of possible plans that constitute the solution space and often selects among mathematically and clinically sub-optimal solutions [8]. These limitations compromise the possibility of fully exploring the trade-offs and available dosimetric options for each patient.

A clear representation of the solution space and the ranges of the trade-offs involved could lead to more informed clinical decision making and better treatment plans [8], [9], [10], [11]. Multi-objective optimization problems generally do not have a unique solution that can satisfy all objectives simultaneously, but rather a set of Pareto optimal solutions, which dominate all other non-optimal solutions. Additionally, each plan on the Pareto front is Pareto optimal, ensuring that a mathematically optimal solution is attained.

The process of evaluating and making decisions regarding radiation dose compromises within the treatment plan that will be delivered includes judgment tasks for the trade-offs involved. Since the planning criteria are usually conflicting, the decision largely depends on the preferences of the decision maker. Given that the decision maker makes a decision after a holistic assessment of the plan, it is important to have information about the relevant ranges of the evaluation parameters and their interdependencies. However, current treatment planning techniques rarely provide this information. In this context, the Pareto front concept has been proposed as an appropriate tool for the comparison of treatment plans [12].

In this study we explored the clinical preferences of the decision makers among plans on the Pareto front. The evaluation was performed using Clinical Grading Analysis (CGA), a method similar to the visual grading analysis used in radiology [13]. It identifies clinically relevant differences between the plans instead of comparing them in an abolute sense. A grading scale is defined in order to classify the relative quality of one plan compared to another (e.g. « much better », or « equivalent »). Our aim was to investigate the possibility of defining a clinically relevant area on the Pareto front. Additionally, we determined if there were differences in the plan choices between the two groups of specialists (radiation oncologists and medical physicists) involved in the process of approving a treatment plan. Our intention was to illustrate an effective method to gain insight about clinical choices using the concept of the Pareto front.

Materials and methods

Patient selection and treatment planning data

We randomly selected five localized prostate cancer cases from the patient database of our center. The study was exempt from further ethics committee approval because the data were anonymized and the results had no impact on the treatment outcome of the patients. Indeed, the number of patients was limited. This choice was done due to the time-consuming process of creating manual Pareto fronts. It was not the aim of the study to provide a universal solution to the question of clinical preference among plans on the Pareto front.

The organs delineated were the Clinical Target Volume (CTV), the Planning Target Volume (PTV), bladder, rectum (defined as the entire volume) and femoral heads. The plans were made according to the RTOG 0415 protocol. Additional information can be found in Supplementary Materials. The rectum was delineated from the recto-sigmoid flexure to the anal canal. The ratio of the overlapped volume of the PTV and the rectum volume to the rectum volume was 13%, 25%, 11%, 10%, and 4% for each plan, respectively. We used the Tomotherapy (Accuray, CA, USA) treatment planning system V5.0 to optimize all the plans in the study that were calculated to be potentially delivered on a Tomotherapy HD device. In all five clinical cases the prescription dose was 78 Gy to the PTV.

We created Pareto fronts of the rectum sparing vs. PTV coverage trade-off for all the cases, based on two clinical evaluation criteria (Figure S1 in Supplementary Materials). Rectum sparing was evaluated through the D50% objective, where D50% is the median dose to the rectum. PTV coverage was evaluated through the V95% objective, where V95% is the percent volume receiving at least 95% of the prescription dose. We focused on this single trade-off in order to simplify the optimization problem and reduce its multi-dimensionality when involving the other organs at risk (OAR). The different plans constituting the Pareto front concerning a specific trade-off were obtained by gradually increasing the penalty values for the rectum objective while keeping the values of all the other parameters of all OARs constant, and the optimization was continued. It was practically impossible to strictly control all the parameters of all the structures involved at the same time. We achieved the best we could by keeping the values of all other parameters of all OARs constant while allowing for dose deviations of 1 Gy maximum on average.

Clinical grading analysis

The trade-off was explored through a sequence of paired choice tasks. In this study the plans under evaluation involved a non-Pareto optimal plan with 0% PTV under-dosage (Plan 0_0), and optimal plans on the Pareto front with 0% (Plan 0), 1% (Plan 1), 2.5% (Plan 2.5), 4% (Plan 4) and 5% (Plan 5) PTV under-dosage (Figure S2 in Supplementary Materials). All the possible pairs of plans were presented for comparison. Nineteen decision makers (nine radiation oncologists and ten medical physicists) from four different radiation therapy centers evaluated the quality of the plans. We chose to ask decision makers from more than one center to avoid any possible bias that might occur due to a single center evaluation. The decision makers received the dose distribution and the DVH of each plan. There were paired comparisons of plans and one plan was defined as the reference, which meant that every time the reference plan was a different one depending on the comparison. They were asked to grade the quality of the plan vis-à-vis the reference plan, on a five-level scale: clearly inferior (-2), slightly inferior (-1), equal (0), slightly superior (1) or clearly superior (2). They were not asked which criteria were used to reach to the decisions.

The comparisons were evaluated with CGA, which was proposed as a tool for quantitative treatment plan comparisons in Radiation Therapy [13]. Similar to the Visual Grading Analysis used in radiology, this method aims to include physical or biological measures used in clinical practice (dose volume metrics such as DVH and dose statistics) as well clinical judgments based on 3D dose distributions. The main contribution of the method is the identification of clinically relevant differences between plans. With CGA, the decision makers visually evaluated the quality of the whole treatment plan and compared pairs of plans presented side by side.

Each of the five comparison plans was analyzed, based on the results of all comparisons and for all decision makers. The results for each comparison plan were counted and the most answered value was chosen for all the decision makers.

The statistical analysis was performed on the whole distribution of answers (and not only on the most answered value) as follows. The Kolmogorov–Smirnov test was used to examine if data were normally distributed. The two groups of decision makers (ROs and MPs) were compared using the Mann–Whitney U test. The Pearson Chi Square (normally distributed data), or the Fisher’s exact test (non-normally distributed data) was used to examine differences in the plan evaluations between the two groups of decision makers. The significance level (α) chosen was 5%.

At the end of the evaluation process of each case the decision makers were additionally shown the Pareto fronts (Figure S2 was presented alone without the dose distributions) and then asked to show the position of their ideal plan on the Pareto front. The decision makers were not allowed to see the plans again and they had to make their decision with only the Pareto front they were shown.

Results

Clinical evaluation

Fig. 1 and Table 1 present the results of the most answered value for each comparison plan for all the decision makers. For all five Pareto fronts the plans with full PTV coverage (Plan 0_0 and 0) were considered equal, with plan 0 being more often preferred by the MPs (30%) than by the ROs (17.8%) (p = 0.029). All the plans with PTV under-dosage up to 5% were considered clearly superior to both plans with full coverage (Plan 0 and Plan 0_0). However, there was a statistically significant difference between the MPs and ROs as far as Plan 2.5 was concerned, with the MPs preferring it more often (p = 0.008 and 0.052 for Plan 0 and Plan 0_0 respectively). Plan 1 was considered inferior compared to Plans 2.5 and 4, but the ROs tended to prefer the Plans 2.5 and 4 more often than the MPs (p = 0.011 and p = 0.049, respectively). Plan 5 was considered inferior compared to Plan 1 with no statistically significant difference among the two groups of decision makers. Plans 2.5 and 4 were considered equal for both the ROs and the MPs. However, there was a statistically significant difference among the two groups, with the ROs marginally evaluating Plan 4 as superior more often than the MPs (p = 0.029). Plan 5 was considered inferior compared to Plan 2.5, with the ROs considering it superior more often than the MPs (p = 0.057). Plans 4 and 5 were considered equal for both groups of decision makers.

Fig. 1

Absolute evaluation of each of the five comparison plans on the Pareto front. The evaluation of a plan was summed over each comparison to obtain a total value. The y-axis is the average value of the summed plan quality over all decision makers (all DM), all radiation oncologists (RO) or all medical physicists (MP).

Table 1

Plan evaluation on all five Pareto fronts among all the decision makers. The value showed corresponds to the choice most often made. Plans on the vertical column are evaluated compared to the reference plans on the horizontal line. The test plan is considered: clearly inferior (-2), slightly inferior (-1), equal (0), slightly superior (1) or clearly superior (2) to the reference plan. As an example, Plan 1 is “clearly superior” to plan 0_0 and to plan 0.

Plan	0_0	0	1	2.5	4
0	=*, **
1	2	2
2.5	2*	2*, **	1
4	2	2	1*	=*
5	2	2	−1	−1*	=

* indicates a statistically significant difference in the overall evaluations between ROs and MPs (accounting for the whole distribution of answers, see statistical analysis in Supplementary Materials for more information).

** The RO results was 1.

For all cases, apart from Case 2, the plans in the central region of the Pareto front (1–2.5% PTV under-dosage) received the best evaluations both among both ROs and MPs. For cases 1, 3 and 4, the ROs tended to prefer Plan 2.5 while the MPs preferred Plan 1 (p = 0.011). For Case 2, with a large rectum-PTV overlap and for which the RTOG constraints were not respected for all plans, there was a preference for the plans with a lower rectum dose.

Choice of plan on the Pareto front

When the Pareto front for each case (except Case 2) was shown to the decision makers, the majority (17 out of 19), chose a plan around the region where the trade-off was more prominent (in most cases around 0.5–1% PTV under-dosage). For Case 2 all the decision makers chose plans in the region with the highest OAR sparing (4–5% PTV under-dosage) even at the cost of poor PTV coverage.

Discussion

In this study, we present a new concept for a clinical evaluation of treatment plans based on Pareto fronts in which we explored and represented a clinically interesting trade-off. Our results were able to define a clinically relevant area on the Pareto front, where plans were also considered clinically equivalent.

The decision makers we consulted were free to compare the plans based on their own criteria as long as the RTOG dose constraints for the OARs were respected. In all cases but one (Case 2), all the plans offered for evaluation were accepted according to the RTOG constraints for all the organs at risk and the PTV. This means that plan evaluation was indeed a question of clinical preference and judgment and not focused on whether a plan was acceptable or not according to clinical constraints. In fact, the second case with 25% overlap between the PTV and the rectum seemed to be an outlier, but we decided to include it because such cases do exist in clinical practical and represent another type of decision making situation.

CGA revealed an area on the Pareto front of the rectum sparing – PTV under-dosage trade-off where plans were considered clinically better for treatment. This area covered the 1 to 4% PTV under-dosage, which means that the extreme regions of the front are removed. Moreover, when data of Case 2 is removed, the region reduces even further to 1–2.5%. Plans in this region were considered equal and better/much better than the corresponding ones outside this region. Plans sparing the rectum were considered clearly superior compared to plans ensuring a good PTV coverage especially in the Pareto front region up to 1% PTV under-dosage. Overall, there was a consensus in each of the two groups of decision makers regarding the preferred region, although the ROs tended to prefer more OAR sparing (regions 2.5–4% under-dosage) while the MPs preferred the 1–2.5% under-dosage region. In other words, the ROs more often preferred plans favoring rectum sparing compared to plans with better PTV coverage, while the MPs were more reluctant to sacrifice PTV coverage especially in the region of more than 2.5% PTV under-dosage. That result corresponds to previous studies [9], [14].

Depending on the case and mainly on the PTV to rectum overlap there were different decision strategies among the decision makers. Both groups generally agreed in their evaluations for Case 2, for which the overlap between the rectum and the PTV volume was large. For this case, not all the plans on the Pareto front respected the RTOG constraints and, in particular, it wasn’t possible to reach full PTV coverage without seriously compromising rectum sparing. In Cases 1, 3, 4 and 5, for which the Pareto fronts have an L shape- binary pattern, the decision makers gave less homogenous evaluations, indicating different clinical preferences..

2D Pareto front analysis of the values of specific decision objectives can be used as a tool to provide the decision maker with additional information on the possibilities available about a particular trade-off. This raises the question of whether the decision maker will make similar choices or use the same criteria if s/he is aware of the trade-off values. Our results suggest that decision makers preferred another region when they recieved the additional information of the full solution space concerning the specific trade-off. Interestingly, when the decision makers were shown the Pareto fronts of each case at the end of the CGA, their choice of the optimal plan was different than their evaluation with CGA. While during plan comparisons they preferred a “balanced” plan with a fair compromise on the main trade-off presented (in the 1 to 4% under-dosage region), they made a different choice when they had a visual insight on the shape of the trade-off through the Pareto front (around 0.5% under-dosage region for all cases except for case 2). It was evident, as most of them expressed, that further under-dosage on the PTV led to a very low gain in rectum sparing. This access to additional information regarding the possibilities about a specific trade-off could be particularly useful during decision making.

As far as we know, this is the first time that a clinical judgement has been evaluated in terms of the inter-observer variability of trade-off preference. Also, there are no studies related to a comparison of plans lying on the Pareto front. Nevertheless, our findings that the ROs prefer lower dose to OAR sparing over PTV coverage are in line with previous studies on clinical judgement [9], [14].

There are several limitations to our study which mainly arise from the complexity of the optimization and the decision making processes. They are described in the following paragraphs.

The number of cases used in this study is rather low (n = 5). This is mainly due to the time needed to determine the Pareto fronts for each case. However, that low number is balanced by the high number of decision makers involved in the study. Our results (decision makers prefer the 1 to 4% PTV under-dosage region, when not knowing the Pareto front and the 0.5% PTV under-dosage region when knowing it) may be biased by such a low number of cases. However, our main result showing how the decision makers changed their mind when provided with additional information about the Pareto front remains valid, no matter whether the chosen region of the Pareto front is approximate or not.4

Our method of manually creating Pareto optimal plans is burdensome and time consuming, but it does enable us to obtain optimal plans. It is also possible with systems generating automatic Pareto fronts, but the quality of the resulting Pareto fronts should be assessed, because of the approximations necessary during the optimization process [15]. It is practically impossible to strictly control all the parameters of all the structures involved at the same time. Therefore, keeping the values of all other parameters of all OARs constant while allowing for dose deviations of, on average, 1 Gy maximum is the best that could be done yet. Such variation might affect the decision making in specific cases, but not to such an extent as to question our results. Additionally, we cannot control or accurately investigate which parameters most affected the decision makers. As far as decision making is concerned there is a big unconscious component in the process which involves not only reasoning but also visual inspection [16]. Our choice of a 1 Gy maximum dose deviation was an acceptable compromise easily handled manually, but does emphasise the importance of keeping all other parameters constant or within acceptable limits.

Complete optimization is a multi-dimensional problem that includes several OARs (rectum, bladder, femoral heads)as well as other parameters such homogeneity, conformity, TCP and NTCP values. Our study was limited to a two-dimensional Pareto front and we do not know what would happen if an additional parameter would have been used for trade-off (e.g. three-dimensional Pareto hyper-plan). The biggest difficulty in deciding upon the best radiation treatment plan for each patient is the lack of universally accepted criteria. The probabilities of possible complications for each individual patient are not currently known or predictable. Since planning criteria usually conflict in some way, the decision largely depends on the preferences of the decision maker, preferences that come from training, experience, and his/her understanding of the patient’s clinical situation. In other words, an evaluation based on a two-dimensional Pareto front is a clear limitation of the whole picture.

In this study we have chosen the D50% as the evaluation parameter for the rectum. During treatment plan evaluation in clinical practice, the high dose region of the rectum DVH is what is usually considered. Recently, the low to intermediate dose region is now also being taken into consideration to account for late effects [17]. Large volumes at lower doses have been shown to correlate with several measures of late rectal toxicity [18], [19].

CGA of treatment plan evaluation shows the subjective preferences of the decision makers. We explored clinical preferences of decision makers from four radiotherapy treatment centers and our results reflect the decision making strategies established in each center. Additionally, they may reflect the experience of the decision makers.

The exploration of decision maker preferences provides insight into clinically interesting areas of the solution space. Information of this kind could prove valuable for automatic treatment plan selection. Combining the Pareto optimality concept with auto planning does provide not only a well optimized plan, based on the quality of the other plans in the database, but an objectively optimal plan on the Pareto front [20], [21]. In addition, the mathematical definition of the decision problem could include decision maker preferences in such a way as to lead the search towards clinically preferred regions.

In conclusion, we assessed the inter-observer variability of decision making using CGA as a tool and we defined a region where mathematically equal plans were also considered clinically equal and preferred for treatment. We also noticed a difference in the region of preference among the ROs and the MPs, with the ROs focusing more on OAR dose sparing while the MPs favored ensuring PTV coverage. This type of information could be valuable for planners and thus homogenize their practice in each center based on requirements established by ROs and MPs.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.