Clinically Oriented Target Contour Evaluation Using Geometric and Dosimetric Indices Based on Simple Geometric Transformations.

PURPOSE: In radiotherapy, geometric indices are often used to evaluate the accuracy of contouring. However, the ability of geometric indices to identify the error of contouring results is limited primarily because they do not consider the clinical background. The purpose of this study is to investigate the relationship between geometric and clinical dosimetric indices.
METHODS: Four different types of targets were selected (C-shaped target, oropharyngeal cancer, metastatic spine cancer, and prostate cancer), and the translation, scaling, rotation, and sine function transformation were performed with the software Python to introduce systematic and random errors. The transformed contours were regarded as reference contours. Dosimetric indices were obtained from the original dose distribution of the radiotherapy plan. The correlations between geometric and dosimetric indices were quantified by linear regression.
RESULTS: The correlations between the geometric and dosimetric indices were inconsistent. For systematic errors, and with the exception of the sine function transformation (R2: 0.023-0.04, P > 0.05), the geometric transformations of the C-shaped target were correlated with the D98% and Dmean (R2: 0.689-0.988), 80% of which were P < 0.001. For the random errors, the correlations obtained by the all targets were R2 > 0.384, P < 0.05. The Wilcoxon signed-rank test was used to compare the spatial direction resolution capability of geometric indices in different directions of the C-shaped target (with systematic errors), and the results showed only the volumetric geometric indices with P < 0.05.
CONCLUSIONS: Clinically, an assessment of the contour accuracy of the region-of-interest is not feasible based on geometric indices alone. Dosimetric indices should be added to the evaluations of the accuracy of the delineation results, which can be helpful for explaining the clinical dose response relationship of delineation more comprehensively and accurately.

Introduction

Contouring of the target and organs-at-risk (OARs) is a key step in radiotherapy, especially with the highly modulated radiotherapy technologies currently in use, such as intensity-modulated radiotherapy (IMRT) and volumetric modulated arc radiotherapy (VMAT). Inaccurate contouring could cause serious systematic errors to the subsequent radiotherapy work. This error has always existed in the subsequent radiotherapy processes used for patients, such as radiotherapy treatment planning or patient positioning.[1-3] The commonly used slice-by-slice manual approach or interpolation-based semi-automatic contouring approach is time-consuming and labor-intensive, and the corresponding results are susceptible to differences between observers. In other words, the accuracy of contouring depends on the residents’ clinical experience and the rational and efficient use of multimodal images, and improper selection of multimodal images may lead to differences in residents’ delineation.[4-6] Another emerging contouring approach is the automatic segmentation technique. Compared with the aforementioned manual delineation method, its contouring speed is faster and it is not affected by the subjective factor, but it is necessary to evaluate the accuracy of the contouring results before its use in clinical applications.

Methods used for the assessment of the accuracy of contouring are generally classified in 2 categories, subjective and quantitative evaluations. The subjective evaluations are only based on the experiences and personal preferences of the evaluators. Evaluators are guided to turn off the original contour display and grade all research contours using 3 levels: useful as test contours (= 1), useful with minor edits (= 2), and not useful (= 3). The definition of minor edits is that the test contours would be acceptable after minor modifications.[7] This evaluation method is deeply affected by the individual differences among the evaluators and requires considerable time. At the same time, most of the contour accuracy studies are performed directly by using quantitative evaluations, which involves the employment of geometric indices to characterize the similarity between the test and the reference contours.[8] Geometric indices extensively used in contour evaluations include distance-type geometric indices (e.g., the maximum (HD), mean (HDmean), and 95% Hausdorff distances (HD95)) and volumetric geometric indices (e.g., the Dice-similarity coefficient (DSC) and the Jaccard coefficient).[9] Recent research has suggested that despite the fact that these geometric indices can be easily calculated, they do not consider the clinical effect and may lack clinical relevance.[10-12] Furthermore, different geometric indices have different properties, but different automatic segmentation studies were not based on a uniform guide or rules to select geometric indices to evaluate the contour results.[13-15] Subject to the assumption of a reference contour, the method used for the clinical assessment of the accuracy of radiotherapy (RT) contours involves the determination and prediction of the deviation of its dosimetric indices based on the dose distribution of the radiation treatment plan.[10,16-18] However, the relationships between geometric and dosimetric indices still need to be studied in more depth.

This study introduced the systematic and random contour errors based on the following geometric transformations: translation, scaling, rotation, and sine function transformation. Based on these transformations, the specific objectives of this study was a) to investigate the correlations between the geometric and the dosimetric indices, and b) to explore the ability of geometric indices to distinguish the contours with the same transformation type but in different directions.

Materials and Methods

Contouring

Four different types of targets were selected for this study: C-shaped target, oropharyngeal cancer, metastatic spine cancer, and prostate cancer (Figure 1). The C-shaped target was delineated on the water phantom according to the American Association of Physicists in Medicine (AAPM) Task Group No. 119 (TG-119) report,[19] and the remaining 3 types of targets were outlined by senior physicians in our research institution according to institutional clinical protocols. The structures of the targets were exported from the treatment planning system (TPS) Raystation (Raysearch, Stockholm, Sweden) in the form of a Digital Imaging and Communications in Medicine (DICOM) file, and the position information of the contours were read by an in-house software developed in Python (version 3.7.3). The information was used to perform the geometric transformations. Subsequently, the transformed structures were imported back to Raystation system in the form of a DICOM file.

Figure 1.

Example of the PTV contours of (A) the C-shaped target, (B) oropharyngeal cancer, (C) metastatic spine cancer, and (D) prostate cancer.

The targets (original contours) before the transformation were regarded as the delineation results by junior residents (test contours), and the transformed targets were regarded as reference contours after systematic and random errors corrections implemented by senior physicians. The contour errors were introduced in the form of geometric transformations. Through the following 7 geometric transformations, systematic errors were only introduced to the C-shaped target. The translational transformations were divided into the following 3 cases: right, anterior, and posterior directions. Based on the location of the original contours at 1 mm intervals, the contours were moved 10 times in each of the right, anterior, and posterior directions to obtain the reference contours. Scaling transformation represented an equidistant expansion or reduction transformation in reference to the position of the original contour. Considering the fast speed of scaling transformation changes, in the patient modeling module of the Raystation planning system, 10 equidistant transformations were performed at 0.5 mm intervals, excluding the anterior and posterior directions. The rotation transformation was based on the use of the origin of the computer tomography (CT) image coordinates as the rotation center points (at 1° intervals) for 10 clockwise rotations. For the sine function transformation, we extracted the coordinate values (x0, y0) of the original contour first, and then used the function y = sinωy0 (ω = 3, 4, 5, 6,…, 12) to conduct periodic transformations 10 times with a fixed amplitude.

In addition, to introduce random errors, we performed geometric transformations on each target 20 times, and 3 of the above geometric transformation methods were used in the same target randomly for each transformation. To be specific, we randomly selected a quarter of all the CT image layers to remain the same without applying any geometric transformation, and the remaining layers were randomly divided into 3 parts. A geometric transformation method was randomly selected and used for each part and the transformation amplitude was randomly selected within the aforementioned (corresponding) amplitude range. In the end, 20 delineation results were obtained for each of these 4 targets.

Geometric Indices

In this study, we chose 5 extensively used geometric indices for the evaluations: 3 distance-type indices HD (maximum, mean, 95%) and 2 volumetric indices (DSC and Jaccard). These 5 geometric indices were calculated by the 3DSlicer (version 4.10.2),[20] which is an open-source software. The calculation of HD was performed on the exported DICOM-RT files. The HD indices calculated by the 3DSlicer represent bidirectional distances, and the bidirectional distance was symmetrical; this type of distance is more stable than the unidirectional distance calculated by other methods.

Dosimetric Indices

The original clinical plans for these 4 targets used IMRT technology. The C-shaped target met the requirements for a simple version in the TG-119 report, the dose of 5000 c Gy received by 90% of the target volume was used as the prescription, and the dose prescriptions for oropharyngeal, metastatic spine, and prostate cancers were 95% of the target volume, which received 5400, 3000, and 5600 c Gy, respectively, and the dose grid was 2 mm. After the geometric transformation, the RT structures were imported into the radiotherapy plan of the original clinical plans, and D98%, Dmean, D2%, conformity index (CI), and homogeneity index (HI) of the planning target volume (PTV) were obtained. According to the International Commission on Radiation Units and Measurements (ICRU) 83 report,[21] these dosimetric endpoints (D98%, Dmean, and D2%) represent the minimum, mean, and maximum doses received by the target. The CI value of target volume is defined according to the following equation,[12,21]

where TV is the volume of prescribed isodose line enclosed volume, PTV is the volume of targets, and TVPTV represents the overlap volume between the target volume and the prescribed isodose line enclosed volume.

The formula of HI suggested by the ICRU 83 report is the following,

In this study, the dose differences (ΔD) of 3 dosimetric indices D98%, Dmean and D2% were calculated and normalized according to their respective clinical goals.[14] Herein, , where x represents the type of dosimetric index. Given that the clinical prescription requirements of the 4 targets were different, they were normalized with their respective prescription doses.

Analysis

Linear regression analysis was conducted with the software SPSS 21.0 (SPSS, Chicago, IL, USA) and involved the calculation of the coefficient of determination (R2, where R denotes the correlation coefficient).[13] The R2 statistic was used to quantify the correlations between the geometric indices (HD (maximum, mean, 95%), DSC, and Jaccard) and the dosimetric endpoints (D98%, Dmean, and D2%). P-values < 0.05 were considered significant. In addition, the geometric indices obtained from the equidistant scaling transformations and the right, anterior, and posterior directions of the C-shaped PTV translation transformations were compared, and the difference between them was tested for statistical significance using the Wilcoxon signed-ranks test in SPSS. From the scatterplots of the geometric versus the dosimetric indices, the feasibility of the assessment of the accuracy of contours with geometric indices was analyzed.

Results

Linear regression analysis was conducted on the geometric indices and dosimetric endpoints (Tables 1-3). Table 1 shows the correlations between the geometric indices and dosimetric endpoints of C-PTV with systematic errors. With the exception of the sine function transformation (R2: 0.023-0.04, P > 0.05), the geometric transformations of the C-PTV had links with the dosimetric indices D98% and Dmean (R2: 0.689-0.988), 80% of which were P < 0.001. Tables 2 and 3 list the correlations between geometric indices and dosimetric endpoints of actual cases with random errors. The geometric indices were correlated with the dosimetric indices D98% and Dmean, and the correlations of HD were weaker than the other 4 geometric indices. The D2% values of the targets were not included in the correlation analysis because of their small variation range.

Table 1.

Linear Regression Analysis Between Geometric Indices and Dosimetric Endpoints of C-PTV With Systematic Errors.

C-PTV	Transformation	HD		HDmean		HD95		DSC		Jaccard
		R2	P-value	R2	P-value	R2	P-value	R2	P-value	R2	P-value
D98%	Right	0.931	< 0.001	0.942	< 0.001	0.936	< 0.001	0.936	< 0.001	0.894	< 0.001
	Anterior	0.49	0.017	0.528	0.011	0.502	0.015	0.511	0.013	0.436	0.027
	Posterior	0.972	< 0.001	0.985	< 0.001	0.982	< 0.001	0.981	< 0.001	0.956	< 0.001
	Expansion	0.966	< 0.001	0.976	< 0.001	0.958	< 0.001	0.962	< 0.001	0.937	< 0.001
	Reduction	0.84	< 0.0017	0.724	0.001	0.788	< 0.001	0.689	0.002	0.754	0.001
	Rotation	0.949	< 0.001	0.949	< 0.001	0.944	< 0.001	0.943	< 0.001	0.912	< 0.001
	Sine	0.023	0.778	0.03	0.828	0.027	0.889	0.04	0.859	0.04	0.861
Dmean	Right	0.917	< 0.001	0.927	< 0.001	0.923	< 0.001	0.919	< 0.001	0.868	< 0.001
	Anterior	0.651	0.003	0.692	0.002	0.672	0.002	0.675	0.002	0.602	0.005
	Posterior	0.959	< 0.001	0.973	< 0.001	0.964	< 0.001	0.967	< 0.001	0.932	< 0.001
	Expansion	0.971	< 0.001	0.988	< 0.001	0.972	< 0.001	0.977	< 0.001	0.956	< 0.001
	Reduction	0.987	< 0.001	0.94	< 0.001	0.965	< 0.001	0.92	< 0.001	0.955	< 0.001
	Rotation	0.927	<0.001	0.929	<0.001	0.922	<0.001	0.921	<0.001	0.881	<0.001
	Sine	0.024	0.779	0.035	0.72	0.032	0.954	0.027	0.808	0.036	0.708

Table 2.

Linear Regression Analysis Between Geometric Indices and D98% of Targets With Random Errors.

Site	HD		HDmean		HD95		DSC		Jaccard
	R2	P-value	R2	P-value	R2	P-value	R2	P-value	R2	P-value
C-PTV	0.578	< 0.001	0.876	< 0.001	0.822	< 0.001	0.877	< 0.001	0.836	< 0.001
Oropharyngeal cancer	0.478	0.001	0.813	< 0.001	0.788	< 0.001	0.819	< 0.001	0.759	< 0.001
Metastatic spine cancer	0.384	0.003	0.95	< 0.001	0.87	< 0.001	0.95	< 0.001	0.907	< 0.001
Prostate cancer	0.549	< 0.001	0.754	< 0.001	0.752	< 0.001	0.75	< 0.001	0.725	< 0.001

Table 3.

Linear Regression Analysis Between Geometric Indices and Dmean of Targets With Random Errors.

Site	HD		HDmean		HD95		DSC		Jaccard
	R2	P-value	R2	P-value	R2	P-value	R2	P-value	R2	P-value
C-PTV	0.555	< 0.001	0.872	< 0.001	0.744	< 0.001	0.875	< 0.001	0.823	< 0.001
Oropharyngeal cancer	0.48	< 0.001	0.788	< 0.001	0.759	< 0.001	0.797	< 0.001	0.734	< 0.001
Metastatic spine cancer	0.418	0.002	0.957	< 0.001	0.877	< 0.001	0.95	< 0.001	0.907	< 0.001
Prostate cancer	0.522	< 0.001	0.794	< 0.001	0.705	< 0.001	0.781	< 0.001	0.744	< 0.001

Table 4 shows the results of the Wilcoxon signed-ranks test analysis between the geometric indices of translation transformation in the right, anterior, and posterior directions, and equidistant scaling transformation in the opposite direction. For the analysis results of the 5 directions of C-PTV, the P-values of HD, HDmean and HD95, were all greater than 0.05, while the P-values for DSC and Jaccard were all less than 0.05, and the differences were statistically significant.

Table 4.

The Wilcoxon Signed Ranks Test Analysis Between the Geometric Indices of Different Directions of Transformations.

Geometric transformation	Parameter	HD	HDmean	HD95	DSC	Jaccard
Right and anterior direction	Z-value	−1	−0.475	−0.73	−2.825	−2.848
	P-value	0.317	0.631	0.465	0.005a	0.004a
Right and posterior direction	Z-value	−0.535	−0.547	−1.604	−2.823	−2.842
	P-value	0.593	0.563	0.109	0.005a	0.004a
Anterior and posterior direction	Z-value	−1.604	−0.47	−1.604	−2.598	−2.121
	P-value	0.109	0.572	0.109	0.009a	0.034a
Scaling transformation	Z-value	−1.586	−1.790	−1.792	−2.505	−2.533
	P-value	0.113	0.074	0.069	0.012a	0.011a

a P-value = the difference was statistically significant.

Figures 2-6 show the relationships between the geometric indices of the targets and the dose differences (98%, mean). Figures 7 and 8 show the relationship between the geometric indices and CI, HI. For the targets with random or systematic errors, the relationships were inconsistent in different geometric indices. In addition, for the water phantom target (C-PTV) with systematic errors, the relationships were nonmonotonic.

Figure 2.

Relationship between the geometric indices of C-PTV and the dose difference. (A-C) and (D-F) show the relationships between the geometric indices of C-PTV after the introduction of systematic errors and the dose difference D98% and Dmean, respectively. (R-HD = dose difference corresponding to HD value after translation transformation in the right direction; R-HDmean = dose difference corresponding to HDmean value after translation transformation in the right direction; R-HD95 = dose difference corresponding to HD95 value after translation transformation in the right direction; R-DSC = dose difference corresponding to DSC value after translation transformation in the right direction; R-Jaccard = dose difference corresponding to Jaccard value after translation transformation in the right direction; A = anterior direction translation transformation; P = posterior direction translation transformation; Ex = expansion transformation; Re = reduction transformation; Ro = rotation transformation; Sine = sine function transformation).

Figure 3.

Relationship between the geometric indices of C-PTV and the dose difference. (A-B) and (C-D) are the relationships between the geometric indices of C-PTV after the introduction of random errors and the dose difference D98% and Dmean, respectively.

Figure 4.

Relationship between the geometric indices of oropharyngeal cancer and dose difference. (A-B) and (C-D) are the relationships between the geometric indices of oropharyngeal cancer after the introduction of random errors and the dose difference D98% and Dmean, respectively.

Figure 5.

Relationship between the geometric indices of metastatic spine cancer and the dose difference. (A-B) and (C-D) are the relationships between the geometric indices of metastatic spine cancer after the introduction of random errors and the dose difference D98% and Dmean, respectively.

Figure 6.

Relationship between the geometric indices of prostate cancer and the dose difference. (A-B) and (C-D) are the relationships between the geometric indices of prostate cancer after the introduction of random errors and the dose difference D98% and Dmean, respectively.

Figure 7.

Relationships between the CI (A-C), HI (D-F) and geometric indices of C-PTV after the introduction of systematic errors.

Figure 8.

Relationships between the CI, HI, and the geometric indices, after the introduction of systematic errors. (A-D) C-PTV, (E-H) oropharyngeal cancer, (I-L) metastatic spine cancer, and (M-P) prostate cancer.

Discussion

Many studies have shown that although it is important to quantify the degree of variation or uncertainty of the contouring, it is more important to determine the dose difference and clinical impact.[10,11,15,17,18,22,23] In an earlier work, Ward van Rooij et al [18] studied the accuracy of automatic delineation of organs at risk in the head and neck regions based on deep learning techniques, while the use of geometric and dosimetric indices allowed them to analyze the correlation between the geometric index SDC (Sørensen-Dice similarity coefficient) and dose difference. They found that there was a weak correlation between the SDC and ΔD for all of the OARs based on automatic segmentation, r = −0.24, P = 0.002, but the correlation was not specific to a certain OAR or a certain patient. This is partly similar with the results described in our study. Ward van Rooij et al evaluated the OARs, and our study evaluated the results of the targets. Dosimetrically, this is a major difference. For the targets, high doses need to be achieved, while for the OARs, high doses need to be avoided. Given that all the beams are directed toward the targets, these volumes are much more sensitive to dosimetric changes in cases where there are contour errors; this effect is thus considerably less prominent in OARs. We found that the geometric indices obtained by geometric transformation were correlated with the dosimetric indices, but for some specific geometric transformation forms, the situations were different, and the correlation was not consistent for the different forms of the geometric transformations. A correlation existed between the geometric indices and dosimetric endpoints in the translation, scaling, and rotation transformations of the C-PTV with systematic errors, but the results for the sine function transformation were not significant or weak. Following the periodic transformation of sine function with fixed amplitude, the C-PTV’s contour changed very little, and the HD values were less than 2 mm (Figure 2B). These outcomes led to small-dose differences, and correspondingly, the correlation between them was weak. At the same time, this re-emphasizes the importance of contour training for junior residents, whereby the repeatability of contouring is high, the contour difference is small, and the dose difference is also small.

The correlation obtained by the C-PTV anterior direction translation was lower than the correlation obtained by the other 2 translation transformations. To avoid high-dose radiation to the surrounding OARs (Figure 1A), physicians try to maintain high-dose areas away from the OARs when designing the original radiotherapy plan. The structures in the low-dose region are less likely to have noticeable dose changes even if its contour varies greatly in the spatial domain.[24] When the translation occurred in this area along the anterior direction, the minimum dose (D98%) of the target in this area was almost unchanged. This resulted in a weaker correlation, and the correlation of Dmean was higher than D98%. In addition, as it can be observed in Figure 7, the HIs are almost unchanged during the anterior direction translation of the C-PTV; additionally, the relationships between the geometric indices and CIs, and the HIs in the cases of other transformation methods are also different. This is consistent with the study by Lim et al,[10] which found that the correlation between geometric indices and dosimetric indices was affected by the goals of the treatment plan. From these studies, it can be shown that the correlation between geometric and dosimetric indices can be affected by many factors, such as the geometric transformation method, the relative positions of the target and OARs, and the constraint goals of the radiotherapy plan.

According to the analysis outcomes of the Wilcoxon signed-ranks in Table 4, for the geometric transformation results of the C-PTV, the distance-type geometric indices HD, HDmean and HD95 cannot express the difference of the translation transformations in the right, anterior, and posterior directions, and the difference at different equidistant scalings. However, there were significant differences between the volumetric geometric indices obtained from the different transformation directions in irregular shape targets. The HD values of the equidistant scaling transformation were the same for C-PTV, but the clinical effect on them were different. As shown in Figure 2B, when HD = 1.547 mm, the dose difference was within 5% for equidistant expansion transformation, while for equidistant reduction transformation it was beyond −5%. The geometric indices obtained from the same type of geometric transformation in different directions were not distinguishable. For a target contour, different types of geometric indices have the same value, and the corresponding dose differences are different. Beasley et al [25] reported that when measured with a suitable spatial metric, the higher the geometric accuracy of the contour is, the smaller the dose difference should be, and vice versa. When the distance-type geometric index value was approximately 3 mm, the dose difference corresponding to HDmean was already close to 30% (see Figures 2 –6). That is why this study asserts that HDmean is not suitable as an index for evaluating contour accuracy alone despite the highest correlation coefficient of HDmean. Compared with HD and HD95, it is too sensitive and the change gradient is too large to reflect the actual clinical situation. Given that there is no independent reference standard for distance-type geometric index values, it is impossible to compare the contour accuracy across different structures. This implies that although 2 different structures have the same geometric index value, they cannot indicate whether the quality of the contour is the same. As shown in Figures 2 –8, the quality of the contour is controversial for the 2 structures (such as oropharyngeal and prostate cancers) with the same geometric index value. Additionally, it is difficult to compare the contour accuracy among different automatic segmentation studies by only using the distance-type geometric index. The article on the study of automatic segmentation of OARs showed that the HD (Liver) = 15.770 ± 1.0 mm had a high accuracy,[26] while another article indicated that an HD = 37.7 ± 13.8 mm also yielded a high accuracy.[8] Although the former value is smaller than the latter, the identification of the value with the higher accuracy cannot be clearly stated owing to the lack of a standard reference.

For volumetric geometric indices, many studies indicated that if the DSC was higher than the normally reported value of 0.7, the agreement between the reference contour and the test contour was considered to be good.[23,26,27] Our research showed that when the DSC and Jaccard values of the anterior direction translation transformation were between 0.5 and 0.7, the corresponding dose differences were also very small (Figure 2C). At the same time, there were some cases in which the DSC values were greater than 0.7, and the corresponding dose difference values were large. These 2 contradictory situations show that it is not reliable to set an acceptable threshold for DSC.

This study introduced the systematic and random geometric errors through translation, scaling, rotation, and sine function geometric transformations; analyzed the feasibility of the clinical evaluation of geometric indices, and determined the ability of geometric indices to identify the direction of transformation. Although geometric indices reflected the geometric differences between test and reference contours, and the correlation between the geometric index and the dosimetric endpoints in this study were relatively high, the relationship between geometric and dosimetric indices was not consistent among different geometric indices, different transformation forms, and different targets. Thus, it was illogical to use only geometric indices to evaluate the clinical acceptability of contour results. In addition, our current research was based on the simulation experiment of geometric transformation. Accordingly, we should explore further the relationship between geometric and dosimetric indices using the actual contouring results cases of the junior residents.

Conclusion

At present, there is a lack of guidance for the evaluation of contours using geometric indices. Therefore, there is a need for a normative framework. We found that the differences between the geometric and dosimetric indices were not consistent. This justifies the inaccuracy arising in instances where only the geometric indices are used to evaluate the results of contouring. The clinical acceptability of contouring results cannot be judged by geometric indices alone. Therefore, we suggest that dosimetric indices should be added to the evaluations of the accuracy of the results of automatic segmentation or contouring training of the residents, which can be helpful in explaining the clinical dose–response relationship of delineation more comprehensively and accurately.