Validation of automated magnetic resonance image segmentation for radiation therapy planning in prostate cancer.

BACKGROUND AND
PURPOSE: Magnetic resonance imaging (MRI) is increasingly used in radiation therapy planning of prostate cancer (PC) to reduce target volume delineation uncertainty. This study aimed to assess and validate the performance of a fully automated segmentation tool (AST) in MRI based radiation therapy planning of PC.
MATERIAL AND METHODS: Pelvic structures of 65 PC patients delineated in an MRI-only workflow according to established guidelines were included in the analysis. Automatic vs manual segmentation by an experienced oncologist was compared with geometrical parameters, such as the dice similarity coefficient (DSC). Fifteen patients had a second MRI within 15 days to assess repeatability of the AST for prostate and seminal vesicles. Furthermore, we investigated whether hormonal therapy or body mass index (BMI) affected the AST results.
RESULTS: The AST showed high agreement with manual segmentation expressed as DSC (mean, SD) for delineating prostate (0.84, 0.04), bladder (0.92, 0.04) and rectum (0.86, 0.04). For seminal vesicles (0.56, 0.17) and penile bulb (0.69, 0.12) the respective agreement was moderate. Performance of AST was not influenced by neoadjuvant hormonal therapy, although those on treatment had significantly smaller prostates than the hormone-naïve patients (p < 0.0001). In repeat assessment, consistency of prostate delineation resulted in mean DSC of 0.89, (SD 0.03) between the paired MRI scans for AST, while mean DSC of manual delineation was 0.82, (SD 0.05).
CONCLUSION: Fully automated MRI segmentation tool showed good agreement and repeatability compared with manual segmentation and was found clinically robust in patients with PC. However, manual review and adjustment of some structures in individual cases remain important in clinical use.

Introduction

Prostate cancer (PC) currently represents 15% of all diagnosed cancers among men. In 2012, almost 1.1 million new cases occurred, which resulted in 307,000 deaths [1]. Magnetic resonance imaging (MRI) is increasingly used in prostate radiation therapy planning (RTP) for both target and normal structure delineation because of its superior soft tissue contrast [2]. The inter-observer variability in defining e.g. prostate apex is smaller with MRI compared to computed tomography (CT) [3]. MRI-based delineation has been associated with reduced late toxicity of PC radiation therapy (RT) as shown by lower urinary frequency and urinary retention toxicity scores [4]. Recently introduced MRI simulation platforms have enabled the use of MRI as the stand-alone imaging modality for prostate RTP [5], [6], [7]. Multiple groups have demonstrated that dose calculation accuracy in MRI-only RTP is comparable to that of CT-based RTP [8], [9]. MRI as the sole imaging modality eliminates potential errors introduced by co-registration with CT or interval changes in organ filling and movement between the two scans [3]. Furthermore, MRI-only RTP is more patient-friendly and resource and cost-efficient than the traditional combination approach [2].

In RT, the technological development of imaging modalities, RTP systems, and linear accelerators have enabled highly conformal dose distributions. In this context, the delineation of target and normal structures in the RTP workflow has become highly important. However, manual segmentation of organs at risk (OAR) can be highly time and resource intensive, and there is a great need to develop and validate automated image segmentation tools (AST). Recently, AST has been adapted in multiple organ sites with deformable-model, atlas-based and deep learning algorithms and reviewed by Balagopal et al. [10] and Boldrini et al. [11]. Initial experience in using AST for PC RT has been reported both for MRI and CT-based approaches with promising findings and dice similarity coefficient values (DSC) for prostate contouring varying between 0.86 and 0.90 [10], [12]. Thus far, these studies have included only a limited number of patients and have not resulted in widespread adoption of AST outside academic hospitals.

The aim of this study was to assess and validate the performance of a fully automated MRI segmentation tool in RTP of PC. To this end, we compared manual and automated OAR segmentation and investigated whether the latter is repeatable in clinical practice.

Materials and methods

Patients

This prospective study was carried out at the Department of Oncology and Radiotherapy of Turku University Hospital (Turku, Finland). The study protocol was reviewed and approved by the Ethics Committee of the Hospital District of Southwest Finland (ETMK Dno 115/1801/2017) and informed consent was obtained from those patients who underwent a second MRI for the repeatability sub-study. Because MRI is part of routine RTP workflow in PC at the performing institution, no consent was needed for patients receiving a single MRI.

We enrolled 65 consecutive men referred to definitive prostate MRI-only RT between September 2017 and August 2018. Patients with newly-diagnosed histologically confirmed local or locally advanced PC (clinical stage T1c-T3bN0-N1) were eligible. All men had trans-rectal ultrasound, digital rectal examination, bone scintigraphy and abdominal CT as part of their diagnostic work-up. Other inclusion criteria were: age between 40 and 90; WHO (World Health Organization) performance status of 0–1; no contraindications for MRI and no prior oncologic treatment except for 4 to 6 months of neoadjuvant anti-androgen therapy. Fiducial nitinol markers (Beampoint AB, Sollentuna, Sweden) were placed in the prostate for all patients according to local protocol. Out of 65 men, 15 had two MRI scans while the remaining 50 had a single MRI. Detailed patient demographics are given in Table 1.

Table 1

Patient characteristics (N = 65) showing the clinical risk groups and use of antiandrogen treatment.

	N (%)
Risk group low	0 (0)
Risk group intermediate	20 (31)
Risk group high	45 (69)
PSA1 < 10 µg/l	30 (46)
PSA 10–20 µg/l	27 (42)
PSA > 20 µg/l	8 (13)
Hormonal treatment	59 (91)
BMI2 (mean, range) [kg/m2]	28 (21–41)
Age (mean, range) [years]	72 (51–83)

Prostate specific antigen.

Body mass index.

Magnetic resonance imaging

All men were requested to use enema either night before or in the morning of study. They were advised to void and then drink 400 ml of water one hour before imaging which was performed with a 1.5 T MRI scanner (Philips Ingenia 1.5 T HP, sw. version 5.3.1, Philips MR Medical Systems International B.V., Best, Eindhoven, The Netherlands). This dedicated MRI RTP platform includes a flat RT-indexed couch top and an external laser positioning system (LAP GmbH Laser Applikationen, Lüneburg, Germany). In pelvic acquisition, the scanner-integrated MRI body and posterior coils are applied together with an anterior coil placed above the patient using a coil holder. The MRI scans were acquired in supine position and total imaging time was 18 min. No endorectal coil or gadolinium contrast was used. In the repeatability evaluation a second scan was obtained on the day of start of RT a median of 8 days (range of 6–15) after the first MRI. Images were rigidly co-registered with 6 degrees of freedom based on the implanted fiducial markers.

Automated and manual segmentation of MR images

The automated segmentation tool (AST) (Philips RTdrive Core 2.0, Philips Medical Systems Netherlands B.V.) was a model-based adaptive algorithm on the Philips MRI scanner console that parallel with the MR simulation process created standard anatomical structures required for RTP of PC. Boundary detection of body, prostate, seminal vesicles (SV), bladder, rectum, femoral heads and penile bulb were based on machine learning from a collection of validated ground truth segmentations. The structures were derived from dedicated T2-weighted turbo spin echo (TSE) images and MRCAT (magnetic resonance for calculating attenuation) source T1-weighted mDIXON [13], [14], [15]. Details on the imaging sequences are found in Supplementary Table 1. Possible inter-sequence organ movement between the source MR images was corrected with an automated rigid 3D image registration during the autosegmentation post processing.

The manual contouring process on MRI followed the ESTRO-ACROP guidelines [16]. All contouring data were divided into three groups: 1) AST, 2) the clinical investigator (CI) and 3) radiation oncologists (RO). The CI was a certified specialist in oncology (A.K.) with 6 years of experience in contouring and RTP of PC. The RO were a mixed group of specialists and residents in radiation therapy having between 6 months to 30 years’ experience. They did the contouring as part of their routine duties. Manual delineations were created with an RTP system (Eclipse™ version 13.6.23, Varian Medical Systems Finland Oy, Helsinki, Finland).

Manual contouring was considered the reference method and always applied in clinical practice. AST contours were generated after all standard manual delineations had been finished, and the auto-generated contours were not used prospectively in RTP of patients. The CI was blinded to the structures contoured by the RO, and vice versa. Furthermore, a radiotherapist visually inspected the AST contours for outliers, i.e. cases where the AST clearly misperformed. For repetitive evaluation AST and CI delineations were performed also on the second MRIs.

Geometrical parameters for evaluation

Structure sets containing the contoured volumes of prostate, SV, bladder, rectum and penile bulb delineated by the CI and RO and calculated by the AST were analysed in Eclipse™ workstation. Contours manually delineated by the CI were used as ground truth. Different metrics, including DSC, absolute volume difference (AVD), centre of mass shift (CMS), and Hausdorff distance (HD95), were determined to quantify the similarity between the auto-segmented and the manually delineated volumes [9], [17], [18]. See supplementary material for definitions of DSC, AVD, CMS and HD95.

Statistical analyses

All data were presented as mean with standard deviation (SD) and range, or counts and percentages. Geometrical parameters were analysed with multiway analysis of covariance (ANCOVA), including observer (AST, CI or RO) and hormones as categorical factors and body mass index (BMI) as a continuous covariate. Only if the main effect was significant (e.g. an observer), were pairwise comparisons applied. Normal distributions of the variables were evaluated from studentized residuals. Natural logarithm transformation was performed to the volumes to achieve normality of distributions. Pearson correlation coefficients were calculated for BMI and DSC difference of SV. All tests were performed as two-sided with a significance level set at 0.05. Confidence intervals of 95% for means were calculated. The analyses were performed using SAS System version 9.4 for Windows (SAS Institute Inc., Cary, NC).

Results

Comparison between automated and manual contouring

A summary of results for structure volumes representing OARs is presented in Fig. 1 and in Supplementary Table 2. The mean total time for contouring CTV and all OARs manually by CI was 26 (SD 7) min. Out of the 65 patients visual inspection detected 8 prostate (12%), 4 rectum (6%), 4 bladder (6%) and 14 SV (22%) outliers in AST delineations. These were omitted from further analysis.

Fig. 1

Tukey boxplots of measured structure volumes for each evaluator. AST denotes to automatic segmentation tool, CI to clinical investigator, and RO to radiation oncologists. Please note the different scales for volumes on y-axes.

The contours of prostate, bladder, and rectum delineated manually by physicians were well comparable with those generated by the AST. The investigated geometrical parameters DSC, HD95 and AVD between the CI or RO and the AST showed no clinically relevant differences in these structures (Fig. 2 and Supplementary Table 3). For SV and penile bulb, the concordance of manual and automated segmentation was somewhat inferior. However, this was in line with differences between the CI and RO as well.

Fig. 2

Mean Dice similarity coefficient (DSC) results for contoured structures for 65 prostate cancer patients. The auto-segmented structures were compared to manually delineated structures of both radiation oncologists (RO) and clinical investigator (CI). Error bars are equal to one standard deviation.

Prostate, bladder and rectum CMSs derived from CI and AST were mainly consistent within a millimetre in all coordinate directions (Fig. 3 and Supplementary Table 4). Two exceptions were revealed: the CMS in anterior-posterior (AP) direction for prostate was 1.2 mm and the CMS in head-feet (HF) direction for rectum was 1.5 mm. In further comparison of AST vs. CI, SV and penile bulb showed CMS > 1 mm more frequently in AP and HF directions while the shift in LR (left-right) direction remained less than a millimetre.

Fig. 3

Comparison between main coordinate directions expressed as Tukey boxplots of centre of mass shift (CMS) for structures delineated by automatic segmentation tool (AST) vs. clinical investigator (CI).

An example of different delineations performed by AST and CI on synthetic CT is seen in Fig. 4.

Fig. 4

A good example of different delineations between automatic segmentation tool (AST) and clinical investigator (CI) on synthetic CT. Colors (light/dark for AST/CI) and DSC (dice similarity coefficient) for delineations are as follows: prostate (red, 0.86), bladder (green, 0.92), rectum (brown, 0.88), penile bulb (blue, 0.80), and seminal vesicles (purple, 0.76). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Repeatability measurements

Out of 65 PC patients, 15 had a second MRI scan performed on the first day of treatment. The individual DSC between the repeated ASTs and manual contours by CI for prostate are presented in Fig. 5. The mean DSC for prostate was 0.89, SD 0.03 (range 0.85–0.94) for AST and 0.82, SD 0.05 (range 0.73–0.89) for CI, respectively. The mean absolute volume difference (AVD) for prostate was 2.8 cm3, SD 1.9 cm3 (range 0.8–6.6 cm3) for AST and 2.4 cm3, SD 1.7 cm3 (range 0.5–6.3 cm3) for CI, respectively.

Fig. 5

Dice similarity coefficient (DSC) of prostate contours in repetitive MRI in 15 prostate cancer patients for automatic segmentation tool (AST) and clinical investigator (CI).

Effect of antiandrogen therapy and body mass index

AST performed as well for patients on hormonal therapy as hormone-naïve patients. No significant differences were found in the investigated parameters between CI and AST. DSCs for prostate were 0.84 and 0.85 (p = 0.85) and for SV 0.56 and 0.60 (p = 0.58), in patients on hormonal therapy and in hormone-naïve patients, respectively. Furthermore, patients on neoadjuvant hormonal treatment had significantly smaller average volumes for prostate (32 ml vs 49 ml, p = 0.004), and for SV (10 ml vs 15 ml, p = 0.023), respectively.

The mean BMI of the study population was 28 (range of 21–41) kg/m2. There was a statistically significant difference in performance of AST between lean and obese men when using BMI 30 kg/m2 as cut-off. Men with BMI ≥ 30 (n = 16) had significantly lower DSC for rectum (0.83 vs. 0.86, p = 0.032) and higher DSC for SV (0.62 vs. 0.52, p = 0.039) than men with BMI < 30 kg/m2 (n = 47). BMI showed no significant effect on the volume of prostate, bladder, rectum or SV.

Discussion

The present study investigated the utility of AST to generate clinically relevant contours for RTP of PC. A commercial model-based algorithm developed for MRI-only and automated RTP workflow was employed. In comparison to the manual segmentations, regarding DSC, AST showed high agreement for prostate, bladder, rectum, and moderate agreement for SV and penile bulb. Visual inspection revealed a few obvious outliers which would had been manually corrected for in a clinical setting and were therefor excluded from the analysis. The CMS showed a systematic 1 mm shift in the AP direction. Compared with visual inspection of the results this difference did not seem clinically relevant. The found HD95 values (in Supplementary Table 3) are in line with those reported by Delpon [19] and Wong [20] which both investigated CT based auto-segmentation tools. Specially, for the prostate the HD95 of 4.6 mm (95% confidence interval 4.3–4.9 mm) is lower than that reported by Wong (6.7 mm), indicating that MRI based delineation may improve accuracy.

Based on the sequential MRI scans of 15 patients AST was highly repeatable with a minimum DSC of 0.85 for the prostate. Our results in Fig. 5 indicate that the repeatability of AST is superior to CI. Due to the daily variability in filling and movement of the other organs and since the co-registration between the image sets was based on fiducial markers the repeatability was not investigated for bladder, rectum, penile bulb and SV. Furthermore, co-registration independent AVD showed similar agreement for AST and CI.

The performance of AST was almost independent of BMI and feasible for both men off and on hormonal therapy even if the latter group tended to have smaller prostate and SV. Although the difference in prostate volumes between those on and off treatment was significant (p = 0.0001) the small number of hormone naïve patients indicates that further studies need to be conducted to confirm our observation. It is, however, appropriate to assume that short-term neoadjuvant hormonal therapy reduced the size of prostate in our patients in accordance with other studies [21].

Previously, MRI-based prostate segmentation has mainly been used for diagnostic applications while only few studies have attempted to delineate prostate and SV for RTP. However, these studies have demonstrated small inter-physician variability (0.7–1.7 mm) [3] and good volume overlap (0.78–0.88) for prostate, bladder and rectum between automatic and manual delineations [12]. In our study, good consistency between automated and manual delineation of prostate, SV and OAR was indeed observed, and the inter-observer variability was comparable to previous reports [3].

It is generally accepted that a DSC > 0.7, 0.5–0.7 or <0.5 denotes good, moderate or poor agreement between reference and test structures, respectively [22]. While our DSC metrics comparing automatic and manual segmentation resulted in good agreement for the most important RTP organs, the findings for the penile bulb and SV showed only moderate agreement. In general, the consistency of segmentation depends among other things on the size and composition of the structures. This explains the moderate DSCs for the smaller organs e.g. SV and penile bulb both between CI and AST as well as CI and RO. In line with this, the penile bulb is poorly visualized even by MRI and prone to subjective concepts about its demarcation from the body of the penis. Langmack et al. [23] found poor agreement for SV in their study with atlas-assisted segmentation based on both CT and MRI. They assumed it to be reflection of both the difficulty in visualizing them and determining which part of them to outline.

In general, there is a notable variability in previously reported DSC-values which reflects the multiple differences in protocols used for RTP. For instance, Dowling et al. [9] reported DSCs denoting moderate agreement for prostate, bladder and rectum of 0.70, 0.64 and 0.63, respectively. Korsager et al. [24], on the other hand, reported a mean DSC of 0.88 for prostate. Delpon et al. [19] compared contours produced from CT images by a radiation oncologist to contours computed by five different automated atlas-based segmentation algorithms and the mean DSC varied between 0.59–0.81 for the bladder, and 0.49–0.75 for the rectum.

A potential limitation of our study is the use of only one delineator (CI) as a reference contouring physician. CI delineation is considered “the gold standard”, but cannot be regarded to represent the absolute ground truth due to inter-observer variability [10]. On the other hand, the comparison of DSC between CI and RO showed high agreement indicating concordance between multiple delineators with varying amounts of experience. In essence, we wanted to assess the feasibility and reproducibility of AST for clinical use. The mean contouring time encompassing clinical targets and OARs was 26 min. Pathamanathan et al. [25] have reported a median contouring time of 9.6 min by therapeutic radiographers for the prostate only using a T2*-weighted MR image. This is in line with our delineation times which include multiple organ structures in addition to the prostate.

Clinical use of AST will most likely reduce the overall contouring time since for most structures the delineations were streamlined with only minor deviations as compared to CI and RO. Further investigations are planned to investigate if the indicated minor corrections in the end would affect dose distributions and treatment. Obvious outliers would anyway need to be edited manually with most corrections needed for SV having an approximate failure rate of 20%. However, AST did not fail contouring of all structures for any of the patients and therefore the overall workload would be reduced in these cases as well. One limitation was that the investigated AST was vendor and machine dependent which restricts its accessibility. Furthermore, for patients unsuited for MRI or those with hip prostheses automatic contouring was not possible.

Here, we have validated a commercially available AST to accurately delineate prostate, bladder and rectum for the purpose of an automated MRI-only based workflow in RTP of PC. Our results indicate high reproducibility of prostate delineation with AST. In a clinical setting attention needs to be paid to the delineated seminal vesicles which might need editing on a more regular basis. Overall, an automated RTP workflow would benefit from an automatized delineation check point to alert for obvious outliers prior proceeding to dose calculation.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Anna Kuisma and Iiro Ranta hold a research grant from Philips MR Therapy Oy with permission to publish. Marko Pesola was and Lizette Warner is employed by Philips MR Therapy Oy (Vantaa, Finland).