Deep learning-based automatic segmentation for size and volumetric measurement of breast cancer on magnetic resonance imaging.

Purpose: In clinical work, accurately measuring the volume and the size of breast cancer is significant to develop a treatment plan. However, it is time-consuming, and inter- and intra-observer variations among radiologists exist. The purpose of this study was to assess the performance of a Res-UNet convolutional neural network based on automatic segmentation for size and volumetric measurement of mass enhancement breast cancer on magnetic resonance imaging (MRI). Materials and methods: A total of 1,000 female breast cancer patients who underwent preoperative 1.5-T dynamic contrast-enhanced MRI prior to treatment were selected from January 2015 to October 2021 and randomly divided into a training cohort (n = 800) and a testing cohort (n = 200). Compared with the masks named ground truth delineated manually by radiologists, the model performance on segmentation was evaluated with dice similarity coefficient (DSC) and intraclass correlation coefficient (ICC). The performance of tumor (T) stage classification was evaluated with accuracy, sensitivity, and specificity.
Results: In the test cohort, the DSC of automatic segmentation reached 0.89. Excellent concordance (ICC > 0.95) of the maximal and minimal diameter and good concordance (ICC > 0.80) of volumetric measurement were shown between the model and the radiologists. The trained model took approximately 10-15 s to provide automatic segmentation and classified the T stage with an overall accuracy of 0.93, sensitivity of 0.94, 0.94, and 0.75, and specificity of 0.95, 0.92, and 0.99, respectively, in T1, T2, and T3. Conclusions: Our model demonstrated good performance and reliability for automatic segmentation for size and volumetric measurement of breast cancer, which can be time-saving and effective in clinical decision-making.

Introduction

Breast cancer is one of the most common malignancies afflicting women worldwide (1). Tumor size has been thought as an indispensable prognostic factor. An accurate preoperative measurement of breast cancer size is essential for surgical resection and the formulation of a chemotherapy regimen (2–4). Furthermore, monitoring the change of tumor volume during treatment is an important reference for response evaluation criteria in solid tumors (5). Thus, it is crucial to measure size and volume accurately in the clinical course.

Medical imaging, which is superior in measuring tumor size and volume, might be used to obtain anatomic information accurately and non-invasively (6–10). Among the imaging methods, magnetic resonance imaging (MRI) is a better diagnostic technique with the highest resolution and quantitative information for preoperative prediction and prognosis evaluation (11–14). However, it takes considerable time and a great deal of expertise to process images by trained radiologists. In addition, due to differences in diagnostic skills, there are inter- and intra-observer variations among radiologists and problems with decision fatigue (15, 16).

Artificial intelligence (AI) aiding medical imaging technologies exceeded the detection capabilities of radiologists in some applications, complemented clinical decision-making, and streamlined preoperative image evaluation. Automated processing by AI computational tools is a more efficient detection approach to measure the volume and the size of a tumor within a reasonable amount of time. It has great reference significance for guiding the clinical development of follow-up treatment plans and avoiding inaccurate measurement incurred by some inexperienced radiologists (17). In addition, some studies indicated that the presence of tumor necrosis correlated with tumor grade, aggressiveness, unfavorable long-term outcomes, and improved response to neoadjuvant chemotherapy (18–20). Measuring the necrosis and the cystic components manually is labor-consuming, but using AI technology improves the efficiency and provides more intuitive parameters for radiologists.

Segmentation plays a significant role in image analysis, including detection, feature extraction, classification, and treatment (21, 22). Automatic and semiautomatic segmentation can alleviate the labor-intensive problems and eliminate the high variability between intra- and inter-observers (23). Moreover, deep learning, as a subset of AI, is a promising method to make a tremendous progress in automatic segmentation by which more reproducible and effective texture features in different fields of image analysis are extracted (24–26). The convolutional neural network (CNN) is a sophisticated deep learning architecture, and it has been successfully applied in various areas of knowledge for digital image segmentation. The U-Net network is a fully CNN with high-performance in graphics processing unit (GPU) computing, requiring fewer training sets, and has higher segmentation accuracy compared with other CNNs (27). Among the U-Net network, Res-UNet is a semantic segmentation model which integrates residual module and U-Net network capable of effectively overcoming excessive parameters and gradient dispersion caused by the deepened network layer (28).

In this study, we developed a deep learning automatic segmentation model based on Res-UNet of preoperative MRI for breast cancer patients and assessed its reliability for size and volumetric measurement. To our knowledge, no reported research has applied deep learning to automatically segment breast cancer and quantify the volume as well as the size on MRI.

Materials and methods

The institutional review board approved this retrospective study and waived the need for written informed consent.

Study design

The workflow of the process is illustrated in , including the following three steps: (1) acquisition of MRI, data annotation, automatic segmentation, image pre-processing, augmentation, and post-processing, (2) designing and building the algorithm, and (3) training and inference.

Figure 1

Workflow diagram.

Patient selection and data annotation

All selected female patients were diagnosed with breast cancer who underwent preoperative breast MRI prior to treatment from January 2015 to October 2021. The inclusion criteria were as follows: (1) diagnosed with breast carcinoma pathologically, (2) underwent MRI prior to treatment, (3) with complete clinical and pathological data, and (4) whose digital imaging and communications in medicine pixel data had no corruption and which were scanned under the same MR protocol. The exclusion criteria were as follows: (1) received any therapy before MRI and (2) non-mass enhancement breast cancer or normal in MRI.

According to the Cancer Staging Manual of the American Joint Committee on Cancer, the system clarified that the tumor (T) stage is based on the size of the invasive components of the longest tumor dimension (in the setting of multiple masses). Our study classified the tumor into three T categories: the size of T1 is not greater than or equal to 20 mm, the size of T2 is larger than 20 mm and not greater than 50 mm, and the size of T3 is equal or greater than 50 mm. A total of 1,000 patients were randomly divided into group 1 (n = 230), group 2 (n = 720), and group 3 (n = 50). The following ratios were used: 80% training cohort and 20% testing cohort to balance the test samples. Thus, we selected 45, 143, and 12 cases relatively for three T categories as testing cohort. In addition, 31 cases with cystic or necrotic changes were enrolled in our study.

MRI acquisition

All patients were scanned using a 1.5-T system (Magnetom Espree Pink; Siemens, Erlangen, Germany), which is equipped with an eight-channel phased-array surface coil for the breast. The patients were examined in the prone position with both breasts positioned in the coil cavity. Axial T1WI [repetition time/echo time (TR/TE), 8.7/4.7 ms; slice thickness, 1.1 mm]. Dynamic contrast-enhanced MRI (DCE-MRI) used a fast, small-angle excitation, three-dimensional imaging (3D-FLASH) sequence and fat-saturated axial T1WI: TR/TE, 4.53/1.66 ms; slice thickness, 1.1 mm. Before the contrast agent was injected, it needed to be scanned one time. After that, the contrast agent, gadopentetate dimeglumine, was injected with a high-pressure syringe at a speed of 2.0 ml/s, and then 30 ml normal saline was injected at the same speed to flush the catheter. Images of each phase were subtracted automatically at the same time.

Delineation of ROIs by iterative workflow

A radiologist used the ITK-SNAP software (www.itksnap.org) to review the first DCE-MRI subtraction images, this being the most critical and the clearest phase of breast cancer evaluation for further analysis. An iterative-label workflow was used to delineate the regions of interest (ROIs) in the early stage to get the ideal labels. It included an initial network model which was trained on our in-house dataset from 100 patients’ ROIs and the pre-trained model which was applied to the remaining patients’ ROIs and achieved coarse labels. After that, two radiologists checked and refined the manual revision. The iterative workflow is shown in .

Figure 2

Iterative labeling workflow.

Image processing

We designed a fully automatic CNN-based segmentation network and built an end-to-end workflow based on Medical Open Network for AI platform, including pre-processing, data loader, augmentation, network building, post-processing, and quantification.

Pre-processing

Three steps including intensity normalization, respacing, and crop patches were performed before the training model. Z-score normalization was suitable for variable intensity ranges. We calculated the intensity ranges, clamped the voxels from 0.5 to 99.5%, calculated the mean and standard deviation (SD) of each case, and used the equation to normalize the images. It is defined as shown in Equation (1):

To automatically adapt to any new dataset, we calculated the mean spacing of all training cohorts to define the standard target. Any data needed to be resized to the target before training and inference. Most cases had the dimension of the width and the height as 384 and the depth from 128 to 320. Due to the limitation of the GPU memory size, it was challenging to send the whole image to the network. In this situation, we first calculated the average area of the lesions and set a minimum cropping patch size which can include the central regions. The cropping patch size must also be a multiple of 2 to be suitable for most regular models. According to the statistics, a patch with a 96*96*96 size was the best choice for our algorithm.

To ensure the balance of positive and negative samples for network training, we randomly selected the cropping patches with the center point at the foreground or background area with half-to-half probability. According to our experiments, crop patches with the likelihood of 2:1 between positive and negative areas can also get a similar performance.

Augmentation

Our algorithm implemented data augmentation to make the model more robust during training steps: random zoom, random scale intensity value, random shift intensity range, random Gaussian noise, random crop fore/background, random rotation with 90°, and random elastic transformation ( ).

Figure 3

Workflow of data augmentation.

Res-UNet network building

U-Net is an overall architecture for 2D and 3D images in medical image processing. Our study designed a robust U-Net-based network called Res-UNet with the residual blocks in the encoder part. shows the architecture of our designed model. In the encoder part, we used residual blocks to extract features. Skip connection was a classical operation from U-Net and might focus on the extracted features from different layer levels. It was well suited for medical images since lesion targets from different scale levels included different features. illustrates the residual blocks. The solid line carrying the layer input to the addition operator was a residual connection. The residual connection might effectively avoid gradient disappearance, especially in deeper layers. We combined the residual connection blocks with the U-Net skip connection to design an efficient network, which might help us obtain the accurate prediction results of lesion segmentation.

Figure 4

Res-UNet architecture.

Figure 5

Residual blocks.

Optimizer and loss function

In our algorithm, we used Adam optimizer, dice similarity coefficient (DSC), and cross-entropy loss. The equations are shown in Equations (2)–(4):

Post-processing

Usually, the tumor is an agglomerate region. We removed the outliers with less than 30 voxels in a connected region to avoid the noise of predicted results for the accuracy. Testing time augmentation is an effective way to improve accuracy in the inference step. We only applied rotation with 90°, 180°, and 270° to repeat the inference in one case to save time. It might improve the DSC of the testing cohort with 1 to 2%. We also tried a multi-model ensemble and trained the same Res-UNet network with different epochs. The ensemble also improved the accuracy by around 1%.

Measurement of pixel level

DSC and intersection over union (IOU) are commonly used metrics in segmentation algorithms. We use these two coefficients to evaluate our segmentation performance. These coefficients are spatial overlap indexes utilizing segmentation in MRI as reproducibility validation metrics. The definition of DSC and IOU are shown as Equation (5) and Equation (6). , show the DSC and IOU performance of our segmentation.

Figure 6

Dice similarity coefficient and intersection over union performance.

Figure 7

Dice similarity coefficient and intersection over union performance.

shows the ground truth and our predicted results. The comparison indicated that case 01 to case 05 get the accurate results with DSC of around 0.9 and IOU of around 0.85. In case 06, a small lesion region was not segmented by the model; thus, the DSC and IOU are 0.0. We thought that the lesion was too small and quite similar to fat. This will be solved with the more various training cohorts.

Figure 8

Six cases showing comparisons between the ground truth and our predicted results.

Measurement of size and volume

We used quantification indexes to calculate shape-based features such as “maximum 3D diameter, 3D mesh volume, minimal diameter, maximal diameter, volume”. The method firstly extracted the largest tumor area, found the maximum connected components, cropped a 3D region with its solid components fitting as an ellipsoid, and then calculated the factors which might influence shape information. These were all based on the pyradiomics library ( ). We also used the Otsu’s method, which is a classical intensity-based method, to divide the cystic degeneration or necrosis region manifested as hypointense in the central or paracentral area on DCE-MRI ( ). It iteratively searched for the threshold that minimizes the within-class variance from the histogram. shows the histogram of the tumor areas calculated by Otsu’s method. Moreover, the maximum value is the threshold to find the interclass variance. After that, we used the threshold as pixel intensity value to segment cystic or necrotic change areas. So far, the central part of the cystic or necrotic areas could be extracted. The outliers, a number of voxels less than 30, were removed in each connected component area.

Figure 9

Workflow of measurement.

Figure 10

Regions of interest of the areas with cystic or necrotic changes. The green part shows the classification from the Otsu’s method.

Figure 11

Histogram of the lesion areas.

Statistical analysis

The automatic segmentation performance was evaluated with DSC. The method performance of classifying the size according to T stage was assessed with accuracy, sensitivity, and specificity. Intraclass correlation coefficient (ICC) was adopted to measure the agreement between the size and the volumetric parameters of the predicted results and the GT results. All statistical analyses were conducted using Python version 3.8 (www.python.org) and SPSS 25.0 software package.

Results

Our improved Res-UNet got the best DSC of 0.89 among different networks. The DSCs of different networks are shown in . The details of DSC and IOU are presented in . The final metrics of the predicted outcomes in the standard-alone test cohort were accuracy = 0.93, sensitivity (T1, T2, and T3 = 0.94, 0.94, and 0.75, respectively), and specificity (T1, T2, and T3 = 0.95, 0.92, and 0.99, respectively. The detailed metrics are shown in – , while – show the details of maximal diameter, minimal diameter, and volume. shows the metrics of cystic or necrotic components including volume and mean intensity. A high concordance of size and volumetric parameters was shown between the deep learning segmentation-based prediction results and the GT segmentation results. For the minimal and maximal diameters, the ICC was greater than 0.95, and for volumetric measurement of mass enhancement breast cancer, the ICC was greater than 0.80 ( ). The trained model took approximately 10–15 s to provide automatic segmentation and volume analysis for each patient, while the average manual segmentation time was at least 15 min.

Table 1

Different networks’ dice similarity coefficient (DSC).

Metrics	UNet	nnUNet	Res-UNet
DSC	0.82	0.887	0.894
GPU memory usage in training	6 GB (batch = 8)	8 GB for normal model32 GB for very big model	11 GB (batch = 8)

GPU, graphics processing unit.

Table 2

Details of dice similarity coefficient (DSC) and intersection over union (IOU).

Metrics	DSC	IOU
Average	0.88	0.80
Standard deviation	0.13	0.15

Table 3

Summary of geometric parameters between the prediction results and GT results.

Geometric parameters	Predict	GT
Maximum 3D diameter(mm)	33.25	34.02
3D mesh volume (mm3)	9,335.38	10,370.29
Minimal diameter (mm)	21.17	21.57
Maximal diameter (mm)	27.41	27.77
Volume (mm3)	9333.08	10416.14

GT, ground truth.

Table 5

Final predicted results of the classification.

Metrics	Small	Medium	Large
Macro average	0.91	0.88	0.89
Weighted average	0.93	0.93	0.93
Accuracy	0.93

Figure 12

Final metrics of the predicted results.

Figure 14

Final metrics of the predicted results.

Table 6

Comparison of the volume and mean intensity between cystic or necrotic components and lesions.

Quantitative parameters	Mean of lesion	Mean of cystic component	Minimum of lesion	Minimum of cystic component	Maximum of lesion	Maximum of cystic component
Volume (mm3)	23,858.41	7,816.06	2,625.14	12.92	253,526.11	128,501.66
Mean intensity	362.29	198.70	204.31	105.69	582.45	321.625

Table 7

Agreement of size and volumetric parameters between deep learning segmentation-based prediction results and GT segmentation results.

	Prediction	GT	Intraclass correlation coefficient
Volume (mm3)	9,333.08 ± 13,409.19	10,416.14 ± 21,928.01	0.840
Maximal diameter (mm)	27.41 ± 13.47	27.77 ± 12.55	0.952
Minimal diameter (mm)	21.17 ± 8.63	21.57 ± 9.06	0.964

GT, ground truth.

Discussion

Our study established a deep learning model based on the Res-UNet network architecture with DSC of 0.89 for the automatic segmentation to improve recognition efficiency and productivity with the speed of 10–15 s for one patient, eliminate inter- and intra-observer variations among breast radiologists as much as possible, and reduce information overload. Our model achieved a good performance with an overall accuracy of 0.93, sensitivity of 0.94, 0.94, and 0.75, and specificity of 0.95, 0.92, and 0.99, respectively, for three T categories in classifying the size of mass enhancement breast cancer. In addition, the model corresponded well with the GT results derived manually by radiologists in terms of size and volumetric parameters. These results implied that our framework might automate certain procedures of the preoperative evaluation for breast cancer. Although the classification capability of our model is powerful and significant, future advances which will be considered through external validation in other institutions or with larger data sets will make it more persuasive for clinical application.

Preoperative breast MRI, for its highest resolution and abundant information, becomes the most promising imaging modality for different AI applications, mainly for lesion detection and classification (12, 29). Automatically detecting and classifying (limited to benign versus malignant) breast lesions on MRI are relatively well-established techniques (30–33). Nevertheless, measuring the volume and the size of mass enhancement breast cancer accurately has important guidance for follow-up therapeutic decisions. In previous studies, some researchers have compared the accuracy of computer-aided detection (CAD) systems and radiologists in measuring the tumor size. The results are mostly reported such that the manual measurement of MRI is better than MRI with CAD (3, 16). However, CAD systems have limited capabilities; they also enable radiologists to process large images efficiently. Therefore, using large sample image data and more intelligent deep learning models based on neural network structures to measure the maximum diameter and solid component volume of tumors can undoubtedly improve the efficiency.

We reported an excellent performance of the model in segmentation, which is in accordance with the previous studies on breast cancer segmentation (34–37). This observation can not only provide precise segmentation and quantitative assessments of breast cancer but also assist in image analysis including detection, feature extraction, classification, and treatment. In most previous studies, tumors were segmented manually, which are prone to inter- and intra-observer variabilities (34, 38, 39). Furthermore, for the 3D medical imaging process, it is difficult and time-consuming for radiologists to measure lesions manually. Automatic segmentation and semi-automatic segmentation will reduce the time as well as improve the reliability. We used automatic segmentation which produced results consistently and reproducibly. What is more, automatically extracting an entire 3D lesion with an irregular shape only takes a few minutes, and the region in the 3D dimension by manual drawing may be discontinuous or not smooth and time-consuming.

Although several prior studies used deep learning to segment breast cancer, they did not measure the volume and the size. To the best of our knowledge, this is the first deep learning study to automatically segment mass enhancement breast cancer and measure the volume and the size on MRI. Our model also analyzed the areas with cystic or necrotic changes. Tumor necrosis has been proposed as a negative prognostic factor in some studies and could be evaluated on MRI comprehensively (40, 41). Differing from prior studies of necrosis as a predictive reference in TNBC, our study aims to automatically delineate and measure the volume of cystic and necrosis areas through our algorithm so that radiologists can intuitively find the changes in tumor components, and this would help them predict the patients’ prognosis (19, 42).

Our study had several limitations. Firstly, although this is a unicentric study with a relatively large sample size, external validation datasets from multiple centers should be set up to test the rationality of the model. Secondly, we did not simultaneously count multifocal or multicentric cancers. Non-mass enhancement breast cancer should also be tried to be divided into regions. Further research is possible in the future to expand the application scope of this model for improvement. Thirdly, from the perspective of methods, the performance of our model can still be improved. Some cases still contain false-positive regions similar to lesions with hyperintensity. We think that a false-positive-remove algorithm may suppress these error regions.

Conclusions

Utilizing a deep learning-based algorithm based on automatic segmentation to measure the volume and the size of mass enhancement breast cancer on MRI is feasible with high accuracy and reliability, thereby reducing the effort and variabilities. Further development will be added in our study for such to be implemented into future clinical practice efficiently.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

WY: conception, design of the study, acquisition of data, analysis and interpretation of data, and drafting the article. FS: critical revision for important intellectual content. ZS, ZT, and GL: establish the automatic segmentation with size and volumetric measurement of breast cancer. HZ, JZ, NT, SG, JD, YL, XB, and JC: acquisition of data. WY and FS: final approval of the version to be submitted. All authors contributed to the article and approved the submitted version.

Funding

This project is supported by the National Natural Science Foundation of China (no. 21575161).

Conflict of interest

Author GL, ZT and ZS were employed by Keya Medical Technology Co., Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Table 4

Final predicted metrics of the classification.

Classification	Precision	Recall	F1-score	support
Small (<20 mm)	0.85	0.94	0.90	50
Medium (20–50 mm)	0.96	0.94	0.95	138
Large (>50 mm)	0.90	0.75	0.82	12