Label-free coronavirus disease 2019 lesion segmentation based on synthetic healthy lung image subtraction.

PURPOSE: Coronavirus disease 2019 (COVID-19) has become a global pandemic and is still posing a severe health risk to the public. Accurate and efficient segmentation of pneumonia lesions in computed tomography (CT) scans is vital for treatment decision-making. We proposed a novel unsupervised approach using a cycle consistent generative adversarial network (cycle-GAN) which automates and accelerates the process of lesion delineation.
METHOD: The workflow includes lung volume segmentation, healthy lung image synthesis, infected and healthy image subtraction, and binary lesion mask generation. The lung volume was first delineated using a pre-trained U-net and worked as the input for the following network. A cycle-GAN was trained to generate synthetic healthy lung CT images from infected lung images. After that, the pneumonia lesions were extracted by subtracting the synthetic healthy lung CT images from the infected lung CT images. A median filter and k-means clustering were then applied to contour the lesions. The auto segmentation approach was validated on three different datasets.
RESULTS: The average Dice coefficient reached 0.666 ± 0.178 on the three datasets. Especially, the dice reached 0.748 ± 0.121 and 0.730 ± 0.095, respectively, on two public datasets Coronacases and Radiopedia. Meanwhile, the average precision and sensitivity for lesion segmentation on the three datasets were 0.679 ± 0.244 and 0.756 ± 0.162. The performance is comparable to existing supervised segmentation networks and outperforms unsupervised ones.
CONCLUSION: The proposed label-free segmentation method achieved high accuracy and efficiency in automatic COVID-19 lesion delineation. The segmentation result can serve as a baseline for further manual modification and a quality assurance tool for lesion diagnosis. Furthermore, due to its unsupervised nature, the result is not influenced by physicians' experience which otherwise is crucial for supervised methods.

INTRODUCTION

The coronavirus disease 2019(COVID‐19) has become a global public health problem and now still affects billions of people's life. According to the World Health Organization, the pandemic has caused over 2 million deaths by April 2021. The typical symptom of COVID‐19 includes cough, fever, and pneumonia after infection. Clinically, computed tomography (CT) scans are commonly used to evaluate the progress and severity of pneumonia , due to their high resolution in three dimensions and broad availability compared to other imaging modalities. Accurate delineation of pneumonia lesions is vital for evaluating disease progression and assessing the severity of infection which is crucial for treatment decision making. However, manual segmentation is time‐consuming and labor‐intensive. Therefore, automatic segmentation methods are highly demanded.

In the past decade, deep learning has shown its tremendous power and potential in various radiological applications, including image segmentation, , disease classification, and synthetic image generation. , , , Since the beginning of the COVID‐19 pandemic, deep learning, incorporating various image modalities like CT, X‐ray, and ultrasound, , , , , has been applied in clinical diagnosis, predicting disease progression, classifying pneumonia types and assessing the severity of infection. , However, existing methods are mostly based on supervised learning which requires substantial data labeling by radiologists as training references. For example, U‐net networks have been used for the classification and segmentation of COVID‐19 lesions in CT scans. , , The results vary greatly among different studies, partially due to the inter‐ and intra‐observer variations in the training lesion labeling by different radiologists.

Compared to supervised learning, unsupervised learning does not require training labeling, and hence gets rid of the burden of manual lesion delineation and the inter‐ and intra‐observer inconsistency. For example, Yao et al. proposed a label‐free pneumonia lesion segmentation method that employed an unsupervised statistical method to simulate infected lungs from healthy ones. Zhang et al. developed an unsupervised method to augment lung images for better follow‐up segmentation training using a conditional generative adversarial network (GAN). However, most of these methods focused on data augmentation with unsupervised networks, and had to rely on supervised networks to train the lesion segmentation process. ,

Cycle consistent GAN (cycle‐GAN) is an unsupervised network that has been widely used in medical image analysis, such as synthetic CT generation , , and image transformation between different MRI sequences. , Inspired by this, we propose a cycle‐GAN‐based unsupervised framework for COVID‐19 lesion segmentation. The cycle‐GAN is used to convert infected lung slices to healthy lung slices by transforming pneumonia lesions into normal lung tissues. Then, the lesion is retrieved by subtracting the simulated “healthy” lung from the original image. The network does not require any image pairing or manual training label, hence can improve efficiency and eliminate the inter‐ and intra‐observer inconsistency otherwise presented in supervised networks.

MATERIALS AND METHODS

Patient datasets

In this study, CT scans of 77 COVID‐19 patients with positive reverse transcription‐polymerase chain reaction results were collected between December 2019 and January 2020 in the First Affiliated Hospital of the University of Science and Technology of China (USTC‐1 dataset). The data was anonymized before any analysis. The patient and CT scan information is listed in Table 1. All CT images were converted to 1×1×1 mm3 spatial resolution and cropped to 256×256 pixels per slice. The image window level was set to [‐800, 100] HU and all images were normalized to [‐1, 1] with a zero background before being fed into network training. We selected 1264 healthy CT slices and 1272 slices with pneumonia lesions from the 77 CT scans as the training dataset.

TABLE 1

Dataset information

Database	Samples	Purpose	Description
USTC‐1	77	Training	CT Images without labeling
USTC‐2	14	Validation	CT Images with labeling
Coronacases	10	Test	CT Images with labeling
Radiopedia	8	Test	CT Images with labeling
UESTC	50	Test	CT Images with labeling

Lung volumes were extracted from CT images using a pre‐trained 2D U‐net lung segmentation network which was trained using 285 patients from a public dataset LUNA‐16. , The average dice similarity of this lung segmentation approach reached over 0.98 in the former study. The extracted lung images were then used as the input of cycle‐GAN. Another dataset including 14 patients from the First Affiliated Hospital of the University of Science and Technology of China (USTC‐2 dataset) was used as the validation set. Three more datasets, including two public databases (Coronacases and Radiopedia) and one obtained from a published study (UESTC), were used as the test set. The details of these datasets are listed in Table 1.

Cycle‐GAN structure

In the training stage, we first used a cycle‐GAN strategy to generate synthetic healthy lung CT slices. The network architecture is illustrated in Figure 1. We denote the infected lung CT slices by domain X and healthy lung CT slices by domain Y, the probability distribution for each domain is referred as and , respectively. The generator denotes the mapping process from domain X to domain Y, and denotes the mapping from domain Y to domain X. are synthetic “infected” and “healthy” lung slices. Two adversarial discriminators and are used to distinguish real input images and synthesized images. The architecture of the generator is a U‐Net variant and consists of eight stages, as shown in Figure 2. Unlike the original U‐net, instance normalization, which can better preserve image details in the image generation process, is applied immediately after each convolutional layer except for the last one. All convolution filters in the generator have a size of 3 × 3 pixels. We set the channel number of the first block to 64. In the encoder part, the width and height of the feature map are halved using convolution with a stride of 2 instead of max pooling. In the first four stages, the channel number is doubled after the feature map passes each layer, while in the last four stages the channel number is fixed to 512. All the feature maps in the encoder part are concatenated with their counterparts in the decoder part. The encoder and decoder parts are symmetric.

FIGURE 1

The scheme of the proposed method. are generators for generating synthetic “Healthy” images and “Infected” images. are three loss functions used for training. are two discriminators that distinguish real and synthetic images

FIGURE 2

The structure of the U‐Net generator. The size of the input and output image is 256 × 256. The encoder and decoder parts each have eight stages. Skip connection is applied to each stage

The discriminator and are implemented by a 70 × 70 Patch‐GAN. The architecture of the discriminator is illustrated in Figure 3. The stride is set to 2 in the first three convolution layers and 1 in the last two layer, and the padding is 1 in all convolution layers. Leaky ReLu activation is applied with a slope of 0.2 after each convolution layer, except the last one. In the first convolution layer, a feature map with 64 channels is generated. After that, the channel number is doubled after the feature map passes each layer. In the last layer, the output is reduced to one channel.

FIGURE 3

The structure of the discriminator. In the 1st layer, convolution and Leaky‐ReLu are applied. In the 2nd–4th layers, convolution, instance normalization, and Leaky‐Reu are implemented. Only one convolution layer is applied in the last layer

The generators and discriminators are trained by solving a min‐max problem: where is defined as: where is the total loss aiming to learn the mapping function between the source and target domain. and are introduced in (1) mainly to weigh the importance of the three losses. After optimization, we set and .

is the loss function of the discriminator calculating the difference between synthetic “healthy” slices and real healthy slices. To maintain stability during the learning process, we here choose L2 loss in the LSGAN as our loss function instead of the sigmoid cross‐entropy in regular GANs. The is defined as: where denotes the learning process on domain X. is used to keep the consistency of the two generators and , and is defined as: where is the ℓ1 norm. Since we only want to convert unhealthy lung CT slices into healthy ones, an identity loss in (1) is designed to keep the image features when a healthy slice is an input into the generator. Identity loss is defined as:

We used the ADAM optimization method to train all the networks, with and . Kernels were initialized randomly with a Gaussian distribution. We updated the generators and the discriminators at each iteration. The input image slice was randomly cropped to patches of 256 × 256‐pixel size. The number of mini‐batches was 1, and the number of epochs was 100. The learning rate was initially set to 0.0002 and linearly decreased to 0 in the last 50 epochs. The training was stopped at the 85th epoch which rendered stabilized optimal dice similarity coefficient (DSC) results. All the hyperparameters were validated on the validation set (USTC‐2). The training was conducted on a Linux machine with an NVIDIA RTX 2080Ti GPU and took 8 h.

Image post‐processing

The post‐processing steps are illustrated in Figure 4. The synthetic healthy image was subtracted from its corresponding real infected image to obtain a different image. The lung CT slices from each patient were reshaped into a 3D volumetric image. Median filtering was applied to suppress noise and remove small islands. Then k‐means clustering was used to segment the lesion from the low‐intensity background. A 5 × 5 Gaussian kernel was employed, slice by slice, to smoothen the lesion edge. Finally, erosion and dilation with a radius of 1 pixel were performed to further remove small and isolated regions. All post‐processing steps, except the Gaussian filtering, were implemented on the 3D volumetric image. The post‐processing procedures were done in MATLAB 2018A and took less than 2 min for each patient. As a comparison, we also compared the k‐means clustering method with Otsu thresholding which is commonly used for thresholding segmentation.

FIGURE 4

The framework of post‐processing. (a–e) shows the image after subtraction of the synthetic “healthy” image from infected image (a), median filtering (b), k‐means clustering with zero background (c), gaussian filtering (d), and hole filling, erosion, and dilation (e). (f) is the ground truth

To summarize the lesion segmentation process, a U‐net previously trained on a publicly available dataset LUNA‐16 was applied to segment the lung volume first. Then a cycle‐GAN network was trained and employed to generate the corresponding synthetic healthy lung images from the infected lung images in the USTC‐1 dataset. Finally, the lesion was obtained after subtraction of the synthetic healthy lung images from the corresponding infected lung images. The subtraction images containing lesions were combined into a 3D volumetric image based on which post‐processing steps were performed to fine‐tune the segmented lesion volume. The whole process is automated and does not need manual labeling of lung lesions as network training references.

Segmentation evaluation

In this study, the DSC, volume precision (vPSC), and volume sensitivity (vSEN) defined in Equations (5)–(7) were used to evaluate the performance of the proposed segmentation method. The in the equations represent the predicted and ground truth lesion volume.

When it comes to the diagnosis, doctors are more interested in whether there is a lesion and where it is rather than how accurate the delineation is. We proposed an evaluation approach to count whether there is any lesion in different subvolumes of the lung. To mimic this process, the lung volume was divided into 12 subvolumes as illustrated in Figure 5. Each subvolume was counted as positive if there was at least one lesion, and negative otherwise. Three additional metrics were used for comparison between the proposed segmentation method and other methods. The detection accuracy (ACC), precision (PSC), and sensitivity (SEN) are defined respectively as:

FIGURE 5

The 12 subvolumes in the lung volume division. The lung volume is divided into 12 regions equally

The paired t‐test was used to statistically compare the proposed method with other methods, that is, nnUNet‐2D, nnUNet‐3D, Cople‐Net, Inf‐Net , and NormNet. As for those comparison methods, the nnUNet‐2D/nnUNet‐3D network was trained and validated on any two of the three datasets (Coronacases, Radiopedia, and UESTC) and tested on the remaining one dataset using a 5‐fold cross‐validation method; for all other networks, the network parameters were directly borrowed from the previous publications to build networks for the tests run on the three datasets.

RESULTS

Table 2 shows the comparisons between the proposed method and several existing supervised and unsupervised methods. The results of our method reach a dice coefficient of 74.8 ± 12.1% and 73.0 ± 9.5% on the Coronacases and Radiopedia datasets, respectively. The volume precision and volume sensitivity are 81.3 ± 8.8%, 73.5 ± 20.5% on Coronacases and 77.3 ± 17.8%, 72.6 ± 11.1% on Radiopedia. As shown in Table 2, our approach is comparable with the supervised Cople‐Net method and outperforms the semi‐supervised Inf‐Net method on these two datasets. The performance of our method on the UESTC database reaches 64.4 ± 19.2%, which is still higher than Inf‐Net. We also compared our method with a state‐of‐the‐art unsupervised label‐free method (denoted as “NormNet” in Table 2). Our method results in higher scores in most indices and is more robust than the “NormNet” method when implemented on different datasets. The results also indicate that k‐means clustering performs better than Otsu thresholding in image post‐processing.

TABLE 2

The comparison between the proposed and existing methods on dice similarity coefficient (DSC), volume precision (vPSC), and volume sensitivity (vSEN)

		Coronacases (10)			Radiopedia (8)
	Method	DSC (%)	vPSC (%)	vSEN (%)	DSC (%)	vPSC (%)	vSEN (%)
SUPERVISED METHOD	nnUNet‐2D	72.2 ± 14.5	86.4 ± 16.1	64.2 ± 16.4	79.2 ± 0.5	75.5 ± 11.2	85.0 ± 6.4
	nnUNet‐3D	77.0 ± 10.9	74.2 ± 14.7	82.2 ± 11.5	73.0 ± 9.6	66.5 ± 17.9	86.2 ± 9.2
	Cople‐Net	76.7 ± 8.2	79.5 ± 14.9	75.9 ± 7.6	77.8 ± 6.7	83.1 ± 14.9	75.6 ± 10.1
	Inf‐Net	59.5 ± 16.8	59.4 ± 19.2	65.2 ± 22.2	64.8 ± 12.0	72.6 ± 17.2	63.0 ± 18.2
UNSUPERVISED METHOD	NormNet	68.7 ± 15.8	85.1 ± 7.0	62.1 ± 22.8	59.4 ± 17.4	60.4 ± 19.7	61.8 ± 18.4
	Proposed (Otsu)	70.9 ± 14.3	68.2 ± 21.1	81.1 ± 9.1	65.3 ± 12.6	65.5 ± 15.6	70.6 ± 16.6
	Proposed (k‐means)	74.8 ± 12.1	81.3 ± 8.8	73.5 ± 20.5	73.0 ± 9.5	77.3 ± 17.8	72.6 ± 11.1
		UESTC (50)			Average(68)
		DSC (%)	vPSC (%)	vSEN (%)	DSC (%)	vPSC (%)	vSEN (%)
SUPERVISED METHOD	nnUNet‐2D	74.8 ± 11.4	80.7 ± 15.6	71.5 ± 11.9	74.9 ± 11.4	80.9 ± 15.3	72.1 ± 13.2
	nnUNet‐3D	69.5 ± 16.8	86.7 ± 13.1	61.1 ± 18.5	71.0 ± 15.5	82.5 ± 15.6	67.1 ± 19.5
	Cople‐Net	84.0 ± 7.1	90.8 ± 5.6	79.1 ± 10.6	82.2 ± 7.8	87.4 ± 9.9	79.1 ± 10.8
	Inf‐Net	41.8 ± 18.4	92.6 ± 3.6	36.6 ± 20.6	47.1 ± 19.6	43.1 ± 22.6	60.1 ± 18.2
UNSUPERISED METHOD	NormNet	61.4 ± 19.4	61.3 ± 26.1	77.6 ± 19.6	62.4 ± 18.6	64.7 ± 23.5	73.5 ± 20.0
	Proposed (Otsu)	62.4 ± 16.2	58.4 ± 19.6	75.0 ± 23.1	64.0 ± 14.8	59.0 ± 17.8	77.3 ± 21.6
	Proposed(k‐means)	64.4 ± 19.2	74.2 ± 19.9	63.2 ± 24.7	66.6 ± 17.8	67.9 ± 24.4	75.6 ± 16.2

Paired t‐test results are shown in Table 3. It is observed that the proposed method is comparable with the supervised methods in most cases. For example, our method is comparable to nnUNet‐2D, nnUNet‐3D, and Cople‐Net on the Coronacases dataset, and comparable to nnUNet‐3D, Cople‐Net, and Inf‐Net on the Radiopedia dataset. In UESTC, however, our method renders an inferior outcome compared to the supervised methods, but still outperforms the unsupervised method NormNet.

TABLE 3

The paired t‐test p‐values for dice similarity coefficient (DSC) comparison between the proposed and compared methods

	Coronacases	Radiopedia	UESTC	Average
nnUNet‐2D	0.614	≤ 0.010	≤ 0.001	≤ 0.001
nnUNet‐3D	0.302	0.618	0.017	0.008
Cople‐Net	0.280	0.060	≤ 0.001	≤ 0.001
Inf‐Net	0.005	0.110	≤ 0.001	≤ 0.001
NormNet	0.271	0.013	0.376	0.151

The proposed method performs well even in small lesion segmentation. Figure 6a shows that a small lesion with only 2‐mm width is correctly delineated. As shown in Figure 6b, our method can also readily separate lesions from the chest wall. Interestingly, our method catches some low contrast lesions which are skipped by radiologists during manual segmentation, as shown in Figure 6c.

FIGURE 6

Segmentation results for lesions of various sizes. (a–c) are segmentation images from different patients. The upper right panel is the zoom‐in of the boxed area in each image. The red outlines are from the proposed method, and the green is the ground truth. The red arrow in (c) points to a sagittal plane across the lesion

Figure 7 demonstrates the performance of our method compared to existing supervised methods. These lesions are varied in shape, size, and position. As shown in Figure 7a, our method localized and delineated, while other methods missed, the lesion close to the trachea. Our method performs better than Inf‐Net and Cople‐Net on a large lesion segmentation as shown in Figure 7b, while is comparable to all existing supervised methods in segmenting small lesions isolated in the lung volume and nearby the chest wall as shown in Figure 7c,d.

FIGURE 7

Results comparison between different segmentation methods. (a–d) show segmentation results from four different patients. The green, red, and blue areas are true positive, false negative, and false positive, respectively. The orange arrow in (a) points to a lesion that can only be detected by the proposed method

As shown in Table 4, the average accuracy of our method on three datasets reaches 79.1%, which is comparable to the supervised methods. Our method outperforms the semi‐supervised Inf‐Net in detection accuracy and precision, except sensitivity. The results for the paired t‐test on detection accuracy are shown in Table 5. As shown both in Tables 4 and 5, there is no significant difference between our method and nnUNet‐2D, and a similar capability for lesion diagnosis can be observed between our method and all supervised methods when performed on database Coronacases and Radiopedia.

TABLE 4

The accuracy (ACC), precision (PSC), and sensitivity (SEN) on three different datasets

	Coronacases (10)			Radiopedia (8)
Method	ACC (%)	PSC (%)	SEN (%)	ACC (%)	PSC (%)	SEN (%)
nnUNet ‐2D	94.2 ± 8.0	93.4 ± 11.9	97.1 ± 6.7	88.3 ± 15.3	81.9 ± 26.0	99.0 ± 3.2
nnUNet ‐3D	83.3 ± 20.0	76.7 ± 26.2	98.0 ± 6.3	87.5 ± 20.5	81.7 ± 29.1	100.0 ± 0.0
Cople‐Net	86.7 ± 15.8	79.5 ± 27.8	98.6 ± 4.5	91.7 ± 12.4	85.0 ± 25.2	95.8 ± 10.1
Inf‐Net	70.0 ± 32.7	69.7 ± 32.7	100.0 ± 0.0	75.8 ± 32.0	74.6 ± 33.9	100.0 ± 0.0
Proposed (k‐means)	90.0 ± 10.2	87.5 ± 20.4	96.4 ± 6.3	86.7 ± 12.6	95.0 ± 15.8	81.6 ± 18.8
	UESTC (50)			Average (68)
	ACC (%)	PSC (%)	SEN (%)	ACC (%)	PSC (%)	SEN (%)
nnUNet ‐2D	76.5 ± 17.7	68.5 ± 24.1	99.3 ± 3.0	80.7 ± 14.5	74.0 ± 24.4	99.3 ± 3.0
nnUNet ‐3D	85.3 ± 12.7	78.8 ± 20.2	98.2 ± 5.2	85.4 ± 14.8	78.9 ± 22.1	98.2 ± 5.2
Cople‐Net	90.8 ± 9.9	85.8 ± 17.8	97.5 ± 7.7	90.4 ± 11.1	84.8 ± 20.2	97.5 ± 7.7
Inf‐Net	54.7 ± 24.5	54.6 ± 24.8	99.7 ± 1.8	59.9 ± 27.6	59.6 ± 28.0	99.7 ± 1.8
Proposed (k‐means)	74.8 ± 22.7	73.8 ± 28.6	92.4 ± 12.9	79.1 ± 20.8	77.7 ± 26.8	92.4 ± 12.9

TABLE 5

The paired t‐test p‐values for detection accuracy comparison between the proposed and compared methods

Accuracy	Coronacases	Radiopedia	UESTC	Total
nnUNet‐2D	0.905	0.096	0.641	0.542
nnUNet‐3D	0.953	0.247	0.004	0.036
Cople‐Net	0.717	0.534	≤ 0.001	≤ 0.001
Inf‐Net	0.297	0.083	≤ 0.001	≤ 0.001

DISCUSSION

In this study, we propose a label‐free method for delineating COVID‐19 lesions using cycle‐GAN. This unsupervised method shows great potential in lesion segmentation without employing labeled data and is validated on different databases. The proposed method is robust and less database‐dependent and effectively eliminates the labeling inconsistency otherwise existing in supervised learning approaches. It can work as an efficient and independent automatic segmentation method or can provide a start point for physicians’ follow‐up refinement.

There are still some limitations to the proposed unsupervised method. First, as shown in Figure 8a, the method may miss some small and low contrast lesions. Second, the lung CT slices delineated by a U‐net were used as the input, which may bear some false positive lesions misclassified from the top part of the liver. On three test sets, the top liver from three patients of a total of 68 (4.34%) was misclassified as a lesion. One example is shown in Figure 8b. Moreover, the sample number in this study might not be sufficiently large, particularly lacking patients with lesions in the top and bottom of the lung volume, potentially resulting in mis‐delineation as shown in Figure 8c.

FIGURE 8

Segmentation limitations for the proposed method. The green contours are manual labels representing the ground truth and the red ones are automatic segmentation results by the proposed method

On the other hand, the network was trained with 2D images due to hardware limitations. This strategy didn't make full use of the three‐dimensional property of CT images. Future studies using 3D image input may improve segmentation accuracy, particularly in terms of the contour continuity along the image thickness direction. Despite this, the proposed unsupervised method still achieved decent segmentation results, with a dice value of 74.8% and an accuracy of 90.0% on the Coronacases public database. In the future, the unsupervised method can be combined with more sophisticated post‐processing methods, such as texture analysis or even additional deep learning networks, to further improve segmentation results.

CONCLUSIONS

In this work, we propose a label‐free approach that can accurately and efficiently delineate the COVID‐19 lesions automatically in CT scans. The training process of the unsupervised network does not rely on any labeled data. The automatic segmentation can provide a start point for further manual refinement and can work as a quality assurance tool for lesion diagnosis. Due to its unsupervised nature, the result is not influenced by physicians’ experiences which otherwise is crucial for supervised methods.

CONFLICTS OF INTEREST

The authors declare no conflict of interest.