Automatic segmentation of COVID-19 from computed tomography images using modified U-Net model-based majority voting approach.

The coronavirus disease (COVID-19) is an important public health problem that has spread rapidly around the world and has caused the death of millions of people. Therefore, studies to determine the factors affecting the disease, to perform preventive actions and to find an effective treatment are at the forefront. In this study, a deep learning and segmentation-based approach is proposed for the detection of COVID-19 disease from computed tomography images. The proposed model was created by modifying the encoder part of the U-Net segmentation model. In the encoder part, VGG16, ResNet101, DenseNet121, InceptionV3 and EfficientNetB5 deep learning models were used, respectively. Then, the results obtained with each modified U-Net model were combined with the majority vote principle and a final result was reached. As a result of the experimental tests, the proposed model obtained 85.03% Dice score, 89.13% sensitivity and 99.38% specificity on the COVID-19 segmentation test dataset. The results obtained in the study show that the proposed model will especially benefit clinicians in terms of time and cost.

Introduction

COVID-19 is a contagious disease that emerged in Wuhan city of China at the end of 2019 and caused a worldwide epidemic. The World Health Organization (WHO) declared COVID-19 a "global epidemic" on March 11, and the disease, which continues to spread rapidly, has reached approximately 260 million total cases and 5.2 million total deaths worldwide as of November 2021 [1].

One of the most important steps in the fight against COVID-19 is to detect infected patients as early as possible and start their treatment. Currently, various tests have been used to detect the disease [2, 3]. However, since these are time-consuming and costly techniques, some researchers have indicated that chest X-ray images can be helpful for detecting COVID-19 as an alternative to this method [4-6]. Today, detection of the disease from radiological images comes to the forefront as one of the fastest ways for diagnosing patients. Therefore, several experimental studies are being conducted to detect COVID-19 from chest X-ray images.

Artificial intelligence is one of the main research areas in computer science and studies carried out in recent years show that artificial intelligence is a technology that can be used in many fields such as agriculture, industry, banking, informatics and health [7-9]. Machine learning (ML) and deep learning (DL) are two sub-branches of artificial intelligence. ML is a type of learning in which computers can learn and improve automatically from experience to perform specific tasks without being explicitly programmed. DL, on the other hand, is a kind of learning which can find a solution to complicated schemes via representational learning [10].

In the literature, there are approaches for feature extraction and classification of COVID-19 disease by utilizing various ML and DL methods on radiological images. Barstugan et al. used the support-vector machine method for the classification of COVID-19 disease and used a database containing a total of 150 computed tomography (CT) images, 53 of which were infected with COVID-19. They preferred to use two, five and tenfold cross-validation techniques during the classification stage. As a result of the experimental tests, they reported that the best accuracy value of 98.71% was achieved when the tenfold cross-validation technique was utilized [11]. Mahdy et al. used 40 chest X-ray images, 25 of which were infected with COVID-19, to diagnose cases of COVID-19. Their results indicate that the support-vector machine method achieved 97.48% accuracy [12]. Shi et al. developed a new model based on the infection Size Aware Random Forest (iSARF) method for the classification of COVID-19 disease and viral pneumonia disease. In their study, where they used a dataset containing CT images of 1658 COVID-19 and 1027 viral pneumonia diseases, they reported that the highest performance of the proposed method was 0.879 when fivefold cross-validation was utilized [13]. Tang et al. utilized the Random Forest machine learning method for automatically classifying the seriousness of the disease from CT images of COVID-19 patients. They reported that the proposed model reached an accuracy value of 0.875 when threefold cross-validation technique was utilized [14]. In another study, Öztürk et al. developed a DL network called DarkNet for early detection of COVID-19 disease. Their network includes 17 convolution layers and is used as a classifier for a real-time object detection system (YOLO). The proposed model obtained an accuracy rate of 87.02% in three class classification and 98.08% in binary classification [15]. Pathak et al. proposed a ResNet-50-based DL model for diagnosing COVID-19 disease. They reported that the accuracy rate of the proposed model in binary classification was 93.02% in their study, where they used 413 COVID-19 and 439 normal or pneumonia images obtained from various sources [16]. Apostolopoulos and Mpesiana proposed a new model using CNN-based transfer learning approaches to classify normal, pneumonia and COVID-19 X-Ray images. The proposed model obtained an accuracy rate of 96.78% in binary classification [17]. In another study, Khan et al. classified the COVID-19 disease using the Xception deep learning architecture. When they tested the model, which called CoroNet, on a dataset containing Covid-19, Pneumonia-bacterial, Pneumonia-viral and Normal images, they reported that they achieved an accuracy rate of 89.5% in the four-class classification and 94.59% in the three-class classification. When they tested on the other dataset containing Normal, Pneumonia and Covid-19 images, they reported that the proposed model reached 90% accuracy in triple classification [18]. Heidari et al. used a transfer learning-based convolutional neural network model for classifying the normal, pneumonia, and COVID-19 infected pneumonia from chest X-ray images. While the model proposed in the study achieved 94.5% accuracy performance in triple classification, it reached 98.1% accuracy performance in two class classification [19]. In another study, Chowdhury et al. used a total of 2905 radiological images including COVID-19, normal and viral pneumonia states and applied a new method they called Parallel-Dilated COVIDNet (PDCOVIDNet). They reported that this method, which uses an extended convolution on the stack of parallel convolution blocks, can capture important features in parallel over the network and increase the classification accuracy. According to their experimental results, they reached an accuracy value of 96.58% [20]. Uçar et al. developed a model in which deep features were extracted in RGB, CIE Lab and RGB CIE color spaces and classified in two stages with various classifiers through pre-trained DL architectures for detecting COVID-19 from chest X-ray images. They stated that the Bi-LSTM network outperformed other classifiers with 92.489% in the experiments [21].

Besides the classification studies, there are also studies carried out for the segmentation of COVID-19 findings through radiological images in the literature. Diniz et al. developed a new method called Residual U-Net, making several changes to the traditional U-Net method, to automatically segment infections caused by COVID-19 disease. The test results indicated that the proposed method provides 77.1% average Dice value performance [22]. In another study, Budak et al. developed a SegNet-based model using the attention gate mechanism for segmentation of COVID-19 infection findings from CT images. They reported that the proposed model achieved a Dice score of 89.61% when fivefold cross-validation was used [23]. Yan et al. developed COVID-SegNet, which is based on deep convolutional neural network, for segmentation of COVID-19 disease. In this study, in which 21,658 CT images of 861 confirmed patients with COVID-19 were used, it was indicated that the proposed model obtained 72.6% Dice score performance in infection segmentation [24]. Fan et al. developed deep learning models, called Inf-Net and its semi-supervised version Semi-Inf-Net, for the automatic detection of COVID-19 infection. They utilized a parallel partial decoder to collect high-level features, while reverse attention and explicit edge-attention were used to set boundaries. When the authors compared the infection segmentation success of the proposed model with well-known segmentation models, the Semi-Inf-Net model produced more successful results with a Dice score of 73.9% and the Inf-Net model with a Dice score of 68.2% [25]. Wu et al. proposed a DL-based model for the segmentation of COVID-19 infection. The proposed model achieved a 78.5% Dice score using 144,167 CT images of 350 normal cases and 400 COVID-19 patients [26]. Qiu et al. proposed a lightweight DL model called MiniSeg, for the segmentation of COVID-19 infection. In the study, they also compared the performance of this model, which stands out with only 83 K parameters and high computational efficiency, in infection segmentation with traditional methods [27]. In another study Zhou et al. built a U-Net-based network using the attention mechanism for segmentation of COVID-19 disease from CT images. As a result of the experimental tests, they reported that the time required for the proposed model to segment a single CT slice was only 0.29 s and achieved a Dice score of 83.1% [28].

The approaches used to diagnose COVID-19 disease have been summarized in Table 1. As can be seen from Table 1, many artificial intelligence-based studies have been carried out to detect COVID-19 disease through radiological images. However, the lack of deep learning-based approaches to effectively solve the problem of rapid and accurate segmentation of COVID-19 disease infections has been the motivation of this study. With the aim of filling current gap, in this study a fast and efficient model without sacrificing performance is proposed by modifying the U-Net architecture, which can segment images precisely using limited amounts of training data. The major contributions of this study are as follows.

Table 1

Summary of the approaches used to detect COVID-19 disease

References	Year	Method	Dataset	Model performance (%)
Barstugan et al. [11]	2020	Support-vector machine	150 CT images	Accuracy: 98.71
Mahdy et al. [12]	2020	Support-vector machine	40 chest X-ray images	Accuracy: 97.48
Shi et al. [13]	2021	Size aware random forest (iSARF)	1658 COVID-19 1027 viral pneumonia disease CT images	Accuracy: 87.90
Tang et al. [14]	2020	Random forest	Chest CT images of 176 patients	Accuracy: 87.50
Öztürk et al. [15]	2020	DL network (DarkNet)	1125 X-ray images	Accuracy: 98.08
Pathak et al. [16]	2020	DL model (ResNet-50)	413 COVID-19 439 normal or pneumonia images	Accuracy: 93.02
Apostolopoulos and Mpesiana [17]	2020	CNN-based transfer learning approaches	1427 X-ray images	Accuracy: 96.78
Khan et al. [18]	2020	DL model (Xception)	1300 X-ray images	Accuracy: 94.59
Heidari et al. [19]	2020	CNN-based transfer learning approaches	8474 X-ray images	Accuracy: 98.10
Chowdhury et al. [20]	2020	Parallel-dilated COVIDNet	2905 radiological images (COVID-19, normal, viral pneumonia)	Accuracy: 96.58
Uçar et al. [21]	2021	Bi-LSTM network	1125 X-ray images	Accuracy: 92.48
Diniz et al. [22]	2021	Residual U-Net	112,873 CT images of 2 datasets	Dice: 77.10
Budak et al. [23]	2021	SegNet-based model	473 CT images	Dice: 89.61
Yan et al. [24]	2020	COVID-SegNet	21,658 CT images of 861 COVID-19 patients	Dice: 72.60
Fan et al. [25]	2020	Semi-Inf-Net	100 CT images	Dice: 73.90
Wu et al. [26]	2021	DL model	144,167 CT images of 350 normal and 400 COVID-19 patients	Dice: 78.50
Qiu et al. [27]	2020	Lightweight DL model (MiniSeg)	3558 CT images of 4 datasets	Dice: 80.06
Zhou et al. [28]	2021	U-Net with attention mechanism	473 CT images	Dice: 83.10

This study provides a significant improvement with rapid and a low-cost diagnosis model by using pre-trained DL models in the encoder part of the U-Net model.

The majority voting method was used to improve the performance of U-Net models in the segmentation of COVID-19 disease infections.

The results obtained from each model were combined with the majority voting principle and the effects on accuracy was evaluated, in the study.

The remainder of the work is organized as follows. In the second section, the dataset and method used in the study are mentioned. In the third section, experimental studies are given. In the fourth section, the results are analyzed and a discussion section is included in which these results are compared with other studies in the literature. Finally, in the fifth section conclusion and future works are presented.

Materials and method

Dataset

Two COVID-19 segmentation datasets taken from the Italian Society of Medical and Interventional Radiology were used in this study [29]. In the first dataset, there were 110 axial CT images of 60 patients infected with COVID-19. The grayscale 512 × 512 images in the dataset were compiled into a single NIFTI file. In addition, the lesions in these images were labeled by radiologists as ground-glass, consolidation and pleural effusion. In the second dataset, 493 of the images were confirmed and segmented by radiologists that there were cases of COVID-19. In this data set, there were 939 slices and 10 CT volumes in 630 × 630 dimensions. Due to the imbalance distribution of the lesions in the datasets, all lesion tags were used with a single tag as COVID-19 lesion in the study. And also, the images in the data sets were converted to jpeg format and resized as 256 × 256. In Fig. 1, sample images of the data sets used in the study are presented.

Fig. 1

Sample images of the COVID-19 CT dataset

Proposed model architecture

In this study, an approach based on the U-Net DL method, which is widely used for segmentation tasks in medical images, is proposed for rapid and efficient detection of COVID-19 disease from CT images. In the proposed approach, the U-Net model was modified and the pre-trained DL models such as VGG16, ResNet101, DenseNet121, InceptionV3 and EfficientNetB5 were used in the encoder (contracting path) part. Thus, in the transfer learning approach, the training process was accelerated and time was saved compared to training a model from scratch by using the knowledge of a network that was previously trained with a large amount of visual data. While selecting these models, their successful performance in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and their computational costs were taken into account. VGG16 architecture is the second in ILSVRC'14, as well as providing low computational cost thanks to its homogeneous network and filters with small-receptive field. ResNet101 ranked first in ILSVRC'15 with low error rates and created residual blocks to solve the degradation problem that may occur in high-depth networks. In addition, even if some layers in the network cannot learn, they proposed the skip connection approach, which provides the flow of information from one layer to the next layer. In DenseNet121 architecture, because each layer is connected to the next layer there is no loss in features as it progresses between layers and thus each layer can access the features of the previous layers. In the InceptionV3 architecture, asymmetric filters and bottleneck layer are used to reduce the computational cost in deep networks. The EfficientNetB5 architecture, on the other hand, provides more efficient results by reducing the size of the model by performing compound scaling.

U-Net models created by using pre-trained models were shown in Fig. 2 and detailed information about the layers used in the U-Net encoder part of these models was presented in Table 2. In addition, since it was desired to build a low-cost and high-performance diagnostic model, a novel method was proposed by combining the results obtained from each model with the majority voting principle. In the proposed method, the results obtained by the modified U-Net models are voted for each pixel and the class with the most votes is assigned. For example in this task, if a pixel is classified as having a COVID lesion by three models and as having no lesion by two models, that pixel is assigned as having a COVID lesion. The block diagram of the proposed majority voting segmentation model is presented in Fig. 3.

Fig. 2

U-Net models created by using pre-trained models

Table 2

Layers of the pre-trained models utilized in the U-Net encoder part

VGG16	ResNet101	DenseNet121 (layer id)	InceptionV3 (layer id)	EfficientNetB5
block5_conv3	stage4_unit1_relu1	311	228	block6a_expand_activation
block4_conv3	stage3_unit1_relu1	139	86	block4a_expand_activation
block3_conv3	stage2_unit1_relu1	51	16	block3a_expand_activation
block2_conv2	relu0	4	9	block2a_expand_activation

Fig. 3

Block diagram of the image segmentation model

U-Net

U-Net, one of the deep learning methods used for image analysis, is a very popular technology that has achieved remarkable success in medical image segmentation [30]. The U-Net model, proposed by Ronneberger et al. to segment biological microscopy images efficiently, is commonly utilized in segmentation tasks of medical images because of its simple and flexible structure and providing high-quality pixel-level segmentation results [31-36]. The U-Net architecture is a symmetrical model and it got this name because when the layers are combined, it evokes the letter U. U-Net architecture has a contracting path (encoder) and an expansion path (decoder). Here, the contracting path is a conventional convolutional network, consisting of a 3 × 3 convolution operation repeated twice. This convolution is then followed by ReLU activation function and a maximum pooling layer with 2 × 2. The number of feature channels doubles with each pooling operation. The purpose of the down-sampling path is to capture the content of the input image so that it can be segmented. This information is then transferred to the expansion path via connections. The number of feature channels is halved after each upsampling step in the expansion path in the architecture. The expansion path consists of four blocks. These blocks include the steps of deconvolution layer, merging with feature map from contracting path, 3 × 3 convolution layer + activation function. Finally, an additional 1 × 1 convolution operation is applied to reduce the feature map to the required number of channels and generate the segmented image. The U-Net architecture is as shown in Fig. 4.

Fig. 4

U-Net architecture

Loss function

In the CT images used in the study, the data are imbalanced since the background covers more pixels than the COVID-19 infected regions. To tackle this problem, in this study Focal Loss and Dice loss functions, which are frequently used in unbalanced data sets, are used together. The loss function used in the study is calculated as in Eq. 1.

Here, represents focal loss function and represents Dice Loss function. Focal Loss is a function that reduces the effect of easy-to-learn examples and focuses on difficult-to-train examples by adding and parameters to the cross entropy loss function [37]. It is calculated by the formula given in Eq. 2.

Here, is used to prevent class imbalance, while is the focusing parameter that adjusts focusing on difficult samples and is usually used as 2.

Dice Loss is a function that works quite well in binary segmentation tasks and helps solve imbalanced training data problems. Dice loss function is calculated with the formula given in Eq. 3 [38].

Here, and indicate pairs of correspondent pixel values of estimation and ground truth, respectively.

Experiments

Training and testing environment

In this study, Google cloud environment was used for all experiments. The computer used for the experiments includes Intel (R) Xeon (R) 2.00 GHz CPU, 12 GB RAM and NVIDIA T80 GPU running on 64-bit Ubuntu operating system. The proposed method was carried out in the Python programming language with segmentation models library which is based on Keras API and TensorFlow platform [39]. In the proposed model, a transfer learning approach was utilized for the initialization of weights [40, 41]. All images used in the study were initially resized at 256 × 256 and normalized using the min–max method. As represented in Table 3, 80% of the dataset is reserved for training and the remaining 20% is reserved for testing and then 10% of the data allocated for the training dataset is reserved for validation.

Table 3

Dataset distribution for training, testing and validation set

	Training	Testing	Validation
Number of images	426	119	48

After several training processes, the hyperparameters of the models were determined by trial and error. In the training phase, the learning rate was set as 0.0001 and the batch size was set as 16; Adam was used as the optimization function. During the training, an early stopping method was preferred to avoid overfitting and the value of early stop was set to 10. Accordingly, the validation accuracy value was calculated for each epoch during the training, and the training was stopped when the validation accuracy did not increase over 10 following epochs. Table 4 represents the hyperparameters of the proposed model in detail.

Table 4

Hyperparameters of the proposed image segmentation model

	Image size	256 × 256
	Normalization technique	Min–max
	Learning rate	0.0001
	Batch size	16
	Optimizer	Adam
	Beta_1	0.9
	Beta_2	0.999
Decoder	Stride(number)	2
	Upsampling layers(number)	5
	Activation function	ReLU, Sigmoid
	Filter sizes	256, 128, 64, 32, 16
	Padding	Same
	Filter size for convolution and upsampling layer	3 × 3, 2 × 2
	Kernel initializer	he_normal

Evaluation metrics

Researchers have used different metrics to evaluate segmentation performance. The most widely used performance evaluation criteria in segmentation of medical images are Dice Coefficient, Specificity, and Sensitivity.

Dice is designed to assess the overlap of estimation results and the ground truth and has a value between 0 and 1. The better prediction result will have a bigger Dice value. It is calculated as in Eq. 4.

Sensitivity, which measures the rate of successful prediction of positive samples, is calculated by the formula in Eq. 5. Here, TP is the number of true positives and FN is the number of false negatives.

Specificity, which measures the proportion of correctly identified negative samples, is calculated by the formula in Eq. 6. Here TN is the number of true negatives and FP is the number of false positives.

Results and discussion

To assess the efficacy of the method proposed in the study, many experiments were carried out and the results obtained in these experiments are presented in Table 5. Table 5 contains the Dice score, sensitivity and specificity values obtained for each modified U-Net model, as well as the results of the majority voting approach recommended in the study. When the Dice score metric, which is commonly used for measuring the performance of the segmentation model, is examined, it is seen that the most successful result was obtained with the majority voting approach with 85.03%. Similarly, in the Sensitivity performance metric, it is seen that the majority voting approach has a significantly higher value than all other models with 89.13%. The highest value for the specificity metric with 99.68% was obtained with the UNet_ EfficientNetB5 model. However, the majority voting approach also achieved a very high value of 99.38%.

Table 5

COVID-19 segmentation results of the proposed method

Model	Dice score (%)	Sensitivity (%)	Specificity (%)
U-Net_ VGG16	84.04	81.67	99.58
U-Net _ ResNet101	82.96	80.66	99.55
U-Net _ DenseNet121	83.33	79.45	99.63
U-Net _ InceptionV3	82.97	79.76	99.59
U-Net _ EfficientNetB5	82.91	77.73	99.68
Majority Voting	85.03	89.13	99.38

Random samples were selected from the dataset for demonstrating the segmentation efficiency of the proposed model, and the visualized results were given in Fig. 5. When Fig. 5 is examined, there are deficiencies in the detection of some small lesions in the single segmentation results of some models. However, the majority voting approach proposed in the study was able to accurately discover almost all lesion areas close to the ground truth. Considering the obtained visual performance results, it has been seen that the majority voting approach can increase the segmentation success.

Fig. 5

Segmentation results of random samples on dataset

The pandemic caused by the COVID-19, which has affected all of the world, has prompted many researchers to develop computer-assisted fully automatic diagnostic systems that can help experts. Table 6 presents the recent studies for the segmentation of the COVID-19 disease and the comparative results of the majority voting approach proposed in this study. When the Dice scores of the studies in Table 6 were examined, it was seen that the method proposed in this study had a very high success rate. However, the performance of the proposed method was compared with Budak et al. [23] and Zhou et al. [28], since it was thought that it would be appropriate to make the fairest performance comparison between those using the same dataset in the studies discussed. Among the studies using the same dataset, the highest Dice score belongs to the model developed by Budak et al. [23] with 89.61. The authors developed a network based on Segnet with an attention gate in this model and used various loss functions such as Dice, Tversky, and focal Tversky while training the network. The model developed by Zhou et al. [28] had a Dice score of 83.01%. The authors integrated an attention mechanism and res_dil block into this network based on the U-Net architecture and used the focal Tversky loss function while training the network. In this study, which has 85.03% Dice score, the encoder part of the U-Net model was modified, and transfer learning-based U-Net models were obtained through pre-trained DL models, and then the results of the single models were combined using the majority voting approach. The total loss function was used by summing the Focal Loss and Dice loss functions, which are frequently used in unbalanced data sets while training the network. In addition, when the table is examined, it is seen that although the Dice score of the method proposed in this study is lower than the model proposed by Budak et al. [23], it has the highest Dice score among U-Net-based architectures. On the other hand, the small number of images in the used dataset is the limitation of this study.

Table 6

Comparison with related studies

Study	Method	Number of images	Dice	Sensitivity	Specificity
Diniz et al. [22]	Residual U-Net	112,288	77.1%	–	99.8%
Budak et al. [23]	SegNet-based model	473	89.61%	92.73%	99.51%
Yan et al. [24]	COVID-SegNet	21,658	72.6%	75.1%	–
Fan et al. [25]	Semi-Inf-Net
Inf-Net	100	73.9% 68.2%	72.5% 69.2%	96.0% 94.3%
Wu et al. [26]	Deep learning-based segmentation	144,167	78.5%	–	–
Qiu et al. [27]	MiniSeg	829	80.06%	90.60%	99.15%
Zhou et al. [28]	U-Net-based network	473	83.1%	–	–
Proposed model	Modified U-Net	593	85.03%	89.13%	99.38%

Conclusion

In this study, a segmentation and DL-based approach has been presented for diagnosing COVID-19 lesions in chest CT images of COVID-19 infected people. In the proposed approach, the U-Net model, which is effective even in very few images, has been modified using deep learning models. Then, a new segmentation result was obtained with the majority voting approach, which was built by utilizing the segmentation results obtained from each modified U-Net model. In the study, the single segmentation performances of each U-Net model and the performance of the majority voting approach were evaluated separately. Experimental studies showed that the majority voting approach offered the best performance with a Dice score of 85.03%. Considering this high segmentation performance, it is thought that the majority voting approach proposed in the study will help to detect COVID-19 infection situations in clinical environments rapidly and cost-effectively.

In the future, it is planned to use more pre-processing methods and discuss attention mechanism-based deep learning networks such as the transformer model for improving the detection of small COVID-19 lesions.