FractalCovNet architecture for COVID-19 Chest X-ray image Classification and CT-scan image Segmentation.

Precise and fast diagnosis of COVID-19 cases play a vital role in early stage of medical treatment and prevention. Automatic detection of COVID-19 cases using the chest X-ray images and chest CT-scan images will be helpful to reduce the impact of this pandemic on the human society. We have developed a novel FractalCovNet architecture using Fractal blocks and U-Net for segmentation of chest CT-scan images to localize the lesion region. The same FractalCovNet architecture is also used for classification of chest X-ray images using transfer learning. We have compared the segmentation results using various model such as U-Net, DenseUNet, Segnet, ResnetUNet, and FCN. We have also compared the classification results with various models like ResNet5-, Xception, InceptionResNetV2, VGG-16 and DenseNet architectures. The proposed FractalCovNet model is able to predict the COVID-19 lesion with high F-measure and precision values compared to the other state-of-the-art methods. Thus the proposed model can accurately predict the COVID-19 cases and discover lesion regions in chest CT without the manual annotations of lesions for every suspected individual. An easily-trained and high-performance deep learning model provides a fast way to identify COVID-19 patients, which is beneficial to control the outbreak of SARS-II-COV.

Introduction

COVID-19 [1] is one of the major catastrophes in the history of mankind. It is important that suspected cases must go on with early diagnosis and detection so that they can recover from this disease. COVID-19 is caused by SARS-II Corona Virus [2] which was first discovered in China at late 2019. The virus majorly transmits through surfaces, human - human contact and through aerosols and droplets. The major impact of COVID-19 includes affecting the lungs with severe pneumonia, suffocation from loss of oxygen and it is still under research. There are few vaccines or drugs introduced to prevent or to cure this disease [3], [4]. But still there are muted variants of corona virus are spreading across the globe [5], [6] the effectiveness of the vaccine is not predicted for this modified virus strand [7]. So in this alarming pandemic situation it is our responsibility as individuals to be safe if any symptoms such as cough, sneeze, fever is present. As there is a probable transmission through aerosols, COVID-19 has significantly high factor.

We must be able to diagnose at earlier stage either through blood test with Reverse Transcription Polymerase Chain Reaction (RT-PCR) [8] or through antibody testing. A large crowd cannot exploit all the tests like RT-PCR. This is a golden method and its sensitivity and specificity is high according to the research. Priority must be given for the people who have all the symptoms. The main short-comes of RT-PCR kit is there are very few in terms of resource, the cost is too high, limited hospitals are having this facility. The time needed for report generation is high which does not meet the isolation of patients rapidly. It is important to find an alternative solutions like antibody testing using CT-scan images by practitioners to diagnose patients with COVID-19. Though they reduce the exploit of resource, there is a high chance of error and cost of testing also crucial.

In order to reduce cost and increase efficiency many solutions were proposed that deals with classifying the chest X-ray images and CT-scan images for diagnosing whether the patient is COVID-19 positive or negative. It would be much helpful if COVID-19 positive cases can be identified at the earlier stage, so that isolating of the patients will be easy. These action can limit the rapid spread of the disease. To perform automatic detection of COVID-19 patients we have designed a decision support tool which helps to automatically detect, classify, and segment the lesion region of lungs of the COVID-19 patients. In this work, we implemented a Fractal COVID-19 Network (FractalCovNet) architecture for classification of X-ray images and segmentation of the CT-scan images.

The FractalNet [9] was first proposed for classification of the images. Here the fractal blocks are used to build a very deep neural network for classification of the ImageNet dataset. Later, it was adopted for segmentation models. In this work, we propose a FractalCovNet using the Fractal modules. The overall framework of the proposed model is shown in Fig. 1 . The proposed approach performs segmentation of CT-scan images using the FractalCovNet architecture. We have also used the FractalCovNet for classification of chest X-ray images using transfer learning.

Fig. 1

Framework of the proposed FractalCovNet for classification and segmentation.

In this work, we have implemented various deep learning models such as ResNet50, VGG16, Xception, DenseNet201 and InceptionResNetV2 using transfer learning for classification of chest X-ray images and their performances are compared with our proposed model. The various segmentation models such as FCN (Fully Convolutional Network), U-Net, DenseUNet, and ResNetUNet are implemented for segmentation of CT-scan images and results are compared with the proposed approach. Metrics such as precision, recall, F-measure, and accuracy are used to evaluate the models. We hope using the models that we created will help in fast and early diagnosis of COVID-19 patient and help in saving life. We are proposing a cost and time effective measure for diagnosis as the medical practitioners suffer due to the huge workload to save each patient.

The organization of the paper is as follows: Section 2 discusses the related works. Section 3 describes the proposed FractalCovNet architecture for segmentation of chest CT-scan images and classification of the chest X-ray images. Section 4 describes the various datasets used, evaluation metrics along with the implementation details. Section 5 discusses the results obtained using the proposed approach and other pre-trained CNN models. Section 6 discusses the advantages of proposed model over other models available in the literature. Section 7 presents the conclusion.

Related Works

The process of automating the identification of COVID-19 patient has started in the early March 2020. But there were lesser number of test samples available for training the models. Some of the initial models developed for automatic detection of COVID-19 was designed using custom CNN models. And others were developed using the pre-trained CNN models like VGG-16 [10], ResNet [11], InceptionResnetV2 [12], etc. The models developed for automatic detection of COVID-19 can be classified into two types: Models developed using CT-scan images and models developed using X-ray images. The processing of X-ray images helps only in detection whether the patient has COVID-19 or not. Whereas the processing of CT-scan images can identify both whether the patient has COVID-19 or not, and also determine the severity of the infection.

There are many segmentation approaches proposed for segmentation of medical images [13], [14], [15]. The U-Net [13] is successfully used in many medical image segmentation approaches. The adaptive context selection approach in [14] uses an context selection method in U-Net for polyp segmentation. It adds a context detection block to process the features from down sampling block before giving as an input to the up sampling block. The approach in [15] proposes a parallel reverse attention network (PraNet) for segmentation of polyp. The parallel reverse attention block provides attention at multiple stages of up sampling to increases the accuracy of segmentation. In this work, we propose a variation of U-Net which uses fractal blocks in the U-Net for segmentation of chest CT-scan images. The features from the down sampling block is passes through an intermediate up sampling and down sampling blocks before provided as an input to the final up sampling block.

The approach in [16] is one of the early COVID-19 classification model developed using SqueezeNet to differentiate COVID-19 affected and non affected CT-scan images. The SqueezeNet consists of blocks called as ”Fire Module” which is a simple bypass connection. All the features from parallel fire modules are concatenated using global average pooling layer. Dual-branch Combination Network (DCN) was developed in [17] to classify the COVID-19 and other diseases infected patients. A lesion attention module was developed to isolate the infected region from the CT-scan images.

COVID-19 pneumonia lesion segmentation network (COPLE-Net) [18] was developed for noise robust lesion region segmentation. It also uses a noise robust DICE loss for segmentation, which helps to overcome the problem of limited dataset images available for training the model. Inf-Net [19] was developed to segment the COVID-19 lung infection automatically from the chest CT-scan images. Inf-Net consist of parallel partial decoder, global map, implicit and explicit edge attention model, and semi-supervised segmentation framework for segmenting the lung infected region of CT-scan images.

A multi-tasking model [20] was proposed for classification and segmentation of the CT-scan images. The automatic classification as well as segmentation tool identify lung region affected by COVID-19 from the CT-scan images. Their proposed work uses a common encoder to represent the disentangled feature which support segmenting and classifying the lesion regions. In [21] automatic segmentation for COVID-19 was implemented using CT-scan images. Here the large scene small object issue of medical images is resolved by using the symmetric property of lungs and tissues, which yield higher accuracy of segmenting the COVID-19 affected regions.

The Multi-Scale Discriminative Network (MSD-Net) [22] is designed to perform multi-class segmentation of the CT-scan images. MSD-Net uses a Pyramid Convolutional Block (PCB) which has varying size of kernels. It also has a Channel Attention Block (CAB) used to focus on the area to be segmented. A weakly supervised deep learning [23] framework is developed to localize the lesion region of lung CT-scan images using pre-trained U-Net model. The segmented images is fed as an input to 3D deep neural network DeepCovNet for classification. The approach proposed in [24] uses Residual block with U-Net for segmentation of CT-scan images. Here the ground truth labels for multiple class in segmentation is obtained from radiologists.

A deep learning based model is proposed for classification of CT-scan images in [25] using GoogleNet InceptionV3 by applying transfer learning. The approach proposed in [8] segment the region of infection using 3D-CNN model. The segmented region is given as input to the pre-trained ResNet-18 model for feature extraction which is used for classification. Here the pre-trained CNN models are used only for classification, but we propose a model for segmentation of lesion region from the CT scan images.

Another approach in [26] uses UNet++ architecture proposed in [27] for medical image segmentation. The UNet++ architecture is similar to the U-Net [13], but here the encoder and decoder are connected through nested, dense skip pathways. The UNet++ is proved to be efficient for various medical image segmentation tasks. Another AI based system is proposed for diagnosis of COVID-19 using UNet++ [28]. Here the segmented image along with help of medical practitioners is used to diagnosis the CT-scan of COVID-19 patients. The interpretation of the results by medical purpose does not provide an automated system for diagnosis of COVID-19. We propose a novel architecture that uses fractal blocks in U-Net architecture. This results in better segmentation of the lesion region compared to basic U-Net based models.

The Deep COVID proposed in [29] uses various deep learning model by applying transfer learning to detect COVID-19 from X-ray images. They have trained various deep learning models like ResNet (18 and 50), SqueezeNet and DenseNet-121. The approach in [30], [31] uses pre-trained Xception [32] model for classification of COVID-19 chest X-ray images by applying transfer learning. Another approach proposed in [33] uses the various pre-trained CNN model for feature extraction. A correlation based feature selection is applied on the extracted features and a Bayesnet classifier is developed for classification. The proposed approach uses FractalCovNet pre-trained on chest CT-scan images, this provides better results as the model is pre-trained on medical images rather than use of ImageNet dataset.

Computer Aided Diagnosis (CAD) [34] was developed to detect the infection occurred by coronavirus using pre-trained VGG-16 model. The diaphragm regions are removed from the X-ray image. Then two different pre-processing methods like histogram equalization and bilateral low pass filter are used to generate a pseudo color image. Here removing diaphragm region improves the performance of the model. The approach in [35] proposed a Bayesian-CNN for classification of X-ray images with human intervention. This model is dependent on human experts for accurate recognition of COVID-19.

In [36] a sequential CNN model is proposed for detecting COVID-19 using chest X-ray images. The proposed model performs better than the other pre-trained CNN models for chest X-ray classification. In an another approach COVID-GAN [37] is designed for generating synthetic chest X-ray (CXR) images. Initially, they generated CXR image from Auxiliary Classifier Generative Adversarial Network (ACGAN) model. The model produces high accuracy of 95% for classification, but it also includes the generated synthetic images. A custom COVID-Net [38] model is developed for classification of chest X-ray images of COVID-19 patients. The model is first pre-trained using ImageNet dataset before fine tuning it for COVID-19 chest X-ray classification.

An Artificial Intelligence (AI) based model is proposed in [39] for diagnosis of COVID-19 patients. Here the attention region of the deep learning model is used to visualise the effective representation of CT-scan images obtained from COVID-19 patients. The AI system acts as an integrated system with human expertise, which again requires time from the medical practitioners. Another approach in [40] proposes Fractal model for feature extraction from CT-scan and X-ray images for segmentation and classification. The fractal model extracts the features and uses Eigen analysis for feature selection and dimensionality reduction. These features are then provided to the CNN model for segmentation and classification.

Though there are many approaches proposed for detection of COVID-19 from the chest X-ray and CT-scan images. Many approaches use CNN models pre-trained using ImageNet dataset for detection of COVID-19. The ImageNet dataset comprises of RGB images that belongs to wide variety of images. The fine tuning of these pre-trained model with larger number of parameters using the CT or X-ray images reduces the performance of the model. This is due to the limited number of CT-scan and X-ray images available from COVID-19 patients. Thus, we propose a custom FractalCovNet architecture for classification and segmentation. We have designed the FractalCovNet using U-Net along with the fractal blocks in the architecture for classification of chest X-ray images and segmentation of lesion in the CT-scan images.

Other segmentation models

In addition to the proposed FractalCovNet model we have implemented the following models for segmentation of CT-scan images. The models are trained using transfer learning and the results are compared with the proposed FractalCovNet model for segmentation of CT-scan images.

Fully Convolutional Network (FCN):

This is a simple custom convolutional network built for the segmentation of the CT-scan images obtained from COVID-19 patients. The model is built using conv blocks and pooling layer to extract the features. At the end of each stage, a lambda layer is added with the linear activation. The linear activated layer is used to transform the result for pixel wise classification of the images. This model has disadvantages due to the bias generated while training the models.

Segnet

The Segnet model consists of an encoder and decoder. The encoder consists of a set of convolution blocks with batch normalization layer used for feature extraction. The continuous blocks of convolution and batch normalisation extracts the normalized feature map. The decoder consists a set of convolution blocks and batch normalization in a decreasing order. The resultant feature set is send to output layer to accurately detect the spots of infection. It compares the pixel wise values to detect the lesions on the CT-scan images.

U-Net

U-Net [13] is a Fully Convolutional Network (FCN) architecture proposed for segmentation of medical images. The U-Net model uses symmetric architecture at the encoder and decoder. The features from the encoder are concatenated to the decoder using skip connections instead of a sum. It consist of three parts: the contracting/down sampling path, Bottleneck layer, the expanding/up sampling path. The contracting path aims to extract the context from the input image to perform segmentation.

ResUNet

The ResUNet [24] model is constructed using residual blocks [11] at the U-Net architecture. The residual blocks helps in easier and fast computation of features by skipping certain convolution blocks. The ResUNet consists of identity blocks and convolution blocks. The identity block in the ResUNet helps to maintain the image dimension and helps in skipping part of the ResUNet architecture.

DenseUNet

The DenseUNet architecture proposed in [41] is divided into two sections: down sampling and up sampling similar to U-Net. But at each step of up sampling and down sampling a dense block is used to process the input features. Then the resultant feature map is concatenated and given as input to the next stage after pooling. The dense block at each step further increases the depth of the model which helps in better segmentation of the input images.

Other classification models

The following pre-trained models are trained for classification of chest X-ray images using transfer learning. We have compared the results obtained by the proposed FractalCovNet architecture with the results obtained by the following pre-trained CNN models.

VGGNet

The VGG [10] is a pre-trained Convolutional Neural Network developed by Visual Geometry Group for solving the ImageNet challenge. The model is trained on the ImageNet challenge (ILSVRC) for classifying the images that belongs to 1000 class. Here two architectures were proposed: VGG-16 which consists of 16 layers and VGG-19 which consists of 19 layers. Both the model consists of a stack of convolution and pooling layers. It was shown that the VGG-16 model provides better classification accuracy. Hence in our work we use the VGG-16 model by extracting features from the final fully connected layers to obtain a feature vector of size 4096.

ResNet

Increasing the number of layers in the deep learning models have increased the performance of the models designed for computer vision task. But, as we increase the number of layers, many difficulties arise while training the model like overfitting, vanishing and exploding gradient problem, etc. To overcome these problems Residual Networks was proposed to solve the ImageNet challenge in [11]. Different ResNet architectures were proposed using 18, 34, 50, 101 and 152 layers. The ResNet-152 model provided best classification results for the ImageNet challenge. In this work, we have chosen the ResNet-50 model due to lesser amount of data available for training.

InceptionResNetV2

The InceptionResnetV2 [12] proposes a deep learning architecture which uses a Inception module. The inception module has a number of parallel convolutions, the output obtained from the convolutions are concatenated and given as input to the next convolution module. There is a filter expansion layer after every Inception block in the model which scales up the dimensionality of the filter bank. The InceptionResnetV2 model consists of both the Inception module and the residual blocks.

Xception

Xception [32] model is a deep convolutional neural network architecture that uses depth-wise separable convolutions followed by point-wise convolutions. The Xception (Extreme Inception) is similar to the Inception module but the Inception modules are replaced by the depth-wise separable convolutions. Every separable convolution layer in the architecture is followed by batch normalization layer.

Proposed FractalCovNet architecture

We propose a novel FractalCovNet architecture for segmentation of CT-scan images and classification of X-ray images. It enables automatic detection of the COVID-19 patients which will help in early treatment of the COVID-19 patients. The FractalCovNet consists of U-Net architecture with fractal blocks. The architecture of the proposed FractalCovNet is shown in Fig. 2 . The architecture shows the various expanding and contracting layers of the model. The FractalCovNet is first trained for segmentation of CT-scan images. This model is then fine tuned for classification of X-ray images using transfer learning.

Fig. 2

The proposed architecture using fractal blocks and U-Net for segmenting lesion region.

FractalCovNet for segmentation of chest CT-scan images

Fractal network was first introduced in [9] for image classification task. It was shown that the very deep neural networks perform well in classification task, irrespective of the usage of residual block. Hence we choose a variant of the basic fractal network for our segmentation task. U-Net [13] was proposed for segmentation of medical images which was found to be successful in various medical image segmentation tasks. It is also used in various image reconstruction tasks. In this work, we propose a variant of U-Net model with fractal blocks for segmentation of lesion regions from CT-scan images of the COVID-19 patients.

A U-Net usually has two path of CNN layers for feature extraction called as expanding path and contracting path. The contracting path down samples the input image into smaller dimensions. While the expanding path up samples the output from the previous layer. The U-Net uses feature concatenation from the contracting path to the expanding path at each level of the model which helps in reconstruction of the images. The contracting path reduces the feature dimension size till the bottleneck layer of the model. The features of the bottleneck layer are used by the expanding path to construct the segmented output images. As the proposed FractalCovNet has two paths for expansion and another two paths for contraction, the architecture has the double U-shape as shown in Fig. 2. The features from the first contracting path flows to the final expanding path through the fractal block.

Let I be the input image for the FractalCovNet model. The input image is passed through a set of convolution layer to in the first contracting path of the model as shown in Fig. 2. Then the output is passed through the bottleneck layer . The output of the bottleneck layer is passed through the expanding path from to . Again the output of the first expanding path is passed through the second bottleneck layer . The output of the bottleneck layer is passed through the contracting path to . is the third bottleneck layer of the model. The layers from to forms the final expanding layer of the FractalCovNet architecture. The final expanding path generates the segmented output image for the given input image I. Features from the first contracting layers is passed through two set of convolutions before reaching the final expanding path.

The expanding paths enables precise localization combined with extraction of contextual information. The U-Net model combines the location information from the down sampling path with the contextual information obtained from the up sampling path. The localisation and context information is obtained from the contracting and expanding path of the model. In this work, each layer in the first contracting path of the model is concatenated with features of the expanding path. These features are processed by a convolutional block and then concatenated with the features obtained from an another down sampling path of the second U-Net. For example we can see that the features from is concatenated from up sampling features from . These features are processed by convolution block at and provided as input to and also . The features at again undergoes the same process and given as input to . This acts like a fractal block, which processes the features at various levels of convolution, thus enables the model to accurately determine the segmentation regions.

After the final convolution and pooling layer we have applied sigmoid activation function to classify the pixel values and to detect the localization of lesion in the image pixel. The output array with dark region deNotes 0’s and white region represent the lesion region denoted by 1’s. The model is trained using cross entropy error calculated for each pixel in the segmented output image. The proposed FractalCovNet model yields lower MAE and higher F-measure value compared to the other deep learning models proposed in the literature.

FractalCovNet for classification of chest X-ray images

The FractalCovNet architecture proposed in Section 3.1 is adapted for classification of chest X-ray images of COVID-19 patients. We apply transfer learning on the FractalCovNet architecture. Fig. 3 shows the architecture for the classification of the chest X-ray images using the pre-trained FractalCovNet model used for segmentation. The input chest X-ray image is passed through the FractalCovNet and an another set of convolution and pooling layers for extracting the features. Let J be the input chest X-ray images for classification. The input is passed through the FractalCovNet model to obtain feature vector and another set of convolution and pooling layers denoted as to extract the feature vector . These features are concatenated and passed through fully connected layers for classification of whether the given input image belong to COVID-19 patients or other cases. The output of the model is defined as:where is the concatenation of the two vectors. We use categorical cross entropy loss function for training the model. The categorical cross-entropy loss function is defined aswhere C is the number of class and is the ground truth label and is the predicted probability. As the FractalCovNet is trained using slices of CT-scan images, it is not suitable for extracting features of X-ray images. Hence we use transfer learning on the FractalCovNet for fine tuning the model for X-ray images. The layers of the FractalCovNet is trainable during the training of classification model. The final fully connected layer produces the final label whether the input chest X-ray image is diagnosed with COVID-19 or not.

Fig. 3

FractalCovNet architecture for classification.

Experiments

Dataset

COVID-19 CT Dataset:

The CT-scan image requires manual annotation of the slices for the CT-scan by radiologist for creating ground truth mask. There are only limited number of datasets available publicly for use. In this work, we use CT-scan image dataset available in [42]. It consists of initially 100 slices of CT-scan images with masks. Later, another 373 CT-scan images was added with masks. Thus we have totally 473 CT-scan images with ground truth mask for lesion region. As we were interested only in segmentation of lesion region we chose only the CT-scan images available with ground truth masks.

COVID-19 chest X-ray Dataset 1

Cohen et al. proposed a collection of X-ray and CT-scan images [43] on 25th March 2020 which comprised of 100 chest X-ray images of COVID-19 patients and 22 chest X-ray images of other disease. Later the number of images in the dataset was improved in [44] which consists of totally 422 images which comprises of chest X-ray/ CT-scan images of patients with different diseases like Acute Respiratory Distress Syndrome (ARDS), COVID-19, Middle East Respiratory Syndrome (MERS), pneumonia, Severe Acute Respiratory Syndrome (SARS) and several diseases related to lungs. Where the number of images with COVID-19 is maximum of 340 images. In addition to the above data we have used normal chest X-ray images from Kaggle repository [45] called ”Chest X-ray Images (Pneumonia)”. It consists of totally 1583 normal chest X-ray images and 4273 pneumonia images. We combined this data with Cohen dataset [44] for classification of COVID-19 from others.

COVID-19 chest X-ray Dataset 2 [46]:

This dataset is obtained from the Kaggle website [46]. It consists of totally 5914 normal/pneumonia chest X-ray images and it also consists of 164 chest X-ray images from COVID-19 patients (6078 images in total) obtained from [47].

Evaluation Metrics

In this work, we use the accuracy, precision, recall, and F-measure as the evaluation metrics to evaluate the performance of classification of chest X-ray images. We have used F-measure/ Dice coefficient, Precision, Recall and Mean Absolute Error (MAE) for evaluating the segmentation results of CT-scan images.

Accuracy

Accuracy measures the number of instances correctly predicted over the total number of instances as in Eq. 3.

Precision

Precision calculates the ratio of number of sample correctly predicted as positive to the total number of samples predicted as positive as in Eq. 4.

Recall

Recall calculates the ratio of number of sample correctly predicted as positive to the total number of samples available in that particular class as in Eq. 5.

F-measure

F-measure is the weighted average of both the Precision and Recall as om Eq. 6.Where TP, FP, TN and FN represent the number of True Positive, False Positive, True Negative and False Negative, respectively. Given a test dataset, TP is the proportion of positive (COVID-19) that are correctly labeled as COVID-19 by the model; FP is the proportion of negative (OTHERS) that are mislabeled as positive (COVID-19); TN is the proportion of negative (OTHERS) that are correctly labeled as normal and FN is the proportion of positive (COVID-19) that are mislabeled as negative (normal) by the model.

Mean Absolute Error

The Mean Absolute Error measures the correctness of the proposed FractalCovNet for segmentation by comparing the ground truth masks with the segmented output images. The MAE is calculated as:where is the value of the segmented image at position . is the value of the ground truth mask at position . Here M and N are the dimension of the images.

Implementation Details

We have implemented the proposed FractalCovNet model for classification and segmentation using Keras with Tensorflow backend. All experiments were performed on Google Colaboratory and Kaggle Kernels. We have used random initialization weights for all the models. The model is trained using Adam optimizer with a learning rate of 1e-4. The batch size and number of epochs were experimentally set to 16 and 100 respectively for all experiments.

For classification the ResNet, DenseNet, InceptionResNetV3, and Xception models are used with transfer learning. These models were pre-trained with ImageNet dataset. The optimizer used for training the model is Adam optimizer with a learning rate of 1e-4. The batch size and number of epochs were experimentally set to 8 and 50 for all experiments. The dataset used was randomly split into two independent datasets with 60%, 20% and 20% for training, testing, and validation respectively.

Results

In this section we discuss the results obtained for classification of X-ray images and segmentation of lesion region from the CT-scan images. We have analysed the results obtained for classification and segmentation by plotting confusion matrix and ROC curve.

Analysis of Classification results

Table 1 shows the accuracy, precision, recall and F-measure obtained for the proposed FractalCovNet and the other pre-trained CNN models for classification of chest X-ray images from Dataset 1. Here ACC represents accuracy, PRE represents precision, REC represents Recall and F1 represents F-measure. Table 2 shows the results for chest X-ray Dataset 2. Table 3 shows the result obtained by training model using both chest X-ray Dataset 1 and Dataset 2. From the results shown in the table we can see that the proposed approach provides better evaluation results than the other deep learning models available in the literature. Thus we can say that the proposed FractalCovNet architecture provides better performance than the other pre-trained CNN models.

Table 1

Results for classification for the dataset COVID-19 chest X-ray Dataset 1.

Models	ACC	PRE	REC	F1
Xception	0.95	0.84	0.91	0.87
VGG16	0.95	0.85	0.86	0.85
ResNet50	0.96	0.90	0.87	0.88
DenseNet201	0.97	0.89	0.91	0.90
InceptionResnetV2	0.96	0.96	0.90	0.84
FractalCovNet	0.99	0.99	0.87	0.92

Table 2

Results for classification for the dataset COVID-19 chest X-ray Dataset 2.

Models	ACC	PRE	REC	F1
Xception	0.99	0.97	0.96	0.96
ResNet50	0.97	0.98	0.82	0.92
InceptionResnetV2	0.98	0.98	0.91	0.94
DenseNet201	0.99	0.97	0.95	0.96
VGG16	0.98	0.95	0.96	0.96
FractalCovNet	0.99	0.99	0.97	0.98

Table 3

Results for classification for the dataset COVID-19 chest X-ray Dataset 1 and Dataset 2.

Models	ACC	PRE	REC	F1
Xception	0.92	0.93	0.88	0.89
ResNet50	0.94	0.89	0.87	0.87
DenseNet201	0.96	0.89	0.86	0.85
InceptionResnetV2	0.95	0.93	0.89	0.89
VGG16	0.87	0.84	0.75	0.82
FractalCovNet	0.98	0.88	0.94	0.92

Fig. 4 shows the confusion matrix for the models trained using the proposed FractalCovNet and the pre-trained CNN models like VGG-16, ResNet-50, InceptionResnetV2, Xception and DenseNet-201 using only the test set of the images. Fig. 4, Fig. 4b shows the confusion matrix of InceptionResnetV2 and Xception, there is a larger number of false positive but no false negative. Fig. 4, Fig. 4, Fig. 4e shows the confusion matrices of DenseNet-201, VGG-16 and ResNet-50 respectively. It is seen that there are few number of false negatives compared to the false positives. Fig. 4f shows the confusion matrix obtained for the proposed FractalCovNet. It is seen that there is only one false negative compared to the false positive, which helps in containment of the COVID-19 spread.

Fig. 4

Confusion matrix of Classification using features extracted from different pre-trained models.

Fig. 5, Fig. 5b shows the Precision-Recall curve and ROC curve plotted for chest X-ray images from Dataset 1 for FractalCovNet and other pre-trained CNN models like ResNet, InceptionResnetV2, VGG, Xception and DenseNet. It is seen that the precision of the proposed FractalCovNet is better than the other pre-trained CNN models. Fig. 5, Fig. 5d shows the plot of training accuracy and loss for Dataset 1. The plot shows that the training loss and accuracy of FractalCovNet is better compared to ResNet model.

Fig. 5

Precision-Recall curve, ROC Curve, Plot of training accuracy and loss for chest X-ray Dataset 1 using proposed FractalCovNet and Resnet model.

Fig. 6a shows the precision curve plot for the proposed FractalCovNet compared to the other pre-trained models like ResNet, InceptionResnetV2, VGG, Xception and DenseNet using chest X-ray images from Dataset 2. It is seen that the proposed FractalCovNet provides better precision and recall compared to the other pre-trained CNN models. Fig. 6b shows plot of ROC curves for chest X-ray images from Dataset 2. Fig. 6c and 6d shows the plot of accuracy and loss for chest X-ray images from Dataset 2. It is seen that the proposed FractalCovNet model converges faster than the other pre-trained models used for classification.

Fig. 6

Precision-Recall curve, ROC Curve, Plot of training accuracy and loss for chest X-ray Dataset 2 using proposed FractalCovNet and Resnet model.

Analysis of Segmentation results

Table 4 shows the results obtained for segmentation using various segmentation models. It shows the MAE, precision and recall scores obtained for the segmentation. It is seen that the proposed FractalCovNet provides better results compared to the other segmentation models available in the literature. The FCN model provides the lowest results compared to the other models. It can also be seen that the ResNetUNet model provides better results compared to the U-Net model, thus the residual blocks are useful than the model developed using only the U-Net architecture.

Table 4

Comparison of results for segmentation of chest CT-scan images.

Models	MAE	PRE	REC	F1
FCN	0.477	0.498	0.698	0.377
Segnet	0.180	0.621	0.720	0.619
U-Net	0.076	0.768	0.630	0.661
DenseUNet	0.074	0.794	0.646	0.703
ResNetUNet	0.074	0.767	0.685	0.712
FractalCovNet	0.064	0.804	0.697	0.745

Fig. 7, Fig. 7b shows the accuracy and the loss values generated while training the segmentation models for segmentation of lesion region from CT-scan images. It is seen that the proposed FractalCovNet model converges faster compared to the other models.

Fig. 7

Training Accuracy and Loss for Segmentation. F-measure curve and Precision-Recall curve for segmentation.

Fig. 7, Fig. 7d shows the F-Measure and Precision-Recall curve plotted for segmentation using the various models. The F-Measure is calculated for segmentation by comparing the segmented results generated by models with the ground truth images. The segmented images shows that the proposed FractalCovNet model performs better segmentation than the other models in the literature. The FCN is very poor at segmentation whereas the U-Net, DenseUNet, and ResNetUNet provides better segmentation compared to the FCN model.

The segmentation results for the sample images are shown in Fig. 8 . The figures shows 6 sample CT-scan images with various levels of infection. It is seen that the proposed FractalCovNet provides better segmentation results and the segmentation is more similar to the ground truth mask region. The ResNetUNet model provides better results compared to the basic FCN and SegNet models. The ResNetUNet model provides better segmented image but not as good compared to the segmented image provided by the FractalCovNet model.

Fig. 8

Results obtained for various segmentation methods.

Discussion

Comparison with other Segmentation models

Table 5 shows the results of the various approaches proposed for COVID-19 CT-scan image segmentation and classification.

Table 5

Comparison of various approaches to COVID-19 CT-scan image segmentation and classification.

Methods	Task	Results	Dataset
GoogleNet InceptionV3 [25]	Classification	Accuracy - 79.3% Specificity - 83% Sensitivity - 67%	COVID-19 and Viral Pneumonia:1065
ResNet + Attention [8]	Segmentation and classification	Accuracy - 86.7%	COVID-19:219 Influenza-A:224 Normal:175
MSD-Net [22]	Segmentation	Dice co-efficient - 74.22%	COVID-19:219 Influenza-A:224 Normal:175
AI model [39]	Segmentation and classification	Specificity - 96.36% Sensitivity - 80.09%	COVID-19:313 others:229
U-Net++ [26]	Segmentation and classification	Per patient result: Accuracy-96% Specificity - 94% Sensitivity - 98%	COVID-19:51 others:55
U-Net++ + CNN [28]	Segmentation and classification	Specificity - 92.2% Sensitivity - 97.4%	COVID-19:723 others:413
DRENet ResNet-50 + FPN [48]	Segmentation and classification	Segmentation: Sensitivity-96% Precision-79%	COVID-19:88 Bacterial Pneumonia:100 Normal:86
DNN + fractal [40]	Segmentation and classification	Segmentation: Accuracy:70% Classification: Accuracy: 83.84%	COVID-19: 100
Inf-Net [19]	Segmentation	MAE: 0.082	COVID-19: 100
DeCoVNet [23]	Segmentation and classification	Segmentation–Hit rate:65.7% Classification: Accuracy: 90.1%	COVID-19: 131
ResidualAtt U-Net [24]	Segmentation multi-class	Accuracy-79% Precision: 82%	Covid-19: 110
COVID-CT-Seg [21]	Segmentation	Dice co-efficient - 78.3%	COVID-19: 201
FractalCovNet	Segmentation	F-measure (dice coefficient) - 74.5%
Precision-80.4%
MAE-0.064	COVID-19:349

Few approaches has been propose to use the pre-trained CNN models for classification of CT-scan images. The approach in [25] uses pre-trained GoogLeNet and the approach in [48] uses pre-trained ResNet50 for classification of CT-scan images. In this work, we have proposed a model for only segmenting the lesion region from CT-scan images. Another AI based approach is proposed in [39] uses CNN for diagnosis of COVID-19 patients, it requires intervention by medical experts. But in our approach we propose an automated FractalCovNet architecture for segmentation of lesion region from CT-scan images of COVID-19 patients.

The approach proposed in [26], [28] uses UNet++ architecture for segmentation and diagnosis of CT-scan images of COVID-19 patients. The UNet++ [26] uses only around 100 images for training and testing, it achieves accuracy of 96% per patient. The approach in [28] uses UNet + CNN with AI assisted model for diagnosis of COVID-19 patients using CT-scan images. In the proposed approach, we perform segmentation of lesion region which produces a F-measure of 74.5%. Thus the U-Net based FractalCovNet architecture provides better results compared to the other U-Net based approaches.

The COVID-CT-Seg segmentation model in [21] uses data augmentation and performs 3D segmentation to achieve dice coefficient of 78.3%. But in our proposed approach we have used 2D-segmentation without any augmentation. Thus the proposed approach performs well compared to the approach used in [21]. The Inf-Net [19] uses reverse attention model with parallel partial decoder. It uses semi-supervised algorithm for segmentation of the lesion to produce MAE error of 0.64. Without the semi-supervised training algorithm it produces an MAE of only 0.082, which is less than our proposed approach. The weakly supervised approach DeCovNet [23] achieves hit rate of 65.7% for segmentation. But the proposed approach provides F-measure of 74.5% which is better than the DeCovNet model.

The approach in [8] uses segmentation and provides segmentation region for classification, it produces an accuracy of 86.7% for the classification model. The approach in [40] produces segmentation accuracy of 70% for CT-scan images. Our approach produces precision socre of 80.4% for pixel-wise segmentation thus it will be able to diagnose COVID-19 with greater accuracy than the other existing approaches. The multi-tasking approach in MSD [22] produces dice coefficient of 74.22% but the proposed approach provides multi-tasking and produces dice coefficient of 74.5% and precision of 80.4%. Thus the proposed approach provides better segmentation results for segmentation of CT-scan images compared to the other state-of-the-art approaches.

Comparison with other Classification models

Table 6 shows the results for the various approaches proposed for COVID-19 X-ray image classification. There are many approaches proposed for classification chest X-ray of COVID-19 patients images using basic machine learning and deep learning approaches.

Table 6

Comparison of various approaches to COVID-19 chest X-ray image classification.

Methods	Task	Results	Dataset
CAAD [49]	COVID-19 or others	Accuracy-78.57%	COVID-19:599 Others-2107
CoroNet [31]	4 class classification	Accuracy-89.60%	COVID-19:290 Pneumonia bacterial:660 pneumonia viral:931 normal:1203
Bayesian ResNet50v2 [35]	COVID-19 or other	Accuracy-89.82%	Normal: 1583 Bacterial Pneumonia: 2786 Viral Pneumonia: 1504 COVID-19: 68
CNN + Bayesnet [33]	COVID-19 or others	Accuracy-91.6%	COVID-19:453 Others:497
DNN + fractal [40]	COVID-19 or others	Accuracy-93.2%	COVID-19:342 Others:341
COVID-Net [38]	COVID-19, Pneumonia or Normal	Accuracy-93.3%	COVID-19: 266 Normal:8066 Pneumonia: 5538
SPP-COVID-19 [50]	4 class classification	Accuracy-94.60%	COVID-19:219 Others:2686
Capsule Networks [51]	COVID-19 or others	Accuracy-95.7%	COVID-19: 266 Normal:8066 Pneumonia: 5538
ResNet-50 [52]	COVID-19 or normal	Accuracy - 96.1%	COVID-19:341 Normal:2800
Xception [30]	COVID-19 or others	Accuracy-97.40%	COVID-19–127 Pneumonia-500 Others-500
CNN [36]	COVID-19 or others	Accuracy-97.56%	COVID-19:165 Normal:497
nCOVnet [53]	COVID-19, Pneumonia or others	Sensitivity-97.62% Specificity-78.57%	COVID-19:42 Others:42
DenseNet-121 [29]	COVID-19 or other	Sensitivity-98% Specificity-75.1%	COVID-19: 184 Others:5000
DarkNet [54]	COVID-19 or others	Accuracy-98.08%	COVID-19–127 Pneumonia-500 Others-500
VGG-16 [34]	COVID-19 or other	Accuracy-98.1%	COVID-19: 415 Others:8059
ResNet101 [55]	COVID-19, pneumonia, bacterial and other virus	Accuracy-98.93%	COVID-19:250 pneumonia, bacterial and others: 1115
OptiDCNN [56]	COVID-19 or others	Accuracy-99.16%	COVID-19:184 Others:5000
Robust DL [57]	COVID-19 or others	Accuracy-99.6%	COVID-19:659 Others:6225
FractalCovNet	COVID-19 or others	Accuracy-99.8%	COVID-19:458 Others:5914

The approaches in [49], [52], [35], [55] uses pre-trained ResNet model for classification of chest X-ray images of COVID-19 and non COVID-19 patients. The approach in [52] uses transfer learning of ResNet50 model and approach in [55] uses ResNet-101 for classification of chest X-ray images. The approach in [35] uses Bayesian model with ResNet50 model for classification. The approach in [33] uses several pre-trained CNN models and uses feature selection for choosing the features and performs classification. The approach in [31], [30] uses Xception architecture and for classification of chest X-ray images.

The approach in [34], [53] uses VGG-16 with transfer learning for classification of chest X-ray images. It is seen that the use of ResNet pre-trained model provides better accuracy when compared to the other approaches. But the proposed approach applies transfer learning on the FractalCovNet architecture for classification of the X-ray images. Table 6 shows that the proposed approach provides better results compared to the models that uses pre-trained CNN using transfer learning for classification of X-ray images. The OptiDCNN [56] uses Biogeography-Based optimisation for training the classification model, it produces an accuracy of 99.16%. But our proposed model uses Adam optimizer for training the model and provides an accuracy of 99.8% which proves the effectiveness of the proposed FractalCovNet architecture.

The lightweight architecture proposed in [50] uses spatial pyramid pooling along with convolution layers for developing a classification model. It provides an accuracy of 94.6% which is lower compared to our proposed approach. Another lightweight module [51] uses capsule module for classification of chest X-ray images and produces an accuracy of 95.7%. A custom CNN model is developed for classification of X-ray images in [36], it produces an accuracy of 97.4%. The light weight models trade-off accuracy for the model complexity. But we need more accurate model for diagnosis of the COVID-19 disease for the containment of the disease. The proposed model provides higher accuracy than the other state-of-the-art approaches. Hence the proposed approach will help in timely diagnosis of the disease.

The Fractal based network [40] produces an accuracy of 93.2% for classification of X-ray images into COVID or non-COVID patients. But the proposed approach using Fractal blocks provides an accuracy of 99.8% for classification of chest X-ray images. The approach in [57] uses Darknet-53 model for classification of chest X-ray images. But it also uses the expertise of medical representative for the classification model, which is very difficult due to the pandemic. The proposed approach provides better results without intervention of human experts. Thus the proposed approach provides better results than the other state-of-the-art approaches for classification of chest X-ray images.

Conclusion

In this paper, we have proposed a novel FractalCovNet for classification and segmentation of chest X-ray images and CT-scan images respectively. The classification of chest X-ray images will help in faster diagnosis of the disease which in turn helps in early diagnosis and containment of the COVID-19 disease in the patients. The segmentation of the chest CT-scan images helps in identifying the percentage of infection which will help to provide timely medical support for the patients. The FractalCovNet is model adapted from the U-Net architecture for segmentation. The U-Net based segmentation have provided promising results of segmentation of CT-scan images of COVID-19 patients. The FractalCovNet proposed in this work is used for both classification and segmentation. The proposed approach provides an accuracy of 99% for classification which is better than the other state-of-the-art approaches. The segmentation model provides MAE of 0.064 and F-measure of 74.5% which is better than the other existing approaches proposed for segmentation of CT-scan images.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.