Automatic prediction of COVID- 19 from chest images using modified ResNet50.

Recently coronavirus 2019 (COVID-2019), discovered in Wuhan city of China in December 2019 announced as world pandemic by the World Health Organization (WHO). It has catastrophic impacts on daily lives, public health, and the global economy. The detection of coronavirus (COVID- 19) is now a critical task for medical specialists. Laboratory methods for detecting the virus such as Polymerase Chain Reaction, antigens, and antibodies have pros and cons represented in time required to obtain results, accuracy, cost and suitability of the test to phase of infection. The need for accurate, fast, and cheap auxiliary diagnostic tools has become a necessity as there are no accurate automated toolkits available. Other medical investigations such as chest X-ray and Computerized Tomography scans are imaging techniques that play an important role in the diagnosis of COVID- 19 virus. Application of advanced artificial intelligence techniques for processing radiological imaging can be helpful for the accurate detection of this virus. However, Due to the small dataset available for COVID- 19, transfer learning from pre-trained convolution neural networks, CNNs can be used as a promising solution for diagnosis of coronavirus. Transfer learning becomes an effective mechanism by transferring knowledge from generic object recognition tasks to domain-specific tasks. Hence, the main contribution of this paper is to exploit the pre-trained deep learning CNN architectures as a cornerstone to enhance and build up an automated tool for detection and diagnosis of COVID- 19 in chest X-Ray and Computerized Tomography images. The main idea is to make use of their convolutional neural network structure and its learned weights on large datasets such as ImageNet. Moreover, a modification to ResNet50 is proposed to classify the patients as COVID infected or not. This modification includes adding three new layers, named, 'Conv', 'Batch_Normaliz' and 'Activation_Relu' layers. These layers are injected in the ResNet50 architecture for accurate discrimination and robust feature extraction. Extensive experiments are carried out to assess the performance of the proposed model on COVID- 19 chest X-Ray and Computerized Tomography scan images. Experimental results approve that the proposed modification, injected layers, increases the diagnosis accuracy to 97.7% for Computerized Tomography dataset and 97.1% for X-Ray dataset which is superior compared to other approaches.

Introduction

According to international reports, Wuhan, China was the focal point from which COVID− 19 started rapidly to attack the world. With its tremendous spreading all over the world, it was classified as a pandemic by WHO. COVID− 19 is a respiratory disease, with common symptoms such fever, cough, short breathing, sore throat, headache, diarrhoea, the vanishing of taste, tiredness, aches, loss of smell, and finally, severe respiratory failure lead to deadness. To the time of writing this paper, there is no approved vaccine to avoid the infection by COVID− 19. coronavirus spreads like wildfire where people can be easily infected from cough droplet of infected persons.

The strategy announced by WHO to stop the virus spreading to healthy persons is to detect and isolate infected persons. Laboratory tests such as PCR, antigen, and antibodies have some limitations concerned with time to get results, the accuracy of results, cost of test, infection phase, and availability of laboratories that can serve the test. For example, only central laboratories can serve the PCR test to detect the COVID− 19 from respiratory swabs. So, medical practitioners resorted to other tests such as X−ray, and Computed Tomography (CT) scan [6, 11]. Based on the fact that COVID− 19 attacks the epithelial cells that line our respiratory tract, X−ray and CT images can be used by the medical practitioner to diagnose infected lungs. Furthermore, most of the emergency clinics have X-beam imaging machines; it could be conceivable to utilize X−Ray to test for COVID− 19 without the devoted test kits. Test images can provide fast detection of COVID− 19, and consequently contribute to control the spread of the disease. X−ray and CT images are frequently used by the clinicians to diagnose pneumonia, respiratory tract infection, enlarged lymph nodes, and abscesses. Therefore, CT scans play an essential role in the diagnosis of COVID− 19 as an advanced imaging modality.

Due to this crisis, computer-aided detection/diagnosis must be employed to help radiologists in the diagnosis process to mitigate the overcapacity of a large number of COVID− 19 patients. The historical conception of image diagnostic systems has been comprehensively explored through several approaches ranging from feature extraction to feature learning. Machine learning researchers and computer scientists play a vital role in the era when COVID− 19 spreads all over the World. One of the breakthroughs of AI is deep learning. It extracts the detailed features from the images. Deep Learning is a combination of machine learning methods that mainly focused on the automatic feature extraction and classification from images.

One of the deep learning techniques is the Convolutional neural network (CNN). Automated features extraction by CNN models is more robust and discriminative than traditional features extraction models. CNN models have the capability of parameter sharing and local connectivity. The layered architecture of CNN is used to extract the features from different layers. CNN initial layers contain generic features (lines, edges, blobs etc.) while later layers contain specific features.

Thanks to Transfer learning, it uses the gained knowledge that solves one problem and applied them to solve different related problems by using a trained model to learn a different set of data. The pre−trained CNNs architectures: AlexNet, GoogleNet, VGG16, VGG19, ResNet− 18, ResNet− 50, ResNet− 101,SENet and ResNet−Inception−v2 consist of 1000 classes, 1.28 million training images, tested on 100 k test images and evaluated on 50 k validation images. They are challenging the accuracy of a human with the best-given results. The networks take an image as an input and produce the object label in the image as an output as well as the probabilities of the object categories. These techniques can train the weights of networks on large datasets as well as fine-tuning the weights of pre−trained networks on small datasets. Due to the small COVID− 19 dataset available, the pre−trained neural networks can be used for diagnosis of coronavirus. However, these techniques applied to chest X−ray and CT images are very limited until now. Hence, the main idea of this paper is to present an automated tool to detect COVID− 19 in both chest X−ray and CT images by transferring the knowledge of the pre−trained deep learning architectures.

In this study, the efficacy of the pre−trained CNNs architecture of ResNet50 is exploited to discriminate COVID− 19 from Non−COVID − 19 X−ray and CT lung scan. We augmented the ResNet50 architecture by inserting three new layers at the end of its architecture to extract the best discriminative features. The three layers are ′Conv′, ′Batch_Normaliz′ and ′Activation_Relu′. Therefore, the feature extraction capability of ResNet50 is enhanced. Experimental results approved that the inserted layers improved the accuracy compared to the original ResNet50. Many experiments are performed to measure the performance and productivity of the proposed model on COVID− 19 chest X−ray and CT scan images. The proposed model reported an accuracy of 97.7% by applying it on the CT dataset and achieved 97.1% by applying it on the X−ray dataset. Comparative results demonstrate that the proposed model is superior to the competitive approaches. In summary, this study provides the following three main contributions: The rest of the work is organized as follows: some recent previous work is discussed in Section 2. Section 3 focuses on the available Covid− 19 datasets, section 4 is dedicated to the proposed model. The experimental results are reported in Section 5. Finally, Section 6 demonstrates the conclusion.

A new reliable algorithm is presented in this work to classify the COVID patients, called COVID-ResNet53. The developed algorithm contains a new CNN architecture which is a modified version of ResNet50 model and we exploit the transfer learning benefits for the modified model.

Comprehensive analysis of eight deep learning models used in the context of COVID patient’s classification is presented, in terms of accuracy.

We achieve the state-of-the-art COVID classification results on COVID19 chest X-Ray and CT scan images dataset.

Literature review

In the last few months, the machine learning-based techniques for identification and detection of COVID− 19 from radiological imaging have been performed. This section reviews recent literature of COVID− 19 disease detection using chest X−ray and CT images. Data is the first step for developing any diagnostic tool. At the emergence of Covid19, there is no collection of COVID− 19 chest X−rays or CT scans designed to be used for computational analysis. The author in [6] released a dataset describe the public database of pneumonia cases with chest X−ray or CT images, specifically COVID- 19 cases as well as MERS, SARS, and ARDS. Data is collected from public sources. Zhao et al. [22] has built a COVID-CT dataset that consists of 349 CT images for COVID− 19 positive and 397 CT images for COVID− 19 negative. This dataset is publicly open for research work.

The work in [11] presents a novel COVID− 19 detecting methodology based on multi-level thresholding and SVM for X−ray images where the average sensitivity, specificity, and accuracy of the lung diagnosis were 95.76%, 99.7%, and 97.48%, respectively. The author in [12] proposed the machine learning-based classification of deep features extracted from chest X−ray images of COVID− 19 and Pneumonia patients using ResNet152. The model achieved an accuracy of 97.3% on Random Forest and 97.7% using XGBoost predictive classifiers.

Two publicly available datasets are used: Chest X−ray Images (Pneumonia) [19] and COVID− 19 public dataset from Italy [5]. A total of 5840 images are used. In [14], the DarkNet architecture was utilized as a classifier for you only look once (YOLO) real-time object detection system. The system implemented by 17 convolutional layers with different filtering on each layer. The object detection accuracy was 98.08% for binary classes and 87.02% for multi-class cases for chest X−ray images on the same dataset in [6]. In [16], SVM is evaluated against 13 number of CNN models for detection of COVID− 19 from X−ray images using deep features resulted from them. The SVM, combined with deep feature of ResNet50, produced the best results compared to other models. ResNet50 plus SVM resulted in accuracy, sensitivity, FPR and F1 score of 95.33%, 95.33%, 2.33% and 95.34%, respectively, for detection of COVID− 19. The author collected the dataset from Kaggle repository (Kaggle, 2020) and the dataset in [6]. In [1], the author developed a deep CNN, called DeTraC (Decompose, Transfer, and Compose), for the classification of COVID− 19 chest X−ray images. DeTraC can resolve irregularities in the image dataset using decomposition mechanism by investigating its class boundaries. High accuracy of 95.12% (with a sensitivity of 97.91%, and a specificity of 91.87%) was achieved by DeTraC.

A combination of two datasets was utilized. Lung CT scans (coronal view) was used in [15] to extract COVID− 19 infected sections from images. A series of procedures, such as threshold filtering, multi-thresholding, segmentation and area feature extraction are implemented to detect the COVID− 19 pneumonia infection from the considered CT. Paper [17] introduced a tuned CNN system to distinguish the COVID− 19 infected patients as infected (+ve) or not (−ve) with a good accuracy rate. In [20], a multi-view fusion model using deep learning network to screen patients with COVID− 19 is trained using CT images with the maximum lung regions in axial, coronal and sagittal views. AUC, accuracy, sensitivity and specificity of 0.819, 0.760, 0.811 and 0.615, respectively, are achieved. In [21], the author presented an evaluation using the chest CTSI of axial-view and considered the images of 102 volunteers (53 men and 49 women with age group of 15 − 79 years). This work also implemented visual inspection based detection. Implemented detection helped to attain Area-Under-Curve (AUC) of 89.2%, Sensitivity of 83.3% and Specificity of 94%. The work in [9] examined the information of 81 patients. Among the 81 patients, 30 patients are evaluated with RT-PCR and 51 patients are evaluated with both CTSI and RT-PCR. This work combined the laboratory analysis (RT−PCR) along with the imaging procedure based on the CTSI. The sensitivity of CTSI based detection of COVID− 19 infection was 98% compared to to RT-PCR sensitivity of 71%(p<.001). The work in [3] examined 219 patient’s information using the radiologist team of China and U.S and their findings are discussed.

This work utilized the RT-PCR and CTSI for the examination. In addition,it presented a classification task using 219 COVID− 19 along with 205 Non−Covid− 19 pneumonia case. The experimental results demonstrated a better sensitivity (93%) and specificity (100%) for both the China and U.S radiologist. Asnaoui and Chawki [8] used convolutional neural network (CNN) models for binary classification of COVID− 19 in CT images. They used pre−trained models namely VGG16, ResNet50 Inception−ResNet−V2, VGG19, Xception, Inception−V3, ResNet50,and MobileNet−V2 for classification. 96% classification accuracy was achieved from MobileNet−V2, Inception−ResNet−V2, and ResNet50. Wang, Kang, et al. [18] used InceptionNet to detect the abnormalities associated with COVID− 19 in lung CT scan images. InspectionNet model was tested on 1065 CT images and 325 infected persons were identified with accuracy of 85.20%. Xu et al. [4] utilized 3D CNN models to discriminate the corona virus infection from Influenza−A viral pneumonia in CT scan images. ResNet was used to extract the features. The accuracy scored from CNN model was 86.70%. Gozes, Frid-Adar, Sagie, et al. [10] used 2D and 3D deep learning models to distinguish the corona virus infection in COVID− 19 patients. CT features are utilized for COVID− 19 classification. A classification accuracy of 99.60% was achieved. Li et al. [13] presented COVID− 19 detection neural network (COVNet) to extract the features from chest CT images for detection of corona virus infection in patients. COVNet was trained over 4357 chest CT images of 3322 patients. The accuracy obtained from COVNet was 95%.

Dataset

One of the prerequisites in image recognition and computer vision research is finding appropriate dataset. Massive amount of training images is one of the essential requirements of deep learning, however; large medical imaging data is an obstacle in the success of deep learning. In the beginning, COVID− 19 lung images datasets were limited as it was a recent emerging disease. Dr. Joseph Cohen established the public GitHub repository [6] where X−ray and CT images of COVID− 19 are collected.

Recently, Walid El-Shafai and Fathi E. Abd El-Samie [7] collected a dataset from many dataset sources on the internet, including Dr. Joseph Cohen dataset and other datasets. The size of the dataset is increased with different augmentation techniques to generate about 17100 X−ray and CT images. This COVID− 19 dataset [7] consists of Non−Covid and COVID cases of both X−ray and CT images. The dataset divided into two main parts, one part for the X−ray images, which includes two separate sub-folders of 5500 Non−Covid images and 4045 COVID images. The other part contains the CT images with 2628 images for Non−Covid and 5427 images for COVID.

Proposed model

The proposed framework for efficient prediction of COVID− 19 for the chest X−ray and CT images present a new CNN model, and it depends on one of the transfer learned and fine-tuned deep pre−trained CNNs models to extract deep features. As depicted in Fig. 1, we introduced a new CNN architecture based on a modification of the ResNet50 architecture and makes use of the pre−trained model of ResNet50. The ResNet50 model architecture is enhanced to suit the Covid19 dataset by adding some layers at its end. X−ray and CT images are taken with a low resolution which may have a variable height to width ratio. Therefore, training and testing dataset images are resized to 224× 224 ×3 for a similar course of action in the developed model architecture.

Fig. 1

The ResNet50 model architecture before and after modifications

ResNet is known to be a better deep learning architecture as it is relatively easy to be optimized and can attain higher accuracy. Furthermore, there is always a problem of vanishing gradient, which is resolved using the skip connections in the network. As the number of layers in the deep network architecture increases, the time complexity of the network increases. This complexity can be reduced by utilizing a bottleneck design. As a consequence, we preferred ResNet50 pre−trained model to build up our framework and excluded other pre−trained networks that have bigger number of layers. A detailed description of the architecture is explained below.

Some modifications are applied to the ResNet50 architecture to reach efficient performance for predicting Covid19. First, we altered the last three layers (fully connected, softmax and classification layers) of the pre−trained ResNet50 architecture in order to adapt them to our classification task. The fully connected layer in the original pre−trained networks are replaced by another fully connected layer, in which the output size represents the two classes in our case, Covid and Non−Covid. Next, three layers, namely, ′Conv′, ′Batch_Normaliz′ and ′Activation_Relu′ are added to the ResNet50 architecture as shown in Fig. 1, to automatically extract the robust features in chest X−ray and CT images. These layers are convolution layer followed by batch normalization layer followed by an activation layer. The addition of the three layers is done as in the following steps Figure 1a depicts the ResNet50 architecture before modifications and Fig. 1b shows the modified architecture after injecting the new layers. The input images are now passed through this modified network to obtain features for each image in the dataset and then classified either to Covid or Non−Covid using the network classifier. The proposed model was trained for the classification of Covid and Non−Covid.

The ′activation_49_relu′ layer is disconnected from ′avg_pool′ layer and connected instead to the newly added ′Conv′ layer.

The newly added ′activation_relu′ layer is connected to ′avg_pool′ layer.

The ′avg_pool′ layer is followed by the last three newly added layers ′fully_connected′, ′softmax′ and ′ClassificationLayer′

Experimental results

In this section, we analyze the effectiveness of the suggested scheme in light of conducted experiments. Our experiments are conducted on the dataset in [7] and applied on each subset, i.e., X−ray and CT subsets, separately. CNN architecture, coupled with new tuning strategy, is deployed by randomly splitting the dataset into 80% in training set and 20% in the testing set. The new injected convolution layer used a filter of size (3 × 3 × 256) when the system is applied on the CT and X−Ray datasets.

Recently, there are some authors focused on testing their algorithm on the only chest X-ray images [5, 12, 14, 19], and [1], and the others using only the CT images [15, 17], and [20], while in this work we tested our algorithm on both the chest X−ray and CT images. Hence the proposed method is compatible with different dataset which nominated it to be more applicable compared to the state of the art techniques. In our conducted experiments, 5427 confirmed COVID− 19 CT images and 2627 confirmed Non-COVID CT images are used. While, 4044 confirmed COVID-19 X−ray images and 5493 confirmed Non-COVID X−ray images are used from the dataset in [7]. The dataset is randomly splitted in to 80% for training and 20% for testing.

We have implemented the proposed system for COVID− 19 diagnosis using Matlab R2020a programming language with a processor of Intel Core i5 and RAM of 6 GB running on Windows 10. The Adam optimizer is utilized for weight updates along with 1e− 4 learning rate and five epochs.

The classification model’s usefulness and productivity were measured using the well-known metrics of accuracy, sensitivity, specificity and precision. Precision is the calculation of the model’s correct predictions all over all predictions. Four major outcomes: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) are necessary to compute the evaluation criteria according to the following equations

A well-known evaluation measure, i.e. accuracy is used to evaluate the effectiveness of the proposed system. Table 1 shows that our proposed model scores 97.1%, 98.9%, 95.7% and 94.5% for the accuracy, Sensitivity, Specificity and Precision measures, respectively, using X−ray dataset. Table 2 shows that the proposed model achieves 97.7%, 98.7%, 95.6% and 97.9% for the accuracy, Sensitivity, Specificity and Precision measures, respectively, using the CT dataset. However, the accuracies achieved by ResNet50 and Resnet101 are 96.8%, 96.8% for X−ray dataset and 96.9%, 95.3% CT dataset, respectively.

Table 1

Results of applying the proposed algorithm on the CT dataset

Model	Accuracy	Sensitivity	Specificity	Precision
Proposed model	97.7%	98.7%	95.6%	97.9%
Resnet50 model	96.9%	96.7%	97.3%	98.7%
Resnet101 model	95.3%	93.8%	98.3%	99.1%
GoogleNet model	96.2%	97.6%	93.3%	96.8%
Alexnet model	93.6%	93.6%	93.5%	96.8%
DenseNet201 model	96.8%	99.4%	91.4%	96%
VGG16 model	88.8%	92.7%	86%	83%
VGG19 model [2]	87.5%	97%	68%	86.2%
InceptionV3 model	97.4%	97.3%	97.5%	98.8%
Resnet50+SVM [16]	97.5%	97.3%	97.9%	99%

Table 2

Results of applying the proposed algorithm on the X−ray dataset

Model	Accuracy	Sensitivity	Specificity	Precision
Proposed Model	97.1%	98.9%	95.7%	94.5%
Resnet50 Model	96.8%	96%	97.4%	96.4%
Resnet101 Model	96.8%	96%	97.4%	96.4%
Googlenet Model	96.5%	96.8%	96.4%	95.1%
Alexnet Model	94.6%	92.7%	96%	94.5%
Densenet201 Model	96.5%	98.9%	94.7%	93.2%
Vgg16 Model	91.8%	89.9%	93.2%	90.6%
Vgg19 Model [2]	88.8%	92.7%	86%	83%
Inceptionv3 Model	96.6%	97.8%	95.7%	94.4%
Resnet50+Svm [16]	95.6%	95.8%	95.5%	93.9%

Figure 2 displays the Accuracy and loss curves for the proposed model using the X−ray and CT datasets. Figure 3 depicts the confusion matrix for the proposed model using the X−ray and CT datasets. The confusion matrix gives an understanding of the proposed methodology and its potential for detailed classification. ResNet101 is 101 layers deep, and ResNet50 is 50 layers deep, so ResNet101 consumes, approximately, twice the time of ResNet50. From the results of Tables 1 and 2, we can conclude that the proposed model accuracy superior to the accuracy of both Resnet101 and Resnet50. Therefore, the suggested scheme is not only efficient in performance, but also it has low time consumption.

Fig. 2

Accuracy and loss curves that are resulted from the proposed model

Fig. 3

Confusion Matrix of the proposed model

The proposed method used pre-trained CNN models to obtain the best performance for the detection of COVID-19. We evaluated the performance results of deep feature extraction based on the ResNet50, ReseNet101, GoogleNet, AlexNet, DenseNet201, VGG16, VGG19, InceptionV3, and (ResNet50+SVM) models. To show the superiority of our model, we compared our proposed model with the previous mentioned powerful pre−trained architectures of CNN. Utilizing these CNN models without its own classification layer enables us to extract features for our target task based on the knowledge of source task.

Tables 1 and 2 show the accuracy, specificity, sensitivity and precision measures for each model. It is observed that the accuracy of our proposed model outperforms the ones of the other 8 models. In addition the proposed method increased the detection accuracy in case of CT and X-ray, from 97.5%, 95.6% to 97.7%, 97.1%, respectively, compared to the sate of the art method in [16] (ResNet50+SVM) and in [2]. According to the results, the proposed algorithm achieved the highest classification accuracy in case of X-ray and CT datasets, i.e. 97.1% and 97.7%, respectively.It is asserted that the accuracy of 97.1% for the X-ray dataset achievable from the proposed algorithm is highly encouraging compared to 88.8% presented from the VGG19 model [2]. Moreover, the proposed method increases the detection accuracy by 10.4% from 87.5% to 97.7% compared to the accuracy presented by the VGG-19 model in the CT dataset. To prove the robustness of the proposed method, the data set is divided into five folds namely, Fold1, Fold2, Fold3, Fold4 and Fold5. A five−fold cross−validation strategy is employed, i.e. five experiments are conducted, in each experiment, four folds are used for training and one for testing. Table 3 shows the results of cross-validation achieved from applying the proposed model on both the CT and X−ray datasets.

Table 3

Results of cross-validation on the CT and X−ray datasets

Evaluation measure	Testing data set	Fold1	Fold2	Fold3	Fold4	Fold5
Accuracy %	CT	98	96.6	97.1	96.5	96.6
	X-ray	96.6	96.1	96.2	96.6	95.1
Sensitivity %	CT	99.4	99	97.9	97.4	98.2
	X-ray	98.4	97.3	98.1	96.2	98.9
Specificity %	CT	95	91.8	95.6	94.5	93.1
	X-ray	95.4	95.2	94.7	96.9	92.4
Precission %	CT	97.6	96.2	97.9	97.3	96.7
	X-ray	94	93.7	93.2	95.8	90.5
Average accuracy %	CT	97
	X-ray	96.12

Conclusion

In this paper, a novel deep transfer learning model is developed for COVID− 19 disease detection based on convolutional neural network and the pre−trained ResNet50 model. Inserting more three layers to the ResNet50 allows extracting more robust features. An improvement in the accuracy has been gained by adding the three proposed layers to the ResNet50 model. An accuracy of 97.7% is obtained by applying the proposed scheme on the CT dataset, and an accuracy of 97.1% is scored by applying it on the X−ray dataset. Experimental results proved the success of our model as a tool can be utilized to diagnose and discriminate Covid− 19 from Non−Covid− 19 patients. Comparative analyses revealed that the suggested scheme performs significantly efficient as compared to other well-known deep transfer learning models such as, ResNet50, ReseNet101, GoogleNet, AlexNet, DenseNet201, VGG16, VGG19, InceptionV3, and (ResNet50+ SVM) models. Moreover, the proposed algorithm accuracy outperforms VGG19 model by 10.4% and 8.5% in CT and X-ray datasets, respectively. As CT scans facility is available in most of the medical institutions; thus, the proposed model can improve the COVID− 19 testing process. Therefore, the proposed model can act as an alternative to various COVID− 19 testing kits.