A deep learning-based COVID-19 classification from chest X-ray image: case study.

The novel corona virus disease (COVID-19) is a pandemic disease that is currently affecting over 200 countries around the world and more than 6 millions of people died in last 2 years. Early detection of COVID-19 can mitigate and control its spread. Reverse transcription polymerase chain reaction (RT-CPR), Chest X-ray (CXR) scan, and Computerized Tomography (CT) scan are used to identify the COVID-19. Chest X-ray image analysis is relatively time efficient than compared with RT-CPR and CT scan. Its cost-effectiveness make it a good choice for COVID-19 Classification. We propose a deep learning based Convolutional Neural Network model for detection of COVID-19 from CXR. Chest X-ray images are collected from various sources dataset for training with augmentation and evaluating our model, which is widely used for COVID-19 detection and diagnosis. A Deep Convolutional neural network (CNN) based model for analysis of COVID-19 with data augmentation is proposed, which uses the patient's chest X-ray images for the diagnosis of COVID-19 with an aim to help the physicians to assist the diagnostic process among high workload conditions. The overall accuracy of 93 percent for COVID-19 Classification is achieved by choosing best optimizer.

Introduction

The Corona Virus Disease 2019 (COVID-19) pandemic, which is one of the worst pandemics in the history of humanity caused by the SARS-CoV-2 virus, is posing significant challenges to healthcare systems around the world and forcing physicians to make quick clinical decisions under pressure. Now there is no proper medicine for prevention or cure of this infectious disease. The World Health Organization declared this COVID-19 disease as pandemic as it spreads quickly all over the countries. COVID-19 primarily affects the lungs and creates breathing problems. We can see ground-glass opacity in the chest X-ray due to fibrosis in the lungs when the corona virus enters into the lungs. Millions of individuals infected worldwide with the novel corona virus (COVID-19), and the number of cases are growing at an exponential rate necessitates the need for a rapid diagnostic system. RT-PCR testing kits are expensive and time consuming process. The use of chest X-ray (CXR) imaging in the diagnosis and monitoring of COVID-19 patients is becoming more common and best practise because of its minimal time and cost effectiveness [1].

COVID-19 severely affects respiratory systems of human which was first discovered in Wuhan, China, in December of 2019. According to the WHO, there have been 440 millions confirmed COVID-19 cases worldwide as of February 28, 2022, with 6 millions death reports. We observe ground-glass opacity in the chest X-ray when the virus enters the lungs due to the presence of fibrosis in the lungs. Artificial intelligence techniques can be used to detect the presence and degree of illness based on the major difference between X-ray images of an infected and non-infected person [2]. The majority of existing research on COVID-19 detection focuses on pre-trained models and standard CNN architecture rather than customized architecture [3]. The early detection of COVID-19 heavily relies on the analysis of lung images. COVID-19 can be diagnosed early with chest computer tomography (CT) imaging and X-rays using radio-logical techniques before the disease spreads to the lungs and causes harm. The disease is easily transmitted from one person to another via water droplets coming of the afflicted person while sneezing, coughing, or exhaling. Almost every country is working very hard to slow down and stop the spread of COVID-19 illness [4].

The polymerase chain reaction (PCR) test is used to identify the presence of infection antibodies in the standard COVID-19 test. Unfortunately, these tests require a high level of precision and are time consuming, with a high risk of false negatives. It goes without saying that a wrong conclusion of the virus's absence might have disastrous consequences. Furthermore, many countries lack the financial and human resources required to deploy COVID-19 tests and testing centers on a wide scale. Chest X-ray image analysis is being considered as an alternative to the PCR test to avoid such difficulties [5]. COVID-19 infected patients' chest X-ray test images are effectively diagnosed using detection models that have been already trained for their better performance [6]. Deep learning Models can be used to find the presence or absence of the COVID-19 infection in lungs rapidly [7]. These intelligence based deep neural network learning models have proved remarkable performance in image-related tasks, including variety of radio-logical environments. They offer a lot of potential in terms of supporting COVID-19 diagnosis, but they require a lot of training data and high end processing computers. Images of both classes differs in task-specific properties when training neural networks for image classification.

There are many classification algorithms widely available based on CNN and it is impossible to ascertain one is superior to the others. LeNet-5, AlexNet, VGG-16, Inception-v1, Inception-v3, ResNet-50, Xception, Inception-v4, Inception-ResNets, and ResNeXt-50 are some of the CNN based top 10 architectures [8]. Transfer learning is a deep learning technique in which a model developed for one task is used as the foundation for another task's model. The hybrid deep learning model is a combination of two CNN models. For example, ResMobNet is the hybrid deep learning model which is a combination of ResNet50 and MobileNet [9]. Predictions of hybrid models are better than individual model as they incorporate texture and morphological features [10, 11]. It is dependent on the application and the nature of the data set provided [4].

Every learning model needs sufficient dataset for good prediction accuracy. If it is not feasible to build a large labeled database for every class (COVID-19 and normal) then prediction goes wrong. Thus, the bias in small datasets and the lack of representative training and tuning data impair the performance of deep learning models. A simple way to deal with these overfitting challenges data augmentation technique is used [12]. Several experiments were conducted using chest X-ray for COVID-19 classification and the results demonstrate that the data augmentation strategy improves the performance of COVID-19 classification [13]. Hyperparameters are the variables which determines the Convolutional neural network structure (and its values are set before training. Number of Hidden Layers and units, weight Initialization, dropout, and activation functions are hyperparameters related to Network structure. Learning Rate, Momentum, Number of epochs, and Batch size are hyperparameters related to Training [14].

The rest of this research article is organized in the following manner. In second section, the Literature review of chest X-ray images for COVID-19 classification and prediction techniques are discussed. The materials and methods, including the proposed methodologies, as well as classification of COVID images are introduced in third section. Then, experimental results are presented and findings are discussed in fourth Section. Finally, fifth Section concludes the article and presents some future work opportunities.

Literature review

Various machine learning and deep learning models are specifically designed by researchers for detecting the corona virus disease from COVID-19 X-ray dataset. Machine learning methods come in handy in critical tasks and does not support automatic feature extraction from images. Henceforth, deep neural network model with automatic feature extraction by computer vision technique can be used to detect COVID-19. The use of a Convolutional neural network in the diagnosis of COVD-19 from digitized chest X-ray image is one of the most common and effective methods and several reviews have been conducted to highlight the recent contributions to the detection of COVID-19.

In recent years, medical image processing with deep learning system has proved to be an effective tool for classification and diagnosis of Chest X-ray for COVID-19. For COVID-19 classification, Keidar et al. achieved 90.3 percent of accuracy with augmentation and normalization of CXR images using ReNet50¸ ReNet152, and vgg16 [1]. Deep learning based automated COVID‑19 Screening Chest X‑ray Classification is designed by Shelke et al. can further classify mild, medium, and severe COVID‑19 [2]. Convolutional neural network architectures like ResNet101, Xception, InceptionV3, InceptionResNetV2, VGG16, and VGG19 are used for COVID-19 classification from chest x-ray images [3]. Hammoudi et al. developed hierarchical classification of COVID-19 from pneumonia viral classification with comparative study for performance evaluation and exceeded 84% of average accuracy [7]. Decompose, Transfer, and Compose (DeTraC) deep Convolutional neural network is developed by Abbas et al. for COVID-19 and SARS classification from CXR and CT images with high accuracy of 93 percent [6].

To classify the chest X-ray image as healthy and COVID-19, Rekha st al. used a transfer learning model with features extracted using XGBoost with CNN [4]. Guefrechi et al. designed effective Deep learning and transfer learning-based detection of COVID-19 from chest X-ray images with ResNet50, InceptionV3, and VGG16. They efficiently trained the model through a small image dataset and fine-tuned models to give high performance in the classification of COVID-19 and pneumonia [5]. Bozkurt et al. Developed deep and handcrafted features-based framework for diagnosis of COVID-19 from chest X-ray images with k-nearest neighbors (kNN), Naive Bayes (NB), and MLPClassifier [15]. Elgendi et al. focused geometric transformation perspective of image augmentation in deep learning networks for detecting COVID-19 [16]. Sousa et al. used CNN-COVID to classify COVID-19 from chest X-ray images and they proved that Chest X-ray model done at low cost and results are quickly obtained over computed tomography [17].

Horry el al. developed a COVID-19 Detection system through transfer learning using multi-modal imaging data with VGG16, VGG19, Xception, Inception, InceptionResnet, Densenet, and Resnet models and produced 86 percent of accuracy for chest X-ray image COVID-19 classification [18]. Momeny et al. improved the generalizability of deep CNN for the detection of COVID-19 with data augmentation and produced 78 percent of accuracy [19]. Ucar et al. designed a system for automated detection of Covid-19 disease using deep fused features from chest radiography images with 92 percent of accuracy [20]. Frid-Adar et al. developed a model that simultaneously detects COVID-19 from both CXR and CT images [21]. Rahaman et al. identified COVID-19 from chest X-ray images using deep learning with transfer learning approach and achieved the classification accuracy of 89 percent using VGG19.

Pereira et al. identified COVID-19 from chest X-ray images and achieved the accuracy of 89 percent for the COVID-19 classification using resampling algorithm to re-balance the classes distribution [22]. Apostolopoulos et al. designed new CNN model from scratch and outperforms to distinguishing the X-ray images between the Covid-19 and non-Covid-19 with classification accuracy of 88 percent [23]. Khan et al. developed CoroNet which is a deep neural network for detection and diagnosis of COVID-19 from chest X-ray images with overall accuracy of 90 percent [24]. Rahimzadeh et al. modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2. The overall accuracy of their model for multi-class classification is 91 percent [25].

Methodology

Most of the researchers used Convolutional Neural Networks for COVID-19 Classification from X-Ray Image dataset. We developed COVID-19 Classification from X-Ray Image dataset using Data Augmentation and Hyper Parameter Tuning. The first phase deals acquisition and pre-process of COVID-19 image dataset, second phase deals the data augmentation. Third phase train the model and fourth phase test the model. Work flow diagram of proposed work is represented in Fig. 1.

Fig. 1

Workflow diagram of proposed work

Data acquisition

Data or Image acquisition is the process of collecting images from an external data sources for subsequent processing. The Cohen/IEEE8023 dataset is the most widely used COVID-19 classification. COVID-19 positive and negative images are collected from Cohen/IEEE8023 dataset which is cited by the majority of COVID-19 researchers. It was one of the first datasets released in April 2020 and was easily accessed by cloning Github repository.

Our dataset consists of 3000 COVID-19 positive images and 3000 normal (COVID-19 negative) images selected form Cohen/IEEE8023 dataset. The images in varying sizes which cannot effectively processed by the algorithm. As a result, all of the images have been scaled to 224 × 224 pixels. Furthermore, all RGB images are normalized by dividing them by 255, bringing all image intensity levels into a single range of [0 to 1]. Image pre-processing is carried out by image re-scaling and normalization. The dataset containing COVID-19 radiography images are acquired from popular Cohen/IEEE8023 dataset which is the real data collected recently. The database includes chest X-ray images for COVID-19 positive cases along with normal images. Figure 2 shows a original sample COVID-19 image and non-COVID-19 normal image.

Fig. 2

Chest X-ray augmented images of COVID-19 and normal

Data augmentation

COVID-19 chest X-ray datasets are obtained from public donations, however they are insufficient for complex CNN training to get more accuracy. This research adopts an augmentation technique without any loss to increase as well as maintain the dataset. To increase the performance of deep learning model we need more data, so augmentation technique is used to increase the dataset and performance accuracy [12]. Ten times the dataset is increased by augmentation using the following algorithm. The parameters of augmentation are, rotation range of 15 degree, shear range of 0.2, height shift range of 0.2, width shift range of 0.2, zoom range of 0.2, brightness range of 0.2 and fill mode as nearest. The images in the dataset are of different sizes and therefore all the images are converted into a size of 224 × 224 × 3 pixels. The following Algorithm is applied for image dataset augmentation.

Transformations such as image rotation, shear, zoom, brightness and re-scale are used for augmentation process. Generated images of COVID-19 and normal using augmentation process are shown in Fig. 2. During image acquisition these properties play important role for successful classification. They increase model robustness and generalizability by expanding the dataset by providing a diverse set of images and it take care of typical fluctuation in CXR image acquisition [1].

The statistics of chest X-ray images included in this study are shown in Table 1. The chest X-ray scan imaging dataset constructed from Cohen/IEEE8023 dataset which consists of a total of 6000 images with balanced dataset of 50% positive COVID-19 and 50% negative COVID-19(normal).

Table 1

Statistics of chest X-ray images

Dataset	Original images	Train images	Test images	Augmented images for training	Augmented images for validation
COVID-19	3000	2400	600	19,200	4800
Normal	3000	2400	600	19,200	4800
Total	6000	4800	1200	38,400	9600

Training, validation, and testing split

To validate the model constructed, the training dataset was split with 80% for constructing the model and the remaining 20% for validation of the constructed model. The test dataset was not shown to the model during the training process and was used to verify the actual performance of the constructed model. The 80% training data was further divided into five subsets to perform fivefold cross validation. Without lossy compression, chest X-ray scan images were used in either the test set or the train set, but not both.

To validate the built model, dataset was split into 80% for training and 20% for testing. The test dataset was not provided to the model during training and was used to verify the generated model's real performance with metrics. To do fivefold cross validation, the 80 percent training data were further separated into five subgroups and one subgroup is used to validate the training model.

Construction of convolutional neural net

The CNN model was built with a sequence of following layers:

Input layer: The image given to the input layer is of size 224 × 224 × 3, where 224 × 224 represents the dimensions of image and ‘3’ represents the RGB channels.

Convolutional layer

The CNN algorithm extracts features from the input automatically with the help of a filter / kernel. The filter of a particular size M × M was designed to span over the entire image. Dot product value is calculated between the pixels of the image covered by the filter and the filter itself at each stride and the multiplied values are added together to produce a single value. When working with a single filter, this method produced a 2D output termed a feature map and this feature map is fed to other layers to learn several other features of the input image. The Rectified Linear Unit (ReLU) activation function is used in order to transform the input received in a node non-linearly.

Pooling layer

This layer is added after the Convolutional layer to reduce the variation in the feature map that can occur when the input is significantly shifted or rotated. The maximum value covered by the pooling filter in the feature map was used in the research, which was done using max pooling. The use of a pooling layer further reduces the amount of parameters and computations necessary to train the model. After pooling, a dropout layer was added to prevent overfitting during training. The Pooling Layer also serves as a bridge between the Convolutional layer and the fully connected layer.

Dense/fully connected layer

The pooling layer's output was flattened to a one dimensional vector and sent into the fully connected layer. This layer adjusts the weights so that it can forecast the probability of each class to which the input belongs. Because probabilities of each class must be found, Softmax activation was used at the last dense layer [4].

Hyperparameter tuning

Hyperparameter tuning was performed on CNN model to improve the performance accuracy. The tuning process was performed using the six hyperparameters: number of neurons, number of epochs, learning rate, dropout rate in the dropout layer, batch size and gradient update optimization algorithms. The values filled in the Table 3 are standard values used by several researchers.

Table 3

Hyperparameter values

S. no	Hyperparameter	Values
1	Epochs	50, 100, 200, 300 and 400
2	Batch size	8, 16, 32 and 64
3	Dropout factor (%)	10, 15, 20, 25, 30 and 35,
4	Learning rate	0.00001 and 0.000001
5	Gradient optimizers	Adam, SGD, RMSProp, Adadelta, Adagrade, Adamax, Nadam, and FTRL

In our model, learning rate is 0.000001 for training and the best epoch was chosen to be 300 with the batch size of 32 which yielded better performance. RMSProp was the best gradient optimizer in most of the versions, but in our versions of CNN, ADAM yielded better results. It was found that dropout probabilities of 25% and 30% resonated well in most of the versions of CNN models. In order to avoid over-fitting and increase the impact of generalization, a dropout of 25% has been used in the CNN model.

Model training and validation

The proposed CNN model is trained and evaluated using augmented chest X-ray images to analyse with pre trained models. The images are distributed in 4:1 ratio across the training and testing datasets with fivefold cross validation. The training and testing images have an input size of 224 × 224x3, and the experiment uses a batch size of 32. The models were trained for 50 to 400 epochs with early-stopping configuration. The model is evaluated on various classification metrics that include accuracy, sensitivity, specificity, recall, F1-score, and confusion matrix. To improve the proposed model ability to generalize well, data set are artificially expanded by generating images in the dataset by leveraging the augmentation technique. Table 3 includes the learning process recorded by the CNN model with respect to the number of epochs, model loss, model accuracy curve, respectively.

Comparison with trained models

The performance of the pre-trained model with respect to the training and validation accuracy curves is shown in Table 2. The overall validation accuracy of the proposed CNN model and the pre-trained model is shown in the Table 4.

Table 2

Summary of CNN model

Layer (type)	Output Shape	Number of parameters
conv2d (Conv2D)	(254, 254, 32)	896
conv2d_1 (Conv2D)	(252, 252, 64)	18,496
max_pooling2d (MaxPooling2D)	(126, 126, 64)	0
dropout (Dropout)	(126, 126, 64)	0
conv2d_2 (Conv2D)	(124, 124, 64)	36,928
max_pooling2d_1 (MaxPooling2D)	(62, 62, 64)	0
dropout_1 (Dropout)	(62, 62, 64)	0
conv2d_3 (Conv2D)	(60, 60, 128)	73,856
max_pooling2d_2 (MaxPooling2D)	(30, 30, 128)	0
dropout_2 (Dropout)	(30, 30, 128)	0
flatten (Flatten)	(115,200)	0
dense (Dense)	(64)	7,372,864
dropout_3 (Dropout)	(64)	0
dense_1 (Dense)	(1)	65
Total trainable parameters:	7,503,105

Table 4

Accuracy for different optimize

Optimizer	Epochs
	50	100	200	300	400
Adam	79.82	84.40	88.65	93.05	92.85
SGD	78.75	83.50	87.30	91.71	91.95
RMSProp	73.55	82.23	86.55	88.72	88.79
Adadelta	76.68	81.50	84.64	86.67	86.59
Adagrade	77.75	81.29	83.75	85.56	85.61
Adamax	75.72	79.46	81.15	83.25	83.37
Nadam	73.55	78.95	83.86	84.57	84.65
FTRL	75.63	79.92	81.52	82.76	82.85

Best value is in bold

Results and discussion

The experiments ware conducted and the accuracy values are tabulated in Table 3 for the epoch values 50, 100, 200, 300 and 400 with different optimizer. Adam optimizer with 300 epochs gives the accuracy of 93.05 percent. The Bar graph drawn for each optimizer in Fig. 3 is the visual representation of the Table 3. For the model evaluation we used accuracy, sensitivity, specificity, recall, F1 score and area under the curve (AUC) for receiver operating characteristic (ROC).

Fig. 3

Accuracy bar graph for different optimizer

Prediction is based on 4800 COVID-19 positive Images and 4800 COVID-19 negative Images as ground truth. 4482 images are correctly classified as COVID-19 positive and 4452 are correctly classified as COVID-19 negative(normal). The values of Accuracy, Sensitivity/Recall/TPR, Specificity/Selectivity, precision/PPV, F1-Score, and AUC are tabulated in Table 4.

From the Table 5, our proposed model shows best accuracy value 93.05. Figure 4: (a) shows confusion matrix of our model prediction for test data with heatmap, (b) shows the train and test accuracy, (c) shows the train and test loss, and (d) shows area under the curve for receiver operating characteristic is 0.93.

Table 5

Summary of CNN models

Ref. Paper	TotalImages	Training	Testing	Validation	Accuracy	Sensitivity / recall /TPR	Specificity / Selectivity	Precision / PPV	F1-Score	AUC
Horry et al. [18]	1475	1180	295	N/A	85.00	83.00	84.00	84.00	83.00	N/A
Momeny et al. [19]	1248	998	125	125	77.60	80.80	91.50	N/A	73.70	N/A
Ucar et al. [20]	1500	1200	300	5-fold	92.49	88.93	94.37	89.03	88.76	91.70
Frid-Adar et al. [21]	1845	1795	50	N/A	91.00	98.00	80.00	98.00	N/A	95.00
Rahamanet al. [26]	860	580	140	140	89.30	89.67	N/A	90.83	88.67	N/A
Pereiraa et al. [22]	1144	802	342	5-fold	87.02	N/A	N/A	N/A	83.00	N/A
Apostolo et al. [23]	3905	N/A	N/A	10-fold	87.66	97.36	99.42	N/A	N/A	N/A
Khan et al. [24]	1248	N/A	N/A	4-fold	89.60	89.92	96.40	90.00	89.80	N/A
Rahimzadeh et al. [25]	9011	N/A	N/A	5-fold	91.40	80.53	94.00	72.83	N/A	N/A
Jain et al. [8]	6432	5467	965	N/A	93.00	88.00	N/A	94.00	90.33	N/A
Nandi et al. [9]	20,425	18,207	2218	5-fold	89.39	N/A	N/A	N/A	N/A	99.99
Proposed work	60,000	38,400	12,000	5-fold	93.05	93.37	92.75	92.78	93.07	93.42

Best value is in bold

Fig. 4

a Confusion matrix, b accuracy curve, c Loss curve, and d ROC with AUC is 0.93

Initially to detect the COVID-19 disease is, the number of labeled image data points was limited. Because of this, there could be chances of over-fitting. Hence, the size of the dataset and the number of image data points used were increased. Better results can be obtained after the dataset size is increased. Currently, our proposed work consists of eight different optimizer used to carry out the mentioned classification tasks.

X-ray images from Chest X-ray machines can be used primarily to detect COVID-19, and physicians can then carry out further diagnosis, saving patients’ hard-earned money. Furthermore, X-ray machines are readily available in large quantities, and in most normal hospitals, the X-ray machine is an inherent element of the setup, so making this available in small hospitals will be simple. Furthermore, compared to RT-PCR, X-ray equipment require less maintenance in terms of chemicals, and hence the cost of operation is lower. There are some negative effects of X-ray scanning, such as people carrying metals and pregnant women. In these circumstances X-ray scanning is not recommended.

We developed a CNN based deep learning framework to identify CXR images as positive or negative for corona virus illness 2019 (COVID-19), with predictions 93.05% accuracy, 92.75% specificity, 93.37% sensitivity and 93.07% F1-score. Despite the fact that the medical image diagnosis system has not yet been approved as a stand-alone diagnosis tool, we feel that it may be utilized as an aid to supplement medical judgement with the benefit of instant results and faster turnaround times. Table 4 shows that some of the reported research results of chest X-ray COVID-19 classification. Our model is comparatively outperforms and yields accuracy of 93.05 percent.

Conclusion and future scope

Mass screening of persons for COVID-19 detection from chest x-ray image could be carried out efficiently with our proposed model. When compare to conventional RT-PCR approach, it will provide faster and more reliable findings with cost-effective. X-ray machines are common in hospitals because they are an important part of the medical equipment setup. As a result, this diagnostic treatment can be made available to people in local areas. Proposed model shows better accuracy, precision, specificity, recall, and F1-score. The CNN based Deep learning network model is very useful in medical image analysis for early detection of COVID-19 by classifying the input CXR images into normal and COVID-19 classes. New CNN based classification model can analyze the chest X-ray images and helps in the diagnosis of COVID-19 with more accuracy. Adam optimizer gives the better accuracy when compared with other optimizer. Augmentation of chest X-ray images and hyper parameter tuning helps to achieve the best accuracy for COVID-19 classification.

Researchers can develop new models with high accuracy for COVID-19 image classification. New models may handles both CT and X-ray images with classification of many classes. An enhanced architecture could be developed to find the severity level of COVID-19 infection. In the future, single chest X-ray image can be used to classify under several classes such as normal, pneumonia, tuberculosis, lung cancer and COVID-19.

Classification Metrics
The total number of real COVID-19 positive images (P) = 4800	(1)
The total number of real COVID-19 negative images (N) = 4800	(2)
Test result that correctly indicates the presence of a COVID-19 condition (TP) = 4482	(3)
Test result that correctly indicates the absence of a COVID-19 condition (TN) = 4452	(4)
Test result which wrongly indicates that a COVID-19 condition is present (FN) = 318	(5)
Test result which wrongly indicates that a COVID-19 condition is absent ( FP) = 348	(6)
Accuracy = (TP + TN)/(TP + FN + FP + TN) = (TP + TN)/(T + F) = (4482 + 4452) / (4800 + 4800) = 0.930625	(7)
Sensitivity = Recall = Hit rate = True positive Rate = TPR = TP/(TP + FN) = TP/P = 4482/4800 = 0.93375	(8)
Specificity = Selectivity = True negative Rate = TNR = TN/(FP + TN) = TN/N = 4452/4800 = 0.9275	(9)
Precision = Positive predictive Value = PPV = (TP)/(TP + FP) = 4482/(4482 + 348) = 0.92795	(10)
F1 score = Harmonic mean of Precision and Sensitivity = 2 × (PPV × TPR)/(PPV + TPR) = 0.93084	(11)
Matthews Correlation Coefficient = (TP × TN − FP × FN)/Sqrt((TP + FP)(TP + FN)(TN + FP)(TN + FN)) = 0.8613	(12)