A deep and handcrafted features-based framework for diagnosis of COVID-19 from chest x-ray images.

Automatic early diagnosis of COVID-19 with computer-aided tools is crucial for disease treatment and control. Radiology images of COVID-19 and other lung diseases like bacterial pneumonia, viral pneumonia have common features. Thus, this similarity makes it difficult for radiologists to detect COVID-19 cases. A reliable method for classifying non-COVID-19 and COVID-19 chest x-ray images could be useful to reduce triage process and diagnose. In this study, we develop an original framework (HANDEFU) that supports handcrafted, deep, and fusion-based feature extraction techniques for feature engineering. The user interactively builds any model by selecting feature extraction technique and classification method through the framework. Any feature extraction technique and model could then be added dynamically to the library of software at a later time upon request. The novelty of this study is that image preprocessing and diverse feature extraction and classification techniques are assembled under an original framework. In this study, this framework is utilized for diagnosing COVID-19 from chest x-ray images on an open-access dataset. All of the experimental results and performance evaluations on this dataset are performed with this software. In experimental studies, COVID-19 prediction is performed by 27 different models through software. The superior performance with accuracy of 99.36% is obtained by LBP+SVM model.

INTRODUCTION

COVID‐19, which is emerging as an upper respiratory and lung infection, first appeared in Wuhan province, China in late 2019. It is mostly infected by airway and contact, and causing severe damage to the infected lungs. The virus has since spread rapidly and has become a global epidemic. The number of cases and related deaths continue to increase every day. One of the major steps to prevent the spread of the COVID‐19 is to effectively scan people suspected of being infected to prevent the virus from spreading further across the population. In this context, any technological tool that allows fast and high‐accuracy scanning of COVID‐19 infection is useful to health professionals. Lung imaging with x‐ray is one of the most widely used and accessible modality for rapid examination of lung conditions. Chest x‐ray images are ready to be quickly analyzed by radiologists. Chest x‐ray images have been proved to be useful in monitoring the effects COVID‐19 causes on lung tissue. Therefore, chest x‐ray images can be used to diagnose COVID‐19. Lung x‐ray radiography is an effective tool for triaging non‐COVID‐19 patients with pneumonia to efficiently allocate hospital resources. However, there are many common features between x‐ray images of COVID‐19 and pneumonia caused by other viral infections, like common influenza. , These common observations in images are ground‐glass opacities, air space consolidation, bronchovascular thickening, vascular thickening, and bronchial wall thickening. In this context, chest x‐ray images of COVID‐19 and pneumonia are similar and contain similar features. This similarity makes it difficult for radiologists to diagnose COVID‐19 cases. A reliable method for classifying COVID‐19 and non‐COVID‐19 chest x‐ray images can accelerate the triage process of non‐COVID‐19 cases and maximize the allocation of hospital resources to COVID‐19 cases.

Real‐world data (like chest x‐ray images) given to models in machine learning (ML) are in the form of feature vectors and these feature vectors are extracted from the raw data. Feature engineering is the process of extracting relevant features from data for a ML model. The count of features is as important as the features themselves. If the features in the feature vectors representing real‐world data given to the model are not sufficient, the model will not be able to perform its main task. If there is a feature vector that contains more than necessary and unrelated features, the model will still not produce accurate results. In the ML process, not only the model but also the features that will represent the data are selected. Well‐selected features facilitate subsequent modeling steps and increase the resulting model's ability to complete the desired task. However, if the features are not selected properly, a much more complex model may be required to achieve the same level of performance. Thus, we study on performances of feature extraction techniques and feature vectors of chest x‐ray images in this study.

The scope and contributions of this study could be summarized as follows:

In this study, a general framework (HANDEFU) that could be used in different computer vision and ML problems is developed.

This original framework supports handcrafted, deep, and fusion‐based feature extraction techniques for feature engineering.

Any feature extraction technique and model could then be added dynamically to the library of software at a later time upon request.

This framework is utilized for diagnosing COVID‐19 from chest x‐ray images.

The user can build a feature extraction method and classification to train the model. Then, all performance evaluations on test data are performed with this software.

The rest of this article is organized as follows. In Section 2, the related works in literature are presented. In Section 3, the methodology which contains stages of the framework is explained. In Section 4, utilized dataset for testing is stated. Then, experimental results are presented and findings are discussed in Section 5. Finally, Section 6 concludes the article and presents some future work opportunities.

RELATED WORK

In the literature, there exist numerous approaches to diagnose COVID‐19 from chest x‐ray images based on various ML, deep learning, and hybrid methods. The studies based on machine and deep learning techniques are gaining increasing popularity for radiology images. When the related studies on this subject are examined, the most common and biggest problem is the lack of a sufficiently large data set. In general, studies have aimed to overcome this problem with various methods. Studies conducted on this subject can be given as follows: related studies using a similar open‐access dataset with this study are summarized as follows. In some studies, different‐sized image samples belonging to the classes may have been taken from the same data set at different times. Therefore, it can be observed that similar techniques give different performances in such a situation. For instance, Chowdhury et al. created this data set, made it available for open‐access, and increased the size of the data over time. They studied the utility of artificial intelligence for accurate detection of COVID‐19 from chest x‐ray images rapidly. They obtained 99.7% classification accuracy with DenseNet201 model (used dataset contained 423 COVID‐19, 1579 normal, and 1485 viral pneumonia chest x‐ray images). Ahammed et al. used a CNN model to detect COVID‐19 positive patients in early phase. Their model achieved 94.03% accuracy. The limitation of this study small number of data (285 COVID‐19 images) was used for training and this is not adequate for deep‐learning and COVID‐19 prediction. Aslan et al. proposed a hybrid model with CNN‐based transfer learning and BiLSTM. Their hybrid architecture gives 98.70% accuracy for COVID‐19 infection detection on chest x‐ray images on this similar data set. Gupta et al. proposed an integrated stacked deep CNN model which name is InstaCovNet‐19. Their model is composed of various pre‐trained models. Their model gives accuracy of 99.08% and 99.52% on three‐class and two‐class classification respectively. Ouchicha et al. proposed CVDNet that is deep CNN model to classify COVID‐19 cases from other cases (normal or pneumonia) using chest x‐ray images. They achieved an average accuracy of 97.20% for two‐class (binary) and 96.69% for three‐class (multiple) classifications. Similarly, Asif et al. proposed the detection COVID‐19 pneumonia patients using deep CNN on same dataset (this consisted of 1345 viral pneumonia, 864 COVID‐19, and 1341 normal images). They obtained the best performance with more than 98% classification accuracy. Cavallo et al. segmented manually lung areas by using polygonal regions of interest from this dataset. They extracted texture features with co‐occurrence matrix by using gray level histogram. They extracted a total of 308 features from per ROI, and 110 COVID‐19 chest x‐ray images were selected for the final analysis. They obtained 91.8% accuracy with ensemble ML and 92.9% accuracy with artificial neural network. The common limitations of these studies are the small number of data and owning to the small number of COVID‐19 samples. Therefore, the lack of images of the COVID‐19 class does not make learning perfect and makes it difficult to detect COVID‐19.

Other related studies could be summarized as follows. Rasheed et al. proposed the use of logistic regression and CNN for diagnosis of COVID‐19. They investigated their method with PCA and without PCA to obtain high accuracy. They obtained overall accuracy between 95.2% and 97.6% without PCA, and between 97.6% and 100% with PCA for identification. In order to detect COVID‐19 from chest x‐ray images, Ahmed et al. proposed a COVID‐Net which is combination of residual network and parallel convolution. They achieved accuracy of 97.99%. Sharifrazia et al. proposed a fusion of CNN, SVM, and Sobel filter to detect COVID‐19 using x‐ray images. They used high pass filter with Sobel to get the edges of the images. The highest classification accuracy achieved with this method is 99.02%. Shankar and Perumal proposed FM‐HCF‐DLF model that is a novel fusion method hand‐crafted with deep learning features for COVID‐19 classification and diagnosis. Their proposed model outcome superior performance with accuracy of 94.08%. Joshi et al. proposed a deep learning‐based system for chest x‐ray images to detect the COVID‐19. The average test accuracy of 97.11% was achieved for multi‐class and 99.81% for binary classification. Jain et al. studied on deep learning‐based CNN models for detection of COVID‐19 on chest x‐ray images. They compared the performance of Xception, Inception V3, and ResNeXt models about accuracy. When compared to other methods, Xception gives the highest accuracy with 97.97%. To detect COVID‐19 patients, Sarker et al. proposed a deep learning‐based approach using the model of Densenet‐121. The test accuracy of 96.49% was achieved for binary classification and 93.71% for multi‐class classification. Turkoglu proposed COVIDetectioNet model for diagnosis of COVID‐19. This model utilizes features selected from combination of deep features. A transfer learning approach was used with a pretrained CNN‐based AlexNet architecture and SVM was used for classification. In experimental results, an accuracy of 99.18% was achieved by using this model. Since the data was imbalanced in some studies, the diversity of the data was increased with the image augmentation technique for the training model. For instance, Rahman et al. achieved 95.11% classification accuracy with DenseNet201 model (used large dataset that contains 18,479 images with 3616 COVID‐19, 6012 non‐COVID, and 8851 normal images). They utilized from augmentation technique (an image rotation‐based technique) to generate training images before CNN models for training. Similarly, Luz et al. utilized from this augmentation technique for effective training. They achieved 93.5% classification accuracy with EfficientNet architecture (used large dataset that contains 13,770 images with 183 COVID‐19, 5521 viral pneumonia, and 8066 normal images).

The common shortcoming of state‐of‐art studies is that most deep learning‐based models are trained on unbalanced datasets and therefore it may lack robustness. There exists a small number of studies on creating a scalable deep learning model using image enhancement/preprocessing techniques and efficient feature extraction. Due to unbalanced data processing and inability to extract necessary features from images, the classification accuracy does not reach the desired level. Also, proposed models in literature cannot guarantee to reproduce the promising results when these are evaluated on a larger dataset. It is a handicap that running a CNN architecture with many iterations on a small dataset leads to overfitting. In order to overcome these limitations, this framework combines image preprocessing with handcrafting, deep and fusion based feature extraction techniques to create a scalable model.

METHODOLOGY

In this study, a general framework (HANDEFU) including different feature extraction techniques and classifiers is composed for use in computer vision and machine‐learning problems as shown in Figure 1. This original framework supports handcrafted, deep, and fusion based feature extraction techniques. The handcrafted based feature extraction methods as LBP, HOG, GABOR are present in the framework. The user is able to build shallow neural network, deep neural network models, and popular deep‐learning techniques through the HANDEFU framework. The framework has a dynamic structure. Therefore, other popular deep learning techniques could be added to the library for deep‐feature extraction. At present, the framework contains popular deep learning techniques as shallow‐fully connected network model, deep‐fully connected network model, shallow‐convolutional network model, deep‐convolutional network model, AlexNet, VGG16, VGG19, ResNet50, InceptionV3, Xception, and DenseNet121 model. Moreover, it is aimed to obtain high performance by creating a robust model by combining handcrafted features and deep learning features. The user can select and combine different deep and handcrafted feature extraction methods as InceptionV3+LBP, VGG19+GLCM, and VGG19+HOG through the framework. It is necessary to classify the handcrafted or deep learning features obtained in the previous stage with the help of a classifier. In this way, a supervised learning or pretrained model is built for prediction. Thus, predictions can be made from test data with an established model. At this stage, the user is given the opportunity to classify the features with other supervised learning techniques through the framework. The classifier options as kNN, Naive Bayes, neural network (simple 3‐layer NN), MLPClassifier, and softmax classifier exist in the framework. These options can be further increased by dynamically adding them to the framework library.

FIGURE 1

General overview of HANDEFU framework

As shown in Figure 1, HANDEFU framework is utilized for diagnosis of COVID‐19 from chest x‐ray images. The user loads the dataset as training and test data. Then, the user is given the opportunity for preprocessing. Then, the user builds a feature extraction method and classification to train the model through the framework. Furthermore, all performance evaluations on test data are performed with this software. All of the stages will be explained in detail as follow.

Preprocessing

Images may need to be preprocessed before the feature extraction. The purpose of the preprocessing at this stage is to facilitate the computing to be done in the later stages of the framework. At this stage, the user is given the opportunity to choose a number of preprocessing options such as smoothing, noise reduction, normalization, and image resizing. The image could be enhanced in different ways. For instance, smoothing reduces undesired details or noise in the image and speeds up following processing. When taken as a single image, some artifacts may cause an inhomogeneity problem in the image. Normalization is the process in which multiple images are placed in an average statistical distribution in terms of size and pixel values. A single image can also be normalized within itself. Thus noisy images are eliminated by intensity normalization. Brightness and contrast can be adjusted through a linear transformation of image intensities with dynamically windowing of each image.

Figure 2 shows a sample of chest x‐ray image before and after preprocessing. Gaussian smoothing and median filtering significantly reduce the noise of the image. The two‐dimensional (2D) Gaussian filter is often preferred for smoothing and noise reduction. Regions with a diagnosis of COVID‐19 are pointed by a radiologist as shown in Figure 2. The radiologist shows the changes made after pre‐processing. Thanks to these filters, ground‐glass opacities and consolidations become clearer. Also, some reticular patterns which are related to COVID‐19 are sharpened after preprocessing as shown in Figure 2. The equation of 2D Gaussian filter is given in Equation (1). denotes variance of the distribution and (x, y) is Cartesian coordinates of the image.

FIGURE 2

A sample of chest x‐ray image (A) before preprocessing and (B) after preprocessing

The sizes of the images in the database used for image classification and pattern recognition may differ from each other. In some feature extraction functions, the number of extracted features depends on the size of the image (e.g., LBP, HOG, and Gabor). In order to conduct methods, the number of features obtained from the images must be the same. For this, all images must be resized to a fixed size. In addition, when the size and resolution of the images are reduced by the resizing process, the time required for processing the images in the framework decreases and the performance of the framework is positively affected. In this study, the images are resized from 10241024 to 250250.

Hand‐crafted feature extraction

The fundamental elements of the pattern recognition process are preprocessing, feature extraction, feature selection, training, and classification as shown in Figure 3. The feature can be basically explained as measurable or observable information about the pattern. Feature extraction enables obtaining characteristic features of the pattern by eliminating irrelevant and excessive information. This is basically a kind of dimensional reduction process. Since very high dimensional data are used especially in image processing applications, using these data in their original form greatly increases the processing time. Thanks to hand‐crafted feature extraction methods, it is ensured that the original data is reduced to smaller sizes by eliminating unnecessary and high‐size information and preserving its characteristic features.

FIGURE 3

An overview of handcrafted‐based feature extraction and classification

Feature extraction, which is one of the pattern recognition stages, plays an important role in the success of the system. Appropriately selected features represent the characteristics of the classes and positively affect recognition success. The purpose of feature extraction is to transport as much of the class information as possible to smaller sizes. In other words, the purpose of feature extraction is to find a T transform that expresses the X space in the best possible way from a k‐dimensional Xobservation space to a smaller l‐dimensional Y feature space as given in Equation (2). Here, numerical features can be in the form of a set of vectors that characterize the class of the pattern.

Figure 3 shows an overview of handcrafted based feature extraction and classification. In this stage, a number of handcrafted based feature extraction methods in HANDEFU framework (e.g., LBP, HOG, Gabor) will be explained in detail.

Local binary pattern (LBP)

LBP feature extraction is used in applications such as texture classification, face recognition, facial expression recognition, gender, and age identification (classification). LBP is a pattern recognition method based on binary coding between each pixel in the image and its neighbors. LBP generates binary codes by comparing the 33 neighborhoods of each pixel in the gray image. The 33 image obtained from each pixel is considered as circular, and the central pixel is accepted as the threshold value. When the neighbor pixel value is compared with the center pixel value, it is encoded in binary as 1 if the value of the neighbor pixel is greater than or equal to the value of the center pixel, else encoded as 0. In this way, the LBP code of the center pixel is obtained by generating an 8‐bit binary code and converting it to a decimal number. This process is computed for each pixel of the image as shown in Figure 4. Places in an image (such as corners, edges, line regions, luminous, or unilluminated regions) represent the features of the image. These textural features of the image could be extracted with the LBP code. In this way, it is possible to obtain different textural features of the images with the generated LBP code.

FIGURE 4

Implementation of the LBP operator

The LBP compares the values between the center pixel at () position and the neighboring pixels surrounding it. LBP code is obtained when an 8‐bit code is created and converted to a decimal number. For each pixel in the image, the binary code is generated by thresholding the neighboring pixels according to the center pixel. The generation of LBP codes is performed by Equation (3). In Equation (3) (, ) is the coordinates of the center pixel, is the gray value of the center pixel, and is the values of the pixels in 33 neighborhood surrounding the center pixel. The function in Equation (4) is the 8 neighborhood values of gray pixels. , ,

Regional histograms could be used to get more efficient results. For this purpose, the x‐ray image is divided into 88 = 64 square regions and histograms are obtained by calculating the LBP codes separately for each region. These histograms are concatenated and used in classification as feature vectors.

Histogram of oriented gradient (HOG)

In recent years, the use of edge orientation histograms within the scope of image recognition studies has gained great momentum. Local gradient orientation histograms (HOG) was first suggested by Dalal and Triggs. HOG is defined as the characteristics of the orientation () and magnitude (m) values of the pixels in image. The purpose of this method is to define the image as a group of local histograms. These histograms contain the number of occurrences of orientations of gradients in a local region of the image. The process phases in implementation of HOG identifiers could be listed as follows.

The image is transformed into a gray level image and the horizontal and vertical gradient values of the gray level image are computed.

Gradient of orientation and magnitude are obtained by using horizontal and vertical gradient values.

Gradient orientations are divided into cells in the size of the image, and the gradient orientations within each cell are labeled with 9 different states of 20°.

Local histograms are created using gradient magnitude and orientations for each specified block.

The created local histograms are normalized.

All histograms are concatenated and used as feature vector.

Horizontal () and vertical () gradient are computed by using derivative masks of the gray level images. The horizontal () and vertical () image gradient are calculated by Equations (5) and (6). In these equations, I() shows the pixel intensity at the point ().

Gradient magnitude (m) and gradient orientations () are calculated by using horizontal and vertical image gradients as follows.

can be divided into equal regions between 0°–180° or 0°–360° depending on the application when computing the histogram. Gradient direction regions are defined as 9 regions with an angle difference of 20° between 0° and 180° in this step. The gradient magnitude of each pixel in the cell is distributed to the histogram regions according to the angle. Thus, a big size histogram is obtained by combining all the histograms created in one block as shown in Figure 5. This histogram is then normalized using by Equation (9). In this equation, is the large histogram vector obtained from a block and v is the normalized HOG feature vector. L2‐Norm is used for normalization. In this stage, a computed HOG feature vector is ready to be given as input of a classifier.

FIGURE 5

General structure of HOG (blocks, cells, histograms, and feature vector)

Gabor feature extraction

This technique shows great similarity with the human visual system in terms of its frequency and orientation characteristics. Gabor wavelets are used in computer vision applications and modeling biological visions (particularly applications about texture descriptor as fingerprint, palm, and face recognition). In wavelet transform, spatial information is obtained as well as the frequency information of the image. In the wavelet transform method, frequency data could be regionalized. For this reason, it performs the transformation of the function not only according to the frequency parameters but also according to the scale parameters when compared with the Fourier transform. This technique, which is especially preferred in texture analysis, uses Gaussian elements modulated by sine and cosine functions. Gabor filter could be applied dimensionally to 1D or 2D images. A Gabor filter is obtained by modulating a sine wave with a Gaussian function. The 2D Gabor filter kernel matrix is calculated by Equation (10). In this equation, the first exponential expression (Gaussian function) is a descending term depending on and , while the second exponential expression is a periodic term containing a complex number. ,

In the equation, is the coefficient that determines the wavelength of the cosine factor. If the coefficient is 1, since the cosine expression will always be 1, the coefficient should be chosen 2 or a larger integer. is used to calculate the values of and and is the orientation angle of the Gabor kernel to be created. The variables and are calculated by the following formula for a given value of . , is the phase angle of the kernel matrix whose angle is to be formed. The filter can be shifted on the x‐axis by changing this value. is the coefficient that determines the standard deviation (SD) of the Gaussian function. allows the SD value to be determined for . If this value is 1, the resulting kernel matrix will be of equal length since it has equal SD for x and y. When a different ratio is chosen, the kernel matrix will be formed in a rectangular shape. The application of the generated Gabor filter to an image is a 2D convolution process and is performed with Equation (12). In the equation, refers to the image to be applied Gabor filter and denotes the Gabor filter. ,

In this study, Gabor feature extraction is performed by applying 40 Gabor filters obtained from 5 scales and 8 orientations as shown in Figure 6. For example, the size of the chest x‐ray images used in experiments is 256256 pixels. A feature vector is with length . and is the factor of downsampling along rows and columns. is the number of scales and is the number of orientations. By using 40 Gabor filters, the dimension of the feature vector for one image is .

FIGURE 6

General structure of Gabor (scales, orientations, and feature vector)

Deep feature extraction

In order to define a model with classical ML techniques or to set up a detection (prediction) system, first the feature vector should be extracted. Experts in the field are required for the extraction of the feature vector. These processes both take much time and keep the expert very busy. For this reason, these techniques cannot process raw data without pre‐processing and expert assistance (as supervised learning). Deep learning (DL) has made great progress by overcoming this problem that those working in the field of ML have been dealing with for many years. Deep learning is a field of ML research that has emerged from studies on artificial neural networks in recent years. Because deep networks do the learning process on raw data, unlike traditional ML and image processing techniques. While processing the raw data, it acquires the required information with the representations which are obtained in different layers.

The DL algorithm is scalable with large data sets and is effective in identifying complex patterns from feature‐rich data sets. The main fields that deep learning is successfully utilized are computer vision, speech recognition, and natural language processing. The reason for the widespread success of deep learning is its method of directly predicting output from the representation set. An important advantage of deep learning over traditional techniques is that it does not require an explicit feature extraction stage. The network takes the raw input and maps it to the desired output. Features are learned automatically by the network without manual intervention. Although these models require high processing power, this is no longer seen as a problem with the development of central and graphical processing power today. DL models are the improvement of traditional networks with an increasing number of hidden layers. Hidden layers greatly improve the performance of these networks. Deep learning techniques can automatically reveal image features that are not obtained from the original images. There is no human intervention or manual determination of the features in the extraction of the features. Deep learning has recently been widely used in pattern recognition and image classification.

It is possible to build different DL models thanks to the development of big data and GPUs. These designed models perform the learning process themselves without the features determined by the user from the input data. It achieves this learning process by discovering different features of the data in different layers. The basic model of these architectures is considered to be CNN (convolutional neural network). CNNs are the improved versions of artificial neural network ANNs. The deep neural network as a result of further increasing the number of hidden layers in ANNs can be defined as CNN. This depth in the CNN was achieved by using 2D filters. In addition to this difference in depth, CNNs perform the learning process in a hierarchical structure. As shown in the figure, defining an object in CNN is possible by discovering a sub‐feature of that object in each layer. CNN as shown in Figure 7, is a mathematical structure consisting of three types of layers, typically convolution, pooling, and fully‐connected layers. While the convolution and pooling layers perform feature extraction, it achieves the final output for classifying the extracted features using the fully‐connected layer. ,

FIGURE 7

Fundamental architecture of the CNN

In digital images, pixel values are stored in a 2D matrix, this is a small convolution matrix called a feature extractor mask that is applied to each point of the image. When one layer feeds its output to the next layer, the extracted features can become hierarchically and increasingly complex. At this stage, with the masks used, learning processes take place in the layers and the values of the layers are computed.

Convolutional layer: The convolution process, which forms the basis of CNNs, aims to apply a filter matrix to the input by the convolution process and use the results for the next layer. This layer is based on traversing small size filters such as 2 2, 3 3, and 5 5 over the entire image. Thus, a new image is obtained by extracting the more distinctive features in the image. The weights of the filter matrix used for the convolution process are determined at the learning stage of the CNN. The filter matrix is shifted with the specified stride and the convolution operation is applied. The result obtained is given as the input for the next layer, if not the last layer. If it is the last layer, it represents the output image. Filter coefficients (f) are multiplied by equal‐sized windows (w) in the image and calculated by taking their sums as in Equation (13). As a result, a new image is obtained based on distinctive high‐level features. ,

Pooling layer: Pooling layer is used to reduce the computational load by reducing the parameters within the network. There are many types of pooling layers. Maximum, average, and L2 pooling options are available in the HANDEFU framework. In maximum pooling, the highest of the pixel values remaining in the filter window, and in the average pooling process, the average of all the pixel values within the filter window is kept as a single value in the output pixel. At the end of the pooling process, the aspect ratio of the image is reduced. In this process, there are losses in pixel values, but these losses create less computational load for subsequent layers.

Dropout layer: The dropout layer is the most commonly used regularization layer in deep learning methods. Dropout is the removal of node from input and hidden layers according to certain rules. Thus, it is aimed to reduce the parameter and computation without loss of performance and it is aimed to prevent overfitting. Dropout layer is generally used after fully connected layers. Thus, there is less information link between nodes and the nodes are least affected by each other's weight changes. As a result, more consistent network models are created.

Batch normalization: Batch normalization is a method used to make the CNN more regular, like the dropout layer. Besides the regulation of the neural network, it also allows the network to resist the vanishing gradient during training. This can reduce training time and provide better performance. For the batch normalization computation, the input is defined as and the training parameters are shown as , . The operation in Equation (14) is the average of the mini batch. The average is calculated by summing all the input in the mini‐batch and dividing them by the total number of input. Equation (15) is variance of the mini‐batch. As a result of these calculations, the general intervals in which the data in the mini‐batch are computed. Normalization of the data is calculated with Equation (16) . And, normalized data is processed with (scaling parameter) and (shifting parameter) in Equation (17). As a result of these operations, a learnable linear equation is obtained. and are used as hyperparameters of the designed network. These parameters determine how much the batch normalization layer will do on the data.

Fully connected layer: The fully connected layer is commonly located at the end of the CNN and is used in classification processes. This layer is often used towards the end of the CNN architecture and to optimize class scores. In addition, the number of these layers may vary in architectures based on deep learning.

Classification layer: Classification layer is the last layer of the CNN model that classification process is performed. The output values of this layer are equal to the number of classes, depending on the number of objects to be recognized. Activation functions are used to learn and predict any continuous and complex relationship between network variables. In this framework, the ReLU activation function exists to avoid linearity in the network. Since multi‐class classification is needed in the fully connected layer in the CNN architecture, softmax function is used as in Equation (18). Softmax classifier is commonly used in this layer based on deep learning architectures. This classifier produces probabilistic values between 0 and 1 for each class. As a result, the highest probability value gives the class predicted by the model.

As shown in Figure 8, the user is able to build a sample deep CNN model consisting of four set convolution‐pooling layers, a flattened layer, three fully connected layers through the HANDEFU framework. Moreover, the framework contains popular deep learning techniques as shallow‐fully connected network model, deep‐fully connected network model, shallow‐convolutional network model, deep‐convolutional network model, AlexNet, VGG16, VGG19, ResNet50, InceptionV3, Xception, and DenseNet121 model as shown in Figure 9. Thus the user is able to build shallow neural network and deep neural network models through the HANDEFU framework. The framework has a dynamic structure. Therefore, other popular deep learning techniques could be added to the library later.

FIGURE 8

A sample deep CNN model built with the HANDEFU framework

FIGURE 9

Existing deep learning techniques in the HANDEFU framework

Fusion‐based feature extraction

Ensemble learning is the creation of a new model by using multiple modeler algorithms together instead of using a single basic learning model. At this stage, it is aimed to obtain high performance by creating a robust model by combining handcrafted features and deep learning features. Feature fusion is achieved by combining multiple feature vectors together. In this context, previously obtained features are combined into a single vector. Here, a feature fusion is obtained by integrating the maximum feature vector. The fusion of features with entropy determines the projection of the method. For instance, deep feature extraction is done by InceptionV3 (Equation 19) and handcrafted feature extraction is done by LBP (Equation 20). Then, feature extraction is combined as a single feature vector as in Equation (21). In order to obtain better features depends on score, entropy is executed on fused (feature vector) on this framework. Thus, user can select and combine different deep and handcrafted feature extraction methods as InceptionV3+LBP, VGG19+GLCM, and VGG19+HOG through the framework.

Classification

In the classification stage, it is necessary to classify the handcrafted or deep learning features obtained in the previous stage with the help of a classifier. In this way, a supervised learning or pretrained model is built for prediction. Thus, prediction can be made from test data with an established model. At this stage, the user is given the opportunity to classify the features with other supervised learning techniques through framework. The classifier options as k‐nearest neighbors (kNN), Naive Bayes (NB), and neural network (NN), MLPClassifier exist in the framework. These options can be further increased by dynamically adding them to the framework library. At this stage, for instance, the user can classify and predict with SVM instead of softmax after extracting features with deep learning (e.g., ResNet50 + SVM). Similarly, the user can choose any classifier after extracting the handcrafted feature. Therefore, user can build different models such as LBP+SVM, LBP+kNN, LBP+NB, LBP+NN, HOG+SVM, HOG+kNN, HOG+NB, HOG+NN through the framework.

EXPERIMENTAL RESULTS

Data set

In this study, an open‐access dataset is utilized for classification of COVID‐19 chest x‐ray images. This dataset consists of chest x‐ray images for COVID‐19 positive cases along with normal and viral pneumonia images. A team of researchers composes this database from the university Qatar, Doha, Bangladesh, Dhaka, and Qatar with their collaborators and doctors. , As shown in Table 1, a total of 3886 chest x‐ray images exist as 1200 COVID‐19 (positive cases), 1345 viral pneumonia, and 1341 normal (negative cases) images. Figure 10 shows image samples of normal (negative), viral pneumonia, and COVID‐19 from this dataset. Regions with a diagnosis of viral pneumonia and COVID‐19 are pointed by a radiologist as shown in Figure 10. In experimental results, this dataset is utilized for training and testing through the framework. All predictions and classification of COVID‐19 chest x‐ray images are performed on this dataset.

TABLE 1

Categorical distribution of the x‐ray images in dataset

Category name	Number of images
Normal	1341
Viral pneumonia	1345
COVID‐19	1200
Total:	3886

FIGURE 10

Sample chest x‐ray images from dataset as the case of (A) normal, (B) viral pneumonia, and (C) COVID‐19

Results

The experimental studies in this study are conducted on a desktop computer with Intel (R) Core‐i7 8700U CPU @ 3.20 GHz, 4GB NVIDIA GeForce GTX 1050 Ti graphics card and 16 GB primary memory. The software was implemented using Python 3.7 and its related libraries (e.g., Tensorflow, Keras, Numpy, Pandas, SciKit Learn, Matplotlib, and Tkinter GUI).

In this section, the performance evaluations of the methods are performed over four performance criteria (metrics) such as Precision (PR), Recall (RE), F1‐Score (F1‐SC), and Accuracy (ACC). These values are calculated over confusion matrices for each class. Confusion matrices are obtained for each model and experiment as shown in Figure 11. In this figure; 0, 1, and 2 for row and column name represents (0=COVID‐19), (1= viral pneumonia) and (2=normal) class respectively. These performance metrics are calculated from confusion matrix and mathematically defined as follows. ,

FIGURE 11

Confusion matrices of six different feature extraction methods and models. (A) LBP+SVM, (B) Gabor+SVM, (C) HOG+SVM, (D) deep CNN, (E) DenseNet121, (F) InceptionV3+LBP

In these equations, True Positive (TP) means that the patient actual with COVID‐19 positive case and system recognized as positive. True Negative (TN) means that the patient actually without COVID‐19 case and the system recognized as the negative. False Positive (FP) means that the patient actually without COVID‐19 case and the system recognized as the positive. False Negative (FN) means that the patient actual with COVID‐19 case and system recognized as the negative.

In this section, performance comparisons of different feature extraction methods and models are performed through the HANDEFU framework as shown in Table 2. Performance comparisons of different feature extraction methods and models (top 5 models are marked in bold according to Acc) are given in Table 2. The user is able to split data for training and testing as 70% is reserved for training and 20% for testing or 80% is for training 20% is for testing in advance. In this section, we performed the performance evaluations for 80%–20%. The methods are categorized as hand‐crafted, deep, and fusion‐based feature extraction. For instance, LBP, HOG, and Gabor‐based texture descriptors are used as hand‐crafted feature extraction techniques with kNN, NB, NN, and SVM classifiers. In this context, 12 different models exist such as LBP+SVM, HOG+SVM, Gabor+SVM and so forth. The user can select any model and run the model on loaded dataset as train and test data through the framework. Similarly, the user is able to build shallow neural network and deep neural network models through the HANDEFU framework.

TABLE 2

Performance comparisons of different feature extraction methods and models (top 5 models are marked in bold according to Acc)

Method	Model	PR (%)	RE (%)	F1‐SC (%)	ACC (%)
Hand‐crafted feature extraction	LBP+kNN	92.94	89.95	94.41	94.76
	LBP+NB	88.13	100	93.68	95.80
	LBP+NN	94.70	99.20	96.88	98.04
	LBP+SVM	98.52	99.42	98.97	99.36
	HOG+kNN	75.24	96.81	84.66	89.04
	HOG+NB	85.58	96.53	90.70	93.96
	HOG+NN	91.70	96.81	94.19	96.14
	HOG+SVM	92.67	98.46	95.47	97.23
	Gabor+kNN	88.06	98.33	92.91	95.37
	Gabor+NB	88.55	99.56	92.02	94.98
	Gabor+NN	90.64	98.58	94.43	96.43
	Gabor+SVM	93.22	98.82	95.94	97.49
Deep feature extraction	Shallow FC Neural Net.	90.72	90.82	90.67	90.62
	Deep FC neural net	92.03	92.20	92.08	92.03
	Shallow CNN	94.85	94.78	94.78	94.73
	Deep CNN	96.12	96.02	96.07	96.02
	AlexNet	95.78	95.88	95.81	95.76
	VGG16	95.90	95.90	95.84	95.76
	VGG19	95.67	95.62	95.59	95.50
	ResNet50	93.50	93.50	93.41	93.32
	ResNet50+SVM	94.82	94.75	94.75	94.70
	InceptionV3	93.70	93.88	93.75	93.70
	Xception	95.88	95.57	95.58	95.50
	DenseNet121	97.31	97.23	97.26	97.17
Fusion‐based feature extraction	InceptionV3+LBP+MLP	96.11	96.65	96.36	96.53
	VGG19+GLCM+MLP	95.96	95.96	95.90	95.85
	VGG19+HOG+MLP	94.17	97.04	95.53	96.37

The framework has a dynamic structure. Therefore, other popular deep learning techniques could be added to the framework later. Nowadays, the most popular deep‐learning techniques exist in frameworks as AlexNet, VGG16, VGG19, ResNet50, InceptionV3, Xception, and DenseNet121 as shown in Table 2. Hereby, performance comparisons of different deep feature extraction models are performed with obtained performance metrics. Furthermore, In fusion‐based category, user can select and combine different deep and handcrafted feature extraction methods as InceptionV3+LBP, VGG19+GLCM, and VGG19+HOG through the framework. All of the performance evaluation results of different models on this dataset are given in Table 2. Thus, we can observe the superior models thanks to comparisons of their performances. The top 5 models among different feature extraction methods and their performance comparisons are presented in Figure 12. Various feature extraction methods and models in this framework are built and executed on this dataset. Each of the model's performance and execution times are recorded. Performance evaluations and execution times of all models are recorded with this software. The top 10 models and their performance comparisons about accuracy and execution time are studied as shown in Table 3. Furthermore, the user can obtain training/test accuracy and training/test loss graphs according to iterations as shown in Figure 13. The graphs of each model are saved by this software.

FIGURE 12

Top 5 models among different feature extraction methods and their performance comparisons

TABLE 3

Top 10 models and their performance comparison about accuracy and execution time

Model	Accuracy (%)	Execution time (t)
LBP+SVM	99.36	5 min 13 s
LBP+NN	98.04	5 min 7 s
Gabor+SVM	97.49	50 min 1 s
HOG+SVM	97.23	6 min 33 s
DenseNet121	97.17	2 h 57 min 43 s
InceptionV3+LBP	96.53	1 h 52 min 04 s
VGG19+HOG	96.37	6h 29 min 16 s
HOG+NN	96.14	11 min 27 s
Deep CNN	96.02	7 h 52 min 52 s
AlexNet	95.76	1h 59 min 27 s

FIGURE 13

Training/test accuracy and training/test loss graphs of the DenseNet121 model

DISCUSSION

In experimental studies, it has been observed that LBP‐based hand‐crafted feature extraction method is more successful in diagnosing COVID‐19 cases for this dataset. As shown in Table 1, LBP+SVM model gives the highest accuracy value with 99.36%. Besides, this model has the shortest execution time (approximately 5 min) according to other methods. Because, regional histograms in LBP give more efficient results. X‐ray image is divided into 88 = 64 square regions and histograms are obtained by calculating the LBP codes separately for each region. These histograms are concatenated and then they have given to the classifier as feature vector. A 3776 size feature vector is obtained for each image in the data set. Similarly, HOG which is another hand‐crafted feature extraction method is studied in experimental results. It is seen from the Table 1 that HOG+SVM model gives better performance with accuracy rate of 97.23%. The model gives very good performance according to accuracy and execution time. The execution time of the model is approximately recorded as 6 minutes as shown in Table 3. The feature vector of any image is obtained as 34,596 with HOG method for 256256 size of image. Gabor which is the other hand‐crafted feature extraction method, is studied in experimental results. In this study, Gabor feature extraction is performed by applying 40 Gabor filters obtained from 5 scales and 8 orientations. By using 40 Gabor filters, the dimension of the feature vector for 256256 size of one image is 163,840. Gabor+SVM model gives better performance with accuracy rate of 97.49% as shown in Table 2. Since the dimension of feature vector obtained with the Gabor model is very large, the execution time of this method takes long time according to other hand‐crafted methods.

In experimental studies, deep feature extraction method and different models are studied on this dataset. Thus, first the simple artificial neural network (shallow‐fully connected network) model was established. Then deep CNNs, which are popular today, are used in COVID‐19 diagnosing. As shown in Table 2, different popular pre‐trained learning techniques such as Deep CNN, AlexNet, VGG16, VGG19, ResNet50, InceptionV3, Xception, and DenseNet121 have been compared on this dataset. Different models are run on the same data, and the performance of each is recorded as shown in Table 2. In the shallow‐fully connected network model, which is the simplest neural network model according to other methods, four hidden layers and 128, 256, 128, and 64 neurons exist in each layer respectively. In the experimental study, the lowest accuracy value was obtained from this model with 90.62%. More hidden layers have been added to the network using convolution neural networks to further improve accuracy performance. For instance, the Deep CNN model consists of four sets of convolution pooling layers, a flattened layer, three fully connected layers, and a classifier. The model has 39,361,539 parameters. In the experimental study, accuracy value of 96.02% is obtained with the Deep CNN model. The DenseNet121 model, which is popular today and is one of the advanced versions of CNN, achieved the highest accuracy value with 97.17%. It was observed that the DenseNet121 model in this study gives superior results than other deep learning models. The InceptionV3+LBP model, which is fusion‐based technique ensemble deep and hand‐crafted features, achieved the highest accuracy value with 96.53%.

In Figure 13, training/test accuracy and training/test loss graphs for 50 iterations of the DenseNet121 are presented. All of the models' evaluation results, confusion matrices, training/test accuracy, and training/test loss graphs could be taken from this software. Both training and test accuracy curves show a rising curve as the number of iterations increases. The loss curve shows the decrease in error rate. It shows that the training process and learning of the network are at a good learning rate. As shown in Figure 13, while the loss value decreases in each iteration, the accuracy rate increases with the given training set and learning occurs.

Performance comparison of the current study with other studies is given as shown in Table 4. In this table, the state‐of‐art/relevant studies with data of various sizes on the chest x‐ray dataset are discussed. The accuracy results of the existing studies in the literature are examined as binary and multiple classes. In addition, other similar studies conducted with the public dataset , used in the experimental study of this study are studied. Chowdhury et al. proposed an image augmentation technique with a transfer technique for detection. They used transfer learning technique with image enhancement technology to increase classification accuracy. SqueezeNet model gives the best accuracy score of 98.3% for this dataset. Ouchicha et al. proposed CVDNet which is a deep CNN model to classify COVID‐19 from other cases for chest x‐ray images. They obtained 97.20% accuracy for two‐class (binary) and 96.69% for three‐class (multiple) classifications. Their method gives promising performance for this small dataset (unbalanced dataset and the number of COVID‐19 cases in the dataset is low). Aslan et al. proposed a hybrid model with CNN‐based transfer learning and BiLSTM. Their hybrid architecture gives 98.70% accuracy for COVID‐19 infection detection on chest x‐ray images on this similar data set. Gupta et al. proposed an integrated stacked deep CNN model which name is InstaCovNet‐19. Their model is composed of various pre‐trained models. Their model gives accuracy of 99.08% and 99.52% on three‐class and two‐class classification respectively. Elaziz et al. proposed to use MRFODE (Manta‐Ray Foraging Optimization based on Differential Evolution) feature selection method and kNN classifier to decide chest x‐ray image whether as a COVID‐19 or non‐COVID‐19 case (for binary classes). Their hybrid model gives 98.09% accuracy for binary classes. There exists large amount of studies in the literature that have studied with unbalanced data and a small number of COVID cases. By considering the results from these studies, it cannot be guaranteed that similar success can be achieved in a larger data and balanced data. In the current study, more COVID‐19 cases are taken from this public dataset and studied with a balanced dataset (1200 COVID‐19, 1345 viral pneumonia, and 1341 normal). Therefore, compared to other studies, more COVID‐19 images are used and high accuracy value is obtained from this more balanced data. Similarly, Ahmad et al. proposed a hybrid ensemble model by studying on a balanced chest x‐ray data. They achieved 96.49% classification accuracy for multiple classes on a balanced dataset. In current study, LBP+SVM model (in the category of hand‐crafted feature extraction) gives the highest accuracy value with 99.36%. Pre‐trained DenseNet121 model (in the category of deep feature extraction) gives the highest accuracy value with 97.17%. In a study similar to the current study, Alam et al. proposed to combine of HOG for feature extraction and CNN techniques. Their novel method proposes fusion of features extracted by HOG and CNN and classification by CNN. Their method gives higher accuracy with 99.49% (for two‐class classification) than individual feature extraction techniques like HOG and CNN. The main difference of current study from this study is the user are able to select and combine different deep and handcrafted feature extraction methods with this framework. Thus HANDEFU framework contains other feature extraction and classification techniques. For instance in experimental study, deep feature extraction is done by InceptionV3, handcrafted feature extraction is done by LBP and classification is done by MLP. The InceptionV3+LBP model achieved the highest accuracy value with 96.53% in fusion‐based category. Unlike other specific studies, this framework provides the opportunity to set up and test different hybrid or fusion based models. In general, when the studies are examined, there are studies in the literature with small size and unbalanced data small amount of COVID‐19 images. In addition, the classification success of the studies was considered as a binary class in some studies. Due to its nature, it has been observed that the binary classification accuracy gives high accuracy results compared to the multi‐class accuracy. Hence, in the current study, a balanced data set was created and acceptable high results are obtained for multiple classes.

TABLE 4

Comparison of the current study with other related studies

Study	Year	Model	Chest x‐ray dataset	Accuracy
Ahammed et al. 9	2020	CNN	2971 images (285 COVID‐19, 1345 viral pneumonia, and 1341 normal)	Multiple: 94.03%
Apostolopoulos et al. 40	2020	VGG19	1428 images (224 COVID‐19, 700 viral pneumonia, and 504 normal)	Binary: 98.75% Multiple: 93.48%
Chowdhury et al. 8	2020	SqueezeNet	487 images (423 COVID‐19, 1485 viral pneumonia, and 1579 normal)	Multiple: 98.3%
Duran‐Lopez et al. 41	2020	COVID‐XNet	6926 images (2589 COVID‐19 and 4337 normal)	Binary: 94.43%
Elaziz et al. 42	2020	MRFODE+kNN	1560 images (219 COVID‐19 and 1341 non‐COVID‐19 images)	Binary: 98.09%
Khan et al. 43	2020	CoroNet	251 images (284 COVID‐19, 330 pneumonia bacterial, 327 Pneumonia viral, and 310 normal)	Binary: 89.60%
Ouchicha et al. 5	2020	CVDNet	2905 images (219 COVID‐19, 1345 Viral pneumonia , and 1341 normal)	Binary: 97.20% Multiple: 96.69%
Ozturk et al. 44	2020	DarkCovidNet	1127 images (127 COVID‐19, 500 viral pneumonia, and 500 normal)	Binary: 98.08% Multiple: 87.02%
Toraman et al. 45	2020	Convolutional CapsNet	2331 images (231 COVID‐19, 1050 viral pneumonia, and 1050 normal)	Binary: 97.24% Multiple: 84.22%
Ahmad et al. 46	2021	Ensemble deep learning	4200 images (1050 COVID‐19, 1050 bacterial pneumonia, 1050 viral pneumonia, and 1050 normal)	Multiple: 96.49%
Alam et al. 47	2021	CNN+HOG+VGG19	5090 images (1979 COVID‐19 and 3111 normal)	Binary: 99.49%
Aslan et al. 10	2021	CNN‐BiLSTM	2905 images (219 COVID‐19, 1345 viral pneumonia, and 1341 normal)	Multiple: 98.7%
Bhardwaj and Kaur 48	2021	Ensemble deep learning	10,000 images (2161 COVID‐19, 2022 viral pneumonia, and 5563 normal)	Binary: 98.33% Multiple: 92.36%
Gupta et al. 11	2021	InstaCovNet‐19	3047 images (361 COVID‐19, 1345 viral pneumonia, and 1341 normal)	Binary: 99.52% Multiple: 99.08%
Jain et al. 19	2021	Xception	6432 images (576 COVID‐19, 4273 viral pneumonia, and 1583 normal)	Multiple: 97.97%
Luz et al. 23	2021	EfficientNet‐B3	13,770 images (183 COVID‐19, 5521 viral pneumonia, and 8066 normal)	Multiple: 93.5%
Rahman et al. 22	2021	DenseNet201	18,479 images (3616 COVID‐19, 6012 non‐COVID, and 8851 normal)	Multiple: 95.11%
Current study	2021	LBP+SVM	3886 images (1200 COVID‐19, 1345 viral pneumonia, and 1341 normal)	Multiple: 99.36%

CONCLUSION

From past to present, infectious diseases are one of the most important threats to human health. COVID‐19 is an infectious disease caused by severe acute respiratory syndrome coronavirus. The rapid spread of the disease and the increase in mortality rates in many countries reveal that an effective treatment method should be developed. At this point, artificial intelligence and ML can contribute to this problem. Especially, there is a serious shortage of experts and traditional pneumonia and COVID‐19 have great similarities. ML aided diagnosing method could be an important milestone towards significantly reducing test time. This technique offers both a lower cost and a more accurate diagnosis and treatment for COVID‐19 and similar diseases. Lung x‐ray radiography is an effective tool for triaging non‐COVID‐19 patients with pneumonia to efficiently allocate hospital resources. However, there are many common features between x‐ray images of COVID‐19 and pneumonia caused by other viral infections, like common influenza. This similarity makes it difficult for radiologists to diagnose COVID‐19 cases.

Feature engineering is the process of extracting relevant features from data for a ML model. The count of features is as important as the features themselves. Well‐selected features facilitate subsequent modeling steps and increase the resulting model's ability to complete the desired task (e.g., detection, classification). Thus, we study on performances of feature extraction techniques and feature vectors of chest x‐ray images. In this study, we develop an original framework (HANDEFU) that supports handcrafted, deep, and fusion‐based feature extraction techniques for feature engineering. The novelty of this study is that image preprocessing and diverse feature extraction and classification techniques are assembled under an original framework. This framework could be used in different computer vision and ML problems. The framework has a dynamic structure. The user is able to build any model with handcrafted, deep, and fusion‐based feature extraction techniques and various classifiers through the HANDEFU framework. At present, the framework contains popular pretrained and deep learning techniques. Any feature extraction technique and model could then be added dynamically to the library of software at a later time upon request. Thus, the user can build a feature extraction method and classification to train the model. All performance evaluations on test data are performed with this software. This framework is utilized for diagnosing COVID‐19 from chest x‐ray images. In experimental studies, COVID‐19 prediction is performed by 27 different models through software. The superior performance with accuracy of 99.36% is obtained by LBP+SVM model. This model has the shortest execution time according to other methods. Because regional histograms in LBP give more efficient results for feature extraction.

In further studies, more handcrafted features‐based and deep learning‐based techniques, and more data will be studied and comparisons will be made. Furthermore, by using this framework in the background, it is planned to develop a mobile and web‐based system aimed at supporting health professionals in their efforts to detect cases of COVID‐19 and other diseases.