Machine Learning-Based Research for COVID-19 Detection, Diagnosis, and Prediction: A Survey.

The year 2020 experienced an unprecedented pandemic called COVID-19, which impacted the whole world. The absence of treatment has motivated research in all fields to deal with it. In Computer Science, contributions mainly include the development of methods for the diagnosis, detection, and prediction of COVID-19 cases. Data science and Machine Learning (ML) are the most widely used techniques in this area. This paper presents an overview of more than 160 ML-based approaches developed to combat COVID-19. They come from various sources like Elsevier, Springer, ArXiv, MedRxiv, and IEEE Xplore. They are analyzed and classified into two categories: Supervised Learning-based approaches and Deep Learning-based ones. In each category, the employed ML algorithm is specified and a number of used parameters is given. The parameters set for each of the algorithms are gathered in different tables. They include the type of the addressed problem (detection, diagnosis, or detection), the type of the analyzed data (Text data, X-ray images, CT images, Time series, Clinical data,...) and the evaluated metrics (accuracy, precision, sensitivity, specificity, F1-Score, and AUC). The study discusses the collected information and provides a number of statistics drawing a picture about the state of the art. Results show that Deep Learning is used in 79% of cases where 65% of them are based on the Convolutional Neural Network (CNN) and 17% use Specialized CNN. On his side, supervised learning is found in only 16% of the reviewed approaches and only Random Forest, Support Vector Machine (SVM) and Regression algorithms are employed.

Introduction

COVID-19 has led to one of the most disruptive disasters in the current century and is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The health system and economy of a large number of countries have been impacted. As per World Health Organization (WHO) data, there have been 225,024,781 confirmed cases of COVID-19, including 4,636,153 deaths as of 14 September 2021. Immediately, after its outbreak, several studies are conducted to understand the characteristics of this coronavirus.

It is argued that human-to-human transmission of SARS-CoV-2 is typically done via direct contacts and respiratory droplets [1]. On the other side, the incubation of the infection is estimated to a period of 2–14 days. This helps in controlling it and preventing the spread of COVID-19 is the primary intervention being used. Moreover, studies on clinical forms reveal the presence of asymptomatic carriers in the population and the most affected age groups [2]. After almost a year in this situation, and the high number of researches conducted in different disciplines to bring a relief, a huge amount of data is generated. Computer science researchers find themselves involved to provide their help. One of the first registered contributions is the visualization of data. The latter was mapped and/or plotted in graphs which allows to: (i) better track the propagation of the virus over the globe in general and country by country in particular (Fig. 1);

Fig. 1

Propagation of COVID-19 over the world

ii) better track the propagation of the pandemic over the time; iii) better estimate the number of confirmed cases and the number of deaths (Fig. 2a, b). Later, more advanced techniques based essentially on Artificial Intelligence (AI) are employed. Bringing AI to go against COVID-19 has served in the prevention and monitoring of infectious patients. In fact, by using geographical coordinates of people, some governments were able to limit their movements and locate people with whom they were in contact. The second aspect in which AI benefits is the ability to classify individuals whether they are affected or not. Finally, AI offers the ability to make a prediction on possible future contaminations. To this purpose, Machine Learning (ML), which is often confused with AI, is precisely used. Beyond the different ML algorithms, Neural Network (NN) is one of the most used to solve real-world problems which gives the emergence of Deep Learning (DL).

Fig. 2

Data-visualization for tracking COVID-19 progress

Deep learning is particularly suited to contexts where the data is complex and where there are large datasets available as it is the case with COVID-19.

In this context, the present paper gives an overview of the Machine Learning researches performed to handle COVID-19 data. It specifies for each of them the targeted objectives and the type of data used to achieve them.

To accomplish this study, we use Google scholar by employing the following search strings to build a database of COVID-19 related articles:We retain all articles in this field which:The result is impressive. In fact, since February 2020, several papers are published in this area every month. As we can see in Fig. 3, India and China seem to be those having the highest number of COVID-19 publications. However, many other countries showed a strong activity in the number of contributions. This is expected as the situation affects the entire world. The different papers appeared from various well-known publishers such as IEEE, Elsevier, Springer, ArXIv and many others as shown in Fig. 4.

Fig. 3

Number of COVID-19 published articles by countries

Fig. 4

Percentage of identified COVID-19 papers in different scientific publishers

COVID-19 detection using Machine learning;

COVID-19 detection using Deep learning;

COVID-19 detection using Artificial intelligence;

COVID-19 diagnosis using Machine learning;

COVID-19 diagnosis using Deep learning;

COVID-19 diagnosis using Artificial intelligence;

COVID-19 prediction using Machine learning;

Deep learning for COVID-19 prediction;

Artificial intelligence for COVID-19 prediction.

Are published in scientific journals;

Propose new algorithms to deal with COVID-19;

Have more than 4 pages;

Are written in English;

Represent complete versions when several are available;

Do not report the statistical tests used to assess the significance of the presented results.

Do not report details on the source of their data sets.

In this paper, the surveyed approaches are presented according to the Machine Learning classification given in Fig. 8. Techniques highlighted in yellow color are those employed in the different propositions to go against COVID-19. We show that most of them are based on Convolutional Neural Networks (CNN) which allows making Deep Learning. Almost half of these techniques use X-ray images. Nevertheless, several other data sources are used at different proportions as shown in Fig. 5. They include Computed Tomography (CT) images, Text data, Time series, Sounds, Coughing/Breathing videos, and even Blood Samples world cloud of the works we have summarized, reviewed, and analyzed in this paper can be seen in Fig. 6.

Fig. 8

Classification of Machine Learning Algorithms

Fig. 5

Proportion of the different data sources used in COVID-19 publications

Fig. 6

A world cloud of the works we have summarized, reviewed, and analyzed in this paper

There are similar surveys on AI and COVID-19 (e.g. in the works of Rasheed et al. [3], Shah et al. [4], Mehta et al. [5], Shinde et al. [6] and Chiroma et al. [7]). What makes this survey different is the focus on specialized Machine Learning techniques proposed globally to detect, diagnose, and predict COVID-19.

The remainder of this paper is organized as follows. In the second section, the definition of Deep Learning and its connection with AI and Machine Learning is given with descriptions of the most used algorithms. The third section presents a classification of the different approaches proposed to deal with COVID-19. They are illustrated by multiple tables highlighting the most important parameters of each of them. The fourth section discusses the results revealed from the conducted study in regard to the techniques used and their evaluation. It notes the limitations encountered and possible solutions to overcome them. The last section concludes the present article.

Artificial Intelligence, Machine Learning and Deep Learning

Artificial Intelligence (AI) as it is traditionally known is considered weak. Making it stronger results in making it capable of reproducing human behavior with consciousness, sensitivity and spirit. The appearance of Machine Learning (ML) was the means that made it possible to take a step towards achieving this objective. By definition, Machine Learning is a subfield of AI concerned with giving computers the ability to learn without being explicitly programmed. It is based on the principle of reproducing a behavior thanks to algorithms, themselves fed by a large amount of data. Faced with many situations, the algorithm learns which decision to make and creates a model. The machine can therefore automate the tasks according to the situations. The general process to carry out a Machine Learning requires a training dataset, a test dataset and an algorithm to generate a predictive model (Fig. 7). Four types of ML can be distinguished as we can see in Fig. 8.

Fig. 7

Machine learning prediction process

Supervised Learning

It is a form of machine learning that falls under artificial intelligence. The idea is to “guide” the algorithm on the way of learning based on pre-labeled examples of expected results. Artificial intelligence then learns from each example by adjusting its parameters to reduce the gap between the results obtained and the expected ones. The margin of error is thus reduced over the training sessions, with the aim of being able to generalize learning in the objective to predict the result of new cases [8, 9]. The output is called classification if labels are like discrete classes or regression if they are like continuous quantities. Within each category, there exists several algorithms [10, 11]. We define below those which was applied in the detection/prediction of COVID-19.

Linear Regression

Linea regression can be considered as one of the most conventional machine learning techniques [12], in which the best fit line/hyperplane for the available training data is determined using the minimum mean squared error function. This algorithm considers the predictive function as linear. Its general form is as follows: with a and b two constants. Y is the variable to be predicted, X the variable used to predict, a is the slope of the regression and b is the intercept, that is, the value of Y when X is zero.

Logistic Regression

Despite its name, Logistic Regression [13] can be employed to perform regression as classification. It is based on the sigmoid predictive function defined as: where z is a linear function. The function returns a probability score P between 0 and 1. In order to map this to two discrete classes ( 0 or 1), a threshold value is fixed. The predicted class is equal to 1 if , to 0 otherwise.

Support Vector Machine (SVM)

Similar to the previously defined algorithms, the idea behind SVM [14, 15] is to distinctly classifies data points by finding an hyperplane in an N-dimensional space. Since there are several possibilities to choose the hyperplane, in SVM a margin distance is calculated between data points of the two classes to separate. The objective is to maximize the value of this margin to get a clear decision boundary helping in the classification of future data points.

Decision Tree

A Decision Tree [16] is an algorithm that seeks to partition the individuals into groups of individuals as similar as possible from the point of view of the variable to be predicted. The result of the algorithm produces a tree that reveals hierarchical relationships between the variables. An iterative process is used where at each iteration a sub-population of individuals is obtained by choosing the explanatory variable which allows the best separation of individuals. The algorithm stops when no more split is possible.

Random Forest Algorithms

Random Forest Algorithms are methods that provide predictive models for classification and regression [17, 18]. They are composed of a large number of Decision Tree blocks used as individual predictors. The fundamental idea behind the method is that instead of trying to get an optimized method all at once, several predictors are generated and their different predictions are pooled. The final predicted class is the one having the most votes.

Artificial Neural Network (ANN)

Artificial Neural Networks is a popular Supervised classification algorithm trying to mimic the way human brain works. It is often used whenever there is abundant labeled training data with many features [19]. The network calculates from the input a score (or a probability) to belong to each class. The class assigned to the input object corresponds to the one with the highest score. A Neural Network is a system made up of neurons. It is divided into several layers connected to each other where the output of one layer corresponds to the input of the next one [20, 21]. The calculation of the final score is based on the calculation of a linear function from the layers weights and an activation function. The weights values are randomly assigned to each input at the beginning and then are learned (updated) by backpropagation of the gradient to minimize the loss function associated with the final layer. The optimization is done with a gradient descent technique [22].

Unsupervised Learning

Unsupervised learning is a type of self-organized learning that learns and creates models from unlabeled training datasets (unlike Supervised Learning). There are two practices in Unsupervised Learning. The first one is the clustering, which is the fact of gathering similar data in homogeneous groups. It is performed by applying one of the many existing clustering algorithms [23]: K-means, Hierarchical clustering, Hidden Markov, etc. The second practice is the dimensionality reduction [24] which consists of the reduction of features in highly dimensional data. The purpose is to extract new features and to find the best linear transformation representing maximum data points by guaranteeing a minimum loss of information.

Deep Learning

As illustrated in Fig. 9, Deep Learning [25, 26] is a branch of AI that focuses on creating large Neural Network models that are capable of making decision based on Machine Learning models, it is a Neural Networks with many hidden neural layers. Indeed, it has been observed that the addition of layers of neurons has a great impact on the quality of the results obtained.

Fig. 9

Classification of Machine Learning Approaches

There are many different deep learning algorithms other than ANN. In the following we define the most used ones and which are applied in the context of COVID-19.

Convolutional Neural Network (CNN)

Convolutional Neural Networks or ConvNets [27, 28] is a type of ANN used to make a Deep Learning that is able to categorize information from the simplest to the most complex one. They consist of a multilayer stack of neurons as well as mathematical functions with several adjustable parameters, which preprocess small amounts of information. Convolutional networks are characterized by their first convolutional layers (usually one to three). They seek to identify the presence of a basic and abstract pattern in an object. Successive layers can use this information to distinguish objects from each other (classification / recognition).

Recurrent Neural Network (RNN)

Recurrent Neural Network [29, 30] is also a type of ANN used to make a Deep Learning where information can move in both directions between the deep layers and the first layers. This allows it to keep information from the near past in memory. For this reason, RNN is particularly suited to applications involving context, and more particularly to the processing of temporal sequences such as learning and signal generation. However, for applications involving long time differences (typically the classification of video sequences), this “short-term memory” is not sufficient because forgetting begins after about fifty iterations.

Generative Adversarial Network (GAN)

GAN [31] is a Deep Learning technique. It is based on the competition of two networks within a framework. These two networks are called “generator” and “discriminator”. The generator is a type of CNN whose role is to create new instances of an object which means that outputs are produced without it being possible to determine if they are false. On the other hand, the discriminator is a “deconvolutive” neural network that determines the authenticity of the object (whether or not it is part of a data set).

Reinforcement Learning

Reinforcement Learning [32, 33] is a method of learning for machine learning models. Basically, this method lets the algorithm learn from its own mistakes. To learn how to make the right decisions, the AI program is directly confronted with choices. If it is wrong, it is “penalized”. On the contrary, if it makes the right decision, it is “rewarded”. In order to get more and more rewards, AI will therefore do its best to optimize its decision-making.

Overview of Machine Learning approaches used to combat COVID-19

Zhang et al. [34] applied Support Vector Machine (SVM) model for COVID-19 cases detection and classification. The clinical information and blood/urine test data were used in their work to validate SVM’s performance. Simulation results demonstrated the effectiveness of the SVM model by achieving an accuracy of 81.48%, sensitivity of 83.33%, and specificity of 100%.

Hassanien et al. [35] proposed a new approach based on the hybridization of SVM with Multi-Level Thresholding for detecting COVID-19 infected patients from X-ray images. The performance of the hybrid approach was evaluated using 40 contrast-enhanced lungs X-ray images (15 normal and 25 with COVID-19). A similar work was done by Sethy et al. [36], in which a combined approach based on the combination of SVM with 13 pre-trained CNN models for COVID-19 detection from chest X-ray images were proposed. Experimental results showed that ResNet50 combined with SVM outperforms other CCN models combined with SVM by achieving an average classification accuracy of 95.33%.

Sun et al. [37] used SVM model for predicting the COVID-19 patients with severe/critical symptoms. 220 clinical/laboratory observations records and 336 cases of patients infected COVID-19 divided into training and testing datasets were used to validate the performance of the SVM model. Simulation results showed that the SVM model achieves an Area Under Curve (AUC) of 0.9996 and 0.9757 in the training and testing dataset, respectively.

Singh et al. [38] used four machine learning approaches (SVM with Bagging Ensemble, CNN, Extreme Learning Machine (ELM), Online Sequential ELM (OS-ELM)) for automatic detection of COVID-19 cases. The performance of the proposed approaches was tested using datasets of 702 CT scan images (344with COVID-19 and 358 normal). Experimental results revealed the efficiency of SVM with Bagging Ensemble by obtaining an accuracy, precision, sensitivity, specificity, F1-score, and AUC of 95.70%, 95.50%, 96.30%, 94.80%, 95.90%, and 95.80%, respectively.

Singh et al. [39] proposed Least Square-SVM (LS-SVM) and Autoregressive Integrated Moving Average (ARIMA) for the prediction of COVID-19 cases. A dataset of COVID-19 confirmed cases collected from five the most affected countries1 was used to validate the proposed models. It was demonstrated that the LS-SVM model outperforms the ARIMA model by obtaining an accuracy of 80%.

Nour et al. [40] applied machine learning approaches such as SVM, Decision tree (DT), and KNN for automatic detection of positive COVID-19 cases. The performance of the proposed approaches was validated on a public COVID-19 radiology database divided into training and test sets with 70% and 30% rates, respectively.

Tabrizchi et al. [41] used SVM with Naive Bayes (NB), Gradient boosting decision tree (GBDT), AdaBoost, CNN, and Multilayer perceptron (MLP) for rapid diagnosis of COVID-19. A dataset of 980 CT scan images (430 with COVID-19 and 550 normal) was used in the simulation and results showed that SVM outperforms other machine-learning approaches by achieving an average accuracy, precision, sensitivity, and F1-score of 99.20%, 98.19%, 100%, and 99.0%, respectively.

Regression Approaches

Yue et al. [42] used a linear regression model for the prediction of COVID-19 infected patients. CT images of 52 patients collected from five hospitals in Ankang, Lishui, Zhenjiang, Lanzhou, and Linxia were used to evaluate the performance of the regression model. Simulation results demonstrated that the linear regression model outperforms the Random Forest algorithm.

Another similar work was done by Shi et al. [43], in which a least absolute shrinkage and selection operator (LASSO) logistic regression model was proposed. The effectiveness of the proposed model was evaluated based on CT images taken from 196 patients (151 non-severe patients and 45 severe patients). Experimental results showed the high performance of the proposed model compared to quantitative CT parameters and PSI score by achieving an accuracy of 82.70%, sensitivity of 82.20%, specificity of 82.80%, and AUC of 89%

Yan et al. [44] proposed a supervised regression model, called XGBoost, for predicting COVID-19 patients. A database of blood samples of 485 infected patients in the region of Wuhan, China was used in simulations and results showed that XGBoost gives good performance by achieving an overall accuracy of 90% in the detection of patients with COVID-19.

Salama et al. [45] used the linear regression model with SVM and ANN for the prediction of COVID-19 infected patients. The effectiveness of the proposed models was assessed based on the Epidemiological dataset collected from many health reports of real-time cases. Simulation results demonstrated that SVM has the lowest mean absolute error with the value of 0.21, while the regression model has the lowest root mean squared error with a value of 0.46.

Gupta et al. [46] proposed a linear regression technique with mathematical SEIR (Susceptible, Exposed, Infectious, Recovered) model for COVID-19 outbreak predictions. It was tested using data collected from John Hopkins University repository taking into account the root mean squared log error (RMSLE) metric. Simulation results showed that SEIR model has the lowest RMSLE with the value of 1.52.

In the work of Chen and Liu [47], Logistic Regression with Random Forest, Partial Least Squares Regression (PLSR), Elastic Net, and Bagged Flexible Discriminant Analysis (BFDA) were proposed for predicting the severity of COVID-19 patients. The efficiency of the proposed models was evaluated using data of 183 severely infected COVID-19 patients and results showed that the logistic regression model outperforms other machine learning models by achieving a sensitivity of 89.20%, specificity of 68.70%, and AUC of 89.20%.

Another similar work was done by Ribeiro et al. [48], in which six machine learning approaches such as stacking-ensemble learning (SEL), support vector regression (SVR), cubist regression (CUBIST), auto-regressive integrated moving average (ARIMA), ridge regression (RIDGE), and random forest (RF) were employed for prediction purposes in COVID-19 datasets.

Yadav et al. [49] used three machine learning approaches (Linear Regression, Polynomial Regression, and SVR) for COVID-19 epidemic prediction and analysis. A dataset containing the total number of COVID19 positive cases was collected from different countries such as South Korea, China, US, India, and Italy. Results showed the superiority of SVR compared to Linear Regression and Polynomial Regression. The average accuracy for SVR, Linear Regression, and Polynomial Regression are 99.47%, 65.01%, and 98.82%, respectively.

Matos et al. [50] proposed four linear regression models (Penalized binomial regression (PBR, Conditional inference trees (CIR), Generalised linear (GL), and SVM with linear kernel) for COVID-19 diagnosis. CT images and Clinical data collected from 106 patients were used in the simulation and results showed that SVM with linear kernel gives better results compared to other models by providing an accuracy of 0.88, sensitivity of 0.90, specificity of 0.87, and AUC of 0.92.

Khanday et al. [51] proposed Logistic regression with six machine learning approaches (Adaboost, Stochastic Gradient Boosting, Decision Tree, SVM, Multinomial Naïve Bayes, and Random Forest) for COVID-19 detection and classification. It was evaluated using 212 clinical reports divided into four classes including COVID, ARDS, SARS, and Both (COVID, ARDS). Simulation results showed that logistic regression provides excellent performance by obtaining 94% of precision, 96% of sensitivity, accuracy of 96.20%, and 95% of F1-score.

Yang et al. [52] proposed Gradient Boosted Decision Tree (GBDT) with Decision Tree, Logistic Regression, and Random Forest for COVID-19 diagnosis. 27 routine laboratory tests collected from the New York Presbyterian Hospital/Weill Cornell Medicine (NYPH/WCM) were used to evaluate this technique. Experimental results revealed the efficiency of GBDT by achieving a sensitivity, specificity, and AUC of 76.10 %, 80.80%, and 85.40%, respectively.

Saqib [53] developed a novel model (PBRR) by combining Bayesian Ridge Regression (BRR) with n-degree Polynomial for forecasting COVID-19 outbreak progression. The performance of the PBRR model was validated using public datasets collected from John Hopkins University available until 11th May 2020. Experimental results revealed the good performance of PBRR with an average accuracy of 91%.

Random Forest Algorithm

Shi et al. [54] proposed an infection Size Aware Random Forest method (iSARF) for diagnosis of COVID-19. A dataset of 1020 CT images (1658 with COVID-19, and 1027 with pneumonia) was used to assess the performance of iSARF. Simulation results demonstrated that iSARF provides good performance by yielding the sensitivity of 90.7%, specificity of 83.30%, and accuracy of 87.90% under five-fold cross-validation.

Iwendi et al. [55] combined RF model with AdaBoost algorithm for COVID-19 disease severity prediction. The efficiency of the boosted RF model was evaluated based on COVID-19 patient’s geographical, travel, health, and demographic data. Boosted RF model gives an accuracy of 94% and F1-Score of 86% on the dataset used.

In the work of Brinati et al. [56], seven machine learning approaches (Random Forest, Logistic Regression, KNN, Decision Tree, Extremely Randomized Trees, Naïve Bayes, and SVM) were proposed for the identification of COVID-19 positive patients. Routine blood exams collected from 279 patients were used in the simulation and results demonstrated the feasibility and effectiveness of the Random Forest algorithm by achieving an accuracy, precision, sensitivity, specificity, and AUC of 82%, 83%, 92%, 65%, and 84%, respectively.

The main characteristics of the predefined Supervised Learning approaches are given in Table 1.

Table 1

Summary of supervised learning approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[34]	SVM	COVID-19 detection	Text data	2	81.48	–	83.33	100	–	–
[35]	SVM with Multi-Level Thresholding	COVID-19 detection	X-ray images	2	97.48	–	95.76	99.70	–	–
[57]	SVM with DT	COVID-19 prediction	X-ray images	2	94.99	–	89.2	93.22	–	–
[36]	SVM with ResNet50	COVID-19 detection	X-ray images	3	95.33	–	95.33	–	95.34	–
[58]	SVM with CNN and RF	COVID-19 detection	X-ray images	2	95.2	100	93.3	100	–	–
[37]	SVM	COVID-19 prediction	Text	2	–	–	100		–	97.57
				2	–	–	93.33	–	–	99.96
[38]	Four machine learning approaches (SVM with Bagging Ensemble, CNN, ELM, OS-ELM	COVID-19 detection	CT images	2	95.70	95.50	96.30	94.80	95.90	95.80
[39]	LS-SVM and ARIMA models	COVID-19 prediction	Time series	2	80	–	–	–	–	–
[40]	SVM with DT and KNN	COVID-19 detection	X-ray images	3	98.97	–	89.39	99.75	96.72	–
[41]	Machine learning approaches (SVM, Naive Bayes, GBDT, AdaBoost, CNN, and MLP)	COVID-19 diagnosis	CT images	2	99.20	98.19	100	–	99.0	–
[42]	Linear regression model and Random Forest	COVID-19 prediction	CT images	2	97	–	100	89	–	–
[43]	Logistic regression model	COVID-19 prediction	CT images	2	82.70	–	82.20	82.80	–	89
[44]	XGBoost	COVID-19 prediction	Time series	3	90	100	–	97	98	–
[45]	Linear regression model with SVM and ANN	Prediction of COVID-19 patients	Text	–	–	–	–	–	–	–
[46]	Linear regression and SEIR	COVID-19 outbreak predictions	Time series	–	–	–	–	–	–	–
[47]	Logistic Regression with Random Forest, PLSR, Elastic Net, and BFDA	COVID-19 prediction	Time series	2	–	–	89.20	68.70	–	89.20
[48]	ML approaches (SVR, SEL, ARIMA, CUBIST, RF, RIDGE	COVID-19 prediction	Time series	3	–	–	–	–	–	–
[49]	Machine learning approaches (SVR, Linear Regression, and Polynomial Regression)	COVID-19 epidemic prediction and analysis	Text	5	99.47	–	–	–	–	–
[50]	Linear regression models (PBR, CIR, GL, and SVM with linear kernel)	COVID-19 diagnosis and severity prediction	CT images and clinical data	2	88	–	90	87	–	92
[59]	Decision Tree	COVID-19 diagnosis	X-ray images	2	98	97	99	97	–	98
[51]	Seven machine learning models (Logistic regression, Adaboost, SVM, SGB, Decision Tree, MNB, and Random Forest)	COVID-19 detection and classification	Text	4	96.20	94	96	–	95	–
[52]	GBDT, Decision Tree, Logistic Regression, and Random Forest	COVID-19 diagnosis	Text	2	–	–	76.1	80.8	–	85.4
[45]	Linear regression model with SVM and ANN	Prediction of COVID-19 patients	Text	–	–	–	–	–	–	–
[53]	PBRR	COVID-19 prediction	Text	–	91	–	–	–	–	–
[54]	iSARF	COVID-19 diagnosis and classification	CT images	2	87.90	–	90.70	83.30	–	–
[55]	Fine–tuned Random Forest model with AdaBoost algorithm	COVID-19 disease severity prediction	Text	5	94	100	75	–	86	–
[56]	ML approaches (Random Forest, Decision Tree, Extremely Randomized Trees, kNN, Logistic Regression, Naïve Bayes, and SVM)	COVID-19 detection	Text	2	82	83	92	65	–	84

Deep Learning Approaches

The most applied method to detect, predict and diagnostic COVID-19 are based on Deep Learning with its different techniques. In the following, we summarize the found approaches in respect of the classification given in Fig. 8. We gather in Tables 2, 3, 4, 5 and 6 are their main features.

Table 2

Summary of convolutional neural networks (CNN) approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[60]	ResNet34	COVID-19 diagnosis	CT images	2	73.10	–	74	67	–	–
[62]	AlexNet	COVID-19 diagnosis	X-ray images	2	98	–	100	96	–	–
			CT images	2	94.10	–	90	100	–	–
[63]	Eight DL models (FCN–8 s, UNet, VNet, 3D UNet++, DPN–92, Inceptionv3, ResNet50, and Attention ResNet50	COVID-19 detection	CT images	2	–	–	97.40	92.20	–	99.10
[64]	UNet++	COVID-19 detection	CT images	2	95.24	–	100	93.55	–	–
				2	98.85	–	94.34	99.16	–	–
[65]	Five CNN models (VGG19, MobileNetv2, Inception, Xception, and InceptionResNetv2)	COVID-19 detection and classification	X-ray images	2	96.78	–	98.66	96.46	–	–
[66]	CNN model	Screening of COVID-19 cases	X-ray images	2	–	–	96	70.65	–	95.18
[67]	Bayesian CNN with Dropweights	COVID-19 diagnosis	X-ray images	4	89.82	–	–	–	–	–
[68]	CAPSNET	COVID-19 diagnosis	X-ray images	2	97.23	97.08	97.42	97.04	97.24	–
				3	84.22	84.61	84.22	91.79	84.21	–
[69]	Six deep learning models (ResNet34, ResNet50, DenseNet169, VGG-19, InceptionResNetV2, and RNN-LSTM)	COVID-19 detection	X-Ray images	3	95.72	–	–	–	–	–
[129]	DenseNet–121	COVID-19 prediction	CT images	2	92	–	–	–	–	–
[70]	Ten deep CNN models (AlexNet, VGG16, VGG19, SqueezeNet, GoogleNet, MobileNetV2, ResNet18, ResNet50, ResNet101, and Xception)	COVID-19 diagnosis	CT images	2	99.51	–	100	99.02	–	99.4
[71]	ResNet+	COVID-19 diagnosis	CT images	3	86.70	80.80	81.50	–	81.10	–
[72]	Deep CNN models (AlexNet and Inception-V4)	COVID-19 diagnosis	CT images	2	94.74	–	87.37	87.45	–	–
[73]	EfficientNetB4 with fully connected neural network	COVID-19 detection and classification	CT images	2	96	–	95	96	–	–
			External dataset		87	–	89	86	–	–
[75]	CNN model with MODE technique	COVID-19 classification	CT images	2	93.50	–	91	91	89.90	–
[76]	Shallow light-weight CNN model	COVID-19 detection	X-ray images	2	96.92	100	94.20	100	97.01	–
[78]	Deep CNN models (MobileNetV2, SqueezeNet) combined with SVM	COVID-19 detection	X-Ray images	3	99.27	100	95	100	97.43	–
[79]	ResNet-50	COVID-19 detection and classification	CT images and clinical data	2	93.02	95.19	91.48	94.78	–	–
[80]	Nine deep CNN models(baseline CNN, VGG16, VGG19, DenseNet201, InceptionResNetV2, InceptionV3, Xception, Resnet50, and MobileNetV2)	COVID-19 classification	X-Ray & CT images	3	92.60	93.85	82.80	97.37	87.98	–
[81]	Six CNN models (Unet, DRUNET, FCN, SegNet, 3D ResNet18, and DeepLabv3)	COVID-19 diagnosis	CT images and metadata	3	92.49	–	94.93	91.13	–	97.97
[130]	Five CNN models (VGG19, ResNet50 V2, Densenet121, Inception V3, and COVID–Net)	COVID-19 diagnosis	X-Ray images	3	–	–	–	–	–	95.3
[82]	Five CNN models (VGG16, InceptionV3, Xception, DenseNet201, and NasNetmobile)	COVID-19 detection	X-ray images	2	99.26	–	–	–	–	–
[83]	Modified InceptionV3	COVID-19 screening	X-ray images	4	76	–	93	91.80	–	93
[85]	Four pre–trained CNN models (ResNet18, ResNet50, ResNet101, and SqueezeNet)	COVID-19 detection	CT images	2	99.40	99	100	98.60	99.50	99.65
[86]	ResNet18	COVID-19 diagnosis	X-ray images	5	88.90	83.40	85.90	96.40	84.40	–
				7	87.66	–	–	–	–	–
[131]	Five CNN models (VGG16, VGG19, InceptionResNetV2, InceptionV3, and Xception)	COVID-19 diagnosis	X-ray images	3	84.1	–	87.7	–	–	97.4
[132]	Three pre-trained CNN models (GoogleNet, ResNet18, and ResNet50) with grid search	COVID-19 detection	X-ray images	4	97.69	95.95	97.26	97.90	96.60	–
[87]	seven pre-trained CNN models (VGG16, VGG19, DenseNet201, InceptionResNetV2, InceptionV3, Resnet50, and MobileNetV2)	COVID-19 detection	X-ray and CT images	4	92.60	93.85	82.80	97.37	87.98	–
[89]	MobileNetv2	COVID-19 detection and classification	X-Ray images	2	99.18	–	97.36	99.42	–	–
[88]	Eight CNN models (CheXNet, DenseNet201, RestNet18, MobileNetv2, InceptionV3, VGG19, ResNet101, and SqueezNet)	COVID-19 detection	X-Ray images	3	97.74	96.61	96.61	98.31	96.61	–
[90]	Modified deep CNN model (combination of Xception with ReNet50V2)	COVID-19 detection	X-ray images	3	91.4	72.8	87.3	94.2	–	–
[91]	DeTraC	COVID-19 detection	X-Ray images	3	95.12	–	97.91	91.87	–	–
[92]	COVID-CAPS	COVID-19 identification and diagnosis	X-Ray images	4	98.30	–	80	98.60	–	–
[94]	VGG16	COVID-19 detection	X-ray images	3	97	–	92	96	92	–
[95]	Deep CNN model	COVID-19 diagnosis	CT images	4	–	–	90.19	95.76	–	97.17
[96]	Truncated InceptionNet	COVID-19 detection	X-ray images	2	98.77	99	95	99	97	–
[97]	Deep InceptionV3	COVID-19 detection	X-ray images	3	98	–	–	–	–	–
[98]	Five CNN models (baseline ResNet, Inceptionv3, InceptionResNetv2, DenseNet169, and NASNetLarge)	COVID-19 diagnosis and classification	X-ray and CT images	2	98	88	90	95	89	–
				3	96	93	90	94	91	–
[101]	Three CNN models (VGG16, DenseNet161, and ResNet18)	COVID-19 diagnosis and analysis	X-ray images	2	98.9	–	–	–	–	–
				3	95.9	–	–	–	–	–
[102]	Eight pre-trained CNN models (VGG16, VGG19, InceptionV3, Xception, InceptionResNetV2, MobileNetV2, DenseNet201, NasNetmobile)	COVID-19 detection	X-Ray images	3	99.01	99.01	99.01	–	99.01	99.72
[133]	Modified AlexNet	COVID-19 detection	X-rays images	3	–	–	94.44	97.27	–	–
[134]	AlexNet, SquzeeNet, ResNet, and DenseNet	COVID-19 detection	X-rays images	2	95	–	–	–	–	–
[135]	ResNet34 and ResNet50	COVID-19 detection	X-rays images	3	72.38	–	–	–	–	–
[106]	3D CNN-based network models	COVID-19 diagnosis	CT images	2	–	–	–	–	–	70
[107]	Four pre-trained CNN models (RESNET50, VGG19, DENSENET121, and INCEPTIONV3)	COVID-19 detection	X-ray images	3	98.71	98	98	–	97.66	–
[136]	Four CNN models (DenseNet169, VGG16, ResNet50, InceptionV3, VGG19, and CTnet10)	COVID-19 diagnosis	CT images	2	94.52	–	–	–	–	–
[103]	Modified EfficientNet	COVID-19 detection and diagnosis	X-Ray images	3	93.9	100	96.8	–	–	–
[136]	Four CNN models (DenseNet169, VGG16, ResNet50, InceptionV3, VGG19, and CTnet10)	COVID-19 diagnosis	CT images	2	94.52	–	–	–	–	–
[103]	Modified EfficientNet	COVID-19 detection and diagnosis	X-ray images	3	93.9	100	96.8	–	–	–
[104]	DenseNet201	COVID-19 detection and diagnosis	CT images	2	96.25	96.29	96.29	96.21	96.29	–
[105]	ResNet50	COVID-19 detection	CT images	3	91.0	–	92.1	90.29	–	–
[109]	Xception	COVID-19 detection	X-ray images	2	97.40	–	97.09	97.29	96.96	–
[108]	Three CNN models (VGG16, VGG19, and ResNet50)	COVID-19 detection	X-ray images	3	98.79	–	–	–	–	–
				3	98.12	–	–	–	–	–
[137]	Seven CNN models (VGG, ResNet, MobileNet, DenseNet, Xception, Attention, and Residual Attention Network)	COVID–19 Screening	X-ray images	2	98	96	100	96	–	–
[110]	15 different CNN models	COVID–19 cases identification	X-ray images	3	89.3	90	89	–	90	–
[111]	EfficientNetB0, 2D curvelet transformation, and CSSA	COVID–19 detection	X-ray images	3	99.69	99.62	99.44	99.81	99.53	–
[112]	MVPNet	COVID–19 detection	CT images	3	98	–	100	65	97	–
[138]	VGG16	COVID–19 diagnosis	X-ray images	3	86	86	86	93	86	–
[139]	Modified AlexNet	COVID–19 detection	X-rays images	3	–	–	94.44	97.27	–	–
[113]	Deep CNN models (EfficientNet and MixNet)	COVID–19 detection	X-ray images	3	95.81	96.80	92.40	–	94.50	–
				3	96.64	96.80	70	–	77.80	–
[114]	Four CNN models( VGG19, DenseNet121, InceptionV3, and InceptionResNetV2) and RNN	COVID–19 diagnosis	X-ray Images	3	99.90	–	99.80	99.80	–	99.90
[115]	Joint CNN model with SVM, random forest, and MLP classifiers	COVID–19 diagnosis	CT images and clinical data	2	83.50	81.90	84.30	82.80	–	–
[116]	Four CNN models (DenseNet121, ResNet50, VGG16, and VGG19)	COVID–19 diagnosis	X-ray images	2	99.33	–	100	98.77	99.27	–
[117]	Three deep CNN models (VGG16, Resnet50, and InceptionV3) and Haralick features	COVID–19 detection	X-ray and CT images	3	93	91	90	–	–	–
[118]	Five Deep learning models (VGG, DenseNet, AlexNet, MobileNet, ResNet, and Capsule Network) with blockchain and federated–learning technology	COVID–19 detection	CT images	3	83	83	96.70	–	–	–
[140]	AlexNet	COVID–19 diagnosis	X-ray images	2	–	–	–	–	–	99.97
[119]	Deep CNN models (modified VGG16, ResNet50, and EfficientNetB0)	COVID–19 detection	X-ray images	3	96.80	–	–	–	–	–
[120]	Five multi-CNN models (Squeezenet, Darknet53, MobilenetV2, Xception, and Shufflenet)	COVID-19 detection	X-ray images	2	91.16	–	–	–	–	96.30
				2	97.44	–	–	–	–	91.10
[121]	Five deep CNN models (ResNet18, ResNet50, ResNet101, VGG16, and VGG19) with SVM and kernel functions	COVID–19 detection	X-ray images	2	94.74	–	91.00	98.89	94.79	99.90
[122]	OptCoNet	COVID–19 diagnosis	X-ray images	3	97.78	92.88	97.75	96.25	95.25	–
[123]	Four deep learning CNN models (Inception V4, VGG 19, ResNetV2 152, and DenseNet)	COVID–19 detection	X-ray Images	2	93	–	–	–	–	–
[124]	VGG–16 with the attention module	COVID–19 detection and classification	X-ray images	3	79.58	91	77	–	83	–
				4	85.43	92	95	–	93	–
				5	87.49	89	92	–	90	–
[125]	Three CNN models (InceptionV3, Xception, and ResNeXt)	COVID–19 detection and analysis	X-ray images	3	97.97	99	92	–	95	–
				3	100	100	100	–	100	–
[126]	CNN models with local binary pattern and dual tree complex wavelet transform	COVID–19 detection	X-ray images	2	98.43	–	99.47	98	98.81	99.90
				2	98.91	–	99.20	99.39	98.28	99.91
[127]	Three CNN models (Resnet50, Shufflenet, and Mobilenet) with GAN	COVID–19 detection	CT images	2	80.82	80.78	–	80.92	80.85	–
[128]	lightweight CNN–tailored deep neural network	COVID–19 detection	X-ray images	2	96.13	93.30	99.40	92.86	96.25	99.08
			CT images	2	95.83	98.13	93.45	98.21	95.73	97.31
[141]	Three deep learning models (CNN, LSTM, and multi–head attention) with Bayesian optimization	COVID–19 prediction	Time series	2	–	–	–	–	–	–
[142]	Nine CNN models (AlexNet, GoogleNet, ResNet50, SeResNet50, DenseNet121, InceptionV4, InceptionResNetV2, ResNeXt50, and SeResNeXt50)	COVID–19 detection	X-ray images	2	98.36	95.76	99.11	98.02	97.4	–
				3	96.99	87.36	94.67	97.43	90.86	–
				3	96.40	84.22	94.67	96.72	89.14	–
				4	95.56	87.09	97.17	95.01	91.85	–
[143]	Six CNN models (SqueezeNet, ResNet, ShuffleNet, DenseNet, InceptionV3, Xception)	COVID–19 detection	CT images	2	99.40	99.60	99.80	99.60	99.40	–
				3	92.90	91.30	93.70	92.20	92.50	–
[61]	Five CNN models (ResNet50, ResNet101, ResNet152, InceptionV3, and Inception–ResNetV2)	COVID–19 detection	X-ray images	4	99.70	98.30	98.80	99.80	98.50	–
[144]	Five CNN models (AlexNet, VGG–16, ResNet50, ResNet101, and ResNet152	COVID–19 detection	X-ray images	4	–	–	–	–	–	96
[145]	Three CNN models (Inception V4, DenseNet161, and ResNet18	COVID–19 diagnosis	X-ray images	3	96.80	95.70	–	–	98.40	98.30
[74]	Deep CNN (Alexnet, Googlenet, and Restnet18) with GAN	COVID–19 detection	X-ray images	2	100	98.20	99.10	–	–	–
			CT images	3	85.2	85.2	85.2	–	85.2	–
			CT images	4	80.6	84.17	80.6	–	82.32	–
[146]	Four CNN models (ResNet18, ResNet50, SqueezeNet, and DenseNet121)	Predicting COVID–19	X-ray images	2	–	–	100	95.6	–	99.6
[77]	CNN model with ranking method and SVM	COVID–19 classification	CT images	2	98.27	97.63	98.93	97.60	99.01	98.28

Table 3

Summary of Recurrent Neural Networks (RNN) approaches for detection, diagnosis, and prediction of COVID-19 case

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[147]	LSTM with NLP	COVID-19 classification	Text	–	–	–	–	–	–	–
[153]	LSTM	COVID–19 Forecasting	Text	–	–	–	–	–	–	–
[154]	LSTM	Forecasting COVID–19 patients	Time series	–	–	–	–	–	–	–
[148]	LSTM	forecasting of COVID–19 cases	Times series	2	–	–	–	–	–	–
[149]	BiGRU–AT model	COVID–19 detection and diagnosis	Breathing/Thermal data	2	83.69	–	90.23	76.31	84.61	–
[150]	LSTM with ResNext+ and slice attention module	COVID–19 detection	CT images	2	77.60	81.90	96.54	85.50	79.30	81.40
[151]	LSTM with CNN	COVID–19 detection	X-ray images	3	99.20	–	99.30	99.20	98.90	–
[152]	BiLSTM with mAlexNet	COVID–19 detection	X-ray images	3	98.70	98.77	98.76	99.33	98.76	99

Table 4

Summary of Specialized CNN approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[155]	DRE-Net	COVID-19 diagnosis	CT images	2	94	96	93	–	94	99
[156]	COVNet model	COVID–19 detection	CT images	3	–	–	90	96	–	96
[157]	3D deep CNN model (DeCoVNet)	COVID–19 detection	CT images	2	90.10	84	90.70	91.10	–	–
[158]	Deep Bayes–SqueezeNet–based system (COVIDiagnosis–Net)	COVID–19 detection and diagnosis	X-ray images	3	98.26	–	–	99.13	98.25	–
[173]	COVIDXception–Net	COVID–19 detection and diagnosis	X-ray images	3	94	95	–	99.7	94	94
[159]	CNN–based DarkCovidNet model	Detection and classification of COVID–19 cases	X-ray images	3	98.08	98.03	95.13	95.30	96.51	–
[161]	Covid–Net	COVID–19 detection and classification	X-ray images	3	93.30	98.90	91	–	–	–
[162]	POCOVID–Net	COVID–19 detection	Sample videos	3	89	88	96	79	92	–
[174]	CovidCTNet with BCDU–Net	Identification of COVID–19 cases	CT images	3	91.66	–	87.5	94	–	95
[175]	ai–corona deep learning model with EfficientNetB3	COVID–19 diagnosis	CT images	2	96.40	–	92.40	98.30	95.30	98.90
[163]	COVID–19Net and DenseNet121–FPN	COVID–19 detection	CT images	2	78.32	–	80.39	76.61	77.0	87
[164]	CoroNet model	COVID–19 detection and diagnosis	X-ray images	2	99	98.30	99.30	98.60	98.50	–
				3	94.59	95	96.90	97.50	95.60	–
				4	89.5	90	89.90	96.40	89.80	–
[165]	CovXNeT	COVID–19 detection	X-ray images	2	98.1	98	98.50	97.90	98.30	–
				3	95.10	94.90	96.10	94.30	95.50	–
				4	91.70	92.90	92.10	93.60	92.60	–
[166]	COVIDLite	detection of COVID–19 Cases	X-ray images	2	99.58	100	99.58	99.34	99.79	100
				3	96.43	97	96	97.89	96	99
[167]	ReCoNet	COVID–19 detection	X-ray images	3	97.48	–	96.39	97.53	–	–
[176]	TV–UNet	COVID–19 detection	CT images	3	86.40	87.10	–	–	–	–
[177]	COVIDPEN	COVID–19 detection	X-ray images	2	96	92	96	–	94	92
			CT images	2	85	81	92	–	86	84
[169]	COVID–CXNET	COVID–19 detection	X-ray images	2	99.04	–	–	–	96	–
[170]	COVIDetectioNet model with AlexNet and SVM	COVID–19 diagnosis and classification	X-ray images	3	99.18	–	–	–	–	–
[178]	COVIDetectioNet	COVID–19 detection	X-ray images	3	96.58	96.58	96.59	–	96.58	–
[171]	CovidSORT	COVID–19 detection	X-ray images	2	96.83	98.75	96.57	–	97.65	–
[179]	CGNet	COVID–19 diagnosis	X-ray images	2	98.75	–	100	97.95	–	–
			CT images	2	99	–	98	100	–	–
[172]	VGG with the convolutional COVID block (CCBlock)	COVID–19 diagnosis	X-ray images	2	98.52	–	98.58	98.49	–	–
				3	95.34	–	98.47	98.98	–	–
[180]	COVID–SDNet	COVID–19 prediction	X-ray images	2	: 97.72	–	–	–	–	–

Table 5

Summary of Generative Adversarial Network (GAN) approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[181]	GAN with Extreme Learning Machine (ELM), RNN, and LSTM	COVID—19 diagnosis and treatment	Time series	–	–	–	–	–	–	–
[182]	GAN with CNN and ConvLSTM	COVID–19 detection	X-ray and CT images	2	99	97.70	100	97.80	99	–
[183]	GAN–DNN	COVID–19 detection	X-ray images	6	100	–	–	–	–	–

Table 6

Summary of other deep learning approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu.	Preci.	Sens.	Spec.	F1-score	AUC
[184]	CHFS	COVID-19 diagnosis	CT images	2	96.07	96.10	96.10	–	96.10	–
[185]	Deep Learning–Based Computer Aided Detection (CAD) System	COVID–19 detection	X–ray images	2	–	-	68.8	66.7	-	-
			CT images	2	–	–	81.50	72.30	–	–
[186]	Multi–task deep learning approach (CNN, MLP, Encoder, and two decoders)	COVID–19 detection and classification	CT images	4	94.67	–	96	92	–	97
[199]	QDE–DF	COVID–19 classification	CT images	3	99.68	–	–	–	–	–
[200]	DTL–MC	COVID–19 diagnosis	Coughing sounds	2	92.85	91.43	94.57	91.14	92.97	–
				4	92.64	89.91	89.14	96.67	89.52	–
[201]	MRFODEk based on Manta–Ray Foraging Optimization with differential evolution	COVID–19 diagnosis	X–ray images	2	94.99	–	–	–	–	–
				2	96.88	–	–	–	–	–
[187]	FCONet with four CNN models (VGG16, ResNet50, Inceptionv3, and Xception)	COVID–19 diagnosis	CT images	2	99.87	–	99.58	100	–	100
[188]	DETL with three CNN models (AlexNet, VGGNet, and ResNet)	COVID–19 screening	X–ray images	4	90.13	–	–	–	–	–
[202]	shuffled residual CNN model	COVID–19 detection	X-ray images	4	99.80	98.36	96.07	99.94	97.20	98.01
[189]	DLBD-COV	COVID-19 diagnosis	X-ray	2	98	98.20	99.10	-	-	-
			CT images	2	99.40	98.90	98.90	-	-	-
[191]	Deep learning models	COVID-19 screening	X-ray images	6	100	–	–	–	–	–
[192]	Stacked ensemble deep learning model	COVID–19 diagnosis	X-ray images	3	98.60	–	–	–	–	–
[193]	CWLD	COVID–19 diagnosis	X–Ray images	2	100	–	–	–	–	–
[194]	stacked auto–encoder detector	COVID–19 diagnosis	CT images	2	94.70	96.54	96.54	94.10	–	94.80
[203]	Deep CNN transfer learning models	COVID–19 diagnosis	X–ray images	3	97.11	97	–	–	–	97
[195]	CAD–based YOLO Predictor	COVID–19 diagnosis	X–ray images	9	97.40	–	85.15	99.06	84.81	–
[197]	Ensemble deep transfer learning models	COVID–19 diagnosis	X–ray images	2	96.15	95.90	96.40	95.8	96.10	–
				3	99.21	99	99	–	99	–
[195]	Deep learning based dual–tasks network (FaNet)	COVID-19 diagnosis and severity assessments	CT images	2	98.28	–	–	–	–	–
				2	94.83	–	–	–	–	–

Wang et al. [60] proposed a deep CNN model, called Residual Network34 (ResNet34), for COVID-19 diagnosis in CT scan images. The effectiveness of ResNet34 was validated using CT scan images collected from 99 patients (55 patients with typical viral pneumonia and 44 patients with COVID-19). Simulation results showed that ResNet34 achieves an overall accuracy of 73.10%, specificity of 67%, and sensitivity of 74%.

Narin et al. [61] used three pre-trained techniques including ResNet50, InceptionV3, and InceptionResNetV2 for automatic diagnosis and detection of COVID-19. The case studies included four classes including normal, COVID-19, bacterial, and viral pneumonia patients. The authors demonstrated that ResNet50 gives the highest accuracy in three different datasets.

Maghdid et al. [62] proposed a CNN model with AlexNet for COVID-19 diagnosis. A dataset of 361 CT images and 170 X-ray images of COVID-19 disease collected from five different sources was used in the simulation. Quantitative results demonstrated that AlexNet achieves an accuracy of 98%, a sensitivity of 100%, and a specificity of 96% in X-ray images, while the modified CNN model achieves 94.10% of accuracy, 90% of sensitivity, and 100% of specificity in CT-images.

Wang et al. [63] employed eight deep learning (DL) models (fully convolutional network (FCN-8 s), UNet, VNet, 3D UNet++, dual-path network (DPN-92), Inceptionv3, ResNet50, and Attention ResNet50) for COVID-19 detection. The efficiency of the proposed models was evaluated using 1,136 CT images (723 with COVID-19 and 413 normal) collected from five hospitals. Simulation results demonstrated the superiority of 3D UNet++ compared to other CNN models.

In CT scan images, UNet++ was employed by Chen et al. [64] for COVID-19 detection. The performance of UNet++ was assessed based on a dataset of 106 CT scan images. Simulation results showed that UNet++ provides a per-patient accuracy of 95.24%, sensitivity of 100%, specificity of 93.55%. A per-image accuracy of 98.85%, sensitivity of 94.34%, specificity of 99.16% were also achieved.

Apostolopoulos et al. [65] proposed five deep CNN models (VGG19, MobileNetv2, Inception, Xception, and Inception ResNetv2) for COVID-19 detection cases. The proposed models were tested using two datasets of 1428 and 1442 images, respectively. In the first dataset (224 with COVID-19, 700 with bacterial pneumonia, and 504 normal), MobileNetv2 approach provided better results with a two-class problem accuracy, three-class problem accuracy, sensitivity, and specificity of 97.40%, 92.85%, 99.10%, and 97.09%, respectively. In the second dataset (224 with COVID-19, 714 with bacterial pneumonia, and 504 normal), MobileNetv2 approach also provided better performance by achieving a two-class problem accuracy, three-class problem accuracy, sensitivity, and specificity of 96.78%, 94.72%, 98.66%, and 96.46%, respectively.

Another deep CNN model was developed by Zhang et al. [66] which is composed of three components (a backbone network, a classification head, and an anomaly detection head). This technique was evaluated using 100 chest X-ray images of 70 patients taken from the Github repository. 1431 additional chest X-ray images of 1008 patients taken from the public Chest X-ray14 data were also used to facilitate deep learning. Simulation results showed that the proposed model is an effective diagnostic tool for low-cost and fast COVID-19 screening by achieving the accuracy of 96% for COVID-19 cases and 70.65% for non-COVID-19 cases.

Another intersting project was done by Ghoshal and Tucker [67], in which a Bayesian Convolutional Neural Networks (BCNN) was used in conjunction with Dropweights for COVID-19 diagnosis and classification.

Toraman et al. [68] proposed a CNN model, called CAPSNET, for fast and accurate diagnostics of COVID-19 cases. CAPSNET model was evaluated using two datasets of 2100 and 13,150 cases, respectively. In the first dataset (1050 with COVID-19 and 1050 no-findings), CAPSNET provided better results by achieving an accuracy, precision, sensitivity, specificity, F1-score of 97.23%, 97.08%, 97.42%, 97.04%, and 97.24% respectively. In the second dataset (1050 with COVID-19, 1050 no-findings, and 1050 pneumonia), CAPSNET provided better performance by achieving an accuracy, precision, sensitivity, specificity, and F1-score of 84.22%, 84.61%, 84.22%, 91.79%, and 84.21% respectively.

Hammoudi et al. [69] investigated six deep CNN models (ResNet34, ResNet50, DenseNet169, VGG19, InceptionResNetV2, and RNN-LSTM) for COVID-19 screening and detection. A dataset of 5,863 children’s X-Ray images (Normal and Pneumonia) was exploited to evaluate the techniques proposed. Simulation results showed that DenseNet169 outperforms other deep CNN models by obtaining an average accuracy of 95.72%.

Ardakani et al. [70] proposed ten deep CNN models (AlexNet, VGG16, VGG19, SqueezeNet, GoogleNet, MobileNetV2, ResNet18, ResNet50, ResNet101, and Xception) for COVID-19 diagnosis. A dataset of 1020 CT images (108 with COVID-19, and 86 with bacteria pneumonia) was used to benchmark the efficiency. Simulation results showed the high performance of ResNet101 compared to other deep CNN models by achieving an accuracy of 99.51%, sensitivity of 100%, AUC of 99.4%, and specificity of 99.02%. Xu et al. [71] proposed a hybrid deep learning model, called ResNet+, based on combining the traditional ResNet with location-attention mechanism for COVID-19 diagnosis. The effectiveness of ResNet+ was evaluated using 618 Computer Tomography (CT) images (175 normal, 219 with COVID-19, 224 with Influenza-A viral pneumonia) and results demonstrated that ResNet+ provides an overall accuracy of 86.70%, sensitivity of 81.50%, precision of 80.80%, and F1-score of 81.10%. It is also revealed that the proposed ResNet+ is a promising supplementary diagnostic technique for clinical doctors.

Cifci [72] proposed two deep CNN model (AlexNet and InceptionV4) for Diagnosis and prognosis analysis of COVID-19 cases. The effectiveness of the proposed models was evaluated using 5800 CT images divided into 80% training and 20% test. It was demonstrated that AlexNet outperforms InceptionV4 by achieving an overall accuracy of 94.74%, a sensitivity of 87.37%, and a specificity of 87.45%. Bai et al. [73] did a similar work by proposin an EfficientNet B4 CNN model with a fully connected neural network for the detection and classification of COVID-19 cases. CT scan images of 521 patients were used in the simulation.

Loey et al. [74] proposed three deep CNN approaches (Alexnet, Googlenet, and Restnet18) with GAN model for COVID-19 detection. The proposed approaches were evaluated using three scenarios: i) four classes (normal, viral pneumonia, bacteria pneumonia, and COVID-19 images); ii) three classes (COVID-19, Normal, and Pneumonia); and iii) two classes (COVID-19, Normal). Experimental results demonstrated that Googlenet gives better performance in the first and third scenario by achieving an accuracy of 80.60%, and 100%, respectively. Alexnet provides better results in the second scenario by achieving an accuracy of 85.20%.

Singh et al. [75] proposed a novel deep learning approach based on convolutional neural networks with multi-objective differential evolution (MODE) for the classification of COVID-19 patients. In addition, Mukherjee et al. [76] proposed a shallow light-weight CNN model for automatic detection of COVID-19 cases from Chest X-rays in a similar manner.

Ozkaya et al. [77] proposed an effective approach based on the combination of CNN model with the ranking method and SVM technique for COVID-19 detection. The case studies included two datasets generated from 150 CT images, each dataset contains 3000 normal images and 3000 with COVID-19. Simulation results showed the high performance and robustness of the proposed approach compared to VGG16, GoogleNet, and ResNet50 models in terms of accuracy, sensitivity, specificity, sensitivity, F1-score, and Matthews Correlation Coefficient (MCC) metrics.

Toğaçar et al. [78] proposed two CNN models (MobileNetV2, SqueezeNet) combined with SVM for COVID-19 detection. The efficiency of the proposed models was validated using a dataset of X-ray images divided into three classes: normal, with COVID-19, and with pneumonia. The accuracy obtained in their work is of 99.27%.

Pathak et al. [79] proposed a ResNet50 deep transfer learning technique for the detection and classification of COVID-19 infected patients. The effectiveness of ResNet50 was evaluated using 852 CT images collected from various datasets (413 COVID-19 (+) and 439 normal or pneumonia). Simulation results showed that ResNet50 model gives efficient performance by achieving a specificity, precision, sensitivity, accuracy of 94.78%, 95.19%, 91.48%, and 93.02%, respectively.

Elasnaoui et al. [80] proposed seven Deep CNN models including baseline CNN, VGG16, VGG19, DenseNet201, InceptionResNetV2, InceptionV3, Xception, Resnet50, and MobileNetV2 for automatic classification of pneumonia images. Chest X-Ray & CT datasets containing 5856 images (4273 pneumonia and 1583 normal) were used to validate the proposed models and results demonstrated that Resnet50, MobileNetV2, and InceptionResnetV2 provide high performance with an overall accuracy more than 96% against other CNN models with an accuracy around 84%. Another similar work was done by Zhang et al. [81], in which a diagnosis COVID-19 system based on 3D ResNet18 deep learning technique with five deep learning-based segmentation models (Unet, DRUNET, FCN, SegNet & DeepLabv3) for Diagnosis and prognosis prediction of COVID-19 cases.

Rajaraman and Antali [82] used five deep CNN models (VGG16, InceptionV3, Xception, DenseNet201, NasNetmobile) for COVID-19 screening. Six datasets of x-ray images including Pediatric CXR, RSNA CXR, CheXpert CXR, NIH CXR-14, Twitter COVID-19 CXR, and Montreal COVID-19 CXR were used to validate the effectiveness of the proposed models. The accuracy obtained was 99.26%.

Tsiknakis et al. [83] proposed a modified deep CNN model (Modified InceptionV3) for COVID-19 screening on chest X-rays. The Modified InceptionV3 was evaluated using two chest X-ray datasets, the first dataset was collected from [84], the second one was collected from the QUIBIM imagingcovid19 platform database and various public repositories. Experimental results showed that the modified InceptionV3 model gives an average accuracy, AUC, sensitivity, and specificity of 76%, 93%, 93%, and 91.80%, respectively.

Ahuja et al. [85] presented pre-trained transfer learning models (ResNet18, ResNet50, ResNet101, and SqueezeNet) for automatic detection of COVID-19 cases. Another similar work was done by Oh et al. [86], in which a patch-based convolutional neural network was proposed based on ResNet18.

Elasnaoui and Chawki [87] used seven pre-trained deep learning models (VGG16, VGG19, DenseNet201, InceptionResNetV2, InceptionV3, Resnet50, and MobileNetV2) for automated detection and diagnosis of COVID-19 disease. The effectiveness of the proposed models was assessed using chest X-ray & CT dataset of 6087 images. Simulation results showed the superiority of InceptionResNetV2 compared to other deep CNN models by achieving an accuracy, precision, sensitivity, specificity, and F1-score of 92.60%, 93.85%, 82.80%, 97.37%, and 87.98%, respectively.

Chowdhury et al. [88] introduced eight deep CNN (DenseNet201, RestNet18, MobileNetv2, InceptionV3, VGG19, ResNet101, CheXNet, and SqueezNet) for COVID-19 detection. A dataset of 3487 x-ray images (423 with COVID-19, 1485 with viral pneumonia, and 1579 normal) with and without image augmentation was used in the validation of the proposed models. Simulation results showed that CheXNet gives better results when image augmentation was not applied with an accuracy, precision, sensitivity, specificity, F1-score of 97.74%, 96.61 %, 96.61%, 98.31%, and 96.61% respectively. However, when image augmentation was used, DenseNet201 outperforms other deep CNN models by achieving an accuracy, precision, sensitivity, specificity, and F1-score of 97.94%, 97.95%, 97.94%, 98.80%, and 97.94%, respectively.

Apostolopoulos et al. [89] proposed a deep CNN model (MobileNetv2) for COVID-19 detection and classification. The efficiency of MobileNetv2 was assessed using a large-scale dataset of 3905 X-ray images and results showed its excellent performance by achieving an accuracy, sensitivity, specificity of 99.18%, 97.36%, and 99.42%, respectively in the detection of COVID-19.

Rahimzadeh and Attar [90] proposed a modified deep CNN model based on the combination of Xception and ReNet50V2 for detecting COVID-19 from chest X-ray images. The proposed model was tested using 11,302 chest X-ray images (31 with COVID-19, 4420 with pneumonia, and 6851 normal cases). Experimental results showed that the combined model gives an average accuracy, precision, sensitivity, and specificity of 91.4%, 72.8%, 87.3%, and 94.2%, respectively. In a similar work, Abbas et al. [91] adapted a Convolutional Neural Network model, called Decompose Transfer Compose (DeTraC). The effectiveness of the DeTraC model was validated using a dataset of X-ray images collected from several hospitals and institutions around the world. As the results 95.12% accuracy, 97.91% sensitivity, and 1.87% specificity were obtained.

Afshar et al. [92] developed a deep CNN model (COVID-CAPS) using on Capsule Networks for COVID-19 identification and diagnosis. The effectiveness of COVID-CAPS was tested using two publicly available chest X-ray datasets. [84, 93] As the results 98.30% accuracy, 80% sensitivity, and 8.60% specificity were obtained.

Brunese et al. [94] adopted a deep CNN approach (VGG-16) for automatic and faster COVID-19 detection from chest X-ray images. The robustness of VGG-16 was evaluated using 6523 chest X-ray images (2753 with pneumonia disease, 250 with COVID-19, while 3520 healthy) and results showed that VGG-16 achieves an accuracy of 97% for the COVID-19 detection and diagnosis.

Jin et al. [95] proposed a deep learning-based AI system for diagnosis of COVID-19 in CT images. 10,250 CT scan images (COVID-19, viral pneumonia, influenza-A/B, normal) taken from three centers in China and three publicly available databases were used in the simulation and results showed that the proposed model achieves an AUC of 97.17%, a sensitivity of 90.19%, and a specificity of 95.76%.

Truncated Inception Net was proposed by Das et al. [96] as a Deep CNN model for COVID-19 cases detection. Six different datasets were used in the simulation considering healthy, with COVID-19, with Pneumonia, and with Tuberculosis cases. It was demonstrated that Truncated Inception Net provides accuracy, precision, sensitivity, specificity, and F1-score of 98.77%, 99%, 95%, 99%, and 97%, respectively.

Asif et al. [97] proposed a Deep CNN model (Inception V3) with transfer learning for automatic detection of COVID-19 patients cases. A dataset consists of 3550 chest x-ray images (864 with COVID-19, 1345 with viral pneumonia, and 1341 normal) was used to test Inception V3. Simulation results proved the efficiency of the Inception V3 by achieving an accuracy of 98%.

Punn and Agrawal [98] introduced five fine-tuned deep learning approaches (baseline ResNet, Inceptionv3, InceptionResNetv2, DenseNet169, and NASNetLarge) for automated diagnosis and classification of COVID-19. The performance of the proposed approaches was validated using three datasets of X-ray and CT images collected from Radiological Society of North America (RSNA), [99] U.S. national library of medicine (USNLM), [100] and COVID-19 image data collection. [84] Simulation results showed that NASNetLarge outperforms other CNN models by achieving 98% of accuracy, 88% of precision, 90% of sensitivity, 95% of specificity, and 89% of F1-score.

Shelke et al. [101] proposed three CNN models (VGG16, DenseNet161, and ResNet18) for COVID-19 diagnosis and analysis. The proposed models were tested using two datasets of 1191 and 1000 X-ray images, respectively. In the first dataset (303 with COVID-19, 500 with bacterial pneumonia, and 388 normal), VGG16 approach provided better results with an accuracy of 95.9%. In the second dataset (500 with COVID-19 and 500 normal), DenseNet161 approach provided better performance by achieving an accuracy of 98.9%.

Rajaraman et al. [102] proposed eight deep CNN models (VGG16, VGG19, InceptionV3, Xception, InceptionResNetV2, MobileNetV2, DenseNet201, NasNetmobile) for COVID-19 screening. Four datasets of x-ray images including Pediatric CXR, RSNA CXR, Twitter COVID-19 CXR, and Montreal COVID-19 CXR were used to validate the effectiveness of the proposed models. Experimental results demonstrated that the weighted average of the best-performing pruned models enhances performance by providing an accuracy, precision, sensitivity, AUC, F1-score of 99.01%, 99.01%, 99.01%, 99.72%, and 99.01%, respectively.

Another similar work was done by Luz et al. [103], which can be considered as an extension of EfficientNet for COVID-19 detection and diagnosis in X-Ray Chest images. It was compared with MobileNet, MobileNetV2, ResNet50, VGG16, and VGG19. Simulation results demonstrated the effectiveness of EfficientNet compared to other deep CNN models by achieving an overall accuracy of 93.9%, sensitivity of 96.8%, and a positive prediction rate of 100%.

Jaiswal et al. [104] employed DenseNet201 based transfer learning for COVID-19 detection and diagnosis. The performance of DenseNet201 was validated using 2492 chest CT-scan images (1262 with COVID-19 and 1230 healthy) taken into account precision, F1-measure, specificity, sensitivity, and accuracy metrics. Quantitative results showed the effectiveness of compared to VGG16, Resnet152V2, and InceptionResNet by providing a precision, F1-measure, specificity, sensitivity, and accuracy of 96.29%, 96.29%, 96.29% and 96.21%, and 96.25%, respectively.

Sharma [105] employed a ResNet50 CNN-based approach for COVID-19 detection. 2200 CT images (800 with COVID-19, 600 viral pneumonia, and 800 normal healthy) collected from various hospitals in Italy, China, Moscow, and India were used in the simulation and results showed that ResNet50 outperforms ResNet+ by giving a specificity, sensitivity, accuracy of 90.29%, 92.1%, and 91.0%, respectively. Pu et al. [106] conducted a similar work.

Alotaibi [107] used four pre-trained CNN models (RESNET50, VGG19, DENSENET121, and INCEPTIONV3) for the detection of COVID-19 cases. A dataset of X-ray images (219 with COVID-19, 1341 Normal, and 1345 with Viral Pneumonia) was used in the experimentation and results demonstrated the better performance of DENSENET121 compared to RESNET50, VGG19, and INCEPTIONV3 by achieving an accuracy, precision, sensitivity, and F1-score of 98.71%, 98%, 98%, and 97.66%, respectively.

Goyal and Arora [108] proposed three CNN models (VGG16, VGG19, and ResNet50) for COVID-19 detection. This technique was evaluated using 748 chest X-ray images (250 with COVID-19, 300 normal, and 198 with pneumonia bacteria) and results showed that VGG19 outperforms VGG16 and ResNet50 by achieving an accuracy of 98.79% and 98.12% in training and testing cases, respectively. A similar work was done by Das et al. [109], in which an extreme version of the Inception (Xception) model for the automatic detection of COVID-19 infection cases in X-ray images.

Rahaman et al. [110] used 15 different pre-trained CNN models for COVID-19 cases identification. 860 chest X-Ray images (260 with COVID-19, 300 healthy, and 300 pneumonia) were employed to investigate the effectiveness of the proposed models. Simulation results showed that the VGG19 model outperforms other deep CNN models by obtaining an accuracy of 89.3%, precision of 90%, sensitivity of 89%, and F1-score of 90%.

Altan and Karasu [111] proposed a hybrid approach based on CNN model (EfficientNet-B0), two-dimensional (2D) curvelet transformation, and chaotic salp swarm algorithm (CSSA) for COVID-19 detection. 2905 real raw chest X-ray images (219 with COVID-19, 1345 viral pneumonia, and 1341 normal) were used. Another similar work was done where a Confidence-aware anomaly detection (CAAD) was proposed based on EfficientNetB0

Ni et al. [112] proposed a CNN model, called MVPNet, for automatic detection of COVID-19 cases. 19,291 pulmonary CT scans images (3854 with COVID-19, 6871 with bacterial pneumonia, and 8566 healthy) were employed to validate the performance of the MVPNet model. Experimental results demonstrated that MVPNet achieves a sensitivity of 100%, specificity of 65%, accuracy of 98%, and F1-score of 97%.

Nguyen et al. [113] employed two deep CNN models (EfficientNet and MixNet) for the detection of COVID-19 infected patients from chest X-ray (CXR) images. The effectiveness of the proposed approach was validated using two real datasets consisting of: i) 13,511 training images and 1,489 testing images; ii) 14,324 training images and 3,581 testing images. Simulation results demonstrated that the proposed approach outperforms some well-established baselines by yielding an accuracy larger than 95%.

Islam et al. [114] proposed four CNN models( VGG19, DenseNet121, InceptionV3, and InceptionResNetV2) and recurrent neural network (RNN) for COVID-19 diagnosis. A similar work was done by Mei et al. [115] with proposing a combination of SVM, random forest, MLP, and CNN.

Khan and Aslam [116] presented four CNN models (DenseNet121, ResNet50, VGG16, and VGG19) for COVID-19 diagnosis. The superiority of the proposed models was evaluated using a dataset of 1057 X-ray images including 862 normal and 195 with COVID-19. Experimental results demonstrated that VGG-19 model achieves better performance than DenseNet121, ResNet50, and VGG16 by achieving an accuracy, sensitivity, specificity, F1-score of 99.33%, 100%, 98.77%, and 99.27%, respectively.

Perumal et al. [117] used deep CNN models (VGG16, Resnet50, and InceptionV3) and Haralick features for the detection of COVID-19 cases. A dataset of X-ray and CT images collected from various resources available in Github open repository, RSNA, and Google images was used in the simulation and results showed that the proposed models outperform other existing models with an average accuracy of 93%, precision of 91%, and sensitivity of 90%.

Kumar et al. [118] used various deep learning models (VGG, DenseNet, AlexNet, MobileNet, ResNet, and Capsule Network) with blockchain and federated-learning technology for COVID-19 detection from CT images. These techniques were evaluated using a dataset of 34,006 CT scan images taken from the GitHub repository (https://github.com/abdkhanstd/COVID-19). Simulation results revealed that the Capsule Network model outperforms other deep learning models by achieving an accuracy of 0.83 and sensitivity of 0.967 and precision of 0.83.

Zebin et al. [119] proposed three Deep CNN models (modified VGG16, ResNet50, and EfficientNetB0) for COVID-19 detection. A dataset of X-ray images (normal, non-COVID-19 pneumonia, and COVID-19) taken from COVID-19 image Data Collection was used to evaluate them. The overall accuracy of 90%, 94.30%, and 96.80% for the VGG16, ResNet50, and EfficientNetB0 were obtained.

Abraham and Nair [120] proposed a combined approach based on the combination of five multi-CNN models (Squeezenet, Darknet-53, MobilenetV2, Xception, and Shufflenet) for the automated detection of COVID-19 cases from X-ray images.

Ismael and Şengür [121] proposed three deep learning techniques for COVID-19 detection from chest X-ray images. The first technique was proposed based on five pre-trained deep CNN models (ResNet18, ResNet50, ResNet101, VGG16, and VGG19), the second deep learning model was proposed using CNN model with end-to-end training, the third and the last technique was proposed using pre-trained CNN models and SVM classifiers with various kernel functions. A dataset of 380 chest X-ray images (180 with COVID-19 and 200 normal (healthy)) was used for validation experimentation and results showed the efficiency of CNN techniques compared to various local texture descriptors.

Goel et al. [122] proposed an optimized convolutional neural network model, called OptCoNet, for COVID-19 diagnosis. A dataset of 2700 X-ray images (900 with COVID-19, 900 normal, and 900 with pneumonia) was employed to assess the performance of OptCoNet and results showed is effectiveness by providing accuracy, precision, sensitivity, specificity, and F1-score values of 97.78%, 92.88%, 97.75%, 96.25%, and 95.25%, respectively.

Bahel and Pillali [123] proposed five deep CNN models (InceptionV4, VGG 19, ResNetV2-152, and DenseNet) for detecting COVID-19 from chest X-Ray images. These techniques were evaluated based on a dataset of 300 chest x-ray images of infected and uninfected patients. Heat map filter was used on the images for helping the CNN models to perform better. Simulation results showed that DenseNet outperforms other deep CNN models such as InceptionV4, VGG19, and ResNetV2-152.

Sitaula and Hossain [124] proposed a novel deep learning model based on VGG-16 with the attention module for COVID-19 detection and classification. Authors conducted extensive experiments based on three X-ray image datasets D1 (Covid-19, No findings, and Pneumonia), D2 (Covid, Normal, Pneumonia Bacteria, Pneumonia Viral), and D3 (Covid, Normal, No findings, Pneumonia Bacteria, and Pneumonia Viral) to test this technique. Experimental results revealed the stable and promising performance compared to the state-of-the-art models by obtaining an accuracy of 79.58%, 85.43%, and 87.49% in D1, D2, and D3, respectively.

Jain et al. [125] proposed three CNN models (Inception V3, Xception, and ResNeXt) for COVID-19 detection and analysis. 6432 chest x-ray images divided into two classes including training set (5467) and validation set (965) were used to analyze the approaches performance. Simulation results showed that Xception model gives the highest accuracy with 97.97% as compared to other existing models.

Yasar and Ceylan [126] proposed a novel model based on CNN model with local binary pattern and dual-tree complex wavelet transform for COVID-19 detection on chest X-ray images. This approach was validated using two datasets of X-ray images: i) dataset of 230 images (150 with Covid-19 and 80 normal) and ii) dataset of 476 images (150 with Covid-19 and 326 normal). Experimental results showed that the proposed model gives good performance by achieving an accuracy, sensitivity, specificity, F1-score, and AUC of 98.43%, 99.47%, 98%, 98.81%, and 99.90%, respectively for the first dataset. For the second dataset, the proposed model achieves an accuracy, sensitivity, specificity, F1-score and, AUC of 98.91%, 99.20%, 99.39%, 98.28%, and 99.91%, respectively.

Khalifa et al. [127] proposed a new approach based on three deep learning models (Resnet50, Shufflenet, and Mobilenet) and GAN for detecting COVID-19 in CT chest Medical Images. In a similar work, Mukherjee et al. [128] proposed a lightweight (9 layered) CNN-tailored deep neural network model. It was demonstrated that the proposed model outperforms InceptionV3.

Hira et al. [142] used nine CNN models (AlexNet, GoogleNet, ResNet50, SeResNet50, DenseNet121, InceptionV4, InceptionResNetV2, ResNeXt50, and SeResNeXt50) for the detection of COVID–19 disease. The efficiency of the proposed models was validated using four scenarios: (i) two classes (224 with COVID–19 and 504 Normal); (ii) three classes (224 with COVID–19, 504 Normal, and 700 with bacterial Pneumonia); (iii) three classes (224 with COVID-19, 504 Normal, and 714 with bacterial and viral Pneumonia) and (iv) four classes (1346 normal, 1345 viral pneumonia, 2358 bacteria pneumonia, and with 183 COVID-19). Experimental results demonstrated that SeResNeXt50 outperforms other methods in terms of accuracy, precision, sensitivity, specificity, and F1-score.

Jelodar et al. [147] proposed a novel model based on LSTM with natural language process (NLP) for COVID-19 cases classification. The effectiveness of the proposed model was validated using a dataset of 563,079 COVID-19-related comments collected from the Kaggle website (between January 20, 2020 and March 19, 2020) and results showed its efficiency and robustness on this problem area to guide related decision-making.

Chimmula et al. [148] used LSTM model for forecasting of COVID-19 cases in Canada. The performance of LSTM was validated using data collected from Johns Hopkins University and Canadian Health Authority with several confirmed cases and results showed that the LSTM model achieves better performance when compared with other forecasting models.

Jiang et al. [149] developed a novel model, called BiGRU-AT, based on bidirectional GRU with an attention mechanism for COVID-19 detection and diagnosis. The performance of BiGRU-AT was assessed using breathing and thermal data extracted from people wearing masks. Simulation results showed that BiGRU-AT achieves an accuracy, sensitivity, specificity, and F1-score of 83.69%, 90.23%, 76.31%, and 84.61%, respectively.

Mohammed et al. [150] proposed LSTM with ResNext+ and slice attention module for COVID-19 detection. A total of of 302 CT volumes (20 with confirmed COVID19 and 282 normal) was used for testing and training the proposed model. According to the results, the proposed model provides an accuracy of 77.60%, precision of 81.90%, sensitivity of 85.50%, specificity of 79.30%, and F1-score of f 81.40%.

Islam et al. [151] introduced a novel model based on the hybridization of LSTM with CNN for automatic diagnosis of COVID-19 cases. The effectiveness of the hybrid model was validated using a dataset of 4575 X-ray images (1525 images with COVID-19, 1525 with viral pneumonia, and 1525 normal). Simulation results showed that the hybrid model outperforms other existing models by achieving an accuracy, sensitivity, specificity, and F1-score of 99.20%, 99.30%, 99.20%, and 98.90%, respectively.

Aslan et al. [152] proposed a hybrid approach based on the hybridization of Bidirectional LSTM (BiLSTM) with CNN Transfer Learning (mAlexNet) for COVID-19 detection. A dataset of 2905 X-ray images (219 with COVID-19, 1345 with viral pneumonia, and 1341 normal) was used in the simulation and results showed that the hybrid approach outperforms mAlexNet model by giving an accuracy, precision, sensitivity, specificity, F1-score, and AUC of 98.70%, 98.77%, 98.76%, 99.33%, 98.76%, and 99%, respectively (Tables 2, 3, 4, 5, 6).

Specialized CNN Approaches for COVID–19

Song et al. [155] developed a deep-learning model, called Details Relation Extraction neural Network (DRE-Net), for accurate identification of COVID-19-infected patients. 275 chest scan images (86 normal, 88 with COVID-19, and 101 with bacteria pneumonia) were used to validate the performance of DRE-Net. Simulation results showed that DRE-Net can identify COVID-19 infected patients with an average accuracy of 94%, AUC of 99%, and sensitivity of 93%.

Li et al. [156] proposed a deep learning method, called COVNet, for COVID-19 diagnosis from CT scan images. A dataset of 4356 chest CT images from 3222 patients collected from six hospitals between August 2016 and February 2020 was used in the simulation and results showed that the proposed COVNet achieves an AUC, sensitivity, and specificity of 96%, 90%, and 96%, respectively. Zheng et al. conducted a similar study [157] by proposing a 3D deep CNN model, called DeCoVNet, for detecting COVID-19 from 3D CT images.

Ucar and Korkmaz [158] proposed a novel and efficient Deep Bayes-SqueezeNet-based system (COVIDiagnosis-Net) for COVID-19 Diagnosis. A dataset of 5949 chest X-ray images including 1583 normal, 4290 pneumonia, and 76 COVID-19 infection cases was employed in the simulation and results showed that COVIDiagnosis-Net outperforms existing network models by achieving 98.26% of accuracy, 99.13% of specificity, and 98.25% of F1-score.

DarkCovidNet was proposed by Ozturk et al. [159] for automated detection of COVID-19. The efficiency of DarkCovidNet was evaluated using two datasets: i) A COVID-19 X-ray image database developed by Cohen JP [84] and ii) ChestX-ray8 database provided by Wang et al. [160]. Simulation results showed that DarkCovidNet gives accurate diagnostics of 98.08% and 87.02% for binary classification (COVID vs. No-Findings) and multi-class classification (COVID vs. No-Findings vs. Pneumonia), respectively.

Wang and Wong [161] proposed a deep learning model, called Covid-Net, for detecting COVID-19 Cases from Chest X-Ray Images. Quantitative and qualitative results showed the efficiency and superiority of the proposed Covid-Net model compared to VGG-19 and ResNet-50 techniques.

In [162], Born et al. proposed POCOVID-Net for the automatic detection of COVID-19 cases. A lung ultrasound (POCUS) dataset consisting of 1103 images (654 COVID-19, 277 bacterial pneumonia, and 172 normal) sampled from 64 videos was used for evaluating the effectiveness of POCOVID-Net model. According to the results, POCOVID-Net model provides good performance with 0.89 accuracy, 0.88 precision, 0.96 sensitivity, 0.79 specificity, and 0.92 F1-score.

COVID-19Net was proposed by Wang et al. [163] for the diagnostic and prognostic analysis of COVID-19 cases in CT images. A dataset of chest CT images collected from six cities or provinces including Wuhan city in China was used for the simulation and results showed the good performance of COVID-19Net by achieving an AUC of 87%, an accuracy of 78.32%, a sensitivity of 80.39%, F1-score of 77%, and a specificity of 76.61%.

Khan et al. [164] proposed a new model (CoroNet) for COVID-19 detection and diagnosis. CoroNet was validated using three scenarios: i) 4-class CoroNet (normal, viral pneumonia, bacteria pneumonia, and COVID-19 images); ii) 3-class CoroNet (COVID-19, Normal and Pneumonia); and iii) binary 2-class CoroNet (COVID-19, Normal and Pneumonia). Experimental results demonstrated the superiority of CoroNet compared to some studies in the literature by achieving an accuracy of 89.5%, 94.59%, and 99% for 4-class, 3-class, and binary 2-class scenarios, respectively.

Mahmud et al. [165] proposed a novel multi-dilation deep CNN model (CovXNeT) based on depthwise dilated convolutions for automatic COVID-19 detection. Three datasets of 5856, 610, and 610 x-ray images were used for evaluating the effectiveness of CovXNeT. Experimental results revealed the performance of CovXNeT compared to other approaches in the literature by providing an accuracy of 98.1%, 95.1%, and 91.70% for the dataset of 5856 images, dataset of 610 images, and dataset of 610 images, respectively.

siddhartha and Santra [166] proposed a novel model, called COVIDLite, based on a depth-wise separable deep neural network (DSCNN) with white balance and CLAHE for the detection of COVID-19 cases. Two datasets of X-ray images: i)1458 images (429 COVID-19, 495 viral pneumonia, and 534 normal) and ii) 365 images (107 COVID-19, 124 viral pneumonia, and 134 normal) were used for testing the effectiveness of COVIDLite. Simulation results revealed that COVIDLite performs for both 2-class and 3-class scenario by achieving an accuracy of 99.58% and 96.43%, respectively.

Ahmed et al. [167] proposed a novel CNN model, called ReCoNet, for COVID-19 detection. The effectiveness of ReCoNet was evaluated based on COVIDx [161] and CheXpert [168] datasets containing 15.134 and 224.316 CXR images, respectively. Experimental results demonstrated that ReCoNet outperforms COVID-Net and other state-of-the-art techniques by yielding an accuracy, sensitivity, and specificity of 97.48%, 96.39%, and 97.53%, respectively.

Haghanifar et al. [169] developed a novel approach, called COVID-CXNET, based on the well-known CheXNet model for automatic detection of COVID-19 cases. The effectiveness of COVID-CXNET was tested using a dataset of 3,628 chest X-ray images (3,200 normal and 428 with COVID-19) divided into two classes including training set (80%)and validation set (20%). Experimental results showed that COVID-CXNET gives an accuracy of 99.04% and F1-score of 96%.

Turkoglu [170] proposed a COVIDetectioNet model with AlexNet and SVM for COVID-19 diagnosis and classification. A dataset of 6092 X-ray images (1583 Normal, 219 with COVID19, and 4290 with Pneumonia) collected from the Github and Kaggle databases was used in the experimentation. Simulation results demonstrated the better performance of COVIDetectioNet compared to other deep learning approaches by achieving an accuracy of 99.18%.

Tammina [171] proposed a novel deep learning approach, called CovidSORT for COVID-19 detection. 5910 Chest X-ray images collected from retrospective cohorts of pediatric Women patients and Children’s Medical Center of Guangzhou, China were used to validate the CovidSORT performance. Simulation results demonstrated that the CovidSORT model provides an accuracy of 96.83%, precision of 98.75%, sensitivity of 96.57%, and F1-score of 97.65%.

Al-Bawi et al. [172] developed an efficient model based on VGG with the convolutional COVID block (CCBlock) for the automatic diagnosis of COVID-19. To evaluate It, 1,828 x-ray images were used including 310 with COVID-19 cases, 864 with pneumonia, and 654 normal images. According to the results, the proposed model gives the highest diagnosis performance by achieving an accuracy of 98.52% and 95.34% for two and three classes, respectively.

Jamshidi et al. [181] used Generative Adversarial Network (GAN), Extreme Learning Machine (ELM), RNN, and LSTM for COVID–19 diagnosis and treatment. Sedik et al. [182] proposed a combined model based on GAN with CNN and ConvLSTM for COVID–19 infection detection. Two datasets of X-ray and CT images were used in the simulation and results showed the effectiveness and performance of the combined model by achieving 99% of accuracy, 97.70% of precision, 100% of sensitivity, 97.80% of specificity, and 99% of F1-score.

Other Deep Learning Approaches

Farid et al. [184] proposed a Stack Hybrid Model, called Composite Hybrid Feature Selection Model (CHFS), based on the hybridization of CNN and machine learning approaches for early diagnosis of covid19. The performance of CHFS was evaluated based on a dataset containing 51 CT images divided into training and testing sets. Simulation results showed that CHFS achieves an F1-score, precision, sensitivity, accuracy of 96.10%, 96.10%, 96.10%, and 96.07%, respectively.

Hwang et al. [185] implemented a Deep Learning-Based Computer-Aided Detection (CAD) System for the identification of COVID-19 infected patients. CAD system was trained based on chest X-ray and CT images and results showed that CAD system achieves 68.80% of sensitivity, 66.70% of specificity with chest X-ray images and 81.5% of sensitivity, 72.3% of specificity with CT images.

Amyar et al. [186] proposed a multi-task deep learning approach for COVID-19 detection and classification from CT images. A dataset of images collected from 1369 patients (449 with COVID-19, 425 normal, 98 with lung cancer, and 397 of different kinds of pathology) was used to evaluate the performance of the proposed approach. Results showed that the proposed approach achieves an AUC of 0.97, an accuracy of 94.67, a sensitivity of 0.96, and a specificity of 0.92.

For COVID-19 pneumonia diagnosis, Ko et al. [187] proposed fast-track COVID-19 classification network (FCONet), which uses as backbone one of the pre-trained deep learning models (VGG16, ResNet50, Inceptionv3, or Xception). A set of 3993 chest CT images divided into training and test classes were used to evaluate the performance of the proposed FCONet. Experimental results demonstrated that FCONet with ResNet50 gives excellent diagnostic performance by achieving a sensitivity of 99.58%, specificity 100%, accuracy 99.87%, and AUC of 100%.

Basu and Mitra [188] proposed a domain extension transfer learning (DETL) with three pre-trained deep CNN models (AlexNet, VGGNet, and ResNet) for COVID-19 screening. 1207 X-ray images (350 normal, 322 with pneumonia, 305 with COVID-19, and 300 other diseases) were employed to validate the proposed model. Experimental results showed that DETL with VGGNet gives a better accuracy of 90.13%.

Elghamrawy [189] developed a new approach (DLBD-COV) based on H2O’s Deep-Learning-inspired model with Big Data analytic for COVID-19 detection. The efficiency of DLBD-COV was validated based on CT images collected from [84] and X-ray images collected from [190] taking into account five metrics such as accuracy, precision, Sensitivity, and computational time. Simulation results showed that DLBD-COV provides a superior accuracy compared to other CNN models such as DeConNet and ResNet+.

Sharma et al. [191] proposed an deep learning model for rapid identifying and screening of COVID-19 patients. The efficiency of the proposed model was validated using chest X-ray images of adult COVID-19 patients (COVID-19, non-COVID-19, pneumonia, and tuberculosis images) and results showed its efficiency compared to previously published methods.

Hammam et al. [192] proposed a stacked ensemble deep learning model for COVID-19 vision diagnosis. The efficiency of the proposed model was validated using a dataset of 500 X-ray images divided into three classes including the training set (80%), validation set (10%), and testing set (10%). Simulation results showed the superior performance of the proposed model compared to any other single model by achieving 98.60% test accuracy. A similar work was done by Mohammed et al. [193], in which a Corner-based Weber Local Descriptor (CWLD) was prpoposed for diagnosis of COVID-19 from chest X-Ray images.

Li et al. [194] proposed a stacked auto-encoder detector model for the diagnosis of COVID-19 Cases on CT scan images. Authors used in their experimentation a dataset of 470 CT images (275 with COVID-19 and 195 normal) collected from UC San Diego. According to the results, the proposed model performs well and achieves an average accuracy of 94.70%, precision of 96.54%, sensitivity of 94.10%, and F1-score of 94.80%. Al-antari et al. [195] introduced a novel model (CAD-based YOLO Predictor) based on fast deep learning computer-aided diagnosis system with YOLO predictor for automatic diagnosis of COVID-19 cases from digital X-ray images. The proposed system was trained using two different digital X-ray datasets: COVID-19 images [84, 88] and ChestX-ray8 images [196]. According to the experimentation, CAD-based YOLO Predictor achieves an accuracy of 97.40%, sensitivity of 85.15%, specificity of 99.06%, and F1-score of 84.81%.

Gianchandani et al. [197] proposed two ensemble deep transfer learning models for Rapid COVID-19 diagnosis. The proposed models were validated using two datasets of X-ray images obtained from Kaggle datasets resource [198] and the University of Dhaka and Qatar University. [88]

Other Machine Learning Approaches

Chakraborty and Ghosh [204] developed a hybrid method (ARIMA–WBF) based on the hybridization of ARIMA model and Wavelet-based forecasting (WBF) model for predicting the number of daily confirmed COVID-19 cases. The effectiveness of ARIMA-WBF was validated using datasets of 346 cases taken from five countries (70: Canada, 71: France, 64: India, 76: South Korea, and 65: UK). Simulation results showed the performance and robustness of ARIMA-WBF in the prediction of COVID-19 cases.

Tuncer et al. [205] proposed a feature generation technique, called Residual Exemplar Local Binary Pattern (ResExLBP) with iterative ReliefF (IRF) and five machine learning methods (Decision tree, linear discriminant, SVM, kNN, and subspace discriminant) for automatic COVID-19 detection. The efficiency of the proposed model was validated using datasets of X-ray images collected from the GitHub website and Kaggle site. Simulation results showed that ResExLBP with IRF and SVM gives better performance compared to other models by providing 99.69% accuracy, 98.85% sensitivity, and 100% specificity.

Tuli et al. [206] developed a novel model based on machine learning and Cloud Computing for real-time prediction of COVID-19. The effectiveness of the proposed model was validated using 2Our World In Data (COVID-19 Dataset) taken from the Github repository (https://github.com/owid/covid-19-data/tree/master/public/data/). Simulation results showed that the proposed model gives good performance on this problem area.

Pereira et al. [207]used MLP with KNN, SVM, Decision Trees, and Random Forest for COVID-19 identification in chest X-ray images. The efficiency of the proposed models was evaluated based on RYDLS-20 database of 1144 chest X-ray images divided into training and test sets with 70% and 30% rates. Experimental results showed the superiority of MLP compared to other machine learning approaches by providing an F1-Score of 89%.

Albahri et al. [208] used a machine learning model combined with a novel Multi-criteria-decision-method (MCDM) for the identification of COVID-19 infected patients. The effectiveness of the proposed model was evaluated based on Blood sample images. Simulation results revealed that the proposed model is a good tool for identifying infected COVID-19 cases.

Wang et al. [209] developed a hybrid model based on FbProphet technique and Logistic Model for COVID-19 epidemic trend prediction. The hybrid model was validated using COVID-19 epidemiological time-series data and results revealed the effectiveness of the hybrid model for the prediction of the turning point and epidemic size of COVID-19.

Ardakani et al. [210] proposed a machine learning-based Computer-Aided Detection (CAD) System (COVIDiag) for COVID-19 diagnosis. The performance of COVIDiag was evaluated using CT images of 612 patients (306 with COVID-19 and 306 normal). Experimental results demonstrated the effectiveness of COVIDiag compared to SVM, KNN, NB, and DT by achieving the sensitivity, specificity, and accuracy of 93.54%, 90.32%, and 91.94%, respectively.

The summary of other Machine Learning approaches is given in Table 7.

Table 7

Summary of other Machine Learning approaches for detection, diagnosis, and prediction of COVID-19 cases

Author (Ref)	Method name	Problem category	Data type	Class	Accu	Preci	Sens	Spec	F1-score	AUC
[204]	ARIMA model and Wavelet-based forecasting (WBF) model	COVID-19 prediction	Time series	5	-	-	-	-	-	-
[205]	ResExLBP with IRF and five machine learning methods (Decision tree, linear discriminant, SVM, kNN, and subspace discriminant)	COVID-19 detection	X-ray images	2	99.69	-	98.85	100	-	-
[206]	Machine learning and Cloud Computing	COVID-19 prediction	Time series	-	-	-	-	-	-	-
[207]	Five machine learning (k-Nearest Neighbors (kNN), Support Vectors Machine (SVM); Multilayer Perceptrons (MLP), Decision Trees (DT), and Random Forests (RF))	COVID-19 detection	X-ray images	7	-	-	-	-	89	-
[208]	MCDM with ML model	COVID-19 detection	Blood sample images	4	-	-	-	-	-	-
[209]	FbProphet technique and Logistic Model	COVID-19 epidemic trend prediction	Time series	-	-	-	-	-	-	-
[210]	COVIDiag	COVID-19 diagnosis	CT images	2	91.94	90.63	93.54	90.32	-	-
[211]	Kalman Filter model	Forecasting and predicting COVID-19 patients	Text	10	-	-	-	-	-	-
[212]	Supervised machine learning Model	COVID-19 detection	X-ray images	2	98.9	96.8	98.4	-	97.6	98.9

Discussion

Machine Learning is the field of AI that has been applied to deal with COVID-19. The finding from this study reveals that:Despite all these contributions, there are still some remaining challenges in applying ML to deal with COVID-19. Actually, handling new datasets generated in real time is facing several issues limiting the efficiency of results. In fact, many of the proposed approaches are based on small datasets. They are, in most cases, incomplete, noisy, ambiguous and with a significant ratio of missing patterns. Consequently, the training is not efficient and the risk of over-fitting is high because of the high variance and errors on the test set. Therefore, the need to build large datasets becomes unavoidable. However, it is not sufficient. In fact, without a complete and standard dataset, it is difficult to conclude which method provides the best results. To overcome that, a deep work of merging existing datasets and cleaning them up, by removing / imputing missing data and removing redundancy, is required.

Techniques of Machine Learning used in this context are several. As shown in Fig. 10, 79% of them are based on Deep Learning, 16% used Supervised Learning, whereas other types of learning are used in only 5% of cases;

Fig. 10

Approaches of machine learning used to deal with COVID-19

Techniques basically known in the field of Unsupervised Learning did not appear in the reviewed papers. However, in case of unlabeled data, deep Learning makes an automatic learning which is a form of an unsupervised learning;

Similarly, techniques of Reinforcement Learning are not explored in the summarized approaches;

The most used technique from Deep Learning is CNN. 65% of DL-based approaches took advantage of this architecture to handle the collected data. As shown in Fig. 11, 17% of them developed new CNN architectures dedicated to COVID-19 data types. The reason is this ability that CNN offers to train multiple layers with nonlinear mappings to classify high-dimensional input data into a set of classes at the output layer. So, given the intensive amount of medical data, CNN emerged as the most suitable solution. Nevertheless, RNN were also present in 6% of approaches and GAN in 2% of them.

Fig. 11

Deep learning approaches used to deal with COVID-19

70% of the Supervised Learning-based approaches opted for the Regression. As we can see in Fig. 12, Regression is made by employing either Random Forest Algorithms or Linear Regression. For its part, classification through SVM technique is applied in 30% of the Supervised Learning based papers.

Fig. 12

Supervised learning techniques used to deal with COVID-19

We have noticed the use of many measures in the evaluation of the proposed approaches. The most recurrent ones are those represented in Fig. 13. In fact, even if we see a balanced result between several metrics, the accuracy seems to take a little more advantage. This is trivial since it is one of the most important metrics in ML which can be used in classification as well as in prediction.

Fig. 13

Metrics used in the evaluation of COVID-19 related approaches

Conclusion

The COVID-19 pandemic has deeply marked the year 2020 and has made the researchers community in different fields react. This paper demonstrated the interest attached by data scientists to this particular situation. It provided a survey of Machine Learning based research classified into two categories (Supervised Learning approaches and Deep Learning approaches) to make detection, diagnosis, or prediction of the COVID-19. Moreover, it gave an analysis and statistics on published works. The review included more than 160 publications coming from more than 6 famous scientific publishers. The learning is based on various data supports such as X-Ray images, CT images, Text data, Time series, Sounds, Coughing/Breathing videos, and Blood Samples. Our study presented a synthesis with accurate ratios of use of each of the ML techniques. Also, it summarized the metrics employed to validate the different models. The statistical study showed that 6 metrics are frequently used with favor to accuracy, sensitivity, and specificity which are evaluated in almost equal proportions. Among the ML techniques, it is shown that 79% of them are based on Deep Learning. In 65% of cases, CNN architecture was used. However, 17% of the reviewed papers proposed a Specialize CNN architecture adapted to COVID-19. Supervised Learning is also present in 16% of cases either to make classification by using mainly SVM or to make regression where Random Forest Algorithms and Linear regression are the most dominant techniques. In addition of them, hybrid approaches are also explored to address the topic of COVID-19. They represent 5% of the reviewed methods in this paper. Most of them mix CNN with other techniques and/or meta-heuristics in order to outperform the classical ones. They demonstrated good performance in terms of accuracy and F1-Score, thus, it would be worth investigating them further. Given this state of the art and the number of techniques proposed, research must now focus on the quality of the data used and their harmonization. Indeed, until now, the studies carried out have been based on different types of datasets and different volumes of datasets. The data considered are overall those present in each country where the disease of COVID-19 has not necessarily evolved in the same way. Thus, it is essential to create benchmarks with real-world datasets to train future models on them.