Performance analysis of data mining algorithms for diagnosing COVID-19.

BACKGROUND: An outbreak of atypical pneumonia termed COVID-19 has widely spread all over the world since the beginning of 2020. In this regard, designing a prediction system for the early detection of COVID-19 is a critical issue in mitigating virus spread. In this study, we have applied selected machine learning techniques to select the best predictive models based on their performance.
MATERIALS AND METHODS: The data of 435 suspicious cases with COVID-19 which were recorded from the Imam Khomeini Hospital database between May 9, 2020 and December 20, 2020, have been taken into consideration. The Chi-square method was used to determine the most important features in diagnosing the COVID-19; eight selected data mining algorithms including multilayer perceptron (MLP), J-48, Bayesian Net (Bayes Net), logistic regression, K-star, random forest, Ada-boost, and sequential minimal optimization (SMO) were applied in data mining. Finally, the most appropriate diagnostic model for COVID-19 was obtained based on comparing the performance of the selected algorithms.
RESULTS: As the result of using the Chi-square method, 21 variables were identified as the most important diagnostic criteria in COVID-19. The results of evaluating the eight selected data mining algorithms showed that the J-48 with true-positive rate = 0.85, false-positive rate = 0.173, precision = 0.85, recall = 0.85, F-score = 0.85, Matthews Correlation Coefficient = 0.68, and area under the receiver operator characteristics = 0.68, respectively, had the higher performance than the other algorithms.
CONCLUSION: The results of evaluating the performance criteria showed that the J-48 can be considered as a suitable computational prediction model for diagnosing COVID-19 disease. Copyright:

Introduction

COVID-19 is a zoonotic, contagious, and infectious disease caused by a virus called acute respiratory syndrome coronavirus 2,[1] which causes serious damage to the respiratory system, creating pneumonia, and in some cases leads to human death.[23] This disease was first reported in December 2019 in Wuhan, Hubei Province in China, and gradually, spread throughout the country and then the world.[4] The most important symptoms of this disease can be mentioned as fever, dry cough, shortness of breath, headache, abdominal pain, gastrointestinal symptoms, vomiting, chest tightness, lung lesions, and respiratory failure.[56] The viral disease can be transmitted through oral and nasal secretions of the infected people when coughing or sneezing or talking to healthy people.[7] The incubation period of this disease in most people is up to 14 days, but in some cases, up to 24 days has been reported.[8] The number of cases and deaths is increasing worldwide, while Iran is among the top 10 countries in terms of the prevalence of the disease.[9] According to data published by the World Health Organization (WHO), by August 16, 2020, the number of cases of corona disease has reached 22,492,312 people, the number of deaths due to this disease has been announced to 788,503 people, in Iran, in this period, there are 352,558 cases and 20,264 deaths due to this disease.[10] Due to the rapid spread of this disease, subsequently, the increasing number of this disease, and deaths worldwide, as well as adverse effects on the economic and social conditions of societies, the use of new and advanced technological methods to quickly diagnose the disease in the early stages, isolation and quarantine cases after that, and effective monitoring are very important in preventing the disease.[1112] Furthermore, due to the ambiguous and sensitive nature of this disease and the lack of proven therapies and the lack of methods for rapid detection of the disease, the use of innovative and noninvasive methods to identify the disease in earlier stages can be very useful and determinative.[13] The most important new and advanced technologies today that have a great role in controlling the epidemic of COVID-19 can be pointed to artificial intelligence (machine learning and deep learning), cloud computing, the internet of things (IoT), big data mining, robotic technologies, and intelligent systems.[1415] The data mining process involves discovering important patterns from the vast amount of data that combines machine learning, statistics, and database systems.[9] Given the huge amount of data available in the field of medicine, this process has so far played a very effective role in the management of various diseases such as prognosis, diagnosis, and treatment.[1617] So far, several studies have been conducted on the application of data mining methods in the management and control of COVID-19 disease with the aim of monitoring and predicting disease trends and epidemiological assessments, providing diagnostic models, determining the best treatment and care model, and predicting the probability of death.[1819] In a review study by Albahri et al., on the role of data mining and artificial intelligence in the detection and diagnosis of COVID-19, the study's quantitative and qualitative criteria were discussed. Finally, the results showed that the use of this technology to provide diagnostic models and identify the most optimal and effective data mining algorithm can be significantly helpful in a timely, effective, and economical diagnosis of the disease.[20] Hence, due to the high prevalence of the disease in our country and the existence of some limitations and scarcity of resources, the purpose of this study was to create an effective and efficient diagnostic model based on comparing the performance of data mining algorithms for COVID-19 disease based on different diagnostic criteria to be able to be a suitable solution for physicians to increase their diagnostic accuracy and reduce the prevalence and death rate in our community.

Materials and Methods

This research methodology consisted of three steps as follows.

Study design and setting

This retrospective and the descriptive-applied study was performed in 2020 in the central hospital for diagnosis and treatment of COVID-19 disease affiliated with Ilam University of Medical Sciences aimed to develop a Clinical Decision Support System based on various data mining techniques and to present the best diagnostic model.

Study participants and sampling

The population was people who had been referred to Mostafa Khomeini Hospital in Ilam city for COVID-19 diagnosis and treatment. The data of 435 COVID-19 and non-COVID-19 cases (positive or negative reverse transcription-polymerase chain reaction) were stored in the electronic medical records (EMRs) of the patient database along with the 40 diagnostic criteria. Out of this number, the information of the 35 cases which had a lot of missing data (more than 70%) was excluded from the analysis. Furthermore, the missing value was imputed with the mean or mode of each variable. The final sample size used in this analysis was 400 COVID-19 and non-COVID-19 cases.

Data collection tool and technique

The data of the 400 cases of patients with the COVID-19 disease and non-COVID-19 were stored on the center's EMRs to be used for data mining. The diagnostic criteria used in the study included demographic findings, clinical and care, medical and personal history, and epidemiological data. The output variable of the study was the same as the diagnostic result of COVID-19 disease, in which the numbers of 0 and 1 were assigned to people who had a negative or positive COVID-19 disease test result.

Ethical consideration

This study was approved by the ethical committee board of Ilam University of Medical Sciences (Ethic code: IR.MEDILAM.REC.1399.294).

Statistical analysis

In this step, before performing the data mining process, due to the number of the diagnostic fields in the research and also to increase the efficiency of the results obtained from different data mining algorithms, some diagnostic criteria that were less important in the research were removed to have a maximum speed to improve the data mining performance. Therefore, because of the existence of two and multistate qualitative variables, the Pearson Chi-square method was used. Using this statistical method, all the diagnostic criteria were weighted according to their degree of importance. The basis of this statistical method is the comparison between the obtained actual value and the expected value, which indicates that the difference between the actual and expected values is statistically significant and the results are completely random.

The formula for calculating the Chi-square along with how to calculate the expected frequency has been shown in Equations 1 and 2, respectively.

In Equation 1, the Fo is the observed frequency and Fe is the expected frequency obtained from Equation 2. Furthermore, in this study, the method of selecting the most important diagnostic criteria is based on the Chi-square criterion, at P < 0.05.

Data mining and performance evaluation

After determining the most important diagnostic criteria based on the Pearson method and Chi-square test, data mining was performed using WEKA 3.9 software. The eight well-known data mining algorithms including multilayer perceptron (MLP), J-48 decision tree, Bays Net, logistic regression (LR), K-star, random forest (RF), Ada-boost, and sequential minimal optimization (SMO) were used to evaluate and generate the best COVID-19 diagnostic model. The 10% fold cross-validation had been considered for measuring the performance criteria. The performance of the sample classification rate of these data mining algorithms was evaluated based on the true positive (TP) (The COVID-19 cases are truly classified as positive by the data mining model), false positive (FP) (The non-COVID-19 cases falsely classified as positive by the data mining model), false negative (FN) (The COVID-19 cases are falsely classified as negative by the data mining model), and true negative (TN) (The non-COVID-19 cases are truly classified as negative by the data mining model), and their performance was evaluated based on various evaluation criteria such as TP-Rate, FP-Rate, Precision, Recall, F-Score, Matthews Correlation Coefficient (MCC), and the area under the receiver operator characteristics (AUC) diagram of each algorithm in WEKA software were examined and compared.

Results

After weighing the research variables, the most important diagnostic criteria were obtained, 21 COVID-19 diagnosing variables had a meaningful correlation with the output class (positive and negative COVID-19 diagnosis) at P < 0.05. Table 1 shows the most important diagnostic criteria, which were weighted using the Chi-square relationship.

Table 1

The most important COVID-19 diagnostic criteria

Variable name	Variable type	Features	χ 2
Lung lesion existence	Binominal	Haven’t	179.21
		Have
Fever	Binominal	Haven’t	113.26
		Have
Contact with suspected people	Binominal	Haven’t	111.26
		Have
O2 saturation in the blood	Poly nominal	>95%	102.4
		85%-95%
		<85%
Rhinorrhea	Binominal	Haven’t	96.4
		Have
Dyspnea	Binominal	Haven’t	90.1
		Have
Digestive sign (diarrhea)	Binominal	Haven’t	81.7
		Have
Nausea and vomiting	Binominal	Haven’t	75.5
		Have
Traveling to high-risk area history	Binominal	Haven’t	63.3
		Have
History of use of immunosuppressive drugs	Binominal	Haven’t	58.2
		Have
History of respiratory failure	Binominal	Haven’t	46.6
		Have
History of respiratory tract infection	Binominal	Haven’t	40.2
		Have
Cough	Binominal	Haven’t	34.2
		Have
History of taking Vitamin D	Binominal	Haven’t	33.6
		Have
Disability sensation	Binominal	Haven’t	32.9
		Have
Chest pain	Binominal	Haven’t	31.1
		Have
Tremor	Binominal	Haven’t	30.8
		Have
Age	Poly nominal	Young	29.7
		Middle-aged
		Adult
Headache	Binominal	Haven’t	27.6
		Have
Consciousness	Poly nominal	Complete	27.3
		Relative
		Without
Throat pain	Binominal	Have	25.5
		Haven’t

Based on the findings have been represented in Table 1, the variable of lung lesions with the highest Chi-square value (χ2= 179.21) was considered as the most important diagnostic criterion for COVID-19 disease. The results of classifying data mining algorithms from people with disease and people without the disease are shown in Table 2.

Table 2

The results of sample classification with COVID-19 disease and non-COVID-19

Algorithm	TP	FP	FN	TN
MLP	211	39	29	121
J-48	221	29	31	119
Bayes net	155	95	1	149
LR	211	39	32	118
Ada-boost	231	19	64	86
RF	216	34	34	116
K-star	218	32	33	117
SMO	210	40	32	116

MLP=Multilayer perceptron, LR=Logistic regression, RF=Random forest, SMO=Sequential minimal optimization, TP=True positive, FP=False positive, FN=False negative, TN=True negative

Based on information obtained from Table 2, the Ada-boost data mining algorithm with the correct classification of the 231 cases (out of 250) of people with COVID-19 disease had the highest classification ability for positive COVID-19 cases, But the Bayesian Net (Bayes Net) with 149 correct samples of people without the disease out of the 150 had the best classification compared to the other data mining algorithms. The results of measuring the TP-Rate, FP-Rate, Precision, Recall, and MCC of all selected data mining algorithms with 10% fold cross-validation are shown in Figure 1.

Figure 1

Comparing the performance of different data mining algorithms

The results of comparing the performance of algorithms based on the TP-Rate, FP-Rate, Precision, Recall, and MCC showed that the Bayes Net with the FP-Rate of 0.147 better than the other algorithms had the higher performance in classifying the samples. The J-48 and Bayes Net with the Precision of 0.85 had the best performance in categorizing the sample, but generally, the J-48 data mining algorithm with the TP-Rate, FP-Rate, Precision, Recall, and MCC of 0.85, 0.173, 0.85, 0.85, and 0.68 had better performance than the other algorithms, respectively. Figure 2 shows the F-Score obtained from different data mining algorithms.

Figure 2

The comparison of the F-Score of the different data mining algorithms

Figure 2 shows that the F-score obtained from the J-48 data mining algorithm was higher than the other algorithms (0.85); therefore, it had a better performance than other algorithms. The Bayes Net (0.761) had the worst performance in this respect. The results of the AUC of each of the data mining algorithms are shown in Figure 3. Horizontal and vertical vertices show a true-positive rate (TPR) and a false-positive rate, respectively.

Figure 3

The receiver operator characteristics of different data mining algorithms

All algorithms used in this study had a ROC value above 0.9 (except the SMO algorithm with AUC = 0.80) and the classification of the research samples was close to TPR, which indicated the high classification strength of the algorithms, but the J-48 classification algorithm with a ROC value of 0.917 with a slight difference compared to the other algorithms had a better performance, relatively.

Comparing the performance of the data mining algorithms based on the different evaluation criteria such as TP-rate, FP-rate, precision, recall, MCC, F-score, and ROC obtained from different algorithms, generally, the J-48 data mining algorithm had a higher performance than other algorithms. Figure 4 shows a pruned J-48 decision tree with the features described below along with a maximum algorithm efficiency based on the most important diagnostic criteria were considered by the tree.

Figure 4

The pruned J-48 algorithm in diagnosing COVID-19

Some important characteristics of the J-48 decision tree algorithm method had been used for sample classification with the highest performance.

The J-48 important characteristics for making tree

Batch size = 100, confidence factor = 0.2, minimum number of objects = 2, number of folds = 3, and number of seeds = 1.

Based on the decision tree drawn, the root node is the presence of lung lesions in the individual, which was considered as the most important diagnostic criterion based on the Chi-square. Based on this decision tree, all COVID-19 disease diagnostic rules were extracted with 85% diagnostic accuracy. The simplest rule derived from this tree was that if a person had a lung lesion, he or she was considered as a person with COVID-19 disease, which was confirmed by the 148 samples of the research. The interpretation of the other rule derived from this decision tree was that if a person had no lung lesions, fever, and blood oxygen saturation >95%, the decision tree algorithm would classify this person with 89% probability in the group with non-COVID-19 disease. In this leaf node, 85 samples were classified, from which the 9 samples were classified incorrectly, and therefore, 76 samples confirmed the diagnosis of non-COVID-19 disease (89%).

Discussion

Today, COVID-19 disease has become a universal problem that has endangered the health, social, economic, and developmental conditions of many countries; thus, WHO in its report has identified the disease as a serious threat to community health.[2122] In such circumstances, the need to use new and advanced technologies to monitor disease trends, track contacts, and identify the disease's patterns as well as to prevent, screen, treat, and monitor the patients has become more realized.[2324] To counter this threat to public health, many countries have emphasized the need to use technological infrastructure.[25] During the COVID-19 epidemic, the promotion of public health potential in the fight against coronavirus requires attention to the fact that the cause of problems and risks threatening public health, as well as solutions to these problems must follow a complex technological adaptation system.[26] The digital health technology revolution in the early 2020s was marked by the epidemic of coronavirus due to the widespread use of artificial intelligence, cloud computing, IoT, and big data mining technologies.[27] Today, the use of innovative and advanced computing technologies in the form of machine learning systems, deep learning, and big data mining, provides deep, effective, and noninvasive analytical capabilities to support the decision of physicians and health policymakers over traditional statistical methods, clinical trials, and evaluations.[28] With the spread of COVID-19 disease and the widespread involvement of many communities, the health-care systems of many countries will not be able to meet the growing needs of patients to diagnose, treat, and care for. In such circumstances, the design and implementation of clinical decision-making systems equipped with custom and effective predictive algorithms will be critical to identify patients and differentiate them from healthy individuals as well as individuals with other similar respiratory diseases.[29] Due to the sensitive, ambiguous, and multidimensional nature of the disease and also the lack of specific treatment, long incubation period, relatively difficult paraclinical evaluations, and differential diagnosis, clinical decision technologies play an important role in improving the quality of diagnosis and provide customized care. Furthermore, increasing the number of infected cases, rapid and effective identification of patients, proper prioritization of health resources, screening, and implementation of quarantine programs help reduce the burden of the disease.[30] Therefore, the main purpose of this study was to evaluate the performance of some data mining models such as MLP, J-48 Decision Tree, Bayes Net, LR, Ada-boost, SMO, RF, and K-star for diagnosing COVID-19 disease and differentiating between infected and healthy individuals to improve the quality and effectiveness of clinical decisions. In the field of data mining applications (machine learning and deep learning), several types of research have been done, among which the comparison of different algorithms to suggest the most efficient ones present an optimal diagnostic model is on the agenda. The results of Alakus study (2020) to design a diagnostic model of COVID-19 based on deep learning showed the precision, F1 score, accuracy, recall, and AUC of 88.66%, 91.89%, 86.75%, 99.42%, and 62.50% of the system to diagnose the disease, respectively.[31] In the study of Narin et al., the performance of three types of artificial neural networks (ANNs) of ResNet50, InceptionV3, and Inception-ResNetV2 was compared to diagnose the COVID-19 disease. The results showed the ANN of ResNet50 with 98% accuracy was more efficient than the other data mining algorithms.[32] In the study of Elaziz et al., they used the machine learning method to analyze the chest image in individuals and the algorithm was able to identify people with COVID-19 disease and healthy individuals based on chest image analysis, the results of the algorithm evaluation showed that the accuracy of 96% and 98% in the two different data sets of images.[33] In a study conducted by Brunese et al., an interpretive deep learning method was developed for the rapid detection of lung and COVID-19, by performing the statistical analysis on 6523 chest photographs belonged to the different institutions, the results of the study showed a 97% accuracy in 2.5 s in diagnosing the disease.[34] In Rodriguez's study, by comparing the two mathematical methods (Gompertz and logistic), and a computational method (ANN), the results showed better performance of the computational models in diagnosing COVID-19.[35] In the present study, based on comparing the performance of the data mining algorithms considering the different evaluation criteria such as TP-Rate, FP-Rate, Precision, Recall, MCC, F-Score obtained, the J-48 data mining algorithm with 0.85, 0.173, 0.85, 0.85, 0.68, and 0.85 of these evaluation criteria, respectively, had a higher diagnostic performance than the other algorithms.

In this research, by introducing a noninvasive, effective, and evidence-based method, we will be able to identify and propose the best diagnostic model for COVID-19 based on the most effective diagnostic criteria. The results of evaluating the performance of the different data mining techniques showed that the proposed diagnostic models based on machine learning have a higher performance than the traditional paraclinical approaches, and also can be used by physicians to improve their diagnostic performance.

Limitation and suggestion

This study had several limitations that need to be addressed. First, this is a backward-looking study that there was some documented data that was erroneous, irregular, uneven, or imbalanced. Thus, we balanced them by manual checking of noisy and abnormal values, errors, duplicates, and meaningless data. Furthermore, by using the Synthetic Minority Oversampling Technique, the bias can be minimized via class balancing. Furthermore, the missing values were imputed with the mean or mode of each variable. To solve the imbalanced data set problem, in which the number of records related to the positive class is significantly lower than the negative, different criteria were measured to assess the performance of each ML algorithm. Second, this was a retrospective study based on a single-center registry system with a limited number of samples. This may limit the generalizability of the proposed model. Nevertheless, we will use a multicenter dataset to perform an external validation of the proposed model to augment the generalized prediction. In the future, the performance of our computational model will be improved if we test more classification techniques at multiorganization datasets. Furthermore, larger cohorts, prospective settings, and clinical trials are needed before elucidating its contribution to improving the outcome of COVID-19.

Conclusion

Given the limited capacity of the health-care industries to deal with the current pandemic, effective prognosis, diagnosis, and triage of patients can help in properly managing, scheduling, and maximize the use of restricted hospital resources. Designing a true and valid diagnostic model may be improving the quality of care and increasing the survival rate of the patients. Therefore, the diagnostic model can greatly contribute to identifying high-risk patients and the adoption of the most effective support and treatment plans. This study may assist clinicians in enabling early detection, effective intervention, and possibly a decrease in the death of COVID-19 patients. This led to decreasing ambiguity by offering quantitative, objective, and evidence-based models for risk stratification, prediction, and eventually episode of the care plan. It offers a better strategy for clinicians to lessening the complications and improved patient survival likelihoods.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.