Human emotion classification based on multiple physiological signals by wearable system.

BACKGROUND: Human emotion classification is traditionally achieved using multi-channel electroencephalogram (EEG) signal, which requires costly equipment and complex classification algorithms.
OBJECTIVE: The experiments can be implemented in the laboratory environment equipped with high-performance computers for the online analysis; this will hinder the usability in practical applications.
METHODS: Considering that other physiological signals are also associated with emotional changes, this paper proposes to use a wearable, wireless system to acquire a single-channel electroencephalogram signal, respiration, electrocardiogram (ECG) signal, and body postures to explore the relationship between these signals and the human emotions. RESULTS AND
CONCLUSIONS: Compared with traditional emotion classification method, the presented method was able to extract a small number of key features associated with human emotions from multiple physiological signals, where the algorithm complexity was greatly reduced when incorporated with the support vector machine classification. The proposed method can support an embedded on-line analysis and may enhance the usability of emotion classification.

Introduction

Human emotional monitoring has many applications; for example, it is essential to assess patients’ progress in rehabilitation, particularly important for those mental affected survivors from traumatic injury [1]. The research on human emotions has been among hot topics in recent years, though the science of human emotions has not yet fully understood. As a result, there are many definitions of emotion. Most commonly, the emotions can be categorized into negative and positive [2, 3]. Researchers may need more than 64 channel electroencephalogram (EEG) signal to analyze the emotions. This method requires costly EEG acquisition equipment as well as high performance workstations to implement the complex features extraction and classification algorithms [4, 5]. A large number of studies have shown that the machine learning is a feasible way to evaluate emotions. The support vector machine (SVM) and deep learning are currently well-known classification methods, and question is still yet existed – how to choose or select the effective features. EEG, a high dimension, nonlinear weak signal, is commonly used for emotion monitoring. Many emotion-related studies suggest that both time domain and frequency domain features may indicate human emotions. To evaluate the emotional state, past studies employed multi-channel EEG signals to extract sufficient features, particularly the feature revealing the differences among channels. By fusing with other features, such as the power spectral density (PSD), short-time power spectrum, and signal asymmetry, a very high dimension feature vector can be formed as the input parameter of the classifier [6, 7].

Due to the weak and high dimensionality characteristics of the EEG, preprocessing is needed before feature extraction and classification, where the electrooculogram (EOG) interference removal is one of the most critical preprocessing required [8]. Although there are many methods to deal with the interferences, those methods generally have high computational complexity [9, 10, 11]. Therefore, the EEG-based emotional classification method is difficult to be implemented experimentally, where the hardware requirements are extremely high, particularly in the real-time applications. Thereby, the experiment may only be implemented in the laboratory environment, which limits the subject’s activity area and results in a low usability [12].

The EEG analysis has been one of the most effective methods to classify emotions; on the other hand there are some easy-to-access physiological signals may also reveal emotional responses, such as respiration, electrocardiogram (ECG), and body posture [13, 14, 15, 16]. During human respiration process, the chest impedance will change periodically due to the air inhalation and exhalation. Clinical studies have shown that respiratory characteristics and emotions also have a certain relationship. For example, in a stable mood, the breathing signal is relatively stable, and in the excited time, the relative change in breathing is relatively dramatic. Similarly, ECG signals, which can be divided into P-QRS-T waves, can also indicate human emotions [17]. The ECG strength and stability of each component wave can reflect the body’s performance and emotional state. Skin conductance research is another hot topic in recent years, it can indicate the human emotion state to some extent, though the reliable and convenient detection method is a challenge [18, 19]. In some other studies, there was a certain relationship between the stability of the head posture and fatigue, but in the emotional research, the consideration of the head posture is very rare [20].

The development of emotional state detection and analysis devices, however, has limit for the suability because most of the researches are mainly conducting theoretical study. Till now, few researcher use wearable online system to analyze the emotions [21]. Based on the above, it is reasonable to combine emotion state-related signals to achieve better classification accuracy, while reducing the complexity of the algorithm and equipment costs.

According to the above reasons, this paper proposes a wearable, wireless body area sensor network to acquire those emotional state-related signals including one channel EEG. Moreover an efficient signal processing method to remove the interference and extract the useful features is presented. According to the analysis results, only key features were chosen as the input of the emotion classifier. Because of the use of easy-to-access physiological signals and the efficient algorithms, the data receiving, signal processing and emotion classification are all implemented in a wearable DSP chip [22]. Finally, this paper employed SVM to train several samples, and compared with the traditional method which only based on multi-channel EEG. The experiment results show that with the greatly reduction of the algorithm complexity, the presented method has a significant improvement of the classification accuracy. That means the human emotion classification can be widely used in many applications with the wearable system, such as psychiatric patients treatment, telemedicine and health care, and other commercial applications.

System design and experiment

Data acquisition and analysis system

In order to illustrate the usability of the method presented in this paper, a custom-made experimental device was employed. The diagram of this system is represented in Fig. 1. The system uses multiple wearable sensors to acquire various physiological signals, and uploads the data via Wi-Fi under the control of main node. With the support from DSP core, the wearable main node performs the data processing and analysis in real time. The multi-function sensor node shown in Fig. 2 enables single-channel EEG and head posture acquisition, and/or ECG and respiratory impedance acquisition. With a tiny-size setup, it can be easily placed in clothing or hat. Two identical nodes use Wi-Fi to communicate with the DSP main node at the same time. This wearable structure greatly enhances the practicality of emotion classification.

Figure 1.

System diagram.

Figure 2.

Sensor node.

In this sensor node, a 24-bit high-precision ADC is used to detect the EEG, ECG and respiratory impedance (RESP) signals, where the EEG and ECG are input directly after the RC filter and the RESP is acquired by using modulation-demodulation method ( 0.2%) [23]. The head posture is tracked by a digital gyroscope and an accelerometer on the board [24]. According to the design indicators and test results, the sampling rate and accuracy of the above physiological signals are shown in Table 1.

Table 1

Signal accuracy and sampling rate

Signal	Accuracy	Rate (Hz)
EEG	0.2 μ V	1000
ECG	4.3 μV	500
RESP	5% V_pp	500
Head posture	0.2Deg/0.004g	100

As shown in Fig. 3, a sensor node was placed in the headband to acquire the single channel EEG signal near the forehead by the medical electrodes, and the right earlobe was chosen as the reference to reduce the ECG interference. The sensor node was used to track the head posture at the same time. Another node was fixed in the chest pocket for ECG and respiratory impedance signal acquiring by a pair of common electrodes. On the other hand, in order to compare with the traditional multi-channel EEG based classification methods, 16-channel EEG signals were acquired by commercial EEG equipment (V-Amp, Germany) during the emotion experiment.

Figure 3.

System set up.

Emotion experiment

The videos provided by Shanghai Jiaotong University SEED laboratory was used to stimulate the different emotions of each subject in this paper [25]. A number of classic native language film slices was chosen and divided into positive, neutral and negative samples. Active films mostly were comedies, disaster and war history movies were defined to negative, and neutral films are sceneries or landscape appreciations.

In each experiment, the subject remained sedentary and watched a 3-mimut length slice while collecting the above physiological signals. Six subjects were studied in this paper; each of them completed 36 different experiments, including 12 positive samples, 12 negative samples and 12 neutral samples. In order to improve the reliability of the data, the sample order was assigned randomly. Figure 4 shows the experiment process and Table 2 lists some slected film slices and their corresponding emotions.

Figure 4.

Experiment process.

Table 2

Selected film slice example

Film slice	Emotion
Tangshan Earthquake	Negative
King of comedy	Positive
Guilin Scenery	Neutral
Lost in Thailand	Positive
Assembly	Negative
China from above	Neutral

Data analysis

Data preprocessing

Data preprocessing is a prerequisite for physiological signal analysis. Usually, the first step is to remove baseline drift [26]. This step is essentially to keep the body at zero potential. Since the baseline drift is very slow, a median filter was employed for the removal in order to improve the efficiency. Furthermore, a 1 60 Hz band-pass filter was applied to suppress the high-frequency interference, a 49 51 Hz band-stop Butterworth filter was chosen to remove the power line interference of 50 Hz.

As described above, the EOG interference contaminated in EEG should be removed. In this paper, a method based on Hilbert-Huang Transform (HHT) was used to filter the EOG contamination, which has been proved effective in previous studies [27, 28].

According to the physical definition, to calculate a meaningful instantaneous frequency, the signal must be with zero average symmetry in any locality, namely Intrinsic Mode Function (IMF) [28]. Therefore, for the EEG signal , it is necessary to decompose it into a linear combination of a series of IMFs and a negligible remainder .

As the EEG signal is usually maintained at 3 30 Hz, we process each IMF component according to the following rule:

where – Processed .

ECG signal is large amplitude interference, so it can be distinguished by the standard deviation. We retain the required EEG signal and replace the large-deviation signal with the mean. This step can be expressed as follows:

where – All minimal and maximal values of th IMF; – The mean of th IMF.

The pure EEG signal without ECG interference is the linear recombination of these new components , Fig. 5 shows the signal processing result. It is clear that with the great filter effect of ECG, the useful EEG signals are almost completely preserved, and the blink frequency feature can be easily obtained [12].

Figure 5.

Experiment process.

Feature extraction and discussing

Table 3 shows the easy-extracted and commonly used features of the detected physiological signals in this paper. Percentage power spectral density is an important frequency domain feature of physiological signals, for example, the EEG signal is divided to (1–4 Hz), (4–8 Hz), (8–13 Hz) and (13–30 Hz) wave [29]. Figure 5 shows 9 experiment results about relationship between and emotions by subject 1. In the positive emotions, the value of might be larger, while in negative emotions, the value of might be small. On the other hand, other three PSD features have the opposite trend. If the is defined as the emotion factor (from negative to neutral to positive monotonous), which is described by Eq. (5).

Table 3

Selected film slice example

Signal	Feature	Symbol
EEG	Percentage PSD	𝐸𝐷_k
	Standard deviation	Estd
	Average power	Epow
	Mean of the absolute value	Emean
	Blink frequency	Fb
ECG	Heart rate	HR
	Heart rate stability	HRstd
	Power	Hpow
RESP	Respiratory rate	RR
	Respiratory stability	RRstd
	Absolute mean of second order difference	𝑅𝑚𝑒𝑎𝑛_𝑑𝑖𝑓𝑓𝑖𝑖
	Percentage PSD	𝑅𝐷_k
	Standard deviation of second order difference	𝑅𝑠𝑡𝑑_𝑑𝑖𝑓𝑓𝑖𝑖
Head posture	Posture stability	Pstd

This result was consistent with some previous research findings where was extracted from multi-channel EEG. Combined with other EEG features, including standard deviation or average power, the accuracy of emotion classification reached 75% by linear SVM. When the difference entropy feature was added, some research results show that the accuracy increased to 85%, but the algorithm complexity became extremely high [30, 31].

As mentioned above, ECG, RESP and head posture features can also reflect to emotional states. For example, the standard deviation of the time interval of ECG pulse peaks can be used as the heart rate stability as denoted in Eq. (6). Figure 6 shows the relationship between ECG feature (average value in 36 samples) and the emotional states from all subjects.

Figure 6.

HRstd average values in different emotions.

where – Time of the th ECG pulse peak; – Standard deviation of .

All six objects showed that the HRstd was higher in positive emotion state, while the negative emotion state was associated with a low HRstd (except for Subject 5). Also, the RESP and head posture features were related to emotional states according to the experiment result. Figures 7 and 8 show the RRstd and Pstd analysis results, where Pstd is the total time of the head significant movement in the experiment. The RRstd analysis result indicates that although it cannot distinguish between neutral and positive emotions, the values were consistently low in negative emotion state. Similarly, Pstd average values were usually high during the neutral emotion except Subject 6, which indicates that subjects were not easy to maintain attention in the neutral emotions.

Figure 7.

RRstd average values in different emotions.

Figure 8.

Pstd average values in different emotions.

Feature selection

This study conducted a test to quantify the relationship between each feature and emotion. In each test, with one of those features was used, and the linear SVM was employed to classify the emotion states. The average classification accuracy of all objects is shown in Table 4. In order to maximize the final classification accuracy, the features that may improve the classification accuracy were chosen in this study, i.e., the test had more than 72.5% accuracy. The features selected for emotion classification were , Estd, Epow, Emean, Fb, HRstd, RRstd, , , and Pstd.

Table 4

Classification accuacy of with another feature

Another feature	Accuracy (%)
-(𝐸𝐷_k only)	72.5
Estd	78.1
Epow	74.6
Emean	73.3
Fb	75.6
HR	69.0
HRstd	84.2
Hpow	70.4
RR	71.9
RRstd	75.5
𝑅𝑚𝑒𝑎𝑛_𝑑𝑖𝑓𝑓𝑖𝑖	85.2
𝑅𝐷_k	72.7
𝑅𝑠𝑡𝑑_𝑑𝑖𝑓𝑓𝑖𝑖	86.0
Pstd	77.6

Classification and discussion

According to the above study, a total of 11 features were chosen for emotional classification in this paper. It can be noted from the analysis of Paragraph 3.2 that neutral and non-neutral emotions can be easily distinguished by the RESP and head posture features; thereby, two SVM was employed to classify emotions as shown in Fig. 9. The first one was used to distinguish between neutral and non-neutral emotions, and the second was used to distinguish non-neutral emotions into positive or negative [32]. For comparison, 16-channel EEG signal was collected during all experiments. A classical SVM classification method based on multi-channel differential entropy and PSD features was used, in which EOG interference in EEG was processed by a classical ICA algorithm. The accuracy of classification and the time consuming by the wearable DSP processor are compared in Table 5. All results show that compared with the classical emotion classification method, the presented method has a significant improvement of the classification accuracy and the time consuming has a great reduction.

Figure 9.

Classification scheme.

Table 5

Classification accuracy and time consuming

Subject	Proposed method		Classical method
	Accuracy (%)	Time (second)	Accuracy (%)	Time (second)
1	92.5	17	89.3	339
2	85.7	18	78.3	342
3	90.5	18	87.4	349
4	90.2	19	85.5	378
5	85.3	18	83.2	351
6	84.2	18	83.7	343

Conclusions and future work

For the purpose to reduce algorithm complexity and costly EEG equipment requirements, this paper presents an innovative method for emotional state classification based on multiple physiological signals, including EEG, ECG, respiration and head posture. Although this method requires the acquisition of multiple types of physiological signals, the total number of signals is greatly reduced and all selected features are easy to extract. In order to reduce hardware cost, an efficient EOG interference removal method was proposed in this paper, which makes it possible to realize all calculations on the wearable system in real time. By using the proposed method, the emotion classification becomes more practical, and users can wear this system to walk or work freely.

According to the validation tests, eleven easy-extracting features from single channel EEG, ECG, respiration impedance and head posture were proved to be related to emotions, which were selected for emotional classification. The experiment results show that compared with the traditional multi-channel EEG-based methods, the proposed method with SVM has a significantly reduction of cost while the classification accuracy has improved greatly.

This paper proves the validity and usability of the emotion classification method based on multi-physiological signals; but there is still a lot of work to be done. Firstly, this paper only discussed the relationship between a portion of features from physiological signals and emotion states briefly. A comprehensive feature selection method will help to enhance the effect of emotional classification. Secondly, the emotion states can be categorized into more types, for example, negative emotion can be fear, sad or anger, at least six primary emotions (happy, sad, anger, fear and disgust) can be identified in fact. Another problem is that the developed system is subject-dependent; the user needs to train the system before emotion classification. In the future, a small sample database will be integrated in this system, so that by using of migration learning, the system might have a better robustness, which is under investigation.