Zhong Yin1, Mengyuan Zhao2, Yongxiong Wang3, Jingdong Yang4, Jianhua Zhang5. 1. Engineering Research Center of Optical Instrument and System, Ministry of Education, Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology, Shanghai, 200093, PR China. Electronic address: yinzhong@usst.edu.cn. 2. School of Social Sciences, University of Shanghai for Science and Technology, Shanghai, 200093, PR China. 3. Engineering Research Center of Optical Instrument and System, Ministry of Education, Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology, Shanghai, 200093, PR China. Electronic address: wyxiong@usst.edu.cn. 4. Engineering Research Center of Optical Instrument and System, Ministry of Education, Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology, Shanghai, 200093, PR China. 5. Department of Automation, East China University of Science and Technology, Shanghai 200237, PR China.
Abstract
BACKGROUND AND OBJECTIVE: Using deep-learning methodologies to analyze multimodal physiological signals becomes increasingly attractive for recognizing human emotions. However, the conventional deep emotion classifiers may suffer from the drawback of the lack of the expertise for determining model structure and the oversimplification of combining multimodal feature abstractions. METHODS: In this study, a multiple-fusion-layer based ensemble classifier of stacked autoencoder (MESAE) is proposed for recognizing emotions, in which the deep structure is identified based on a physiological-data-driven approach. Each SAE consists of three hidden layers to filter the unwanted noise in the physiological features and derives the stable feature representations. An additional deep model is used to achieve the SAE ensembles. The physiological features are split into several subsets according to different feature extraction approaches with each subset separately encoded by a SAE. The derived SAE abstractions are combined according to the physiological modality to create six sets of encodings, which are then fed to a three-layer, adjacent-graph-based network for feature fusion. The fused features are used to recognize binary arousal or valence states. RESULTS: DEAP multimodal database was employed to validate the performance of the MESAE. By comparing with the best existing emotion classifier, the mean of classification rate and F-score improves by 5.26%. CONCLUSIONS: The superiority of the MESAE against the state-of-the-art shallow and deep emotion classifiers has been demonstrated under different sizes of the available physiological instances.
BACKGROUND AND OBJECTIVE: Using deep-learning methodologies to analyze multimodal physiological signals becomes increasingly attractive for recognizing human emotions. However, the conventional deep emotion classifiers may suffer from the drawback of the lack of the expertise for determining model structure and the oversimplification of combining multimodal feature abstractions. METHODS: In this study, a multiple-fusion-layer based ensemble classifier of stacked autoencoder (MESAE) is proposed for recognizing emotions, in which the deep structure is identified based on a physiological-data-driven approach. Each SAE consists of three hidden layers to filter the unwanted noise in the physiological features and derives the stable feature representations. An additional deep model is used to achieve the SAE ensembles. The physiological features are split into several subsets according to different feature extraction approaches with each subset separately encoded by a SAE. The derived SAE abstractions are combined according to the physiological modality to create six sets of encodings, which are then fed to a three-layer, adjacent-graph-based network for feature fusion. The fused features are used to recognize binary arousal or valence states. RESULTS: DEAP multimodal database was employed to validate the performance of the MESAE. By comparing with the best existing emotion classifier, the mean of classification rate and F-score improves by 5.26%. CONCLUSIONS: The superiority of the MESAE against the state-of-the-art shallow and deep emotion classifiers has been demonstrated under different sizes of the available physiological instances.
Authors: Mohammad Mahdi Shiraz Bhurwani; Kenneth V Snyder; Muhammad Waqas; Maxim Mokin; Ryan A Rava; Alexander R Podgorsak; Kelsey N Sommer; Jason M Davies; Elad I Levy; Adnan H Siddiqui; Ciprian N Ionita Journal: Proc SPIE Int Soc Opt Eng Date: 2021-02-15
Authors: Muhammad Awais; Mohsin Raza; Nishant Singh; Kiran Bashir; Umar Manzoor; Saif Ul Islam; Joel J P C Rodrigues Journal: IEEE Internet Things J Date: 2020-12-10 Impact factor: 10.238