Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual learning environments.

In this work, the affective state of users in virtual learning environments is assessed/recognized in terms of continuous arousal and valence dimensions, making use of multimodal information (audio, text and video), whenever any of these modalities are available. In general, virtual learning environments where these three modalities are all the time, are not common; at some moments only the video modality is available, while in others only text or/and video and/or audio. Different approaches using feature-level fusion and decision-level fusion are proposed for multimodal recognition with missing data. Recognizing according to available modalities is studied following the ideas of dropout from neural networks and of variable input length from recurrent neural networks. This proposal is innovative because it represents emotions in the continuous space, which is not common in virtual education; and makes use of the available modalities in a virtual environment in a given moment, which is very common in virtual learning environments because the people are not speaking or writing all the time.

Introduction

Virtual education has been gaining great popularity around the world due to cheapening of communication devices, globalization of the use of the internet, advances in connectivity and universal access to educational resources. In addition, governments are every day more interested in virtual education for the lower costs compared with classroom education, accessibility, ease of schedules, among other aspects. Additionally, recent advances in communication technology make virtual education available to all countries.

According to [1, p. 1] “There is empirical evidence in the psychological literature that affective states affect cognitive processes like memorizing and decision making.”. In general, the relationship between emotional states and the learning process has been widely studied in the psychological field, and several scientific studies link emotions to education [2], [3], [4], [5], [6], [7]. According to these studies, emotions affect the way people learn, they are very correlated with essential elements of student's self-regulated learning, such as interest, motivation, and strategies of learning [8].

Recognizing the affective states in virtual education has particularities that must be taken into account for an accurate assessment, mainly, the sources of information for recognizing affective states should not be obtrusive nor invasive for not perturbing the learning process. For example, many works use electroencephalograms for recognizing emotions in people [9], [10], [11], [12], which cannot be used in an educational context because this sensor can be annoying for students, disturbing the natural learning process. Sources of information must allow the free development of the learning process, and these could be cameras, microphones, text from chats, reviews, etc. Another important consideration is that in virtual learning environments not all the sources of information (also called modalities) are available at every moment. In a virtual learning environment, a student is not all the time talking or writing. Due to this, the system for recognizing the affective states in virtual education must consider the availability of modalities at all times.

During the recognition of affective states, the more information available then the more accurate the recognizing can be. A single instant can not reflect all the context of a situation, periods of time must be considered rather than single moments. In a single instant, an image can suggest that a person is asleep, but in a video, a sequence of images can show that it was only a blink. This work aims to recognize affective states in periods of time instead than on specific moments, using information from all the sequence of moments that compose the period of time.

No one modality is good enough for recognizing an emotion in an arousal and valence emotional model, some are very accurate for arousal while others are more for valence [13], which also depends on the situation. Mixing multiple modalities help strengthen weaknesses among themselves, and therefore, having a more robust affective state recognizer. Multimodality offers more points of view for taking a decision. In our proposal, we consider three modalities: audio, video and text, which are not-intrusive or invasive to the learning process and can be captured from affordable devices, such as integrated cameras or microphones.

The main contribution of this work is to propose and compare approaches for the recognition of emotional states in terms of arousal and valence dimensions from various non-intrusive and affordable modalities, using only the available modalities at each specific moment, oriented to the context of the virtual education.

The rest of the paper is structured as follows: Section 2 exposes the theoretical framework of this work. In Section 3 related works are reviewed and compared with our approach. The proposed approach is described in Section 4. Experiments and implementation details are given in Section 5. In Section 6 results and analysis are provided. Finally, in Section 7 some conclusions, discussions and future works are presented.

Theoretical framework

Multimodal data fusion

Modalities represent different sources of information, which can suggest different information, different points of view. To obtain a global perspective, the system must have all the available information. In multimodal emotion recognition, different modalities can predict different affective states. Each modality has advantages and weaknesses: audio modality is better than video modality in poor lighting conditions, and text modality can be better than audio modality for predicting valence dimension under certain circumstances. Since modalities can have flaws in specific situations, information from other modalities must be considered for improving recognition in those scenarios. The idea behind multimodal fusion is to strengthen the weaknesses of each modality using the joint information to obtain a more robust system.

For multimodal data fusion, there are two main approaches: feature level fusion and decision level fusion. Fig. 1 graphically shows the differences between them. The first consists of fusing the characteristics extracted from the different modalities (several works use concatenation as a fusion technique), and then, using a model for predicting the target from the fused characteristics. The second requires unimodal models before fusion for making separate predictions of each modality, and then, fusing these predictions using an ensemble model to obtain a final prediction.

Figure 1

Differences between decision level and feature level fusion.

Models for representing of emotions

The representation of emotions has been studied during the last century by different psychologists. There are two groups of models for representing emotions: categorical and continuous models. Categorical models use discrete values for representing emotions, and these models are more intuitive for humans. For example, Ekman proposed a model with 6 discrete emotions [14]: Anger, Disgust, Fear, Happiness, Sadness and Surprise. In Table 1 various categorical models are exposed.

Table 1

Examples of categorical emotion models (table extracted from: [15]).

Theorist	Basic Emotions
James (1884)	Fear, love, grief, rage
McDougall (1926)	Anger, elation, fear, disgust, subjection, wonder, tender-emotion
Watson (1930)	Love, fear, rage
Arnold (1960)	Love, fear, anger, aversion, courage, dejection, desire, despair, hate, sadness, hope
Mowrer (1960)	Pain, pleasure
Izard (1971)	Fear, anger, contempt, joy, distress, guilt, interest, shame, surprise, disgust
Plutchik (1980)	Fear, Acceptance, anger, anticipation, disgust, sadness, surprise, joy
Ekman, Friesen, and Ellsworth (1982)	Fear, anger, disgust, joy, sadness, surprise
Gray (1982)	Rage and terror, anxiety, joy
Panksepp (1982)	Fear, expectancy, rage, panic
Tomkins (1984)	Fear, anger, interest, contempt, disgust, distress, shame, surprise, joy
Weiner and Graham (1984)	Happiness, sadness
Frijda (1986)	Desire, happiness, interest, surprise, wonder, sorrow
Oatley and Johnson-Laird (1987)	Anger, disgust, anxiety, sadness happiness

The second group consists of continuous models with one or various dimensions for representing affective states. Empirical evidence suggests that continuous models are more accurate with respect to discrete models [13]. In this work, the arousal-valence dimensional model is used, the valence dimension for the pleasure level and the arousal dimension for the excitement of the person. In Fig. 2, an example of a map of some emotional terms for an arousal-valence continuous model is presented (various psychologists propose different mappings).

Figure 2

Example of mapping of discrete emotions to the valence-arousal dimensional model (figure taken from: [16]).

Related works

Multimodal emotion recognition has several results recently. The authors of [17] make an amazing job of reviewing unimodal and multimodal approaches for emotion recognition for the modalities of audio, video and text. The paper [18] presents a review where most of the papers work with categorical models for the representation of the emotions, and just a few utilize continuous dimensions.

The work in [19] proposes a two steps approach for recognizing emotions in educational contexts, the first step indicates if in a period of time there is the presence of an educative relevant emotion, and if there is, the second step recognizes that emotion. They use video and physiological signals (heart rate, breath, skin conductance and temperature) as modalities. These last modalities are obtrusive for the learning process due to the devices required for the measure, additionally, those devices are not affordable for all students of virtual education. Salmeron also experimented with low intrusive modalities for affect recognition in [20], using interactions with mouse and keyboard from students in order to predict emotions and valence states in SAM [21] scale, but they do not consider the availability of these modalities at every moment. The investigation published in [22] uses the audio modality for detecting emotions for classifying positive and negative emotional clips. They extract a set of features based on the feature set proposed in the Interspeech challenge [23], and classify the audio signal in categories using a linear support vector machine (SVM) model. They reported 75.51% of accuracy. The paper [24] extracted features from audio, visual and text modalities, in order to recognize categorical emotions. They used different datasets as training set and testing set for visual and textual modalities, but the audio was trained and tested with the same dataset. They concatenated the characteristics from the different modalities, and then used SVM as a classifier. They reported 87.95% of average accuracy. In [25], features for two modalities (video and text) were extracted using neural networks. They also used the Interspeech feature set for audio features. Then, they concatenated the features and used a recurrent multiple kernel learner as a classifier. They reported 76.85% of average accuracy for the IEMOCAP dataset [26]. These three works performed very well for unimodal and multimodal emotion recognition in the categorical model, but they require that the three modalities are available all time, which is not always the case in real conditions.

The authors of [27] use video modality (head movements) for predicting values over five dimensions (arousal, expectation, intensity, power and valence). They use optical flow on faces for detecting head movements, then use a Support Vector Regression (SVR) as a model for predicting values for the five dimensions. They reported 0.094 MSE (mean squared error) on the SEMAINE dataset. The investigation [28] claims that including events such as smiles, head shakes or laughter can improve the prediction of values over the affective dimensions. They utilize a string-based method for fusing the features, used SVR as predictor and reported 0.19 MLE (mean linear error). The work [29] uses audio, visual and text modalities on the Audio/Visual Emotion challenge (AVEC) 2017. They present good results, close to 0.09 RMSE (root mean squared error), for each dimension. However, data do not reflect real conditions in virtual learning environments, they do not deal with missing data, and models used for extracting features and predicting affective states are complex neural networks that have high requirements of computational resources and time for predicting, which is not the best option for scalability.

A similar work to our approach is proposed in [30] that also deals with missing data for multimodal emotion recognition, which is a different problem to other approaches for the problem of the missing labels. They exposed various ways for dealing with missing data at the fusion level, and used one for the experiments. However, even though they use the arousal-valence dimensional model (a continuous model), they predict quadrants and not continuous values in that space. In this way, their approach is considered as a categorical model for representing emotions, which the precision can be improved if a continuous model is used, as it is claimed in [13]. They reported 55% of accuracy on the CALLAS expressive corpus [31], predicting one of the four quadrants in the arousal-valence dimensional model (see Fig. 2). Particularly, the work presented in [30] compares various approaches for dealing with missing data, being the majority voting approach the best one. This is an approach only possible for categorical classifications and not for continuous models. For that reason, it is not possible a direct comparison between this work and ours (our target is an emotional continuous representation). A similar approach for continuous models is averaging the outputs, strategy, which is used for our “base model” (explained in the “our approach” section).

Table 2 presents a comparison between these works and our approach based on the following criteria:

Table 2

Comparison with related works.

	A	B	C	D	E
[19]	X
[20]	X		X		X
[22]			X
[24]	X		X
[25]	X		X
[27]		X	X
[28]	X	X	X
[29]	X	X	X
[30]	X		X	X
Our work	X	X	X	X	X

More than 1 modality.

The target is the continuous bi-dimensional space of arousal and valence.

Uses non-intrusive and affordable modalities.

Works with missing data.

Oriented to virtual education.

Several recent works meet criterion A. Multimodal approaches have been shown to have advantages over unimodal approaches. On the other hand, the recognition of discrete emotions is a traditional approach that is still very used, especially for works that focus on specific affective aspects and not on the whole emotional spectrum (see criterion B). However, nowadays works focusing on continuous models starting to be much more frequent. Criterion C is related to non-intrusive modalities. As [17] shows, several works use non-intrusive modalities. Criterion D refers to dealing with the availability of input data modalities at all times. Missing data is the most frequent case in virtual learning environments, the learner is not writing or speaking all time, making these two modalities available only at certain moments. Finally, criterion E is related to the context of application (virtual education). Few works for multimodal emotion recognition focusing in this field have been published, compared with other areas. In our work, this orientation is implicit in the dataset used, which is the most similar to the real data in a virtual learning environment.

A big difference between those works and our approach is considering the availability of modalities. In our work, we deal with the availability of modalities at every moment for reflecting the reality in virtual learning environments. During the search, it was not found a reason why not considering missing data for multimodal emotion recognition, it is not an exclusive problem of the virtual educational field. Also, in the literature, there are works for the problem of missing labels, as is the case of [32], but which is different from the problem solved in this paper about missing modalities in input data.

According to the literature review [18], our approach is the only one that includes the next characteristics: uses non-intrusive and affordable modalities (audio, video and text), deals with missing data (missing modalities) at all time in order to reflect the reality in the virtual education, and represents emotions using a continuous model for improving representation precision.

Our approach

The main objective of this work is the creation of a system for multimodal recognition of affective state, representing emotions in the continuous model of arousal and valence, using non-intrusive and affordable modalities, dealing with the availability of the modalities at every moment, reflecting the real conditions in a virtual learning environment. The proposed general procedure for affective state recognition from audio, face and text modalities consists of 4 major steps: feature extraction, feature selection, fusion and recognition, as is shown in Fig. 3. This general procedure is based on the common practices observed in the literature [24], [25], [27], [29], [33], [34], [35].

Figure 3

General procedure.

The general idea is to extract features from each modality, select only those features relevant to our objective, fuse the data from the three modalities (audio, text and video), and then recognize the affective state based on the multimodal data. The main contributions of this work are inside the purple steps in Fig. 3. Each step of the general procedure is defined below.

Feature extraction

In this subsection, the feature extraction process of the general procedure is explained. From each modality, relevant features are independently extracted. As the objective of our work is not to investigate the extraction of unimodal characteristics or an unimodal affective state recognition, but rather the fusion of multiple modalities, the unimodal approaches were adapted from related works that showed good results. In concrete, the approaches presented in [24], [25] are used as the base for our feature extraction process. The specific implementation details are given in the experiment section.

Audio

For the audio modality, the features from the Interspeech Compare 2013 feature set are selected, as proposed in [25]. This set consists of a total of 6373 features, including low-level descriptors and statistical metrics of them, like mean and standard deviation. This standardized feature set contains different features that include the “most common and at the same time promising feature covering prosodic, spectral and voice quality features” [23]. Some of them are the root mean square frame energy, the pitch frequency, and the harmonics-to-noise ratio. Also, it includes statistical descriptive metrics like the arithmetic mean, standard deviation, kurtosis, skewness, minimum and maximum values, among others.

Video

For the video modality, 68 facial landmarks for every frame of each video are extracted using the dlib frontal face detector and face predictor [36]. There are better face detectors that work with no frontal faces and more accurate landmarks predictors that work better with overlapping objects or with bad lighting conditions, but this model works fine and requires little time, a very precious feature for scalability in virtual learning environments. In addition, the distances of each landmark to each other normalized by the height of the face detected are used as features for each frame. As a result, 2278 features per frame were extracted. For summarizing, the features of the sequence of frames that represents each video are averaged resulting in a total of 2278 features for each video.

Text

For the text modality, the Senticnet [37] database was used as a knowledge base for extracting the affective features from text, specifically, the pleasantness, attention, sensitivity, aptitude, polarity, primary mood and secondary mood features were extracted for each word. Additionally, the AffectiveSpace resource [38] was used for extracting 100-dimensional vectors from each word. As a result, 105 continuous features and 2 categorical features were extracted from each word. Finally, the percentiles 0, 25, 50, 75 and 100 were used for continuous features and the sum of occurrences of each category for categorical features were used, resulting in 541 features per text: 5 percentiles for each continuous feature: 105*5 = 525, and 8 for each categorical feature because there are 8 classes: 2*8 = 16. Thus, 525+16 = 541.

Feature selection

The feature selection process is guided by three filters, the macro-algorithm of this process can be seen in Fig. 4. The first filter removes features that present low variance, these features are almost constants with low information (see lines 1-3). The second filter eliminates features that can be represented by others in the set. This degree of explanation is calculated using the variance inflation factor (VIF), which is defined as (see lines 4-13 in the algorithm): where is the R-squared metric of a model that predicts the variable j from the other variables in the set. The VIF filter prevents from multicollinearity of features. VIF value is calculated for each feature, and if the maximum VIF is higher than a threshold, then the feature with maximum VIF is eliminated and the VIF calculation process starts again. The third and last filter prunes features by their importance for predicting the target. Models as decision trees provide information on the importance of features. Best features can be selected by adjusting a threshold for this importance (this filter is explained in lines 14-19). A model that provides this information is fitted, in this case, random forest, then all variables below an established threshold are removed from the set.

Figure 4

Feature selection macro algorithm.

Fusion and recognition

The fusion and recognition steps depend on the fusion approach, and they can be carried out together or separated. As mentioned in the theoretical framework section, there are two strategies for data fusion: feature level fusion and decision level fusion. Decision level fusion requires unimodal models for making predictions before fusion, feature level fusion does not. This section proposes several fusion and recognition approaches, and is divided into two parts, the definition of some concepts required by our approaches and the presentation of our approaches.

Definition of preliminary concepts

In this section are defined what is a unimodal model and a base model, concepts that will be used by our approaches.

Unimodal models

The decision level fusion strategy needs the definition of unimodal models. They are required by the approaches that use a decision level fusion strategy to make the fusion of modalities. Each one of them receives as input the features extracted from its modality, and outputs values for arousal and valence dimensions. Since the target is two continuous values (one for arousal and one for valence), metrics selected for evaluation and comparison are and the root mean squared error (RMSE) scores. RMSE is defined as:

The score is defined as:

Where is the predictions, y the true values, is the mean of y, and n the number of samples.

is limited to one at the top, a higher value is better. RMSE is limited to 0 at the bottom, a lower value is better. Intuitive interpretations of the metrics are: is how good is the model compared to the model that always predicts the mean of the true values. RMSE is the mean error in the original units.

Additionally, the relative error of standard deviation was added as a metric for measuring how similar the dispersion of the predictions is to the dispersion of the labels. Like any other error, the lower the better. Relative error is defined as:

As our target is bidimensional, the euclidean norm is used and its formula is as follows:

Where SRE is the relative error of standard deviation, is the standard deviation of the arousal labels, is the standard deviation of the valence labels, is the standard deviation of the arousal predictions, is the standard deviation of the valence predictions.

Base model

In this paper, three approaches for multimodal recognition dealing with missing data are proposed, and a base model is implemented in order to test the results. The base model is based on a decision level fusion, so it requires a model for each modality before fusion. In addition, it consists of aggregating the results of the three unimodal models, averaging each dimension. As average can be performed over one, two or three values, it can deal with the variability number of available modalities at all time. Also, this average is weighted by the calculated performance of each unimodal model, in order to give more relevance to better performing models. The prediction of this model is defined as:

Where is the number of available modalities at moment i ( ranges from 1 to 3), is the predicted arousal for the moment i and modality j, is the predicted valence for moment i and modality j, and is the weight given to modality j according to its performance. These weights are calculated as:

Where error is a metric of the error of the model, and k iterates over the modalities available at moment i. In other words, the denominator is the sum of 1-errors of models from modalities available at moment i. These weights are pre-computed before prediction calculation and are static. The output of this model is a two-dimensional point that indicates the affective state in the arousal-valence dimensional model (see Fig. 2).

Our fusion and recognition approaches

First approach: feature-level fusion with zero padding

The first approach uses feature level fusion and consists of concatenating the features extracted from each modality, filling with zeros the missing data, and using a model for predicting arousal and valence values. This approach intends to replicate the idea of dropout, widely used in the neural networks field. Fig. 5 shows the process when video modality is not available.

Figure 5

First approach when video modality is not available.

Second approach: decision level fusion with zero padding

Since this approach is based on decision level fusion, it requires unimodal models for obtaining the predictions that are going to be fused. This approach is very similar to the previous one, but consists of filling with zeros (padding) the predictions that are not available, then concatenating these predictions, and using a model for generating a final prediction. Fig. 6 shows the process when video modality is not available.

Figure 6

Second approach when video modality is not available.

Third approach: decision level fusion with dynamic input

This approach consists of using models that allow inputs of varying length, in this way, there is no need to fill the input, but rather it can be passed regardless of its length. Specifically, recurrent neural networks are used, which through weight sharing allow sequences of any length to be input. These types of networks are unrolled according to the length of the input sequence, having a memory state that allows the next element in the sequence to know information about the past element(s), and thus generate a prediction with the information of the last element. This memory state stores information of all the previous elements. Fig. 7 shows the process when video modality is not available.

Figure 7

Third approach when video modality is not available.

Setup of the experiments

This section describes experiments performed and the implementation details of each step of our general procedure.

The approaches presented in [24], [25] are taken as bases, and their implementations are adapted for our specific purposes. These works were selected due to their good results for the recognition of the categorical emotions. These works propose Enterface and IEMOCAP as datasets, however, in our case different datasets were used for each modality (see artificial dataset subsection).

For this modality, the work presented in [25] is taken as a base. As proposed in this work, IEMOCAP was used as the dataset, Interspeech as the feature set, and OpenSmile as the software for extracting the features.

The work presented in [24] proposes to use 66 landmarks, we instead use 68 landmarks for ease of implementation and the difference is not significant. We also used the distances between every pair of these landmarks as features, resulting in 2278 features per face. Additionally, we normalized these distances by the height of the detected face. With this modification, it is sought that the characteristics are comparable between different faces regardless of the distance they are from the camera, since the closer they are, the larger the recognized face will be, and therefore, the greater the distances will be. The neutral face preprocessing that they propose was not used. The dataset used for this modality is hci-tagging [39]. For summarizing features from each frame of a video (a video is a sequence of frames), the average of the features is used, resulting in 2278 features per video. Other approaches for summarizing were analyzed: summation, quartiles (0,25,50,75,100), max and min. The average was the one with the best performance, but summation and quartiles have similar performances to average.

Authors of [24] propose using EmoSenticSpace [40] as a knowledge base, however, this resource is not available, and instead, we used AffectiveSpace [38] that also provides 100-dimensional vectors of terms describing the affective information. They propose using the polarity score provided by Senticnet [37] as an additional feature, instead we used polarity and the other six features that Senticnet provides: pleasantness, attention, sensitivity, aptitude, primary mood and secondary mood, as explained in the previous section. The negation approach that they proposed was not used. The dataset used for this modality is arousal-valence Facebook Posts [41].

The feature selection process consists of three sequential filters: low variance, high VIF and low importance. For the first filter, the traditional variance was used and the threshold selected is a minimum of . For VIF, linear regression models were used as predictors, and a maximum VIF of 10 was used to admit for any feature. For the feature importance filter, random forest models were built for extracting the importance, and the minimum importance accepted was . Only those features that meet these requirements were kept.

The feature selection process is performed for each of the two dimensions (arousal and valence), and each modality (3 modalities). The total features before the selection process are: for audio: 6373, for video: 2278, and for text: 541.

For audio modality, the first filter removed 621 features that presented variance lower than . The second filter is expensive in time, especially for audio modality due to the big quantity of features. For each one, it is necessary to build a model for calculating its VIF value, then at first VIF step 5752 models must be calculated, in the second step 5751, and so on. Even thinking this process is executed just one time, it is very time-consuming. For that reason, a heuristic was used for speeding up the procedure: eliminate 1000 features by step instead of 1 (as it is done is traditional VIF filtering) until a limit, then 500 by step, then 250 by step, and so on. At the end of the VIF filter, only 2752 features were considered as not highly correlated. The final filter was only passed by 70 features for arousal and 65 for valence.

The first filter did not eliminate variables for video modality, all features have a considerable variance. VIF filter showed that facial features are very highly correlated, therefore only 21 passed this filter, 2257 were removed for being highly correlated. As few features passed the VIF filter, feature importance is high due to the low number of variables, all the 21 features passed the third filter for both dimensions.

As for video modality, the variance filter for text modality eliminated no features evidencing acceptable variance for every descriptor. 366 features were selected after the VIF filter, indicating that 175 features are highly correlated with the others in the set, thus were removed. 223 were considered relevant for predicting arousal and 200 for valence.

The feature selection process is very expensive in time, especially because of the calculation of VIF values for a large set of features. Despite that, as it is executed just one time, it does not represent a big problem.

Unimodal models

Since this model is not part of our main contribution, no in-depth research or experimentation was done. Techniques compared for every modality are: Random forest (RF), support vector machines (SVM) and Partial Least squares regression (PLS). RF and PLS, both allow multiple output as needed in this work, on the other hand, SVM does not support multiple output. Hence, two models were trained, one for arousal and one for valence, then the predictions are concatenated and metrics calculated. Some bias were found on training data, it was eliminated by removing high-density points in data: truncating each point in data to maximum one or two samples, depending on the modality. Labels closer than 0.05 in the euclidean distance are considered as the same point, and only one or two samples are accepted per point for reducing high-density points in the dataset that included bias in the data.

RF performed very well in training data, but had problems for generalizing, predictions in the test set presented low variance and were biased to the center of the space. On the other hand, PLS performed badly compared to RF in the train set, however, it presented better generalization with the test set. PLS was chosen as the model for the three modalities.

Text modality showed very bad results for the arousal dimension, which impoverishes the results of the model. It gives clues that recognizing arousal from social media posts is a very difficult task, and textual data may not be very informative about this specific affective dimension.

Artificial dataset

There was a great difficulty finding datasets that meet four requirements: 1) contain the three modalities (Audio, Video and text), 2) annotated in the arousal-valence space, 3) labels cover the majority of the arousal-valence space, 4) data in the English language. These four requirements do not assure good quality of data, but they are the minimal characteristics required to test our approaches. No dataset with the four characteristics was found, some of the explored datasets were: IEMOCAP, Enterface [42], Mosei [43], SEMAINE [44], MOSI [45], MOUD [46], MELD [47], Humaine [48], [49], CK [50], CK+ [51], DEAP [52], AMIGOS [53], [54], AFEW-VA [13]. Thus, it was decided to look for unimodal datasets that meet these requirements and as far as possible that they reflect the real conditions of virtual education, with the objective of fusing them for creating an artificial dataset that have the required features.

For the audio modality, IEMOCAP [26] was selected because it is a conversational dataset similar to circumstances in virtual learning environments, where learners use their voice primarily to conversationally communicating with others. Also, the data acceptably cover the space, as is shown in Fig. 8.

Figure 8

Distribution of data from IEMOCAP, hci-tagging and arousal valence Facebook posts datasets, over the arousal-valence space (horizontal axis: valence, vertical axis: arousal).

For the video modality, mahnob hci-tagging dataset [39]1 was chosen because in the dataset various participants were asked to watch movie fragments, and after watching the fragments, annotate their affective states using the self-assessment manikin. This is close to reality in virtual learning environments, where learners interact with content and obtain an affective state. In addition, data covers very well the space, as can be observed in Fig. 8. The disadvantage of this dataset is the number of samples: only 513.

For the text modality, it was difficult to find a dataset that contains arousal and valence annotations. Many datasets contain only valence annotations because this problem has been much more attacked in NLP for product reviews, marketing, etc. To the best of our knowledge, the more similar dataset to the real settings in virtual learning environments that is annotated in terms of arousal and valence is the arousal valence Facebook posts [41]. An advantage of this dataset is that it covers decently the space, as is shown in Fig. 8.

Fig. 8 shows that a dataset that meets the four conditions previously established can be built from these three datasets. For dealing with multimodality, datasets were fused using the following idea: if two samples from different datasets are assigned the same point in the arousal-valence dimensional model, those two samples express the same affective state, therefore, they can be joined in one sample with the same assigned value in the space.

Thus, the fused samples of the resultant artificial dataset are not related to a single individual; however, since the samples of the three modalities express the same affective state, they can be fused.

In practice, the process is the following: for each point in one modality, if there exists one point in each of the other two modalities that are closer than 0.05 (using Euclidean distance) to this original point, they are considered as a sample for the fused dataset. In other words, if the three points from different modalities are closer than 0.05 from each other, the samples are fused and considered as a sample for the resulting artificial dataset.

If a sample is already fused with the other two from the other modalities, it is not considered anymore for including no repetition of samples in the dataset. Data that could not be merged into trimodal examples are merged into bimodal or left unimodal.

The final artificial dataset contains only 141 points with the three modalities and 3426 that are incomplete (only one or two modalities). This is translated in few trimodal samples for base models (that requires complete data), and an acceptable number of samples for strategies dealing with missing data.

For each approach, a “naive” strategy was implemented for comparing the same approach with and without dealing with missing data. The “naive” approaches consist of using the same approaches without dealing with missing data. Experiments were performed by approach. The metrics used for comparison are RMSE, score and std relative error. For every approach, RF, SVM and PLS regression models are compared, and for some, neural networks are implemented. Every model was tuned using a grid search approach.

First approach

Neural network models, RF, SVM and PLS models were experimented for this approach. The final neural network model used is a dense layer with a relu activation followed by a dropout layer and another dense layer at the end, with a tanh activation for prediction. Dropout and regularization strategies were used for avoiding overfitting. The optimizer used in this case is the Adam optimizer. The loss function used was the part after 1- in the formula. Contrary to what one might think, RF showed better results than the neural network model for this approach.

Second approach

For this approach, neural network models, RF, SVM and PLS models were explored. The final neural network model used is 3 LSTM layers with a relu activation each followed by a dropout layer. At the end, a dense layer with a tanh activation is used for prediction. Dropout and regularization strategies were used for avoiding overfitting. The optimizer used in this case is the Adam optimizer. The loss function used was mean squared error.

Third approach

As this approach is based on recursive neural network models, various architectures of this type of neural network were analyzed: simple recurrent, GRU and LSTM, obtaining the best results with the LSTM architecture. The final neural network model used is a LSTM layer with a relu activation followed by a dropout layer and a dense layer with a tanh activation for prediction. Dropout and regularization strategies were used for avoiding overfitting. The optimizer used in this case is the Adam optimizer. The loss function used was the part after 1- in the formula. Recurrent neural networks can receive varying length sequences as input, however, all sequences inside a batch must have the same length. For this reason, training and testing batches were formed appending randomly selected samples with equal length. All batches have a fixed size. 100 trimodal, 100 bimodal and 100 unimodal batches were formed for training and testing, random sampling is performed without replacement.

Models for predicting errors

The results of the approaches were not good enough. It was decided to add at the end of every approach a model for predicting errors of the fusion model, after that, predicted errors with predictions were summed for getting the final predictions, as proposed in [55]. Models considered for this method were SVM, RF and PLS. This method improved the performance of almost every approach, except for the second approach where no error model could improve the results of the fusion model, then, the fusion model is taken as the final model.

Results

Different results were obtained for the different unimodal models, and for the different multimodality fusion approaches dealing with missing data. Best results with unimodal models are for audio modality: =0.3, RMSE=0.36, SRE=0.3; video: =0.07, RMSE=0.54, SRE=0.39; and text: =0.43, RMSE=0.51, SRE=0.25.

Audio modality performed better than the other two modalities, the reason for this can be found in the data or in the characteristics chosen to represent it. PLS regression was the best model for every one of the three modalities. SVM showed poor results in comparison with the other two techniques. RF showed comparable results to those of PLS in the test set, but they were biased to the center of the space, and low variance was present as evidenced by the SRE metric. In addition, huge overfitting was observed using this model. RF training performance was excellent, but in the testing set the results were very different, in contrast, PLS performed similarly in the training and the test sets.

“Naive” approaches (without dealing with missing data) gave different results to those of the proposed approaches. In fact, proposed approaches gave better results in general. Intuitively, “naive” approaches should perform better than proposed approaches because they do not have to deal with missing data. This lack of performance from “naive” approaches can be explained by the low quantity of complete data (only 141 samples) compared to the quantity of complete and incomplete data (3567 samples).

Table 3 resumes the results of “naive” and proposed approaches, and compares them with other works. Although the results of this work are not directly comparable with other works because we deal with missing data, the results from other works are exposed for having a reference.

Table 3

Summary of comparison of continuous emotion approaches (B: base approach; F: First approach, feature level fusion with zero padding; S: Second approach, decision level fusion with zero padding; T: Third approach: decision level fusion with dynamic input, wavg: weighted average) for each modality (A: audio, V: video, T: Text) with different metrics (N/A: not apply, MLE: mean linear error).

	Modals.	Data	model	R2	RMSE	SRE	MLE
B “naive”	A,V,T	Artificial	wavg,PLS	0.370	0.334	0.404	N/A
F “naive”	A,V,T	Artificial	PLS,PLS	0.300	0.355	0.192	N/A
S “naive”	A,V,T	Artificial	PLS,PLS	0.400	0.326	0.401	N/A
T “naive”	A,V,T	Artificial	NN,PLS	0.346	0.340	0.400	N/A
B	A,V,T	Artificial	wavg,RF	0.394	0.357	0.338	N/A
F	A,V,T	Artificial	RF,RF	0.443	0.345	0.300	N/A
S	A,V,T	Artificial	NN	0.387	0.371	0.259	N/A
T	A,V,T	Artificial	NN,NN	0.281	0.409	0.474	N/A
[27]	V	Semaine	SVM	N/A	0.310	N/A	N/A
[28]	AV	Semaine	SVM	N/A	N/A	N/A	0.190
[29]	A,V,T	avec 2017	NN	N/A	<0.1	N/A	N/A

As mentioned before, a comparison between our approaches and other works can not be directly performed because significant differences exist. Our approaches deal with missing data, also different datasets and different metrics are used. Works [27], [28] use Semaine as the dataset, we did not use this dataset because it does not contain text modality, thus, a speech-to-text process is required. Authors of [29] use AVEC 2017 dataset, this dataset is a subset of SEWA and contains the three modalities, where the text modality corresponds to the transcriptions of speech. This dataset was not used because it is in German, and is a bit different than the real conditions that happen in virtual learning environments. They obtain excellent results reaching an RMSE lower than 0.1, approximately. They use complex neural network models for extracting features from modalities and recognizing affective state, but those models are very time expensive and cannot be implementable in virtual education platforms due to scalability issues.

RF and PLS regression obtained the best results for almost every approach, and in some cases gave better results even than neural networks models. RF showed better results in terms of and RMSE for almost every approach, however, when predictions were plotted they were very central. Points far from the center of the space were never predicted, which was the reason for including SRE as a metric, for measuring the dispersion of predictions avoiding selecting models with high and RMSE but that are biased to the center of the space.

The base approach was included in the system to reference the results. However as mentioned before, they did not perform as expected. The performance of these base models is relatively close to the results of proposed approaches, noting differences that are relevant but not by much.

The best approach according to metrics is the first approach: feature-level fusion with zero padding using two RF models, one for predictions and one for predicting errors. Generally, RF showed centered biased results, and for this case, SRE is not so high showing a relatively good variance of predictions discarding center bias. In addition, is the highest observed.

These results showed that unimodal models do not contribute much to the system, as observed in the results of decision level fusion approaches, and that the direct fusion of the extracted characteristics is more successful than having an intermediate model. In fact, the unimodal models did not achieve encouraging results except for the audio modality.

Conclusions

In this work, an unobtrusive multimodal system for recognizing affective states in terms of arousal and valence that deals with missing data in similar conditions to the real settings in virtual learning environments is proposed.

The biggest difficulty of this investigation was on obtaining good quality data that allow the evaluation of the three modalities (audio, video and text), which is annotated in terms of arousal and valence, that covers all the arousal-valence space, in the English language, and that reflects the real conditions in virtual learning environments. Many datasets were analyzed, as none of the datasets met the requirements, there was a need to create an artificial dataset. The resulting dataset acceptably covers the space, contains data from the three modalities, which can be used to simulate missing data and reflects real conditions in virtual learning environments. However, the dataset is not of the best quality, some biases to the center of the arousal-valence space were found working with the data, and the set contains a low quantity of complete samples (trimodal).

Despite the artificial dataset is not of the best quality, it could be used for comparing the approaches with and without dealing with missing data-oriented to virtual education. Approaches for dealing with missing data could help improving results when little amount of complete data is available, as is the case. In our artificial dataset, only 141 complete samples were available, on the other hand, 3567 complete and incomplete samples were accessible. Approaches dealing with missing data showed better results on this dataset.

It is concluded that the affective states of students in virtual education can be extracted using unobtrusive sources of information, while the learners are studying and interacting in the platform. This affective information can improve virtual learning platforms, letting them know more information about the context of students for using better teaching strategies, or for giving better and more personalized content recommendations. In addition, this information can be used for enhancing the selection of learning resources for a virtual course.

As this work was oriented to virtual learning environments, the time component is very important for obtaining scalability since the number of users can go up to various thousands. Recently, due to the contingency caused by Covid-19, the majority of students moved to virtual education. Thus, the scalability requirement constrained us to use efficient lightweight models that give good results such as RF, SVM, PLS and light neural networks.

For future works, other approaches for multimodal fusion that deal with missing data could be explored, which consider the domain, the time and scalability needs. In addition, datasets from virtual learning environments should be obtained. Finally, other features that consider other information types for predicting the affective states could be analyzed. There was found a low quantity of works similar to our in the literature, research on multimodal emotion recognition dealing with missing data appears to be very few. This work encourages researchers and opens doors to exploit the advantages of multimodal emotion recognition with missing data in virtual education.

Declarations

Author contribution statement

C. Salazar, E. Montoya-Múnera, J. Aguilar: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.