The relation of expression recognition and affective experience in facial expression processing: an event-related potential study.

The present study investigates the relationship of expression recognition and affective experience during facial expression processing by event-related potentials (ERP). Facial expressions used in the present study can be divided into three categories: positive (happy), neutral (neutral), and negative (angry). Participants were asked to finish two kinds of facial recognition tasks: one was easy, and the other was difficult. In the easy task, significant main effects were found for different valence conditions, meaning that emotions were evoked effectively when participants recognized the expressions in facial expression processing. However, no difference was found in the difficult task, meaning that even if participants had identified the expressions correctly, no relevant emotion was evoked during the process. The findings suggest that emotional experience was not simultaneous with expression identification in facial expression processing, and the affective experience process could be suppressed in challenging cognitive tasks. The results indicate that we should pay attention to the level of cognitive load when using facial expressions as emotion-eliciting materials in emotion studies; otherwise, the emotion may not be evoked effectively.

Introduction

When individuals see a face, two main types of information can be inferred. The face is identified as a specific stimulus belonging to a unique individual, taking into account change in appearance or aging. Second, facial expression is interpreted from its emotional content, which sets modality for social interactions.1 Facial expression plays an important role in expressing human emotion and in understanding the emotion of others. The ability of facial expression recognition requires the ‘seer’ to analyze and detect special expression states from the expression images or video frames, and then to ascertain the subject’s specific emotional state.

During the process of expressional recognition, people are easily affected by a facial expression. Which then evoke relevant emotions. As a result, emotional expressions have been widely used in all kinds of emotion studies as useful emotion-eliciting materials. The use of facial expression for measuring people’s emotions has dominated psychology since the late 1960s when Ekman reawakened the study of emotion by linking expressions to a group of basic emotions.2 Ekman and Friesen3 developed the affective facial picture system for emotional studies, which made it convenient for researchers to choose stimuli materials for their studies and make the study results comparable.

The reason that facial expression can be used as an emotion-eliciting material is that people easily experience the same emotion when they recognize an expression. However, a problem confronted us when emotional facial pictures were used as emotion-eliciting materials in previous studies.4 When participants were asked to finish an expression identification task, such as comparing the emotional states of two faces, the emotion-evoking effect of emotional faces usually could not be done. Though many methods have been tried (ie, prolong stimulation time, show more trials, or use more participants), no obvious solution to this problem has been found. Why? Emier and Kiss5 found that cognitive load (a term that refers to the load on working memory during instruction) attenuates event-related potentials (ERPs) to facial expressions. So, we hypothesized that the ERP deflection maybe suppressed by the cognitive load. A reasonable interpretation is that the processes of emotion experience and expression recognition are independent. It means that even if people recognize the expression correctly, the relevant affective experience may not be produced in challenging cognitive tasks.

The field of cognitive neuroscience provides methods that allow us to investigate the neural mechanism of face recognition.6 Functional MRI (fMRI) and high-density ERPs were most used in present studies. Functional MRI based on measurements of the BOLD signal measures the hemodynamic response to neural activity and, consequently, has relatively poor temporal resolution but good spatial resolution. EEG and ERP (as well as magnetoencephalography [MEG]) measure the integrated, synchronous electrical activities of neurons and have excellent temporal resolution but poor spatial resolution. Although ERPs are limited in terms of spatial resolution compared with functional neuroimaging, they provide an excellent and more precise metric of the time course of neural activity.7 Relevant evidence supporting the functional specificity of brain mechanisms responsible for emotional face processing is offered by psychophysiological studies using ERPs.8,9 Single-cell, neuroimaging, and lesion studies have shown that the orbitofrontal cortex and amygdala play prominent roles in processing facial emotional expressions,10 and other prefrontal areas–the right anterior cingulated, right inferior parietal cortex, ventromedial occipitotemporal cortex, inferotemporal cortex, and the hippocampus–are also involved in the analysis of faces and facial expressions.11–13 These studies indicate that the way we process facial expressions is different from how we perceive other objects.

Studies using emotionally implicit tasks have shown that the anterior P200 component, which is considered an index of the attention-related process, was larger for negative stimuli than for positive stimuli.14 N2 (about 250–350 ms from onset) is another sensitive ERP component of negative facial expressions.15–17 Negative stimuli usually evoked larger amplitude than did neutral or positive ones. Another important ERP component is P3 (or late positive complex, LPC), which signals the cognitive evaluation of the meaning of stimuli.18,19 These ERPs were evoked in emotional-eliciting tasks and were widely used in studies.

Traditional studies tend to believe that an emotional experience will be evoked as soon as we see an emotional expression, which is why we use expressions as emotion- eliciting materials. The dissociations between facial identity and facial expression processing, as well as between facial expressions and structural features of facial stimuli, have been well-documented by the cognitive model of face recognition proposed by Bruce and Young.20 The model outlines seven distinct types of information that can be derived from the face: pictorial, structural, semantic, identity, name, expression, and facial speech. Research on facial structure suggests that the structural and semantic features of the face are processed independently.9 Based on the analysis above, we hypothesized that the processes of expression recognition and affective experience were uncoupled in facial expression processing, meaning that even though people recognized a facial expression correctly, they may not have experienced the emotion behind the facial expression. The current study was initiated to test this hypothesis with neurologically healthy individuals. We measured ERPs while participants performed facial-processing tasks. We can speculate about the emotion-eliciting effect by analyzing the neural response to different emotional faces in different tasks (easy and difficult).

Methods

Participants

As paid participants, 14 college students (8 women, 6 men) aged 22.8 to 27.4 years (mean age, 24.2) participated in this study. All subjects were healthy, right-handed, with normal or corrected-to-normal vision, and none reported a history of affective disorder. Each subject signed an informed consent form for the experiment. This study was performed in compliance with the author’s institution’s policies related to the use of animal and/or human subjects and human-derived material in China (2009.12). The experimental procedure was in accordance with the ethical principles of the 1964 Declaration of Helsinki.

Materials

All emotional facial pictures were taken from the College Students’ Facial Expression of Emotion (CSFE) system in this study. This system was developed in a key laboratory of mental health, the Chinese Academy of Sciences, to avoid the cultural bias of emotional inducement found in Chinese participants when International Affective Picture System (IAPS)21,22 was used. In the college students’ facial expression of emotion scale, all facial pictures were divided into seven groups: disgust, surprise, neutral, happy, sad, angry, and fear. More details about this scale are accessible in Wang and Luo.23 Three categories of facial pictures were used: positive (happy), neutral (neutral), and negative (angry). Each category included 20 pictures (10 male faces, 10 female faces). Before the formal study, 30 college students were asked to make a decision about these pictures (happy, neutral, angry), and report their arousal level from 1 (extremely negative) to 9 (extremely positive). The correct answer rates about these pictures (happy, neutral, angry) were 100%. The arousal level was about the same.

All stimuli were presented in pairs and were placed in six groups (same emotion: both happy, both neutral, both angry; different emotion [not involved in analysis, only for control]: neutral vs positive, neutral vs negative, positive vs negative). The stimuli were shown in pictures sized 240 × 140 pixels (when running E-Prime software, the whole screen is in 640 × 480 pixels). All pictures were in grayscale; the background was black (Figure 1).

Figure 1

Stimuli materials for example in our experiment. Each stimuli consists two facial pictures. These two facial pictures are of the same gender. The emotional valence are varied in six types: same (neutral), same (happy), same (angry), and different (neutral, angry), different (neutral, happy), different (angry, happy).

Tasks

The present study included two kinds of tasks. The first task consisted of six blocks of 50 trials. In this task, participants had to decide whether faces presented in pairs were the same or different in their emotional properties (positive, negative, neutral), and press corresponding keys when the pictures disappeared (same, positive, 1; same, negative, 2; same, neutral, 3; different, none). The stimuli were presented 1000 ms (1000 ms is long enough for them to make a decision).

The second task contained six blocks of 50 trials. In this task, participants had to decide whether faces presented in pairs were the same or different in their emotional properties (positive, negative, neutral), and press corresponding keys as soon as possible when the pictures appeared (same, positive, 1; same, negative, 2; same, neutral, 3; different, none).

To motivate them to respond properly, participants were told that the best six (task 1 and task 2) respondents would be rewarded with a book that they could select themselves. Each subject participated in both tasks, with the order of these two tasks counterbalanced between subjects. In order to set the difficulties of different tasks, we used two tasks. The procedures of these tasks were about the same; the only difference was the time to respond. It is hard to create different task difficulties in just one task, therefore two tasks were used in the present study.

Stimuli and apparatus

Subjects were seated in a quiet room approximately 100 cm from a computer screen (Dell 15-inch CRT monitor, 80-Hz refresh rate) with the horizontal and vertical visual angles below 5°. Before the study, all subjects were told that they should keep their eyes fixed in the middle of the screen during the entire process. Each subject participated in both tasks, with the order of the two tasks counterbalanced between subjects.

In task 1, each trial was initiated by a 250-ms presentation of a small white cross (+) in the center of a black screen, followed by a stimulus picture with a fixed interval of 1000 ms. After that, a black screen that lasted no longer than 2000 ms appeared and participants were to press corresponding keys during this period. The black screen was terminated by pressing a specific key or was terminated when it had lapsed for 2000 ms. In task 2, each trial was initiated by a 250-ms presentation of a small white cross (+) in the center of a black screen, followed by a stimulus picture that last no longer than 1000 ms. Participants were asked to press corresponding keys as soon as possible during this period. After that, a black screen with a fixed interval of 2000 ms appeared.

ERP recording

High-density ERPs were recorded from each participant using a 128-channel geodesic sensor net (Electrical Geodesics Inc, Eugene, OR, USA), coupled to a high-input impedance amplifier. An electroencephalogram continuously recorded and sampled at 250 Hz. Wherever possible, impedances were reduced to less than 50 KΩ prior to recording. Vertical electroculograms were recorded at the left orbital rim; horizontal electroculograms were recorded at the right orbital rim.

ERP averaging

The data were analyzed offline with the software NetStation (Electrical Geodesics Inc). Trials with incorrect responses and trials with electroculogram artifacts (electroculogram voltage exceeding 50 μV) were excluded from the average. The data were filtered with a lowpass of 30 Hz. Electroencephalogram activity for correct responses in each valence condition was overlapped and averaged separately. ERP waveforms were time-locked to the onset of stimuli and the average epoch was 1200 ms, including a 200-ms prestimulus baseline. As shown by the ERPs’ grand-averaged waveforms, those elicited by three conditions (positive, neutral, negative) showed prominent differences from each other in task 1. These differences were largest at the central and frontal sites (Figure 2, left side). In contrast, the grand-averaged ERPs of task 2 displayed no obvious differences during three valence conditions (Figure 2, right side). Studies on facial recognition have revealed that the face typically elicits a larger negative deflection approximately 170 ms after stimulus onset, 24–26 known as N170, and reflects perceptual processing of structural information from faces in specialized occipital-temporal brain areas27,28 and studies on emotion29,30 found that the prefrontal cortex was responsible for these process. Therefore, we selected T5, T6 for the analysis of N170, and we selected the following 10 sites for statistical analysis of other ERP components: Fp1, Fp2, Fz, F3, F4, F7, F8 (seven frontal sites) and Cz, C3, and C4 (three central sites). All sites were selected according to the international 10–20 system. For both tasks, the mean amplitude (mean value in a selected period) and peak latency (from stimulus onset to the peak of each component) in the components (N170 [140–190 ms] in occipital-temporal brain areas; P2 [175–225 ms], N2 [350–450 ms] in frontal sites; P3 [350–450 ms] in central sites) were measured and analyzed. The mean amplitudes and peak latencies were determined by the NetStation software (Electrical Geodesics Inc). Repeated ANOVAs were conducted for the amplitude and latency of each component. Analysis of variance factors were valence conditions (positive, neutral, negative) and electrode sites. Bonferroni correction was applied for multiple post-hoc comparisons.

Figure 2

Averaged ERPs at Fz, Cz, and Pz. The waveforms in the three conditions (happy, neutral, angry) showed significant difference in task 1 (easy task), however, no differences were found among them in task 2 (difficult task).

Results

Behavioral performance

The mean reaction times (from when the stimuli disappear to when participants respond) in task 1 for positive, neutral, and negative were 219.3 ms, 218.2 ms, and 229.8 ms, respectively. A one-way analysis of variance showed there was no main effect of valence [F(2,26) = 1.073, P > 0.05]. Reaction times (from when the stimuli appear to when participants respond) in task 2 for positive, neutral, and negative were 374.3 ms, 359.1 ms, and 347.9 ms, respectively. Similar to task 1, there was no main effect of valence [F(2,26) = 0.877, P > 0.05]. False responses were rare in task 1, because nearly all subjects achieved 100% accuracy in the three kinds of emotional stimuli. False responses in task 2 were 6.33%, 5.94%, and 6.86%, respectively. No significant effect was found among these valences [F(2,26) = 0.683, P > 0.05]. Because of the different characteristics of our research tasks, the comparison between these two tasks was meaningless. Behavioral results are presented here only for reference.

ERP analysis: task 1

N170 component was elicited by all three valence conditions in task 1, but no significant main effect was found in N170 for mean amplitude [F(2,26) = 0.077, P > 0.05] and peak latency [F(2,26) = 0.912, P > 0.05]. P2 is another early component elicited by all three valence conditions; however, no significant main effect was found in P2 for mean amplitude [F(2,26) = 0.216, P > 0.05] and peak latency [F(2,26) = 0.132, P > 0.05].

Amplitude effect for valence was observed in N2 [F(2,26) = 4.794, P < 0.05]. The largest N2 amplitude was recorded in anterior electrode sites (Fp1, Fp2, Fz, F3, F4, F7, F8). A subsequent pairwise comparison for valence and amplitude showed that the positive condition elicited smaller N2 amplitude than did the negative [F(1,13) = 6.981, P < 0.05] and neutral [F(1,13) = 7.828, P < 0.05] ones; there was no difference between positive and neutral conditions in mean amplitude [F(1,13) = 1.440, P > 0.05] and peak latency [F(2,39) = 2.640, P > 0.05].

Another important ERP component is P3, which was observed principally in the central electrode sites (Cz, C3, C4). A significant main effect of valence was found for the P3 amplitude [F(1,13) = 7.563, P < 0.05]. A subsequent pair-wise comparison for valence and amplitude showed that the negative condition elicited higher P3 mean amplitude than did the positive [F(1,13) = 6.222, P < 0.05] and neutral [F(1,13) = 4.630, P < 0.05] ones; no significant main effect was found for P3 peak latency [F(2,26) = 1.022, P > 0.05] either.

ERP analysis: task 2

In task 2, early components (N170 and P2) in different valence conditions were similar to the features in task 1. No significant main effect was found in N170 for mean amplitude [F(2,26) = 0.085, P > 0.05] and peak latency [F(2,26) = 0.280, P > 0.05]. No significant main effect was found in P2 for mean amplitude [F(2,26) = 0.09, P > 0.05] and peak latency [F(2,26) = 0.160, P > 0.05].

In task 2, no significant main effect was found among the three valence conditions in N2 in mean amplitude [F(2,26) = 0.301, P > 0.05] and peak latency [F(2,26) = 0.236, P > 0.05]. There was no main effect of valence for P3 in mean amplitude [F(2,26) = 0.125, P > 0.05] and peak latency [F(2,26) = 0.089, P > 0.05].

Anterior hemispheric asymmetries test

The valence hypothesis favors the concept of left hemisphere processing for positive emotions and right hemisphere for negative emotions, regardless of processing mode (ie, regardless of perception or expression of emotion and regardless of input channel). There are several proponents of this theory and numerous supporting studies.31 In our study, we selected 14 anterior sites for comparison: left hemisphere (F3, AF3, F1, F5, FC1, FC3, FC5) and right hemisphere (F4, AF4, F2, F6, FC2, FC4, FC6) (Figure 3, black).

Figure 3

Electrode sites selected for comparison (black). We selected the AF3, F1, F3, F5, FC1, FC3, FC5 electrode sites in left hemisphere, and the AF4, F2, F4, F6, FC2, FC4, FC6 electrode sites in the right hemishphere.

A comparison between left and right hemispheric electrodes was performed in N2 mean amplitude. In task 1, the negative condition elicited a higher N2 mean amplitude than did the positive [F(1,13) = 7.162, P < 0.05] condition in the right hemisphere; the positive condition elicited a higher N2 mean amplitude than did the positive [F(1,13) = 6.461, P < 0.05] condition in the left hemisphere. This means that the emotion was evoked well in task 1. In task 2, no main effect was found in N2 between the hemispheres in positive [F(1,13) = 1.163, P > 0.05] and negative [F(1,13) = 0.779, P > 0.05] emotions. This means that emotion was not evoked effectively.

Discussion

ERP studies of adult face processing have revealed that the face typically elicits a larger negative deflection approximately 170 ms after stimulus onset,24 known as N170, and reflects perceptual processing of structural information from faces in specialized occipital-temporal brain areas.27,28 However, Eimer and Holmes8 showed that the face-specific N170 component was unaffected by facial expression. In our study, no significant effect was found among the three valence conditions for N170 in both tasks. The results were consistent with the findings of Eimer and Holmes.8 Frontal P2 activation within 200 ms is indicative of rapid detection of typical stimulus features.32 The results (task 1 and task 2) showed that early visual processing (reflected by P2) was similarly affected by all three valence conditions, probably because the stimuli pictures of faces were equal in size and resolution. Early ERP components within 200 ms (ie, N1 and P2 in this study) were sensitive to the physical features of stimuli, such as size and color. The results in N1 and P2 indicated that the stimulating materials used in the study were strongly consistent.

Compared with positive and neutral pictures, negative pictures often included threatening content, which can engage attention rapidly and automatically.33,34 Accordingly, in the present study, we used the detection of negative (angry) faces. Consistent with this interpretation, higher N2 amplitude for negative conditions in frontal sites in task 1 was observed in the study. This suggests that negative stimuli could be interpreted as more important in evolution and could recruit more physiological and psychological resources compared with other stimuli,17 which contributed to the higher amplitude of N2. All the results showed that the ERPs elicited by emotional faces were active during the 300–500 ms period, which was consistent with the emotional studies using emotional pictures.35,36 However, no significant effect was found for the negative condition in all ERP components (N2, P3) in task 2. ERPs elicited by these valence conditions showed great similarity, which was quite different from the features in task 1. The results in task 2 suggest that the emotional aspect of emotional faces did not affect the psychological process.

The enhanced ERPs in response to emotional faces were typically interpreted as reflecting differential processing of emotionally meaningful and neutral stimuli in the cortical visual systems.37 These results indicated that the emotional aspect of facial pictures showed a significant effect on the process of facial recognition in task 1; that is, participants had relevant emotions evoked when they identified the emotional expressions in facial processing. Contrary to the finding of task 1, there was no difference in amplitudes or latencies in N2 for different valence conditions in task 2. The results indicated that, even if participants had identified the expressions correctly, no relevant emotions were evoked during this process. In other words, emotion was not experienced simultaneously when people recognized the facial expression in task 2.

In the present study, both tasks were facial expression recognition tasks: the main difference between them was the different degree of cognitive endeavor used during the process. In task 1, participants were to identify the facial expression shown and press the corresponding keys, which was an easy task because the expressions could be identified easily. However, task 2 was a challenging task because participants were to identify the facial expressions separately, compare them, and then press relevant keys as soon as possible. During this process, participants put more effort into expression recognition. It was found that the emotion experience and expression identification were not synchronous in this study. The influential factor was the cognitive involvement during the process. If more endeavors were put into expressional recognition, the emotion process could be suppressed. Overall, the results supported our hypothesis of the relationship between affective experience and expressional recognition in the facial expressional process.

These findings suggest that the emotional experience was not synchronous with identifying expressions. The affective process could be suppressed by challenging cognitive tasks. This results not only confirmed Eimer and Kiss’s5 findings that cognitive load attenuates ERPs to facial expressions, but also further extended their results. The results indicate that we should pay attention to the difficulty of cognitive tasks when using facial expression as emotion-eliciting materials in emotion studies; otherwise, the emotion may not be evoked effectually.

Limitations and future directions

Although a number of studies have shown that affective effects are not lateralized even with long stimulus presentations and in the absence of rapid response demands. However, lateralization analysis is still a useful and valid method in measuring experience of emotions. Future researches should try to find more precise methods to measure different emotion effects.

Appendix 1

Identification numbers of CSFE pictures presented in this study

	Female	Male
Happy	f170, f175, f189, H36f, H48f, H62f, H64f, H65f, H66f, H79f	H2m, H4m, H14m, H16m, H68m, H95m, m124, m138, m139, m141
Neutral	A30f, A78f, A79f, D78f, S43f, S54f, S88f, S90f, S92f, S93f	m162, m165, m166, m168, m169, m179, m181, s2m, s4m, s7m
Angry	f1, f2, f3, f4, f5, f12, f34, f37, f38, f39	A5m, A22m, m3, m4, m11, m12, m16, m28, m29, m38