Recognition of Contextually Threat-Related Scenes is Enhanced by Preceding Emotionally Incongruent Facial Expression.

Previous studies on intentional and incidental face memory have investigated the effects of emotional facial expression on facial recognition itself. However, it is still uncertain whether facial expression influences later recognition of other emotional stimuli, such as emotional scenes. To address this issue, participants during the encoding phase were presented with emotional scenes together with facial expressions. The emotion of the scenes was either congruent or incongruent with that of the facial expression. In order to increase the attention towards faces/facial expressions, the task was related to faces rather than scenes, for example, the participants were asked either to memorize the facial identities (intentional face learning, Experiment 1) or to identify the gender of the faces (incidental face learning, Experiment 2). Subsequently, the participants were asked to perform an unexpected old/new recognition task regarding the scene pictures. In general, the results showed that recognition of threat-related scenes was enhanced by a preceding emotionally incongruent facial expression irrespective of intentional or incidental face learning. The findings indicate that facial expression influences recognition of contextual threat-related scenes. Copyright:

Introduction

Facial expression conveys a wide variety of information about others’ emotional and motivational states and, therefore, is critical in everyday social interactions. Facial expression influences how well the face is encoded, which influences subsequent steps of memory, such as recognition (Balconi & Mazza, 2009; Calvo & Beltrán, 2013, 2014; Hagemann et al., 2016; Holmes et al., 2008; Müller-Bardorff et al., 2016; Johansson et al., 2004); Satterthwaite et al., 2009; Wagner et al., 2007). For example, it has been shown that threat-related expressions (e.g., fearful and angry expressions) increase face recognition when individuals are intentionally asked to learn the faces during the encoding phase (i.e., intentional face learning; Adolphs, 2002; Bartholow et al., 2009; Lander & Metcalfe, 2007; Langeslag et al., 2009; Leppänen & Hietanen, 2004; Nomi et al., 2013; Pinabiaux et al., 2013; Sessa et al., 2011; Wang, 2013). When participants were asked to perform face memory-unrelated tasks (e.g., identify the gender of a face) during the encoding phase (i.e., incidental face learning), but later to have a sudden recognition regarding faces, better recognition for threat-related compared to neutral faces was reported, similar to the findings regarding intentional learning (Grady et al., 2007; Keightley et al., 2011; Lin et al., 2015a; Wagner et al., 2007). However, some studies found enhanced recognition of happy or even neutral faces (Hagemann et al., 2016; Shimamura et al., 2006). These discrepancies may be due to whether recognized faces are presented with the encoded expressions or changed into a new expression during the recognition phase (Lin et al., 2015b; Hagemann et al., 2016) and/or whether emotional expressions are shown in the encoding or the recognition phase (Chen et al., 2005). Nevertheless, increased recognition of threat-related faces might be associated with evolutionary distinctiveness (Pratto & John, 1991), as distinctiveness theory has suggested that distinctiveness enhances stimulus memory (Rawson & Overschelde, 2008; Schmidt, 1991).

In daily life, however, there are many circumstances in which facial expression is not viewed in isolation, but as a part of surrounding environments. Facial expression might not only influence face processing, but also influence how we perceive the surrounding environments. Suppose you are staying in a dark forest with your friend. When he/she shows a fearful expression, you might think of something dangerous in your surroundings. The perception of emotional facial expression might influence the perception of contextual environments and thus, the later retrieval of these environments. An investigation of this issue might yield a better understanding of the role of evolutionarily distinctive stimuli, as elicited by emotional (e.g., threat-related) facial expression, on memory and thus, give more insight into the distinctiveness theory.

A recent study by Bowen and Kensinger (2017) investigated whether facial expression influenced recognition of surrounding words. In this study, an emotional (e.g., negative, positive, and neutral) facial picture or a scene picture was presented together with a neutral word. The participants were asked to indicate whether the prompted stimulus was a facial picture or a scene picture and meanwhile, to memorize the words. Subsequently, the participants were asked to recognize the words that were presented without any facial or scene pictures. However, the results did not show the effect of preceding facial expression on word recognition.

In the study by Bowen and Kensinger (2017), the participants were asked to identify the stimulus category (e.g., a scene or a face) during word memorization. Considering that identification of stimulus category was much easier than word memorization, the participants may have paid more attention to memorizing the words than to identifying the faces and/or facial expressions, which generally reduced the effect of facial expression. More importantly, the study only investigated the effect of facial expression on the recognition of neutral, but not of emotional stimuli. Wieser and Keil (2014) showed that steady-state visually evoked potential (ssVEPs) responses were larger during incidental encoding of threat-related compared to positive and neutral contextual scenes when the contextual scenes were presented with threat-related faces, and these effects did not occur when the scenes were presented with some other facial expressions (e.g., positive and neutral expressions). The findings may indicate that the effect of facial expression on the encoding of contextual environments is affected by the emotion of the environments.

Moreover, previous studies have shown that the emotions between contextual environments and target faces are integrated during the late time ranges in the encoding phase. Thus, the encodings of both contextual environments and target faces are suggested to be related to emotional (in)congruence between faces and environments (Gu et al., 2013; Ho et al., 2015; Kato & Takeda, 2017; Lin & Liang, 2019a; Xu et al. 2017). The encoding, which is the first step of memory, might influence later steps, such as retrieval. Thus, the retrieval of contextual environments might also be influenced by emotional (in)congruence between target faces and contextual environments. More importantly, distinctiveness theory proposes that distinctiveness enhances stimulus retrieval, and expectancy incongruence is a phenomenon of distinctiveness (Schmidt, 1991). For instance, Porubanova et al. (2014) showed that everyday concepts that were incongruent with common sense/expectations (e.g., stone mirrors) were more memorable than concepts that were congruent with the expectations (convertible cars). Regarding emotion-related expectancy incongruence, our recent study (Lin & Liang, 2019b) showed that recognition of neutral pictures was enhanced, when they had been expected in negative (i.e., the expectancy-incongruent) rather than neutral (i.e., the expectancy-congruent) condition. The findings might suggest that the distinctiveness, as elicited by expectancy incongruence, influences later recognition of target stimuli. Emotional congruence and incongruence between target faces and contextual environments might be considered as expectancy congruence and incongruence, respectively, as facial expressions that are congruent with contextual environments in emotional contents are in accordance with common sense. However, no empirical studies have directly investigated the effect of distinctiveness, as elicited by emotional incongruence, on later recognition of contextual environments.

Taken together, the present study aimed to investigate whether facial expressions influence later recognition of contextual environments (e.g., scene pictures) and whether the effect regarding recognition is dependent on the emotion of the environments. To address this issue, the participants in the encoding phase were presented with (threat-related and neutral) scene pictures together with facial expressions. Threatrelated scenes and faces were used because threat is thought to elicit stronger emotional responses as a result of evolutionary distinctiveness (Batty & Taylor, 2003; Caharel et al., 2005; Righart & de Gelder, 2006; Stekelenburg & de Gelder, 2004). Previous studies also used threatrelated faces and scenes to investigate the issues regarding emotional (in)congruence (Righart & de Gelder, 2006; Wieser & Brosch, 2012). The emotion of the scene pictures was either congruent or incongruent with the faces. In order to increase the attention towards the facial expressions, the task during the encoding phase was related to the faces rather than the scene pictures. Given that previous studies regarding face recognition often used the intentional and incidental face learning tasks, the present study also used intentional (Experiment 1) and incidental face learning tasks (Experiment 2) during the encoding phase. Participants were not informed that there would be a recognition task regarding scene pictures until the end of the encoding task. In the recognition phase, emotional scene pictures were presented again, without any facial expressions. Participants were asked to judge whether the prompted scene pictures had been presented or not in the preceding phase. In accordance with distinctiveness theory and related empirical studies, we predicted that emotional incongruence between target faces and contextual scenes might enhance the distinctiveness of contextual scene pictures during the encoding phase and thus, the recognition of these scene pictures in the later recognition phase.

Experiment 1

The aim in Experiment 1 was to use an intentional face learning task to investigate whether emotional (in)congruence between target faces and contextual scenes influenced later recognition of emotional scenes. As mentioned above, we predicted that recognition of threat-related scenes would be enhanced by preceding emotionally incongruent facial expressions.

Participants

Thirty healthy undergraduate students (ranging from 18 to 23 years old, 15 males and 15 females) were recruited in Guangdong University of Finance via advertisement in return for a compensation of gifts worth approximately 5 RMB. All participants were right-handed and had normal or corrected-to-normal vision. No participants reported a history of neurological or psychiatric diseases. All participants gave written informed consent. The study was conducted with approval from the local ethics committee.

Stimuli

In the present study, the stimuli included emotional faces and scenes. Regarding emotional faces, sixty grey faces were selected from the Chinese Facial Effective Picture System (CFEPS; Wang & Luo, 2005). Facial pictures portrayed 30 identities (15 males and 15 females) with a threat-related (e.g., fearful) and a neutral expression each. They were cropped similarly around the face outline and centered, so that the eyes, nose, and mouth were at similar positions for all the faces. Non-facial portions (e.g., neck, shoulders, and distant hair) were removed. Facial pictures were adjusted to a size of 2.60 (horizontal) × 3 cm (vertical), and aligned in brightness and contrast by means of Photoshop CS6. The background color was then set to black. These faces were separated into two sets (30 identities each) according to the expression of the faces.

Regarding emotional scenes, they were selected from Chinese Affective Picture System (CAPS; Bai et al., 2005). The stimuli consisted of 90 threat-related and 90 neutral scene pictures. All the pictures were adjusted to a size of 15.28 × 11.11 cm and had the resolution of 28.35 px / cm. The visual non-emotional features (e.g., luminance, contrast, and composition) were also adjusted appropriately. The contents of threatrelated scenes included animals, people, landscapes, and objects that can elicit threat, and the contents of neutral scenes included ordinary and neutral animals, people, landscapes, and objects. According to normative ratings on emotional valence reported in Bai et al.’s (2005) study, threat-related scenes used in the present study were rated as more unpleasant than neutral scenes (threat-related: M = 3.15, SD = 1.56; neutral: M = 5.54, SD = 1.61; t(179) = 18.74, p < .001, d’ = 1.51). For arousal, the ratings were higher for threat-related compared to neutral scenes (threat-related: M = 4.96, SD = 2.35; neutral: M = 3.95, SD = 1.98; t(179) = 11.09, p = .004, d’ = 0.31).

One hundred and twenty pictures (60 threat-related and 60 neutral) were used as old scenes and presented during encoding and recognition phases. Each emotional category of old scenes was pseudorandomly separated into two sets according to normative valence and arousal ratings. These four sets of old scenes, combining with two facial expression sets, were used to create four experimental conditions: threat-related-congruent (threat-related faces were presented with emotionally congruent scenes), threat-related-incongruent (threatrelated faces were presented with emotionally incongruent scenes), neutral-congruent (neutral faces were presented with emotionally congruent scenes) and neutral-incongruent (neutral faces were presented with emotionally incongruent scenes). The other 60 scenes (30 threat-related and 30 neutral) served as new ones and were presented only during the recognition phase. For each emotional category of scenes, old and new scenes were similar in valence and arousal ratings (p > .05).

Procedure

Participants sat in a comfortable chair in a dimly lit and sound-proof room approximately 80 cm directly in front of a 15 in. computer monitor with a screen resolution of 1024 × 768 px. Stimulus presentation and behavioral data collection were controlled by E-Prime 1.0 software (Psychology Software Tools, Inc., Sharpsburg, PA, USA). All stimuli were presented against a dark background.

The experiment consisted of an encoding phase and a recognition phase. For the encoding phase, each trial started with a white fixation cross for 300 ms, followed by a blank screen ranging from 1000 to 1400 ms (M = 1200 ms). Subsequently, a face was presented concomitant with a picture for 2500 ms. In order to reduce the overlapping between faces and pictures, the pictures were presented in the center of the screen, while the faces were presented in the bottom half of the scene, so that essential parts of the picture were not concealed. This manipulation was in accordance with Pierguidi et al.’s (2016) study. During the presentation of the faces, the participants were asked to memorize the facial identities regardless of scenes. The participants were also told that an identity would be presented for four times, with a threat-related expression twice and a neutral expression twice. Participants were instructed to memorize the identity of the faces in particular, irrespective of their expressions. The presentation sequence of experimental conditions was randomized across participants. The next trial started after another blank screen for 700 ms.

The recognition phase immediately followed the encoding phase. Participants were informed about the recognition of scenes rather than faces at this point in the experiment. For the recognition phase, each trial started with a white fixation cross for 300 ms. Following a blank screen ranging from 1000 to 1400 ms (M = 1200 ms), either an old or a new picture was presented for 2500 ms. Importantly, all the pictures were presented in isolation, without any faces. The participants were asked to identify whether the prompted scene had been presented in the preceding encoding phase (i.e., old scenes) or not (i.e., new scenes) by pressing the “F” and “J” keys using the index finger of the left and right hands. The instructions also emphasized speed and accuracy. Response assignments were counterbalanced across participants. The subsequent trial started after another blank screen for 700 ms.

Experimental procedure in the coding recognition phases in Experiments 1 and 2. Participants were asked to memorize the facial identities regardless of the scenes in Experiment 1 and to indicate the gender of a face in Experiment 2.

In the encoding phase, each face was presented two times, once in the congruent condition and once in the incongruent condition. Each scene picture was presented once. In the recognition phase, all the scenes (including old and new ones) were presented once. Therefore, there were 120 trials (30 trials for each experimental condition) in the encoding phase and 180 trials in the recognition phase (including 120 old scenes and 60 new scenes). The actual experiment lasted approximately half an hour. Experimental procedures in the coding phase and in the recognition phase are shown in Figure 1.

Data Recording and Analysis

Response accuracies and times were recorded for old and new scenes during the recognition phase. Response accuracy refers to correctly judging old scenes as “old” or new scenes as “new.” For data analysis, response accuracies and times were analyzed in 3 × 2 repeated-measures analyses of variance (ANOVAs) with congruency and novelty (congruent, incongruent, new) and emotion (threat-related, neutral) as withinsubject factors. Means and their SDs across the conditions are presented in Table 1.

Results

Response accuracy. The analysis of response accuracy showed a main effect of congruency and novelty, F(1, 29) = 12.40, p < .001, partial η2 = .30, and emotion, F(1, 29) = 7.03, p = .013, partial η2 = .20. Response accuracy was generally higher for new scenes than for emotionally incongruent and congruent scenes and for threat-related scenes than for neutral scenes.

These main effects were further quantified by a two-way interaction between congruency and novelty and emotion, F(1, 28) = 19.78, p < .001, partial η2 = .41. Further analysis showed that for threat-related scenes, the effect of congruency and novelty was not significant (p > 005). For neutral scenes, there was an effect of congruency and novelty, F(1, 28) = 24.04, p < .001, partial η2 = .45. Post hoc t-tests showed that response accuracy was higher for new scenes compared to emotionally incongruent and congruent scenes (both ps < .001).

Notably, the differences were quite large between old and new neutral scenes. This might have influenced the ANOVA model, thus reducing the significance levels of other effects, for example, the effects between congruent and incongruent scenes. Thus, we excluded the new scenes and only analyzed old scenes by a 2 × 2 repeated-measures ANOVA with congruency (congruent, incongruent) and emotion (threat-related, neutral). This revealed a two-way interaction between those two factors, F(1, 28) = 4.37, p = .045, partial η2 = .13. Further analysis showed that for threat-related scenes, response accuracy was higher when the scenes had been encoded with emotionally incongruent compared to congruent faces, F(1, 28) = 8.71, p = .006, partial η2 = .23. For neutral scenes, however, the effect of congruency was not significant (p > .05). Therefore, the inclusion of new scenes reduced the differences between congruent and incongruent threat-related scenes.

Response time. The analysis of response time showed a main effect of emotion, F(1, 29) = 34.09, p < .001, partial η2 = .54. Response time was shorter for threat-related compared to neutral scenes. Other main effects or interactions were not significant (p > .05). Notably, the exclusion of new scenes did not influence the other effects.

Mean Response Accuracy, Response Time (ms) and SD for All the Experimental Conditions in Experiment 1

Discussion

The present study investigated whether emotional (in)congruence between target faces and contextual scenes influences sudden recognition of scenes by an intentional face learning task. In line with the hypotheses, the results showed that response accuracy was higher for threat-related scenes when they had been encoded with emotionally incongruent compared to congruent faces. However, there was no effect of emotional incongruence for neutral scenes. The findings may suggest that the effect of facial expression on scene recognition is dependent on the threat quality of the scenes when the faces and/or facial expressions are attended to. In addition, we also found that for neutral scenes, response accuracy was higher for new scenes than for old scenes, suggesting that participants had a bias towards recognizing neutral scenes as new. However, this bias was not significant for threat-related scenes/

Experiment 2

In Experiment 1, we found that emotional (in)congruence between target faces and contextual scenes influenced the recognition of threatrelated scenes when an intentional face learning task was used to increase the attention towards facial expression during the encoding phase. In Experiment 2, we investigated the effect of emotional incongruence on emotional contextual scenes when an incidental face learning task was used. As mentioned above, we predicted that emotional incongruence between target faces and contextual scenes would also increase later recognition of scenes, in particular threat-related scenes.

Participants

Thirty healthy undergraduate students (ranging from 18 to 23 years old, 15 males and 15 females) were recruited in Guangdong University of Finance via advertisement in return for a compensation of gifts worth approximately 5 RMB. The participants did not participate in Experiment 1. All participants were right-handed and had normal or corrected-to-normal vision. No participants reported a history of neurological or psychiatric diseases. All participants gave written informed consent. The reported study was conducted with approval from the local ethics committee.

Stimuli

The stimuli were the same as in Experiment 1.

Procedure

The experimental procedure was similar to Experiment 1, except that the task during the encoding phase was an incidental face learning task. During the presentation of the faces, the participants were asked to indicate the gender of the faces by pressing the “F” and “J” keys using the index finger of the left and right hands, respectively. Response assignments were counterbalanced across participants.

Data Recording and Analysis

Data recording and analyses were similar to those in Experiment 1. Means and their SDs across conditions are presented in Table 2. In addition, to examine whether the effect of congruency and novelty differed between Experiments 1 and 2, we further performed a 2 × 3 × 2 (Emotion [threatrelated, neutral] × Congruency and Novelty [congruent, incongruent, new] × Experiment [Experiment 1, Experiment 2]) repeated-measures ANOVAs separately on response accuracy and response time. To exclude the influence of new scenes, response accuracy and response time were also separately performed a 2 × 2 × 2 (Emotion [threat-related, neutral] × Congruency [congruent, incongruent] × Experiment [Experiment 1, Experiment 2]) repeated-measures ANOVAs.

Results

Response accuracy. The 3 × 2 ANOVA on response accuracy did not show any main effects or interactions (p > .05). Similar to Experiment 1, when we only analyzed old scenes in a 2 × 2 repeated-measures ANOVA, the results showed a main effect of congruency, F(1, 29) = 4.44, p = .044, partial η2 = .13. Response accuracies were generally higher for emotionally incongruent scenes than for congruent scenes.

Response time. The 3 × 2 ANOVA on response time showed main effects of emotion, F(1, 29) = 33.89, p < .001, partial η2 = .54, and congruency and novelty, F(1, 29) = 4.51, p = .015, partial η2 = .13. Response time was generally longer for new scenes than for emotionally incongruent and congruent scenes and for threat-related compared to neutral scenes. The interaction between those two factors was not significant (p > .05).

Mean Response Accuracy, Response Time (ms) and SD for All the Experimental Conditions in Experiment 2

When we only analyzed old scenes using a 2 × 2 repeated-measures ANOVA, the results showed a main effect of emotion, F(1, 29) = 31.68, p < .001, partial η2 = .52. Response time was shorter for threat-related scenes compared to neutral scenes. Other main effects or interactions were not significant (p > .05).

Differential Effects of Congruency, Novelty, and Emotion Between Experiments 1 and 2

Response accuracy. The abovementioned 2 × 3 × 2 ANOVA showed that the three-way interaction among emotion, congruency and novelty, and experiment was significant, F(1, 29) = 4.40, p = .014, partial η2 = 0.07. A separate analysis for each emotion category showed that for threat-related scenes, there was no interaction between congruency and novelty and experiment (p > .05). For neutral scenes, however, the interaction between those two factors was significant, F(1, 29) = 5.65, p = .006, partial η2 = .16. Thus, the findings suggest that the effect of congruency and novelty was different between Experiment 1 and 2 for neutral scenes but not for threat-related scenes. Further analyses showed that response accuracy was generally higher in Experiment 2 than in Experiment 1 for old scenes (congruent: F[1, 29] = 4.00, p = .055, partial η2 = .12; incongruent: F[1, 29] = 4.28, p = .048, partial η2 = .13), and for new scenes, response accuracy was higher in Experiment 1 than in Experiment 2, F(1, 29) = 4.43, p = .044, partial η2 = .13. In addition, the main effect of experiment or the other interaction associated with the factor experiment was not significant.

When new scenes were excluded, the 2 × 2 × 2 ANOVA showed a main effect of emotion, F(1, 29) = 8.70, p = .006, partial η2 = .23, and congruency, F(1, 29) = 9.70, p = .004, partial η2 = .25. Response accuracy was generally higher for emotionally incongruent scenes than for congruent scenes and for threat-related scenes than for neutral scenes. No interactions were significant (p > .05). Taken together, the 2 × 3 × 2 and 2 × 2 × 2 ANOVAs suggest that the abovementioned interaction between congruency and novelty and experiment was only related to new neutral scenes, whereas response accuracy was not significantly different for old scenes between Experiment 1 and 2.

Response time. For response time, the abovementioned 2 × 3 × 2 and 2 × 2× 2 ANOVAs did not show a main effect or interaction of experiment (p > .05). Thus, the effects on response time were not significantly different between Experiment 1 and 2.

Discussion

The present study investigated whether emotional congruence between faces and scenes influenced later recognition of scenes when an incidental face learning task was performed to increase the attention towards the faces/facial expressions. In line with the hypotheses, the results showed that response accuracy was generally higher for both threat-related and neutral scenes and for scenes that have been encoded with emotionally incongruent compared to congruent facial expression. However, as the effect of congruency was not significantly different between Experiments 1 and 2 and as Experiment 1 did not show an effect of congruency for neutral scenes, the congruency effect seemed to be more prominent for threat-related scenes than for neutral scenes, even though the interaction between congruency and emotion was not significant. Taken together, similar to Experiment 1, the present study also suggests that recognition of threat-related scenes is influenced by a preceding facial expression, in particular an emotionally incongruent facial expression

General Discussion

The present study investigated whether facial expression influenced later recognition of emotional scenes when intentional and incidental face learning tasks (Experiment 1 and 2, respectively) were performed to increase the attention towards facial expression during the encoding phase. Results generally showed that response accuracy was higher for threat-related (e.g., fearful) scenes when they had been encoded with emotionally incongruent compared to congruent faces. The findings generally indicate that recognition of threat-related scenes is enhanced by a preceding emotionally incongruent facial expression, and that this effect is independent of face-related tasks. However, the effect of congruency was not significant for neutral scenes, suggesting that the emotion of facial expression does not influence recognition of neutral scenes.

The findings regarding threat-related scenes in the present study are in line with our hypotheses, which suggested better recognition of emotional scenes that had been encoded with emotionally incongruent facial expressions. The current findings might also be in line with distinctiveness theory, which suggests that stimulus retrieval can be enhanced by distinctiveness (Schmidt, 1991). Specifically, expectancy incongruence is a phenomenon of distinctiveness. In the present study, emotional congruence and incongruence between faces and scenes might be considered as expectancy congruence and incongruence, respectively, as emotional congruence is more consistent with common sense/expectation than emotional incongruence. Therefore, the findings in the present study might suggest that emotional incongruence between target faces and contextual scenes increases the distinctiveness of the scenes during the encoding phase and thus, the recognition of the scenes in the later recognition phase.

However, different from our hypotheses and distinctiveness theory, the present study did not show an effect of emotional congruence for neutral scenes. As mentioned above, scene pictures were more distinctive in the incongruent condition than in the congruent condition. More importantly, threat-related faces are also thought to be more distinctive than neutral faces. Due to limited attention resources and task demands, threat-related faces should reduce the distinctiveness of scene pictures to a larger extent than neutral faces, irrespective of emotional (in)congruence. Thus, for threat-related scenes, the distinctiveness of congruent scenes was reduced by threat-related faces. Given the increased distinctiveness of incongruent scenes and the reduced distinctiveness of congruent scenes, there was a strong effect of emotional incongruence on later recognition of threat-related scenes. For neutral scenes, however, emotional incongruence increased the distinctiveness of incongruent neutral scenes, whereas threat-related facial expression reduced the distinctiveness of those neutral scenes. The reversed effect of distinctiveness might reduce the effect of emotional incongruence on later recognition.

More importantly, while previous empirical studies have suggested that distinctiveness of target stimuli influences their retrieval, the present study was the first to demonstrate the effect of distinctiveness, as elicited by emotional incongruence, on retrieval even for the inattentive contextual stimulus. However, this effect regarding distinctiveness seems to be different between target and contextual stimuli. For instance, our recent study (Lin & Liang, 2019b) showed the effect of expectancy incongruence on recognition of neutral rather than negative target stimuli, suggesting that the effect of distinctiveness, as elicited by expectancy/emotional congruence, on retrieval occurs for nondistinctive but not for distinctive target stimuli (e.g., negative scenes). However, the present study demonstrated that the effect of distinctiveness on retrieval of contextual stimuli might only occur for distinctive contextual stimuli (e.g., threat-related scenes) rather than for nondistinctive stimuli (e.g., neutral scenes). Taken together, the findings of our previous study and the present study might extend the distinctiveness theory, in which distinctiveness, as elicited by expectancy/emotional congruence, influences retrieval of both target and contextual stimuli, whereas the effect of distinctiveness occurs for nondistinctive target stimuli but for distinctive contextual stimuli. However, given that there are several differences between our previous study (Lin & Liang, 2019b) and the present study (e.g., paradigms and stimuli), it is difficult to ascertain the reasons for the differential distinctiveness effects between target and contextual stimuli. Future studies might investigate the related issues in more detail.

Different from the findings in the present study, Bowen and Kensinger (2017) did not find an effect of preceding facial expressions on subsequent recognition of words. In that study, the participants were asked during the encoding phase to identify whether the stimulus was a scene or a face as well as to memorize the words. Considering that the identification task was much easier than the memory task, the participants may have had a main focus on memorizing the words rather than identifying the faces. Therefore, the participants may have paid little attention on the facial expressions, leading to the general reduction of the effects of facial expression. More importantly, Bowen and Kensinger’s study did not investigate the congruency effect between faces and words, which might be important in influencing the recognition of words. In the present study, the participants were asked to focus on the faces rather than the scenes during the encoding phase by intentional and incidental face learning tasks. Emotional (in)congruence between faces and contextual scenes were manipulated. With more attention towards the faces and facial expressions, there was an effect of emotional congruence particularly for threat-related scenes.

In addition, the results showed that in Experiment 1, recognition of neutral scenes was generally better in the new condition than in the old condition, suggesting that the participants had a response bias of judging neutral scenes as new. This bias was stronger for threat-related scenes in this experiment or even for neutral scenes in Experiment 2. In Experiment 1, the participants were asked to memorize the facial identities. The participants may have anticipated about face retrieval in the near future and thus, have recalled the faces before they have known that the recognition task was regarding the scenes. This recall might distract unconscious encoding and consolidation of the scenes. Moreover, this distraction might be stronger for neutral scenes than for threat-related scenes, as threat-related stimuli are distinctive and thus, are resistant to distraction (Lin et al., 2017). Thus, in the later recognition task, participants were more likely to judge old neutral scenes as new. Different from Experiment 1, Experiment 2 asked the participants to judge the gender of the faces. In this case, the participants did not anticipate about an upcoming task. Unconscious encoding and consolidation of the scenes were thus not affected. This might be the reason that there was no response bias for neutral scenes in Experiment 2.

Finally, we would like to mention some limitations in the present study and suggest the directions for future studies. Firstly, the present study investigated the effect of emotional congruence only in the context of threat-related and neutral emotions. Future studies might investigate the related issue in the context of other emotions, such as other negative emotions (e.g., disgust and sadness) and positive emotions. Secondly, according to the findings in Bowen and Kensinger’s (2017) study and in the present study, the effect of facial expression on recognition of other stimuli was not shown when the participants paid much more attention towards other stimuli than the faces during the encoding phase. Therefore, it is still uncertain as to whether the effect regarding facial expression emerges when the attention towards the faces was similar to the attention towards other stimuli (e.g., participants are asked to memorize both faces and other stimuli). Future studies may further investigate this issue. Finally, the present study essentially investigated the effect of facial expression on recognition only of contextual emotional scenes. In future studies, the effect of facial expression on recognition of some other contextual emotional stimuli or body expressions might also be investigated.

Conclusion

In general, the present study found that recognition of threat-related scenes was enhanced by preceding emotionally incongruent facial expression for both intentional and incidental learning tasks, and the effect of emotional incongruence was reduced for neutral scenes. Taken together, the findings indicate that facial expression influences later recognition of contextual threat-related scenes, irrespective of task demand.