Distinctive Effects of Within-Item Emotion Versus Contextual Emotion on Memory Integration.

This study investigates the links between memory and emotion and, more specifically, how emotions can impact the integration mechanism. The authors' hypotheses were based on a dynamic conception of memory (Versace et al., 2014; Macri et al., 2018), and stated that an emotion coming from the stimulus (within-item emotion) should enhance the integration of the stimulus features, and that an emotion coming from the context (contextual emotion) should improve integration of the item and its context. In two experiments, the participants performed an associative memory task in which they undertook three kinds of recall: item (memory for a target item), location (spatial position of a target item), and association recall (association of a target item and its location). In the first experiment, the emotion was introduced by the target stimuli (neutral or negative words), while in the second experiment, contextual emotion was introduced by means of an odorant dispenser (negative or neutral odorant) placed under the participant's chin and only neutral words were used. In both experiments, target items were words objects or animals that were either typically associated to a sound or not typically associated to a sound). The results confirm that emotions act in different ways on the integration mechanism depending on how they are introduced to the participant: within-item emotion enhances item recall itself by strengthening the link between its components, while contextual emotion favors the integration of the item with its location. Copyright:

Introduction

Studies investigating the effect of emotion on memory have traditionally used different ways to induce emotional processing, with emotion being either a component of the stimulus itself or not a property of the stimulus, but instead a property of the experimental context or the participant's emotional state. These methods bring about major differences in the way emotion impacts memory performance.

There is ample literature on memory for emotional stimuli. For example, Levine and Pizarro (2004) demonstrated that emotional memories are more sustainable, enriched with more details, and more indelible than memories without emotional content. They explained this phenomenon in terms of the activation of the amygdala, which facilitates consolidation and long-term memory reinforcement. On a behavioral level, Earles et al. (2016) showed that negative events have higher recall rates than positive ones, while Kensinger and Schacter (2006) found that negative events were remembered more precisely and more accurately than positive ones. In such studies, where emotion was introduced through the items, authors have often proposed explanations in terms of attentional mechanisms: information with a high emotional content captures attention more effectively and is therefore remembered better than nonemotional information (Bower, 1981; Kensinger & Corkin, 2004).

Concerning the effects of emotion introduced as a contextual feature, Bower (1981) used the phrase “mood-state dependent” retrieval to express the fact that the recall rate is higher when the emotional states are similar at the times of memorization and retrieval than when they are different. Bower also studied the mood congruency effect, which reflects the tendency of individuals to remember information more easily when this information and their current state share the same emotional content. These two phenomena are, in fact, very similar to the contextual effects commonly observed in memory research (see Baddeley, 1993). The explanation of the effects obtained is therefore not specific to emotion. Moreover, these studies have not proposed any satisfactory explanation of these results in terms of mechanisms that are thought to play a significant role in memory efficiency.

According to Act-In, a memory model proposed by Versace et al. (2009; 2014), memory contains multiple traces which reflect all the sensorimotor experiences of the individual, distributed across multiple neuronal systems, and which code the multiple components of the experiences. In such an approach, the emergence of specific knowledge requires multi-component integration at the level of the individual trace during encoding, that is, the components have to be integrated within the same trace. Therefore, the effectiveness of the re-emergence of specific traces depends on the level of integration of the features within the trace. The notion of level of integration refers to the strength of the link between the components of the trace. The more integrated the features, the more likely it is that the trace as a whole will be dissociated from other traces and therefore be specifically retrieved. Among the factors likely to strengthen the level of integration of the various elements in an experience, Versace et al. (2014; see also Versace & Rose, 2007) propose the subject's emotional state.

However, according to Macri et al. (2018), to understand the origin of the effects of emotion on memory, it is necessary both to take into account the origin of the emotion and to make a distinction between item memory (memory for the target items) and source memory (memory for the contextual information). They proposed that an emotion coming from the stimulus (within-item emotion) enhances integration of the stimulus features and that an emotion coming from the context (contextual emotion) improves integration of the item with its context. However, no study so far has directly tested this hypothesis, even though the available results do argue in its favor. For example, Mather (2007) proposed an object-based framework to formalize the literature on the effects of within-item emotion on memory performance. This framework states that emotion coming from a stimulus attracts attention and enhances the binding of the constituent features of the item itself. Mather suggests that this binding applies to the features processed during perception and maintains them in the longterm memory in the case of within-object components. In turn, this attraction either impairs or does not affect the associations between the item and other items or its context. This interpretation is supported by a large body of literature investigating item memory, source memory, and associative memory (Adelman & Estes, 2013; Cook et al., 2007; D’Argembeau & Van der Linden, 2004; Kensinger & Schacter, 2006; Mather & Nesmith, 2008; Rimmele et al., 2011; Touryan et al., 2007).

Fewer studies have investigated the effects of contextual emotion. In Macri et al.’s (2018) experiment, participants memorized pictures displayed in a 16-square grid while the emotional context was manipulated to be negative or neutral by means of an olfactory dispenser diffusing a negative or neutral fragrance that had been pre-tested to confirm the respective valences. The participants then performed three potential types of recall: verbal item recall, location recall (selecting the different squares in which the words had appeared), and recall of the association between the item and its location (by selecting the items at the bottom of the screen and moving them to the squares in which they had initially appeared). The results revealed that the participants achieved higher recall rates for location and association recall with the negative than with the neutral odor, while no significant gain was shown for item recall. These results confirm the authors’ hypothesis that contextual emotion improves integration of the item with its context. However, their experiment did not directly test the hypothesis that an emotion coming from the stimulus enhances the integration of stimulus features.

The present study was therefore designed to test this hypothesis and to confirm Macri et al.'s (2018) results. The same task as in Macri et al. was used. However, item, source, and memory for the association between the item and its source were evaluated with a new type of stimulus. Pictures were replaced by words to prevent bias due to the complex features of pictures, such as visual saliency, color, or shape, which can affect performance. In the first experiment, we compared the influence of negative compared to neutral words on item, source, and association recall. To evaluate item integration, half the words designated objects typically associated with sounds.

Our hypothesis was that the presentation of a word should reactivate all the perceptual components encoded in the memory traces associated with the word. Therefore, if the word designates an object typically associated with sounds, then the content of this reactivation should have more components. For example, on the one hand, the words “ambulance” and “spider” are considered negative words. The word “ambulance” is strongly associated with the sound of its siren and is described as a negative sonorous word. By contrast, the word “spider” is not typically associated with a sound and is described as a negative non-sonorous word. On the other hand, the words “diary” and “flipper” are considered neutral. The word “flipper,” which is strongly associated with the noises made by the machine when it is played, is a neutral sonorous word whereas the word “agenda,” which is not typically associated with a sound, is a neutral non-sonorous word. If within-item emotion does improve the integration of the stimulus features, then the more multimodal an item (typically, sound items in this experiment), the more it should benefit from the emotion. No effect of within-item emotion on the recall of the association between the items and their positions was expected in this experiment. In the second experiment, emotion was introduced by means of a negative odor (contextual emotion). We therefore expected to replicate Macri et al.'s (2018) results and observe an effect of contextual emotion limited to the integration of items with their position, and no effect of emotion on item integration.

Method

Item Selection

The items were selected from Bonin et al.’s (2003) 866-word database. A total of 432 words, preselected as the most neutral and negative items in the database, were subjected to a pretest conducted with 11 participants (seven women and four men; M = 23.9 years, SD = 2.5). These 432 items were rated on a five-point Likert-type scale for perceived valence, sonority, odor, concreteness, imagery, and occurrence (see Appendix A), enabling us to fine-tune the selection to obtain 96 neutral and 48 negative items. Each experiment used 96 stimuli from this selection: 48 neutral and 48 negative words for the experiment introducing emotion through the stimuli, 96 neutral items for the experiment introducing emotion through the olfactory context. In Experiment 1, the aim was (a) to select two subsets, one of 48 neutral and the second of 48 negative words, that differed only in terms of emotional valence and that were equivalent on all other aspects; and (b) within each 48-word subset, to rate 24 words as sonorous and the other 24 as nonsonorous. In Experiment 2, the aim was to select 48 sonorous neutral words and 48 nonsonorous neutral words, and for each 48-word subset to differ from the other only in terms of sonority (see Appendix B). This was done in order to reduce the effect of confounding variables. For example, in the second experiment, if one subset contained words that were rated as more strongly associated with an odor than the words in the other subset, then presenting an odor during the task could have favored the recall of these words.

Experiment 1

Participants

A total of 24 nonsmoking students (19 women and five men; M = 21.8 years, SD = 1.5) from the University of Lyon 2 participated in the experiment. Each student provided written informed consent. All the participants had normal or corrected-to-normal vision and presented no neurological, psychiatric, or olfactory disorders.

Stimuli and Material

Forty-eight neutral and 48 negative items were used for this procedure. The 48 negative items were selected to obtain 24 words designating manufactured items and 24 words designating natural items, and this balance was also implemented for words designating sonorous and nonsonorous items. The same selection criteria applied to the neutral items. The experiment was conducted on a Macintosh computer with 21.5 in. screen. OpenSesame software (Mathôt et al., 2012) was used to set up and run the experiment.

Procedure and Design

The participants performed an associative memory task in which 12 series of eight words (four sonorous and four nonsonorous words per series) were randomly presented in a 16-square grid for 2 s, with an interstimulus interval of 500 ms. The participants were instructed to remember both the words and their locations. After each series of eight words, the participants were asked to perform one of three kinds of recall. In the item recall, the subjects had to verbally report the words independently of their position. In the location recall, they had to click on the squares (eight out of 16) where the words had appeared, and in the association recall, they had to drag the eight words, which were displayed again at the bottom of the screen, to the squares in which they had initially appeared. Four series of eight words were presented for each of the three kinds of recall. Half of the four series displayed neutral stimuli and the other half presented negative stimuli. The words in each series were presented in a random order for each participant. Across participants, words were used in each of the three types of kinds of recall. The same type of recall could not be repeated more than twice in succession. The participants were familiarized with the task by means of a training phase in which the three types of recall were required. For each participant, the total duration of the experiment was about 40 minutes.

Results

The mean percentages of correct responses were calculated across subjects for each experimental condition (see Table 1). For each type of recall, analyses of variance (ANOVAs) were performed on percentages of correct recall with emotion type (negative vs. neutral words) and sonority (sonorous vs. not sonorous) as within-subject factors.

Item recall. The analysis did not reveal any significant difference between negative and neutral words, F(1, 23) = 2.56; p = .91, or between sonorous and nonsonorous words (F < 1). However, it showed a significant interaction between emotion type and sonority, F(1, 23) = 5.50; p = .03; η2 = .19 (see Figure 1).

Experiment 1 – Mean Percentages of Correct Responses (CR) in Each Experimental Condition

Mean percentage of correct responses as a function of emotion and sonority for item recall in Experiment 1. Error bars represent the SEM.

As expected, the sonorous words benefited more from the presence of emotion than the nonsonorous ones. Planned comparisons showed a significant difference between negative and neutral words for sonorous items (p = .004; d = .59), but not for nonsonorous ones (p = .98).

Location recall. The analysis showed that the correct recall rate was higher for the locations of negative words (M = 91%; SD = 10%) than for those of neutral words (M = 64%; SD = 17%), F(1, 23) = 112.01; p < .001; η2 = .83.

A significant effect also appeared for sonority, with a higher recall rate being found for locations displaying sonorous words (M = 85%, SD = 14%) than nonsonorous words (M = 0.70%, SD = 14%), F(1, 23) = 23.66; p < .001; η2 = .51. The interaction between sonority and emotion was also significant, F(1, 23) = 16.67; p < .001; η2 = .42. As Figure 2 shows, the difference between sonorous and nonsonorous words appeared for neutral words, F(1, 23) = 29.01; p < .001; η2 = .56, but not for negative ones (F < 1).

Association recall. The results did not show any significant main effect or any interaction involving the associations of words and locations (all Fs < 1).

Discussion

Emotions coming from the item significantly enhanced the recall rate of sonorous items, thus supporting our hypothesis that within-item emotion facilitates the integration of the item features. Therefore, the more multimodal an item, the more it benefits from emotion. By contrast, within-item emotion did not improve the recall of the association of the items and their positions (an item of contextual information). Moreover, recall rates for the locations of the sonorous items were higher only when the items were neutral. Therefore, when emotion is derived from the item, it seems to outweigh the sonority effect.

A limitation of this study is the fact that we did not anticipate the impact of sonority on the recall of location. By interfering with it, it could prevent a more clear-cut effect of emotion from emerging. The study could therefore be replicated while manipulating another feature, such as a gesture associated with the word.

To confirm our hypothesis that emotion has specific effects depending on its source (within-item or contextual), we performed a second experiment in which emotion was contextual and was introduced by means of a negative odor. We therefore expected to observe an effect of emotion that would be limited to the integration of items with their positions.

Experiment 2

Participants

Twenty-four different non-smoking students (22 women and two men; M = 21.8, SD = 2.3) volunteered based on the same eligibility criteria as in Experiment 1.

Stimuli and Material

Each of the 96 neutral words selected was presented in the 16-square grid during the experiment. Amongst these 96 words, 48 had also been used in the first experiment, while the other 48 were specific to this experiment. There were equal numbers of words designating manufactured and natural items (48 of each), as well as equal numbers of sonorous and nonsonorous words. Each stimulus had a neutral valence. The emotion was induced by means of two flasks containing odorous substances (caraway seed; Carvone-d; 16724; 1.96% for the neutral odor; and alcohol; 1-Decanol; 8174; 50.91% for the negative odor). These were selected after a pretest phase conducted for a previous experiment and involving 10 participants who characterized them in terms of their emotional valence (neutral/unpleasant). A chinrest was used during the experiment so that the participants could watch the pictures and inhale the smell.

Mean percentage of correct responses as a function of emotion and sonority for location recall in Experiment 1. Error bars represent the SEM.

Procedure and Design

The participants followed the same procedure as in Experiment 1. Half of the participants performed the first six blocks with the negative odor and the last six blocks with the neutral one, whereas this order was reversed for the other half of the participants.

After having inhaled the contents of the odorant flask, the participants placed their heads on a chinrest and the flask containing the odor was placed under their noses so that they could inhale the odor while performing the task. A few minutes later, the participants performed the last six series following the same procedure but with the second odorant flask. For each series, the odor was administered both during the presentation of the words as well as during the recall phase. The words associated with the different blocks of odors and the different types of recall were counterbalanced across participants so that each word could be associated with each odor and each type of recall. The participants were familiarized with the task by means of a training phase in which the three types of recall were required.

Results

The mean percentages of correct responses were calculated across subjects for each experimental condition (see Table 2). For each type of recall, analyses of variance were performed on the percentages of correct recall, with emotion type (negative vs. neutral words) and sonority (sonorous vs. nonsonorous) as within-subject factors.

Item recall. The analysis did not reveal any significant difference between neutral and negative induction or between sonorous and non-sonorous items (all Fs < 1).

Location recall. The analysis only revealed a significant effect of sonority, as the correct recall rate was higher for sonorous words (M = 83%; SD = 19%) than for nonsonorous words (M = 62%; SD = 21%), F(1, 23) = 27.567; p < .001; η2 = .55.

Association recall. The results showed a higher recall rate for words presented during the negative induction (M = 50%; SD = 16%) than during the neutral induction (M = 25%; SD = 14%), F(1, 23) = 70.663; p < .001; η2 = .75. The analysis did not reveal a significant effect of sonority or an interaction between valence and sonority.

Experiment 2 – Mean Percentages of Correct Responses (CR) in Each Experimental Condition

Discussion

The results of the second experiment support the interpretation suggesting that an emotion coming from the context seems to improve the integration of contextual features. This will be discussed further below.

One point of interest that was not anticipated was the effect of sonority on the recall of the locations of the stimuli. The locations for sonorous words were recalled better than those of nonsonorous words. In the previous experiment, the same effect was found for neutral words. This effect might be specific to sound. Indeed, sound localization as a primary cognitive function of our species might explain why the locations of sonorous words are better recalled even when the sound is not directly present in the scene.

General Discussion

The main objective of this study was to demonstrate that, depending on its source, emotion can influence different forms of integration. An emotion coming from the stimulus (within-item emotion) enhances integration of the stimulus features, and an emotion coming from the context (contextual emotion) improves integration of the item with its context.

In the first experiment, we used within-item emotion (neutral or negative words), with half of the items typically being associated with a sound in memory. If within-item emotion really does improve the integration of the stimulus features, then the more multimodal an item (typically, sound items), the more it should benefit from the emotion. As expected, in the case of item recall, we observed a significant interaction between emotion type and sonority, showing that the sonorous words benefited more from the presence of emotion than the nonsonorous ones. By contrast, no effect of within-item emotion was observed for the recall of the association of the items and their positions. The effect of emotion found on the recall of locations is more likely to be due to attentional effects, as has been reported more than once in the literature (Mather & Nesmith, 2008), thereby showing the high impact of arousal (and not valence) on the recall of locations. Concerning the effect of sonority on the recall of locations, an explanation in terms of attentional mechanisms cannot be the only one, given that this effect only appeared for neutral words. Since sound is an intrinsic (mnesic) property of items, the positions associated with a sound item are remembered better. However, in the case of negative items, this effect is hidden by the attentional effect of sonority.

In the second experiment, a contextual emotion was introduced by means of an olfactory dispenser diffusing negative or neutral fragrances. The main goal was to replicate the results of Macri et al. (2018), with an effect of contextual emotion on the recall of the association between the items and their positions, but not on the recall of the items, irrespective of their sonority. The results completely confirmed our expectations. The NEVER model (Bowen et al., 2018) proposes that, compared to positive events, negative events trigger an augmented encoding of sensory details, and that these valence-based differences occur even when the events induce the same level of arousal. Even if the authors do not specify the origin of the emotion, they nevertheless refer to negative events, which suggests that the emotion comes from the encoding episode in its entirety. This idea of an increased encoding of sensory details is consistent with our proposition. The main difference is that we propose that negative events do not only increase the encoding of sensory details, but that they also facilitate their integration, which is essential if this multisensory encoding is to be effective. However, one of the differences between the NEVER model and the Act-In model is that the former predicts different results depending on the positive or negative valence of the events whereas, according to Act-In, the presence of any emotion (i.e., arousal) should facilitate this integration. It would therefore be interesting to replicate our Experiment 2 with positive stimuli.

Contrary to the NEVER model, Clewett and Murty (2019) proposed that high-arousal states will elicit greater memory selectivity, regardless of the level of behavioral activation (defined as a motivational state of increased vigor, energization that promotes exploration and novelty-seeking, and low arousal). By contrast, in states of high behavioral activation, they predict greater memory integration and/or broadening. Although this idea is particularly interesting, the difficulty here is whether a negative odorant would lead to a decrease or an increase in behavioral activation. By decreasing behavioral activation and increasing arousal, the negative odorant could lead to a narrowing of attention and less memory integration. However, if we limit ourselves to the approach and/or avoidance reactions associated with emotions, then it is possible that a negative odorant may be associated with an avoidance reaction and therefore a high level of behavioral activation and, consequently, according to Clewett and Murty (2019), to a high level of integration of all the elements of the encoding episode.

It therefore seems essential to replicate this research by trying to dissociate to the effects of valence from the effects of arousal. According to the Act-In model, on which our hypotheses are based, valence is thought to be able to bind the elements of the episode, provided, of course, that the level of activation of these elements is sufficiently great to modify the content of the memory traces.

According to Yonelinas and Ritchey (2015), emotional events are remembered better than neutral events, and this emotion advantage becomes particularly pronounced over time. Moreover, they assume that these time-dependent effects impact upon recollection rather than on familiarity-based recognition. This is consistent with the idea that emotion facilitates the integration of the item with contextual information, thus allowing the context to provide cues that lead to the reactivation of the item itself (e.g., Tulving & Thomson, 1973).

The results of the two experiments reported in this article therefore shed light on the different ways emotions can influence memory processes, and are consistent with our hypotheses of a specific impact of emotion on the integration mechanism, although future studies would require a manipulation that would allow for a direct measurement of integration.

However, the nature of the trace is significantly different depending on how emotion is introduced to the participant and it leads directly to an improvement of different kinds of retrievals (see Figure 3). When emotion is induced by means of an item (see Figure 3, left), only the integration of the components of this item is improved by the emotion. These components are, on the one hand, the physical properties of the item (its shape, color) but can also be properties associated with the item in memory (e.g., the sound associated with the word). When emotion is introduced via the context (see Figure 3, right), only the integration of contextual features (such as the spatial position) with the item is enhanced by the emotion.

Representation of the effects of emotion on the integration of the different properties of an event within the memory trace of this event.

Conclusion

To conclude, despite its limitations and the need for further studies in order to investigate and strengthen our claims, the current study has direct theoretical implications, mainly in the way the impact of emotion on memory as a function of its source is described. From a more applied point of view, it is clear that our results can be used in cases where cognitive deficits are associated with integration difficulties. For example, the associative deficit hypothesis proposed by Naveh-Benjamin, (2000) in the field of aging states that, with age, it becomes more difficult to create associations between an item of information and its contextual features, or in other words, to create a strong link between the different components of the memory trace related to the information. The present study opens up the possibility of using emotion to try to remedy this, and therefore represents a stepping stone for further research into normal and pathological aging.

Appendices

Questions used in the rating of the stimli and the Likert scales for each answer.

p values obtained for the t-test used to compare the different subsets of words.