Less is more: Removing a modality of an expected olfactory-visual stimulation enhances brain activation.

In recent years, multisensory integration of visual and olfactory stimuli has extensively been explored resulting in the identification of responsible brain areas. As the experimental designs of previous research often include alternating presentations of unimodal and bimodal stimuli, the conditions cannot be regarded as completely independent. This could lead to effects of an expected but surprisingly missing sensory modality. In our experiment, we used a common functional magnetic resonance imaging (fMRI) study design with alternating strong unimodal and bimodal olfactory-visual food stimuli, in addition to a slight overhang of the bimodal stimuli in an effort to examine the effects of removing a visual or olfactory congruent stimulus for older people (41-83 years). Our results suggest that the processing of olfactory and visual stimuli stays intact over a wide age-range and that the utilization of strong stimuli does not lead to superadditive multisensory integration in accordance with the principle of inverse effectiveness. However, our results demonstrate that the removal of a stimulus modality leads to an activation of additional brain areas. For example, when the visual stimulus modality is missing, the right posterior superior temporal gyrus shows higher activation, whereas the removal of the olfactory stimulus modality leads to higher activation in the amygdala/hippocampus and the postcentral gyrus. These brain areas are related to attention, memory, and the search of the missing stimulus. Consequently, careful attention must be paid to the design of a valid, multimodal sensory experiment while also controlling for cognitive expectancy effects that might confound multimodal results.

INTRODUCTION

Humans rely on their senses for perceiving their everyday environment. As our environment is multisensory in nature, several stimuli reach our brain at once, leading to a massive amount of information that needs to be processed simultaneously. Not only does the brain need to process stimuli that belong to different sources, information from different modalities belonging to the same source need to be processed as well. In a food‐related context, visual and olfactory stimuli play an important role in processing food stimuli by influencing each other and together help form a holistic perception (Morrot et al., 2001; Seigneuric et al., 2010; Spence et al., 2010; Zampini et al., 2007; Zellner & Kautz, 1990; W. Zhou et al., 2010). The process of combining different sensory inputs and creating a holistic percept is called multisensory integration. Multisensory integration effects are of special importance for older people as they can compensate for the decline of separate modalities (Laurienti et al., 2006; Mahoney et al., 2011).

Several brain areas have been identified as multisensory brain regions that are responsible for olfactory‐visual integration. The left rostromedial orbitofrontal cortex (OFC), the posterior intraparietal sulcus (IPS), as well as the right superior temporal sulcus (STS), and posterior cingulate cortex were identified when the stimuli presented in an experiment were congruent (Gottfried & Dolan, 2003). Several other regions, including the inferior parietal sulcus (IPS), the amygdala, or the inferior frontal gyrus (IFG), were established as olfactory‐visual integration areas (Ripp et al., 2018; Sijben et al., 2018; Stickel et al., 2019). However, in functional magnetic resonance imaging (fMRI) studies, multimodal brain areas are not always activated or detected (Stevenson et al., 2014): A common criterion to identify these regions is superadditivity. This criterion assumes that the bimodal activation cannot be explained by the addition of the unimodal activations alone and that, therefore, integration must take place (Calvert, 2001; Stevenson et al., 2007). Besides the reason, that whole‐brain SPM is often not sensitive enough to detect superadditivity (Beauchamp, 2005; Stevenson et al., 2007), not in all bimodal stimulation conditions integration processes are equally strong. This can be traced back to the principle of inverse effectiveness which states that the more effective an individual stimulus is, the less effective the benefit is of combining them (Meredith & Stein, 1983), thus leading to mere additive or subadditive enhancement and ultimately not to superadditive processing (Stein & Stanford, 2008).

Not only does the stimulus‐dependent effectiveness of multisensory integration lead to complex, multimodal processes in the brain. Also, top‐down effects such as attention or task‐learning modulate bottom‐up processes (Choi, Lee, & Lee, 2018). For example, a visual stimulus that was previously presented with an odor can lead to activations in olfactory brain areas even if no odor is presented anymore (Gottfried, Smith, Rugg, & Dolan, 2004; Karunanayaka et al., 2015). Previous priming in the study or priming due to human experience could result in formed expectations and memory effects (Smeets & Dijksterhuis, 2014) based on a specified network of brain activations.

On the other hand, top‐down or cognitive effects could also occur if a stimulus is expected to be multimodal (e.g., bimodal olfactory‐visual stimulation) but then one modality is missing. This fact is supported by behavioral studies examining modality‐switching effects or violated modality expectations (Spence et al., 2001; Turatto et al., 2002). Examining the underlying neural network, an auditory–visual PET study indicates that a missing visual cue compared to a bimodal stimulation leads to a higher level of brain activation in the middle prefrontal gyrus and superior temporal gyrus (Kang et al., 2006). Additionally, when tasting something unexpected, (e.g., when a vocal cue suggests a different taste), brain areas related to taste, reward, and attention demonstrate significantly higher activations in comparison to when an expected taste is presented. Significant deactivation in the fusiform gyrus has been observed when the taste stimulus is unexpected versus expected (Veldhuizen et al., 2011). The authors of studies examining violated expectations of pain found activations in brain regions that redirect attention (Colloca et al., 2019) as well as in brain areas related to memory and associative learning (Zeidan et al., 2015). Similar to violated expectations are prediction errors that mostly cause activations in the striatum, insula, thalamus, and fronto‐medial structures (D'Astolfo & Rief, 2017; Garrison et al., 2013). Further, with studies using the oddball paradigm it was demonstrated that the fronto‐parietal network, which also plays a role in working memory, is activated for unexpected visual conditions (Mccarthy et al., 1997). Interestingly, Stevens et al. (2000) found a high degree of overlapping areas, particularly in the middle frontal gyrus and anterior cingulate, when comparing auditory and visual oddball tasks. The cingulate cortex, which is also linked to visual or auditory oddball paradigms as well as prediction errors, was shown to be activated in inattentive odorant detection due to a discrepant olfactory event (Sabri et al., 2005). Therefore, the aforementioned brain areas seem to work in a modality‐independent manner (Stevens et al., 2000).

Nevertheless, studies examining brain activations concerning missing odorous stimuli are lacking. In olfactory‐visual brain imaging studies the experimental designs often include alternating presentations of unimodal and bimodal stimuli. Therefore, the conditions cannot be regarded as completely independent from each other which could lead to effects of an expected but surprisingly missing sensory modality and to the activation of a neural network comparable to the above‐mentioned. To the best of our knowledge, in previous studies regarding olfactory and visual stimulation only multisensory integration between those modalities regarding superadditivity were analyzed (Gottfried & Dolan, 2003; Stickel et al., 2019). This subsequently raises the question, whether a similar study design would also elicit similar brain activations if the stimulation is unimodal and not—as maybe expected—bimodal.

In an effort to answer this question, the present study therefore focused not on the interaction effect of bimodal olfactory‐visual stimuli but on the effects of the removal of a stimulus. To execute this, we used a slightly adapted, but commonly used, fMRI study design with alternating unimodal and bimodal olfactory and / or visual food‐related congruent stimulation with older subjects. To be able to focus on the effects of a missing stimulus rather than multisensory integration, we used clear and strong stimuli that were established during pilot studies. We applied a slight overhang of bimodal conditions to render the expectation to see and smell something (unimodal visual: unimodal olfactory: bimodal = 1:1: 1.5). To exclude age‐related differences in stimulus processing, we scanned middle‐aged (41–64 years) as well as older participants (75–83 years). The aim of our study was to analyze whether a common multisensory fMRI study design with alternating bimodal and unimodal olfactory‐visual stimuli elicits brain activations in cognitive areas if a stimulus is missing. We hypothesized that we would detect brain activations related to top‐down effects if a stimulus is missing.

MATERIAL AND METHODS

Participants

In total, 33 healthy, well phenotyped Caucasian participants of the enable‐cohort (Brandl et al., 2020) were included in our study. Twenty of them were from the middle‐aged group (mean age 52.2 years, SD 6.1 years, age range: 41 to 64 years, 14 females), 13 of the subjects were from the older‐aged group (mean age 77.6 years, SD 2.5 years, age range: 75 to 83 years, 4 females). All participants were healthy (see exclusion criteria in the Supporting Information 1.1), right‐handed (verified using the handedness questionnaire; Oldfield, 1971), exhibited no signs of depression (BDI scores between 0 and 16, mean 3.88 [SD 4.63]; Beck et al., 1996), were tested for their cognitive performance (participants achieved a MoCA score higher than 26 or a score of 25 that is just below the threshold for a normal result, mean 28 [SD 2]; Nasreddine et al., 2005), and had normal olfactory function (middle‐aged group: MONEX‐40 score > 26, old group: MONEX‐40 score ≥ 20, mean 32.8 [SD 2.6]; Freiherr et al., 2012). Additional data of the comprehensive phenotyping procedure for those subjects were recently published (Brandl et al., 2020).

The study was approved by the ethics committee of the Medical Faculty of the Technical University of Munich (TUM) and has been carried out in accordance with the Declaration of Helsinki. All participants provided written informed consent prior to study participation.

Stimulation

In a pilot study, a food preference questionnaire was answered by 375 participants. Participants were asked to rate 96 food items based on the liking of the food item on a 7‐point scale. Due to the results of this questionnaire, we selected the two most attractive (apple and strawberry), the two most aversive (liver and liquorice), and the two most ambiguous food items (lamb and shrimp). Here, ambiguous food items are food items which cover a large range on the liking scale—meaning that certain participants preferred the food item while other participants disliked the food item. In addition, prior to the functional imaging study, the participants were asked to rate the six selected odors on an 11‐point visual analog scale based on their pleasantness and intensity. According to the individual responses, the odors which were rated as the most pleasant and unpleasant were selected for each participant as well as the ambiguous odor which had a hedonic rating between these two odors while also having similar intensity ratings. We were thereby able to optimize the study design regarding pleasantness of the presented stimuli by accounting for individual odor preferences, as an average odor hedonic rating does not necessarily match individual impressions. The ambiguous odor was used to create an overhang of the bimodal conditions by keeping the ratio of unpleasant and pleasant stimuli across all participants. The odors were biomimetic aroma recombinates (Dunkel et al., 2014)—apple (Steinhaus et al., 2006), strawberry (Schieberle & Hofmann, 1997), liver (Straßer & Schieberle, 2014), liquorice (Wagner et al., 2016), lamb (Rota & Schieberle, 2005), and shrimp (Mall & Schieberle, 2017). All odors were perceived strong in intensity (means across groups on a 0–10 scale, [SD]): apple 8.93 [1.19], strawberry 6.72 [2.52], liver 6.91 [2.19], liquorice 5.84 [2.52], lamb 6.55 [2.71], and shrimp 5.00 [2.96]). Odor intensity did not differ significantly between our age groups (F(1, 21) = 1.652, p = .213) as well as no interaction effect between odor intensity and age group (F(5, 105) = 0.301, p = .911) was observed (see Supporting Information 1.2). In addition, participants had to choose a matching picture out of six presented pictures for each odor. This decision was up to personal preferences and experiences as all the pictures represented the same food item as the simultaneously presented odorant. This additionally ensured that bimodal stimulation was congruent and, overall, we achieved an individually adapted study design with equal conditions for both age groups.

Olfactory stimulation during the pilot and the fMRI study was conducted using an olfactometer (Lundström et al., 2010) with a constant airflow of 2 L/min. For the visual stimulation during the fMRI sessions, pictures were projected onto a screen in the scanning room, which could be seen by the participant through a mirror attached to the head coil. A standardized and precise stimulus presentation in addition to recording of the participant's feedback, scanner triggers, and onset times was achieved using the software E‐Prime 2.0 (Psychology Software Tools, Pittsburgh, PA).

Experimental design

The functional imaging study consisted of two scanning sessions with a duration of 20 min each, which were separated by a 6‐min resting session. This allowed the participant to relax and prevented odor adaptation and habituation processes. Afterwards, two anatomical scans, namely, a FLAIR and an MPRAGE, were acquired. Before the start of the experiment, we informed the participants that we would present them odors, which were sometimes presented together with a picture. Participants were asked to inhale through their noses as soon as they saw a yellow cross and to rate the pleasantness of the odor after each presentation. When the participants did not detect an odor, they were asked to provide a neutral rating.

Figure 1 shows the experimental design of the two functional imaging sessions. During the stimulation, subjects perceived one out of three odors (attractive, aversive, and ambiguous) or no odor (distilled water), paired either with or without the selected congruent picture for 3 s. The ambiguous stimulation was only presented in the bimodal condition, resulting in eight different conditions (Figure 2, top). Each condition was presented eight times during one scanning session, leading to 16 presentations of each condition in total. Therefore, the study consisted of 32 unimodal visual (16 attractive, 16 aversive), 32 unimodal olfactory (16 attractive, 16 aversive), 48 bimodal (16 attractive, 16 aversive, 16 ambiguous), and 16 baseline presentations. Since the ambiguous stimulation only existed as a bimodal stimulus combination, an overhang of bimodal conditions was created that possibly caused certain expectations. Conditions in a session were randomly ordered. A yellow cross with a duration of 300 ms signaled the onset of the stimulus. A visual analog scale was presented immediately after the stimulus presentation to rate the pleasantness of the odor. Participants used the fingers of their right hand and a feedback device. Participants had 4.5 s to provide their feedback. After the stimulus evaluation phase, a rest phase for an average of 10 s (jittered between 8–12 s) appeared, during which a black fixation cross was shown.

FIGURE 1

Experimental design. A yellow fixation cross (duration 300 ms) signals the arrival of sensory stimulation. For 3 s, participants were presented an odor or no odor and saw or did not saw a congruent picture (for the different conditions, see Figure 2). After stimulus presentation, participants had 4.5 s to rate odor pleasantness on a visual analog scale. A black fixation cross with a jittered duration of 8–12 s separated one trial from the next trial

FIGURE 2

Experimental stimulus combinations and resulting conditions. Each combination of attractive vs. aversive with visual vs. olfactory was presented eight times per session. In addition, an ambiguous bimodal combination was also presented eight times per session. This combination was not included in our analysis, but severe to evoke the expectation of a bimodal representation in the subject without familiarization to a special combination. Resulting conditions combined aversive and attractive stimuli to get pleasantness‐independent results

Breathing behavior

To examine, whether the different conditions influence the breathing behavior of the participants, we conducted a pilot study with 16 participants of our cohort. Participants were presented all six food items in a unimodal visual, unimodal olfactory, and bimodal combination. In addition, the baseline condition was presented. The MLT1133 Piezo Respiratory Belt Transducer (MR) (ADInstruments, Oxford, UK) was used to measure changes in the thoracic or abdominal circumference elicited by respiration. Similar to the fMRI study, a yellow fixation cross (duration of 300 ms) signaled a new condition. Participants were asked to breathe in, as soon as they saw the crosshair. Each condition was presented in a randomized order only once for 3000 ms. Between the end of one condition and the next yellow fixation cross was a pause of more than 10 s. No statistical differences for the inhalation time (T i) and the total breath duration (T tot) between the different conditions could be established (RMANOVA with corrected degrees of freedom using Huyn–Feldt estimates of sphericity: T i: F(9.62, 221.36) = 0.72, p = .71, η 2 = 0.03; T tot: F(13.69, 314.93) = 0.87, p = .60, η 2 = 0.04). We therefore can conclude that breathing patterns were independent of the presented condition. Therefore, breathing behavior was considered unlikely to influence differences in the brain activation due to the different stimulus presentations.

fMRI data acquisition and preprocessing

Magnetic resonance images were obtained using a 3‐T Siemens Verio MRI scanner at TUM rechts der Isar Hospital. Blood oxygenation level dependent (BOLD) images were acquired using T2*‐weighted echo‐planar imaging (EPI) sequences with a repetition time (TR) of 2450 ms, an echo time (TE) of 30 ms, a flip angle of 80°, a field of view (FoV) of 192 mm, a pixel size of 2.46 mm × 2.46 mm, slice thickness of 2.5 mm, and a matrix size of 78 mm × 78 mm; 40 slices were acquired per volume in an interleaved manner. The slice package was tilted 20° upwards relative to an axial slice through the anterior/posterior commissure (AC/PC). Anatomical scans were obtained using a high‐resolution three‐dimensional T1‐weighted gradient sequence (MPRAGE) with an inversion time (TI) of 900 ms, an echo time (TE) of 2.98 ms, a repetition time (TR) of 2.3 ms, a flip angel of 9°, a field of view (FoV) of 196 mm, pixel size of 1 mm × 1 mm and a slice thickness of 1.2 mm.

DICOM images were converted to NIFTI files using the dcm2niix converter (Li et al., 2016). Functional imaging analyses were carried out using statistical parametric mapping SPM12 (Welcome Trust Centre for neuroimaging, London, UK, http://www.fil.ion.ucl.ac.uk/spm) implemented in Matlab R2020a (The MathWorks, Inc., Natick, Massachusetts, USA). After manual adjustment of the origin of all images to the anterior commissure, the images were slice‐timed to the middle image slice and realigned to correct for head movements. The anatomical image was coregistered to the mean functional image and segmented using all six SPM12‐delivered tissue probability maps. The resulting forward deformation maps were used for normalization of all functional images and the anatomical image to the MNI (Montreal Neurological Institute) reference space. Finally, the functional images were spatially smoothed with 8 × 8 × 8 mm full‐width‐at‐half maximum (FWHM) Gaussian kernel.

Statistical analyses

Behavioral analysis

After each stimulation, participants were immediately asked to rate the pleasantness of the odor on a 100‐mm visual analog scale. Each participant therefore rated the odor of each condition 16 times. We eliminated ratings with a reaction time faster than 200 ms, as these could be the results of fast guesses (Whelan, 2008) and calculated the mean values for each condition per participant. We tested for main and interaction effects with a three‐way mixed ANOVA (Chambers et al., 1992). To further test for significant differences between the ratings, we used paired t‐tests with Bonferroni correction. Statistical analyses were performed using RStudio (RStudio, PBC, Boston, Massachusetts, US, www.rstudio.com) and Python 3.7.1 (python software foundation, Wilmington, Delaware, US https://www.python.org/).

Functional imaging data

As we were interested in the difference between unimodal and bimodal (olfactory and visual) stimulation independent of the pleasantness of the stimuli, we presented the participants aversive as well as attractive food‐related odors and images but regarded these for the analysis as a combined condition. This way, we were able to generalize the odor and/or the picture to the whole bandwidth of food‐related stimuli. In total, there were 32 unimodal visual, 32 unimodal olfactory, and 16 baseline presentations. Additionally, 32 bimodal stimulations (attractive and aversive) were included in the analysis. The ambiguous bimodal stimulation was not included in the analysis, as no unimodal ambiguous stimuli were presented. However, the use of this stimulation led to an overhang of bimodal conditions (48 in total), forming higher expectations in participants to smell and see something concordantly. As a result, we had four different conditions: the unimodal visual condition (V), the unimodal olfactory condition (O), the bimodal condition (OV), and the baseline (BL). During the baseline condition, neither an odor nor an image was presented. Therefore, in addition to be regarded as a control condition, the baseline can be regarded as a condition in which two stimulus modalities are missing.

Four regressors of interest, corresponding to the onset times of the different conditions (duration times of 0) were convolved with a canonical hemodynamic response function (HRF). Subject‐specific movement parameters and a high pass filter with cut‐off at 128 s were included into a general linear model (GLM) using an event‐related analysis procedure.

At the first level, we created basic contrasts of all four conditions ([1], [0 1], [0 0 1], and [0 0 0 1]). We included those contrasts in a factorial design model with the two factors group (two levels: middle‐aged and old) and modality (four conditions as levels). Regions of interests (ROI) were extracted directly from our contrasts or the AAL3 atlas (Rolls et al., 2020) using the SPM toolbox marsbar version 0.44 (Brett et al., 2002). Reported significances for beta estimates were calculated using marsbar toolbox. Coordinates are reported in MNI space (Montreal Neurological Institute). Activations that survived a threshold of p < .05 corrected for whole‐brain volume are reported. The conjunction analysis is reported at p < .001 uncorrected for whole‐brain comparison. To investigate effects between unimodal or baseline stimulations against bimodal stimulation, we used a cluster threshold of 10 to reduce false positive activations.

RESULTS

Behavioral results

A three‐way analysis of variance on the pleasantness ratings yielded significant main effects for pleasantness category (attractive, aversive) (F(1, 31) = 150.24, p < .001) and for modality (olfactory, bimodal) (F(1, 31) = 10.42, p = .0029) but not for age group (middle‐aged, old) (F(1, 31) = 0.045, p = .83). The pleasantness ratings during the experiment confirm that the chosen odors for each participant were indeed attractive or aversive in the unimodal odor condition (O) and the bimodal olfactory‐visual condition (OV) (atO: 8.96 [1.17], atOV: 9.03 [1.22], avO: 4.78 [1.57], avOV: 4.31 [1.71]; mean [SD]). In addition, the ambiguous bimodal condition showed the highest variance (3.04 [VAR]) across all participants (amOV: 5.32 ± 1.74). As we only asked to rank the pleasantness of the odor, neutral values for the unimodal visual combinations (atV: 6.23 [0.44], avV: 5.98 [0.43]) as well as for the control condition (BL: 6.09 [0.27]) validate our sensory stimulation. Significant differences exist among the scores for the bimodal conditions (avOV and atOV: t(32) = −11.27, p = 1.12e−12; avOV and amOV: t(32) = −2.75, p = .0096; amOV and atOV: t(32) = −10.029, p = 2.11e−11) (Figure 3).

FIGURE 3

Boxplot of pleasantness ratings of the presented odors collapsed across all participants

No significant interaction effect existed for age group × modality × pleasantness (F(1, 31) = 0.72, p = .40). However, the interactions between the age group and the pleasantness category (F(1, 31) = 6.44, p =.016) and between the modality and the pleasantness category (F(1, 31) = 9.43, p = .004) were significant, whereas the interaction between the age group and the modality was not significant (F(1, 31) = 1.40, p = .25). Results indicate that the interaction between the pleasantness category and the modality is disordinal and occurs independently from the age group. Directly comparing bimodal with unimodal olfactory conditions revealed a reinforcement of the aversiveness, implying that aversive odors were perceived significantly less pleasant if presented with a congruent image (t(32) = −3.54, p = .001). This was not significant for attractive odors (t(32) = 0.96, p =.34). The effect of bimodal versus unimodal stimulation did not significantly differ between the age groups. In addition, above‐mentioned results revealed that older people rate perceived odors less extreme concerning their pleasantness (Figure 4).

FIGURE 4

Interaction plots for three respective factors modality, pleasantness category and age group. ns: nonsignificant, *p < .05, **p < .01

fMRI results

Age effects and superadditivity

Main effects of unisensory stimulations together with contrasts of activated brain areas compared with the baseline condition confirmed correct sensory stimulation (see Supporting Information). No significant differences between the age groups were established which shows that the behavior stays intact over our range of ages and allows us to combine the age groups for further analyses. Additionally, no superadditive brain activation was detected. Similarly, when comparing bimodal against unimodal stimulation, no further brain areas other than those corresponding to the added modality were activated significantly higher in the bimodal stimulation than in the unimodal stimulations or in the unimodal compared to the BL condition (e.g., if visual stimulation is added to olfactory stimulation, visual brain areas will be activated in addition to previous activated olfactory brain areas but no additional brain areas demonstrated significantly higher activations). Detailed results can be found in the Supporting Information.

Removal of a stimulus

In contrast to previous studies examining the multisensory integration processes, this study considered the effects of removing a stimulus, that is, not presenting an expected stimulus. Therefore, we regarded brain areas with a higher activation during unimodal presentation compared with bimodal presentation (see Figure 5 and Table 1).

FIGURE 5

Significant brain activations (p < 0.05 FWE whole‐brain comparison, extended cluster threshold of k = 10) if unimodal and baseline conditions are compared with the bimodal condition. A missing olfactory stimulus (green) leads to activations in the left hemisphere, especially in postcentral (cluster (C) 3) and superior temporal gyrus (C6) and the hippocampus (C8). In addition, the cuneus (C2) was significantly activated. A missing olfactory (green) as well as a missing visual stimulus (red) lead to activations in brain regions adjacent to the right supramarginal and angular gyri (C4 and C5). If both stimuli were missing (baseline condition, blue), the BOLD signal in left hemisphere regions was significantly enhanced. Clusters labeled according to the AAL3 atlas (Rolls et al., 2020): C1 L. inferior parietal sulcus, C2 L. cuneus, C3 L. postcentral gyrus, C4 R. rolandic operculum adjacent to r. angular and supramarginal gyrus, C5 R superior temporal gyrus adjacent to R. angular and supramarginal gyrus, C6 L. superior temporal gyrus, C7 L. parahippocampal gyrus, C8 L. hippocampus (L. middle temporal gyrus). For detailed information, see Table 1

TABLE 1

Regions that are less activated in bimodal stimulation compared with unimodal or baseline condition

		MNI coordinates (mm)				Part of cluster
Anatomical region	Cluster size k	X	Y	Z	Peak Z
V > OV
Left amygdala	54	−27	−4	−25	5.59	8 (6)
Left middle temporal gyrus		−45	−4	−28	5.28	8 (6)
Left fusiform gyrus		−36	−22	−19	4.97	8 (6)
Left hippocampus						8 (6)
Left postcentral gyrus	64	−51	−10	23	5.57	3
Left rolandic operculum		−57	−7	11	4.89	3
Left postcentral gyrus		−42	−13	35	4.89	3
Left parahippocampal gyrus	26	−24	−31	−19	5.40	7
Left fusiform gyrus		−36	−31	−16	4.87	7
Left superior temporal gyrus	47	−45	−10	−13	5.25	6
Left superior temporal gyrus		−51	5	−7	5.09	6
Left cuneus	26	−12	−82	32	5.21	2
Left inferior parietal gyrus	22	−57	−43	35	5.06	1
Right rolandic operculum	11	36	−34	20	5.02	4
O > OV
Right superior temporal gyrus	26	63	−49	17	6.51	5
BL > OV
Left superior temporal gyrus	42	−42	−13	−16	5.46	6
Left middle temporal gyrus		−48	−4	−16	4.89	6
Left inferior parietal gyrus	13	−60	−46	38	5.17	1
Left postcentral gyrus	10	−33	−16	32	4.96	3

Note: We present location, coordinates as well as the Z and p values for local maxima within the activated clusters (according to AAL3's local maxima labelling). For clusters with more than one local maximum, the anatomical region in which most of the cluster is located (according to AAL3's cluster labelling) is written in bold. The column on the right partitions the brain areas in clusters as in Figure 5 and clusters for the ROI analysis. As cluster 6 and 8 are merged for ROI analysis, the ROI number is given in brackets for these cases. p < 0.05 FWE‐corrected for whole‐brain comparison with an extended cluster threshold of k = 10.

When the unimodal visual condition was compared with the bimodal condition ([V > OV]), the rCBF significantly increased in the left inferior temporal gyrus, left postcentral gyrus, left superior temporal gyrus, left cuneus, left parahippocampal gyrus and the left hippocampus / amygdala (for coordinates and statistical values, see Table 1). In the right hemisphere, only the rolandic operculum adjacent to the right supramarginal and angular gyrus was significantly activated. In contrast, if we investigate brain areas that are more activated when the visual stimulus was missing ([O > OV]), a significant difference (p < 0.05 FWE) of the BOLD signal was only observed in the posterior part of the right superior temporal gyrus adjacent to the right supramarginal and angular gyrus.

Considering the nature of the baseline, the baseline cannot only be regarded as a control condition but it can also be regarded as a true condition, in which not only one modality but both are missing compared to the bimodal condition. Therefore, the contrast [BL > OV] can detect brain areas that are activated if the cues of both modalities are missing. During the BL condition, significant activations in the left postcentral gyrus, the left inferior parietal gyrus adjacent to the supramarginal gyrus, and the left superior temporal gyrus adjacent to the hippocampus were observed.

A conjunction analysis of the three contrasts [O > OV], [V > OV], and [BL > OV] revealed significant activations (p < 0.001 uncorrected, k = 10) in the middle and superior temporal gyri including the right angular gyrus and right rolandic operculum as well as hippocampal regions (Table 2). The cluster around the right angular gyrus overlaps with cluster 5, which is the cluster around the right rolandic operculum with cluster 4. The cluster with its maximum in the left middle temporal gyrus and hippocampus covers parts of clusters 6 and 8.

TABLE 2

Regions that are significantly activated in a conjunction analysis with the contrasts [O > OV], [V > OV], and [BL > OV]

		MNI coordinates (mm)
Anatomical region	Cluster size k	X	Y	Z	Peak Z
[O > OV] ∩ [V > OV] ∩ [BL > OV]
Left middle temporal gyrus	45	−45	−1	−16	4.10
Left middle temporal gyrus		−45	−13	−22	3.69
Right angular gyrus	48	51	−52	23	4.08
Right superior temporal gyrus		60	−49	17	3.40
Right rolandic operculum	12	39	−37	23	3.79
Right hippocampus	11	27	−10	−19	3.57
Left hippocampus	12	−36	−13	−10	3.57
Right parahippocampal gyrus	13	18	−19	−16	3.55
Right cerebellum		12	−25	−22	3.21
Right middle temporal gyrus	11	51	−34	−1	3.31

Note: We present location, coordinates as well as the Z and p values for local maxima (according to AAL3's local maxima labelling) within activated clusters. p < 0.001 uncorrected for whole‐brain comparison with an extended cluster threshold of k = 10.

ROI analysis for established clusters

To get further insight into special brain areas which are activated more in the unimodal conditions than in the bimodal conditions, we directly compared the brain activation for a whole cluster instead of doing voxel‐wise comparisons between the different conditions using multiple t‐tests. Therefore, we created clusters for a ROI‐analysis for each of the previously established brain areas (see Figure 6). As clusters 6 and 8 are overlapping (see Figure 5 and Table 1), we combined them for the ROI analysis resulting in seven clusters in total. Significant main effects of vision and olfaction as well as for contrasts between the conditions are presented in Table 3. As we compared different clusters as well as different contrasts in these clusters, we utilized a rather conservative threshold of p < 0.001 for significant differences. With regard to the main effects, all clusters with the exception of cluster 5 showed significant effects for the missing olfactory cue. Cluster 5, however, had a significant main effect for the missing visual cue. All clusters with the exception of cluster 2 were activated significantly more regarding the contrast [BL vs. OV]. All clusters showed significant differences for [V vs. OV] and clusters 4, 5, and 6 showed significant differences for the contrasts [O vs. OV]. If the non‐bimodal conditions are considered, there were no differences for the contrasts [V vs. BL]. All brain activations with the exception of cluster 5 showed significances comparing [O vs. V] and only cluster 1 comparing [O vs. BL].

FIGURE 6

Beta estimates of clusters of brain areas that were significantly more activated in unimodal or baseline conditions than in the bimodal condition. V, O, OV, and BL are the conditions

TABLE 3

p values of statistical significance test for contrast in ROIs

	Main effects		Condition versus bimodal			Between conditions
Cluster	Missing visual cue	Missing olfactory cue	BL vs. OV	V vs. OV	O vs. OV	O vs. V	V vs. BL	O vs. BL
	[−1 1 –1 1]	[1 –1 −1 1]	[0 0 1 –1]	[1 0 –1 0]	[0 1 –1 0]	[1 –1 0 0]	[1 0 0 –1]	[0 1 0 –1]
1	0.0789 ns	9.41∙10−11 ***	5.78∙10−9 ***	3.91∙10−8 ***	0.0170*	2.59∙10−5 ***	0.946 ns	6.72∙10−5 ***
2	0.533 ns	1.42∙10−5 ***	0.00500**	1.21∙10−8 ***	0.00750**	5.64∙10−5 ***	0.0130*	0.233 ns
3	0.709 ns	7.09∙10−9 ***	3.10∙10−5 ***	7.31∙10−10 ***	0.114 ns	8.88∙10−7 ***	0.0430*	0.00160**
4	0.153 ns	2.96∙10−9 ***	1.22∙10−6 ***	3.22∙10−8 ***	5.45∙10−4 ***	4.63∙10−4 ***	0.452 ns	0.0343*
5	1.37∙10−6 ***	0.00860**	3.71∙10−7 ***	3.22∙10−4 ***	2.64∙10−9 ***	0.0817 ns	0.0827 ns	0.804 ns
6	0.262 ns	2.31∙10−9 ***	9.48∙10−7 ***	2.67∙10−11 ***	9.55∙10−4 ***	3.81∙10−5 ***	0.274 ns	0.00520**
7	0.432 ns	1.28∙10−7 ***	1.20∙10−4 ***	1.40∙10−9 ***	0.0658 ns	2.91∙10−6 ***	0.0188*	0.0179*

Note: Clusters are built based on significant activations in the brain if comparing unimodal or baseline conditions against the bimodal condition. [. . . .]: contrasts for [V O OV BL], ns: non‐significant.

p < 0.05.

p < 0.01.

p < 0.001 (uncorrected).

DISCUSSION

The aim of this study was to determine whether a common, multisensory fMRI study design with alternating bimodal and unimodal olfactory‐visual stimuli elicits activations in cognitive brain areas when a stimulus is missing. As such, our study adds to the previously conducted multisensory integration studies by involving bimodal olfactory‐visual stimulation (Gottfried & Dolan, 2003; Ripp et al., 2018; Stickel et al., 2019) but we analyzed our data using a different perspective. To enhance the effects of a missing expected stimulus, we added bimodal ambiguous stimulations to create an overhang of bimodal conditions. This triggered expectations and cognitive preparation in participants to see and smell something. As odor preferences are subjective (Herz, 2006), group average hedonic ratings do not necessarily match individual ratings. Therefore, we used individually adapted attractive as well as aversive congruent stimuli. Using attractive and aversive stimuli enabled pleasantness‐independent results. The degree of attractiveness as well as the strong intensities for each odor were confirmed using behavioral ratings. During our study, participants attended to a yellow fixation cross, signaling that they needed to inhale and prepare for the possible arrival of the stimuli. Participants knew the study design and were aware of the different conditions. As multisensory integration effects are especially important for older people as they can compensate for the decline of certain modalities (Laurienti et al., 2006; Mahoney et al., 2011), we examined middle‐aged and older participants. Our results suggest that there are comparable olfactory‐visual processes over a wide range of age (see Supporting Information).

No superadditivity for strong and clear stimuli

As our goal was to focus on the removal of a stimulus and not multisensory integration, we chose clear stimuli that were strong in intensity in accordance with the principle of inverse effectiveness (Meredith & Stein, 1983). Due to the fact that less stringent criteria could reflect linear summations of neurons instead of multisensory integration processes (Calvert, 2001) and in accordance with previous olfactory‐visual studies, we applied superadditivity as a criterion for multisensory integration. No significant effects of superadditivity could be established in contrast to existing olfactory‐visual studies (Gottfried & Dolan, 2003; Stickel et al., 2019). In addition, adding a stimulus ([OV > V] and [OV > O]) did not reveal further brain areas other than those already corresponding to the added sensory modality. Recent reviews focused on the effect of age on multisensory integration effects. The results of these studies provide hints that age increases the integration of multisensory input, even if particularly older studies came to opposite results (de Dieuleveult et al., 2017; Freiherr et al., 2013; Mozolic et al., 2012). Therefore, one could assume that for older people we should detect enhanced multisensory integration processes. However, our results suggest that the activation in brain areas due to the visual stimulus is independent of the presence of an olfactory stimulus and vice versa (interaction effect). That means the strong and clear stimuli elicited no superadditive multisensory integration following the principle of inverse effectiveness. Nevertheless, additive or subadditive multisensory integration could have occurred (Stein & Stanford, 2008) but was not detected by the conservative superadditivity criterion, or activation was not strong enough in additional brain areas. In summary, we can conclude that depending on the design of olfactory‐visual fMRI studies, multisensory superadditive brain areas are not always detected as was shown before for other senses (Beauchamp, 2005; Laurienti et al., 2005; Stevenson et al., 2014).

Cognitive brain areas activate if a stimulus is missing

Unlike other studies which only focused on the interaction effect of bimodal olfactory‐visual stimuli, we took an inverse analysis approach by exploring the effects of a removed stimulus. We therefore performed comparisons between unimodal and bimodal conditions. Significant brain activations were found in several brain areas using a whole‐brain analysis. This enabled us to form clusters and analyze the brain activations in more detail using ROI analyses. Our experiment demonstrated that brain areas related to attention, memory, and search are activated if unimodal or control conditions in contrast to bimodal conditions are presented in a common alternating unimodal and bimodal olfactory‐visual fMRI study design. As participants prepared to see and smell something because they expected this stimulus combination, a unimodal or control contrast violated their expectations. Comparable unexpected taste or pain stimuli were examined in previous fMRI studies. The authors of these studies observed enhanced activation in brain regions related to the corresponding modality, reward, attention (Veldhuizen et al., 2011), sensory discrimination, working memory, and associative learning processes (Colloca et al., 2019; Zeidan et al., 2015). Our data enhance the results of the aforementioned studies by showing those processes for a food‐related olfactory‐visual setting.

Special attention must be paid when removing a visual stimulus. Kang et al. reported inverse contrasts for auditory–visual stimulations using PET imaging (Kang et al., 2006). In accordance with our findings, adding a visual stimulus to an already existing stimulus (here auditory, [AV > A]) did not lead to a significantly higher activation in brain areas. The opposite contrast on the other hand [A > AV] revealed higher activations in the right middle and superior temporal gyrus. This contrast and finding is analog to our contrast [O > OV] that showed significantly higher activation in the right superior temporal gyrus adjacent to the right angular and supramarginal gyrus (cluster 5). In contrast to an argumentation that this is due to inhibitory effects, we argue, that this brain area is more activated because of the missing stimulus: In the whole‐brain analysis (see Table 1), the right angular and supramarginal gyrus showed significantly higher activations if the visual stimulus was removed [O > OV]. This brain area only revealed significant differences for the unimodal olfactory but not for the unimodal visual or baseline condition compared with the bimodal condition. The ROI analyses (see Table 2) established significant differences for these contrasts as well, but not between the nonbimodal conditions. The conjunction analysis revealed significant activation in the right angular gyrus, showing that this region is activated if any stimulus is missing. We therefore assume that the angular gyrus could play an important role if a stimulus is unexpectedly missing, especially if it is a missing visual cue. Previous studies indicate that the right superior temporal gyrus as well as the right supramarginal gyrus are related to visual search behavior (Ellison et al., 2004; Gharabaghi et al., 2006; Ten Brink et al., 2016). In the visual search paradigm, participants search for a particular picture among similar ones. In contrast, in our experiment the visual stimulus is apparently missing. Nevertheless, the previous results could explain, that the angular gyrus, superior temporal gyrus and supramarginal gyrus were activated due to search and attention effects.

For the sake of simplicity, we present our conclusions about the function of further established brain areas due to a combined analysis of whole‐brain activation, ROI, and conjunction analysis before discussing them. The left inferior parietal sulcus (cluster 1), the left postcentral gyrus (cluster 3), and the left parahippocampal and fusiform gyri (cluster 7) showed no significant differences if the visual stimulus is missing ([O > OV]) but if the olfactory stimulus is missing ([V > OV] and [BL > OV]). In addition, no significant differences were observed between the unimodal visual and baseline conditions and the conjunction analysis reveals no significant activations within these clusters. Therefore, these brain areas seem to be responsible for the search of an odor stimulus. The right rolandic operculum adjacent to the right angular and supramarginal gyrus (cluster 4) as well as the left superior to middle temporal gyrus and hippocampus (cluster 6, including cluster 8) showed significant activations in the ROI analyses if any condition with one or two missing modalities is compared with the bimodal condition, although a missing olfactory stimulus leads to greater differences. For whole‐brain analysis, only the conditions [V > OV] and [BL > OV] revealed significances. Parts of clusters 4 and 6 became significant during the conjunction analysis. This leads to the suggestion that these brain areas are activated if an expected stimulus—especially an olfactory one—is missing. The conclusions only taken from our combined results will now be brought together with previous findings.

The inferior parietal lobule (cluster 1 and 4) was shown to be part of a multimodal network for attention processes due to changes in sensory stimulation (Downar et al., 2000; Macaluso et al., 2002) (also see oddball paradigms: Linden et al., 1999; Mccarthy et al., 1997) or unexpected stimuli (Veldhuizen et al., 2020; Zeidan et al., 2015). Other studies have linked this region to violated memory expectations (Colloca et al., 2019; O'Connor et al., 2010; Zandbelt et al., 2013). As such, the search for a missing cue can be interpreted as a shift of attention due to the violation of an expectation or a discrepancy between sensory and cognitive events. Closely linked to these processes are memory processes, especially recollective ones. The lateral parietal cortex including the left postcentral gyrus (cluster 3) is involved in such recollective processes as well as working memory (Vilberg & Rugg, 2008). Therefore, the effects of the missing cue can also be led back to memory processes. In addition, hippocampal regions are involved in several memory processes (Burgess et al., 2002). Moreover, it has been shown that the hippocampus (cluster 6) and parahippocampal regions (cluster 7) are involved in olfactory memory processes (Cerf‐Ducastel & Murphy, 2006; Ergorul & Eichenbaum, 2004; Goodrich‐Hunsaker et al., 2009; Kareken et al., 2003) and the hippocampus in rapid olfactory learning processes (Karunanayaka et al., 2015). Both regions were activated in our study if especially an olfactory stimulus was missing, suggesting that memory retrieval took place and odors were recalled that were surprisingly not detected. As the hippocampal connectivity with the primary olfactory cortex is stronger than with visual areas (Zhou et al., 2021), this could lead to the enhanced activity if the olfactory stimulus was missing in contrast to a missing visual stimulus. However, activation of the parahippocampal gyrus and hippocampus are also related to sniffing (Koritnik et al., 2009; Simonyan et al., 2007). If an olfactory stimulus was missing, participants could increase sniffing behavior to be sure of its nonexistence. This could also lead to enhanced brain activity in nonolfactory conditions. In our study, though, none of the other reported brain areas for sniffing was significantly activated. In addition, our pilot study confirmed similar breathing patterns between the conditions and participants were explicitly asked to breathe in during the yellow fixation cross signaling the arrival of a condition and otherwise breathe normally. Nevertheless, as we did not measure breathing during the fMRI study, we cannot exclude such effects.

Contrasts in the left cuneus (cluster 2) were significant if the unimodal visual condition was compared with the bimodal or unimodal olfactory condition. The unimodal visual condition showed the highest activation of all contrasts. No parts of this cluster showed significant activations in the conjunction analysis. The cuneus is part of the visual system and has been linked to working memory tasks (Lagopoulos et al., 2007; Michels et al., 2008). In addition, left cuneus activity was found in visual search tasks and was related to a shift of attention (Makino et al., 2004). The connection to our findings must be explored in further studies, as our results suggest that there was a higher activity in conditions where no visual search occurred, and no visual working memory had to be activated.

Overall, our results indicate that if participants receive a clear visual cue together with a clear olfactory stimulus that is high in intensity, brain areas corresponding to these modalities are activated. If on the other hand the condition is not bimodal, brain areas related to memory, search and attention are activated. This could be due to an unfulfilled expectation, the search of the missing stimulus, or an attention shift (see Figure 7). In accordance with findings from oddball studies (Sabri et al., 2005; Stevens et al., 2000), the search systems we established for olfactory and visual stimuli partially overlap. However, the strength of the activation differs in these regions. In addition, more brain areas are activated if the missing stimulus is the odor compared to the picture. This could be a result of a dominance of visual over olfactory stimuli—so called visual dominance (Morrot et al., 2001; Parr et al., 2003). If participants do not see anything, they may indeed trust the missing visual cue more than they trust a missing odor. This missing self‐confidence with regards to odors could be due to two facts. First, to detect a smell, humans have to sniff. The sniffing process is a precondition for successful odor perception, and it could be that odors are not always detected due to a missing inhalation. Second, odors are, in contrast to pictures, processed on an unconscious level and information does bypass the thalamus (Doty, 2001). Therefore, it is more complex to realize, interpret, and react to odors. In addition, it was shown that connectivity of memory areas (hippocampus) with the primary olfactory cortex is higher than with visual areas (Zhou et al., 2021). Future research could change the images to more vague visual stimuli to adapt olfactory and visual self‐confidence during the experiment. As such, regions that were now related especially to olfactory search may or may not be activated for visual search. Another point influencing the specific brain activations could be due to the fact that, in our study, participants only rated the odor pleasantness but not pictures. This could have drawn increased attention towards the olfactory stimulation. To specifically distinguish between the olfactory and visual search system, further research is needed.

FIGURE 7

Schematic overview of processes in the brain if an expected stimulus is missing. Due to priming or experience, expectations are generated. If a stimulus modality is missing, this could lead to effects of an unfulfilled expectation, to the search of the missing stimulus or an attention shift. As a result, additional brain areas related to memory, search or attention are activated

CONCLUSION

To study multisensory olfactory‐visual integration, in fMRI studies usually designs with alternating unimodal and bimodal conditions are applied and multisensory brain regions identified. However, these study designs may also elicit expectations in participants about the upcoming stimuli. In our study, we analyzed the effects of a missing olfactory or visual stimulus using an adapted, common fMRI design for older people with clear and strong stimuli in addition to an overhang of bimodal conditions. First, we found out, that the processing of unimodal and bimodal olfactory‐visual stimuli as well as the effects of a missing stimulus stay intact over a wide age‐range. Second, we showed that multisensory superadditive brain areas are not always detected in olfactory‐visual stimulation. In addition, multimodal conditions do not always activate additional brain areas than those directly belonging to the modalities. Third, not only multisensory aspects but also several more can play a crucial role in olfactory‐visual alternating unimodal and bimodal experiments. These can be attributed to memory effects, effects for the search of a stimulus, expectations, and their possible violation according to the next condition in addition to effects related to an attention shift. These aspects can lead to higher cognitive brain activations in conditions with a missing modality, especially if the missing stimulus is the odor. It is important to note, that the normal baseline conditions can also lead to such effects. Consequently, careful attention must be paid to the design of a valid multimodal sensory experiment.

CONFLICT OF INTEREST

None.

ETHICS APPROVAL STATEMENT

The study was approved by the ethics committee of the Medical Faculty of the Technical University of Munich (TUM) and has been carried out in accordance with the Declaration of Helsinki. All participants provided written informed consent prior to study participation.

Data S1: Supporting Information

Click here for additional data file.