Smelling directions: olfaction modulates ambiguous visual motion perception.

Senses of smells are often accompanied by simultaneous visual sensations. Previous studies have documented enhanced olfactory performance with concurrent presence of congruent color- or shape- related visual cues, and facilitated visual object perception when congruent smells are simultaneously present. These visual object-olfaction interactions suggest the existences of couplings between the olfactory pathway and the visual ventral processing stream. However, it is not known if olfaction can modulate visual motion perception, a function that is related to the visual dorsal stream. We tested this possibility by examining the influence of olfactory cues on the perceptions of ambiguous visual motion signals. We showed that, after introducing an association between motion directions and olfactory cues, olfaction could indeed bias ambiguous visual motion perceptions. Our result that olfaction modulates visual motion processing adds to the current knowledge of cross-modal interactions and implies a possible functional linkage between the olfactory system and the visual dorsal pathway.

In our daily life, we are constantly exposed to multiple sensory inputs from different sensory modalities of varying reliability. Yet we could effortless integrate these parallel sensory signals and maintain a unified perception that allows us to interact with the external world. For example, in a buffet dinner we often use not only vision but also olfaction to serve ourselves with the most desirable food. It is well known that visual cues such as colors12345, shapes6, pictorial images78 or even abstract symbols9 could modulate olfactory perceptions. However, surprisingly little is known about whether the reverse modulation - olfactory on visual perception – can occur10.

Recent studies began to reveal that olfaction can influence visual object perception. In one study11, subjects were asked to search for a visual object embedded in a photograph full of objects. It was shown that the visual object was explored faster in the presence of the congruent odor. Similar olfactory modulations on visual object perception have also been discovered during binocular rivalry12 – a well established phenomenon with alternating perceptions of two competing objects that are presented to each eye. The authors have shown that an odorant congruent to one of the competing objects prolongs the time that object is visible and shortens its suppression time. These olfactory modulations on visual object perception can even occur in the absence of conscious visual awareness12 and have lateralization properties, smelling an odor from one nostril modulates the timing of rivalry perceptions more for the objects presented in the contralateral visual field relative to those presented in the ipsilateral visual field13. Those results suggest an early convergence between olfactory system and visual system, specifically the visual ventral stream for object representation14. Yet, it remains to be tested whether visual motion processing, which is associated with the visual dorsal stream, can be modulated by olfactory cues as well. Addressing this question will help us better understand interactions between olfactory and visual system.

It has been shown that the brain integrates multisensory information by weighting each modality in proportion to its reliability, compatible with a statistical optimal scheme151617181920 (e.g., Bayesian or maximum-likelihood estimation212223). In vision-olfaction interactions, since vision undoubtedly prevails olfaction in a normal circumstance, such reliability-based multisensory integration theory asserts that vision should modulate olfactory perception (as reported in numerous previous studies, for a review see10) while olfaction could hardly influence visual perception, unless the reliability of visual information is compromised (e.g., during binocular rivalry1213). For visual motion signals, the reliability refers to the levels of directional ambiguity, which is often manipulated by varying the relative percentages of coherent moving dots and random moving dots2425. Here, we tested whether the perception of ambiguous visual motion direction is influenced by the presence of olfactory cues that are associated with directions by classical conditioning pairing26. We hypothesized that if there is a interaction between visual dorsal pathway and olfactory system, olfactory cues should be able to bias visual motion direction perception. On the other hand, if olfactory cues could not modulate visual motion direction perception, this would indicate that olfaction-vision interaction might be restricted to the visual ventral stream.

Results

Experiment 1: odors can bias ambiguous visual motion perception

First, we would like to see how the perception of ambiguous visual motion directions was influenced by the presence of odorant stimuli, which paired with motion directions on a trial-by-trial basis. Each trial was consisted of two phases: a pairing phase followed by a testing phase. Experimental procedures are schematically illustrated in figure 1. In experiment 1, we paired leftward moving dot patch with the smell of banana and rightward moving dot patch with the smell of fennel (Fig. 2A). At the end of each experiment, behavioral data was sorted according to testing phase. We found that when tested with ambiguous leftward moving dot patch, the overall probability of participants perceived rightward motion was 18.9% ± 3.4% (mean ± SEM) for banana-smell trials, and 35.0% ± 5.6% for fennel-smell trials. The difference was statistically significant (t (13) = 3.7, p = 0.003). A similar pattern held true for the rightward motion perceptions. When tested with ambiguous rightward moving dot patch, the probability of perceiving a rightward motion was 55.6% ± 5.0% for the banana-smell and 68.5% ± 4.7% for the fennel-smell. Again, the percentages of perceiving a rightward motion was significant larger in fennel-smell trials as comparing to banana-smell trials (t (13) = 3.3, p = 0.005). A two-factor repeated measures ANOVA was conducted, including odorants (banana vs. fennel) and motion directions (leftward vs. rightward) as factors. There were significant effects on odorants (F(1,52) = 9.3, p = 0.0037) and motion directions (F(1,52) = 54.5, p < 0.001), but no significant interactions (F(1,52) = 0.11, p = 0.74). Taken together, the results from experiment 1 showed that odorants influenced the perceptions of ambiguous visual motion directions in a way that is consistent with the established odor-motion direction pairing.

Figure 1

Experimental procedures.

During the pairing phase subjects viewed coherently moving dot patches while sniffing the associated banana-like and fennel-like odorants sequentially. During the testing phase they indicated the perceived direction of an ambiguous moving dot patch after a third sniffing. In experiment 1 the banana-smell was paired with the leftward motion while the fennel-smell was paired with the rightward motion; in experiment 2 the reversed pairing was implemented; in experiment 3 both leftward and rightward directions were paired with the same neutral air-smell (from purified water). In experiment 4 the motion direction–odorant pairing was randomized from trial to trial. (The pictures of banana and water were taken by author S.K. The picture of fennel was taken by author T.Z.)

Figure 2

Odorants modulate ambiguous visual motion direction discrimination.

(A): When pairing leftward motion with banana-smell and rightward motion with fennel-smell, subjects exhibited more frequent perceptions of rightward motion in fennel-smell trials relative to banana-smell trials. (B): When pairing leftward motion with fennel-smell and rightward motion with banana-smell, subjects exhibited more frequent perceptions of rightward motion in banana-smell trials relative to fennel-smell trials. (C): When pairing both leftward and rightward motion with the same neutral air-smell the perceived motion directions were not significantly different across conditions. (D): Similar olfactory modulations on visual motion perception when the pairing relationships were randomized. Note only ambiguous rightward motion was used during the testing phase.

Experiment 2: olfactory modulation on visual perception was contingent on the odor-motion direction pairing

To confirm that the olfactory modulation on visual perception was specific to the associated pairing between odor-motion direction instead of a general perceptual and/or choice bias that was incidentally induced by the presence of smells, we conducted experiment 2 with the second group of subjects. This experiment followed the same procedures as experiment 1 except that we reversed the smell-motion direction pairing (Fig. 2B). When leftward motion direction was paired with smell of fennel and rightward motion direction with smell of banana, we found that modulation effects were reversed accordingly. Again, repeated measures ANOVA found significant effects of odorants (F(1,36) = 14.1, p < 0.001) and motion directions (F(1,36) = 127.6, p < 0.001) but no significant interactions either (F(1,36) = 0.39, p = 0.54). Specifically, with the reversed pairing, subjects perceived rightward motion more often in the banana-smell trials relative to the fennel-smell trials (t (9) = 3.1, p = 0.013 for leftward motion signals; and t (9) = 3.0, p = 0.015 for rightward motion signals). This result suggests that the observed modulations on ambiguous visual motion direction perceptions by olfactory cues were contingent upon the pre-established odor-motion direction association.

Experiment 3: control for potential unidentified non-olfactory cues

One possibility is that the biases in direction perception between odors might reflect some artificial un-identified non-olfactory cues in our design. For example, any auditory, tactile, or visual cues might be interfering direction perception and causes the perceptual bias. To address this concern, we recruited an third independent group of subjects and asked them to perform the same task, except that both leftward and rightward motion directions were paired with the same neutral smell of air (from purified water). The subjects were instructed in a similar way as in the previous experiments. As one can see, in figure 2C, for both leftward and rightward ambiguous testing motion signals, there were no significant differences in direction perception between the two neutral smell conditions (t (9) = 0.03, p = 0.97 for leftward motion; and t (9) = 0.59, p = 0.57 for rightward motion). Correspondingly, the two-way ANOVA revealed no significant modulations (F(1,36) = 0.14, p = 0.72) and interactions (F(1,36) = 0.11, p = 0.75) either, but a significant effect on motion directions remained (F(1,36) = 110.1, p < 0.001).

Experiment 4: control for randomization of odorant-motion direction pairing

In previous experiments, the fixed pairing stimuli of smell and visual motion direction were introduced in each trial. This procedure might potentially induce response or decisional bias into participants' direction judgments. To rule out this possibility, we introduced randomized odorant-motion direction pairings on a trial-by-trial basis. During the testing phase, only the ambiguous rightward motion (paired with either banana-like or fennel-like smells) was presented to subjects so that the total number of trials was equal to those in the previous experiments. This design gave rise to 4 total task conditions (2 odorants × 2 pairings). With repeated measures ANOVA treating odorants and pairing as two independent factors, we found significant interactions (F(1,24) = 8.7, p = 0.007) between odorants and pairings, but no significant effects on odorants (F(1,24) = 0.04, p = 0.84) and pairing relationships (F(1,24) = 0.19, p = 0.66). This means olfactory modulation on visual motion perception followed a very specific pattern: the presence of a smell (either banana- or fennel-like) biased participants towards rightward motion perception if it was paired with a rightward motion via associative conditioning, and vice versa for leftward motion (Fig. 2D).

Discussion

The current study demonstrates that olfactory cues can influence ambiguous visual motion direction discrimination. This finding adds to the sparse evidence for the cross-modal interactions that weak senses, such as olfaction, are capable of modulating dominant visual perception. Our result of olfaction-visual motion interactions complements recent studies of olfaction-visual object perception interplays1112. These results indicate that the olfactory information processing and the visual information processing are functionally linked. Olfaction is more likely interacting with vision at multiple pathways, at least including ventral stream related to object recognition and dorsal stream related to motion representation.

According to the Bayesian framework for multisensory integration, the unified percept is the result of weighted estimates across modalities212223. Since the weights are proportional to the reliability in each modality and in primates visual perception generally prevail olfactory perception, previous studies on vision-olfaction interplays reported mainly the visual modulation on olfactory perception while the reversal observations are relatively rare (for review, see10). In present study, we revealed this modulation effects by using ambiguous visual motion signals as probe to test the olfactory influences on visual motion processing. By manipulating the ambiguity levels of visual motion direction, we compromised the quality of visual signals to a large extent and therefore made the reliability comparable across senses, which is crucial in our experimental design. This similar concept of reducing visual reliability was largely employed in previous cross-modal studies using the binocular rivalry paradigms12132728, in which degraded visual perception arises because of alternating (bistable) perceptions29. Taken together, these experiments including our current study, point to a general rule that reducing the reliability of dominant sense facilitates our appreciations of cross-modal interactions.

In our task design, we used visual motion signals instead of previous widely used visual color/shapes or pictorial images3456781112. In odor detection/discrimination tasks subjects showed enhanced performance when an odor was presented together with a congruent color/picture in comparison to a non-matching one7. On the other hand, in visual object perception tasks, the presence of a semantically congruent odor facilitated visual object perceptions1112, as comparing to the case of the non-congruent odor. All these previous studies used the visual attributes processed by the visual ventral pathway. Therefore, interplays between olfaction and vision in these studies might indicate a functional merging between the visual ventral stream and the olfactory system. Here, in the current study, with the olfactory modulation on the visual motion signals, we extend the current understandings of cross-modal interactions to the visual dorsal pathway. We should point out that such changes in visual appearance by means of Pavlovian conditioning have been already demonstrated previously303132. For example, Haijiang et al (2006) described a “cue recruitment” experiment in which an association between an unrelated cue and the rotating direction of a Necker cube was established during the training phase. Subsequently, on test trials the presence of this unrelated cue became effective in biasing participants' direction judgment of the ambiguous rotating cube. In the current study we could as well have employed similar bi-stable motion signals to pair with olfactory cues. However, our point here was not to replicate Haijiang et al's findings but rather was attempted to test whether olfactory cues could influence visual motion processing in general. Similar effects have also been observed in audiovisual interaction studies333435. It was reported that the association between sound and visual motion was easily acquired within a short period of time and sound was able to induce illusory visual motion perception for both static33 and moving visual objects3435. These data and ours suggest that the visual system can flexibly utilize new signals (e.g., auditory or olfactory) as “cues” to build perceptions after exposure to correlated input signals via the form of multisensory association,which can develop very rapidly3637.

We would like to point out that the results of current study are based on behavioral experiments. Therefore, we can only say there is a direct/indirect functional link exists between dorsal visual pathway and olfactory system, but we are not sure about the underlying neural mechanism yet. The linkages between these two sensory systems could in principle have taken place either in the earlier/lower areas or in the later/higher cortical areas. In line with the early merging hypothesis, olfaction-vision interaction has shown to be subconscious12 and nostril-visual field specific13. Furthermore, TMS stimulation over human V1 improves olfactory discrimination38, and olfactory stimulation selectively modulates the OFF bipolar cells in the retina of zebra fish39. Alternatively, the cross-talk between these two senses could also be a result of top-down projections from higher cortical areas such as the orbitofrontal cortex40, which receives converging input from both the primary olfactory cortex and higher-order visual areas41 and has been suggested to play important roles in integrating olfactory and visual information74243. A recent neurophysiologic study44 revealed that neurons in a motion-sensitive cortical area named middle temporal (MT) became selective for the orientations of static arrows after monkeys had learned the association between arrow orientations and different motion directions – a property not seen prior to the associative learning. In the current study the associations between the odorants and motion directions were arbitrarily established by means of classical Pavlovian conditioning. Obviously, there was a lacking of natural “congruency” between the directions of moving dots and the smells of odorants. We thus consider it unlikely that a lower level interaction across these two sensory systems had occurred. Instead, we are inclined to speculate that the interplay between the olfactory system and the visual dorsal stream is more likely to be operating at the semantic/cognitive level that originates from higher cognitive brain areas4043. However, we admit that without further evidence we cannot rule out the roles of early cortical processing. For example, in audiovisual interaction, studies have suggested relatively low level perceptual processing could be involved in the paired association of arbitrary pitched sound and visual motion within a short period4546. Be it at high levels or not, the flexible cross-modal interactions reported in this study via short-term associative learning are consistent with the general view of a highly plastic brain. This extraordinary plasticity during multisensory integration may provide implications into the field of sensory substitution studies4748.

In summary, the present study has shown that, after establishing an association between odorants and motion directions, olfactory cues can bias perceptions of ambiguous visual motion directions. These results extend our understandings on vision-olfaction couplings and suggest a functional interaction between the olfactory system and the visual dorsal pathway. Further investigations involving brain imaging or extracellular recording techniques are needed to address precisely at which neural level the interaction takes place and what the underlying neuronal mechanism are.

Methods

Subjects

A total of forty-one subjects (21 females and 20 males, mean age ± SEM = 24.0 ± 0.79 years) volunteered to participate in the experiments. We randomly divided them into four groups, with each group participated in one of the four experimental designs. All subjects have reported to have normal olfaction and normal or corrected-to-normal vision. They were naive with respect to the scientific purpose of this study. All experimental protocols were approved by the Ethics Review Committee of Institute of Psychology, CAS. All the experimental procedures described below were conducted in accordance with institutional ethical guidelines and informed consent was obtained from all subjects.

Apparatus and stimuli

Subjects were seated in a dimly lit room. They fixated their heads with a chinrest, which was 57 cm away from the CRT monitor (refresh rate: 60 Hz). The generation of visual stimuli and experimental tasks were designed in Matlab, using the Psychophysics Toolbox extensions4950. For visual fixation spot, a gray circle (diameter: 0.25deg) was presented in the center of the screen on a black background. Moving dot patches consisted of 100 dots (diameter: 0.05deg) that were randomly located within an invisible circular window (diameter: 7.5deg). The moving dot patch was presented above the central fixation spot with an eccentricity of 7.5deg. There were two kinds of moving dot patches. In the coherent dot patch, all the dots were moving in the same direction (either left or right, at a speed of 3 deg/s). In the ambiguous dot patch, only a fraction of dots were moving in the same direction while others were moving randomly. Varying the fractions of randomly moving dots allows us to manipulate the ambiguity levels of motion directions. For each subject we pre-tested their direction discrimination performance as a function of fractions of random dots to obtain a psychometric curve. The fraction that gave rise to approximately 60% correct performance was then used for the ambiguous moving dot patch in the subsequent main experiment. We used only one coherence level instead of the full range (say, from 0% to 100%) as typically was done to obtain a psychometric function. Here we were constrained by relative longer inter-trial intervals with olfactory stimuli, and more critically, olfactory processing is vague and fuzzy and it may saturate (or lose sensitivity) after a limited number of trials with repeated exposure to odorants (see below).We chose the ambiguity level of 60% performance based on the empirical experience which was constrained by the fact that, on the one hand, the reliability of visual motion signals should be sufficiently low (so as to be comparable with olfactory cues), and on the other hand, we need to at least reserve some directional information for subjects in the motion direction discrimination task.

We used similar procedures as described in the previous study13 to generate olfactory stimuli and deliver them to participants, except that in the current study the olfactory stimuli delivered to both nostrils were identical. In brief, we had two kinds of odorants: banana-like odor (isoamyl acetate, 1% in propylene glycol) and fennel-like odor (anethole, 1% in propylene glycol). They were contained in two separate 40 ml glass bottles, each fitted with a Teflon nosepiece. Each odorant was delivered to both nostrils of the participants using a computer-controlled air-dilution olfactometer. Participants were encouraged to inhale with their nose and exhale through their mouth. All olfactory stimuli were suprathreshold to all subjects, and this was validated by post-experiment oral reports.

Task design

The behavioral task was the same for subjects in each experimental condition: they were required to indicate the perceived direction of the moving dot patch by means of keyboard press (left key for leftward motion and right key for rightward motion).

Each trial was consisted of two successive phases: a pairing phase and a testing phase (Fig. 1). Each trial started when subjects fixed their gaze at the central gray fixation spot for 1 s, followed by a coherently moving dot patch appeared in the visual periphery for 3 s during which subjects were instructed (the fixation spot transiently turn red) to inhale a specific smell (e.g., banana-like). After a brief inter-stimulus-interval (ISI), a coherent moving dot patch with opposite moving direction would appear at the same location for another 3 s and subjects were asked to inhale a different smell (fennel-like). Note that the order in which the smell-motion direction pair appeared was randomly interleaved during the pairing phase. The subsequent testing phase began after a second ISI of 1.5 s. During the testing phase, the ambiguous moving dot patch would appear and subjects were requested to inhale for a third time before they gave a keyboard response to indicate the perceived direction. The directions of ambiguous moving dot patch could be either left or right and the accompanying odorants could be either banana or fennel smell, and this gave rise to 4 total task conditions (2 directions × 2 odorants). The motion directions and odorants were balanced out and randomly interleaved from trial to trial. There is a 30 s break between each trial with continuous exposure to air puffs to flush the olfactory memory. Each subject performed a total of 160 trials (120 in the pre-test and 40 in the subsequent experiment). Since there were two directions, if the participants made their responses or decisions based not on the testing visual motion but on the previewed pairing of the smell and the motion, then we should expect no difference between left and right testing stimuli. In fact this turned out not to be the case in our data (see Results).

In experiment 1, we paired the banana smell with leftward moving coherent dot patch (while the fennel smell with rightward moving coherent dot patch).To verify our observations in experiment 1, in experiment 2 we reversed the odor-direction pair. That is, we paired the banana smell with rightward moving coherent dot patch. As a control experiment to rule out any potential unidentified non-olfactory cues that contributed to our results, in experiment 3 we paired both directions with the same neutral smell from the purified water which was otherwise suggested to the participants as containing different smells with low intensity. In experiment 4, we intermixed the pairing relationships (motion-odorant mappings were randomly interleaved from trial to trial) to discourage any response and/or decision biases.

Data analysis

In experiment 1–3, our task designs involved 2 (motion directions) × 2 (odorants) conditions. In each condition we counted the percentages of the rightward motion perceptions for each subject. These percentages were then averaged across subjects in each condition. We used both a two-way repeated measures ANOVA and a post-hoc paired t-test to statistically test the modulations of olfactory input on motion perception. P values less than 0.05 were considered statistically significant and marked with one asterisk, and less than 0.01 was marked with two asterisks. In experiment 4 the two-factor repeated measures included 2 odorants and 2 pairing relationships (while the direction was fixed as the ambiguous rightward moving patch). Otherwise the analysis and statistical tests were identical as those in previous experiments.

Author Contributions

S.K. and T.Z. designed the experiments. S.K. conducted the experiments and collected the data. S.K. and T.Z. analyzed the data. S.K. prepared figures 1–2. S.K. and T.Z. wrote the manuscript. All authors reviewed the manuscript.