What can crossmodal aftereffects reveal about neural representation and dynamics?

The brain continuously adapts to incoming sensory stimuli, which can lead to perceptual illusions in the form of aftereffects. Recently we demonstrated that motion aftereffects transfer between vision and touch.(1) Here, the adapted brain state induced by one modality has consequences for processes in another modality, implying that somewhere in the processing stream, visual and tactile motion have shared underlying neural representations. We propose the adaptive processing hypothesis-any area that processes a stimulus adapts to the features of the stimulus it represents, and this adaptation has consequences for perception. This view argues that there is no single locus of an aftereffect. Rather, aftereffects emerge when the test stimulus used to probe the effect of adaptation requires processing of a given type. The illusion will reflect the properties of the brain area(s) that support that specific level of representation. We further suggest that many cortical areas are more process-dependent than modality-dependent, with crossmodal interactions reflecting shared processing demands in even 'early' sensory cortices.

Introduction

Aftereffects are a powerful behavioral paradigm used to infer how information is represented in the brain and how neural populations and circuits change over time – neural dynamics. In the case of motion aftereffect paradigms, for example, an observer stares at visual motion such as a drifting grating (the adapting stimulus) for a period of seconds. When this visual stimulus is suddenly changed to a static visual grating (the test stimulus), the observer sees this stationary stimulus as if it were moving opposite the direction of the original motion for a short period of time., From this simple behavioral paradigm, we gain critical insights into both underlying neural representation and neural dynamics. First, visual motion perception relies on competing representations in opponent directions. Second, extended processing of a stimulus leads to changes in the brain, which we refer to as the adapted brain state. Where in the brain are circuits changing during adaptation? In other words, what is the site of these neural dynamics?

One intuitive answer is that visual motion aftereffects arise from local dynamics in visual cortex. Motion aftereffects also exist in the auditory and tactile domains, suggesting that such neural dynamics are a general property of cortico-cortical or thalamo-cortical circuits. However, we recently demonstrated that adaptation to tactile motion can lead to visual motion aftereffects, and visa versa. Further, visual motion adaptation leads to auditory motion aftereffects., As such, these crossmodal aftereffects challenge the simple explanation that motion aftereffects arise from unisensory cortex alone. What properties about adaptation and representation are needed to explain how crossmodal aftereffects occur?

Adaptive Processing Hypothesis

Aftereffects reveal that extended processing of incoming sensory information changes the brain, and there are measurable consequences of this change in subsequent perception. What is the site of these neural dynamics? A naïve view is that dynamics are expressed only in a final integration stage, for example a single ‘higher’ order cortical area, where modulatory flexibility is inherent to its function. In the classic view, the explicit computational goal of this area is ‘bimodal’ integration. Cross-modal aftereffects would emerge, then, as the product of dynamics at this convergent center.

In contrast, we propose that any area or circuit that processes a stimulus is changed by that stimulus and that these dynamics are a functional property of areas throughout the system—the adaptive processing hypothesis. For example, motion-responsive neurons are found in many places short of the “motion processing area” MT, including V1, V2, and V3, and are also found in parietal areas. Thus, motion aftereffects likely originate not from adaptation in one area or circuit but from many stages of processing both in early sensory areas and in higher level areas.

A corollary of the adaptive processing hypothesis is that at each level of processing, different aspects of the incoming stimulus are adapting, reflecting the underlying dimensions represented by those neural populations. For example, V1 responses reflect orientation, scale, and motion properties at a specific location, with increasing receptive field size and tuning properties in V2, V3, and MT. This view implies that different aftereffects might be observed across retinotopic locations based on the relative contribution of early and later areas in processing the subsequent test stimulus.

Thus, in adaptation paradigms, the subsequent test stimulus can be thought of as a probe of the adapted state. For example, following 10 sec of visual motion adaptation, presenting a static grating leads to retinotopic aftereffects of short duration with low illusory velocity. Following the same adaptation, presenting a dynamic grating instead leads to aftereffects in more spatial locations, which have faster velocity and longer duration. Importantly, the same adapted brain state can give rise to several different perceptual aftereffects. Similarly, following adaptation to a face, observers have stronger aftereffects when tested on upright vs. inverted faces, but also show aftereffects when tested with simple T-shaped stimuli., The critical insight is that the adapted brain state will have consequences on a subsequently presented test stimulus to the extent that that test stimulus depends on processing in those adapted areas. This framework helps explain the well known fact that aftereffects depend on the relationship between the adapting and test stimuli.

In the case of crossmodal aftereffects, these paradigms simply use one modality to probe the adapted state induced by extended processing in another modality. For example, we recently demonstrated that visual and tactile motion adaptation lead to aftereffects in the other modality. Based on the framework outlined above, processing tactile motion depends on circuits that were previously adapted by visual motion processing. Similarly, the processing of visual motion depends on circuits adapted by tactile motion. Crossmodal motion aftereffects reveal that visual and tactile motion perception rely on partially shared neural substrates.

Process-Selective Cortical Circuits

One reason why there might be a site of shared processing between visual and tactile motion comes from an argument for efficient processing. If there is a neural circuit that is specialized to extract motion trajectories from spatio-temporal patterns of spiking input, it might be efficient to route information that requires that processing through that circuit. Indeed, visual motion and tactile motion appear to processed in overlapping (or at least adjacent) areas.,

However, this logic does not extend to auditory motion, which does not activate area MT (reviewed in ref. 15). One possible account for this discrepancy is that visual motion and tactile motion share a similar input pattern, where a grid of sensors in the retina or skin receives spatial information over time. Interestingly, auditory motion information does not arrive by a grid of spatial sensors but by interaural temporal differences, suggesting these stimuli access other brain areas organized to more efficiently perform a different computation.

Several other examples of utilizing specialized processing circuits across modalities exists. For example, TMS studies have shown that fine spatial orientation judgments utilize V1 for visual as well as tactile stimuli. Further, fMRI evidence has shown that fine scale orientation judgments in vision and tactile modalities both activate early visual cortex, visual and haptic shape activate lateral occipital areas, and even haptic exploration of faces is suggested to activate the visual face-selective FFA. If these areas are fundamentally contributing to the perception of orientation, shape, and faces in both modalities, then we would predict crossmodal aftereffects will be found. More generally, these data support process-selective cortical circuits, rather than stimulus-selective cortical circuits. Indeed, emerging evidence that the neocortex is more multisensory than previously believed,, also suggests that defining areas by sensory modality might not accurately describe the underlying representation.

Conclusions

Aftereffects reveal that any incoming sensory information leads to changes in neural dynamics. Typically when we sense the world, we sample the continuous stream of input with rapid exploratory patterns. Eyes saccade 3 times per second, and maintaining steady fixation eventually causes the world to turn flat gray. Similarly, skin sweeps over surfaces, and without changing stimulation we cease to notice contact, e.g., with clothing. While active sensing rapidly samples different aspects of the physical world, adaptation paradigms force extended processing of a single aspect of the physical world (for adaptation with brief durations see ref. 22). In a sense, this extended processing during adaptation may accentuate the neural mechanisms and perceptual consequences that are continually operating on at a more rapid timescale.

Crossmodal aftereffects provide several insights about these adaptive mechanisms. Specifically, we suggest that adaptation is happening at all neural sites that are involved in processing the stimulus, e.g., by renormalizing competing representations to reflect the incoming sensory information. The test stimulus can be thought of as a probe of this adapted state – the extent of shared substrates in processing determines what aftereffects properties will be observed. Areas may be more process-dependent, rather than stimulus-dependent, with crossmodal interactions following automatically in cases with shared processing demands.