The cognitive neuroscience of crossmodal correspondences.

In a recent article, N. Bien, S. ten Oever, R. Goebel, and A. T. Sack (2012) used event-related potentials to investigate the consequences of crossmodal correspondences (the "natural" mapping of features, or dimensions, of experience across sensory modalities) on the time course of neural information processing. Then, by selectively lesioning the right intraparietal cortex using transcranial magnetic stimulation, these researchers went on to demonstrate (for the first time) that it is possible to temporarily eliminate the effect of crossmodal congruency on multisensory integration (specifically on the spatial ventriloquism effect). These results are especially exciting given the possibility that the cognitive neuroscience methodology utilized by Bien et al. (2012) holds for dissociating between putatively different kinds of crossmodal correspondence in future research.

Introduction

The last couple of years have seen a marked growth of interest in the topic of crossmodal correspondences (see Marks, 2004; Spence, 2011, for reviews). Crossmodal correspondences have been defined as a tendency for a sensory feature, or attribute, in one modality, either physically present or merely imagined, to be matched (or associated) with a sensory feature in another sensory modality (see Parise & Spence, in press; Spence, 2012). Crossmodal correspondences have now been documented between pretty much every pair of sensory modalities (see Spence, 2012).

Crossmodal correspondences affect behavior in a wide range of experimental paradigms—everything from the redundant target paradigm through to the Implicit Association Test (see Spence, 2011). But how should such phenomena be explained at the computational/neural level? At the computational level, Bayesian coupling priors appear to provide a powerful tool to model the effects of crossmodal correspondences on multisensory integration (Ernst, 2007; Parise & Spence, 2009; Spence, 2011). The latest cognitive neuroscience research is now starting to provide answers regarding the time course and cerebral localization of these, often surprising, cross-sensory mappings (Bien, ten Oever, Goebel, & Sack, 2012; Sadaghiani, Maier, & Noppeney, 2009; Seo, et al., 2010; see also Kovic, Plunkett, & Westerman, 2009).

One critical research question in this area concerns whether crossmodal correspondences are all of a kind or whether instead there are different kinds/mechanisms of correspondence at work (Spence, 2011). One might, for example, want to know whether those correspondences that are based on the internalization of the correlations between stimuli/dimensions present in the environment are represented differently than the semantic correspondences associated with our tendency to use the same terms to describe, for example, certain sounds as having a “high” frequency while at the same time also using the same descriptor when talking about certain elements in a perfume or perhaps wine aroma (so-called “high notes”). While discriminating between different kinds of crossmodal correspondence has sometimes been possible using behavioral techniques (Gallace & Spence, 2006), it is undoubtedly true that more robust insights into the shared vs. distinct mechanism(s) underlying various examples of crossmodal correspondence are likely to emerge from a better understanding of the neural substrates underpinning these effects.

In their latest research, Bien et al. (2012) utilized a simplified version of the psychophysical paradigm developed by Parise and Spence (2009) demonstrating enhanced multisensory integration for crossmodally corresponding pairs of auditory and visual stimuli (small or large circles and low- vs. high-pitched sounds; with integration being assessed by means of the magnitude of the spatial ventriloquism effect). Using event-related potentials (ERPs), Bien et al. (2012) looked for any differences in the pattern of neural activation on those trials where crossmodally congruent vs. incongruent pairs of stimuli were presented. A relatively early neural effect of crossmodal congruency was observed on the P2 component. Seo et al. (2010) have also demonstrated similar effects when comparing the ERPs elicited by crossmodally congruent vs. incongruent combinations of abstract visual symbols and orthonasally presented odors (see also Kovic et al., 2009).

Interestingly, the magnitude of the P2 component at parietal recording sites correlated with the effects reported behaviorally in the study of Bien et al. (2012). These researchers then used source localization to infer the likely source of this congruency effect as originating in right intraparietal sulcus, a site that was then temporarily lesioned by applying offline transcranial magnetic stimulation (TMS). Crucially, this intervention led to the elimination of the effect of crossmodal congruency on participants' behavioral performance (not to mention on the ERP signal). The research of Bien et al. (2012) builds nicely on the earlier psychophysical research of Parise and Spence (2009) while also complementing the results of a study by Sadaghiani et al. (2009) that highlighted differing neural substrates for natural, metaphorical, and linguistic mappings (or crossmodal correspondences). That said, given that Bien et al. (2012) did not fit psychometric curves to their data, it is still an open question whether the effects induced by TMS are due to the reduced integration of crossmodally congruent stimulus pairs or else to a horizontal shift in perceptual space.

In an earlier TMS study by Muggleton, Tsakanikos, Walsh, and Ward (2007), lesioning of right intraparietal cortex using TMS knocked out Stroop-like interference effects in a group of synesthetic participants. However, before jumping to the conclusion that the same neural substrates may underlie both crossmodal correspondences and synesthesia, it is worth noting that the cortical site targeted in the two studies was actually separated by a few centimeters. Of course, the fact that TMS was not applied over a control site in the study of Bien et al. (2012) also means that it is not possible to know exactly how specific (in neural terms) their TMS effects really were.

The temporary lesioning approach to the study of crossmodal correspondences highlighted by Bien et al. (2012) complements previous anecdotal reports that damage to other neighboring regions (e.g., the angular gyrus, located within the temporal–parietal–occipital, region) results in the loss of “the takete–maluma effect” (see Figure 1). In the future, our understanding of the mechanism(s) underlying the many forms of crossmodal correspondence that have been documented behaviorally in recent years (see Parise & Spence, in press; Spence, 2011, 2012) are likely to be greatly enriched by further cognitive neuroscience research using techniques such as TMS. Indeed, one might even go so far as to consider whether certain coupling priors underpinning crossmodal correspondences (e.g., between auditory and visual stimuli) might not be encoded in the intraparietal sulcus itself (cf. Sadaghiani et al., 2009).

Figure 1.

The takete–maluma effect (first introduced by Köhler, 1929). The majority of participants say that the word “takete” better fits with the angular shape shown on the left, whereas the word “maluma” fits better with the rounded shape displayed on the right. Even those individuals from cultures without any written language have been shown to exhibit this effect (see Bremner et al., in press). The only exceptions to this generalization appear to be those suffering from damage to the angular gyrus (Ramachandran & Hubbard, 2003) and certain individuals suffering from autism spectrum disorder (Oberman & Ramachadran, 2008; Ramachandran & Oberman, 2006).