Crossmodal correspondences: Innate or learned?

"Are Chimpanzees synaesthetic?" An affirmative answer to this question appeared recently in a Nature commentary on a study by Ludwig, Adachi, and Matzuzawa (2011) that demonstrated crossmodal correspondences in both chimpanzees and humans. Here we question the claim that chimpanzees are synaesthetic. We also question the claim that certain crossmodal correspondences are innate. We suggest an alternative account for the crossmodal correspondence between auditory pitch and visual lightness in terms of the internalization of correlations present in the environment. We highlight the limitations of such natural correlation approaches to the study of crossmodal correspondences as well as how such claims could potentially be tested in future research.

Introduction

Ludwig, Adachi, and Matzuzawa (2011) recently published the results of an experiment in which chimpanzees and normal human participants had been trained to make speeded manual discrimination responses using a touch screen. On each trial, a small central white or black square was briefly-presented on a computer monitor. Next, two larger squares (one black, the other white) appeared on the upper part of the screen. The participants simply had to touch the square of the matching colour as rapidly and accurately as possible.

After sufficient training (always a concern with animal studies), the chimps were able to respond as rapidly as their human counterparts. However, on every trial, when the two squares were presented on the screen, a high or low tone was presented in the background (see http://www.pnas.org/content/suppl/2011/11/30/1112605108.DCSupplemental for a video). Both groups of participants responded significantly more rapidly to the white (black) square when the high (low) tone was presented than when the mapping was reversed. Crossmodal matches between distinct sensory dimensions, like here between auditory pitch and visual brightness, have been variously documented in human adults and infants over the years, and can be encompassed under the label of crossmodal correspondences (see Spence, 2011, for a review). At the very least, Ludwig et al.'s (2011) results provide one of the first examples that animals exhibit crossmodal correspondences much like the rest of us. They also place the chimp somewhat above the ‘clever’ dogs tested by researchers in Vienna. For while the latter canines could apparently match the sight and sound of a dog of the appropriate size, their gaze to a large or small dog image (tested in a preferential looking paradigm) wasn't influenced by the presentation of a high or low pitched tone (Faragó et al., 2010). The latter result was taken to suggest that man's best friend is simply not smart enough to make such symbolic, or abstract, cross-sensory matches.

The fact that chimps do is important for a number of reasons: First, because such results challenge linguistic accounts of crossmodal correspondences (see Martino & Marks', 1999, semantic coding hypothesis or the idea that these might be ‘metaphorical’ mappings), at least for chimps. Ludwig et al.'s (2011) results are all the more important in that they provide perhaps the first compelling demonstration that animals can match features between modalities in a way that is not simply explainable by their picking-up on an amodal stimulus property, such as shape (as demonstrated by research on the crossmodal transfer of shape information between vision and touch; e.g., Ettlinger & Wilson, 1990; Meltzoff & Borton, 1979; Parker & Easton, 2004), nor are they reducible to the pick-up of some sort of sensory equivalence between, for example, auditory and visual stimulus intensity (see Lewkowicz & Lickliter, 1981).

That said, Ludwig et al. (2011) are certainly not the first to have looked for such apparently arbitrary crossmodal correspondences in animals: Ettlinger (1961) tried, and failed, to show metaphorical crossmodal matches in monkeys. And while Premack and Premack (2003, pp 216–218) make the dramatic claim that monkeys systematically match round to soft and pink, and angular to rough and red in their book on the origins of intelligence, we have been unable to find any independent empirical support for this claim.

Innate correspondences

Ludwig et al. (2011) argue that there was unlikely to have been any correlation between the dimensions of pitch and brightness in the chimp's sensory environment. As such, they could not have learned this crossmodal association or correspondence: Hence the authors came to the dramatic conclusion that such crossmodal matching must be innate. Similar claims for innateness have also been made for certain other crossmodal correspondences (Shepherd, 2012). Other researchers who are unable to find any obvious environmental source for a particular correspondence between particular sensory features (or dimensions) often use the term ‘synaesthetic’ instead (here meaning that the correspondence involves a surprising or unexpected pairing of sensory dimensions, e.g., see Walker et al., 2010) rather than calling the phenomenon that they have ‘discovered’ a crossmodal correspondence (Spence, 2011) or a natural crossmodal mapping (Evans & Treisman, 2010). This places one on a slippery slope that leads from the finding that chimpanzees' behaviour is influenced by a ‘synaesthestic’ mapping between the senses to the much broader claim that chimpanzees are synaesthetes. For many researchers, the latter term primarily refers to rare cases where certain sensory or conceptual inducers elicit idiosyncratic yet consistent, involuntary conscious sensory concurrents (see Deroy & Spence, Forthcoming). The lack of evidence regarding the occurrence of a conscious synaesthetic concurrent should be sufficient to question the appropriateness of applying the term to chimpanzees. Furthermore, Lewkowicz (2011) recently articulated the concern held by many researchers that there simply isn't enough genetic material to code all those abilities/behaviours that researchers seemingly want to suggest are innate, including probably crossmodal correspondences. It is even doubtful whether we need to posit an innate basis for these correspondences is doubtful given the evidence that at least adults are able to learn novel associations between initially uncorrelated sensory dimensions very rapidly (Ernst, 2007).

Now where might the crossmodal correspondence between pitch and lightness have been learned? In most natural environments, the source of illumination comes from above. Humans and presumably many other species (e.g., chimps) have been shown to internalize such information in terms of a “light-from-above” prior (Adams, Graf, & Ernst, 2004). Add to this the fact that smaller bodies or objects will, generally-speaking, make higher-pitched sounds when struck, sounded, voiced etc. than larger objects, and further that smaller objects are more likely to be found in the sky than large objects (elephants and whales were presumably never going to fly). The suggestion that emerges is that there may indeed be an environmental association between lightness, pitch and size: Smaller higher-pitched objects are, statistically-speaking more likely to be located somewhat closer to the source of ambient illumination and hence to be better illuminated and appear visually lighter than larger objects.

Of course, many such suggestions regarding the environmental origins of such ‘surprising’ crossmodal correspondences (see also Walker et al., 2010) have something of the unsatisfactory feel of the “just-so” stories popularized by evolutionary psychologists. They are certainly hard to prove without some kind of environmental sampling of the statistics of the natural environment (see Geisler, 2008, for a review). One other possibility to consider here is that given that crossmodal correspondences are transitive, once the two former statistical regularities are learned (lightness–size; size–pitch), the mind/brain will also acquire the correspondence between lightness and pitch. The two hypotheses (direct or indirect acquisition) might generate different models and predictions. What they both confirm, however, is that looking carefully for the environmental source of such surprising correspondences, once demonstrated, may be preferable to many than the alternative claim that such phenomena are innate, or for that matter, that chimps are synaesthetic.