Consciousness is more than meets the eye: a call for a multisensory study of subjective experience.

Over the last 30 years, our understanding of the neurocognitive bases of consciousness has improved, mostly through studies employing vision. While studying consciousness in the visual modality presents clear advantages, we believe that a comprehensive scientific account of subjective experience must not neglect other exteroceptive and interoceptive signals as well as the role of multisensory interactions for perceptual and self-consciousness. Here, we briefly review four distinct lines of work which converge in documenting how multisensory signals are processed across several levels and contents of consciousness. Namely, how multisensory interactions occur when consciousness is prevented because of perceptual manipulations (i.e. subliminal stimuli) or because of low vigilance states (i.e. sleep, anesthesia), how interactions between exteroceptive and interoceptive signals give rise to bodily self-consciousness, and how multisensory signals are combined to form metacognitive judgments. By describing the interactions between multisensory signals at the perceptual, cognitive, and metacognitive levels, we illustrate how stepping out the visual comfort zone may help in deriving refined accounts of consciousness, and may allow cancelling out idiosyncrasies of each sense to delineate supramodal mechanisms involved during consciousness.

Introduction

Cognitive neurosciences largely focus on vision, and the field of consciousness studies is no exception. Counting the number of peer-reviewed articles published per year with the keyword “consciousness” or “awareness” in the visual, auditory, tactile, and olfactory modalities, one can note two trends. First and foremost, starting from the 1990s, the publication rate increases exponentially in all modalities, which attests to the growing importance of consciousness studies in cognitive neurosciences over the last 30 years (Fig. 1). In addition, looking at the publication rate across modalities, one cannot but notice the overwhelming dominance of visual studies compared with all other modalities: in 2015, there were three times more studies of visual consciousness compared with auditory consciousness, and around 12 times more compared with studies of olfactory and tactile consciousness put together. We see two main reasons for this “visual bias” in consciousness research. The first one belongs to recent history: the pioneers who initiated the scientific study of subjective experience in the 1990s relied on the relatively comprehensive understanding of the visual system as a starting point to the quest for consciousness, assuming that other forms of consciousness may have the same properties: “We made the plausible assumption that all forms of consciousness (e.g. seeing, thinking, and pain) employ, at bottom, rather similar mechanisms and that if one form were understood, it would be much easier to tackle the others. We then made the personal choice of the mammalian visual system as the most promising one for an experimental attack. This choice means that fascinating aspects of the subject, such as volition, intentionality, and self-consciousness, to say nothing of the problem of qualia, have had to be left on one side” (Crick and Koch 1990b).

Figure 1.

Number of published articles per year containing the keywords “consciousness” or “awareness” together with “vision” or “visual” (purple), “audition” or “auditory” (red), “touch” or “tactile” (blue), “olfaction” or “olfactory” (green), and “multisensory” or “multimodal” (black). Results were obtained from PubMed (www.ncbi.nlm.nih.gov/pubmed). Fitting was obtained using local regression with R (2016) and the ggplot2 package (Wickham 2009).

The second reason is pragmatic: researchers can rely on a vast arsenal of experimental paradigms allowing the display of visual stimuli below perceptual threshold (Kim and Blake 2005; Dubois and Faivre 2014), while it remains an experimental challenge in all other modalities [but see exceptions like D’Amour and Harris (2014) for vibrotactile masking; Gutschalk ; Dupoux ; Faivre , for auditory masking; Sela and Sobel (2010) for olfactory masking, Stevenson and Mahmut (2013), for olfactory rivalry, or Zhou for binaral rivalry]. As most research on perceptual consciousness relies on a contrastive approach (Baars 1994), the capacity to present stimuli below the threshold for consciousness is crucial, and the difficulty to do so in nonvisual domains can be an impediment to research. We acknowledge that this difficulty could simply be due to methodological reasons, with less effort put onto the design of audio or tactile masking compared with visual masking. However, it could also be due to physiological differences between senses. One possibility is that the transition between conscious and unconscious states is gradual in vision, allowing for fine-grained estimations of unconscious processes, and more abrupt in nonvisual senses, so that subtle changes in the stimulation pattern map onto sudden variations in terms of conscious experience. The possibility that the limen of consciousness (i.e. the transition between unconscious and conscious states) is thinner for nonvisual senses warrants empirical exploration, as it may underlie different properties of consciousness for visual vs. nonvisual modalities.

In what follows, we summarize four lines of research focusing on the multisensory nature of consciousness. Namely, we review the evidence showing that signals from distinct sensory modalities interact even when they are not perceived consciously due to psychophysical manipulations affecting the content of consciousness (“Multisensory interactions below the perceptual threshold for consciousness” section), or due to variations in vigilance states decreasing the level of consciousness (“Multisensory interactions at low levels of consciousness” section). Multisensory interactions between subliminal stimuli or during low levels of consciousness challenge the view that consciousness is a prerequisite for multisensory processing. We then address the role of multisensory interactions in the formation of bodily self-consciousness (BSC), defined as the subjective experience of owning a body and perceiving the world from its point of view (“Multisensory interactions for bodily self-consciousness” section). Finally, we review how the metacognitive self, defined as a second-order monitoring of mental states, applies to different sensory modalities, and conjunctions of sensory modalities (“Supramodal properties of metacognition” section). We conclude by arguing that a multisensory study of consciousness may allow deepening our understanding of subjective experience beyond the idiosyncrasies of visual consciousness.

Multisensory Interactions below the Perceptual Threshold for Consciousness

Most researchers consider conscious experience to be multisensory in nature: For instance, it is difficult to appreciate our favorite dish dissociating the visual, somatosensory, olfactory, auditory, and gustatory sensations evoked by it [this applies both to perception and mental imagery, see Spence and Deroy (2013)]. Despite its apparent tangibility, the actual nature of multisensory conscious remains a topic of philosophical debate [see O’Callaghan (2017), for six differing ways in which conscious perceptual awareness may be multisensory], and some wonder if consciousness is indeed multisensory or rather a succession of unisensory experiences (Spence and Bayne 2014). As scientists, we need to assess how different sensory signals are bound together into a single unified object (i.e. object-unity), but also more globally to assess whether different objects from different senses fit into a single unified experience [i.e. phenomenal unity, see Bayne and Chalmers (2003); Deroy ]. Based on the apparent multisensory nature of conscious experiences, it has been argued that conscious processing plays a role in merging together inputs from different sensory modalities (Baars 2002). Yet, recent evidence suggests that cross-modal interactions exist in the absence of consciousness [reviewed in Deroy ]. The majority of studies contributing to this recent reevaluation of cross-modal consciousness have focused on cross-modal interactions occurring outside of visual consciousness, as visual stimuli can be easily presented subliminally (Kim and Blake 2005; Dubois and Faivre 2014). For example, several sensory modalities can interact with a visual stimulus rendered invisible by interocular suppression [binocular rivalry: Alais and Blake (2005); or continuous flash suppression: Tsuchiya and Koch (2005)]. Cross-modal interactions were demonstrated between subliminal visual stimuli and audition (Conrad ; Guzman-Martinez ; Alsius and Munhall 2013; Lunghi ; Aller ), touch (Lunghi ; Lunghi and Morrone 2013; Lunghi and Alais 2013, 2015; Salomon ), smell (Zhou ), vestibular information (Salomon ), and proprioception (Salomon ). These studies have shown that cross-modal signals facilitate consciousness of a congruent visual stimulus, facilitating its entry into consciousness earlier compared either to visual only stimulation or to incongruent cross-modal stimulation. Moreover, in order for this “unconscious” cross-modal interaction to occur, the cross-modal signals need to be matched. Thus, when the features of a suppressed visual stimulus (e.g. orientation, spatial frequency, temporal frequency, and spatial location) match those of a suprathreshold stimulus in another modality this may facilitate visual consciousness, indicating that the cross-modal enhancement of visual consciousness only occurs when the different sensory signals can be interpreted as arising from the same object. This suggests that these unconscious cross-modal interactions reflect a genuine perceptual effect, which is unlikely to be mediated by a cognitive association between the cross-modal signals. In fact, it has been shown (Lunghi and Alais 2013) that the orientation tuning of the interaction between vision and touch is narrower than the subjects’ cognitive categorization: the interaction between visuo-haptic stimuli with a mismatch of only 7° does not occur, despite the fact that observers cannot consciously recognize the visual and haptic stimuli as being different. These results suggest that signals from nonvisual modalities can interact with visual signals at early stages of visual processing, before the emergence of the conscious representation of the visual stimuli (Sterzer ). Visual stimuli can also interact with subliminal tactile signals: a study by Arabzadeh has reported that a dim visual flash is able to improve tactile discrimination thresholds, suggesting that visual signals can enhance subliminal tactile perception. A similar visuo-haptic interaction has been reported in a study showing that congruent tactile stimulation can improve visual discrimination thresholds (Van Der Groen ). These studies indicate a tight interplay between visual and tactile consciousness. To our knowledge, genuine sensory cross-modal interactions with nonvisual subliminal auditory and olfactory signals have not yet been reported, even though unconscious integration of auditory and visual stimuli occurs after conscious associative learning (Faivre ).

Taken together, these results show that different sensory modalities can contribute to unimodal consciousness, as cross-modal signals can enhance consciousness in a particular modality. What is the function of this cross-modal contribution to consciousness? We suggest that cross-modal interactions might facilitate consciousness of relevant sensory stimuli. Besides facilitating an efficient interaction with the external world under normal circumstances, this mechanism could also be particularly useful in conditions in which the unisensory information is weak or impaired, e.g. during development (when the resolution of the visual system is coarse) or in case of sensory damage.

Multisensory Interactions at Low Levels of Consciousness

The previous section has outlined how unconsciously processed stimuli from different sensory modalities interact, and affect the formation of perceptual consciousness. These findings pertain to situations in which perceptual signals are not consciously experienced during wakefulness, but a few studies show that multisensory interactions may also occur under reduced levels of consciousness. There are several states which are characterized by a decreased level of consciousness: a naturally recurring state occurring during sleep, a pharmacologically induced state under anesthesia, and a pathological state in the case of disorders of consciousness (DOC) (Laureys 2005; Bayne ). Humans in low-level states of consciousness have closed eyes most of the time. Therefore, while visual paradigms may be highly appealing for studying consciousness under normal wakefulness, during altered states of consciousness nonvisual signals may be more suitable.

One of the earliest works examining interactions between different sensory modalities in states of diminished consciousness focused on the feasibility of multisensory associations during pharmacologically induced sleep (Beh and Barratt 1965). Specifically, it was shown that a tone can be successfully paired with an electrical pulse during continuous uninterrupted sleep – which as far as we know is the first evidence for multisensory interaction during human sleep. It took approximately 30 years until the study was revisited, this time during natural human sleep (Ikeda and Morotomi 1996). Again, a tone and an electrical stimulus were associated, yet only during discontinuous slow wave sleep (SWS) and not during Stage 2 sleep, implying that multisensory associations might be sleep stage dependent. To disentangle the influence of different sleep stages on multisensory interactions, a more recent study investigated auditory–olfactory trace conditioning during nonrapid eye movement (NREM) and rapid eye movement (REM) sleep (Arzi ). Sleeping humans were able to associate a specific tone with a specific odor during both NREM and REM sleep. Intriguingly, although associative learning was evident in both sleep stages, retention of the association during wakefulness was observed only when stimuli were presented during NREM sleep, but was irretrievable when stimuli were presented only during REM sleep. This may suggest that although new links between sensory modalities can be created in natural NREM and REM sleep, access to this new information in a different state of consciousness is limited, and sleep stage dependent.

Multisensory associative learning has also been demonstrated in awake neonates (Stamps and Porges 1975; Crowell ; Little ) and in sleeping newborns (Fifer ). One to two days old sleeping infants presented with an eye movement conditioning procedure, learned to blink in response to a tone which predicts a puff of air (Fifer ). Taken together, these findings suggest that despite the different nature of adult and newborn sleep (de Weerd and van den Bossche 2003; Ohayon ), sleeping newborns can also rapidly learn to associate information from different sensory modalities.

A similar auditory–eye movement trace conditioning was used as an implicit tool to assess partially preserved conscious processing in DOC patients (Bekinschtein ). Vegetative state (VS)/unresponsiveness wakefulness syndrome (UWS) and minimally conscious state (MCS) are DOC that can be acute and reversible or chronic and irreversible. VS/UWS is characterized by wakefulness without awareness, and no evidence of reproducible behavioral responses. In contrast, MCS is characterized by partial preservation of awareness, and reproducible though inconsistent responsiveness. Accurate diagnosis of consciousness in DOC has significant consequences on treatment and on end-of-life decisions (Giacino ; Laureys 2005; Bayne ). Some individuals diagnosed with VS or with MCS were able to associate between a tone and an air puff – an ability that was found to be a good indicator for recovery of consciousness. Intriguingly, when the same paradigm was applied to subjects under the anesthetic agent propofol, no association between the tone and the air puff was observed (Bekinschtein ), suggesting that multisensory trace conditioning may not occur under propofol anesthesia. This result is in line with previous findings showing that multisensory neurons during wakefulness become unimodal under propofol anesthesia (Ishizawa ). That said, there is considerable evidence for multisensory interactions at the neuronal level under other anesthetics (Populin 2005; Stein and Stanford 2008; Cohen ; Costa ).

Put together, the few studies that investigated multisensory interactions at low levels of consciousness confirm that information from different sensory modalities can be associated unconsciously. Nevertheless, such unconscious interactions are only found under specific conditions and states of reduced consciousness, suggesting that the ability to associate stimuli from different modality is probably state-specific.

Multisensory Interactions for Bodily Self-Consciousness

The studies we reviewed so far manipulated perceptual signals or examined cases in which consciousness was absent due to neurological damage or sleep. Although this line of work yields some important findings about consciousness and its neural correlates, it overlooks an important aspect of consciousness, namely the “Self,” the subject or “I” who is undergoing the perceptual experiences (Blanke ; Faivre ). Recent research on BSC has uncovered that the implicit and pre-reflective experience of the self is related to processing of bodily signals in multisensory brain systems [for reviews see Blanke (2012); Blanke ; Ehrsson (2012); Gallagher (2000)]. Neurological conditions and experimental manipulations revealed that fundamental components of BSC, such as owning and identifying with a body (body ownership), and the experience of the self in space (self-location) are also based on the integration of multisensory signals [e.g. Ehrsson (2007); Lenggenhager ]. For example, tactile stimulation of the real hand/body coupled with spatially and temporally synchronous stroking of the viewed virtual hand/body give rise to illusory ownership over the virtual hand [rubber hand illusion (RHI)] or body [full body illusion (FBI)] as measured by subjective responses as well as neural and physiological measurements [Botvinick and Cohen (1998); reviewed in Blanke (2012) and Kilteni ]. Such bodily illusions may impact perception, implying a tight link between the subject and the object of consciousness. A recent study based on the RHI revealed that perceptual consciousness and BSC are intricately linked, by showing that visual afterimages projected on a participant’s hand drifted laterally toward a rubber hand placed next to their own hand, but only when the rubber hand was embodied (Faivre ). These results suggest that vision is not only influenced by other sensory modalities, but is also self-grounded, mapped on a self-referential frame which stems from the integration of multisensory signals from the body.

Others have used sensorimotor correlations giving rise to the sense of agency to establish a sense of ownership over a hand (Sanchez-Vives ; Tsakiris ; Salomon ) or a body (Slater ; Banakou and Slater 2014). These findings suggest that BSC is a malleable and ongoing process in which multisensory and motor signals interact such that congruent information from different bodily senses form a current spatio-temporal representation of the self [reviewed in Blanke (2012); Ehrsson (2012); Sanchez-Vives and Slater (2005)]. As BSC is thought to be implicit and pre-reflective, one would expect that it should not require consciousness of the underlying sensory signals. Indeed, it has recently been demonstrated that the FBI, based on visuo-tactile correspondences, can be elicited even when participants are completely unaware of the visual stimuli or the visuo-tactile correspondence (Salomon ). This suggests that subliminal stimuli or unattended bodily signals may still contribute to the formation of BSC.

The above studies show that exteroceptive signals are fundamental in establishing the sense of self. In parallel, several theories have suggested a central role for interoceptive signals arising from respiratory and cardiac activity in BSC (Damasio 1999; Craig 2009; Seth 2013; Park and Tallon-Baudry 2014). Indeed, recent experimental work has linked interoceptive signals to BSC. For example, two experiments employed a paradigm similar to the RHI and FBI, but substituted the tactile stimulation with cardio-visual stimulation: when participants saw their hand/body flashing synchronously with their heartbeat, they reported illusory ownership over the seen limb/body compared with an asynchronous condition (Aspell ; Suzuki ). Furthermore, changes in the neural processing of heartbeats during the FBI (Park ), as well as when facial stimuli are modulated by heartbeat (Sel ), and learning of interoceptive motor tracking have been reported (Canales-Johnson ), suggesting that interoceptive processing is tightly linked to BSC. Beyond their influence on BSC, recent work has shown that cardiac activity and related neural responses also affect visual consciousness, suggesting the possibility of interactions between processing of bodily signals for BSC and perceptual consciousness (Faivre ). Notably, Park ) have shown that spontaneous fluctuations of neural responses to cardiac events (heartbeat evoked responses) are predictive of stimulus visibility. Furthermore, visual stimuli presented synchronously with the heartbeat were found to be suppressed from consciousness through activity in the anterior insular cortex, suggesting that the perceptual consequences of interoceptive activity are suppressed from consciousness (Salomon ). Taken together, these studies indicate that internal bodily processes interact with exteroceptive senses and have substantial impact on both our perceptual and bodily consciousness.

In summary, the sense of self is formed through the combination of multisensory information of interoceptive and exteroceptive origins at a pre-reflective level. While often neglected in classical perceptual consciousness studies, this fundamental multisensory representation of the organism is critical to our understanding of both bodily and perceptual consciousness and their associated neural states.

Supramodal Properties of Metacognition

As described above, the sense of self includes the feeling of owning a body, based on the integration of exteroceptive and interoceptive bodily signals. Another crucial aspect of mental life that relates to the self is our capacity to monitor our own percepts, thoughts, or memories (Koriat 2006; Fleming ). This capacity, long known as a specific form of introspection and now referred to as metacognition is measured by asking volunteers to perform a perceptual or memory task (first-order task), followed by a confidence judgment regarding their own performance [second-order task; Fleming and Lau (2014)]. In this operationalization, metacognitive accuracy is quantified as the capacity to adapt confidence according to first-order task performance, so that confidence is on average high after correct responses, and low if an error is detected. Metacognition has a crucial function in vision (e.g. confidence in spotting a flash of light), as well as in other nonvisual modalities (e.g. confidence in hearing a baby crying, or smelling a gas leak). As for the study of perceptual consciousness, the study of perceptual metacognition also suffers from a visual bias, most studies using single, isolated and almost exclusively visual stimuli. As a result, the properties of metacognition across senses remain poorly described. Notably, an open question is whether metacognitive processes involve central, domain-general neural mechanisms that are shared between all sensory modalities, and/or mechanisms that are specific to each sensory modality, such as direct read out from early sensory areas involved in the first-order task. Answering this question requires one to assess metacognition also for nonvisual stimuli. Recently, two studies met this challenge by examining perceptual metacognition in the auditory domain. First, it was shown that metacognitive performance in an auditory pitch discrimination task correlated with that in a luminance discrimination task (note that correlations between tasks relying on vision or memory were not found, see Ais ). These results are supported by another study showing that confidence estimates across a visual orientation discrimination task and an auditory pitch discrimination task can be compared as accurately as within two trials of the same task, suggesting that auditory and visual confidence estimates share a common format (de Gardelle ). These similarities between auditory and visual metacognition were further supported by another study, in which metacognitive performances during auditory, visual, as well as tactile laterality discrimination tasks were found to correlate with one another (Faivre ). Metacognitive performances in these unimodal conditions also correlated with bimodal metacognitive performance, where confidence estimates were to be made on the congruency between simultaneous auditory and visual signals. Modeling work further showed that confidence in audiovisual signals could be estimated from the joint distributions of auditory and visual signals, or based on a comparison of confidence estimates made on each signal as taken separately. In both cases, these models imply that confidence estimates are encoded with a supramodal format, independent from the sensory signals from which they are built. Jointly, these results suggest that perceptual metacognition involves mechanisms that are shared between modalities [see Deroy for review].

General Discussion

The dominance of vision in consciousness studies may have led to an inadvertent confounding of what we know about visual consciousness and what consciousness is: subjective experience. Conscious experiences evoked by stimuli from different modalities share common features yet each kind of sensory consciousness also has particular qualities. For instance, while sensory perception appears continuous, the temporal sampling in olfaction is in the scale of seconds dictated by sniffing rate, while it is in the scale of hundreds of milliseconds in vision through eye movement, and is almost continuous in audition or other senses such as proprioception (Sela and Sobel 2010; Van Rullen ). Another example among many is the role of attention and its link with consciousness, which is likely to vary across sensory modalities [for a special issue on this topic, see Tsuchyia and van Boxtel (2013)]. At the neural level, most accounts of consciousness assume that visual, tactile, auditory, or olfactory conscious experiences stem from the transduction of sensory signals of completely different nature (i.e. electromagnetic, mechanical, or chemical, respectively), followed by the processing of subsequent neural activity by a common mechanism referred to as a neural correlate of consciousness (NCC: Crick and Koch 1990a; Koch ). Considerable progress has been made regarding the description of NCCs, mostly through the measure of brain activity associated with seen vs. unseen visual stimuli. Among the likely NCC candidates, some predominate, like the occurrence of long-range neural feedback from fronto-parietal networks to sensory areas (Dehaene and Changeux 2011), local feedback loops within sensory areas (Lamme and Roelfsema 2000), or the existence of networks that are intrinsically irreducible to independent subnetworks (Tononi ). If one accepts that a common mechanism enables consciousness in different modalities, a NCC candidate should apply to all senses, and not reflect idiosyncratic properties of visual consciousness only. While these limitations are now often recognized (Koch ), a clear disambiguation between the processes which are idiosyncratic to visual consciousness with those related to consciousness in other modalities, or supramodal processes is central to advancing the field. One step forward in identifying NCCs that are not idiosyncratic of each sensory modality is to rely on a multisensory contrastive approach, whereby one contrasts simultaneously – with the same participants, paradigms, and measures – brain activity associated with conscious vs. unconscious processing in different senses. Within such framework, a supramodal NCC may be defined as the functional intersection of all unisensory NCCs, and consists in the common mechanism that enables consciousness, irrespective of the specificities inherent to each sensory pathway.

Beyond independent subjective experiences in distinct sensory modalities, one may consider consciousness as a holistic experience made of numerous sensory signals bound together (Nagel 1971; Revonsuo 1999; Bayne and Chalmers 2003; Deroy ; O’Callaghan 2017; but see Spence and Bayne 2014). Although often discussed in philosophy, this crucial aspect of phenomenology has remained outside of empirical focus until recently, with notable current theories stressing the importance of consciousness for information integration [reviewed in Mudrik ]. The results reviewed in “Multisensory interactions below the perceptual threshold for consciousness” and “Multisensory interactions at low levels of consciousness” sections clearly show that multisensory interactions occur in the complete absence of consciousness, which challenges the theoretical predictions that consciousness is a prerequisite for multisensory processing. Nevertheless, the existence of interactions between senses does not imply that two signals from distinct modalities are integrated into a bimodal representation. To test if unconscious processing allows bimodal signals to be processed as such, future studies should assess whether bimodal subliminal signals are subject to the classical laws of multisensory integration. In particular, future experiments should assess if the strength of multisensory integration increases as the signal from individual sensory stimuli decreases (i.e. inverse effectiveness), and if the combinations of two unimodal signals produce a bimodal response that is bigger than the summed individual responses (i.e. superadditivity).

Besides perceptual consciousness, the results reviewed in “Multisensory interactions for bodily self-consciousness” and “Supramodal properties of metacognition” sections show that in addition to shaping perceptual experiences, multimodal interactions are also central for self-representations, as they give rise to the “I” or subject of experience (i.e. BSC), and can be processed to form a sense of confidence during the monitoring of one’s own mental states (i.e. metacognition). Thus, beyond overcoming the “visual bias” in the current literature, we argue that studying consciousness in different sensory modalities, and most importantly with combinations of sensory modalities, can reveal important aspects of perceptual consciousness and self-consciousness that would otherwise remain obscured.

Conflict of interest statement. None declared.