Supralinear and Supramodal Integration of Visual and Tactile Signals in Rats: Psychophysics and Neuronal Mechanisms.

To better understand how object recognition can be triggered independently of the sensory channel through which information is acquired, we devised a task in which rats judged the orientation of a raised, black and white grating. They learned to recognize two categories of orientation: 0° ± 45° ("horizontal") and 90° ± 45° ("vertical"). Each trial required a visual (V), a tactile (T), or a visual-tactile (VT) discrimination; VT performance was better than that predicted by optimal linear combination of V and T signals, indicating synergy between sensory channels. We examined posterior parietal cortex (PPC) and uncovered key neuronal correlates of the behavioral findings: PPC carried both graded information about object orientation and categorical information about the rat's upcoming choice; single neurons exhibited identical responses under the three modality conditions. Finally, a linear classifier of neuronal population firing replicated the behavioral findings. Taken together, these findings suggest that PPC is involved in the supramodal processing of shape.

Introduction

Our experience of the world depends on integrating signals through multiple senses. In many instances we can recognize an object independently of the modality by which we acquire the sensory signal, implying that our knowledge about things can be accessed and triggered through a number of sensory pathways (Quiroga et al., 2005). In psychophysical studies, humans, non-human primates, and rodents demonstrate heightened accuracy of perceptual judgements when multiple cues are combined, and in some studies the increase in accuracy approximates that of a statistically optimal observer (Alais and Burr, 2004, Battaglia et al., 2003, Ernst and Banks, 2002, Fetsch et al., 2009, Guo et al., 2014, Jacobs, 1999, Raposo et al., 2012). Neuroscientists, to date, do not have a full understanding of how the processes of convergence, fusion, and generalization across sensory modalities are realized, though much attention has been focused on the problem (Battaglia et al., 2003, Drugowitsch et al., 2015, Ernst and Banks, 2002, Fetsch et al., 2013).

Where in the brain are distinct sensory channels combined? Since object recognition is usually attributed to neocortical processing, here we examine the posterior parietal cortex (PPC), situated between primary somatosensory and visual cortices and a target of both (Akers and Killackey, 1978, Krubitzer, 1995, Reep et al., 1994, Whitlock et al., 2008, Wilber et al., 2014). PPC is a good candidate for visual-tactile convergence (Olcese et al., 2013), although in rodents the degree of multisensory convergence within it is debated (Erlich et al., 2015, Harvey et al., 2012, Licata et al., 2017, Lippert et al., 2013, Raposo et al., 2014).

In the present study, we linked the psychophysical and neurophysiological approaches by measuring the efficiency of multisensory integration through a behavioral test, while simultaneously examining neuronal processing in the PPC. As a psychophysical gauge of integration, we measured the capacity of rats to judge the orientation of an object by the visual sensory channel, by the tactile sensory channel, and by both channels together. The results show that the same functional knowledge of object orientation could be evoked under varying sensory conditions, yet the most reliable judgment arose from the two channels operating together. Next, as a neurophysiological gauge of multisensory integration, we measured the degree to which the activity of PPC neurons recorded during the behavioral task could account for the rats’ supramodal recognition of object orientation.

Results

Visual-Tactile Orientation Categorization Task

We designed a visual-tactile categorization task that involves a solid object. The discriminandum, pictured in Figure 1A, had a circular boundary (98 mm diameter) and raised parallel bars (width and depth of 3.5 mm), alternately colored white and black. It was thus accessible as both a visual and a tactile square wave grating. Rats were trained to encounter the object to judge its orientation on each trial, categorizing orientations in the range of 0°–45° as horizontal, and orientations in the range of 45°–90° as vertical (Figure 1B; also see STAR Methods and Figure S1). Figure 1C illustrates in a schematic manner the sequence of events in the behavioral task. Each trial started with a head poke that triggered the opening of an opaque gate, followed by either a tactile (T) trial (in complete darkness, with the animal allowed to touch the object with its snout and whiskers), a visual (V) trial (under illumination but with a transparent panel in front of the object to prevent contact), or a visual-tactile (VT) trial (with the object illuminated and accessible by touch). After sampling the stimulus, the rat turned its head toward one spout (L or R) and licked. The boundary angle, 45°, was rewarded randomly on left or right. The three stimulus conditions (V, T, and VT) were randomly interleaved with 32% likelihood per trial. The remaining 4% were control trials (neither visual nor tactile access to the stimulus).

Figure 1

Visual-Tactile Orientation Categorization Task

(A) Photographs of the object examined by the rat. Shown are front-side view (left) and exactly front view (right), the latter approximating the perspective of the rat.

(B) Schematic of the orientations of the stimuli and rule of the categorization task. Rats were trained to categorize orientations from 0° to 45° (solid red) in one category and orientations from 45° to 90° (solid green) in another. When tested with −45° to 0° and 90°–135° (red and green stippling, respectively), they immediately generalized the rule, suggesting that the categories corresponded to horizontal (H) and vertical (V); see Figure S2C.

(C) Sequential steps in the behavioral task. Each trial started with a head poke that interrupted a light beam and triggered the opening of an opaque gate, followed by either visual, tactile, or visual-tactile access to the object. After probing the stimulus, the rat turned its head toward one spout, in this illustration left for vertical and right for horizontal. On the right side of the figure, the sequence of actuators and sensors is shown schematically. See Figure S1A for the experimental setup.

As a first step, we characterized the rats’ performance on V and T (unimodal) trials and compared the performance to that on VT trials. Next, we analyzed their accuracy in these three conditions to determine whether they merged or else kept separate the visual and tactile sensory information on visual-tactile trials. Finally, we explored the neuronal correlates of the convergence of sensory channels.

Improved Performance through Merging of Touch and Vision

To quantify rats’ performance, we used a cumulative Gaussian function to fit psychometric curves to the choice data of each rat in each modality (see STAR Methods). Figure 2A reveals that rats differed in their modality-specific acuity: rat 3 performed better in the T condition (left), while rats 12 and 1 performed better in the V condition (middle and right). All three example rats performed better in the VT condition than in V or T alone (in this figure, and throughout the report, green traces correspond to the T condition, blue to V, and red to VT). The boost in performance in the VT condition indicates that, when both modalities were available, rats did not select and rely upon the single, preferred channel; instead, they benefited from the access to both channels of sensory input (Figure S2B).

Figure 2

Supralinear Performance through the Merging of Touch and Vision

(A) Three example rats that, notwithstanding differences in modality-specific acuity, each showed better performance on VT trials as compared to V and T trials. In this and all figures, green traces correspond to the T condition, blue to V, and red to VT. Black curves are predicted psychometric curves from the Bayesian cue-combination model. Error bars show 95% binomial confidence intervals. See Figure S2A for tests of significance.

(B) Pale curves give the performance of 12 rats in each modality. Dark data points and curves show the average over all rats in each modality. Error bars show 95% binomial confidence interval. Inset shows the improved performance in the VT condition versus V alone (blue) and T alone (green), as a function of orientation, averaged across rats.

(C) Comparison of the thresholds predicted by linear combination of V and T signals based on Bayesian cue combination model (ordinate) with measured thresholds on VT trials (abscissa). Each point corresponds to one rat. Error bars indicate 95% confidence intervals by bootstrapping.

(D) Comparison of mutual information between behavioral choice and stimulus category in VT trials (ordinate) with the mutual information between behavioral choice and stimulus category summated from V and T trials (abscissa). Each point corresponds to one rat. Error bars indicate 95% confidence intervals by bootstrapping.

(E) Synthesis of psychometric curve parameters under the three modality conditions. Each point corresponds to one rat. Abscissa denotes the rat’s point of subjective equivalence. The histograms are the summed distributions across individual rats, with bin size 2°. Ordinate denotes the maximum slope of the rat’s psychometric curve (slope is the change in proportion of trials in which orientation was judged as vertical per degree of angle change). The histograms are the summed distributions across individual rats. Circled points are the average values.

(F) Same analysis as in (C) for six subjects (rats 2–7) tested in a block design where seven sessions with only V and T trials were followed by seven sessions with only VT trials. Most rats showed supralinear combination of V and T signals, indicating that sensory channel integration did not depend on the modality sequence of trials. Error bars show 95% confidence interval.

(G) Response times averaged across all test sessions for all rats. Data from V, T, and VT trials are pooled. Data show mean ± SEM. See Figures S2E and S2F for interquartile ranges.

Figure 2B shows the psychometric fits for 12 rats (pale curves, colored according to modality). The average performance across rats is illustrated by the thick curves. Rats showed, on average, good orientation acuity using their whiskers and snout (T) and slightly better performance using vision (V). Performance was markedly superior in the VT condition, indicating multisensory enhancement, consistent with Figure 2A. The standard deviation, , of the cumulative Gaussian function underlying the psychometric curve is a robust measure of acuity, with a small implying a steep psychometric curve slope and high acuity. Individual rats performed best on bimodal trials, independently of the individual’s preferred modality: the decrease in in the VT versus V and T conditions was significant for all rats (Figures S2A and S2B, bootstrap test, p < 0.001).

In the VT condition, rats also showed reduced lapse rates compared to V and T (i.e., fewer errors on “easy” trials, those near the 0° and 90° orientations, p = 0.000 for all conditions in average curve; Figure S2A, bootstrap test) and a more accurate detection of the horizontal and vertical boundary, assessed by the point of subjective equality (PSE closer to 45°, p < 0.02 for V and T in average curve compared to VT; Figure S2A, bootstrap test). The inset in Figure 2B highlights the improved performance in the bimodal condition versus each of the unimodal conditions, averaged across rats. Overall, VT benefit was greater versus T than versus V, an outcome of the better V versus T unimodal performance. Benefit varied with stimulus orientation. At 45°, performance can be no better than chance; thus, multisensory integration provided no benefit. At the cardinal orientations, 0° and 90°, performance in V and T conditions was high, so that multisensory integration could offer limited benefit. At the intermediate orientations, 20°–35° and 55°–70°, the benefit obtained by combining information from the two modalities was greatest.

Did rats learn reward contingencies specific to the stimulus orientations upon which they were trained, or did they execute the task by spontaneously sorting any encountered stimulus, whether familiar or novel, into the categories of horizontal and vertical? To distinguish between the possibilities, in ten rats who were well trained with stimuli in the range of 0°–90°, we introduced trials with stimulus orientations ranging from 90°–135° and −45° to 0° (Figure 1B, stippled sectors) interleaved with the familiar 0°–90° trials in the same session. In the first such session, each rat performed equally well on the new versus the familiar stimuli (bootstrap test, all rats p > 0.29; Figure S2C). Thus, rats seemed to solve the task by measuring the difference between the encountered orientation and the cardinal orientations of “horizontal” and “vertical.” They were able to generalize this categorization rule to new stimuli upon the first instance, consistent with the ability of rats to generalize visual object recognition to novel views (Tafazoli et al., 2012). In further behavioral sessions, we included the full range of stimulus orientations. For the purpose of analysis, stimuli in the ranges of 90°–135° and −45° to 0° were reflected about the vertical and horizontal axes, respectively, allowing all orientations to be treated as 0°–90°.

Merging of Modalities Is Supralinear

We next compared the performance observed in bimodal trials to that predicted by the optimal combination of two unimodal channels. Optimal performance was defined as the linear combination of the two independent signals, based on a framework of Bayesian decision theory (Ernst and Banks, 2002, Fetsch et al., 2013, Landy et al., 2011). Specifically, we fit the psychometric data with a cumulative Gaussian function yielding two parameters: the point of subjective equality (the orientation aligned to the mean of the best-fitting cumulative Gaussian function) and the threshold (the Gaussian function standard deviation, ). In the Bayesian framework, the standard deviation in the VT condition is related to the standard deviations of the unimodal conditions as follows:

Solving the equation for gives the optimal combined threshold from single-modality estimates. Lower corresponds to better accuracy. Single-rat psychometric curves based on this model of optimality are represented by the black traces in Figure 2A. To illustrate all rats together, Figure 2C plots the measured threshold on VT trials versus the threshold predicted by linear combination of V and T signals: 9 out of 12 rats combined V and T signals in a supralinear manner (similar to rat 1 in Figure 2A), 2 in a linear manner (similar to rat 3 in Figure 2A), and 1 sublinearly (rat 12 in Figure 2A).

As an additional approach to comparing observed performance with that predicted by a linear model, we treated V and T as two channels that provide the rat with streams of information. We then compared the total amount of information possessed by the rat on VT trials with the sum of the information possessed, separately, on V and T trials. This approach assumes that the information present within the sensory channels is converted directly into a choice (Adibi et al., 2012) and allows us to quantify this information according to the rat’s behavior (e.g., 1.0 bits of information supports 100% accuracy, 0 bits supports 50% accuracy). In practice, we computed the mutual information (MI; see STAR Methods) between stimulus category (horizontal or vertical) and the rat’s behavioral choice (left or right reward spout) in each modality separately as well as in the combined condition. Then we computed the quantities predicted by the linear combination of V and T signals. In theory, the observed merging of V and T information could be exactly linear, reflecting the optimal combination of two independent signals: sublinear, reflecting redundancy between the two signals, or supralinear, reflecting synergy. If the summated quantities of V plus T information were to exceed 1.0 bits, then the rat would possess more information than it could express by its choices and the model would break down; this case never occurred.

Figure 2D shows that 8 out of 12 rats combined V and T signals in a strongly supralinear manner, 2 rats close to supralinear (one error bar overlies the diagonal), 1 in a linear manner (both error bars overlie the diagonal), and 1 in a sublinear manner. In summary, two analyses—Bayesian prediction of underlying threshold and MI—both suggest that in the majority of rats the signals carried by the two sensory channels were not merged independently but were combined in a synergistic manner (the two measures are correlated, linear regression R = 0.6; Figure S2D).

The availability of two channels not only improved the performance but also allowed rats to more reliably detect the horizontal and vertical boundary. Figure 2E plots their boundary detection (PSE, the stimulus orientation at which left and right choices were equally likely) along with performance (estimated by maximum psychometric curve slope). In the VT condition, the distribution of PSE was closer to the 45° line and curve slope was greater; average values are in the black circles.

In standard testing, V, T, and VT trials were randomly interleaved. It could be argued that supralinear performance on VT trials occurred because rats’ motivation was higher: inasmuch as unimodal trials were more difficult, rats might have spent fewer attentional resources on them, with the knowledge that an easier bimodal trial would soon follow. However, in a different set of experiments, we tested a group of trained rats (rats 2–7) on a block design where seven unimodal-only sessions were followed by seven bimodal-only sessions. This configuration would be expected to minimize any loss in motivation on unimodal trials, since the rat could not choose to attend preferentially to bimodal trials. For this block design, measured values remained close to or better than Bayesian optimal estimates (Figure 2F), indicating that supralinear summation in the interleaved sessions was not well explained by enhanced motivation on VT trials.

Response Time Is Correlated with Trial Difficulty

Response times were measured as the time from first possible sensory access to the object (opening of the opaque panel) until the first lick. Considering all sensory conditions and stimulus orientations, response times were 435–734 ms (interquartile range; Figure S2E). With trials sorted according to stimulus orientation, it becomes apparent that response times were longer for orientations close to the category boundary, 45° (Figure 2G). Thus, rather than undersampling the difficult orientation due to reduced reward likelihood, the rats in fact seemed to try to acquire additional evidence. The distributions differed little according to trial modality condition (Figures S2F and S2G).

Stimulus Orientation and Stimulus Category Are Encoded by Neurons in Posterior Parietal Cortex

We recorded neuronal activity in the left PPC of trained rats (Figure 3A) after they reached a stable level of performance in the behavioral task. For most recording sessions, the set of stimulus orientations was reduced to angles between 0° and 90° in 15° steps (hence seven orientations) to allow a greater number of trials (and therefore neuronal samples) at each orientation. Rats usually completed 300–500 trials per recording session. In Figure 3B, the action potential waveform and interspike interval histogram (ISI) of a single unit is illustrated (upper plots; ISIs for a large set of sampled neurons are given in Figure S3C). The raster plot shows the unit’s firing on 250 (randomly selected out of a session of 524) trials, with trial sequence conserved. Modality conditions (V, T, and VT) are labeled by color and were randomly interleaved. Firing rate, computed in a window of 1 ms and smoothed with a Gaussian ( = 50 ms), is plotted with the same color code. The neuron had a stable firing rate of about 2 spikes/s as the rat awaited the trial onset. About 200 ms after trial onset, the neuron’s firing rate ramped upward and remained at about 8 spikes/s until 1 s after trial onset. Similar temporal profiles characterized V, T, and VT trials.

Figure 3

Encoding of Stimulus Orientation and Stimulus Category in Posterior Parietal Cortex

(A) Upper: the target area, PPC (green), is located between visual cortex (blue) and the vibrissal region of somatosensory cortex, Bf. Lower: histological section demonstrating recording sites in the LPtA zone of PPC according to Paxinos and Watson (2007).

(B) Upper left: example PPC neuron action potentials (150 single waveforms and their average, black). Upper right: interspike interval histogram. Lower: raster plot showing the unit’s firing on 250 randomly selected trials. Modality conditions (V, T, and VT) are labeled by color. Firing rate, computed in a window of 1 ms and smoothed with a Gaussian ( = 50 ms), is plotted with the same color code. Shaded bands are ±SEM.

(C and D) Raster plots (left plots), peri-event time histograms (PETHs, middle plots, smoothed with a Gaussian kernel  = 50 ms), and angular tuning curves (right plots, separated by modality) of four example PPC neurons. Error trials and trials with 45° orientation are excluded. Trials are grouped by stimulus orientation (see color key). Tuning curves show average firing rate for trials grouped with the same stimulus orientation in a 300 ms window preceding the reward spout lick. Data points show the average firing rate per angle ±SEM. The curves are cumulative Gaussian fits (see STAR Methods).

We examined the encoding of stimulus orientation and stimulus category (the horizontal category versus the vertical one) by grouping trials with the same orientation together. Figure 3C shows raster plots (left plots) and peri-event time histograms (PETHs, middle plots) of two example PPC neurons, with responses in all modality conditions combined. Neuronal data are aligned to the trial onset (defined as the onset of the opening of opaque panel). Error trials and trials with 45° orientation are excluded in these plots. The angular tuning curves (right plots, separated by modality, based on neuronal data in a 300 ms window preceding the reward spout lick) highlight that the two neurons’ firing rates were modulated by stimulus orientation, the first firing at a progressively higher rate as angle increased from 0° to 90°, and the second firing at a progressively lower rate from 0° to 90°. By contrast, Figure 3D shows two neurons that carried a large signal about stimulus category: firing rates deviated according to the upcoming action (left or right). The angular tuning curves indicate that they were poorly modulated by stimulus orientation within a choice category. The orientation and category information carried by these neurons is shown in Figure 5.

Figure 5

Joint Information about Stimulus Orientation and Category in Posterior Parietal Cortex

Data points illustrate for 622 neurons the values of conditional mutual information computed from the start of the trial until the response lick. Information was measured between firing rate and stimulus angle conditional on stimulus category as well as the information between firing rate and category conditional on stimulus angle. Neurons numbered as 1–4 correspond to the same neurons of Figures 3C and 3D, top to bottom. Dashed ellipse in main plot highlights a small set of neurons that carried particularly high conditional category information; dashed ellipse in marginal histogram indicates the same set of neurons.

Responses in Posterior Parietal Cortex Are Supramodal

The tuning curves of Figure 3C and 3D suggest that, while neuronal firing could vary according to orientation (first pair of neurons) or category (second pair of neurons), it was largely invariant to differences in modality (in right plots, compare green, blue, and red curves). For the first quantitative test of modality invariance, we merged the data across all stimulus orientations. In Figure 4A, each point represents the average firing rates of one neuron (total, 622 neurons) in a 300 ms window preceding the reward spout lick in V, T, and VT trials, separately. The proximity of nearly all points to the diagonal line attests to the supramodal character of these neurons: a given neuron’s overall engagement in the task was nearly equal under the three modality conditions.

Figure 4

Supramodal Responses in Posterior Parietal Cortex

(A) Each point denotes the average firing rates of one neuron (total, 622 neurons) in a 300 ms window preceding the reward spout lick in V, T, and VT trials, separately. Points are clustered about a line (orange) corresponding to equal responses under the three modality conditions. Data are not separated according to orientation.

(B) Separated by angles (colors), points indicate the firing rate of all neurons, again in the 300 ms window preceding the reward spout lick, in T (abscissa) versus V (ordinate) trials. R is the Pearson linear correlation coefficient.

(C) Upper plots: mean firing rates on T (abscissa) versus V (ordinate) trials for each of the six tested orientations for two neurons. Linear regression coefficient and slope are reported on the plot. The neuron in the left panel was also illustrated in the lower plot of Figure 3C and the neuron in the right panel from the upper plot of Figure 3D. Data points show the average firing rate per angle ±SEM. Lower plots: histograms are the distributions of the linear regression coefficients for all neurons (left) and slopes for neurons with statistically significant regression coefficient (right).

(D) Upper histogram: modality selectivity for a population of 622 PPC neurons. Positive index values correspond to higher firing rate for V trials and negative values to higher firing for T trials. Lower histogram: category selectivity for the same population of neurons. Positive index values correspond to higher firing rate for leftward choices and negative values to higher firing for rightward choices. See Figure S4 for time course of selectivity index measured for each neuron.

Next, we separated the data according to orientation to find out whether invariance to modality persisted across all orientations. Figure 4B illustrates the firing rate of all neurons, again in the 300 ms window preceding the reward spout lick, in T (abscissa) versus V (ordinate) trials. The independence of firing rate on modality held across the six stimulus orientations; the Pearson linear correlation coefficient, R, between firing rate on V and T trials ranged from 0.96 to 0.99.

To obtain a population summary, for each neuron we considered the firing rates on V and T trials as two coordinates and plotted the resulting response values (again, in the 300 ms window preceding the reward spout lick) for each of the six tested orientations, as illustrated for two neurons in the upper plots of Figure 4C. The left plot shows a neuron whose response is modulated by stimulus orientation and the right plot shows a neuron whose response is modulated by the choice category associated with stimulus orientation. For all neurons, we computed the strength and slope of the V versus T linear regression through the six points. A neuron can yield linear regression with coefficient R = 1 and regression slope also 1 only if it satisfies two conditions: variation in firing rate according to stimulus orientation and/or category, and equivalent firing rate for the two modalities. The histograms in Figure 4C (lower plots) are the distributions of the regression coefficients (left) and slopes of the statistically significant coefficients (right); both distributions show a peak near 1, highlighting the similarity in response in the two sensory modalities.

Neurons Show Greater Selectivity for Stimulus Category Than for Stimulus Modality

The nearly equal responses of neurons on V, T, and VT trials imply that the studied population did not inform the brain about the modality in which the object was presented, though modality selectivity of many PPC neurons has been reported in recent work (Raposo et al., 2014). We further examined the problem by comparing stimulus modality selectivity to stimulus category selectivity in the same population. To carry out the comparison, we employed a selectivity measure (Feierstein et al., 2006) based on receiver operator characteristic (ROC) analysis. The measure defines how well the firing rate of a given neuron can be used to classify the modality of the stimulus (visual versus tactile) as well as the rat’s categorization of the stimulus (measured by its choice, right versus left). The metric is proportional to the area under the ROC curve, which is scaled from −1 to 1. Figure 4D, upper histogram, shows the distribution of modality selectivity for the population of 622 PPC neurons. Just 5% (30 out of 622) were selective (p < 0.01; permutation test) for stimulus modality, and the analysis does not point to an overall preference for either modality among these 30 neurons. To test whether modality preference was modulated by the rat’s upcoming choice, we did the same analysis with right-choice and left-choice trials separated, doubling the number of tests (2 per neuron). In the 1,244 tests, the same 5% proportion was selective.

Figure 4D, lower histogram, shows the distribution of category selectivity for the same population of neurons; over 50% of neurons (318 out of 622) were significantly selective (p < 0.01; permutation test) for the upcoming choice of the rat. A greater number of neurons (219 out of 318 neurons) was selective for turns to the right (index < 0) than to the left (index > 0), suggesting overall opposite-side preference in left PPC. In sum, this population of neurons did not appear to be informative about the modality of the stimulus but instead generated a robust signal about stimulus properties—orientation and the choice category associated with that orientation—irrespective of modality (also see Figure S4).

While Figures 4C and 4D considered the category and modality signal separately in single neurons, it was of interest to know how the properties of stimulus orientation and category were carried jointly. Two alternative coding schemes might hold: (1) the properties of stimulus orientation and category were carried jointly, such that no neuron could be cleanly classified as an “orientation-coding” or a “category-coding” neuron, or (2) the population of neurons separated into two distinct functional classes, “orientation” and “category.” Selecting between the models is non-trivial because there are inherent correlations between task parameters: since stimulus orientation predicts its perceived category, a neuron that encodes purely angle will also have responses that vary according to category; likewise, a neuron that encodes purely category will also have responses that vary according to angle.

To disentangle the two correlated coding properties, we developed a conditional information theoretic measure. This measure extracts the information that a neuron carried about stimulus category (assessed by choice) conditional on a specific stimulus angle. For instance, suppose that a rat judged 80% of 30° trials as horizontal (giving 80% accuracy); the analysis would estimate how much information neuronal responses carried about the perceived category of the stimulus, conditional on the 30° orientation. The same measure can be applied to extract the information that a neuron carried about angle conditional on the perceived category (again, assessed by choice) in that set of trials. In this way we could estimate the information a given neuron conveyed about both signals, while removing the effect of task correlation.

The mutual information was computed in a 200 ms window, shifted in 50 ms steps, from the start of the trial until the response lick, and the maximum value was taken. The scatterplot in Figure 5 illustrates the joint conditional information values for 622 neurons. The plot can be interpreted as showing how much additional information a neuron carried about category (abscissa) or angle (ordinate) conditional on a fixed value of the other feature. The marginal distributions (the quantities of one type of conditional information, pooled across all neurons) are made evident by projecting the scatterplot upward and rightward to form histograms. About 50% of neurons (315 out 622, p < 0.01) carried significant information about category conditional on angle (blue triangles and blue histogram projection); a small set of neurons carried a very strong category signal (green dashed ellipses). About 45% of neurons (284 out of 622, p < 0.01) carried significant information about stimulus angle conditional upon perceived category (brick-colored triangles and histogram projection). About 38% of neurons (239 out of 622, p < 0.01, bootstrap test) carried significant conditional information about both orientation and choice (stars and ochre-colored histograms and squares). This seemingly counter-intuitive form of coding signifies that a neuron fired at a significantly different rate according to the rat’s choices, while also firing in a graded manner for angle steps within a category; the firing rate difference across the category boundary was larger than the firing rate differences within category. The circled neurons, labeled 1–4, correspond to the four neurons illustrated in Figures 3C and 3D in order from top to bottom. Around 42% (262 out of 622) of neurons carried neither conditional angle nor conditional category information (gray squares and histograms).

The bands around the diagonal are a boundary of ≥0.05 bits of information (corresponding to 1 standard deviation of the whole dataset); the fact that 78% of neurons that carried information about both angle and category were clustered between the bands indicates that PPC neurons tended to express both stimulus coding and category coding. The simultaneous representation of both task properties suggests that PPC is involved in a supramodal stimulus-to-category transformation.

Choices Are Encoded by Neuronal Population Activity

To this point, we evaluated each neuron as an independent unit. However, it is likely that the rat executes the behavior using signals carried in a collective manner by a large set of neurons (Safaai et al., 2013). Thus, we sought to characterize the extent to which the rats’ performance could be supported by the message contained within populations of PPC neurons. We used a linear discriminant classifier (Duda et al., 2012, Vencatasawmy and Krzanowski, 2002; see STAR Methods) to obtain an estimate of the population signal. The training data for the classifier consisted of the single-trial responses, in a 300 ms epoch before the response lick, of a population of simultaneously recorded neurons across a set of trials that did not include the test trial; after training, the classifier predicted the rat’s choices on the test trial on the basis of the population’s response. The train and test cycle was repeated to yield a predicted choice on every trial. To challenge the robustness and generality of the classifier across modalities and stimulus angles, trials with the specific orientation and sensory modality of the test trial were omitted from the training set. A classifier trained without trials of the angle and modality of the tested trial could successfully decode population activity only if that population carried a robust sensory code (i.e., that generalized from the training trials’ angles to the test trial angle) and a supramodal signal (i.e., that generalized from the training modality conditions to the test trial modality condition).

As schematically illustrated for one typical session (Figure 6A), the classifier was able to determine a boundary that could separate the population’s activity into two states associated with the rat’s two choices. In this example, PPC population activity was decoded as a left (circle) or right (triangle) choice on test trials with stimuli at 0° in T condition after training with trials with orientations from 15° to 90° in V and VT conditions. The decoded trials (dark red points) correspond to 0° stimulus.

Figure 6

Choices Are Encoded by Neuronal Population Activity

(A) Illustration of the linear discriminant classifier used to decode choices from the population activity. The clouds of circles and triangles show the sets of response population vectors produced in different trials of a single recording session. Circles are associated with turning to the left spout, triangles with the right spout. Symbols representing data from training trials have no border, while the symbols representing data from test trials have a black border and are grouped within the dashed boundaries. Since the test data fall on the correct side of the discrimination boundary (near the training trials of the same symbol), the performance in this schematic dataset would be perfect.

(B) Actual trial-by-trial performance of the decoder for each session separately (ten sessions from five rats) is in gray. Black is average across sessions. Decoder performance is measured based on neuronal activity aligned to the time of first response lick; 100 ms windows shifted in 50 ms steps. True decoder performance surpasses 90% correct.

(C) Average psychometric curves (see STAR Methods for the fit) based on decoding neuronal responses (solid lines) and the rats’ observed psychometric curves (dashed lines). Data are presented as mean ±SEM. See Figure S5B for individual sessions and modalities.

Figure 6B evaluates the classifier’s real performance on a trial-by-trial basis for each session separately (ten sessions from five rats). We assessed classifier performance as the rats approached and examined the object, considering neuronal activity in a 100 ms window shifted in 50 ms steps (activity aligned to the time of first response lick). About 1 s before the response (i.e., before trial onset), the decoder’s performance was around 50%. This suggests that the PPC population did not manifest a choice bias before encountering the stimulus. Performance gradually increased to >90% on average by around 250 ms before the response lick. Peak decoder performance measured in 5 rats ranged from 91% to 99% (9 sessions), with 1 session at 70% (p < 0.001, permutation test). To make sure the decoder’s performance did not result from the activity of only a few neurons with strong choice-related signals, we examined the classifier weight associated with each neuron. Almost all neuronal weights were non-zero in each session, indicating that every neuron in the population contributed to the classifier’s performance (Figure S5A).

By separating the classifier’s performance according to stimulus orientation, we obtained in Figure 6C psychometric curves based on decoding neuronal responses (solid lines), and we compared them to the rats’ observed psychometric curves (dashed lines). Across different modalities, the signal carried by the PPC population closely paralleled behavior. Notably, since for each trial the classifier was trained using only trials with modality different from the test trial, we can conclude that the population code was supramodal and could account for the rats’ behavior (Figure S5B). The signal carried by the PPC population also closely paralleled behavior when the classifier training set included the modality of the test trial (Figure S5C).

Discussion

Unimodal Perceptual Capacities and Their Supralinear Combination

This study presents a new behavioral paradigm to investigate whether and how rats combine information from multiple sensory modalities. Results obtained in the two single-modality conditions offer novel insights. Rats were able to perform fine orientation discriminations in the purely visual (V) condition, confirming that the visual capacities of Long-Evans rats might be widely underestimated (Zoccolan, 2015). In the purely tactile (T) condition, rats were also able to perform fine orientation discriminations using their snout and whiskers. In an earlier report (Polley et al., 2005), rats learned in a single trial to avoid shock by escaping the arm of a maze containing protruding vertical bars on the wall; horizontal bars marked the safe arms. Orientation detection was whisker dependent, and resolution was on the order of 45°. In our study, in both modalities rats were sensitive to 5° angle differences (e.g., non-overlapping error bars for neighboring angles in Figures 2A and 2B). Within our set of rats, some performed better in the V condition and others in the T condition. Because of such individual variability, generalizations about rats’ comparative acuity in vision and touch cannot be made.

We used each rat’s psychometric curves in the V and T conditions to compute the psychometric curve expected in the VT condition by optimal Bayesian combination of the two unimodal channels. The majority of rats exhibited actual VT performance significantly better than that predicted by the standard Bayesian ideal-observer model. We refer to this deviation from the model as “supralinearity.” Additional analyses found that the orientation information that rats demonstrated in the VT condition was greater than the sum of the quantities of information they demonstrated in the V and T conditions, a second confirmation of supralinear combination of the two channels.

Supralinear behavioral combination of sensory channels has been found in previous studies (Fetsch et al., 2009, Fetsch et al., 2011, Kiani et al., 2013, Raposo et al., 2012). It could occur if the information available on unimodal trials were only partially exploited, while the information available on bimodal trials were completely exploited. The unimodal signal would then be underestimated by the experimenter, and the accuracy on bimodal trials would appear to surpass that predicted by the linear sum of (underestimated) unimodal signals (Raposo et al., 2012). Partial exploitation of information on unimodal trials might stem from reduced motivation: rats are sensitive to overall reward rate, and unimodal trials yield lower average reward rates than do bimodal trials. Therefore they might attend less to unimodal trials, preferring to invest their attentional resources in bimodal trials, where the investment would be more frequently rewarded. With lower expected net reward on unimodal trials, rats may give priority to speed over accuracy (Drugowitsch et al., 2014, Drugowitsch et al., 2015). In a bounded evidence accumulation framework, the greater speed could be realized by lowering the decision threshold, thus prematurely truncating the processing of unimodal stimuli (Kiani et al., 2008, Kiani et al., 2013).

However, our results do not support the notion of non-optimal use of information on unimodal trials. In all modality conditions, rats’ response time was nearly 150 ms longer on trials with orientation near the 45° category boundary (Figure 2G), indicating that they did not truncate the collection of evidence on more difficult trials. Similarly, response time distributions were similar across the three modality conditions (Figure S2F), suggesting an attempt to accumulate sensory evidence on unimodal trials just as on bimodal trials. Finally, we tested rats with sessions comprised of a single modality condition. In V-only or T-only sessions, rats could not choose to attend preferentially to VT trials, so unimodal performance should come close to its maximal possible value. Yet, comparing across sessions, rats still combined V and T signals in a supralinear fashion to guide VT performance (Figure 2F).

If supralinear combination is not well explained by enhanced motivation on VT trials, what account might better explain the results? Both cue-combination models employed in our study (Bayesian decision theory and MI) assume independence of the sensory channels. The observed supralinearity could result from a violation of that assumption—visual and tactile signals might not be processed independently in bimodal trials. The presence of one signal might affect how the other is acquired by the sensorimotor system. For instance, the visual cue might help the rat palpate the surface with its whiskers more efficiently, boosting the tactile signal (Prescott et al., 2011). Further studies analyzing head and whisker kinematics will be required to assess this hypothesis. A second possible form of interaction is synergy between the two processing pathways within cortex. Inhibition between cortical regions could allow activity evoked through one modality to suppress the non-specific noise in the other modality’s stimulus representation. Evidence of direct functional connections between sensory cortical processing regions is emerging. In the mouse, activation of auditory cortex by a noise burst drives local GABAergic inhibition in the primary visual cortex, via cortico-cortical connections (Iurilli et al., 2012). Similarly, brief pulses of auditory noise sharpen orientation selectivity of L2/L3 pyramidal neurons in primary visual cortex (Ibrahim et al., 2016).

Posterior Parietal Cortex Participates in the Generation of a Supramodal Representation

In search of a cortical processing region that might be involved in the construction of the supramodal object representation revealed by behavioral testing, we focused on PPC. Primate PPC receives input from cortical areas representing multiple sensory modalities and communicates back to these territories (Cavada and Goldman-Rakic, 1989, Leichnetz, 2001). PPC in rats is held to be homologous to primate PPC, as it shares the characteristic reciprocal connectivity with sensory, frontal, temporal, and limbic cortical regions (Akers and Killackey, 1978, Miller and Vogt, 1984, Reep et al., 1994, Whitlock et al., 2008, Wilber et al., 2014). It is thus well positioned to receive unimodal sensory signals (Kolb and Walkey, 1987, Olcese et al., 2013, Reep et al., 1994) and transmit a processed form of those signals to downstream cortical regions.

In our behavioral task, the signals present in PPC—and thus distributed to its targets—appear to incorporate two key properties. The first is a fully supramodal representation of the episode. Each neuron’s firing rates were the same under the V, T, and VT conditions. Indeed, the orientation and category signal carried by neurons in the three modality conditions could not be distinguished except by its greater robustness in VT trials—mirroring the greater behavioral robustness in VT trials (Figure 6C). The second is a representation of the episode that, neuron by neuron, ranged from orientation tuning to category tuning. Since orientation and perceived category were correlated, the two forms of coding were distributed along a single dimension, such that an individual neuron could show graded steps in firing rate within an orientation category (thus, stronger angle tuning) or else a large step in firing rate around 45° (thus, stronger category tuning). In order to quantify the message carried by populations of PPC neurons, we trained a linear classifier to distinguish the rat’s choice based on the population’s firing. When tested, the classifier replicated the rat’s behavior on over 90% of trials (Figure 6B). Considering together the two key properties, we suggest that PPC contributes to the formation of a supramodal representation of the object and participates in the percept-to-action transformation carried out by a network of cortical regions. This interpretation is based on correlations between trial-by-trial neuronal firing and whole-animal behavior; evidence for a direct causal role of PPC might emerge from external manipulations of this region during execution of the behavior.

Our experiment accentuates the high degree of task-dependent plasticity inherent in the function of PPC. While PPC could receive orientation-tuned signals from earlier stages of visual cortical processing (Wang et al., 2012), tactile orientation tuning has not been reported in barrel cortex and may be generated in PPC to meet specific behavioral demands. Furthermore, the reward rule boundary of 45° was set arbitrarily, and its neuronal correlate must also be generated to meet behavioral demands. These observations lead to the conclusion that the functional properties of PPC neurons are not hardwired but rather emerge through a learning process in order to solve the requirements of ongoing tasks.

Formation of Neuronal “Knowledge” about Real Objects

Numerous studies have trained subjects to associate the sensory data originating in two modalities by some arbitrary rule (Sarko et al., 2013). In a recent study in which rats judged the quantities of auditory and/or visual pulses in a train, many neurons in PPC responded differentially on trials of the two modalities; indeed, modality itself could be decoded from neuronal firing (Raposo et al., 2014). This contrasts with the modality-free coding of the object in our study.

We hypothesize that supramodal coding is a consequence of the nearly simultaneous arrival in PPC of congruous signals, through two sensory channels, about a real object. Coherence between modalities, as in our study, might better reflect the statistics of the real world. Vision and touch have likely evolved together to explore the shape, form, and spatial properties of the environment. A rodent might need to maneuver through oriented bars, whether those bars are seen or felt. In general, the brain’s capacity to call up knowledge about things independently of sensory input channel (Quiroga et al., 2005) requires a stage of processing in which the same information is encoded by both channels, and PPC seems to contribute to this abstraction of stimuli from sensory domains.

STAR★Methods

Key Resources Table

Contact for Reagent and Resource Sharing

Further information and requests for resources and reagents should be directed to and will be fulfilled by the Lead Contact, Mathew E. Diamond (diamond@sissa.it).

Experimental Model and Subject Details

Twelve male Long–Evans rats (Charles River Laboratories, Calco, Italy) were used. They were caged in pairs, except for implanted rats which were housed alone. They were maintained on a 12/12 hr light/dark cycle; experiments were conducted during the light phase. Upon arrival they were 8 weeks old, weighing approximately 250 g, and typically grew to over 600 g over the course of the study. They had free access to food in the cage. To promote motivation in the behavioral task, rats were water-deprived on days of training/testing. During each session they received 17-22 mL of pear juice diluted in water (1 unit juice: 4 units water) as reward. After the session they were given access to water ad libitum for 1 hr, though they were rarely thirsty. Rats were examined weekly by a veterinarian. Protocols conformed to international norms and were approved by the Ethics Committee of SISSA and by the Italian Health Ministry (license numbers 569/2015-PR and 570/2015-PR).

Method Details

Apparatus

The main chamber of the apparatus, custom-built in opaque white Plexiglas, measured 25 × 25 × 37 (H × W × L, cm) (Figure S1A). The rat started a trial by interrupting an infrared beam detected by a phototransistor (Figure 1C). Beam interruption triggered fast opening of an opaque panel (through a rotational motion of 40° in 75 ms), actuated by a stepper-motor, uncovering a circular hole (diameter 5 cm) in the front wall through which the rat could extend its head to see and touch the object. The stimulus was 3 cm behind the opaque panel (further details below) and the reward spouts were 2 cm lateral to the edge of the stimulus.

The apparatus was in a Faraday cage which, with door closed, provided acoustic, visual, and electromagnetic isolation. An array of 12 infrared emitters (λ = 850 nm, OSRAM Opto Semiconductors GmbH, Germany) illuminated the stimulus port to permit the investigator to monitor behavior and to execute video recording without providing visible illumination for the rat. On V and VT trials, a pair of 6 white LED arrays illuminated the stimulus. Two infrared-sensitive video systems (Point Grey Flea, Edmund Optics, Barrington, NJ) registered the rat’s actions. The first camera, equipped with a macro lens (Fujinon TV HF25HA-1B Lens, Fujifilm, Tokyo) mounted 25 cm above the stimulus delivery area (distance with respect to the center of stimulus), monitored the rat’s interaction with the object. In some sessions, this camera was set to 250 f/s to monitor head, snout and whisker position and movement during behavior. The second camera provided a wide-angle view (Fujinon HF9HA-1B Lens, Fujifilm, Tokyo) and monitored the entire setup, illuminated with adjustable infrared LEDs, at 30 f/s.

Reward spouts included custom-made infrared diode sensors interrupted by the tongue. Only the licking signal from the correct spout triggered the pump motor (NE-500 programmable OEM; New Era Pump Systems, mounted on a vibration-cancellation pedestal) to extrude the reward, 0.05 mL per trial of diluted pear juice. Licking marked the end of the trial, accompanied by the closure of the opaque front panel. The default setting of the ‘window of opportunity’ from opaque panel opening to initiate reward collection was 2 s, but could be varied by the experimenter to influence the cadence of the rats (e.g., in a slow rat, the window could be reduced to promote urgency). Before the next trial began, the motor on which the stimulus was mounted rotated to generate the next orientation.

Rats are sensitive to reward volume and will quickly develop a strong side preference if the two pumps deliver unequal quantities. Reward volume is a function of the number of steps executed by the pump motor. To make sure that equal volumes of reward were delivered at both ports, we regularly calibrated the pumps according to the following procedure. A computer program ran the stepper motor for 5,000 steps and we measured the volume of the displaced water both from the measurement marks on the syringes and a graduated cylinder. We repeated this step 3 times for each pump separately.

Custom made software was developed using LabVIEW (National Instruments, Austin, TX). An AVR32 board (National Instruments) and multiple Arduino Shields (National Instruments) acquired all sensor signals and controlled the motors, LEDs, and the reward syringe pumps. All the sensors, actuators (including motors and pumps) and lights were interfaced with the computer program allowing full control over a wide range of parameters governing the flow of the training and testing. Although fully automatic, the software allowed the experimenter to modify all the parameters of the task and control the lights, sensors and motors online as needed.

Visual-tactile Stimulus Presentation

The stimulus was a black and white square-wave grating within a circular 9.8 cm-diameter circumference, built in-house by a 3D printer (3D Touch, BFB Technologies, Figure 1A). It was mounted on a stepper-motor and rotated to generate the trial’s intended orientation (Figure 1B). Within behavioral testing sessions each stimulus orientation was sampled from a uniform distribution in 5° steps between −45° and 135° and presented in a semi-random fashion (sampling without replacement). In neuronal recording sessions, stimuli were sampled in 15° steps from 0° to 90° to allow a greater quantity of neuronal data for each orientation.

The depth and width of the grooves was 3.5 mm. Visual acuity is measured in cycles per degree (cpd), an assessment of the number of lines that can be seen as distinct within a degree of the visual field:where is the width of each line in the stimulus and is the distance from the eye. Considering the 3 cm distance behind the opaque panel (Figure S1A), at the moment of panel opening each cycle of the 14 cycles would occupy ° of visual angle, for a total stimulus coverage of about 117°. The spatial frequency of the gratings would be cycles per degree of visual angle. As the normal visual acuity of Long–Evans rats has been estimated as ∼1 cpd (Prusky et al., 2002), the bars would be expected to be resolvable. Rats also have a large depth of focus, from 7 cm to infinity (Powers and Green, 1978). The width of the binocular field directly in front of the rat’s nose, generally considered the animal’s binocular viewing area (Mei et al., 2012), ranges from approximately 40°-110°, depending on head pitch (Wallace et al., 2013). The 117° stimulus should thus completely cover the rat’s binocular visual field.

The apparatus allowed stimuli to be explored in three different conditions. In the visual-only condition (V), ambient light was set to 10.24 cd/mm2 (Konica Minolta LS-100 luminance meter, Tokyo) and a transparent panel in front of the illuminated stimulus prevented tactile contact. In the tactile-only condition (T), the transparent panel was rotated out of the head-poke and the rat leaned forward to examine the stimulus with whiskers and snout in the dark; ambient light measured as 0 cd/mm2. In the visual-tactile (VT) condition, rats were able to both see and touch the stimulus. In a control condition, rats could neither see nor touch the stimulus (transparent panel in front of the stimulus and luminance set to 0).

The stimulus stepper-motor was controlled through a feedback system with a digital step counter to maintain the exact desired orientation and prevent possible changes due to the rat’s contact with the stimulus, e.g., the rat could not rotate the stimulus with its paws. They contacted the object with the macrovibrissae (the long whiskers arranged in rows A-E) and the microvibrissae, which are much shorter and are clustered around the nose and lips (Brecht et al., 1997). Rats mostly projected their whiskers forward and made very small movements (see Movie S1) perhaps engaging in “foveal” whisking (Berg and Kleinfeld, 2003). In the many thousands of examined video-recorded trials taken from the entire set of rats, no observation was made of a T or VT trial in which the animal reached out to contact the object with its paws. As evident in Movie S1, due to the structure of the head-hole (or for implanted rats, head-slit), it was essentially impossible for the rats to extent their paws toward the stimulus delivery area to reach the stimulus; rather they held onto a horizontal bar inside the box arena and extended their head toward the stimulus. The head opening did not extend vertically downward far enough to allow the rat to comfortably pass its paw between its head and the edge of the opening.

Behavioral Task and Training

Duration of training to reach stable performance was typically 4–6 weeks, with 1 session per day, and varied according to individual differences in rate of learning. The training protocol proceeded across the following stages:

Stage 1: Handling

For half an hour each day, for 5–7 days, the investigator held and petted the rat and fed it by hand and a dropper pipette. There was no water restriction.

Stage 2: Training to Head-poke and Collect the Reward

From this stage onward, a water restriction schedule was implemented, whereby the rat collected reward in the apparatus and had free access to water immediately after each session and during the weekend. As the first step of this stage, the rat learned to interrupt the LED beam with its head to open the opaque panel. With panel open, the rat was allowed to examine the stimulus by V and T signals; only the cardinal orientations, 0° and 90°, were presented. After opening of the opaque panel, reward was available at either spout. Rats learned in < 10 trials that reward was triggered by licking. This stage usually lasted 1 day.

Stage 3: Implementation of the Stimulus Rule on Cardinal Orientations

Training continued in the VT condition. Only the cardinal orientations, 0° and 90°, were presented. The rule associating grating orientation with the side of reward availability was implemented, however if the first choice was wrong, the rat was allowed to proceed to the opposite (correct) spout, where the reward was dispensed immediately. The first lick was regarded as the correct/incorrect response for performance measures. By this ‘error remediation’ protocol, the rat began to detect the relationship between stimulus features and reward location. This not only kept it motivated in the task but also gave it further opportunities for stimulus/reward association since it could sample the stimulus during error remediation. This stage usually lasted 1-5 days.

Stage 4: Introducing Delay as a Punishment

As before, only the cardinal orientations, 0° and 90°, were presented. Stimulus examination was in the VT condition. In this stage, a short delay (0.5–2.5 s) was introduced if the rat licked the incorrect spout, after which it could obtain the reward to the opposite (correct) spout. This stage usually lasted 1-3 days.

Stage 5: Rewarding Correct Response Only

As before, only the cardinal orientations, 0° and 90°, were presented. Stimulus examination was in the VT condition. Now the rat was rewarded only when it licked the correct spout as a first choice, and the trial ended with a delay of 1-2 s (as punishment) if it licked the incorrect spout. Faced with gradually increasing error costs across Stages 3-5, they performed above 75% correct in as few as 2 to 7 days (Figure S1C). Typically the rat completed 200–300 trials in these initial training stages.

Stage 6: Training to Discriminate Cardinal Orientations in Interleaved Modality Conditions

The rat was presented with interleaved V, T, and VT trials with equal probability in a random sequence. In the first session of this stage, rats could perform with accuracy of 70%–90% in the V and T conditions, suggesting that the percept of the object in the previous multimodal (VT) condition was naturally extended to the unimodal V and T conditions. This stage usually lasted 1-5 days (Figure S1C).

Stage 7: Training to Discriminate the Full Range of Orientations in Interleaved Modality Conditions

In this stage, all of the orientations were presented within each category (i.e., 0°-45° versus 45°-90°; Figure 1B) in 5° angular steps. At 45° the reward was given with 50% probability at either spout. The rat received equal numbers of trials in each orientation and in each modality. This stage usually lasted 5-14 days (Figure S1E).

With stable performance achieved, the subsequent sessions were taken as final test data. Behavioral measures were taken with stimuli in 5° steps, however in neurophysiological recording sessions, in order to collect as much neuronal data per condition as possible, the resolution was reduced to 15° steps. Rats usually completed 300–500 trials per recording session. Similar to the standard task, trials of all modalities were interleaved.

Random Modality Training

We trained four rats in the random modality protocol to assess whether multimodal enhancement could have been caused by training. The difference between the random modality and standard protocols was the presentation of randomly interleaved V, T, and VT trials with equal probability from Stage 2 onward; in other words, Stage 6 did not constitute a separate stage in this training protocol. Progress in Stages 3–5 was more difficult for these rats and required ∼3–5 weeks of training. This probably occurred because discovering the rule of the task and stimulus/reward contingency was more complex: rats might have assumed that stimulus modality (e.g., absence or presence of the light and transparent panel) was a salient fact and had to learn to ignore this factor (Figure S1D).

Control Experiments

Exclusion of Olfaction

Since olfaction has been implicated in the performance of object recognition tasks ostensibly dependent on other sensory modalities (Astur et al., 2002), we carried out five control procedures. First, we washed the stimulus object with soap and water before and after each training session. In addition, before each session, we wiped the stimulus surface and grooves with a multi-purpose disinfectant solution (Virkon, DuPont, Wilmington, DE). Second, once rats learned the easy conditions (Stage 6), for 3–4 sessions we paused the experiment every 10 trials, wiped the stimulus with the disinfectant solution and put it back, letting the rat continue the task. Third, we substituted different copies of the stimulus on different trials, which would confuse the rat if it were using olfactory cues. Fourth, we printed an object where tactile features (raised bars) were absent but the stimulus was otherwise identical to the standard stimulus (the visual gratings were 2D printed on the object). With this stimulus, rats could only perform the categorization task above chance level in V and VT trials but not on T trials. Importantly, multisensory enhancement was not observed—the performance in VT trials was similar to that of V, confirming not only the absence of olfactory cues, but also a lack of visual distortion due to the transparent screen used on V trials. Finally, on randomly selected trials the stepping motor rotated the stimulus 180° in a random fashion. Doing so maintained the orientation but would cause a left-right reversal of any olfactory cues.

Exclusion of the Motor’s Acoustic Noise

If detectable, the noise made by the stepping motor could inform the rat of the change in orientation between trials according to its duration. First, we sought to assess whether the sound of the motors fell within the acoustic sensitivity range of Long-Evans rats. We recorded sounds (LAN-XI type 3052; Bruel and Kjaer, Nærum, Denmark) during all possible movements of the stimulus motor and examined the frequency spectrum. Figure S1B shows the frequency spectrum of the motor noise in comparison to the measured hearing range of rats reported previously. The highest frequencies generated by the motor were below 500 Hz signifying that rats, which possess the higher-frequency hearing characteristic of mammals (Heffner et al., 1994, Kelly and Masterton, 1977, Masterton and Diamond, 1973, Ölveczky, 2011), would be expected to be insensitive to such sounds.

Nevertheless, as a further caution we implemented an algorithm that generated a series of 2–4 random clockwise and counterclockwise rotations, with variable degrees of rotation, before concluding at the desired orientation. Change duration was thus uncoupled to tested orientation. In a final control experiment, in several sessions we set the object orientation by hand between trials. A nearby motor, unconnected to the object, set orientations uncorrelated with the one accessible in the apparatus. The rats responded to the orientation of the manually controlled orientation, not the inaccessible motor-controlled orientation, indicating that they used the intended visual and tactile cues, not the motor noise.

Catch Trials

During each test session, in 4% of trials the rat was not allowed to either see or touch the stimulus—the transparent panel remained in front of the stimulus and white light was off. Performance was not significantly different from the expected chance performance of 50% (Figure S2H).

Acquisition of Neuronal Data

After rats showed stable performance in the behavioral task (> 75% correct and consistent slope of psychometric curve over the most recent 15 sessions), they underwent a surgical operation for electrode implantation.

Five of the Long-Evans rats trained in the behavioral task were used for neuronal recordings. Two days prior to surgery rats were housed individually to habituate them to the cage conditions and had access to food and water ad libitum.

Microelectrode devices

Depending on the desired duration of the neuronal recordings and intended neuronal dataset rats were implanted either with fixed microwire array electrodes or custom-made movable arrays. Fixed Microwire arrays (ZIF-Clip, Tucker-Davis Technologies, Alachua, FL) were comprised of 16 or 32 polyimide-insulated tungsten wires of 33 μm diameter, 250 μm electrode spacing and 375 μm row spacing (Figure S3A). Each wire was individually laser-cut to the desired length and configuration with a 45° angle of cut at the electrode tip. The impedance of the each wire was 100-300 kΩ (tested at 1 kHz in saline), and around 150-400 kΩ when measured in vivo. Reference and ground wires were connected to the arrays with impedance 20 kΩ. The arrays were designed in a way that wires with two different lengths were interleaved. This resulted in half of the electrodes having 400 μm tip-length difference with the others. Through this configuration we could record simultaneously from two different depths with a single array.

As an alternative method, we developed a custom-made miniaturized movable microdrive for microwire arrays (Figure S3B). The 16 or 32 channel microwire array used in these drives had similar build specification and electrical properties as the fixed arrays. However, the electrodes were connected to the implant connector via a flexible printed circuit board that allowed the array end to be advanced or retracted by a screw. The arrays were advanced 50-100 μm after each recording session.

General Surgical Procedures

Animals were anesthetized with Isoflurane (2.5% for induction and craniotomy, 1.5% for maintenance) delivered through a snout mask. Anesthesia was maintained by monitoring respiration as well as foot pinch responses throughout the surgical procedure. The animals were placed in a stereotaxic apparatus (Harvard Apparatus, Holliston, MA). Ophthalmic ointment (Epigel, Ceva, Libourne, France) was applied to keep the eyes moistened throughout surgery.

In order to keep the heart rate constant and prevent mucus secretion in the airways, animals were injected with atropine (1.5 mg/kg) one hour after the onset of anesthesia. Half of the dose of analgesic (Rymadil, 5 mg/kg) was injected one hour after the onset of anesthesia and the rest of the dose was injected at the end of the surgery before waking the animal. Antibiotic (Baytril, 5 mg/kg) was also injected before waking the rat.

Rats were then given antibiotic enrofloxacin (Baytril; 5 mg/kg delivered through the water bottle) for up to 48 hr after surgery. During the one-week recovery time, rats had unlimited access to water and food. Recording sessions in the behavioral apparatus began thereafter.

Electrode Implantation

After shaving and sterilizing the scalp with iodine solution, lidocaine topical gel was applied to provide local analgesia before performing scalp incisions. Then the skull was cleaned and three to four anchoring screws were fixed in the skull to support the dental cement and the microdrive. One of the screws was advanced deep enough to touch the dura and served as the reference and another one served as the ground (in some cases reference and ground wires were shorted).

The craniotomy was made above the left PPC (centered 3.8 mm posterior to bregma, 2.5 mm left of midline, ∼2 mm anteroposterior; ∼3.5 mm mediolateral in size). Although the craniotomy was made as small as possible, in the case of larger craniotomies we performed additional steps to minimize brain dimpling during electrode insertion. First, dura mater was removed using a small hypodermic needle (guage-25) whose tip was bent to form a small hook. Then a drop of sterile Vaseline ointment was put in the middle of the craniotomy and the surgical cyanoacrylate adhesive (Histoacryl, B.Braun, Melsungen, Germany) was applied directly to the pial surface bordering the edge of the cranial opening. This procedure fastened the pia mater to the overlying bone edge and the resulting surface tension prevented the brain from depressing under the advancing electrodes. The hydrophobic ointment in the middle of craniotomy prevented the spread of tissue adhesive on the brain.

With the brain anchored to the bone, the 16 or 32 channel microwire arrays were inserted by slowly advancing a stereotaxic micromanipulator (SM-25C, Narashige). While lowering the arrays during implantation, the quality of raw signals was monitored and the detected spikes were clustered and sorted online using the OpenEx toolbox (Tucker-Davis Technologies). The array was fixed at a depth of ∼1100 μm below the pial surface, where it became possible to distinguish the spontaneous firing of action potentials. The depth of the recording site is consistent with an electrode tip position in cortical layer 4. However our analyses and conclusions do not depend on the precise laminar localization of the neurons.

Once at the desired depth, the remaining exposed brain surface was either covered with bio-compatible silicon (KwikSil, World Precision Instruments, Sarasota, FL) or a custom-made antibiotic containing Vaseline based ointment depending on the type of electrode array implanted—fixed or movable microdrive respectively. The fixed array or the body of microdrive was then attached to the skull by dental cement (SEcure Starter Kit, Sun Medical, Moriyama, Japan).

Electrophysiological Recordings

After passing through a unity-gain digital headstage amplifier chip (Intan Technologies, Los Angeles, CA), physiological recordings were digitized at a sampling rate of 25 kHz directly inside the headstage clip (ZCD-32, Tucker-Davis Technologies). Digitized signals were then routed to a digital commutator (ACO-32, Tucker-Davis Technologies). This active commutator prevented twisting or binding of the headstage cable while reliably tracking rotation on the headstage cable. The signals were passed to a digital headstage manifold (PZ-4, Tucker-Davis Technologies) through a single cable for transfer to a digital signal processor base station (RZ-2, Tucker-Davis Technologies) via optical fibers, where they were amplified and stored on a computer. Together with the neuronal data, all of the event-related signals from the sensors, lights and motors in the behavioral apparatus were sent synchronously to the RZ-2 DSP, digitized and stored.

Histology

At the conclusion of the series of electrophysiological recording sessions, the rat was deeply anesthetized with Urethane (1.5 mg/kg). To mark the final positions of electrode tips, electrolytic microlesions were made by passing 10 μA current for a through each electrode for 10 s to achieve optimal charge density necessary for lesioning. After lesioning, the animal was perfused transcardially with 4% paraformaldehyde. The brains was extracted and postfixed in 4% paraformaldehyde for 24–48 hr and then in sucrose solution (15% solution then 30% solution). After postfixation, 30 μm coronal sections of the brains were cut on a microtome (SM2010-R Sliding microtome, Leica, Wetzlar, Germany). Slides were Nissl-stained and electrode trajectories were reconstructed under a microscope.

Quantification and Statistical Analysis

Analysis of Behavioral Data

We analyzed the behavioral data in MATLAB (MathWorks, Natick, MA) and LabVIEW.

Generation of Psychometric Curves

To quantify a single rat’s performance, we fit psychometric curves to its choice data separately for each modality. For a given orientation, we calculated the proportion of trials categorized as vertical. Ideally rats would categorize all trials with angle greater than 45° as vertical and all trials with angle less than 45° as horizontal. For 45° trials, choices should be evenly distributed between vertical and horizontal. However, task difficulty grows in the vicinity of 45°, such that real performance is better described by a sigmoid function with an inflection point at the point of subjective equality (PSE), the orientation at which subjects report the stimulus with equal likelihood as horizontal or vertical. In unbiased rats, the PSE should be at 45°. We generated psychometric functions using a cumulative Gaussian function with the general form given in equation below based on (Wichmann and Hill, 2001). The parameter estimation was then performed in MATLAB using maximum-likelihood estimation:

The two-parameter function , is defined by a cumulative Gaussian distribution, as follows:where is the stimulus orientation, is the lower-bound of the function , and is the lapse rate. Often, and are considered to arise from stimulus-independent mechanisms of guessing and lapsing. is the mean of the probability distribution that determines the displacement along the abscissa of the psychometric function–a reflection of the subject’s bias–and is the standard deviation of the cumulative Gaussian distribution. determines the slope of the psychometric function, a common measure of acuity. To generate the psychometric curves, we utilized the data corresponding to the range of orientations from −45° to 135°, with data from orientations −45°–0° and 90°–135° reflected along the horizontal and vertical axes, respectively (e.g., responses to 115° trials were merged with responses to 75° trials). This allowed all data to fit onto one sigmoid, extending from 0° to 90°.

Test of Significance for the Fitted Psychometric Curves

Errors around the performance value for each orientation and modality condition were expressed as a 95% binomial proportion confidence interval computed by approximating the distribution of errors about a binomially-distributed observation, , with a normal distribution:where is the proportion of correct trials (Bernoulli) and is the number of trials.

For statistical tests of significance, we performed a non-parametric test based on bootstrapping, as follows. We computed a distribution of the peak slope values from the first derivatives of the fitted functions based on 1,000 resamples of the performance data. We then performed pairwise comparisons between all the slope values generated via bootstrapping from fitted psychometric function of each sensory condition, calculated the overlap between the distributions and computed the p-values. The same procedure was repeated for tests of significance for other psychometric curve parameters such as PSE and lapse rates (Figure S2A).

Modeling Behavioral Data with Bayesian Cue Combination

According to the linear ideal–observer model of cue integration (Alais and Burr, 2004, Ernst and Banks, 2002, Jacobs, 1999, Körding and Wolpert, 2006, Landy et al., 2011), the bimodal sensory signal, , arises from linear summation of two unimodal signals as follows:where is visual orientation signal and is tactile orientation signal. If each is considered a Gaussian random variable with mean, , and variance, , then the optimal estimate of , can be computed by setting the weights (, ) proportional to the reliability (i.e., inverse variance) of and .

The bimodal reliability will then be equal to the sum of the unimodal cue reliabilities:

We fitted the psychometric data with a cumulative Gaussian function yielding two parameters: the point of subjective equality (mean of the best fitting cumulative Gaussian function) and the threshold (as its standard deviation, ). Solving the equation above for gives the optimal combined threshold from two unimodal thresholds.

Information Theoretic Analysis of Behavioral Data

To quantify the information extracted by the rat in each sensory modality we performed a mutual information analysis. We assumed that the information extracted by the rat about the stimulus orientation was converted directly into a choice; following this assumption, the rat’s behavioral accuracy allows a direct estimate of the signal present in the sensory system. The quantity of information that the behavioral response (left or right) conveys about the stimulus category (horizontal or vertical) can be quantified by Shannon’s mutual information formula (Cover and Thomas, 2012, Shannon, 1948)where is the probability of presentation of a given stimulus category (horizontal or vertical), is the conditional probability of the rat’s response (right or left choice) given the category of stimulus, and is the marginal probability of response (rat’s choice to left or right) unconditional on the stimulus category. All of the information values in equation above were computed using Information Breakdown Toolbox (Magri et al., 2009).

Neuronal Data Analysis

Spike detection and sorting were performed using automatic MATLAB clustering algorithms (Wave_Clus; see Quiroga et al., 2004). Our intention was to analyze only well-separated single units based on interspike interval histogram with refractory period as well as stable waveform and firing rates throughout the recording session. However, single-tip electrode recordings, unlike tetrode recordings, can allow some multi-unit records to leak into the dataset. Multi-unit recordings, however, would be expected to show reduced functional tuning (e.g., orientation or category) since the distinct properties of single-units would be pooled and diluted. The high degree of tuning present in most recordings is a further argument for generally robust single-unit sorting. An initial analysis of neuronal responses included a continuous-time data analysis approach. We first convolved the spike train of each neuron (with 1 ms resolution) with Gaussian kernels to obtain spike density functions. Kernels were corrected for the edge effect. Peri-event time histograms (PETHs) were computed by considering spike times separately in relation to two specific events, depending on the intended analysis: stimulus onset (defined by the opening of the opaque panel) and time of first lick. The spike trains for these epochs were aligned to the stimulus onset or to the first response lick, respectively. Onset-aligned or first lick-aligned time-dependent spike density functions, which give an estimate of the instantaneous firing rate, were used for the rest of the analysis explained below.

For some analyses we computed Shannon’s Mutual Information (Shannon, 1948), hereafter referred to simply as information. Information measures how much knowledge of the neuronal response reduces an observer’s uncertainty (or entropy) about the parameter of interest. In this formulation, the amount of information which can be extracted from the firing rate of a neuron , about the task-related parameter, , can be computed as:where is the probability of presentation of a given task parameter, , is the conditional probability of observing a neuronal response, , given the presentation of the task parameter, , and is the marginal probability occurrence of neuronal response, , among all possible responses unconditional on the task parameter. For example when measuring information about stimulus orientation, is the probability of a given stimulus orientation.

When estimating the information in the neuronal response, we were concerned about spurious information values caused by the inherent correlations between task parameters. Therefore, to determine whether neurons encode the stimulus orientation in a graded manner, we computed the stimulus information that could not be explained by other possible parameters (like future action of the animal). Specifically, we measured conditional stimulus information:likewise,where, is the grating orientation, is the binary action of the animal (left or right), , is the neuron’s firing rate, , is the joint mutual information between firing rate and stimulus orientation and rat’s choice, is the information between firing rate of the neuron and stimulus orientation, conditioned on animal’s decision. To solve the complementary inherent correlation, we computed , the mutual information between firing rate and rat’s choice, conditioned on stimulus orientation.

To quantify neuronal selectivity for category and modality, we used an ideal observer decoding based on ROC analysis (Green and Swets, 1966). PETHs from the pre-decision epoch were constructed from spike trains by averaging firing rates in 1 ms bins and smoothing with a 50 ms Gaussian kernel. Trials were grouped according to two different variables: stimulus category (assessed by the rat’s response, left or right) and the sensory modality available for sensing orientation (visual versus tactile). The ROC analysis was then carried out on the distribution of the group of trials from the smoothed spike trains (Feierstein et al., 2006, Raposo et al., 2014). Category and modality preference were calculated from the area under the ROC curve (AUC) and defined as This value could range from −1 to 1, where −1 indicates that the neuron always fired more preceding right choice and 1 means that the neuron always fired more preceding left choice. Similarly a modality preference of −1 indicates that neuron always fired more for tactile trials and a modality preference of 1 means that the neuron always fired more for visual trials. The modality preference was calculated for leftward and rightward choices separately and was then averaged.

In order to determine whether the activity of a PPC neuronal population encoded the rat’s choices on a trial by trial basis, we used Fisher linear discriminant analysis (LDA) (Duda et al., 2012, Vencatasawmy and Krzanowski, 2002) as a classifier. The classifier finds the most effective linear boundary to separate the activity of the population of neurons according to the rat’s choices, left versus right. The classifier was trained on population responses in a 300 ms epoch before the reward spout lick and made trial-by-trial predictions of rats’ choices according to the following procedure.

First we performed a dimensionality reduction using singular value decomposition (SVD) to identify the dimensions in neuronal space that captured the most variance in the data. For each recording session the data were summarized into a matrix, , with size whose columns correspond to the average of smoothed, z-scored population responses of neurons in T trials in a single session (N = 65, 32, 66, 61, 83, 53, 82, 36, 81, 63). SVD performed singular value decomposition of matrix , such that , where is the diagonal matrix whose diagonal elements are non-negative singular values, and and are left and right singular vectors respectively. Then we chose the first columns of U that correspond to the 80% explained variance and projected onto as follows:

Then by reprojecting the n-dimensional population vector onto a line where the samples of each choice are optimally separated, the LDA found the optimized projection directions whereby the between-class variance in the data is maximized relative to the within-class variance. Defining and the centers of the cluster of points corresponding to choice left and choice right, respectively, the within-class scatter matrix is given by:

And the optimal direction that separates the points of class 1 and class 2 can be shown to be:

Next, we assigned each trial to one of the two classes according to the minimum Euclidean distance in the direction of (v).

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Chemicals, Peptides, and Recombinant Proteins
Isoflurane	Merial	AP/DRUGS/220/96
Epigel	Ceva	N/A
Atropine	ATI	AIC n. 101948014
Antibiotic (Baytril)	Bayer	AIC n. 100155062
Antipain (Rimadyl)	Zoetis	AIC n. 102191119
Topic antibiotic	Dechra	AIC n. 102881012
Local anesthetic (lidocaine)	Molteni Farmaceutici	AIC n. 005638010
Paraformaldehyde	Sigma-Aldrich	158127
Cresyl Violet	Sigma-Aldrich	C5042
SEcure Starter Kit dental cement	Sun Medical, Moriyama	N/A
Bio-compatible silicon	KwikSil, World Precision Instruments	N/A
Histoacryl surgical cyanoacrylate adhesive	B. Braun	N/A
Experimental Models: Organisms/Strains
Long–Evans rats	Charles River Laboratories	Cat# 2308852
Software and Algorithms
LabView 2014	National Instruments	http://www.ni.com/download/labview-development-system-2014/4735/en/
OpenEx toolbox	Tucker-Davis Technologies	http://www.tdt.com/openex.html
MATLAB v 2015b	MathWorks	https://www.mathworks.com/products/matlab/
Wave_clus	By Quiroga et al., 2004	https://github.com/csn-le/wave_clus
Other
Digital TDT recording system	Tucker-Davis Technologies (TDT)	model:RZ2 BioAmp Processor
Microdrive	Custom built; Designed by SISSA Mechatronics Lab	http://www.tdt.com/tdt-microdrive.html