Generalized associative representations in parietal cortex.

Making associations between sensory stimuli is a critical aspect of behavior. We previously found that neurons in the lateral intraparietal area (LIP) of Macaca mulatta reflect learned associations between directions of moving visual stimuli. Individual LIP neurons might encode associations only for specific stimuli, such as motion directions; alternatively, they may encode more general associations whenever animals must decide between discrete alternatives. To test this, we asked whether LIP neurons encode learned associations between pairs of arbitrarily chosen static shapes and, in a separate task, whether the same neurons also encode associations between motion directions. Our experimental design dissociated the visual associations from the movements used to report those associations. We found robust encoding of the learned pair associations between shapes, and shape-pair-selective neurons tended to be selective for direction associations. These findings suggest that representing generic categorical outcomes may be a fundamental role of parietal neurons.

The primate brain is adept at rapidly making and breaking associations between sensory stimuli. Associative learning has long been considered a function of frontal and temporal brain areas[1,2]; however, a recent study from our laboratory suggests that parietal neurons also play a role[3]. In that study, monkeys learned to group directions of visual motion into two 180°-wide “categories” separated by an arbitrary direction boundary. After the animals learned to categorize directions, the firing rates of neurons in the lateral intraparietal area (LIP) tended to be much more similar for directions within than between the trained categories. Importantly, these neuronal associations could be reversed after the animals were re-trained with an orthogonal category boundary[3].

Parietal neurons might encode associations only for specific visual attributes, such as directions of motion: LIP receives input from multiple cortical visual areas[4], and inputs from the dorsal and ventral visual streams are anatomically segregated[5]. Alternatively, LIP neurons may encode more general categorical associations whenever animals need to decide between discrete alternatives. This generalized view could encompass both categorical representations[3] and the outcome of perceptual decision processes[6-9] that are encoded by parietal neurons. For example, deciding the direction of a moving random dot pattern in a perceptual decision paradigm ultimately involves a discrete, categorical assignment (e.g., “right” vs. “left”).

To test the generality of associative representations, we examined whether parietal neurons also represent associations for stimulus features other than direction. Under certain experimental conditions, some LIP neurons are selective for visual features such as shape[10,11] or color[12]; here, we trained monkeys to associate pairs of arbitrarily selected static shapes. In separate blocks of trials, the animals also performed the direction-categorization task[3]. We found that the activity of LIP neurons indeed reflected the learned shape-pair associations, and those same neurons also tended to encode the learned direction categories. Thus parietal neurons may provide categorization signals that are generic with respect to visual stimuli.

RESULTS

Delayed shape-pair association task

We trained two monkeys on a delayed shape-pair association task in which they signaled, by releasing their hand from a touch-sensitive bar, whether two sequentially presented shapes belonged to an associated pair. Six shapes were arbitrarily grouped into three pairs (Fig. 1a).

Figure 1

Behavioral paradigm. (a) Monkeys associated six shapes into three pairs. Different pairings were used for each monkey. (b) Delayed shape-pair association task. After monkeys fixated their gaze and gripped a touch-sensitive bar, a “sample” shape appeared in the receptive field (RF). After a subsequent delay period, a “test” shape appeared in the RF. If the sample shape and the test shape belong to the same associated pair, the monkey must release the touch-bar to receive a juice reward. If the sample shape and the test shape do not belong to the same pair, the monkey must maintain his hold on the touch-bar throughout the test period and a second delay period until the associated shape appears (test 2), when he must release the touch-bar to receive juice reward. (c) Monkeys’ mean session performance (chance = 50%; dashed line). Error bars indicate ± 1 s.e.m.

Each trial of the delayed shape-pair association task (Fig. 1b) started with the monkeys fixating their gaze on a central point and gripping the touch bar. A randomly-selected “sample” shape was presented for 650 ms, and following a 1,500 ms delay, a “test” shape was presented for 650 ms. On half of the trials, the test shape was the paired associate of the sample shape; on the other half of trials, one of the four non-associated shapes was randomly selected as the test shape. To receive a juice reward, the monkey had to release the touch-bar when the test shape was associated with the sample shape. If the test shape was not the pair associate, the monkey had to withhold release until the associated shape was subsequently presented. Because the monkey cannot know in advance whether a trial will end with a paired or unpaired test shape, an important feature of this task is that the identity of the sample shape and associated shape are not confounded with the animal’s manual response.

For all six of the sample shapes, the monkeys performed with high accuracy – > 90% accuracy for monkey H and > 85% accuracy for monkey I (Fig. 1c).

Individual neurons are selective for the shape-shape pairings

We recorded from 161 LIP neurons in two monkeys (monkey H, n = 94; monkey I, n = 67) while they performed the delayed shape-pair association task. We analyzed the neuronal activity during four epochs within a trial: “fixation” (1–500 ms before sample-shape onset), “sample” (80–730 ms following sample-shape onset, shifted to account for visual latencies), “delay” (731–2230 ms following sample-shape onset), and “test” (80–300 ms following test-shape onset; truncated earlier if the animal responded before 300 ms). We did not select neurons based on shape selectivity, but most neurons responded selectively among the six shapes (sample period: 132/161, delay period: 118/161, test period: 83/161, compared to 4/161 during the fixation period; Kruskal-Wallis test, P < 0.01), consistent with previous studies[10,11]. However, the shape selectivity of many neurons reflected the learned pairings. The activity of three single neurons during the shape-pair task is shown in Fig. 2a–c. The six traces correspond to the mean activity elicited by each of the six sample shapes on correct trials, and associated pairs of shapes are indicated by traces of the same color. The three neurons show differences in overall firing dynamics throughout the trial (typical for LIP neurons), but for all three, the activity tended to be more similar between associated shapes than between non-associated shapes. This effect was evident during the sample and delay periods, and could even extend into the test period, when the monkey had to decide whether or not to respond to the test.

Figure 2

Example responses of LIP neurons. (a–c) Average activity evoked by the six sample stimuli for three LIP neurons. Neuronal responses are sorted by the identity of the sample shape. Same-color traces correspond to associated pairs of shapes.

We assessed the statistical significance of the shape-pair associations for single neurons using nested analysis of variance (ANOVA) in which the main variables were the three shape-pairs and nominal variables were the two sample shapes within each shape-pair. The nested design tests whether neuronal responses to both shapes within a pair are distinct from responses to the other shapes; neurons that respond selectively to only one shape within a pair would not be considered shape-pair selective by the nested ANOVA. The majority of neurons showed a significant effect of shape-pair during the sample and delay epochs (sample: 103/161, delay: 101/161, test: 60/161, compared to 5/161 during fixation; P < 0.01).

To quantify the strength of shape-pair encoding for each neuron, we calculated eta-squared (η2), the proportion of variance explained by the pairs in the nested ANOVA (Fig. 3a). Explained variance values could range from 0, indicating that none of the variance in single-trial spike rates was explained by the shape-pair identities, to 1, indicating that all of the variance was explained by the shape-pair identities. The mean explained variance among the 161 neurons was highest during the delay period (sample: 0.1507, delay: 0.1759, test: 0.0819, compared to fixation: 0.0189).

Figure 3

Shape-pair selectivity across the population of LIP neurons. (a) Shape-pair selectivity for all neurons, as quantified by the proportion of variance explained by pairs in the nested ANOVA (η2). Black points indicate neurons with significant shape-pair selectivity (nested ANOVA, P < 0.01); non-selective neurons are in gray. Points are arbitrarily shifted along the horizontal access for clarity. (b) Best pairing scheme. The learned pairing scheme is indicated by black, pairing schemes that include one learned pairing are indicated by gray, and pairing schemes that include no learned pairings are shown in white.

While a preponderance of neurons encoded the learned shape-pair associations, we also asked whether the associated pairings were the best possible pairings for each cell. For six shapes, there are 15 possible unique combinations of three pairs: the actual pairing scheme used in the experiment, six combinations in which one of the three associated pairs is included, and eight combinations in which none of the associated pairs are included. We determined the best pairing scheme for every cell by calculating which of the 15 combinations yielded the lowest P-value in the nested ANOVA (Fig. 3b). We only included neurons for which the best pairing scheme (of the 15 possible) was statistically significant (sample: 125, delay: 109, test: 73; nested ANOVA, P < 0.01, Bonferroni corrected). During the sample, delay, and test epochs, the actual learned pairing scheme was most frequently the best pairing scheme (sample: 32/125 (26%), delay: 40/109 (37%), test: 17/73 (23%), versus 1/15 (7%) expected by chance), and the schemes in which all or one learned pair were encoded were significantly more frequent than those in which none of the learned pairs were encoded (Chi-square test; P < 0.0025 for sample, delay, and test epochs).

Behavioral controls

It is possible that systematic differences in performance between pairs of shapes could reflect differences in attention or expected reward that could in turn modulate the firing of LIP neurons[13,14], and thus mimic shape-pair selectivity. Systematic differences in fixational eye movements or position or covert planning of saccades that could occur at the end of trials could also influence neuronal firing[15,16]. We used regression analysis to test the effect of these potential behavioral “contaminants” on neuronal firing. In short, the neuronal selectivity for shape-pairs could not be explained by any of the behavioral parameters that we examined (see Supplementary tables 1–5). We also found weaker shape-pair selectivity on error trials compared to correct trials (see Supplementary data).

Individual neurons encode associations for multiple types of stimuli

Our lab previously found that LIP neurons encode associations among directions of motion that have been grouped together through training[3,17]. A key question is whether individual LIP neurons are “specialized” and encode only shape associations or direction associations, or whether they can show generalized encoding of both types of associations. Associations could be specific to only one type of visual stimulus, perhaps reflecting the apparent anatomical segregation of inputs to LIP from the dorsal and ventral visual pathways[5]. Alternatively, if individual neurons encode both types of associations, this would suggest that LIP neurons play a general role in categorical assignments regardless of the specifics of the visual stimuli.

To examine this question, for 78 of the 161 LIP neurons (n = 45, monkey H; n = 33, monkey I), the animals alternated between the shape-pair association task and a modified version of the direction-categorization task[3], using six directions of motion divided into two groups of three directions (Fig. 4a–c). The activity of two single neurons is shown in Fig. 5a–b. The neurons responded selectively for both the associated shape-pairs and the direction categories.

Figure 4

Delayed direction-categorization task. (a) Monkeys grouped six motion directions into two categories in a modified version of the delayed match-to-category paradigm of Freedman & Assad[3]. (b) Delayed match-to-category task. After monkeys fixated their gaze and gripped a touch-sensitive bar, one sample motion patch appeared in the receptive field (RF). After a subsequent delay period, a test motion patch appeared in the RF. If the sample and test directions belong to the same category, the monkey must release the touch-bar to receive a juice reward. Otherwise, the monkey must maintain his hold on the touch-bar throughout the test period and a second delay period, until a second test stimulus belonging to the same category appears (test 2), when he must release the touch-bar to receive reward. (c) Monkeys’ mean performance across all sessions (chance = 50%; dashed line). Error bars indicate ± 1 s.e.m.

Figure 5

Selectivity for shape-pairs and direction-categories. (a, b) Responses of two example LIP neurons tested with both the shape-pair task (above) and the direction-category task (below). Same-color traces correspond to associated shapes/directions. (c) Explained variance (η2) for shape pairs vs. direction categories for all 78 neurons tested with both tasks. Solid line is regression fit; dashed line has a slope of 1. (d) Time course of explained variance for shape pairs (magenta) and direction categories (black), averaged across all 78 neurons tested with both tasks. Error bars are ± 1 s.e.m. Note that explained variance during the fixation period is slightly higher for the shape task than the direction task because there are three pair predictors for shape and only two category predictors for direction.

For each of the 78 neurons, we quantified whether the neuron was selective for zero, one, or both types of associations using a nested ANOVA. Many neurons were selective for both the shape and direction associations (sample: 36/78, delay: 27/78, test: 14/78, compared to 0/78 during the fixation period; separate nested ANOVA for shape pairs and for direction categories, P < 0.01). Moreover, we did not find evidence for distinct populations of LIP neurons encoding only one type of association. Rather, during the sample and delay intervals, the probability that a shape-pair selective neuron was also selective for direction categories was significantly higher than the probability that a non-shape-pair-selective cell was selective for direction categories (sample: 36/48 pair-selective neurons were category selective, vs. 11/30 non-pair-selective cells, P = 7.6×10−4; delay: 27/42 vs. 12/36, P = 0.0064, test: 14/28 vs. 15/50, P = 0.080; chi-square test).

Individual neurons have similar strengths of association selectivity across tasks

We also examined the relationship between the strength of selectivity for associated shape-pairs and direction categories. We used the proportion of firing-rate variance explained by associated shape pairs or direction categories as our measure of the strength of association. If individual neurons encoded only associated shape pairs but not direction categories, we might find a negative correlation between the two measures across the 78 LIP neurons. Instead we found a significant positive correlation between the strength of selectivity for shape pairs and direction categories: (sample: r2 = 0.21; delay: r2 = 0.38, test: r2 = 0.11, P < 0.005in all cases, vs. fixation: r2 = 0.008, P = 0.44). Fig. 5c shows this relationship for the first half of the delay period, which had the strongest selectivity for each task individually (early delay: r2 = 0.45, P = 2.0×10−11).

We compared the time course of associative signals for the 78 neurons recorded in both tasks by calculating the variance explained by shape pairs and direction categories in 100-ms windows stepped every 50 ms (Fig. 5d). For both tasks, the associative signals rose within the early sample and were sustained throughout the delay and test periods.

DISCUSSION

Pair-associate signals have been observed in inferotemporal and perirhinal cortex[18-20], and also prefrontal cortex[21]. The strength of the associative signals we observed in LIP is at least comparable to the strength of signals reported in frontal and temporal areas, suggesting that parietal neurons contribute to a distributed network that supports learned associations during visual stimulation and working-memory periods. Given its position in the visual cortical hierarchy, parietal cortex may also be a source of the weaker associative signals found in earlier visual areas[22].

We previously hypothesized that LIP neurons may be specialized for forming associations between different directions because LIP receives inputs from areas specialized for motion processing[5,23] as part of the dorsal visual stream[24]. Shape-pair encoding is an important advance for a critical reason. Unlike motion, for which there is a continuous, parametric relationship between directions that could serve as the substrate for developing associations between directions[25], the shape-pairings were completely arbitrary. Thus the shape-pair association task provides strong evidence for the generality of associative representations in LIP; if LIP neurons can “learn” to associate arbitrarily chosen shapes, they should be able to associate any two visual stimuli. Not only could LIP neurons form associations between arbitrarily chosen shapes, in a separate task many of those same neurons encoded learned associations between directions of motion. Thus LIP neurons can form associations for different types of visual features as well as encode arbitrary associations within a single feature space.

But perhaps these results argue that we should focus less on LIP neurons as associating specific features and think more about LIP neurons as representing generic categories. Whenever an animal is confronted with a visual task with discrete alternatives – category A vs. category B, or pair A vs. pair C – the firing of many LIP neurons is likewise discrete, or categorical. This view could provide a unifying framework for understanding other prominent findings about parietal neurons. For example, it has been argued that parietal neurons participate in perceptual decisions[6-9] or encode specific cognitive variables such as numerosity[26-29]. However, these findings have generally been made in the context of tasks that require a choice between discrete, categorical alternatives “motion to the right vs. left,” or “three vs. four items,” etc. We thus propose that these neuronal signals, whether they are labeled “categorical” or “decisional,” are related[30].

It has recently been proposed that decisional signals might be encoded in parietal cortex in an “intentional framework”, based on the particular movements that an animal uses to signal its decision[31]. In our experimental paradigm, the categorical outcome of a trial was explicitly dissociated from the movement that the animal used to report that outcome; the animals did not even know whether they would make a movement until the test interval. Thus the categorization signals that we observed in LIP cannot be explained by a movement/intention-based framework[31]. Our experiment does not address whether other types of categories/decisions might be encoded in an intentional framework. For example, decisions for which movement can be predetermined might be efficiently encoded in an intentional framework, whereas in our behavioral paradigm the animals had to defer decision-related movement planning until after the delay period. However, a recent perceptual decision experiment that did allow for movements to be predetermined on some trials also found categorical/decision-related activity in LIP that could not be ascribed to movement planning per se[32]. It remains an open question whether other types of decisions may be formed in an intentional framework.

A final question is the relationship of these categorical signals to the well-known spatial selectivity of LIP neurons[33,34]. In our experiment, the visual stimuli were confined to the receptive field, so spatial selectivity cannot explain the category selectivity. This suggests that categorical signals may be orthogonal to spatial signals[17]. However, an open question is whether there may be an even broader conceptual framework that could unify categories, decisions, and space in parietal cortex.

ONLINE METHODS

Behavioral tasks

Monkeys were trained to indicate whether a test stimulus belonged to the same pair or category as a previously presented sample stimulus. Animals fixated throughout the tasks (± 1.4–1.75° square window).

For the shape-pair association task, the shapes were static, equated for number of illuminated pixels, and bounded within a 4°-square. Shapes were bidirectionally paired, such that when shape A was presented as the sample, shape A’ was the matching test, and vice versa. Stimuli were centered in the receptive field (RF) of each neuron.

The organization and timing of the direction-categorization task were the same as for the shape-pair association task (Fig. 4b). Motion stimuli were circular patches (4–9° diameter) of 0.1°-square dots moving at 12°/sec in one of six evenly spaced directions (60° apart), with 100% motion coherence.

Shape and direction tasks were presented in alternating blocks of 66 correct trials. Recording sessions alternated between starting with the shape or motion task. Neurons were included in the analysis if at least two blocks of trials were recorded for a task.

Electrophysiological recording

Two male monkeys (Macaca Mulatta, 10.5–14 kg) were implanted with a recording chamber, head post, and scleral search coil. All surgical and experimental procedures were in accordance with Harvard Medical School and National Institutes of Health guidelines. Electrophysiological recordings were made from single neurons using tungsten microelectrodes (FHC) and a guide-tube/grid system. Spike times were recorded with 1-ms resolution. Vertical and horizontal eye position was sampled at 200 hertz.

We tested each neuron with a memory-delayed saccade task and mapped its receptive field with a sparse noise stimulus while the animal fixated[35]. Neurons were classified as LIP neurons if they had spatially-selective delay activity in the memory-delayed saccade task or were located between such neurons in the same electrode penetration. Neurons were not pre-screened for shape or direction selectivity.

Statistical analysis

Significance thresholds were P < 0.01. We used Bonferroni’s correction in the case of multiple comparisons. Data analysis and statistical tests were performed using MATLAB (MathWorks).