How Do Visual and Parietal Cortex Contribute to Visual Short-Term Memory?

Visual short-term memory (VSTM) enables the representation and manipulation of information no longer present in the sensorium. VSTM storage has long been associated with sustained increases in univariate activity (eg, averaged single-neuron spike counts or fMRI activation levels) across a broad network of frontal and parietal cortical areas (for review, see D’Esposito and Postle, 2015). More recently, several research groups have used multivariate analytical techniques to “decode” or infer the identity of a remembered visual stimulus from multivariate fMRI responses measured in human visual cortical areas (eg, V1–V4) during the delay period of a VSTM task, even though these areas typically do not show sustained increases in activity during VSTM storage (Harrison and Tong, 2009; Serences et al., 2009; Riggall and Postle, 2012; Emrich et al., 2013; van Bergen et al., 2015). However, virtually all of these studies have used simple designs that require participants to remember information over a blank delay period. In many real world scenarios, information must be stored despite a constant barrage of dynamic and unpredictable sensory input. How does the brain accomplish this goal?

A recent human neuroimaging paper by Bettencourt and Xu (2016) attempted to answer precisely this question. In their Experiment 1, participants were shown a sequence of two tilted gratings and retroactively cued to remember the orientation of either the first or the second grating. Following a blank delay period, participants judged whether the orientation of a probe grating was tilted slightly clockwise or anticlockwise of the remembered orientation. During the first half of experimental blocks, participants remembered the cued orientation over a blank delay (no-distractor blocks). During the second half of blocks, the delay period was filled with a sequence of task-irrelevant images (photographs of faces or gazebos; distractor blocks). Using fMRI and a multivariate pattern classification algorithm, the authors attempted to decode the orientation of the remembered grating from delay-period activation patterns measured in visual and parietal cortex as a function of distractor presence. In particular, the authors focused on two regions-of-interest (ROIs). The first encompassed retinotopically organized visual areas V1–V4, which typically support robust decoding of remembered stimuli (Harrison and Tong, 2009; Serences et al., 2009). The second encompassed portions of the superior intraparietal sulcus (sIPS), where overall activation levels have been shown to track the amount of task-relevant information that must be enumerated (Nieder et al., 2006; Harvey et al., 2013), tracked (Drew and Vogel, 2008), attended (Mitchell and Cusack, 2008), or stored in VSTM (Todd and Marois, 2004; Xu and Chun, 2006).

On the assumption that areas V1–V4 would be recruited during sensory processing of the distractor photographs, Bettencourt and Xu (2016) reasoned that stimulus-specific delay period activation patterns in these regions would be particularly susceptible to interference. By contrast, insofar as sIPS is less involved in sensory processing, stimulus-specific delay period activation patterns in measured this region should be robust to interference. Thus, Bettencourt and Xu (2016) hypothesized that decoding performance in V1–V4 would be reduced during distractor blocks relative to no-distractor blocks, whereas decoding performance in sIPS would be unaffected by distractor presence. This is precisely what was found: during no-distractor blocks, both V1–V4 and sIPS supported above-chance decoding of the cued orientation, replicating earlier findings (Ester et al., 2009; Harrison and Tong, 2009; Serences et al., 2009; Christophel et al., 2012; Emrich et al., 2013; van Bergen et al., 2015). During distractor blocks, decoding performance in sIPS remained well above chance levels (and statistically indistinguishable from decoding performance on no-distractor blocks), whereas decoding performance in V1–V4 was reduced to chance. Additionally, participants’ behavioral performance was statistically equivalent during distractor and no-distractor blocks, suggesting that the loss of stimulus-specific information in V1–V4 during distractor-present blocks had a negligible effect on memory performance. The authors interpreted these findings as evidence that sIPS, and not V1–V4, has a privileged role in mediating VSTM storage, particularly when distracting visual information is present.

This conclusion is problematic, because it rests on the assumption that if a particular region contributes VSTM, then activation patterns measured in that ROI during distractor blocks should support above-chance decoding of a remembered stimulus. However, chance-level decoding need not imply that a particular region does not contribute to VSTM (or any other related function). Indeed, one alternative possibility is that V1–V4 encode stimulus-specific representations during distractor blocks, but at a level of anatomical or physiological granularity that is inaccessible to the multivoxel decoding approaches used in this study (for a recent example of information existing at the level of single neurons, but not multivoxel activation patterns see Dubois et al., 2015). Thus, the results of this experiment provide only indirect support for the conclusion that sIPS has a privileged role in mediating VSTM storage.

In a subsequent experiment (Experiment 3), Bettencourt and Xu (2016) attempted to replicate these findings while mixing the distractor and no-distractor conditions within the same block of trials. If distractors invariably disrupt stimulus-specific VSTM representations in V1–V4, then the results of this experiment should be identical to their first experiment: both V1–V4 and sIPS should support above-chance decoding during no-distractor trials, but only sIPS should support above-chance decoding during distractor-present trials. However, in this version of the task both V1–V4 and sIPS supported above-chance decoding of the cued orientation during distractor trials. Bettencourt and Xu (2016) suggested that these results could reflect a strategic choice: if distractors selectively interfere with VSTM representations in V1–V4, then participants may choose to “disengage” these regions when they are certain that distractors will be present. This would explain why decoding performance in Experiment 1, where distractor and no-distractor trials were blocked, fell to chance levels when distractors were present. This seems unlikely for several reasons. First, recall that in Experiment 1 participants’ memory performance was equivalent during distractor and no-distractor blocks, even though decoding performance in V1–V4 fell to chance levels when distractors were present. Thus, activation patterns in V1–V4 made no discernable contribution to overall memory performance. It is therefore unclear why participants would choose to engage these regions under any circumstance (assuming this is indeed a strategic choice), particularly if doing so is metabolically costly. Second, as Bettencourt and Xu (2016) note, most real-world scenarios require VSTM representations to be maintained despite a constant barrage of dynamic and unpredictable sensory input. Thus, the results of Experiment 1, when distractor presence was entirely predictable, may reflect a unique (and artificial) set of circumstances rather than a general property of the neural systems supporting VSTM.

If V1–V4 and/or sIPS contribute to VSTM performance, then delay period activation patterns in these regions should correlate with participants’ memory performance (Emrich et al., 2013; Ester et al., 2013; van Bergen et al., 2015). In Experiment 4, Bettencourt and Xu (2016) tested this possibility by comparing activation patterns measured with fMRI with memory performance measured outside of the scanner. In both tasks, participants remembered the orientation of a single (masked) grating over a blank delay period. On each trial, the to-be-remembered grating was assigned one of six possible orientations (10°, 40°, 70°, 100°, 130°, or 160°). In the scanner, participants judged whether the orientation of a probe grating presented at the end of the trial was tilted clockwise or anticlockwise of the remembered orientation (as in Experiments 1 and 3). During behavioral testing, participants reported whether the remembered grating matched the orientation of a subsequent probe. During mismatch trials, the probe orientation was tilted ±30°, ±60°, or 90° relative to the remembered orientation. Bettencourt and Xu (2016) reasoned that representations that are more similar (eg, ±30° apart) should take longer to discriminate and should be harder to decode than representations that are less similar to one another (eg, ±90° apart). Consequently, decoding performance for similar pairs of remembered orientations should be inversely correlated with behavioral response latencies for the same pairs. That is, decoding accuracies should be low, and response latencies should be high, for similar relative to dissimilar pairs of orientations. Indeed, pairwise decoding accuracies were negatively correlated with pairwise response latencies in both V1–V4 (r = −0.70) and sIPS (r = −0.59). This result suggests that both V1–V4 and sIPS contribute to VSTM storage. Instead, Bettencourt and Xu (2016) argued that whereas the correlation between behavioral response times and neural activation pattern discriminability in sIPS likely reflects VSTM storage, the correlation between the same variables in V1–V4 likely reflects lingering sensory processing of the to-be-remembered stimulus. However, both correlations were generated using activation patterns measured during the same delay period interval, and both correlations remained strong when activation patterns measured during the early part of the memory delay (ie, those most likely to include contributions from lingering sensory responses) were omitted from the analysis. In light of these observations, it seems quite unlikely that the correlation reported in V1–V4 reflects sensory processing, whereas the correlation reported for sIPS reflects VSTM storage. Finally, note that no distractors were presented in this experiment. It would be interesting to know whether correlations between memory performance and activation patterns in V1–V4 and sIPS are modulated by distractor presence.

In our view, the data reported by Bettencourt and Xu (2016) provide only modest support for the assertion that sIPS has a central or privileged role in mediating VSTM storage. This conclusion rests on a single null result (chance-level decoding performance in V1–V4 during predictable distractor-present blocks in Experiment 1), and the results of subsequent experiments instead support the conclusion that both V1–V4 and sIPS contribute to VSTM storage, both in the presence (Experiment 3) and absence (Experiment 4) of distractors. These latter results agree with a growing body of evidence suggesting that VSTM is mediated by coordinated activity across multiple cortical regions. For example, recent studies have documented feature-specific VSTM representations in a multitude of visual, parietal, and prefrontal cortical areas (Sprague et al., 2014; Ester et al., 2015). Critically, some of these areas show classic signatures associated with VSTM, such as elevated delay period activation, but many others do not (Riggall and Postle, 2012; Ester et al., 2015). Moreover, artificial perturbations (eg, via transcranial magnetic stimulation) of neural populations within visual (van de Ven et al., 2012), parietal (Hamidi et al., 2008), and prefrontal (Fregni et al., 2005) cortex during VSTM alter memory performance, suggesting a functional role for each of these regions. Determining what role(s) these different regions play in mediating VSTM storage under different contexts is an important goal for future research. However, we suspect that even simple mnemonic behaviors, such as active storage of a single item over a short delay, depend on coordinated activity between a multitude of cortical areas, rather than just one or two “privileged” sites.