Partial dissociation in the neural bases of VSTM and imagery in the early visual cortex.

Visual short-term memory (VSTM) and visual imagery are believed to involve overlapping neuronal representations in the early visual cortex. While a number of studies have provided evidence for this overlap, at the behavioral level VSTM and imagery are dissociable processes; this begs the question of how their neuronal mechanisms differ. Here we used transcranial magnetic stimulation (TMS) to examine whether the neural bases of imagery and VSTM maintenance are dissociable in the early visual cortex (EVC). We intentionally used a similar task for VSTM and imagery in order to equate their assessment. We hypothesized that any differential effect of TMS on VSTM and imagery would indicate that their neuronal bases differ at the level of EVC. In the "alone" condition, participants were asked to engage either in VSTM or imagery, whereas in the "concurrent" condition, each trial required both VSTM maintenance and imagery simultaneously. A dissociation between VSTM and imagery was observed for reaction times: TMS slowed down responses for VSTM but not for imagery. The impact of TMS on sensitivity did not differ between VSTM and imagery, but did depend on whether the tasks were carried concurrently or alone. This study shows that neural processes associated with VSTM and imagery in the early visual cortex can be partially dissociated.

Introduction

Visual short-term memory (VSTM) and visual imagery are both functions of the visuo-spatial working memory (VSWM) (Baddeley, 2003; Cornoldi and Vecchi, 2003). While VSTM refers to processes associated with maintaining visual information beyond the presentation of the target stimulus (e.g. Baddeley, 2003), imagery is a form of sensory experience occurring in the absence of perceptual input (Kosslyn, 1994). VSTM and visual imagery are closely related, as imagery enables the content of memory to be consciously experienced in the “mind's eye”. Indeed, there is much evidence to indicate that they both involve visual cortical neurons which encode incoming sensory information (Sparing et al., 2002; Slotnick et al., 2005; see e.g. Postle (2006) for review; Serences et al., 2009; Harrison and Tong, 2009; Van de Ven et al., 2012; Albers et al., 2013). For example, imagery of visual stimuli is associated with neuronal firing in the same neurons that are activated by the visual presentation of those stimuli (Kreiman et al., 2000), and the content of VSTM can be decoded from the activity patterns of the visual cortex (Harrison and Tong, 2009; Serences et al., 2009; Emrich et al., 2013). Thus imagery, VSTM, and the perception of external visual input all make use of overlapping resources in the visual cortex (cf.; “sensory–recruitment” model of working memory; e.g. Awh and Jonides, 2001; Postle, 2006; D'Esposito, 2007).

On the cognitive level, this overlap has been explained in terms of the visual cache, a component of the visuospatial sketchpad, which is involved in the maintenance of both VSTM and imagery content (Logie, 1995). In his model, Logie (1995) proposed a functional structure of the visuospatial sketchpad (VSSP), a structure previously elaborated by Baddeley and Hitch (1974). In this model, the VSSP comprises two subsystems; the inner scribe, a system responsible of actively rehearsing information that are spatially organized and/or in motion; and the visual cache, a system responsible of passively holding visual information related to forms and colors. Whereas VSTM representations based on a single input object are held in the visual cache, mental imagery generation depends on the visual buffer (Kosslyn and Thompson, 2003). In this view, after a mental image has been generated, its maintenance relies on the visual cache (Borst et al., 2012) where also VSTM content is actively rehearsed (Coleman and LeFevre, 2002). In addition, imagery requires the engagement of the central executive that enables the mental image to be maintained in conscious experience (Logie, 1995).

Even though VSTM and mental imagery may share neural/cognitive resources, they are nevertheless two distinct psychological processes that can be dissociated behaviorally. For example, dynamic visual noise presented concurrently with visual imagery generation impairs subjects’ performance in imagery tasks, whereas no impairment is found when visual noise is presented during a VSTM task (Quinn and McConnell, 1996; Andrade et al., 2002; Zimmer and Speiser, 2002). Recently, VSTM and mental imagery have also been dissociated with respect to their impact on the detection of concurrently perceived visual stimuli (Saad et al., 2013a). Whereas imagery suppresses external visual information (cf. Perky (1910)), VSTM appears to facilitate the encoding of matching visual input (e.g. Soto et al., 2005).

These dissociations raise the questions of where in the brain their processing diverges. The objective of the present study was to assess whether this divergence is present in the early visual cortex (EVC). This was tested using transcranial magnetic stimulation (TMS) as a probe of visual cortical activation state (e.g. Van de Ven and Sack, 2013; Silvanto and Muggleton, 2008; Sandrini et al., 2011; Cattaneo et al., 2010). We assessed two aspects of participants' performance: 1) accuracy of the memory for the original cue (conventional VSTM task); and 2) accuracy of the mental image. In the “alone” condition, either VSTM or imagery was assessed, of which participants were informed before each block. In the “concurrent” condition, participants were informed only at the end of the trial whether they would be asked to perform a comparison task relative to their mental imagery, or relative to their memory of the original memory cue. Thus maintenance of the original memory cue as well as engagement in imagery was required on all trials. The concurrent condition was carried out to understand how imagery and VSTM might interact when they are engaged simultaneously. In order to equate the assessment of VSTM and mental imagery we intentionally used the same task for both conditions. The difference, however, resides in the instructions. To monitor participant’s compliance with task instructions we administered a questionnaire at the end of the experiment addressing cognitive processes used during VSTM and imagery. We hypothesized that any differential effect of TMS on VSTM and imagery would indicate that their neuronal bases differ at the level of EVC.

Materials and methods

Participants

23 participants (9 females; mean age 25 years) with normal vision participated in the experiment. All were naïve to the aim of the study and provided written informed consent, in agreement with the Declaration of Helsinki and approved by the ethics committee of Aalto University. Participants received a monetary reward for their participation.

Stimuli

Stimuli and task were controlled by E-prime v2.0 (Psychology Software Tools Inc., Pittsburgh, USA; http://www.pstnet.com/eprime.cfm). All stimuli were sinusoidal luminance-modulated gratings (with a diameter of 5° of visual angle; generated with Matlab), presented foveally from a viewing distance of 57 cm on a gray background. The spatial frequency of the gratings was 1.44 cycles/degree. All gratings were vertical in orientation. The Memory/imagery cues had a Michelson contrast of either 0.2, 0.3, 0.4, or 0.5 and participants needed to hold the contrast in memory/imagery (specific instructions are described in the next section). The test cue presented at the end of each trial had a Michelson contrast that differed from the Memory/imagery contrast by either ±0.06 (“difficult” difficulty level) or 0.09 (“easy” difficulty level) of Michelson contrast. (For example, for a memory/imagery cue of 0.2, the test cues could be either 0.14 or 0.26 Michelson contrast (for the ±0.06 difference condition) and 0.11 or 0.29 Michelson contrast (for the ±0.09 difference condition). The mask was a uniformly black circle with the same diameter as the gratings (as used in previous studies; Saad and Silvanto, 2013a,b). The stimuli were presented on 22-inch screen with 1600×1200 pixel resolution.

Experimental sessions

3 types of blocks were run:

VSTM alone-assessment of memory for the original memory cue. In these blocks, participants were instructed to hold the cue contrast in memory, without engaging in imagery throughout the trial. In other words, participants were not required to maintain a conscious mental image throughout the delay period (in other words, phenomenal experience of the qualia of the memory content was not required during maintenance). At the end of the trial, they were required to judge whether the test cue was of lower or higher contrast than the original memory cue. We refer to this block as “VSTM alone”-block.

Imagery alone; assessment of accuracy of the conscious mental image. In these blocks, participants were asked to maintain conscious mental image of the original imagery cue throughout the delay period, where the mental image contrast was compared to the test cue. In other words, the contrast judgment was based on an online inspection of the mental image. As in these blocks the actual memory for the original memory cue was never assessed, we refer to it as “Imagery alone”-block, as strictly speaking it does not contain a conventional VSTM task. Thus the main difference between the two tasks is that, in “Imagery alone”, participants are asked to use visual imagery to perform the task, by keeping the imagery cue in phenomenal experience throughout the delay period. This was not required in “VSTM alone” condition.

Concurrent VSTM and imagery – assessment of either VSTM accuracy and accuracy of the conscious mental imagery (similar to Slotnick et al., 2012; Saad & Silvanto 2013a). In these blocks, participants were informed at the end of the maintenance period whether memory or imagery would be assessed.

Within the same day of testing, two sub-sessions were carried out for each participant. In session 1, conditions 1 and 2 were run (see Fig. 1A). For both conditions, 2 blocks of 32 trials were run for both TMS conditions (Early Visual Cortex, Sham). In Session 2, condition 3 was run in 4 blocks of 32 trials for both TMS conditions (see Fig. 1B). The order of sessions was counterbalanced, as was the order of blocks within each session. Each block contained Memory/imagery main cues of four different contrasts (0.2, 0.3, 0.4, and 0.5 Michelson contrast). The contrast difference between the test cue and the main cue was either ±0.06 or ±0.09.

Fig. 1

Timeline of an experimental trial. At the start of each trial, participants were presented with a cue (a vertical grating). The task involved maintaining the contrast of the grating by holding it in memory and/or forming a conscious mental image of it and maintaining it throughout the maintenance period. TMS pulse train was applied 2.5 s after the onset of the maintenance period. At the end of each trial, participants were asked to judge the test cue contrast relative to VSTM/imagery content (i.e. is the test cue of lower or higher contrast). (A) In “VSTM alone” and “Imagery alone” blocks, the assessment of memory and imagery were carried out in separate blocks. (B) In “concurrent” blocks, participants were informed at the end of the trial whether memory for the original memory cue would be assessed, or whether they should perform the contrast discrimination task relative to their conscious mental image.

General procedure

Each trial began with a fixation point (1 s), followed by the Memory/imagery cue (300 ms). The cue was a vertical sinusoidal grating and had a Michelson contrast of either 0.2, 0.3, 0.4, or 0.5. To avoid any afterimage induction by this cue, a mask (a uniformly black circle, appeared after the offset of the cue for 100 ms). The imagery/memory contrast (depending on the experimental condition) was then assessed by a forced choice task. On each trial, participants were asked to judge whether the contrast of the test cue was lower (press 1) or higher (press 2) than the original memory cue or their mental image.

TMS stimulation and site localization

TMS was delivered using a Magstim rapid2 (Magstim super Rapid Plus, Magstim company, UK) using a figure-of-eight 70-mm air-cooled coil. The coil was held using a custom-made magic-arm and placed tangentially on the skull. To stimulate the early visual cortex, the coil was placed 2 cm above the inion and 0.5 cm laterally on the right hemisphere, and the coil position was slightly adjusted such that participants reported phosphene in the location where stimuli would appear in the main experiment (see e.g. Pascual-Leone and Walsh (2001), Campana et al. (2002) for this approach). Participants who did not perceive phosphenes (n=9) were stimulated with the above coordinates. For Sham TMS, the coil was placed above the central parieto-occipital (POz) electrode region, and foam was used to increase the distance between the coil and the skull. For half of the participants, MRI scans were available and these were used to confirm that the location of stimulation was in the vicinity of the calcarine sulcus via a neuro-navigation system. Phosphene thresholds were measured for each participant using a Modified Binary Search Paradigm (MOBS; Tyrrell and Owens, 1988). TMS intensity was adjusted for each participant such that an intensity of 90% of the phosphene threshold was used. Participants who did not perceive phosphenes were stimulated with an intensity of 65% of machine output (as used in a prior study; Saad and Silvanto, 2013b). None of the participants reported phosphenes during the experiment. On each trial, a pulse train (consisting of five pulses applied at 10 Hz; i.e, pulse gap of 100 ms; (e.g. Ashbridge et al., 1997; Campana et al., 2002, 2006; Muggleton et al., 2003; Saad and Silvanto; 2013b) was applied 2.5 s after the onset of the 4 s maintenance period. This specific time window was used in order to allow sufficient time for the generation of the mental image and to avoid a close temporal proximity of the TMS pulse train with the test cue.

Questionnaire assessing task strategy

At the end of the experiment, participants were asked to fill a questionnaire assessing their cognitive strategies during the experiment and ensure that instructions were followed. The questions were as follows: For VSTM: “Please describe in detail how you memorized the original cue; what strategy or process did you follow until asked to judge your memory of the cue”. For imagery: “Please describe in detail how you formed the mental image and what strategy or process did you follow until asked to judge your image of the cue?”. For Condition 3, participants were asked the following question: “Where you able to memorize and make a mental image of the main cue?” Please describe in detail how you memorized the memory/imagery cue and made a mental image of it; what mental strategy or process did you follow until asked to judge your memory/imagery of the cue? A representative response for VSTM and imagery were as follows:VSTM: “I memorized the memory cue and waited until the test without doing any mental process; somehow it was unconscious. At test, I tried to dig out the memory”. Imagery: “I made a mental image of the imagery cue, kept looking at it in my head through focusing and repetition the whole time, until the test”.

On the basis of this questionnaire, five participants were excluded from the data analysis. Two participants were excluded because they had not used imagery when required. Three participants reported using the same maintenance process across all sessions (e.g. using imagery even in VSTM alone blocks and vice versa). The rest of the participants (n=18) reported having followed task instructions.

Results

Three participants were removed due to reaction times being of 3 SD above the mean across conditions; therefore both the reaction times analysis and sensitivity analysis were conducted on 15 participants.

Overall effects of VSTM and imagery on sensitivity

Fig. 2(a–d) shows the mean (n=15) sensitivity (d′) for VSTM and imagery as a function of TMS site and difficulty level. We initially carried out an ANOVA into which all independent variables were entered. This 2×2×2×2 ANOVA, with task (imagery or VSTM), condition type (alone or concurrent), TMS site (EVC or sham), and difficulty (easy or difficult) revealed a main effect of difficulty (F (1,14)=63.62; p<0.001), condition type (F (1,14)=63.75; p=0.02), and a 2-way interaction between condition type and TMS (F (1,14)=21.36; p<0.001). None of the other main effects or interactions were significant.

Fig. 2

Mean (n=15) of sensitivity (d′) as a function of TMS site and contrast difficulty level. (A) VSTM in “alone” condition. (B) Imagery in the “alone” condition. (C) VSTM in “concurrent” condition. (D) Imagery in “concurrent” condition. TMS facilitated the sensitivity of both Imagery and VSTM when conducted separately in the alone condition. Error bars indicate ± SEM from which between-subjects variance has been removed.

To understand the nature of these effects, we carried out post-hoc comparisons. In these t-tests we collated the data across tasks (Imagery, VTSM) and the difficulty levels (easy or difficult) as neither factor was involved in significant interactions in the ANOVA. These pairwise comparisons revealed that, in the alone condition, EVC-TMS enhanced the sensitivity relative to sham (t (14)=−5.80; p<0.002); in contrast, in the concurrent session, EVC-TMS did not modulated the sensitivity relative to sham (t (14)=1.3; p=0.42).

To investigate whether baseline performance (i.e. sham TMS condition) were modified across condition type we conducted a 2×2×2 ANOVA in which we entered condition type (alone or concurrent), task (VSTM or imagery), and difficulty level as independent variable. This revealed a significant effect of difficulty level (F (1,14)=16.27; p=0.001). However, no other main effect or interaction was found (highest p-value 0.9).

In summary, EVC-TMS enhanced the sensitivity of both VSTM and imagery when conducted separately. In contrast, TMS had no impact on sensitivity in the concurrent condition. The baseline performance level of imagery and VSTM did not differ, and was not modulated by the task (i.e. alone or concurrent).

Overall effects of TMS on reaction times

Fig. 3(a–d) shows the mean (n=15) median reaction time during VSTM and imagery conditions as a function of TMS site and contrast difficulty level.

Fig. 3

Mean (n=15) of median reaction time (ms) as a function of TMS site and contrast difficulty level. (A) VSTM in “alone” condition. (B) Imagery in the “alone” condition. (C) VSTM in “concurrent” condition. (D) Imagery in “concurrent” condition. TMS significantly slowed down reaction times for VSTM but not for imagery. The Error bars indicate SDs from which between-subjects variance has been removed (Loftus and Masson, 1994).

We initially carried out an ANOVA into which all independent variables were entered. This 2x2x2×2 ANOVA, with task (imagery or VSTM), condition type (alone or concurrent), TMS site (EVC or sham), and contrast difficulty (easy or difficult) revealed a main effect of condition type (F (1,14)=26.61; p<0.001), difficulty level (F (1,14)=17.50; p=0.001), a 2-way interactions between Task and TMS site (F (1,14)=9.5; p=0.009), and a 2-way interactions between difficulty level and TMS site (F (1,14)=8.64; p=0.01). None of the other main effects or interactions were significant.

As task was interacting with TMS site, we conducted separate ANOVAs for each task selectively in order to investigate these effects. For VSTM, we conducted a 2×2×2 ANOVA with condition type (alone or concurrent), TMS site (EVC or sham), and contrast difficulty (easy or difficult). This revealed a main effect of condition type (F (1,14)=26.4; p<0.001), difficulty level (F (1,14)=5.8; p=0.031), and TMS site (F (1,14)=9.23; p=0.009). None of the interactions were significant. The main effect of TMS indicates that TMS induced a slowing down of RTs for VSTM.

For imagery, we conducted a 2×2×2 ANOVA with condition type (alone or concurrent), TMS site (EVC or sham), and contrast difficulty (easy or difficult). This revealed only a main effect of condition type (F (1,14)=28.5; p<0.001). None of the other interactions were significant. Thus TMS had no effect on RTs for imagery.

In sum, these results show that TMS applied at EVC increased reaction times relative to Sham in the VSTM task. No such effect was found for imagery.

Discussion

The aim of this study was to examine whether the neural bases of VSTM and imagery are dissociable in the early visual cortex. VSTM and imagery differed in terms of the level of phenomenal awareness of the memory/imagery content during the maintenance period. In “imagery” conditions, participants were asked to perform the task by keeping the imagery cue in phenomenal experience throughout the delay period; this was not required in “VSTM alone” condition. Our results can be summarized as follows: in the “alone” condition (i.e. when participants knew in advance of each block whether VSTM or mental imagery would be assessed at the end of the trial), TMS over the early visual cortex increased the sensitivity of both VSTM and imagery. In contrast, TMS had no effect on sensitivity in the “concurrent” condition (where both VSTM and imagery maintenance was required on each trial). A dissociation between VSTM and imagery was present in the reaction times. Whereas TMS increased reaction times for VSTM, no such effect was found in the imagery task. Condition type did not interact with the impact of TMS on RTs. Thus while the impact of TMS on sensitivity was similar for VSTM and imagery (but differed between alone and concurrent conditions), the impact of TMS on reaction times was different for VSTM and imagery, independently of condition type.

The facilitatory effect of TMS on sensitivity in the “alone” condition is consistent with previous findings on the overlap of imagery and VSTM in the EVC (Kosslyn et al., 2006, Gains et al., 2009), and prior demonstrations of TMS-induced facilitations in VSTM and imagery paradigms (Cattaneo et al., 2012; Silvanto and Cattaneo, 2010; Silvanto and Soto, 2012). For example, phosphene studies have shown TMS to facilitate the features contained in both imagery (Sparing et al., 2002; Cattaneo et al., 2011) and in VSTM (Silvanto and Cattaneo, 2010).

However, the results of this study beg the question of why the effects on sensitivity and reaction times were qualitatively very different. It is generally assumed that the impact of TMS on these two measures occurs via the same mechanism. TMS is believed to act by indiscriminately activating neurons in the targeted region, thereby adding noise to the highly organized pattern of neural activity associated with perceptual processes (see e.g. Walsh et al., 2003; Ruzzoli et al., 2010). This can slow down reaction times, as more time is needed to accumulate the necessary level of evidence required for the discrimination judgment, due to the increased amount of noise. Induction of noise can also reduce sensitivity by reducing the quality of the sensory representation on which the discrimination judgment is based. Of these two measures, reaction times are generally more sensitive to TMS-induced disruption, possibly because even small amounts of noise can slow down evidence accumulation, whereas the sensory signal might have sufficient redundancy to deal with low noise levels (Walsh et al., 2003). In this view, whenever accuracy is reduced, an effect on reaction times should also be observed, as the latter is more susceptible to disruption. Our results are inconsistent with this, as TMS induced a general slowing down of RTs only for VSTM (for both alone and concurrent condition), whereas it facilitated sensitivity for both VSTM and imagery, but only in the “alone” condition,

To account for these results, it thus appears to be necessary to postulate that reaction times and sensitivity reflect distinct neural processes. What might these be? Successful performance in the discrimination task consisted of at least two components: firstly, maintaining an accurate memory/imagery representation, and secondly, accessing it for conscious inspection for comparison against the test cue. The key difference between items in VSTM and imagery is that the latter are already in the conscious domain (Logie, 1995), and thus do not require a separate stage of retrieval before they can be compared with the test cue. In contrast, VSTM content needs to be consciously accessed for the discrimination task to be performed. TMS in the present study had two distinct effects: one on the actual memory/imagery representations (reflected in enhanced sensitivity, and found for both VSTM and imagery), and a second on the process of conscious retrieval (reflected in reaction times). It might be that for the latter, TMS had no impact in the imagery task because the mental image is already in the conscious domain and therefore such retrieval is not needed. However, VSTM content needs to be re-accessed in order for the discrimination task to be performed. In this study VSTM maintenance did not entail efforts in regenerating an image of the cue once the iconic memory had faded. For this reason, the VSTM task (but not the imagery task) involves a separate stage of retrieval of accessing memory content and the ease of this process might have been affected by TMS.

Interestingly, whereas EVC TMS facilitated VSTM and imagery in the “alone” condition, it had no impact when they were carried out concurrently. This indicates that the memory/imagery trace is in a different neural state when the two are engaged simultaneously, compared to “VSTM alone” and “Imagery alone” conditions. One possibility is that, during simultaneous VSTM and imagery, the underlying memory trace on which both items are based is stronger. It could be that with sufficiently strong representation, TMS is no longer able to enhance it further. However, what argues against this explanation is that baseline level of performance of either VSTM or imagery was not higher in the “concurrent” condition relative to the “alone” condition, which this view would predict.

Another possibility is that VSTM and imagery involve different representations and that during the maintenance period, these interact. If the imagery representation is derived from the VSTM representation, then one would expect VSTM content to constantly update the mental image. This interaction could be bidirectional, with the mental image itself strengthening the underlying VSTM trace. An interaction between VSTM and imagery might have modulated the nature of both the VSTM and imagery traces, rendering them both differentially susceptible to TMS relative to the “alone” condition. In this view, VSTM and imagery would be based on partly distinct representations.

In summary, the key finding of the present study is that TMS had a differential impact on the reaction times of VSTM and imagery, dissociating these processes at the level of the early visual cortex. While the current literature often emphasizes the visual cortical overlap in neural resources for VSTM and imagery, our study demonstrates that differences between these two cognitive functions exist not only at the “high” level of executive functions (e.g. Logie, 1995), but also at the level of the visual representations.

Contribution(s)

Designed research: ES, JS; Performed research: ES, MW; Analyzed data: ES, JS; Wrote the paper: ES, JS.

Conflict of Interest

The authors declare no competing financial interests.