Severe distortion in the representation of foveal visual image locations in short-term memory.

The foveal visual image region provides the human visual system with the highest acuity. However, it is unclear whether such a high fidelity representational advantage is maintained when foveal image locations are committed to short-term memory. Here, we describe a paradoxically large distortion in foveal target location recall by humans. We briefly presented small, but high contrast, points of light at eccentricities ranging from 0.1 to 12°, while subjects maintained their line of sight on a stable target. After a brief memory period, the subjects indicated the remembered target locations via computer controlled cursors. The biggest localization errors, in terms of both directional deviations and amplitude percentage overshoots or undershoots, occurred for the most foveal targets, and such distortions were still present, albeit with qualitatively different patterns, when subjects shifted their gaze to indicate the remembered target locations. Foveal visual images are severely distorted in short-term memory.

The representation of visual spatial locations in short-term memory has been one of the most classic means for studying the neural mechanisms of cognitive control in modern-day systems neuroscience (1–6). Experimentally, reporting a remembered target location with a saccadic eye movement has proven extremely useful in demonstrating how different cortical and subcortical brain regions may maintain a memory trace of visual targets (7), and it has also been equally important for oculomotor control studies in dissociating sensory and motor responses (8–11).

One mechanism for short-term memory maintenance in multiple brain areas is the persistence of neural activity associated with remembered stimulus locations, even in the absence of sensory drive (7, 12). Such persistence, with temporal drift, can help explain a variety of distortions in memory-based task performance, for example, as a function of how long a location needs to be maintained in short-term memory (12). In addition, such persistence can reveal certain systematic biases with respect to whether stimuli occupy a single visual quadrant or multiple visual quadrants (13), which in turn enables linking visual field asymmetries in perception to representations of remembered target locations. For example, visual performance along the horizontal and vertical retinotopic meridians is different (14–16), and this difference is maintained in tasks involving short-term memory (17). The fact that such visual meridian effects may be related to tissue magnification in visual cortical areas (18–20) might then suggest that other known distortions in short-term memory tasks, such as foveal biases in remembering peripheral target locations (21–23), may also be related to how visual space is represented in topographic maps.

If remembering visual target locations depends on both how visual space is topographically represented as well as on how memory information is neurally maintained, then an important remaining open question is whether foveal visual locations are recalled veridically or not, given the normally very high acuity nature of foveal vision in humans. Here, we investigated this question and found that remembering a foveal visual location as near to the line of sight as 0.1° (6 min arc) is subject to severe distortion, even after very short memory delay intervals. This paradoxical foveal distortion in visual short-term memory emerges with or without a constant landmark being available during the response phase of the task, and, perhaps most importantly, it is also qualitatively different depending on whether the remembered location is reported with an eye movement response or with another response modality.

Results

We asked human subjects to fix their gaze on a small, central spot. During fixation, we briefly flashed, for 59 ms, a similar spot at a pseudorandom location between 6 min arc and 12° of eccentricity. After a memory interval of 300–1100 ms, the fixation spot disappeared, and a response cursor appeared in its place. The subjects moved the response cursor via a button box to the remembered location of the flash, and they were free to move their eyes during cursor control (). In a previous analysis (11), we reported a general overshooting behavior in the subjects’ reported locations for foveal targets. Here, we analyzed the spatial layout and geometry of localization performance in detail, and we found a large two-dimensional distortion in the reported locations for targets presented at foveal visual eccentricities, very near to the line of sight. Such distortion also occurred: when we changed the response modality for indicating the remembered locations; when we maintained a central visual landmark during the subjects’ response execution; and also when we asked the subjects to indicate the remembered locations with eye movements.

A Strong Diagonal Bias and Eccentricity Overshoot in Remembered Foveal Target Locations.

Fig. 1 shows all possible target locations that we tested in our button cursor movement experiment (). We densely sampled foveal eccentricities in all directions, and we also sampled extrafoveal locations. In the Left two panels of Fig. 1 (labeled “Foveal targets”), we plotted, for two example foveal target locations (black circles), a sampling of response indications by the subjects (faint red dots indicating individual trials). The average response location for each target, our variable of interest in this study (), is indicated by a saturated red circle. For the purely diagonal target at an eccentricity <15 min arc (Leftmost panel in Fig. 1; “Diagonal”), the average memory-guided response was directionally accurate, but it exhibited an amplitude overshoot. For the second foveal target location at an angle of 30° below the right horizontal meridian (and still with an eccentricity <15 min arc), the average response not only overshot the target (11), but it was also strongly biased in direction toward the diagonal axis (Middle panel; compare red and black lines). This diagonal bias largely disappeared for the third example target location shown in Fig. 1 (Rightmost panel); this target location still had a direction of 30° from the horizontal meridian, but it now was at a much larger extrafoveal eccentricity >5° (Rightmost panel). Therefore, when remembering foveal visual target locations, our subjects exhibited a strong and diagonal bias for oblique locations, along with an eccentricity overshoot (11), which was much reduced at larger extrafoveal eccentricities.

Fig. 1.

A strong distortion in remembered foveal target locations. (A) All target locations tested in our short-term memory button cursor experiment. Each dot shows a potential remembered target location on a log-polar plot. In each trial, a brief flash appeared at one location. After a few hundred milliseconds, the display was blanked, and the subjects had to move a cursor to the flash’s remembered location. (B) Example response biases in remembered target locations. The Left two panels show two example foveal targets at an eccentricity <15 min arc. Each target is shown as a black circle (with a black line connecting display center to the target). The faint red dots show example response locations. For the Leftmost panel, the responses overshot the target, but their directions were (on average) correct (the red circle shows the average response location). For the Middle panel, a similar foveal target eccentricity had a nondiagonal direction from the horizontal meridian; here, the responses were additionally strongly biased toward the diagonal axis. The blue arrow shows the mislocalization that took place (connecting target location to average response position). The Rightmost panel shows an example nondiagonal target location (similar to the Middle panel: 30° direction from the horizontal meridian), but now at a much larger eccentricity (note the different eccentricity scale). In this case, the responses were directionally accurate, and there was no overshoot.

To visualize results across all target locations and eccentricities, we plotted, for each target, a vector connecting its true location and the average response location (similar to the blue arrow in the Middle panel of Fig. 1). The results, across data from all seven subjects, are shown in Fig. 2. Noncardinal foveal target locations were strongly skewed toward the diagonal direction in each visual quadrant, and this happened at all tested foveal eccentricities (Fig. 2). Note how there was always an amplitude overshoot, although such an overshoot was quite small for the target locations along the cardinal axes (purely horizontal and purely vertical directions). Moreover, the amplitude overshoot increased in size for the diagonal foveal targets, for which the direction errors were minimal (the response vectors for the purely diagonal targets were still largely diagonal, resulting in minimal direction errors). Therefore, complementary to known foveal biases in visual short-term memory representations for peripheral, extrafoveal targets (21–23), we saw strong overshoots in remembering target locations with small, foveal visual eccentricities. Most critically, we concurrently observed a severe diagonal bias in all noncardinal foveal target locations.

Fig. 2.

Distorted representation of foveal visual image locations in short-term memory. (A) Systematic mislocalization of remembered foveal target locations as revealed by a quiver plot. The origin of each vector is the true target location, and the end of it (the arrowhead) is the average indicated location (similar to the blue vector shown in the Middle panel of Fig. 1). There was an amplitude overshoot for all tested foveal target locations (outward arrows from the origin in all directions). Moreover, there was a strong diagonal bias for noncardinal target locations. Cardinal targets on the horizontal and vertical meridians were associated with the least errors. (B) Same data as in (A) but after remapping all targets and responses to one quadrant (and after including parafoveal eccentricities). Note how the cardinal targets between 2 and 3° eccentricity were associated with a foveal bias (undershoots), but oblique targets at the same eccentricities were still associated with amplitude overshoots. (C) The gray region shows the data from (B), and the rest of the figure shows results from more eccentric locations. The red dashed line shows that noncardinal (oblique) targets flipped from overshoots to undershoots at an eccentricity of 4°. shows consistent results from individual subjects.

For further characterizing this strong diagonal bias, we exploited the general symmetry of the effect shown in Fig. 2. We, therefore, remapped the target and response locations to one quadrant, for easier representation, and we plotted the memory-based mislocalizations. The results are shown in Fig. 2. The strong diagonal bias at foveal eccentricities was robustly observed. Interestingly, a qualitative flip in memory-based localization error from being a response amplitude overshoot at foveal eccentricities to a response amplitude undershoot at higher eccentricities (the known foveal bias) occurred along the cardinal axes at an eccentricity of 2° (Fig. 2); that is, overshoots in memory-based response localization occurred at less than 2° target eccentricities, whereas foveal biases (undershoots) occurred at more than 2° target eccentricities. However, noncardinal locations still exhibited amplitude overshoots at eccentricities larger than 2°. For example, in Fig. 2, eccentricities between 2 and 3° showed amplitude overshoots for noncardinal remembered target locations, but they showed foveal biases (undershoots) at cardinal target locations. The flip from overshoots to foveal biases in memory-based target localizations, for noncardinal target locations, occurred at an eccentricity of 4° (red dashed line in Fig. 2). In terms of direction errors, the diagonal biases for noncardinal target locations persisted even for eccentricities >10°, although they were milder than in the foveal visual representation (Fig. 2; also see Fig. 3).

Fig. 3.

Direction and amplitude errors when remembering foveal and extrafoveal visual target locations. (A) For all remapped target and response locations (as in Fig. 2 ), we plotted direction error as a function of target direction from the horizontal meridian. Direction error was the difference between the average response direction and true target direction. The blue curve shows results from foveal target eccentricities. Cardinal target directions were associated with minimal, but nonzero, direction errors. Noncardinal target directions were associated with a strong diagonal bias, with near-horizontal targets having positive direction errors and near-vertical targets having negative direction errors (shaded regions). Direction error was practically zero at purely diagonal target locations. For extrafoveal target locations (green and orange), cardinal direction errors were essentially eliminated. Noncardinal diagonal biases were much milder than in foveal eccentricities. (B) Same analyses but for response amplitude gain (ratio of average response eccentricity to true target eccentricity). Foveal target eccentricities (blue) exhibited strong overshoots, which peaked at purely diagonal directions. Eccentric targets (>3°) were associated with foveal biases (undershoots) (21–23). Eccentricities between 2 and 3° (green) had mild overshoots for noncardinal directions, as in Fig. 2. Error bars denote SEM across all observations. shows the results from individual subjects, and shows the effects of memory interval length on response distortions.

To further quantify the direction and amplitude errors in memory-based target localization, we binned target locations into different direction and amplitude bins. For foveal targets (<2° eccentricity), we created a running average of direction bins (15° bin widths and with running steps of 5°). In these bins, 0° direction would represent horizontal target locations and 90° direction would represent vertical target locations (as in the remapped representations of Fig. 2 ). For each direction bin, we then calculated the average direction error between response direction and target direction (). The resulting curve is shown in Fig. 3 (<2° target eccentricity). Other than for purely cardinal target locations, foveal targets close to the horizontal meridian (with directions less than 45°) were associated with positive direction errors (which systematically decreased in amplitude with increasing target direction), and foveal targets close to the vertical meridian (with directions larger than 45°) had negative direction errors. These results are consistent with an overall directional bias toward the diagonal axis (shaded regions in Fig. 3; consistent with Fig. 2). For purely diagonal targets, the direction errors were minimal.

Interestingly, horizontal target locations (0° direction on the x-axis in Fig. 3) were associated with lower absolute direction errors than vertical target locations (90°) (0.654° ± 0.2305° SEM vs. 1.5° ± 0.3043° SEM direction error; an ∼2.3-fold difference). This observation is reminiscent of better visual performance on the horizontal versus vertical visual meridian studied at larger peripheral eccentricities (14, 16, 17), but here it was found in a memory-guided localization task and also extended to the foveal visual image region.

As for amplitude errors, we used the same directional binning as in Fig. 3 and now measured the ratio of average response eccentricity to target eccentricity. For foveal targets (<2° target eccentricity), there was up to ∼35% amplitude overshoot when remembering foveal visual target locations (Fig. 3). The overshoots were largest for the purely diagonal targets (Fig. 3), for which the direction errors were minimal (Fig. 3), and the overshoots were the smallest for purely cardinal target locations. Moreover, overshoots for purely horizontal targets were less strong than overshoots for purely vertical targets (8.1 ± 1.18% SEM vs. 15.8 ± 1.48% SEM; a 1.95-fold difference), again reminiscent (in terms of superior horizontal vs. vertical meridian performance) of visual meridian effects reported at larger eccentricities (14, 16, 17). All of these results are consistent with the observations summarized in Fig. 2.

The strong diagonal biases and amplitude overshoots that we observed were also most significant for foveal visual target locations. For larger remembered target eccentricities (extrafoveally), diagonal biases for oblique targets were still present in Fig. 3 (see the data points for 2–3° target eccentricities in green and >3° target eccentricities in orange), but they were much smaller in size than for foveal targets. For example, at 30° and 60° directions from the horizontal meridian, the absolute direction errors for targets >3° in eccentricity were 1.74° ± 0.241° SEM and 1.14° ± 0.251° SEM, respectively, but they were 6.29° ± 0.255° SEM and 5.52° ± 0.257° SEM for targets <2° in eccentricity (Fig. 3). This was a factor of 3.6–4.8 times difference. Moreover, in terms of response amplitudes, targets at eccentricities >3° were consistently associated with known foveal biases (amplitude undershoots) (21–23), which is the exact opposite of the overshoots that we observed for targets <2° in eccentricity.

Therefore, remembering recently presented visual target locations at foveal visual eccentricities is associated with strong distortions. These distortions are much larger in size, relative to the target eccentricities (and also opposite in sign in terms of amplitude errors), than distortions of more eccentric target location representations.

We also confirmed that the results in Figs. 2 and 3 were robust at the individual subject level. For example, shows representations of memory-based foveal target mislocalizations from four individual example subjects (in a format similar to that used in Fig. 2). Moreover, show all individual subject curves from the analyses of Fig. 3 for the foveal targets. All subjects showed a clear direction bias toward the diagonal axis, and they all showed amplitude overshoots, which were weakest for the purely cardinal target locations. Moreover, even though memory intervals in our task were relatively short (<1100 ms; ), we still observed a larger amplitude of mislocalization errors with larger memory interval lengths (), and this effect was not explained by a potential difference in response reaction times ().

Finally, we conducted a further control experiment in which we addressed a potential concern that our effects may have been influenced by the strong bias to sample foveal target locations across trials. Specifically, our target locations in Fig. 1 were more often foveal than extrafoveal, and with dense sampling (including of the 45° diagonal lines). While we did this on purpose, in order to explore the representation of foveal space in great detail, it could be argued that the subjects may have exhibited distorted reports just because of the skewed sampling to include more foveal targets. As a result, in our control experiment, the target locations were more often extrafoveal (two-thirds of the time), and we also never sampled purely diagonal foveal or extrafoveal locations (). We still replicated all of the results of Figs. 1–3, whether with a subject who had participated in the original experiment () or with completely new subjects (). Therefore, there is a large and robust distortion in the representation of the foveal visual image region in short-term memory.

Persistence of Foveal Image Distortions in Short-Term Memory with a Foveal Visual Landmark.

In the above results, there was no visual landmark (other than the response cursor) during the final response phase of the task, and the subjects were also free to move their eyes during this phase. Moreover, cursor movement could take up to several seconds to reach the location intended by the subjects ( and ). We, therefore, asked whether the distortion that we observed so far (Figs. 1–3 and ) was caused by any or all of the above factors. We conducted a second experiment, which only differed in the response modality. In this mouse pointing variant of the experiment, the trial ended with the persistence of the central fixation spot on the display, and the appearance of a computer mouse pointer (at a random location; see ). The subjects were instructed to maintain gaze on the central spot and use the computer mouse to point and click at the remembered target location. We used a similar range of foveal and extrafoveal target locations (Fig. 4). Therefore, in this version of the task, we maintained gaze position near the original location (thus maintaining, on average, retinotopic correspondence during the visual presentation and response phases of the trials); we kept a foveal visual landmark on the display during the response phase, just like there was a foveal visual landmark at target flash onset; and, finally, we ensured a much faster reaction time than with cursor movement ().

Fig. 4.

Similar distortion of foveal visual location memory with a different response modality and a foveal visual landmark during the response phase. (A) Target locations from our mouse pointing task. The trials were identical, except that during the response phase, the central fixation spot remained visible, and subjects were to maintain gaze on it. The subjects responded with a mouse click at the remembered target location. (B) A similar diagonal bias as in the original experiment (Fig. 2) can be seen. (C and D) Analyses similar to those of Fig. 3 showing similar results to the original experiment, but the errors were generally smaller in the current experiment. Error bars denote SEM across all responses. shows the influence of memory interval length on the response distortions, and shows results from individual subjects.

We replicated all of the results from the first experiment. Specifically, even though the localization errors were smaller, in general, we still found a diagonal bias for remembered foveal visual eccentricities (Fig. 4 ). This bias was also significantly weaker for extrafoveal locations (Fig. 4), again consistent with Figs. 1–3. Similarly, foveal visual eccentricities were associated with substantial overshoots of target locations, which were absent, and sometimes replaced by undershoots (known foveal biases), at extrafoveal eccentricities (Fig. 4). Moreover, when we evaluated the influence of memory interval duration like we did above, we again found an increased overshoot distortion with longer memory intervals as in the first experiment (), and this effect was not explained by altered reaction times (). Finally, all of these results were robust at the individual subject level (): all subjects showed a diagonal direction bias in the fovea, and all but one subject showed amplitude overshoots. Importantly, four of seven subjects who performed this version of the task also performed the earlier one, and highlight one subject’s results in both variants of the experiment (with consistent observations).

Therefore, all of the primary results from Figs. 1–3 and were replicated with an altered response modality in which we maintained retinotopic correspondence during stimulus presentation and response execution, and when we also maintained visual correspondence by keeping the foveal visual landmark persistent on the display during both phases of the task.

Qualitatively Different Memory-Based Foveal Distortions when Reporting Target Locations with Saccades.

We finally asked whether distortion of the foveal visual image region during our memory-guided localization task was specific to manual responses. We repeated the same mouse pointing experiment (Fig. 4), but this time while removing the fixation spot at trial end (and not showing any mouse pointer). Subjects were instructed to make a saccade, including microsaccades (11, 24), toward the remembered target location, making this task the same as that used in a large number of classic short-term memory studies. The subjects readily made the memory-guided saccades, even for small foveal eccentricities, as we recently described (11, 24). Moreover, their reaction times were much faster than in the manual versions of the task (). So, we asked whether their endpoints also revealed strong distortions of the foveal visual image region during oculomotor-based memory localization.

Fig. 5 shows the mislocalizations that we observed. It was immediately evident that the foveal visual image region was also severely distorted. However, there was an upward bias in small saccade endpoints, which would have been masked by remapping all quadrants into a single quadrant (like we had done above). Therefore, in Fig. 5, we remapped the target and response locations into one hemifield, rather than into one quadrant, in order to maintain representation of lower visual field target locations. Foveal targets in the lower visual field (like upper visual field foveal targets) were now associated with an upward bias in small eye movement endpoints. Therefore, oculomotor-based reporting of target locations from short-term memory was still associated with systematic distortions of the foveal visual image region, but the nature of the distortions was qualitatively very different.

Fig. 5.

Indicating remembered foveal visual target locations by small saccades is also strongly distorted, but in a qualitatively different manner. (A) We repeated the same memory task, but trial end was marked by the removal of the central fixation spot with no other targets; subjects made a saccade toward the remembered target location (which was now blank). We plotted average (foveal) saccade endpoint in relation to true target location, similar to Fig. 2. However, to avoid masking the upward bias that was evident, we now remapped targets and responses into one hemifield instead of one quadrant. Targets in the lower visual field were associated with upward displacements in saccade endpoints, which would have been masked with remapping to a single quadrant. Amplitude overshoots were still evident. (B) Response direction from the horizontal meridian as a function of target direction for the right and left visual hemifields. Foveal targets (blue curve) in short-term memory were associated with a systematic upward bias (shaded regions). More eccentric targets did not show such a strong bias (green and orange). (C) Response amplitude gain for the same target locations. Foveal targets (blue curve) were associated with amplitude overshoots, as in Figs. 1–4. The overshoots got progressively stronger for more upward target locations (consistent with the idea of an overall upward bias for downward targets effectively shortening the intended saccades). Error bars denote SEM across all responses. show individual subject results, as well as demonstration of the qualitative difference in foveal memory distortion between saccadic and manual responses.

To summarize the upward bias that we observed with saccadic memory-based localization, we picked, for all foveal target eccentricities (<2°), different target directions from the horizontal meridian, and we plotted the direction of the final response as a function of them (blue curve in Fig. 5). Response directions were consistently biased upward (shaded regions in Fig. 5), except for target directions that were near the purely upward meridian: an upward bias for purely upward targets would not bias the response direction, but it would increase the amplitude overshoot. Indeed, in Fig. 5, we plotted the response amplitude gain for the same data, and we found that the amplitude overshoots associated with remembering foveal visual targets were the largest for the target directions that were near the purely upward direction (blue curve). Thus, oculomotor-based readout of foveal visual target locations from short-term memory was still associated with a strong distortion (including amplitude overshoot), but it was dominated by an upward bias rather than a diagonal one.

For extrafoveal target eccentricities, the upward bias was much smaller in amplitude than for foveal targets (green and orange in Fig. 5), consistent with a significantly milder distortion of extrafoveal eccentricities in our earlier experiments as well (Figs. 1–4). Also consistent with these earlier experiments was the change from amplitude overshoots to amplitude undershoots for extrafoveal targets, which we still observed in the saccade version of the experiment (green and orange in Fig. 5). Finally, the results in Fig. 5 were robust at the individual subject level (), and we confirmed that within individuals that performed all three variants of the experiments (), the qualitative change in distortion pattern for foveal remembered visual target locations between manual and saccadic responses was still clearly visible ().

Discussion

We observed a seemingly paradoxical distortion of the highest acuity region of the visual image in short-term memory. Attempting to report a remembered foveal visual location after only a few hundred milliseconds from its presentation was associated with significant overshoots in its reported eccentricity, as well as strong biases toward the diagonal axes for noncardinal target locations. Relative to the original target eccentricities, the distortions in the foveal visual image region were much larger than distortions of extrafoveal eccentricities normally suffering from severe visual acuity losses with sensory stimulation. Oculomotor readout of short-term memory for foveal visual target locations was also distorted, but with a qualitatively different pattern of a global upward bias, suggesting that readout modality matters a great deal in studies of visual short-term memory.

While the distortions that we observed may be generally related to topographic neural map representations, these distortions are not trivially explained by a simple model of a more diffuse neural trace of the visual target representation in a foveally magnified area, such as the primary visual cortex (25, 26) or the superior colliculus (27). For example, readout of a foveally magnified neural map (at different locations) should be similar whether there is a wide hill of activation (diffuse memory trace) versus a narrower hill of activation (which might result from actual stimulus onset). Rather, anisotropies in neural representations of the visual field that go beyond foveal magnification might be at play as well (18–20). Indeed, anisotropies in visual field representations also exist in brain areas relevant for oculomotor readout in our task, as exemplified by the large difference between how the superior colliculus represents the upper and lower visual fields (28).

In the visual cortex, anisotropies are related to behavior (18–20, 29–31), as elegantly highlighted in the theoretical framework of Previc (32). For example, visual performance on the horizontal meridian in a variety of tasks is better than performance on the vertical meridian (14, 16), and also better than performance at oblique locations (33–35). Besides our diagonal biases, our manual response results with short-term memory for foveal target locations demonstrate similar visual meridian patterns (e.g., compare the direction and amplitude errors in Fig. 3 between the horizontal and vertical meridians), and they are in line with evidence that visual meridian performance differences in the periphery persist in short-term memory tasks (17). In fact, our manual response results can also suggest that anisotropies in representing visual field locations in cortex (18–20) extend well into the small-eccentricity foveal representations, which makes sense given the large amount of neural tissue magnification associated with foveal eccentricities.

In that regard, it is interesting that differences in perceptual (30, 33–38) and oculomotor (35) performance often demonstrate a so-called oblique effect, in which performance is worse at oblique directions. In our case, the diagonal bias gave the best directional error performance at oblique directions in the fovea (e.g., Figs. 2 and 3). However, amplitude errors were the largest for the oblique directions (Fig. 3). Thus, the term oblique effect may, in principle, be used to describe our results as well, although, given the diversity of possible effects (e.g., having the best directional errors on the diagonal versus observing the worst amplitude errors), we prefer to use the term diagonal bias instead of oblique effect. Other authors have also done so (39) in a different context.

Another reason for our interest in oblique effects is that these earlier studies almost invariably employed a two-interval forced choice paradigm. In such paradigms, a reference stimulus is first presented, followed by a second one. It could be the case that the memory trace of the first stimulus is distorted, as revealed by our results, and that it is this distortion that gives rise to oblique effects in these experiments. As mentioned above, at eccentric locations, visual meridian performance differences indeed occur in a short-term memory task (17). Moreover, memory distortions can emerge very rapidly, even with memory intervals as short as 50 ms (40). Thus, classic oblique effects with 2-interval forced choice paradigms may be mediated by distortions in short-term memory like the ones that we have described here.

It may also generally be the case that evoking oculomotor or manual reporting from nonvisual stimuli would invoke distortions, by virtue of nonvisual neural activity that may be needed for successful task performance. For example, in an anti-saccade task, in which subjects generated a saccade to the diametrically opposite location from a visual stimulus, a mild diagonal bias, not unlike ours, was observed for extrafoveal target eccentricities (39). In this case, the distortion emerged with a transformation of the visual signal into a separate movement command, which may or may not require recruitment of short-term memory. We even found, in earlier work, repulsion away from the fovea at near eccentricities and toward the fovea at far eccentricities around the time of microsaccade generation (41), suggesting that any manipulation of neural activity in a logarithmically warped and anisotropic topographic map may be sufficient to cause distortions like the ones that we saw in our study.

Our oculomotor results are also intriguing because they may relate to debates on whether saccades in darkness are associated with an upward shift in gaze. Such an upward shift in gaze is a robust phenomenon in monkeys (42, 43), but it has been questioned whether humans might also show it (e.g., see related experiments in refs. 43, 44). If such a shift is related to an anisotropy in the representation of the upper visual field by oculomotor structures (28), then also saccades (and not just gaze position) may have upward biases when triggered without visible targets. In our case, complete darkness was not even required to evoke an upward bias in memory-guided saccade endpoints. All that was needed was a saccade to a blank. Importantly, small saccades on the scale of microsaccades were not previously tested for such upward biases under memory guidance. It is possible that it was easier to see upward biases for small saccades exactly because of large foveal magnification, similar to how visual meridian effects (with or without memory) might reflect anisotropies in cortical tissue magnification; our weak extrafoveal effects (relative to monkeys) might, in turn, suggest that humans have larger foveal magnification than monkeys. Either way, it is fascinating that other behaviors involving saccades can also be associated with upward biases in the eye movement endpoints, as in the case of visual search (45). According to the framework of Previc (32), this may be a purposeful functional specialization related to the fact that eye movements allow sampling of the far visual environment.

In any case, the fact that we still observed upward biases for our humans’ small memory-guided microsaccades is consistent with past observations on the role of active vision at the foveal scale in oculomotor control areas (46, 47). It is also important to remind that saccades (including small ones) are more accurate with visual guidance than without, as we showed recently with the same subjects (and with monkeys) in variants of our memory-based task that had a visual saccade target in the final response phase (11, 24). Therefore, our oculomotor results were not reflective of a mere general sloppiness of saccades.

Our saccade results can also help rule out the possibility that our manual response results were due to a range effect, or the idea that subjects were simply biased toward the midpoint of the range of eccentricities that we tested. Since saccades do not suffer from a range effect in the memory-guided paradigm (48) (but see ref. 49 for other viewing contexts), our observation that saccades still overshot the target at small eccentricities suggest a genuine distortion in memory-based localization, even in manual tasks. This is also consistent with earlier conclusions in extrafoveal memory recall (22). It is also consistent with the results of our control experiment of , in which we significantly altered the statistics of the possible target locations in the experiment while still observing the same mislocalization patterns.

Based on the qualitative difference between our manual response and saccadic results, we also believe that readout modality matters a great deal in studies of short-term memory. This is an important point to consider given the large number of studies of short-term memory that rely on saccadic responses. Indeed, our results suggest that manual responses in the same task might reveal very different distortions from saccadic responses. Such different distortions may be associated with visual topographic map anisotropies (Fig. 6), which can be very distinct from oculomotor map anisotropies (Fig. 6).

Fig. 6.

Differences in short-term memory distortions across response modalities may be related to differences in topographic anisotropies within the circuits used for readout. (A) With manual reporting, anisotropies in cortical visual areas may dominate, with diagonal biases reflecting overrepresentation of the visual meridians (blue) at the expense of noncardinal locations (red). Here, overrepresentation is schematized as smaller visual receptive fields (circles); note that lower visual field receptive fields are schematized as also being smaller than upper visual field receptive fields. Evidence for such visual field anisotropies exists (18–20, 35) and is consistent with our observations. (B) With saccadic reporting, location readout might be more dependent on oculomotor maps, which can have very different anisotropies. For example, there are smaller response fields in the foveal and upper visual field (blue circles) representations of the superior colliculus (27, 28, 59). This could potentially result in different memory-based distortions with saccadic readout, as we saw in Fig. 5.

Finally, it may also be the case that the fixation spot during the memory interval acted as a visual landmark, which can distort target localization (50, 51). However, we saw similar distortions with and without a foveal visual landmark during the response phase in our first two experiments. We think that the retinotopic reference frame itself is what induces outward expansion for near locations and foveal biases (21–23, 52, 53) for far locations. To consolidate this idea, it would be interesting in the future to identify foveal visual performance asymmetries (like visual meridian effects) even without a memory task.

In all, our findings highlight the importance of studying multiple aspects of foveal visual processing, particularly at the level of neural mechanisms, since foveal vision, along with its associated active scanning behaviors, is particularly relevant for fully understanding the entire spectrum of human visual function.

Materials and Methods

We performed previously-unpublished analyses on behavioral data that were collected earlier as part of another study (11), and we also collected new data for an additional control experiment (the results of which are shown in ). The earlier study was concerned with memory-guided microsaccades and their associated neural responses in the superior colliculus; the behavioral parts provided necessary control conditions in that study, but they were not analyzed in detail. Here, we analyzed the behavioral parts anew, to reveal properties of foveal visual location recall from short-term memory.

Laboratory Setup.

The laboratory setup was the same as that used in previous publications (11, 41, 54). Briefly, subjects were seated 57 cm from the front of a CRT monitor having a refresh rate of 85 Hz, and their heads were stabilized with a custom-built device (41). We tracked eye movements at 1000 Hz using a video-based eye tracker (EyeLink 1000). The display had a resolution of 41 pixels per degree (11, 54, 55) and it was luminance-calibrated. Stimuli used in this study consisted of white dots having 97 cd/m2 luminance, which were presented over a uniform gray background of 21 cd/m2 luminance (54).

Subjects.

We collected data from a total of 12 subjects across the main experiments of Figs. 1–5. Each variant of the memory paradigm (Figs. 1–5) was performed by a total of seven subjects, and four subjects participated in all three variants (in separate sessions). For the control experiment of the button cursor movement variant (), we collected data from a total of four subjects (one from the original 12 of the earlier experiments, and three new ones experiencing the task for the very first time; two of the new subjects were females). In the main button cursor movement task, 1 subject was female; in the mouse pointing task, 1 subject was female; and in the memory-guided saccade task, 3 subjects were female. The subjects provided written informed consent prior to their participation, and they were financially compensated for their time.

All experiments were conducted after approval from ethics committees at the Medical Faculty of the University of Tübingen, and the experiments were in line with the Declaration of Helsinki.

The subjects’ ages ranged from 23 to 39 y.

Behavioral Tasks.

Button cursor movement task.

Each trial started with a central fixation spot (7.3 × 7.3 min arc) presented over a uniform gray background. After a random interval of 750–1250 ms, we briefly flashed a similar spot, for 59 ms, at one of 300 possible locations (indicated in Fig. 1), spanning eccentricities of 0.1° (6 min arc) to 12°. A total of 176 out of the 300 target locations had both a horizontal and vertical eccentricity absolute value of less than 1°. After a memory period of 300–1100 ms, the fixation spot disappeared, and a response cursor (a crosshair in the shape of a plus sign, having dimensions 59 × 59 min arc) appeared at the center of the display (in place of the initial fixation spot). The subjects pressed one of four buttons on a response box to move the cursor along all four possible cardinal directions, and they then pressed a fifth (middle) button to indicate that the cursor was at the remembered target location. Each button press during cursor movement moved the cursor by 1.46 min arc along a cardinal direction, and prolonged button presses resulted in one of two faster cursor speeds depending on press duration (11). All of this meant that reaction times in this task could be quite long (), and we allowed subjects up to 20 s for finalizing their response. The subjects were free to move their eyes during cursor movement and final report. After the subjects finalized their response (by pressing the fifth button), the true target location appeared as feedback on how accurate their report was. Each subject participated in five to six sessions of 250–350 trials each. We collected a total of 11,249 trials across subjects.

Control version of the button cursor movement task.

The task was the same as the button cursor movement task, except that we changed the sampled target locations. In the original version, we (deliberately) sampled the fovea densely and also included foveal samples on the 45° pure diagonal lines (Fig. 1). The extrafoveal target locations were sparser and did not include samples on the 45° diagonals. In this control version of the task, we investigated whether such a sampling strategy influenced our results. We made the foveal targets a minority of trials; we did not sample any locations directly on the 45° diagonals; and we also sampled the extrafoveal target locations more densely than in the original experiment. Specifically, we had three target eccentricities (0.75°, 9°, and 11°) and 12 directions from the horizontal (equally spaced around the circle; ). The experiment was otherwise unchanged. We analyzed 540 trials per subject in this variant of the experiment (collected in one session per subject).

Mouse pointing task.

The task was similar to the original button cursor movement task, except that we kept the central fixation spot visible at trial end (and until subject response). Instead of a cursor, the computer mouse pointer became visible to indicate that it was now time to report the remembered target location. The mouse pointer appeared at a random location 1–5° in eccentricity away from the true target location. This allowed faster reaction times than in the button cursor movement task, and it also made the subjects’ task somewhat easier, especially since they were now required to maintain their gaze near the central fixation spot at all times while responding (even for the peripheral targets). The subjects’ reporting requirement was to point the mouse pointer at the remembered target location and click the mouse button as quickly as possible. They then received feedback by the reappearance of the target at its original location. As expected, the reaction times in this task were significantly shorter than in the button cursor movement task (), since moving the mouse pointer could be achieved faster. We used a total of 480 target locations, 296 of which had both a horizontal and a vertical eccentricity of less than 1°. Subjects participated in five sessions of 380–500 trials each (except for one subject who completed four sessions). We collected 16,213 trials in this task.

Memory-guided saccade task.

The task was identical to the mouse pointing task above (including in the target locations). However, at trial end, the fixation spot was removed, and no other stimulus was presented. The removal of the fixation spot was the cue to generate a memory-guided saccade toward the remembered target location. After a prolonged delay well beyond the saccadic reaction times, the target reappeared as feedback to the subjects. We collected four to five sessions, of 480–600 trials each, per subject, for a total of 16,844 trials.

Data Analyses.

Eye movements were analyzed in detail earlier (11, 24), including with monkey subjects exhibiting similar effects to the humans (11). Here, we used the already labeled data from these studies to investigate the saccadic distortions shown in Fig. 5 and .

For manual responses (whether with button cursor movement or mouse pointing), we created tables of reported locations along with true locations. We excluded rare trials in which the subjects missed seeing the flash, for example, due to microsaccadic suppression at flash onset time (56). In these cases, subjects typically indicated display center (no cursor movement) or pressed a far location and in the wrong quadrant. Such trials were very few (∼0.4%).

We defined amplitude and direction errors as explained in Fig. 1. Specifically, the average response location across trial repetitions had a vector from display center of a given eccentricity and direction (red in Fig. 1). The direction error of the response vector was the difference between this vector’s direction from the horizontal meridian and the direction of the vector connecting the display center to the true target location. As for amplitude error, we divided the eccentricity of the average response vector by the eccentricity of the true target location vector. A ratio >1 represented a response amplitude overshoot, and a ratio <1 represented a response amplitude undershoot (e.g., foveal biases for eccentric extrafoveal locations).

For summarizing distortions in foveal location short-term memory, we connected the true target location with the average response location using a blue vector (as in the Left panel of Fig. 1). Across all target locations, this resulted in global quiver plots (e.g., Fig. 2) of distortions. We summarized these distortions in a more compact way by remapping all four quadrants into a representative quadrant, since the manual response tasks revealed symmetry across quadrants (e.g., Fig. 2). To do this, we reflected the left visual hemifield and its responses to the right hemifield, and we reflected the lower visual field and its responses to the upper visual field. We also did this for the control experiment of . For the saccadic version of the task, it was immediately obvious that the pattern of distortion was not symmetric across all four quadrants (e.g., Fig. 5). Rather, it was symmetric across the right and left visual hemifields only. Therefore, we remapped responses only across the right and left visual hemifields, and we did not reflect the lower visual field into the upper visual field. This would have masked the global upward bias in memory-guided saccade endpoints that we observed (e.g., Fig. 5).

For the manual response tasks, we also created eccentricity and direction bins of the target locations. Because of the sampled locations, the eccentricity bins of 2–3° and >3° had fewer direction samples than the <2° eccentricity bin (e.g., Fig. 1). Therefore, we did not do further direction binning beyond the categorical direction bins that existed in our sampled target locations. This is also why we did not connect the data points for these extrafoveal eccentricities with connecting lines in Fig. 3 (and similar figures) as we did for the <2° eccentricity bin. For the foveal eccentricities that were critical for this study, we had sufficient direction samples for further direction binning. Thus, we created a running average across directions, with direction bin widths of 15° and sampled in steps of 5°. For example, to plot direction error as a function of target direction from the horizontal meridian (e.g., Fig. 3), we took all target and response reports with eccentricity <2° and target direction from the horizontal within a given direction bin. We then plotted the average direction error. We used a similar procedure for amplitude errors. For the analyses of , we also repeated the same procedures but only for trials with either short or long memory intervals. Trials were classified as having short or long memory intervals based on a median split of the time between stimulus flash end (at the beginning of a trial) and trial end (i.e., onset of the response cursor or mouse pointer) across all trials (subsequent reaction times were similar in both groups of trials, as shown in ).

For the saccade task, we used similar eccentricity and direction binning. For clarifying the upward bias in direction errors, rather than plotting direction error as a function of direction bin, we plotted the average response direction as a function of direction bin (e.g., Fig. 5). We also explicitly plotted both right and left visual hemifield results (e.g., Fig. 5) to demonstrate that the memory-guided saccade direction biases were upward in both hemifields (justifying the hemifield remapping strategy in Fig. 5). For displaying the individual subject results (), we pooled the hemifields again just to simplify the figures.

In all main figures, we plotted results across subjects by pooling their responses together. This was justified because there were a similar number of trials collected per individual subject. However, in the supplementary figures, we also showed all results after first analyzing each subject individually and then averaging the individual subjects’ averages. As expected, the conclusions were the same with either approach.

In all figures, we also provided descriptive statistics with indications of central tendencies and variances around them. We focused on systematic errors in memory recall for our analyses. This was important for emphasizing that the effects that we reported here could not be attributed to variability in fixational eye position due to fixational eye movements. Specifically, since gaze continuously shifts subliminally during fixation, it may be argued that errors in memory-based target localizations were due to fixational instability. However, if memory-based localization performance was entirely explained by fixational eye movements, then no systematic errors would be observed in our experiments (only nonsystematic, or variability, errors would occur). This is because average eye position will be on the fixated target, by definition. For example, in our earlier work (57, 58), we found that fixational saccades optimize eye position on the fixated target remarkably well, such that average gaze position remains highly accurate.