Build-up of serial dependence in color working memory.

Serial dependence, how immediately preceding experiences bias our current estimations, has been described experimentally during delayed-estimation of many different visual features, with subjects tending to make estimates biased towards previous ones. It has been proposed that these attractive biases help perception stabilization in the face of correlated natural scene statistics, although this remains mostly theoretical. Color, which is strongly correlated in natural scenes, has never been studied with regard to its serial dependencies. Here, we found significant serial dependence in 7 out of 8 datasets with behavioral data of humans (total n = 760) performing delayed-estimation of color with uncorrelated sequential stimuli. Moreover, serial dependence strength built up through the experimental session, suggesting metaplastic mechanisms operating at a slower time scale than previously proposed (e.g. short-term synaptic facilitation). Because, in contrast with natural scenes, stimuli were temporally uncorrelated, this build-up casts doubt on serial dependencies being an ongoing adaptation to the stable statistics of the environment.

Introduction

Our perception depends on past experiences[1]. Serial dependence—how our current estimates are biased towards immediately previous ones—has been described experimentally using many different paradigms[2-19]. In particular, paradigms including delayed-estimations of different visual features[2], such as orientation[9,11,15,19], numerosity[20], location[3,21,22], facial identity[23] or body size[13]. It has been speculated that these ubiquitous attractive biases are a consequence of the world’s tendency to be stable, and have the functional role of averaging internal noise[2,11,14,24]. Some have further argued that serial dependence could be of adaptive nature, changing its strength depending on the stimulus statistics[2,11,14,16,24], similarly to contraction biases[25-28]. The separation of time scales in which serial dependence and contraction biases operate suggests distinct mechanisms[18], which could still possibly interact[29], and we will focus here on serial dependence. Color, which is strongly correlated in natural scenes[30], has never been studied with regard to its serial dependencies, possibly due to its strong systematic biases[31-33]. Similar to other perceptual biases for other visual features[34,35], these systematic color biases adapt to stimulus statistics in the course of one experiment[33]. This suggests that typical perceptual bias adaptations occur in time scales of minutes to hours. Slow adaptation of serial dependence, however, has never been characterized. If serial biases are also subject to adaptation with a similar time scale, when exposed to long sessions with uncorrelated stimulus statistics they should decrease or, in case of not being adaptive, they should remain stable. In fact, a recent study supports the latter hypothesis: in an auditory working memory task, with sound frequencies sampled from uniform, Gaussian or bimodal distributions, serial biases were not affected by the stimulus distribution[18]. Here, we address serial dependence in delayed-estimation color tasks. Based on an undefined mean stimulus color in this task, contraction biases are not confounding our assessment of serial dependence. By controlling for the known systematic biases in color perception we characterize for the first time, contrary to our hypothesis, a slower dynamics of increasing serial dependence through the experimental session, despite uncorrelated stimulus statistics.

Results

We analyzed 8 datasets that are freely available online (see “Methods” section), with a total of n = 760 subjects performing variations of the same, delayed estimation of color task (Fig. 1a, Supplementary Table S1). We found that, across experiments, the reports of the subject were attracted to the previous target color for relative distances between previous and current trial target color of up to 90° in all experiments. Significant serial dependence occurred in all individual datasets for relative distances of up to 90° (two-tailed t-test on the average folded error; Cam-CAN: t(648) = 6.19, p = 1.08e−09; Van der Berg I: t(12) = 3.33, p = 0.006; Van der Berg II: t(12) = 7.29, p = 9.67e−06; Oberauer & Li: t(18) = 3.06, p = 0.007; Foster et al. I: t(11) = 4.04, p = 0.002; Foster et al. II: t(20) = 8.52, p = 4.36e-−08; Bays et al.: t(11) = 3.78, p = 0.003), except for the dataset collected by Souza et al.[36] (t(20) = 1.54, p = 0.14). See Supplementary Fig. S1 for more details.

Figure 1

Serial dependence in color. (a) Top, illustration of the prototypical experimental design. All experiments were variations of a delayed-estimation of color task as in ref.[37], differing mostly on set size and number of trials (see Table S1 and original references for more details). Briefly, subjects were shown a disc colored randomly in each trial. After a blank delay (of different durations, Supplementary Table S1), subjects were asked to report the color on a wheel rotated by a random angle in each trial. Bottom, serial dependence was simulated as a drift towards the previous trial trace in a diffusion process. In purple, 50 trials with a stimulus feature (purple triangle) close to the previous trial trace (gray) and in black, 50 far trials. Thick lines represent the averages of each condition, which are attracted to previous trial stimulus for trials that are close by. (b) Serial bias in the delayed-estimation of color task for all datasets. We found significant serial dependence relative to the previous report in all datasets (two-tailed t-test, t(7) = 6.5, p = 0.0003), except for the dataset collected by Souza et al.[36] (p = 0.14). Black bar on the top marks points where the mean was above zero (bootstrap, p < 0.05). (c) Left, error to target stimulus reveals systematic biases on simulated trials. Middle, serial dependence calculated separately for trials simulated with and without systematic bias. Right, folded version of serial dependence removes all systematic biases without any additional preprocessing. (d), same as (c) for trials of Foster et al. I[38].

Folding serial bias curve removes systematic biases

Delayed-response reports are subject to systematic biases, which are particularly strong in the case of color[32]. It has been argued that it is necessary to model and remove the systematic bias prior to estimating serial dependence[3,9,22]. Here, we applied a model-free strategy that corrects serial dependence by “folding” the serial bias plot (Fig. 1b). We tested this method in surrogate data obtained using a computational modeling approach. We simulated each delay of two consecutive trials as a diffusing memory trace[39] using a simple random walk simulation (see “Methods” section). On top of independent Gaussian errors responsible for diffusion, we added serial dependence as another source of error that accumulated incrementally at each time step, and two other sources of distortion (see “Methods” section): (1) systematic biases derived from inhomogeneities of the task space[32], and (2) systematic rotational biases (e.g. a constant clockwise error[3,9,15]). Figure 1c shows the effect of these systematic biases on serial bias estimation (see Supplementary Fig. S4 for the effect of each systematic bias separately). We simulated n = 1,000 trials and n = 100 subjects with (gray) and without (dashed black) systematic biases. As previously reported[9], we found that systematic biases contribute an additive overall shift to the serial bias function. This shift precludes the correct identification of attractive and repulsive serial bias regimes, and complicates comparison across subjects. We found that a simple processing of the data allowed for a model-free correction of systematic-bias-induced shifts: we “folded” the serial bias curve by collapsing all negative distances between consecutive targets on positive values, while also inverting for these trials the sign of the behavioral error (folded error, see “Methods” section). This method effectively removes all systematic biases introduced in simulated trials (Fig. 1c, right). For illustration purposes, we show the application of this method in one dataset (Foster et al. I[38]) with similar systematic biases (Fig. 1d, left), that led to a shifted serial bias function (Fig. 1d, middle) and finally a folded version, without systematic biases (Fig. 1e, right). See Supplementary Fig. S1 for the same analyses on each experimental dataset. Thus, our simulation approach validated the folding approach to correct for systematic biases in serial bias estimations, and allow for across-subject comparisons (Fig. 1b, Supplementary Fig. S4).

Color serial dependence builds up in the course of an experimental session

Serial dependence, some argue, reflects the world’s tendency to be stable[2,14,24]. The reasoning is that because similar stimuli usually elicit similar behavior, the brain would incorporate mechanisms to exploit these patterns[24]. Along these lines, a recent study has shown that systematic biases in color working memory change in the course of an experimental session to adapt to stimulus statistics[33], arguing that systematic biases seen in delayed-estimation of color reflect real-world statistics. If similar adaptive plasticity operated for serial dependence in the time scale of the experimental session, we would expect to see a reduction of serial biases as one is exposed to a sequence of uncorrelated stimuli. To our knowledge, the stability of serial dependence within an experimental session is yet to be characterized. To address this question, we divided each session in three thirds and computed serial dependence relative to the previous trial report for each subject and experiment the first and last thirds (Supplementary Fig. S2). When averaging all experiments together, we found that, contrary to our hypothesis, there was stronger serial dependence in the last than in the first part of the experimental session (Fig. 2a). To further characterize this serial dependence build-up, we used the folded error on each trial as a scalar measuring the evolution of serial dependence in the course of the session (see “Methods” section). Figure 2b illustrates this analysis using a sliding window of 75 trials for Cam-Can[40,41] dataset and of 200 trials for the Foster et al. I dataset[38], both showing a clear increase of serial dependence as the session progressed. To test this effect across subjects and experiments, we obtained the regression slope of the folded error as a function of trial number. We computed this slope for each subject and for all experiments. Figure 2c shows that serial dependence build-up was positive for all but 1 dataset (Bays et al.[42]), significant for 2 datasets individually (Cam-Can[40,41], p = 0.008, 95% CI [0.1, 0.6], and Foster et al. I[38], p < 1e-6, 95% CI [0.18, 0.36], bootstrap) and for all combined (p = 0.007, 95% CI [0.04, 0.2], bootstrap t-test on the 8 averages across experiments, which treats with equal weight each study despite their very disparate number of subjects, or combining subjects from all experiments, p = 0.006, 95% CI [0.1, 0.5]; bootstrap t-test, which weighs equally each subject irrespective of study).

Figure 2

Serial bias builds up during a session. (a) Serial biases computed using first third (black) and second third (green) of the trials for two example experiments: Cam-Can[40,41] and Foster et al. I[38]. Black bars on the top mark where curves are significantly different, p < 0.05, permutation test. (b) Both experiments show a significant increase in serial dependence through the session computed with a sliding window of 75 trials for Cam-Can and 200 for Foster et al. I. (c) For each subject, we computed the slope of serial dependence over the course of the session (without averaging). We found that serial-bias build-up was significant in two experiments (marked with black error-bars: Cam-Can, p = 0.008; Foster et al. I, p < 1e−6). Error-bars were calculated from bootstrap distributions and unless stated otherwise (all experiments in (c), red), are standard errors of the mean.

We then tested if this serial bias build-up was related with subjects getting familiar with the task, in which case one would expect to see an improvement in performance through the session, or related to subjects feeling tired, which should be reflected in worsening of performance. To this end, we calculated the fraction of guesses as a proxy of tiredness or engagement. We classified as guesses those trials with error > 90° and counted them in independent windows of 20 trials to obtain the fraction of guesses. Importantly, these trials were excluded from all the other analyses (above). We then computed the slope of change of the fraction of guesses in the course of the sessions for each participant. We found a decrease in guess rate through the session in 2 datasets (CamCan, p = 7.1e−20 and Van der Berg I, p = 0.008) and an increase in another 2 datasets (Souza et al., p = 0.02 and Bays et al., p = 1.1e−06). However, these trends did not correlate with serial bias build-up for any dataset independently (p > 0.35, linear regression, Figure S3a), combining subjects from all datasets (p > 0.35, linear regression) or averaging across experiments (p > 0.2, linear regression, Supplementary Fig. S3c). We also checked the evolution of performance by measuring the mean squared error through the session. Serial bias build-up was also not correlated with the subjects’ squared error (excluding guess trials) trend during the session (Supplementary Fig. S3). These control analyses show that serial bias build-up during experimental sessions was not associated with trends in performance dynamics as a result of subjects getting familiar with the task or tiredness. Together, these results show that serial dependence is not stable on the time scale of one experimental session, as previously assumed, and it also discards a mechanism that adapts to stimulus statistics. Instead, our result suggests the involvement of slowly accumulating plastic mechanisms in serial dependence of color delayed-estimations.

Discussion

We provide the first evidence of serial dependence in color working memory. Serial dependence had been characterized with great detail for other visual features[2], and in particular in spatial working memory[3,21,22]. Several common features of color and spatial working memory suggest that serial dependence could also be similar in color: simultaneously memorized stimuli interfere attractively when presented at close distances[43,44], and memory precision decreases with memory period duration[3,45,46]. These commonalities are in contrast with the differences of neural representations. While spatial representations consolidate early in the visual pathway[47], complex transformations in color representations occur as color information travels from the photoreceptors in the retina, to visual cortex, and into association cortex[48]. The fact that serial dependence is similar for color and spatial working memory thus suggests that it depends on inter-trial interferences that occur at processing stages with representational maps equally distant from the corresponding perceptual map, and this points at higher color processing stages. A candidate region for this is the inferotemporal cortex, where continuous neuronal representations of color of circular shape on the two perceptual cardinal axes (yellowish-bluish and greenish-reddish axis) have been found[49,50]. Recent evidence suggests that similar color representations, maintained through working memory delays, may reach up to the monkey prefrontal cortex[51].

The analogy of color and angular location neural representations motivated us to simulate color working memory similarly to spatial working memory of angular locations[52]. We simulated the angular memory trace in the memory period as a diffusion process[39] with a drift toward the previous trial memory trace that introduces serial dependence[53]. We used this model to test the concerns about the impact of systematic biases in the estimation of serial dependence. This is a general concern that has been raised for other visual features[3,9,21,22], but in the case of color it may be particularly important for the marked perceptual systematic biases that have been reported[32]. We therefore incorporated strong systematic biases in the reports of our model simulations, and we developed new analysis strategies avoid their impact on the estimation of serial dependence (Supplementary Fig. S4). One typical strategy for systematic bias removal is to low-pass filter the responses as a function of stimulus feature[3,9,21,22]. This approach depends on parameters that are often subjectively decided (e.g. size of sliding window). In addition, removing systematic biases incorrectly, for example when subjects do not have systematic biases, can introduce extra biases in otherwise clean data (Supplementary Fig. S4). We showed that by folding the serial dependence function, one can reduce the impact of systematic biases on serial dependence without adding biases in unbiased data and without specifying arbitrary parameter values. We therefore conclude that this analysis allows a more robust estimation of serial dependence in behavioral studies.

Theoretical models have proposed that short-term subthreshold mechanisms in inter-trial intervals underlie serial dependence in delayed-estimation of location[53-57]. In this class of models, neural activity in previous trial’s mnemonic representations engage plasticity mechanisms that leave a selective trace in the network’s synapses. This trace interferes with neural activations in the next trial by biasing the neural representation of the new stimulus towards the previous memorized location. These dynamics explain most experimental findings of serial dependence in spatial working memory[3,57], and our simplified modeling approach is consistent with this mechanistic substrate[57]. However, our finding that attractive serial biases build up in the course of an experimental session is not explained by these models. Indeed, the short-term synaptic plasticity mechanisms invoked so far operate in time scales of a few seconds, much shorter than the time scale of the experimental session. Our results reveal that additional mechanisms, accumulating in a time scale of 10’s of minutes or hours, are also responsible for the instantiation of serial dependencies in delayed-estimation of color tasks. Possible mechanisms are changes in plasticity efficacy itself (i.e. “metaplasticity”), modulating synaptic release probability over the experimental session. We tentatively speculate that habit-related endocannabinoid modulation of synaptic release[54,58] could mediate serial dependence build-up as a non-adaptive result of task habituation.

The build-up of serial dependencies during an experimental session has further implications for how we interpret their functional role. In analogy with other behavioral biases, such as contraction biases[25] or systematic biases[59], serial dependence has been proposed to respond to the organism’s quest to adapt to the effective statistics in the environment[24]. If serial dependence was an adaptation to exploit the world’s tendency to remain stable[24], and this adaptation could occur in the time scale of hour fractions (as recently shown for systematic biases in delayed-estimation of color tasks[33]), memorizing a sequence of uncorrelated stimuli should decrease serial dependence in the course of an experimental session. Alternatively, if hard-wired mechanisms underlie serial dependence, we would not expect any change. Instead, our results show that serial dependence builds up during the session. Importantly, this is not related to the adaptive dynamics of contraction to the mean (as presented stimuli had undefined mean, Methods) or systematic biases (Supplementary Fig. S3). Thus, our results suggest that serial dependence does not respond to an active adaptation to the statistics of visual stimuli in the environment (at least in the time scale of hours) but instead may reflect a plasticity mechanism driven by repeated selective neuronal activations in the circuit.

Methods

Participants and design

We analyzed 8 datasets that are freely available online (Supplementary Table S1), with a total of n = 760 subjects performing variations of the same, delayed-estimation color task (Fig. 1a). We will briefly describe the general experiment and Supplementary Table S1 summarizes the specifics of each task, for detailed descriptions please refer to the original studies[36,38,40-42,60,61]. On each trial, a set of colored stimuli (varying from 1 to 8 stimuli) were briefly shown. After a delay period of roughly 1 s (see Supplementary Table S1 for details), during which stimuli were no longer visible, subjects had to report the target color of a cued location. These color reports correspond to angles (i.e. degrees) on a color wheel rotated by a random amount on every trial, to avoid a spatial memory strategy. There was no color rendering calibration procedure in any of the 8 datasets (Supplementary Table S1). We don’t expect this to influence our results, since all experiments sampled colors randomly and independently on a trial-by-trial basis, so any potential impact of color inhomogeneities on serial biases is expected to be averaged out with large numbers of trials. Finally, the overall experimental design is free of contraction biases, because the stimulus mean is undefined given the uniform distribution of color locations on the circle.

Serial dependence analysis

As in previous studies (e.g. ref.[11]), serial dependence was measured by calculating the mean error as a function of distance between current and previous target () in sliding windows with size and in steps of . Most of the studies, except for Foster et al.[38], were multi-item working memory. On trials with more than one stimulus, we use the target stimulus as reference.

Folded errors

We corrected for the effect of systematic biases in color estimation[32] by computing a folded version of a typical plot (e.g. ref.[11], see “Discussion” section) as follows. Folded errors were obtained by multiplying trial-wise errors () by the sign of : . We then computed serial bias curves as mean folded errors versus the absolute values of . We also obtained an overall measure of the strength of serial dependence by computing the mean folded error across all trials. Positive mean folded errors should be interpreted as attraction towards the previous stimulus and negative mean folded errors as repulsion away from the previous location.

Serial dependence build-up

Serial dependence build-up was measured by splitting each session’s trials in three equal consecutive thirds and computing serial dependence separately. Serial dependence build-up was also measured by using the whole session data. For the whole session data analyses, we computed the folded error on each trial (by multiplying each error with the sign of the target-to-target distance, see above). For each session, we fitted a regression slope of folded error against trial number and averaged across sessions, keeping one slope per subject and experiment.

Simulating consecutive trials

We simulated the memory trace in trial k as a random walk, (t = 0…T, with T the duration of the delay period). Each random walk (Eq. 1) started at its corresponding stimulus location, . On top of independent Gaussian errors responsible for diffusion (controlled by the parameter σ = 0.006 deg), we added serial dependence as one more source of error that accumulated incrementally at each time step. This increment was obtained from a derivative-of-Gaussian (DoG) function of the distance between the instantaneous memory trace and the previous trial’s stimulus (i.e. starting or ending location of the previous trial simulated trace , with t = 0 or t = T, respectively) (Eq. 2, refs.[9,11,15,23]). The DoG curve was defined by parameters a = 0.09 deg, c = , and w = 0.007 deg−1 (Eq. 2), similar to ref.[9].

The response in each simulated trial was then derived from the last point of the memory trace , by applying two sources of distortion: (1) systematic biases derived from inhomogeneities of the perceptual space were simulated as , with constants chosen to fit the data by visual inspection (Fig. 2c,d, left); (2) systematic rotational biases (e.g. a constant clockwise error[3,9,15]) were simulated by adding a constant arbitrary angle

Statistical tests

All error-bars and confidence intervals were computed from bootstrapped distributions. Significance was assessed with 1-sample t-tests or permutation tests at P = 0.05.

Supplementary file1 (PDF 993 kb)