Seeing the past: afterglow effects on familiarity judgments are category-specific.

According to several computational models, novel items can create a learning mode with dynamics favorable to new learning, and not to memory retrieval. In line with that idea, a new item in a recognition test has been found to create a bias toward calling subsequent items new as well. Here, we tested whether this bias, which we termed the afterglow effect, is indeed caused by a general learning mode, or is caused by perceptual overlap between preceding and current items. In two experiments, we show that a preceding recognition judgment biases the current one, but only if the preceding and current items are of the same perceptual category. In contrast, we did not find strong bias effects from perceptually novel fractal images, as would be predicted if novel items induce a learning mode that then biases recognition judgments. We conclude that the afterglow effect is more likely to reflect perceptual phenomena than a learning mode. We suggest how this can be reconciled with what is known about familiarity at the neural level.

It has long been suggested that the brain can switch between learning and retrieval modes on the basis of incoming information (Hasselmo et al. 1995; Meeter et al. 2004). This is proposed to happen through neuromodulatory processes. Novel items would stimulate the release of neuromodulators that stimulate bottom-up separation of patterns and would boost LTP, which would then facilitate the storage of a new, unique pattern because of a learning mode. On the other hand, familiar input would inhibit the release of these neuromodulators, down-regulating learning and bottom-up pattern separation, and increasing pattern completion processes. These pattern completion dynamics, which are proposed to be governed by the hippocampus, would be ideal for retrieval of stored patterns (for review, see e.g., Rolls 2013). While theories proposing learning and retrieval modes (Hasselmo et al. 1995; Meeter et al. 2004) focused on the neurobiology and were rather unprecise at the behavioral level, they did make a crucial prediction: that a learning mode would linger for a few seconds after being set off by a novel item. This would mean that a novel item would not only alter the way it is stored itself, but might also affect the storage of stimuli that came after it.

Some behavioral evidence supporting this predication has indeed been found. Encoding of novel pictorial stimuli was found to enhance the likelihood that an item that resembled recent items was judged to be new (Duncan et al. 2012), as would be expected when the preceding new item would induce a learning mode and enhance pattern separation. A familiar item, on the other hand, increased the likelihood that the similar item was judged to be old, as would be expected when the preceding familiar item induced a retrieval mode that facilitated pattern completion. These findings were interpreted as showing that such novel stimuli induced a lingering encoding mode that affected subsequent mnemonic decisions for a few seconds, which matches the time that a novel item was theorized to induce a learning mode (Meeter et al. 2004).

This idea was corroborated by Malmberg and Annis (Malmberg and Annis 2012; Annis et al. 2018) in a more traditional word-recognition task. Using the preceding recognition decision as their independent variable, they found that a preceding “new” answer increased the likelihood that the next item would also be judged “new,” independently of whether it was truly new or not. It thus seems that a new item (a lure) induces some bias to judge subsequent items as new, perhaps through inducing a learning mode, and old items a bias to judge subsequent items as old. Indeed, in five experiments they showed that only the bias to call an item new or old was affected, sensitivity of recognition judgments was not. We will refer to this bias induced by the preceding item as the afterglow effect.

In a follow up experiment, the same authors found that when judging frequency of exposure, these afterglow effects were found to be stronger when stimuli belonged to a category that, presumably, contained more similar exemplars (landscapes) than categories with exemplars that were deemed less similar to one another (objects) (Annis and Malmberg 2013). This dependence on perceptual factors challenges the learning mode assumption, as this would predict a more general learning mode guided by pattern separation or completion principles in the hippocampus. A perceptual source of the afterglow effect is also made more plausible by the fact that similar effects have been found in purely perceptual judgments. For example, when observers must judge the attractiveness of faces in a tinder-like setting (i.e., viewing faces in fast, sequential order), preceding faces have a similar afterglow effect in that a face is judged more attractive when the preceding face was judged attractive, than when the preceding face was judged unattractive (Taubert et al. 2016). Similar effects were found for basic visual features such as orientation (Fischer and Whitney 2014) and numerosity (Cicchini et al. 2014). These effects have been explained by reference to the notion of a continuity field (Fischer and Whitney 2014). Since the visual world is mostly constant but our perception of it may rapidly change (e.g., because we move), our visual system would be biased to increase the perception of continuity between preceding and current input. Crucially, this would only operate when current perception is similar to what was perceived just before. In line with this, preceding faces only influence how attractive the current face is judged if it is similar in, for example, having the same gender (Liberman et al. 2014).

In the memory domain, it is possible that afterglow effects are similarly influenced by a bias to see current input as similar to the preceding one. Yet, the question remains whether afterglow effects reflect the inducement of a general learning mode by novel experiences, or whether they are more akin to perceptual effects, or both, as the theories are not mutually exclusive. A crucial difference between these two explanations is whether the effect depends on perceptual similarity. A learning mode explanation would predict that it does not, as learning and retrieval modes are theorized as a general bias of the hippocampal processing toward either learning or retrieval. A continuity field explanation would predict that afterglow effects do depend on similarity. Conversely, a learning mode explanation would predict that a stimulus that is wholly novel to the participant (e.g., a unique fractal) would thus induce a stronger learning mode and generate a stronger afterglow effect than a stimulus that resembles stimuli already seen by the participant (e.g., a picture of a landscape). A continuity field explanation would not predict such an effect.

Here, we therefore contrasted effects of novelty and stimulus similarity on afterglow effects. Participants studied stimuli of two different categories, scenes and fractals, in a continuous recognition task in which they had to rate for each stimulus whether they recognized it as having seen it before in the experiment (old) or not (new). Fractals were novel to all participants and were rarer as a category than landscapes were, hence creating a stronger novelty experience. We then tested two predictions, one generated by the learning mode hypothesis, one by the continuity field hypothesis: We tested each of these predictions with a separate analysis, as each hypothesis could in principle be true independent of the other hypothesis. Moreover, we tested whether any afterglow effect would be a factor more of preceding item status (as Duncan et al. 2012, found), or of the preceding recognition decision (as Malmberg and Annis 2012, found). In experiment 2, we ran a similar experiment, but now with a different nonpictorial category (words) with the same frequency as the landscape pictures to further examine the effect of category overlap.

The learning mode hypothesis would predict that highly novel items, such as new fractals, would exert a stronger effect on subsequent recognition judgments than less novel items, such as a new landscape image.

The continuity field hypothesis would predict that the old/new status of the preceding item would only affect recognition judgments to the current one if the two items were similar, i.e., of the same category.

Results

Experiment 1

The afterglow effect is a bias, not a change in sensitivity. The standard way of separating bias from sensitivity, signal detection theory, has been criticized as inadequate (Macmillan and Creelman 1991; Grider and Malmberg 2008). Our main analyses were therefore performed on the raw data (though see Supplemental Materials for a signal detection analysis confirming that within this framework of analysis indeed only bias was affected). Figure 1A,B show the percentage of items judged correctly to be old or new as a function of current old/new status and accuracy, when preceding and current item are of the same category, or of different categories, respectively. We first performed a RM ANOVA testing the learning mode hypothesis, in which we looked at the effects of current item status, preceding item status and preceding category specifically for landscape items. The afterglow effect would show as a two-way interaction between current and preceding item status. This interaction was indeed present (F(1,28) = 5.38, P < 0.05, partial η2 = 0.16). The learning mode hypothesis would predict that this interaction would be qualified by a three-way interaction including preceding category, as it predicts that the afterglow effect is stronger after fractals than after landscape items. This predicted three-way interaction between preceding category, preceding item status and current item status was not there (F(1,28) = 1.76, P = n.s.)—if anything, the afterglow effect seemed stronger for preceding landscapes.

Figure 1.

(A,B,D,E) show accurate responses for old and new items, as a function of preceding item status (old or new) and whether the current and preceding item were of the same category (A,D) or of a different category (i.e., a fractal preceded by a landscape; B,E). (C,F) show the same data, recomputed as an Afterglow effect measure. This was computed as mean accuracy when current and preceding item had the same status (old/old or new/new), minus accuracy when current and preceding item had a different status (old/new or new/old). A higher value indicated a stronger afterglow effect. This is shown as a function of current and preceding item category. A–C show data of Experiment 1, D–F of Experiment 2. Error bars represent standard errors of the mean.

To test the continuity field hypothesis, we performed another ANOVA. We now collapsed over current category, and analyzed performance as a function of current item status, preceding item status, and whether current and preceding category were the same (category similarity). The continuity field hypothesis predicts that the afterglow effect is stronger when the current and preceding category are the same, which translates statistically into a three-way interaction. This interaction was indeed there: the RM ANOVA testing the category hypothesis did reveal a three-way interaction between category similarity, preceding item status and current item status (F(1,28) = 7.81, P < 0.01, partial η2 = 0.22). To get more insight in the factors driving this interaction we ran two 2 × 2 ANOVAs, one where the same category was presented and one where different categories were presented. When we compared trials of the same category, we found no main effect of “preceding status” (F(1,28) < 1), a borderline significant main effect of “current status” indicating a somewhat conservative response bias (new > old ; F(1,28) = 4.20, P = 0.05, partial η2 = 0.13), and the interaction between preceding status and current status (F(1,28) = 7.23, P < 0.05, partial η2 = 0.21) that shows the afterglow effect. When comparing only trials of different categories, we found no main effect of preceding status (F(1,28) < 1), a main effect of current status (new > old; F(1,28) = 12.92, P < 0.01, partial η2 = 0.32), and no interaction between preceding status and current status (F(1,28) < 1), so no afterglow effect.

The afterglow analysis, where we computed the effect of the preceding item's old/new status on the current decision for each category separately (see Materials and Methods) is shown in Figure 1C. This analysis showed no main effect of preceding category (F(1,28) < 1), a main effect of current category (F(1,28) = 4.28, P < 0.05, partial η2 = 0.15) and an interaction between preceding category and current category (F(1,28) = 5.82, P < 0.05, partial η2 = 0.20). These findings show that the preceding trial influenced the current decision both when current and preceding items were landscapes and when both were fractals, but not when category switched from the preceding to the current trial, replicating and extending preceding findings.

The analyses above were performed using actual preceding item status as dependent measure (i.e., old vs. new). To see whether it is actual status or the recognition decision that affects the current decision, we performed another 2 × 2 × 2 analysis with only the more frequent landscape items, using as independent variables current status (old/new), preceding status (old/new), and preceding recognition decision (old/new). This analysis showed no three-way interaction (F(1,28) = 2.83, n.s) but did reveal two-way interactions between current status and preceding recognition decision (F(1,28) = 11.64, P < 0.001, partial η2 = 0.30), and preceding status and preceding recognition decision (F(1,28) = 11.64, P < 0.05, partial η2 = 0.14) (see Supplemental Fig. S2, panels A and B). This analysis shows that it is the preceding recognition decision, not the preceding item status, that is most closely tied to the afterglow effect.

Experiment 2

Figure 1D through 1F show the results of Experiment 2. Here, we only tested the continuity field hypothesis, using the same 2 × 2 × 2 RM ANOVA as for Experiment 1 with current item status, preceding item status, and category similarity as factors. The RM ANOVA revealed a three-way interaction between these three factors (F(1,24) = 5.21, P < 0.05, partial η2 = 0.18). To get more insight in the factors driving this interaction we again ran two 2 × 2 ANOVAs, one where items were of the same category on the current and preceding trial (Fig. 1C) and one where items were of different categories (Fig. 1D). In the ANOVA with same-category trials, we found no main effect of preceding status (F(1,24) < 1) or of current status (i.e., no specifically conservative or liberal response bias; F(1,24) = 2.55, n.s., partial η2 = 0.10), but we did find the interaction between preceding status and current status that reflects the afterglow effect (F(1,24) = 11.90, P < 0.01, partial η2 = 0.33). In the ANOVA with different-category trials, we found a main effect of preceding status (F(1,24) = 7.02, P < 0.05, partial η2 = 0.23), with better accuracy when the preceding item was old, no main effect of current status (F(1,24) = 0.04, P < 0.01, partial η2 = 0.001), and no interaction between preceding status and current status (F(1,24) < 1)—i.e., no afterglow effect.

We again analyzed our afterglow effect measure to ascertain that the afterglow effect was seen for both categories. This analysis showed no main effect of preceding category (F(1,24) = 1.99, n.s., partial η2 = 0.08) or of current category (F(1,24) < 1), but an interaction between preceding category and current category (F(1,24) = 5.21, P < 0.05, partial η2 = 0.18) showing that the afterglow effect was only found when the preceding and current category was the same (Fig. 1F).

We also performed an analysis to test whether the preceding item status or the preceding recognition decision that mattered most. We performed a 2 × 2 × 2 × 2 ANOVA with as factors category similarity, preceding item status, preceding recognition decision, and current item status. The afterglow effect is reflected in the three-way interaction between category similarity, current item status, and preceding item status. However, when preceding recognition decision was also included in the analysis, this three-way interaction was borderline not significant (F(1,24) = 4.23, P = 0.051, partial η2 = 0.15). Instead, the three-way interaction between category similarity, current item status, and preceding recognition decision was significant (F(1,24) = 31.87, P = 0.000, partial η2 = 0.57) (see Supplemental Fig. S2, panels C and D). Thus, again, the afterglow effect reflected the preceding recognition decision more than preceding item status.

Experiment 2 thus corroborates findings in experiment 1 in showing that the preceding recognition decision influenced the current decision specifically when the stimulus category was matched (i.e., when on both the current and the preceding trial a fractal or a landscape was shown, old/new accuracy was higher), even when the category is nonpictorial (words).

Discussion

Here, we show that afterglow effects on recognition memory are not driven by a general learning mode, as the perceptually novel fractal images in Experiment 1 did not yield larger afterglow effects on landscape images than other landscape images did. Rather, we show that such effects are modulated by categorical similarity, supporting the continuity field hypothesis. In Experiment 2, where we only tested category effects, we replicated this effect of category and extended it to the recognition of words: landscape–landscape and word–word sequences yield afterglow effects but sequences in which the category changed did not. Word–word sequences in fact seemed to yield the same effects as landscape–landscape.

In our design, unlike the findings of Duncan et al. (2012), perceptual novelty did not yield an afterglow effect. The assumption that these types of effects are dependent on a lingering learning mode driven by pattern separation and completion mechanisms is thus not substantiated by our findings. This adds to the evidence (for review, see Schomaker and Meeter 2015) that a learning mode fluctuating as a function of novelty on a time course of seconds in unlikely to exist in humans (although exposure to novel environments may boost learning somewhat on a time scale of tens of minutes—e.g., Schomaker et al. 2014). Rather, we show that afterglow effects are strongly affected by categorical similarity as is predicted by a continuity field hypothesis (Fischer and Whitney 2014). This was already suggested by Annis and Malmberg (2013), although similarity in their study was not manipulated directly in their study. Our second experiment additionally showed that this does not need to be a pictorial input, as the afterglow effect is also present when recognizing previously encountered words.

Annis and Malmberg (2013) explained the afterglow effect within the Retrieving Efficiently from Memory model (REM, Shiffrin and Steyvers 1997) by reference to the notion of cue features carrying over from one trial to the next. If the preceding cue was successful in generating a recognition signal on the preceding trial, it would also help generate such a signal in the current trial. The authors constructed a variant of REM to include this account, which could explain the main effects—if anything it overestimated the magnitude of the afterglow effect. Since it predicted that the afterglow effect was stronger when the preceding and current item are similar, it can probably also account for our results.

The idea of a continuity field (Fischer and Whitney 2014) on the other hand, has not been formalized, to our knowledge, in a computational model. It can be summarized as that our perception is some weighted average of the current and of preceding input, provided that that preceding input is similar enough to the current one. Applied to recognition memory, that could be rephrased as that a recognition judgment is some weighted average of the familiarity of the current input, and that of recent similar inputs. This may seem a stretch, but it is quite consistent with how familiarity is thought to be implemented in the brain. A growing literature as supported the idea that many neurons in the perirhinal cortex, an area in the medial temporal lobe that is suggested to pass perceptual signals to the hippocampus, are responsive to whether current inputs are new or not recently seen. When a current input does not elicit a strong perirhinal response, this is interpreted by the brain as a signal that the current input is familiar (Xiang and Brown 1998; Brown and Banks 2015; Scofield et al. 2015). It is possible that perirhinal responses reflect the newness of both current and recent inputs. This would mean a stronger “new” signal when the preceding item was new and similar to the current one, and a stronger familiarity signal when the preceding item was similar to the current one and familiar. Crucially, such an account would suggest that the effect of the preceding item is a bias, not a change in sensitivity (since it does not depend on whether the current item is actually familiar or novel), and that it is dependent more on whether the preceding item is thought to be new or old, than whether it is truly new or old (since the recognition decision would presumably reflect previous perirhinal responses more accurately than actual item status). Both suggestions are consistent with our results. The account would also make the afterglow effect a special case of the perceptual biases caused by preceding items. This is consistent with the fact that in primates perirhinal cortex is not only a memory-related, but also a visual perceptual area (e.g., Murray et al. 2007).

Both Annis and Malberg's (2013) account and the one offered above rely on the idea that the afterglow effect reflects familiarity processes, as opposed to recollection processes (Mandler 1980). It is possible that a recognition test relying more strongly on recollection (e.g., associative recollection) would reflect different processes, and yield different results. This has not been tested so far.

In sum, our findings show that the afterglow effect (or sequential dependency effects), where a judgment made to a preceding item in a sequence affects judgments of a current item, is not affected by the novelty of the preceding item, but is dependent on the preceding item being of the same category as the current one. This means that the presence of a general lingering learning mode that is governed by pattern separation and completion processes is unlikely to explain the afterglow effect. We here propose that these afterglow effects are governed by a continuity field that is modulated by perceptual similarity.

Materials and Methods

We ran two experiments in which participants were asked to encode and subsequently recognize new and old images from two categories (fractal or landscape in Experiment 1, and word or landscape in Experiment 2).

Participants

Twenty-nine volunteers participated in Experiment 1 (23 women, mean age 19.64 (SD: 1.68)). Participants were students of the Vrije Universiteit Amsterdam who received partial credit toward a course requirement. Ethical approval was obtained before start of the experiments from the ethical committee (VCWE) of the faculty of Behavioral and Movement Sciences of Vrije Universiteit Amsterdam.

Stimuli

Stimuli consisted of 289 images of outdoor landscapes and 73 fractals taken from (Rangel-Gomez et al. 2015). They were presented in the middle of the screen at half-width of the 21-inch screen.

Experimental design

We executed a 2 × 2x2 × 2 within-subject design with independent variables preceding category (fractal or landscape in Experiment 1, word or landscape in Experiment 2), current category (fractal or landscape in Experiment 1, word or landscape in Experiment 2), preceding item status (old or new), and current item status (old or new). These variables were compared with the dependent variable accuracy on the recognition task. The experiment was programed using OpenSesame (Mathôt et al. 2012) and participants were tested on a desktop computer in a secluded booth. After signing informed consent and reading an instruction, participants completed four practice trials where they were tested using one new fractal, one old fractal, one new landscape and one old landscape, in randomized order. The old images were images used as examples in the instructions. After being able to ask questions about the experiment, participants started the experiment. They studied the images in random order in nine blocks of 40 items (32 landscapes and eight fractals) while performing a continuous recognition task. For each image they indicated whether they had seen it before (by pressing “z” on the keyboard), or whether it was new (by pressing “m” on the keyboard). Half of the images in each block were indeed new (i.e., not shown before), and the other half were randomly picked from all images seen in one or more of the preceding blocks (since selection was random, so an “old” image could also be repeatedly picked). The reason for this arrangement was that if we would sample without replacement, the “old” items would necessarily be mostly recent items (since older items would all already have been picked). This would mean that the task would get a heavy short-term memory component, which we wanted to avoid, and repetition of “old” items is not uncommon in continuous recognition (Hannay and Malone 1976). The images seen in the practice trials were used as “old” in the first block. After answering, a blank screen was shown for 500 msec and participants proceeded to the next trial. After each block, feedback was given about the accuracy in that block.

Analyses

We compared the four variables described above with the dependent variable accuracy on the recognition task. We decided to use raw accuracy of responses as dependent measure, as opposed to sensitivity measures such as d-prime because the afterglow effect is a bias effect affecting hit and false alarm rates in the same direction and to approximately similar extent. False alarm rates were rescored to correct rejection rates so that in both cases a higher number would reflect better performance (moreover, since hit and correct rejection rates were of approximately the same magnitude, and afterglow effects would show up as an easily interpretable crossover interaction). A traditional signal detection analysis was also run, and is reported in the Supplemental Materials. It confirmed that afterglow was a bias and not a sensitivity effect.

A full four-way analysis of accuracy can be found in the Supplemental Materials, but results will not be interpreted here.

To be able to specifically address our hypotheses, we used three 2 × 2 × 2 repeated measures (RM) ANOVAs. The first RM ANOVA was set up to address our novelty hypothesis. Here, we only considered landscape pictures in the current condition and examined the effects of preceding category, preceding item status and current item status on accuracy on performance on landscape pictures in the recognition task. To address the continuity field hypothesis, we collapsed preceding and current category (same or different) and contrasted this factor with preceding item status and current item status. When a significant three-way interaction was found in these models, we ran subsequent 2 × 2 post-hoc analyses to further define the results.

Finally, to be able to analyse the afterglow effect for the specific categories (as opposed to collapsing across the two), we computed an afterglow effect measure, which computes the effect of the preceding item's old/new status on the current decision for each category separately. This was computed as p(correct|Oc & Op) + p(correct|Nc & Np) − p(correct|Oc & Np) − p(correct|Nc & Op), where Oc refers to current “old,” Np to preceding “new,” etc. We then compared the afterglow measures for preceding category (fractal or landscape in Experiment 1, word or landscape in Experiment 2) and current category (fractal or landscape in Experiment 1, word or landscape in Experiment 2) in a 2 × 2 RM ANOVA. Alpha was set at 0.05 throughout.

Twenty-five volunteers participated in Experiment 2 (21 women, mean age 19.24 (SD: 1.75)). Participants were again students of the Vrije Universiteit Amsterdam who received partial credit toward a course requirement. Ethical approval was obtained before start of the experiments from the ethical committee (VCWE) of the faculty of Behavioral and Movement Sciences of Vrije Universiteit Amsterdam.

In Experiment 2 the categories were comprised of words and landscapes instead of fractals and landscapes. The same 246 landscape pictures were used, and in addition 246 Dutch concrete nouns (4 to 12 letters) were used, taken from a preexisting list in our laboratory (Schomaker et al. 2014).

Experimental design and analyses

The experimental design was very similar to Experiment 1, with the following differences: The practice phase now had 12 trials, the experiment now consisted of 10 blocks of 48 trials and the division between words and landscapes was now 50/50 instead of 20/80 (20/80 had been adopted to test the learning mode hypothesis, which was not tested in Experiment 2). Analyses were the same as for the continuity field analysis described for Experiment 1.