Recollection-dependent memory for event duration in large-scale spatial navigation.

Time and space represent two key aspects of episodic memories, forming the spatiotemporal context of events in a sequence. Little is known, however, about how temporal information, such as the duration and the order of particular events, are encoded into memory, and if it matters whether the memory representation is based on recollection or familiarity. To investigate this issue, we used a real world virtual reality navigation paradigm where periods of navigation were interspersed with pauses of different durations. Crucially, participants were able to reliably distinguish the durations of events that were subjectively "reexperienced" (i.e., recollected), but not of those that were familiar. This effect was not found in temporal order (ordinal) judgments. We also show that the active experience of the passage of time (holding down a key while waiting) moderately enhanced duration memory accuracy. Memory for event duration, therefore, appears to rely on the hippocampally supported ability to recollect or reexperience an event enabling the reinstatement of both its duration and its spatial context, to distinguish it from other events in a sequence. In contrast, ordinal memory appears to rely on familiarity and recollection to a similar extent.

Episodic memory represents sequentially unfolding events, each associated with a particular temporal and spatial context (Moscovitch et al. 2006; Conway 2009; Howard et al. 2014). Time, therefore, is a crucial aspect of episodic memory, and key to the conscious reexperience, or recollection, of past events (Tulving 1985, 2002; Nyberg et al. 2010). An important distinction specific to the relationship between the hippocampus and episodic memory is the dual-process memory theory, which we integrate here for the first time with temporal processing. The dual-process account draws a distinction between recollection (“remembering”) an event, which is often accompanied by a rich set of contextual information such as the temporal and spatial context, and familiarity, or “knowing” that an event had happened, which is based on a sense of familiarity with a presented cue, but bereft of its context at acquisition (Tulving 1985; Skinner and Fernandes 2007; Hudon et al. 2009). With regard to hippocampal activity, research has shown that whereas recollection can elicit hippocampal activation, familiarity is more frequently associated with a network of regions in the medial temporal lobes that do not include the hippocampus (Ranganath et al. 2004; Bowles et al. 2007; Diana et al. 2010; Koen and Yonelinas 2014).

Space and time both provide fundamental contexts for episodic memories (Howard and Eichenbaum 2013). This intrinsic connection between the two modalities motivates a novel hypothesis that memory for the passage of time may be strongly linked to “recollection,” just as memory for the spatial context in which an event occurs. In contrast, temporal memory should be less associated with the less contextual memory type of “familiarity.” Here, we develop this hypothesis by relating it to behavioral and neural studies in humans and rodents on spatial and temporal memory, and test it using a real world virtual reality task based on Google Street View.

Our hypothesis connecting temporal representation and recollection is supported by studies showing that the hippocampus, which is crucial for recollection (Eichenbaum et al. 2007; Moscovitch et al. 2016), is also implicated in spatial and temporal memory, suggesting that the same network of neural substrates may underlie the encoding of both time and space into episodic memory. The role of the hippocampus in spatial navigation is well established (Burgess et al. 2002; Spiers and Maguire 2007; Spiers and Barry 2015). Research in rodents suggests that the capacity for spatial navigation is supported by hippocampal place cells and grid cells, which code the structure of a particular space and track an animal's own position in it (O'Keefe and Dostrovsky 1971; O'Keefe and Nadel 1978; Hafting et al. 2005). Place-cell-like activity has also been observed in humans (Ekstrom et al. 2003; Miller et al. 2013). Supporting this view is evidence that lesions to animal hippocampi interfere with spatial navigation (Morris et al. 1982; Whishaw et al. 1995; Riedel et al. 1999), and patients with hippocampal damage similarly suffer from deficits in spatial navigation and self-localization (Worsley et al. 2001; Maguire et al. 2006; Bartsch et al. 2010; Goodrich-Hunsaker et al. 2010). Therefore, the hippocampus has clear ties to representing spatial environments and contributing to spatial navigation and learning. Keeping track of the traversed path is equally important in terms of both distance and time, yet it is less well understood how time is tracked and remembered during spatial navigation, and how events that occur along a path are incorporated into this temporal representation.

Recently, evidence for hippocampal cells concurrently tracking the passage of time and the distance travelled during navigation was reported in rodents, providing insight into a possible mechanism of such temporal processing (Pastalkova et al. 2008; MacDonald et al. 2011; Jacobs et al. 2013; Kraus et al. 2013; Eichenbaum 2014; Salz et al. 2016). In addition, evidence from human neuroimaging studies suggests that patterns of hippocampal activation track the ordinal sequence of events (Tubridy and Davachi 2011; Hsieh et al. 2014; Davachi and DuBrow 2015) and carry information about the duration of different events (McEchron et al. 2003; MacDonald et al. 2012; Barnett et al. 2014). No previous studies, however, have attempted to explore how temporal duration is encoded into event representations in episodic memory and retrieved as part of a broader context of an event memory.

Though a variety of evidence supports the idea that the hippocampus is involved in coding both ordinal and temporal information, Howard and Eichenbaum (2013) proposed a view that relates both types of information processing to the hippocampus but differentiates the manners in which they are represented. Specifically, they argue that the ordinal representation of a sequence of events is devoid of information about the duration of individual events and merely stores the sequential position of individual events from an episode. The temporal representation, however, retains the entire timeline of an episode, incorporating the durations of individual events, but also allowing for the extraction of ordinal level information. Based on this account, we can predict that the ability to recall the duration of an event (relying on the temporal representation) would be enhanced when the spatiotemporal context can be reinstated (recollection), while this would have a smaller impact on the capacity to retrieve ordinal information. In contrast, ordinal information can be extracted from a less detailed representation, therefore relying on recollection to a lesser degree. The present study seeks to investigate the behavioral implications of this hypothesis by using recollection and familiarity measures as proxies for hippocampal and extra-hippocampal representation. Based on predictions from Howard and Eichenbaum (2013), we expect that event duration information may be more recollectively dependent than ordinal information alone.

To bring together these multiple research streams, and link the capacities to encode time and space with the posited hippocampal mechanism of detailed recollection, we created a real-world virtual reality spatial navigation task. We based this task on the rodent time-cell paradigm in which subjects navigated along a preset route with specific temporal gaps between segments of the journey (Kraus et al. 2013). We extended this type of task to human participants with the aim of investigating the encoding and retrieval of event duration and sequential position from long-term memory. The encoding of duration and sequential position was made incidental in the present task to allow us to explore whether temporal information is incorporated into event memories without explicitly attending to the time elapsed. To our knowledge, this is the first study to investigate long-term memory for temporal information of events nested in a sequence biasing participants’ attention toward event durations. While retrospective duration judgments have been examined in the past (Block and Zakay 1997; Zakay and Block 2004), participants in these studies were often asked to estimate duration after the presentation of each stimulus, and, more importantly, stimuli were not event-like and embedded into a continuous spatial context, which is how we experience events in real life. A recent study provided evidence that spatial boundaries shape memory for the temporal order in which two objects were presented, regardless of the spatial and temporal distance separating them (Horner et al. 2016). However, the participants in this study were asked to discriminate between only two objects and the spatial boundaries were within a smaller scale environment. Here, we use an extended sequence during large-scale navigation such that the distances between individual intersections may serve as retrieval cues for ordinal memory.

In addition to dissociating the traversal of spatial and temporal dimensions, we manipulated the degree of active interaction with the environment. Active engagement with a task and volitional control result in superior memory in a variety of domains (MacLeod et al. 2010; Voss et al. 2011; Ozubko et al. 2012). Interval timing was shown to rely, in part, on areas typically associated with motor control, suggesting that sensorimotor feedback during a temporal interval may benefit the encoding of duration information (Merchant and Yarrow 2016). We thus made the experience of the passage of time during stops active for half of participants, and passive for the other half. We controlled participants’ movement through space so that they were led along the route at a set pace to keep temporal aspects of the task consistent. This manipulation allowed us to investigate whether active interaction with the environment enhances memory nonselectively (i.e., both duration and temporal order in both tasks), or whether memory for duration may be selectively enhanced when only that aspect of navigation is made active.

In summary, we aimed to investigate how memory for temporal duration is influenced by the subjective quality of the memory for the event in which it is embedded, and if this differed for memory for the ordinal position of events during spatial navigation. We also aimed to investigate the effect that active experience of the passage of time may have on temporal and ordinal memory. Participants learned a route with eight stops of different durations in a naturalistic virtual reality version of Chicago, a city with which they were not familiar, based on Google Street View. After learning the route, participants engaged in a recognition memory task, where eight old intersections and eight new intersections were intermixed and participants indicated for each whether they were recollected, familiar, or new. They were then shown pairs of intersections and asked to indicate where they waited longer (duration discrimination), and which of the two intersections was closer to the end of the route (ordinal discrimination). Participants also sorted the intersections by their route order and by increasing duration, and estimated the duration spent waiting at each intersection (see Fig. 1 for an overview of the methods, and Material and Methods for detailed procedures). We predicted that we would observe an improvement in performance on both ordinal position and duration discrimination as a function of the distance between the events’ ordinal positions and difference between their durations, respectively, based on the pattern separation literature showing performance improvement as a function of the distance between stimuli (Holden et al. 2012). Furthermore, we predicted that accuracy at discriminating event duration and ordinal position would be significantly better when participants reported recollecting (“reexperiencing”) an event in comparison to having a sense of familiarity (“knowing”) alone. We also predicted that temporal memory would benefit more from recollection than ordinal memory, as self-reported reexperiencing would suggest more extensive processing of contextual information and the incorporation of temporal information into this contextual representation.

Figure 1.

(A) First-person view screenshots from two of the intersections along the route. In the passive wait condition, participants simply waited while stopped at an intersection. In the active wait condition, participants held down the space bar for the duration of the wait. (B) Map of the entire route illustrating how the degree of temporal and ordinal separation was calculated. Temporal separation was the difference between durations of each pair of intersections compared, and ordinal separation was the difference in the sequential positions between each pair of intersections. (C–F) Memory tasks in the order performed by the participants. (C) Recognition memory: participants were prompted to respond whether each item was new, reexperienced, or familiar. (D) Temporal and ordinal discrimination: each intersection was paired with every other intersection and participants indicated at which they waited longer (duration discrimination), or which was closer to the end of the route (ordinal discrimination). (E) Duration and sequence order: participants ordered images of all the intersections according to their duration (i.e., shortest to longest) and sequential position along the route. (F) Time estimation: participants typed the number of seconds they believed they had spent waiting at each of the intersections.

Results

All items incorrectly endorsed as “new” in the recognition memory task (misses) were removed from all subsequent analyses. Prior to any analyses, all intersections that were marked as guesses (rated 1 on the confidence scale) in the temporal and ordinal discrimination tasks were removed from the analysis for each of the two tasks. In the active condition, these measures excluded a total of 5.8% of all responses in the ordinal discrimination task (5% misses, 0.8% guesses), and 12.2% of all responses in the duration discrimination task (3.4% misses, 8.8% guesses). In the passive condition, these measures excluded 8.4% of all responses in the ordinal discrimination task (6.4% misses, 2.0% guesses), and 10.7% of all responses in the duration discrimination task (6.4% misses, 4.3% guesses). Data from two participants in the active condition were excluded from analyses in the duration discrimination task because their mean proportion correct was significantly below chance, suggesting the task was completed incorrectly. Their data from the ordinal discrimination task were included in the analyses.

Because mean accuracy on the duration sorting task was not significantly above chance (M = 0.125; P = 0.445) in the passive condition, and there were large and significant differences between duration and order sorting tasks in both conditions, these results were not easily interpretable. Due to this, we report data from the ordering tasks in the Supplemental Material (see Supplemental Fig. S1).

Recognition memory

On the recognition memory task in the active condition, the mean proportion of false alarms (lures incorrectly recognized as “old”) was 0.06 (SD = 0.06) and the proportion of misses (old items incorrectly recognized as “new”) was 0.02 (SD = 0.03). Of the correct old responses, the proportion of reexperience (recollection) responses was 0.80 (SD = 0.29) and the proportion of know (familiarity) responses was 0.20 (SD = 0.29). In the passive condition, the proportion of false alarms was 0.05 (SD = 0.11), and of misses, it was 0.02 (SD = 0.03). Of the correct old responses, the proportion of reexperience responses was 0.77 (SD = 0.27), while the proportion of know responses was 0.23 (SD = 0.27). Independent samples t-tests indicated no significant differences in the proportions of R responses, misses, or false alarms between the two conditions (all P-values >0.4).

Duration and ordinal discrimination

The overall proportion correct on the duration discrimination task in the active condition was 0.72 (SD = 0.14) and 0.62 (SD = 0.17) in the passive condition, collapsed across all durations and distances. Two one-sample t-tests indicated that performance in both the active (t(21) = 7.26, P < 0.001, d = 1.55) and the passive (t(22) = 3.33, P = 0.003, d = 0.69) conditions was significantly better than chance levels of 50%.

The overall proportion correct on the ordinal discrimination task in the active condition was 0.95 (SD = 0.05) and 0.92 (SD = 0.12) in the passive condition, collapsed across all durations and distances. One-sample t-tests indicated that performance in both the active (t(23) = 41.64, P < 0.001, d = 8.50) and the passive (t(22) = 16.12, P < 0.001, d = 3.36) conditions was significantly above chance levels of 50%.

Multilevel modeling

We used participants’ recognition memory judgments to sort the trials on the duration and ordinal discrimination tasks. Trials were divided according to whether participants had previously reported recollection or familiarity for each of the intersections in each pair. This allowed us to divide trials into two categories: where both intersections were reported as reexperienced (Both-R), and where both were reported as familiar (Both-K). We did not include mixed trials (R–K) to enable us to investigate pure familiarity and recollection. The full analysis comparing Both-R to Some-K (where either one or both of the intersections in a pair were familiar) revealed largely the same pattern of results and is reported in the Supplemental Material (Supplemental Fig. S3). For each ordinal discrimination comparison, the number of road segments travelled between the two intersections being compared on a given trial was calculated (Fig. 1B). This number was designated as the degree of ordinal separation. For each duration discrimination comparison, the difference in durations was calculated (Fig. 1B), and designated as the degree of temporal separation.

To test the primary outcome measures, duration and ordinal discrimination, in relation to recollection and familiarity, while accounting for degree of separation and degree of control (i.e., active versus passive), we fitted a logistic regression model to the data. A two-level trial-by-trial model was used to account for participants being nested within the passive and active conditions. This approach has the advantage of increasing statistical power relative to averaging per participant, as each trial is considered as an individual observation. The calculation of power is more complicated compared with single-level designs due to the complexity of the interaction between the sample sizes across the two levels (Mathieu and Chen 2011; Mathieu et al. 2012). Because no similar study has been run before, a priori power calculation was not possible in this case, as parameter estimates were not available. Post hoc power analyses are not recommended, as all significant outcomes will indicate sufficient power. On the basis of simulations, however, Mathieu et al. (2012) suggest that Level 1 sample size >18 will afford power of over 80% to detect cross-level interactions with Level 2 sample size of around 35 (Mathieu et al. 2012). As we used a trial-by-trial model, the power of the tests reported here can be considered sufficient. Here, our Level 2 sample size was 47 participants. Each participant, nested within the level 2 predictor of the active versus passive condition, had between 14 and 28 trials, or “observations” within the level 1 predictor (the number varies depending on the number of guesses or errors, as explained above). This outcome suggests that our analyses have sufficient power to detect the effects reported, as our sample sizes across the two levels were adequate.

Participants were entered as a random effects term and a random intercept and slope were estimated for each participant. We first fitted a trial-by-trial logistic regression model to our data to predict the likelihood of an incorrect (0) or correct (1) outcome in the binomial distribution of scores from five factors: (1) task type (duration versus ordinal discrimination), (2) ordinal or temporal separation (mean normalized to allow for direct comparison), (3) recognition memory category (Both-R or Both-K), (4) passive versus active wait condition, and (5) participants’ confidence in their response. An interaction term was entered between the recognition memory category and task type (ordinal versus duration discrimination) to test whether there was a significantly different effect of RK on duration relative to ordinal discrimination. We then performed a Wald's chi-square test on the model (Hauck and Donner 1977) in the linear model output to assess the significance of each factor. The effect size of each model is reported as the conditional R2 statistic (R2c), which is interpreted as the variance captured by both fixed and random factors. This analysis revealed that there was a significant difference in performance between duration and ordinal discrimination (χ2(1) = 30.42, P < 0.001), a significant main effect of temporal or ordinal separation (χ2(1) = 34.62, P < 0.001), and a significant main effect of confidence (χ2(1) = 26.55, P < 0.001). There was also a significant main effect of the recognition memory category (χ2(1) = 10.98, P = 0.001), but no significant main effect of passive versus active wait (χ2(1) = 2.06, P = 0.151). The interaction between task type and recognition memory category did not reach significance (χ2(1) = 2.16, P = 0.141). The model effect size was R2c = 42.0% (the number of trials entered into the model was 1719) (Fig. 2).

Figure 2.

Performance in the duration and ordinal discrimination tasks. (A) Duration discrimination performance as a function of temporal separation between intersections, P < 0.001 (B) Ordinal discrimination performance as a function of ordinal separation between intersections, P < 0.001. (C,D) Performance on duration and ordinal discrimination tasks as a function of their R–K category. There was a significant main effect of R–K category for duration (P = 0.007), but not for order (P = 0.388). (E) Performance on duration and ordinal discrimination when performance was not significantly different between the two tasks (P = 0.115). There was a significant main effect of R–K for duration (P = 0.012), but not ordinal discrimination (P = 0.287). The error bars represent standard error.

To compare performance against chance in the duration discrimination task for pairs of Both-R and Both-K intersections, as well as in the duration discrimination task for pairs of Both-R and Both-K intersections, we examined the 95% confidence intervals for performance on the two discrimination tasks (see Supplemental Fig. S2). Confidence intervals are more suitable for the multilevel modeling approach reported here, as they reflect the strength and direction of the effect shown without averaging across participants (Gardner and Altman 1986; Altman 2000). Performance was not reliably above chance only in the Both-K category in the duration discrimination task (where the lower 95% CI was below 0.50), while it was consistently above chance in all other cases. This suggests that retrieval of duration information was not reliably above chance when the intersection was merely familiar, but the retrieval of ordinal information was possible by relying on both recollection and familiarity.

To test whether the inclusion of additional interactions between factors captured any additional variance, we added interaction terms between passive versus active wait, recognition memory category, and task type (duration versus ordinal discrimination). This analysis revealed no difference between our reported model and this extended one (χ23 = 3.47, P = 0.325), suggesting that our model with only the interaction of interest captured sufficient variance in the data. While we failed to observe a significant interaction to support the notion that recollection has a greater contribution to memory for duration relative to memory for order, these analyses are limited by performance differences between the ordinal and duration discrimination tasks. Specifically, performance was significantly better on the ordinal discrimination task (Fig. 2A–D), making the interpretation of the relative effects of recollection and familiarity on memory for duration and order difficult due to different distributions of scores.

To address this issue of confounds of task difficulty, we reran the model, but included only bins of comparable performance on both tasks. We entered the four bins with best performance on the duration discrimination task (temporal separation between 6 and 10.5 sec) and the three bins with lowest performance on the ordinal discrimination task (ordinal separation between 1 and 3 road segments). This way, performance was matched as closely as possible, with the inevitable caveat that the numbers of trials differed due to the design of the task (258 trials for duration, 594 for ordinal discrimination). We ran the same model as above, but the factor accounting for temporal and ordinal separation was not entered, as the range would not be representative of that experienced in the study. Critically, while performance was still numerically greater in the ordinal discrimination task, this analysis revealed no significant difference in performance between the duration and ordinal discrimination tasks (χ2(1) = 0.199, P = 0.655), but a significant interaction between recognition memory category and task type (duration versus ordinal discrimination) (χ2(1) = 7.93, P = 0.005). There was a significant main effect of recognition memory category (χ2(1) = 14.13, P < 0.001), and a significant main effect of confidence (χ2(1) = 32.88, P < 0.001), but no significant main effect of passive versus active wait (χ2(1) = 0.002, P = 0.963). The effect size of this model was R2c = 38.0% (the number of trials entered into the model was 852). The significant interaction between recognition memory category and task type suggests that when performance on the two tasks was comparable, there was a significantly greater effect of recollection on memory for duration, relative to memory for order (Fig. 2E). As above, we established that there was no difference in the significance level of any of the main factors and interactions if an interaction term was added between passive versus active wait, recognition memory category, and task type (χ23 = 0.475, P = 0.924), suggesting that our model with only the interaction of interest captured sufficient variance in the data.

Based on this significant interaction, we examined the data from the duration and ordinal discrimination tasks in separate models to establish the relative contributions of recollection and familiarity to performance in each of the individual tasks. We examined both the data that were entered into the model with equated performance and the full data sets for both tasks. Based on the significant interaction when performance was not significantly different, we report the data from bins with matched performance here. Both models revealed the same patterns of results, and the outcomes for the full data sets are reported in Supplemental Material. Performance (0 = incorrect, 1 = correct) was modeled for each of the tasks as a function of (1) recognition memory category, (2) passive versus active wait, and (3) confidence in each response. Participants were again entered as a random factor. Because this was a subset of the data, the factors for temporal and ordinal separation between pairs of intersections were not included, as the range would not be reflective of that experienced during the navigation task.

(χ2(1) = 1.13, P = 0.287), and no significant main effect of passive versus active wait (χ2(1) = 0.126, P = 0.722), but a significant main effect of confidence (χ2(1) = 23.23, P < 0.001). The effect size of this model was R2c = 29.1%.

On the duration discrimination task, there was a significant main effect of the recognition memory category (χ2(1) = 6.30, P = 0.012), and a significant main effect of confidence (χ2(1) = 4.27, P = 0.039). There was no significant main effect of passive versus active wait (χ2(1) = 1.06, P = 0.302). The effect size of this model was R2c = 37.6%.

Collectively, these analyses suggest that memory for event duration was significantly better when both intersections were subjectively reexperienced, while the same was not true for memory for their order, where familiarity was sufficient to discriminate between two ordinal positions.

To establish that neither ordinal position nor duration of wait were influencing the tendency to reexperience an event, we examined reexperience responses as a function of ordinal position and wait duration. We performed two χ2 analyses in which we (1) calculated the frequency of R and K responses for each of the wait durations and (2) calculated the frequency of R and K responses for each of the ordinal positions. We found no evidence that longer durations were more likely to be subjectively reexperienced (χ2(7) = 8.771, P = 0.270). Conducting this test separately confirmed that this was the case in both the active (χ2(7) = 7.897, P = 0.342) and the passive conditions (χ2(7) = 4.811, P = 0.683). The same analysis, performed on frequencies of R and K responses as a function of ordinal positions, revealed a marginally nonsignificant main effect of ordinal position, such that more recent positions tended to be associated with more R responses (χ2(7) = 13.901, P = 0.053). Splitting this analysis by condition, however, revealed that this finding was driven by the passive condition, where the effect was trending (χ2(7) = 13.759, P = 0.056), while it was not significant in the active condition (χ2(7) = 5.770, P = 0.567). This indicates that there was no systematic or statistically robust variation between ordinal position and the tendency to recollect the event that occurred at that ordinal position.

To further compare temporal and ordinal discrimination, we ran a consistency analysis in which we compared participants’ performance on duration and ordinal discrimination tasks for the “same” pairs of intersections. We correlated performance (correct versus incorrect) for the same pairs for intersections for each participant. We then compared these correlation coefficients to 0—if the correlation between the two tasks was not significantly different from zero, this would suggest that different processes contribute to the ability to distinguish between durations and ordinal positions. In the active condition, the mean correlation across participants was r = −0.075, and in the passive condition, it was r = 0.044. These correlations were not significantly different from zero (t(28) = −0.302, P = 0.765). As some participants achieved a perfect score on the ordinal discrimination task, calculating a correlation was not possible.

Because we were not able to correlate performance on the two tasks for all participants, we followed up with a consistency analysis in which we calculated the proportion of trials where each participant (1) accurately discriminated both duration and order, (2) accurately discriminated duration, but not order, (3) accurately discriminated order, but not duration, and (4) inaccurately discriminated between both types of information for the same pairs of intersections. We entered these four categories into a repeated-measures ANOVA (see Supplemental Materials Fig. S4). There was a significant main effect of category (F(3,129) = 140.10, P < 0.001, η2 = 0.765). There was also a significant difference between the active and passive condition (F(1,43) = 33.217, P < 0.001, η2 = 0.436), and a significant interaction between category and condition (F(3,129) = 4.912, P = 0.003, η2 = 0.103). This analysis suggests that participants were significantly more likely to accurately discriminate between two intersections’ ordinal positions, but not their durations, than the opposite direction. A speculative possibility based on these analyses is that participants may have primarily accessed route-general ordinal information, which enabled them to subsequently access event-specific temporal information.

This interaction does not appear to be supported by a bidirectional underlying mechanism, however. An exploratory analysis reported in Supplemental Material (Supplemental Fig. S5) suggests that performance on duration discrimination cannot be predicted from the distance in ordinal positions, and performance on ordinal discrimination cannot be predicted from the difference in intersection durations.

Finally, we correlated each participant's “mean” performance on the duration discrimination task with their mean performance on the ordinal discrimination task. There was a significant positive correlation between performance on the two tasks (r = 0.317, P = 0.034), indicating that participants who performed better in one domain, also tended to perform better in the other domain.

Time estimation

We investigated participants’ explicit ability to estimate the elapsed temporal intervals (Fig. 3). We calculated the bias in participants’ temporal estimates by subtracting the actual wait time from their estimated wait time. In the active condition, participants tended to overestimate the time spent waiting, while in the passive condition participants underestimated the time. The average overestimation (bias) in the active condition was 4.50 sec (SD = 3.34), whereas the average underestimation in the passive condition was 2.80 sec (SD = 1.15).

Figure 3.

(A) Participants’ estimates of waiting times at each intersection in the active (dark gray) and passive (light gray) conditions. (B) Time estimates split by recognition memory category in each condition. (C) The difference between estimated and actual time ranks, split by recognition memory category in each condition. The number 0 would indicate no error in the rank of the estimate. The error bars represent standard error.

We entered participants’ estimates into a separate trial-by-trial multilevel generalized linear model in which we predicted participants’ bias on each of their estimates from the time actually spent waiting at that intersection, passive versus active wait, and recognition memory category (i.e., “reexperienced” or “familiar”). There was a significant main effect of passive versus active wait (F(1,54) = 32.50, P < 0.001), a significant main effect of wait duration (F(1,296) = 135.161, P < 0.001), and a significant main effect of recognition memory category (F(1,329) = 7.298, P = 0.007).

There was also a significant interaction between passive versus active wait and recognition memory category (F(1,329) = 6.116, P = 0.014), and a significant interaction between passive versus active wait and wait duration (F(1,296) = 10.359, P = 0.001). There was no significant interaction between recognition memory category and wait duration (P = 0.849), and no significant three-way interaction (P = 0.967). These results suggest that active waiting actually resulted in greater overestimation of wait durations, and this effect was further increased for recollected, but not familiar, intersections. A visual inspection of Figure 3B suggests that wait estimates for recollected intersections were consistently overestimated, but may follow the trend of actual wait durations more accurately than familiar intersections.

As a way to test this trend, we calculated the rank of each of the participants’ estimates. Doing so abolished the issue of highly different ranges in which estimates were made in the passive and active wait conditions and standardized the estimates for each participant. We ran the same model, now predicting the difference in ranks between the actual correct rank and the ranking of each participant's estimate (see Fig. 3C). A difference of 0 would therefore indicate perfect performance (each estimate would correspond accurately to the rank of the actual wait time). We found a significant main effect of the recognition memory category (F(1,208) = 5.491, P = 0.020), but no significant main effect of active versus passive wait (F(1,64) = 1.113, P = 0.295), and no significant main effect of the actual time (F(1,314) = 0.021, P = 0.886). There was a significant interaction between recognition memory category and the actual time spent waiting (F(1,328) = 3.897, P = 0.049). No other interactions were significant (all P > 0.20).

Discussion

The present study used an incidental encoding paradigm embedded in a virtualized real-world environment to show, for the first time, how information about the duration of individual events in a sequence is encoded in episodic memory and how it differs from memory for order. We showed that in an incidental encoding setting, the ability to recollect an event is critical for the retrieval of its duration, but not for its sequential position. We also showed that temporal estimates were more finely tuned when participants actively interacted with the environment and reflected actual wait times more accurately when an intersection was reported as reexperienced.

Recollection, familiarity, and temporal versus ordinal memory

Our findings show that participants’ ability to distinguish between memories for two durations was significantly enhanced when they reported reexperiencing both events with their associated spatiotemporal contexts. In the ordinal discrimination task, however, performance was approximately equal regardless of participants’ subjective sense of recollection (Fig. 2C,D). This pattern of results was reflected in the interaction between task type (ordinal versus duration) and recognition memory category (Fig. 2) when there was no significant difference in performance between the two tasks (Fig. 2E). This suggests that both recollection and familiarity support, or contribute to, memory for order, while recollection primarily contributes to memory for event duration.

The present research, therefore, fits well with the ideas presented in the theoretical framework proposed by Howard and Eichenbaum (2013), which differentiates between ordinal and temporal representations of a sequence of events. While there is established evidence for hippocampal involvement in representing ordinal information for events that occurred within the same sequence (Hsieh et al. 2014; Ranganath and Hsieh 2016), there is also a growing body of empirical work suggesting a crucial role for hippocampal oscillations in encoding, and possibly creating, the temporal representation for such events. For example, Pastalkova et al. (2008) argue that sequential firing of cell assemblies in the hippocampus enables the reinstatement of a particular spatiotemporal position on an internal timeline, or a “jump back in time,” accompanied by a perceptually vivid sense of reexperiencing. The data from the present experiment on delayed recall allow us to speculate that while specialized hippocampal (“time”) cells may track the time elapsed during empty temporal intervals, memory for duration can only be reliably recovered when participants reported the ability to reexperience the event. This finding fits with the idea that recollection is the recovery of a particular state associated with a unique spatiotemporal context (Howard and Eichenbaum 2013), which is presumed to be mediated by the hippocampus (Eichenbaum et al. 2007).

Our data provide behavioral evidence to support the distinction between temporal and ordinal representations. The ability to access duration information appears to rely on a rich, perceptually vivid episodic representation to a much greater extent than does the ability to retrieve ordinal information. For duration discrimination, individual event durations must be retrieved and then compared. For ordinal discrimination, event-specific details are not critically involved and a coarser representation suffices as a basis on which discriminations can be made. This is consistent with the observation that memory for ordinal position in a familiar environment is not dependent on the hippocampus (Rosenbaum et al. 2000, 2004). The finding that discrimination between sequential positions was, nonetheless, numerically better when events were reported as reexperienced is still congruent with the view that distinguishes between the relation of recollection and familiarity to memory for temporal order. While the capacity to reexperience two events is not critical for the ability to distinguish between their serial positions, the addition of a temporal representation from which ordinal information can be extracted is likely to enhance the quality of memory and subjective confidence. As a result, it is also not surprising that subjective confidence in the decision is a significant predictor of performance; indeed, it has been argued that medial temporal lobes also carry a code for confidence (Rutishauser et al. 2015), which may interact with subjective perceptual vividness.

Our findings on memory for duration mirror those in delayed recall for temporal order. For example, Sadeh et al. (2014) showed that only items reported as recollected, and not as familiar, showed a temporal contiguity effect or a reactivation of surrounding items (Kahana et al. 2008), an effect thought to be supported by the hippocampus. Our results, showing that participants’ ability to distinguish between sequential positions and event durations increased as a function of spatial and temporal separation, respectively (Fig. 2A,B), are also consistent with two recent studies showing that distance in both time and space is represented by the hippocampus (Nielson et al. 2015; Deuker et al. 2016).

One possible trivial explanation for the improvement in performance corresponding to the degree of temporal separation is that participants were simply exposed to intersections with longer temporal intervals for more time, making it possible for them to rely on their knowledge only for those specific intersections when making temporal comparisons. We show, however, that the proportion of “reexperience” responses does not increase as a function of wait duration, and that accuracy in duration sorting is approximately equal for the shortest and longest durations (Supplemental Fig. S1). Instead, it is possible this effect is driven by salience as both the very short and the very long durations are equally well remembered. Even in the passive wait condition, where participants’ time estimation curve was completely flat (Fig. 3A), they were nonetheless able to place the shortest and longest wait durations to the appropriate ends of the scale (Supplemental Fig. S1).

We predicted that actively holding a key for different temporal intervals between periods of navigation would enhance memory for duration. There was a trend toward better performance when waiting was active (Fig. 2A), suggesting that motor/perceptual feedback, or simply attention, enables the formation of stronger temporal memory traces, highlighting the importance of active interaction with the environment in the encoding of temporal aspects of new routes. It is important to note, however, that active waiting only appeared to have a significant main effect on explicit estimates of wait times. Participants’ estimates in the active condition reflected the actual durations more accurately than those in the passive condition (Fig. 3). In both conditions, however, subjectively reexperienced intersections were associated with a smaller degree of error in the “relative” positions of their estimates. That is, while the durations at reexperienced intersections were consistently overestimated in the active condition (Fig. 3B), they more correctly reflected the duration's position or rank in the range of all durations.

A possible alternative explanation for the finding that memory for time is enhanced when the wait is active is related to the production effect, according to which performing a task enhances later memory (MacLeod et al. 2010; Ozubko et al. 2012). If this were the only explanation, however, we would predict that both temporal and ordinal discrimination would be improved by an equal degree. Instead, it appears that this benefit is task-specific, such that actively experiencing the passage of empty temporal intervals selectively enhances memory for duration, but not the sequence of events. This manipulation had a small effect, however.

Contextual binding of sequence and duration

Our results are consistent with a broader body of work suggesting that recollection is crucial for the retrieval of source memory when the event is initially encoded as an arbitrary association, but familiarity is sufficient for the retrieval of events where items and contexts are unitized at the time of encoding (Diana et al. 2008). A recent paper showed that while recollection was crucial for the retrieval of a specific, but arbitrary, context of an event (weather), familiarity was sufficient for the retrieval of their sequence (Persson et al. 2016). Our findings are consistent with this view, highlighting the notion that events are encoded along as parts of the sequence in which they had occurred. We further extend this body of work by showing, for the first time, that the encoding of event duration requires the creation of a contextual association, which may rely on recollection at retrieval.

It is possible that temporal information is reliably incorporated into the context of an episodic memory only when the memory is perceptually rich and vivid. Alternatively, it is possible that the sense of recollection might rely on the ability to retrieve the entire temporal and ordinal context of the episode. The apparent recollection-dependent ability to retrieve an event's duration is aligned with the notion that the hippocampus creates arbitrary associations between all manners of features (Moscovitch 2008). In the present study, the duration of individual events was arbitrary to the particular spatial location (i.e., randomized across participants) and disconnected from the environment, which may partly account for the difference in performance. In contrast, the order of the locations had a particular structure which enhanced later retrieval of this information. Ordinal position appears to be encoded much more readily than temporal information, which may be especially salient in the context of navigation. Integrating a predetermined structure into the durations may enhance memory for duration in a similar way, and possibly enable these memories to be retrieved based on familiarity alone as well. Conversely, if ordinal information were made unstructured (nonmeaningful; i.e., visual scenes were shown in a randomized order), memory for the order may become recollection-dependent. These possibilities offer important and interesting avenues for future research.

An important follow-up is to extend our behavioral findings to the brain. That is, though we have speculated about hippocampal representation of time, we only obtained subjective measures of remember/know in our experiment. From a theoretical perspective, it is important to determine whether the hippocampus is implicated reliably in distinguishing between memories for duration and whether the demonstrated distinction between recollection and familiarity also applies to information about time, the same way it does to information about other stimuli that are relationally integrated into a larger context.

Based on the existing literature, there are two prominent possibilities for how time may be integrated into our episodic memories. One possibility is that the proposed hippocampal “time cells” track the passage of time and locally integrate this information in addition to the code for location and space and ordinal information (Howard and Eichenbaum 2013; Eichenbaum 2014). A second option is that a separate temporal signal is received by the hippocampus from another cortical or subcortical structure (Howard and Eichenbaum 2013; Meck et al. 2013). Strong candidate structures are the striatum (Meck et al. 2008; Adler et al. 2012, 2013; Hsieh and Ranganath 2015), parietal cortex (Danckert et al. 2007; Davis et al. 2009; Wiener et al. 2010; Vicario et al. 2013), and the insula (Wittmann et al. 2010). The striatum in particular has been suggested to code for a conjunctive representation of an event and its temporal properties in a manner akin to time cells (Adler et al. 2012; Howard and Eichenbaum 2013). Striatal involvement may play a key role in explicit temporal judgments, as the strongest effect of active versus passive wait emerges in time estimates provided by participants. Exploring striatal–hippocampal interactions in interval timing would provide key evidence for the resolution of this argument.

Conclusions

The findings reported here show, for the first time, how information about duration is encoded into episodic memory in a rich virtual reality navigation paradigm. We show that the pattern of results describing memory for time closely resembles previously reported findings on temporal memory and that the reexperience of a past event appears to play an important role in enabling the retrieval of temporal information. The subjective reinstatement of an event's spatiotemporal context appears to support the retrieval of temporal information, consistent with current theories of hippocampal function. In contrast, ordinal memory appears to rely on familiarity and recollection to a similar extent, raising important questions about the nature of the representation of these dimensions in event memory.

Materials and Methods

Participants

Forty-seven participants took part in the study. Twenty-four took part in the active condition (17 female, average age 21.3 yr, SD = 3.7) and 23 in the passive condition (15 female, average age 19.3 yr, SD = 1.4). All were recruited from the University of Toronto, and received either partial course credit or monetary compensation for their time. All participants were fluent in English, right-handed, with no history of neurological or psychiatric disorder, and unfamiliar with the city of Chicago. All participants provided their informed consent and were either paid $10 per hour for their participation or received partial course credit. This study was approved by the ethics committee at the University of Toronto.

Experimental design and procedure

Virtual navigation task

A virtual rendering of the city of Chicago was used in this study. The navigation software was written in MATLAB v7.5.0.342 and used the PsychToolbox v3.0.10 (Brainard 1997). The software used first-person images from Google Street View to allow navigation through a virtual Chicago. Load times for images were ∼70 msec and after each image was presented for 100 msec, it was crossfaded gradually with the next image over a period of 200 msec to provide the illusion of fluid motion.

Participants were automatically guided along a route in a first-person, street-level perspective through a virtual Chicago (Fig. 1A). Because the participants were not familiar with the city prior to the experiment, none of the effects can be attributed to, or confounded by, remote spatial memory. A single route through the city was designed so that there were equal numbers of left and right turns and that the travel time between intersections was roughly equal (see Fig. 1B).

Navigation was paused at eight intersections along the route, of which four were followed by a left turn and four, by a right turn (Fig. 1B). The range of pause durations was 1.5–12 sec with a 1.5-sec interval between each possible pause duration. The single route was repeated seven times by each participant to maximize learning and to simulate repetitive training in rodent studies (Pastalkova et al. 2008; MacDonald et al. 2011, 2012). Each of the “pause” intersections was associated with the same pause duration on all seven repetitions of the route, but these durations were randomized for each individual participant, so that effects of spatial salience could not be assumed to contribute to the performance on the temporal memory task.

Participants were told to pay close attention to the route and were not prompted to pay attention to the time spent waiting at each intersection. They were told to imagine that they were passengers in a car and that their only task was to remember as many details as possible about the route and the intersections where they waited. On occasion (one intersection per repetition, starting with the second repetition), participants had to indicate the direction in which the journey should proceed after the wait by using the arrow keys (left, right, straight). Six out of eight intersections were thus sampled over the course of the experiment for each participant. Which intersection was sampled during which repetition was randomly selected, but the same intersection was never sampled twice for a single participant. Regardless of whether a participant indicated the correct or incorrect direction, they were always guided along the correct route. The mean accuracy on this task was 0.87 (SD = 0.15) in the passive condition and 0.83 (SD = 0.19) in the active condition. The difference in accuracy was not significant (t(40) = 0.752, P = 0.456; note, the degrees of freedom do not correspond to the number of participants because navigation data files were missing for two participants in the passive task and for three participants in the active task).

Navigation on each route lasted ∼4.5 min. While the timing of each intersection was closely constrained, there was slight variability caused by differences in response time in those trials where participants responded with the direction following the stop. After reaching the end of the route after each repetition, participants were allowed to initiate the next trial at their own pace; some took a brief break prior to each repetition. None of the participants, however, took more than 35 min to complete the task.

Passive versus active waiting conditions

The participants in the passive condition were taken along the route as passengers and did not have to navigate actively so as to closely control the timing of the task. They were instructed that they would, on occasion, be prompted for the direction in which the car should continue, with the aim of ensuring that they were paying attention to the task and attempting to learn the route. They were instructed to simply pay attention to the route and that the car would stop at certain intersections as if waiting at a red traffic light, and that these intersections would be the same on every repetition. The waiting periods were indicated by red horizontal bars at the top and the bottom of the screen (Fig. 1A).

The experimental procedure in the active condition was virtually identical to that in the passive condition in most respects. The key difference was that in the active condition participants were instructed to hold down the space bar on the keyboard for the entire duration of each stop. If they waited for longer than 500 msec to press the key, or released it prematurely (before the interval elapsed), a warning appeared on the screen, instructing them to press the key (Fig. 1A). Similarly, if they waited for longer than 500 msec to release the space bar after the red bars had disappeared from the screen, a text box appeared on the screen, alerting them to release the space bar so the journey could continue. By adding this textbox, we hoped to motivate participants to be as accurate as possible so as to control the duration of each stop. If they pressed as soon as the bars appeared and released as soon as they disappeared, no warning was shown so that the two conditions were visually matched.

Behavioral testing and analysis

After completing all seven repetitions of the route, participants completed 5 memory tasks: (1) recognition memory, (2) duration discrimination, (3) ordinal discrimination, (4) duration and sequence ordering, and (5) time estimation tasks (see Fig. 1C–F for all tasks).

Recognition memory task

First, participants were asked to complete a recognition memory task in which they were shown images of all 8 intersections at which they had stopped and 8 lures in a randomized order, yielding 16 trials in the task (Fig. 1C). The lures were never-before-seen images of intersections from the same part of Chicago navigated during the task. For each image, they were asked to respond whether they could reexperience (R) waiting in that intersection, whether they knew (K) the intersection, or whether it was a new (N) intersection. Instructions were given such that “reexperience” and “know” ratings would correspond to estimates of recollection and familiarity (see Tulving 1985). Following each of these decisions, they were asked to also provide their confidence on a scale from 1–6 that the intersection was old (if they responded with R or K) or new.

Duration discrimination task

In the duration discrimination task, all eight intersections at which the participants had waited were paired together and shown pairwise in a randomized order, generating 28 trials in total (Fig. 1D). The side of the screen on which each intersection was displayed was randomized on each trial. Participants were asked to indicate using the left and right arrow keys at which of the two intersections they had waited longer (i.e., the left or right intersection). Following this decision, they were again asked to provide a confidence rating on a scale from 1–6, where 1 would indicate “completely guessing” and six would indicate “completely sure.”

Ordinal discrimination task

In the ordinal discrimination task, images of all intersections at which participants had waited were again paired and displayed side by side in a newly randomized order, generating 28 trials in total (Fig. 1D). Here, participants were asked to indicate which of the two images was closer to the end of the route. As in the previous task, they were asked to provide a confidence rating on a scale from 1–6.

The ordinal and duration discrimination tasks were presented in counterbalanced order but were always preceded by the recognition memory task, since the discrimination tasks did not contain lures and would reveal which intersections were old.

Duration and sequence ordering tasks

Following these tasks, participants were shown images of all eight intersections at which they had waited and were asked to complete two ordering tasks: the duration and the sequence ordering task (Fig. 1E). In the sequence ordering task, they were asked to drag the images into the correct sequential order. In the duration ordering task, they were asked to order the images according to their duration, from the intersection where they waited the shortest amount of time to the one where they waited the longest. The order of these tasks was counterbalanced across participants.

Time estimation task

In the final test, participants were shown, in a random order, images of all 8 intersections at which they had waited (Fig. 1F). They were asked to estimate the time they spent waiting at each by typing the number of seconds and to provide a confidence rating between 1 and 6 for each estimate.