New Trends in Episodic Memory Assessment: Immersive 360° Ecological Videos.

How best to measure memory in a reliable and valid way has been intensely debated in neuropsychological literature. Specifically, classical neuropsychological tests often fail to predict real-life performance or capture the multifaceted nature of memory function. To solve these issues, there has been a growing emphasis on the use of more ecological memory assessment. In this scenario, several virtual reality based tools have been developed to evaluate memory function. The aim of the current perspective is to discuss critically the possibilities offered for episodic memory assessment by one of the most innovative trends in the technology field, i.e., 360° videos. Immersivity, egocentric view and realism appear to be crucial features of 360° videos enabling them to enhance the ecological validity of classical assessment tools of memory abilities.

The State-Of-The Art in Memory Assessment

How best to measure memory in a reliable and valid way has been intensely debated in neuropsychological literature. Classical neuropsychological tests have proved to have an adequate clinical predictive value. However, the correlation between these tests and subjective memory complaints and/or everyday memory functioning appears to be quite inadequate (see for example, Chaytor and Schmitter-Edgecombe, 2003; Reid and Maclullich, 2006). The majority of neuropsychological tests (including memory assessment) have been developed following a “construct-driven approach”; starting from a solid theoretical paradigm, they evaluate abstract constructs without any reference to the real-life performance or behavior (Parsons, 2015; Parsons et al., 2017). In the last 20 years, however, there has been a growing emphasis on the use of a more ecological neuropsychological assessment. This is the so-called “function-led approach,” that implies the direct observation of behavior for the definition of a more ecologically valid neuropsychological testing (Parsons, 2015; Parsons et al., 2017). A representative example of this approach in the memory domain is the Rivermead Behavioural Memory Test (RBMT; (Wilson et al., 1986, 1989), which includes a series of daily-life tasks (i.e., recognize a picture, remember an appointment, encode, and store a route, etc.) to evaluate memory abilities in people following brain injury. Beyond the issue of ecological validity, neuropsychological tests appear unreliable in capturing the “complexity” of memory functioning (and its components). For instance, episodic memory has been traditionally defined as the memory of personally experienced events in their spatiotemporal context along with their perceptual details (Tulving and Murray, 1985; Tulving, 2002). The multiple components of episodic traces (what, when, where, spatial details) are merged through a process known as binding (Kessels et al., 2007), dependent upon the medial temporal lobes (Tulving and Markowitsch, 1998; Eichenbaum, 2000). So far, episodic memory assessment is usually conducted in clinical settings by asking older adults to remember a verbally presented story (Spinnler and Tognoni, 1987) or a list of words (Delis et al., 1987). Consequently, classical memory tests usually evaluate the memory components in isolation, thus failing to offer a global picture of the true essence of episodic retrieval (Tulving, 2002).

In this context, virtual reality (VR) appears to be a suitable technology for a function-led assessment of cognitive functioning (Rizzo et al., 2004; Bohil et al., 2011; Parsons, 2015; Parsons et al., 2017). Indeed, VR allows the development of a new class of ecologically valid neuropsychological tools able to simulate real-life situations. Moreover, this approach permits advantages in terms of time consumption, economic cost, precise control over the delivery of stimuli, and safety (Bohil et al., 2011). A growing body of VR-based tools has been developed to specifically evaluate episodic memory functions (Burgess et al., 2001; Spiers et al., 2001; Parsons and Rizzo, 2008; Plancher et al., 2008, 2012; Repetto et al., 2016; Parsons and Barnett, 2017; Picard et al., 2017). Within virtual environments, participants can be immersed in digital scenarios that represent everyday situations (i.e., a kitchen, a park, a museum); in these highly ecological situations, simple tasks can easily be implemented to evaluate the multifaceted nature of episodic memory.

For example, Burgess et al. (2001) exploited the potentiality of VR to investigate the neural basis of episodic recollection. In this study, participants were immersed in a highly ecological virtual town to follow a route where they received 16 objects from two people in two places. During a functional neuroimaging resonance, the memory of the objects, places and people was tested. A further example was offered by Piolino’s team, who developed a VR-based test for assessing the different components of episodic memory (Plancher et al., 2008, 2010). Participants were immersed in a virtual town (with either passive or active navigation), where they encountered various episodes (i.e., a man playing a guitar, a barking dog, etc.) to be remembered in their specific spatiotemporal context.

Furthermore, within virtual environments, it is possible to make the episodic memory task more complex and closer to the real life situation by adding intervening executive tasks (Parsons and Barnett, 2017); for example, Parsons and Barnett (2017) asked participants to shop for a list of items in a virtual grocery, ignoring items that were not present on this list as well as other irrelevant announcements.

However, the assessment of episodic memory in virtual environments has already been tested in clinical settings. Plancher et al. (2012) carried out a study involving healthy participants, amnesic mild cognitively impaired patients, and Alzheimer’s patients. One of the goals was to benchmark the virtual-reality episodic memory task against traditional neuropsychological tools for the assessment of episodic memory and subjective memory complaints. The virtual test correlated better with subjective memory complaints than classic memory tests did; importantly, the virtual task was also able to characterize the different cognitive profiles of the three populations, thanks to its suitability in capturing the multi-componential nature of the episodic memory.

To further enhance the ecological validity of neuropsychological testing and to deeply evaluate the multiple aspects of episodic memory, one possibility is offered by one of the most innovative trends in the technology field, i.e., 360° videos.

The Potential Use of 360° Videos for Memory Investigation

Videos with 360 degrees are spherical videos recorded by special cameras with omnidirectional lenses, able to collect images from all the way around the space. Usually 360° videos are recorded in natural life settings and displayed through a head-tracked Head Mounted Display (or on a low-cost cardboard viewer connected to a smartphone provided with a gyroscope). During the playback, a user can control the viewing direction by means of moving the head in a very realistic way (looking up one can see the sky/roof; looking down, one can see the floor/ground; and if one wants to see what happens on the left/right, it is sufficient to turn the head accordingly, as in real life situations). Therefore, these videos offer the viewers the ability to feel inside the represented environment and be the protagonist of action that unfolds before their eyes (Broeck et al., 2017). Lastly, it should be noted that no advanced programming skills are required to customize 360° videos; thus they can easily and economically be adopted in clinical settings without specific technical effort.

For these capabilities, 360° videos are often considered a special kind of VR, even though several features of 360° videos differ from graphic-based VR (Slater and Sanchez-Vives, 2016), as will be discussed later in this section.

Because of better availability and technological advancement of the necessary equipment (i.e., omnidirectional camera and HMD/cardboard viewers), the use of 360° videos is rapidly growing in several fields, such as immersive journalism, brand advertisement, and live sports. Many online platforms and social networks, such as YouTube, Facebook, and Vimeo, are now ready to support these videos, thus contributing to their sharing among the general public. Conversely, little effort has been put forward at a scientific level to understand the cognitive processes underlying this special 360° visual experience, or its capabilities for the investigation or assessment of cognitive abilities such as memory.

A recent study in this direction has been presented by Robertson et al. (2016) who used 360° videos to investigate how visual memory for panoramas are formed in the brain. In two behavioral experiments, authors asked participants to study novel 360° panoramic environments representing real urban scenes of a city, actively explored by means of a VR headset. Crucially, the study conditions varied in relation to the type of visual stimulus presented: in the naturalistic condition, the scenes were seamless and displayed continuously, exactly as they had been recorded from the real environment; in the morph condition, the scenes included pictures from different environments, gradually morphed within the main panorama. Results of the first study (study 2 in the research report) indicated that associative memory was higher for scenes studied in the naturalistic condition than in the morph condition. The second experiment (study 4 in the research report) specifically assessed the impact of perceptual priming (a screenshot of the studied panorama vs. a black screen) on memory for the correct position of a scene image. Again, the perceptual priming effect was superior in the naturalistic condition. All in all, these findings demonstrated that, even at the peripheral stage of visual processing (i.e., within the retina), individuals experience only discrete fleeting views of the environment; in the brain these images are merged together to create the perceptual experience of a coherent continuous panorama. Interestingly, the recognition of this seamless view has a positive impact on visual memory. These first results on the relationship between the naturalistic view of a panoramic scene and visual memory suggest that 360° videos may be the ideal tool for future investigations of memory processes, as they provide a visual experience very close to the natural visual exploration of the environment. More generally, we posit that 360° videos could effectively address the issue of ecological validity in memory assessment, thanks to several of its features.

The first one is the possibility to experience the environment from an egocentric perspective. According to Rubin and Umanath (2015) a visual perspective is crucial in distinguishing between the process of “remembering” and that of “knowing”: the egocentric view allows one to recollect memories of events and relive them during recall [in fact, this process is often referred to as mental time travel (Tulving, 1983; Suddendorf et al., 2009)]. Importantly, in this view, first-person (e.g., experiencing the scene from one’s own eyes) and third-person perspectives (e.g., experiencing the scene as an observer) are both egocentric in that they place the person remembering the scene relative to the spatial context in which the event occurs. Memory research has underlined that recent autobiographical memories are more likely retrieved from a first-person perspective, whereas older memories are more often associated with an observer perspective (Nigro and Neisser, 1983; Rice and Rubin, 2009). While recording the 360° videos, the experimenter could choose the egocentric view he/she is interested in: placing the camera over one’s head creates the illusion of being the protagonist of the scene (first-person perspective); conversely, placing the camera in a given position within the environment allows the scene to be experienced in the role of external observer (third-person perspective). This flexibility gives the chance to adapt the panoramic video to the actual needs of the researcher, targeting different memory processes.

The second element related to the ecologic validity is the realism. Realism is at least twofold: first, it can be intended as a perceptual similarity to the real world (Lovett et al., 2015; Parsons, 2015); second, it could be referred to as behavioral realism, that is the extent to which a user responds to the represented environment in the same way he/she would respond to the real one (Freeman et al., 2000). As far as perceptual realism is concerned, 360° videos, being recorded in real-life situations, are exact representations of the real world; therefore, they would not suffer from the pitfalls encountered when creating computer-generated virtual environments. With respect to behavioral realism, ergonomic research has demonstrated that a 360° panorama can outperform VR. In particular, Higuera-Trujillo et al. (2017) directly compared the psychological responses evoked by three simulated environmental set-ups (photograph, 360° panorama, and VR) with responses elicited in a real environment. Specifically, authors measured the affective and emotional states triggered by the different environmental set-ups and discovered that 360° panoramas produced results closest to reality. Psychological studies confirmed that 360° videos are able to induce emotions, assessed by both self-reports and psychophysiological parameters. Chirico et al. (2017) administered 360° videos and traditional 2D videos displaying awe-inspiring and neutral content. During the video exposure, they recorded physiological parameters and after the experience, they asked participants to rate the level of awe. Results indicated that panoramic videos enhanced the intensity of awe experienced, compared to 2D videos, and also were able to activate more the parasympathetic system. Similarly, panoramic videos are reported to stimulate the psychophysiological correlates of anger arousal (Macedonio et al., 2007).

The third feature to consider is the immersivity, defined as the possibility to experience the environment from an immersive perspective, while isolated from the real world (Coelho et al., 2006; Diemer et al., 2015). Immersivity is supported by the use of a head-mounted display to view the videos. Immersivity and realism are two of the sub-components of presence. Although there are several slightly different conceptualizations of this construct (Lombard and Ditton, 1997; Riva et al., 2004, 2011; Coelho et al., 2006), presence can be defined as “the sense of being there” in either a virtual or physical environment (Riva and Mantovani, 2012). The capability to induce the sense of presence is highly relevant in the assessment of memory, as it has been demonstrated that the former directly affects the latter (Sutcliffe et al., 2005; Slobounov et al., 2015). Makowski et al. (2017) asked people who had just watched a movie in a cinema to rate the sense of presence and emotional experience during the projection. Crucially, they correlated these self-reports with people’s ability to recall specific factual memories (details of the movie) and temporal memories (order of the scenes) and found that higher levels of presence were associated with better factual memory (but not temporal memory). More importantly, the impact of emotion on memory recall was mediated by presence, confirming its crucial role in memory encoding.

Although 360° videos are as powerful as computer-generated VR in relation to their egocentric view and immersivity, and panoramic videos could even outperform VR in relation to realism, one feature recognized as a strong point in VR is missing in 360° videos: active navigation (Repetto et al., 2016). Indeed, videos do not allow one to choose the direction of navigation, which is defined during recording and cannot be actively changed during the playback. In a computer-generated virtual environment, the user can move in any direction, approach objects and people, and change the path of navigation; however, when watching a 360° video, the user is automatically conducted through the environment and forced to follow the direction of navigation selected during the video recording. Nonetheless, the user can actively explore the environment by moving his/her head and directing attention toward the portion of the 360° visual field he/she is interested in. The active exploration is enhanced if the video is recorded with moving viewports (i.e., the camera moves within the environment while recording) (Broeck et al., 2017). Lastly, a further disadvantage of the 360° video compared to computer-generated VR is the lack of interaction with the elements of the environment.

Future Directions

As discussed above, immersivity, egocentric view and realism all concur to enhance the ecological validity of a video, and all are involved in episodic memory. As such, we propose that panoramic videos could be used as an assessment tool for episodic memory abilities (see Figure ). In this approach, 360° videos would be the means to present the stimulus to be remembered; similar to more classical assessment methodologies, the outcome measures would be recall of details of the video (the what-when-where components of the episodic recall, as suggested by Tulving, 2002) and recognition of the encoded information. Additionally, the 360° videos also could offer the possibility to collect eye-tracker measures, potentially useful to identify the pattern of visual exploration typical in good and bad memory performers.

An example of 360° video for episodic memory assessment. The participant, wearing a head-mounted display, is immersed within the 360° scenario and (s)he is free to move his/her head to explore the environment. The participant is prompted to watch the video and remember as much details as possible (about WHAT happens, WHEN it happens, and WHERE it happens). Written informed consent was obtained from the depicted individual for the publication of this image.

These special videos have already been used to evaluate cognitive abilities in a clinical setting, in particular for the assessment of executive function in Parkinson’s disease (Serino et al., 2017). In this study authors administered the 360° version of the Picture Interpretation Test (Rosci et al., 2005) to Parkinson’s disease patients and healthy controls, discovering that this tool correlated with neuropsychological measures of executive functions and was able to reliably discriminate between the two groups (patients vs. healthy controls).

In the same way, panoramic videos could be employed for the assessment of episodic memory in other populations, such as elderly and psychiatric patients. The memory assessment of the elderly is particularly important for detection of early memory decline and the identification of degenerative disorders: indeed, an ecological memory evaluation (by administering videos representing real-life scenarios) could help identify those subtle amnestic deficits that usually escape formal memory assessment, yet account for the subjective memory complaints (Balash et al., 2013) sometimes reported by the elderly. Furthermore, it has been proposed that episodic and autobiographical memory impairments documented in some psychiatric diseases could be due to a specific deficit in scene construction, that is, “the process of mentally generating and maintaining a complex and coherent scene or event. This is achieved by the retrieval and integration of relevant informational components, stored in their modality-specific cortical areas, the product of which has a coherent spatial context, and can then later be manipulated and visualized” (Hassabis and Maguire, 2007, p. 299). Some authors have investigated this construct by asking patients to imagine personal life situations (Raffard et al., 2010), but the possibility to immerse patients in real-life scenes, targeting specific emotions, could be much more helpful in providing objective and controllable clues to memory performance.

Author Contributions

SS and CR conceived and wrote the paper.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.