Movies and narratives as naturalistic stimuli in neuroimaging.

Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.

While the vast majority of neuroimaging-based research into human cognition still employs traditional experimental paradigms involving non-naturalistic stimuli, the use of movies and narratives as naturalistic stimuli is increasingly extending the reach of human neuroimaging to phenomena that are otherwise difficult to investigate (Astolfi et al., 2009; Gaebler et al., 2014; Hanke et al., 2014; Hu et al., 2017; Marussich et al., 2017; Mueller et al., 2012; Ogawa et al., 2013; Spiers and Maguire, 2007; Wilf et al., 2017). In one good example, since movies and narratives elicit emotions more robustly than pictures do (Westermann et al., 1996), applying them as stimuli in neuroimaging studies has greatly advanced understanding of the neural basis of emotions (Matsukawa et al., 2017; Saarimäki et al., 2016).

In this review, we focus on describing research in areas wherein naturalistic stimuli have created particular opportunities for research beyond those offered by more traditional paradigms. These include human memory, attention, language, emotions, and social cognition. This is thanks to the naturalistic stimuli allowing, for example, depiction of natural social interactions evolving over multiple time scales rather than still pictures, presentation of dynamic dialogue and speech rather than individual words or disconnected sentences, and presentation of events and objects within their context rather than as lists of to-be-memorized items. The complexity and richness of naturalistic stimuli at the same time pose methodological challenges, however, neuroimaging method development is rapidly advancing and opening new exciting avenues for research in this area.

There are other excellent reviews, some of them very recent, on the present topic. These previous reviews focused on methodology that makes it possible to study brain activity in virtual environments (Spiers and Maguire, 2007; Maguire, 2012), on the use of naturalistic stimuli in the contexts of visual and memory system, social cognition, and clinical studies (Sonkusare et al., 2019), on the use of naturalistic stimuli in developmental studies (Vanderwal et al., 2019), and the neural mechanisms underlying virality of narratives (Jaaskelainen et al., 2020). Thus, each of the previous reviews has a vantage point different and complementary from the present review where we examine how movies and narratives have been recently utilized to advance scientific understanding in the areas of human memory, attention, language, emotions, and social cognition. Here we predominantly focus on describing findings obtained with functional magnetic resonance imaging (fMRI). Studies from other modalities such as EEG and iEEG have been referenced to highlight the relevant findings.

Use of movies and narratives in neuroimaging studies of memory

Encoding of naturalistic stimuli into long-term memory

Neuroimaging studies with naturalistic stimuli have provided new information about the neural basis of memory encoding. In one early study, emotionally arousing aversive and neutral film clips were shown to subjects during positron emission tomography (PET). Activity in the right amygdala and orbitofrontal cortical areas at the time the subjects viewed the emotionally arousing movie clips was correlated with clips’ enhanced memorization, demonstrated three weeks later (Cahill et al., 1996). This suggests that amygdala activity during emotionally arousing life events causes differential memorization of such events relative to non-emotional ones. These findings provide one good example on how naturalistic stimuli, via eliciting robust emotions, have allowed study of phenomena that are difficult to investigate with traditional paradigms wherein for example the subjects are presented with lists of to-be-memorized items.

Studies since then have investigated which brain structures are involved in memory encoding of non-emotional movie events. The data analysis in many of these studies has utilized inter-subject correlation (ISC) of brain activity wherein subjects’ brains are aligned and correlations are calculated between all pairs of subjects (or subgroups of subjects) for fMRI activity time series for each voxel. The ISC is very well suited for analysis of fMRI data since it is a model-free approach, where one does not need to build predictors of specific events in the rich and complex movie stimuli as in case of general-linear model (GLM) based analyses (Hasson et al., 2004). Further, the ISC analysis was found to give similar results as a GLM analysis in the context of an fMRI study with traditional stimuli (Pajula et al., 2012).

Some studies have tested which aspects of brain activity at the time of encoding predict later behaviorally measured recollection of movie content. In one fMRI study, significant ISC of hemodynamic activity in the parahippocampal gyrus, superior temporal gyrus (STG), temporoparietal junction (TPJ), and anterior temporal pole (aTP) areas during movies’ viewing predicted memorization of their content three weeks later (Hasson et al., 2008a). In another study, ISC of electroencephalographic (EEG) activity, estimated to originate in sensory and extra-sensory areas, predicted subsequent memorization of audiovisual narratives (Cohen and Parra, 2016). Together, these studies suggest that STG, TPJ, aTP, and sensory cortical areas contribute to the memory-encoding process alongside the hippocampus as one experiences everyday life events.

The influence of prior knowledge on memory encoding is another question addressed in various neuroimaging studies utilizing naturalistic stimuli. Functional connectivity between the ventromedial prefrontal cortex (VMPFC) and hippocampus was enhanced during watching of the last 15 min of a movie only when encoding to memory was rendered more demanding by the first 80 min having been presented in scrambled rather than intact form the day before (van Kesteren et al., 2010). Interestingly, the functional connectivity enhancement persisted over subsequent rest. In contrast, when the movie’s beginning was viewed intact, only enhanced ISC of the VMPFC was observed. These findings were considered to support the notion that interplay between hippocampal and VMPFC is needed when there are weak schemata to guide encoding. In another study, the VMPFC and other default-mode network (DMN) structures interacted more strongly with the hippocampus during natural viewing when congruent contextual cues about the movie preceded it by a day rather than minutes (Chen et al., 2016). Taken together, these results suggest that the VMPFC and hippocampus interact as one navigates day-to-day life when long-term memory content is utilized in processing of dynamically unfolding events that are hard to predict with existing schemata.

Finally, naturalistic stimuli have helped uncover how the brain re-encodes memory representations in light of new information. When subjects recognized that a given video segment belonged with certain other video segments, the patterns of activity exhibited during these segments’ viewing started to resemble each other in the hippocampus and VMPFC (Milivojevic et al., 2015). In line with these findings, specific distributed activity patterns in the medial prefrontal cortex (MFPC) emerged during watching of specific segments of Memento (dir. Christopher Nolan, 2000) wherein previously seen events are reinterpreted in light of new information (Kauttonen et al., 2018). Thus, it seems that memory schemata are supported by distributed patterns of activity in these structures. These findings are also highly relevant, in that reinterpretation of prior events in light of new information is vital in day-to-day life. In the context of movies, the results shed light on what might happen in the viewer’s brain after it is revealed that, for example, the butler committed the murder, when all the pieces of the puzzle suddenly click together.

The neural basis of recalling naturalistic events from long-term memory

Neuroimaging studies have shown how the human brain recalls naturalistic events from long-term memory, which are contextually meaningful unlike to-be-memorized items in more traditional paradigms. In one study, researchers compared both recall accuracy and brain activity hours, weeks, and months after the initial presentation of a movie. Behavioral findings indicated that memories became more “gist-like” after delays of weeks and months as compared to delays of hours (Furman et al., 2012). During retrieval, brain activity extended across a broad set of structures associated with autobiographical memory retrieval, including hippocampal and DMN structures. These activations grew weaker after the months-long delay, further supporting the idea that memories become more gist-like after lengthy intervals. Recall performance was significantly correlated with activity in the hippocampus, lateral temporal, occipito-temporal, and prefrontal areas – but only after the delay of months. Similar single-neuron activity has been shown during both encoding and recall of movie clips in the hippocampus and entorhinal cortex of epileptic patients, thus suggesting that neuronal firing patterns that repeat across encoding and recall in these structures supports episodic memory (Gelbard-Sagiv et al., 2008).

Other studies have echoed the finding that recall of movie events activates the autobiographical memory network (Chen et al., 2017; St-Laurent et al., 2016). Autobiographical memory is something that would be difficult to study with traditional stimulus/task paradigms. Narratives presenting personally relevant autobiographical memories have been reported to involve the DMN structures and hippocampus more strongly than fictitious narratives (Loughead et al., 2010). Furthermore, across subjects who watched an episode of a TV program and freely re-called it immediately afterward, the parahippocampal cortex and other DMN structures showed greater similarity during recall than during the initial viewing (Chen et al., 2017). A brain network-level analysis, based on fMRI data, further suggested that interplay between anterior hippocampus and precuneous is especially relevant for free retrieval of memories consolidated during natural viewing (Ren et al., 2018).

Intriguingly, the type of recall task modulates which areas of the autobiographical memory network are more involved than others: recalling the events’ temporal order activates the hippocampus (Kwok et al., 2012; Lehn et al., 2009) and precuneus (Kwok et al., 2012), recall of spatial information activates frontal-parietal areas (Kwok et al., 2012), that of perceptually rich details activates the right hippocampus (St-Laurent et al., 2016), and movie-scene recognition involves the MPFC (Kwok and Macaluso, 2015b; Kwok et al., 2012). True/false ratings for a movie seen earlier activated the precuneus, lateral temporal areas, VMPFC, and hippocampus (Mendelsohn et al., 2010; Risius et al., 2013). Retrieval of memories associated with emotionally traumatic film, in turn, resulted in activation of precuneous, retrosplenial cortex, and cingulate cortex (Gvozdanovic et al., 2017). Together, findings lend further support to the conclusion drawn above that memory schemata are supported by distributed patterns of activity in DMN structures (especially in MPFC but also in precuneous) and hippocampus.

The cortical hierarchy of temporal receptive windows in short-term memory

Besides shedding light on the neural underpinnings of long-term memory encoding and recall, naturalistic stimuli have helped uncover entirely new short-term memory mechanisms in the brain. The discovery of a hierarchy of temporal receptive windows (TRW) in the human brain constitutes the first of these new advances. The TRW has been defined as “the length of time before a response during which sensory information may affect that response”(Hasson et al., 2008b). For example, grasping the meaning of the sentence “Jill kicked the ball” requires its subject (“Jill”) to be kept in memory. One thereby connects the subject with the verb and the object, hence reaching an understanding that it was Jill who kicked the ball. An even longer TRW is needed for understanding that Jill is the same person referred to earlier in the narrative.

The shortest TRWs have been observed in the brain’s sensory areas and the longest in DMN structures (Chen et al., 2016; Hasson et al., 2008b; Lerner et al., 2014, 2011). Whereas the former are known to process lower-level aspects of narratives, such as syllables and individual words, the DMN structures seem to keep track of the evolving narrative (see below). Thus, the hierarchy of TRWs seems to mesh with the increasing need to accumulate information over larger timescales as one moves from processing transient elements (consonant–vowel groups, syllables, etc.) toward more slowly evolving aspects of the narrative (such as its plot).

The manner of conducting the aforementioned studies is rather insightful. The one with scrambled content showed that, while a “backwards-played” movie resulted in significant ISC only in the brain’s primary visual areas, as the movie was reordered with increasing window lengths, significant ISC appeared for increasingly high-order cortical regions (Hasson et al., 2008b). The ISC indicates that the brain area is processing the naturalistic stimulus similarly across subjects. As this occurs with increasing window length, it can be inferred that the ISC is caused by the availability of information over longer time constants, thus signaling that the respective brain areas are processing information accumulating over longer window lengths. A subsequent study, in which reordering of audio narratives utilized several segment lengths, showed an analogous hierarchy of TRWs (Lerner et al., 2011). Interestingly, TRW hierarchies spontaneously emerge in deep recurrent neural network models also (Shi et al., 2018). The existence of TRWs is further supported by evidence that the precuneus, along with the right posterior superior temporal sulcus (pSTS), supports cumulative plot formation during listening to a narrative (Tylén et al., 2015). These findings are complemented by ISC being stronger at lower frequencies in hierarchically higher cortical areas, which could reflect progressively longer temporal receptive windows (Honey et al., 2012; Kauppi et al., 2010). This is because the areas processing the slowly developing aspects of the narrative (e.g., the plot) are not synchronized by the rapid sensory stimulation elements, and vice versa.

Taken together, these findings indicate that TRWs grow in duration with ascent in the cortical hierarchy, which seems to occur in conjunction with increasing complexity of the receptive fields. Further, increasing TRW length might reflect a hierarchy with rising neuronal adaptation time constants as one progresses from sensory to association areas of the brain (Himberger et al., 2018; Jaaskelainen et al., 2011; Zhou et al., 2018). The hierarchy of TRWs might underlie the human brain’s ability to make sense of information that accumulates over multiple timescales. Interestingly, the hierarchical TRWs are rescaled to the speech rate: when the speaker is talking slowly, the TRWs are prolonged, relative to those with higher speech rates (Lerner et al., 2014). In a finding that parallels this, TRWs differ between speech and music on the basis of the timescales for conveying information (Farbood et al., 2015). Accordingly, rather than there being a hierarchy of TRWs with fixed durations, the TRWs reflect accumulation of temporally distributed information with flexibility as to how quickly or slowly the window of integration is closed. This points to the interesting yet unexplored question of whether and how the TRWs might be related to event segmentation.

The neural basis of event segmentation

Event segmentation is another short-term memory mechanism that can be studied very effectively with naturalistic stimuli. The mechanism involves processing points of discontinuity in a movie/narrative, such as changes in time, location, or protagonists (Zwaan et al., 1995). Such points of discontinuity constitute perceptual event boundaries that result in memorization of an ongoing naturalistic stimulus as distinct events. For example, a discussion with a colleague in the break room is memorized as an event separate from a subsequent staff meeting or driving to the grocery store. Movies and narratives that are naturally composed of such events have supplied us with powerful tools to uncover what takes place in a subject’s brain when one such event ends and another begins.

Neuroimaging studies have shown that there are peaks in the brain’s hemodynamic activity in the hippocampus at event boundaries (Ben-Yakov and Dudai, 2011; Ben-Yakov and Henson, 2018; Ezzyat and Davachi, 2011; Magliano and Zacks, 2011; Swallow et al., 2011; Zacks et al., 2010). This hippocampal activity predicts recall accuracy (Ben-Yakov et al., 2013), with hippocampal activity being lower during presentation of familiar event sequences (Ben-Yakov et al., 2014). Further, activation patterns in the right hippocampus predicted subjects’ judgments of the temporal distance between two events in an audio narrative (Lositsky et al., 2016). The hippocampus even seems to hold character- and location-specific representations during movie-watching (Milivojevic et al., 2016). This finding lines up well with the observations of memory-schemata being supported by distributed activity patterns in hippocampus (and DMN structures, see above). Together, these findings suggest that the hippocampus supports memory encoding of events that have come to an end, such that the episodic information accumulated during such an event does not interfere with processing of the information encountered in the course of the next event.

In addition to the hippocampus, processing of event boundaries activates a host of other structures. Event boundaries have been observed to activate the VMPFC, striatum, precuneus, cingulate cortex, anterior-lateral temporal cortex, visual cortices, lateral occipital cortex, posterior parietal cortices, and TPJ (Ben-Yakov and Dudai, 2011; Betti et al., 2013; Ezzyat and Davachi, 2011; Speer et al., 2007; Whitney et al., 2009; Zacks et al., 2001, 2011, 2010). Furthermore, the inferior parietal lobule (IPL) has shown differential responses to movie frames obtained from an event boundary vs. non-boundary section. The IPL activity differed also between passive viewing of a movie and viewing during which the participant marked event boundaries (Kwok and Macaluso, 2015a). The cerebellum too seems to support integration of events into event sequences (Lahnakoski et al., 2017).

There is a division of labor with regard to event-boundary types. The middle frontal gyrus has shown activity upon changes in characters’ interactions, the hippocampus responded to spatial changes, and parts of the superior frontal gyrus were activated during changes in protagonists’ goals (Zacks et al., 2010). Elevated difficulty in behaviorally predicting the action five seconds later in movie clips that depict day-to-day life co-occurred with striatal and midbrain activity (Zacks et al., 2011). This suggests that these regions contribute to triggering an updating signal in the brain at event boundaries. Machine-learning algorithms have been able to classify listening to narratives with restaurant vs. airport scripts from script-specific patterns of brain activity in DMN structures (Baldassano et al., 2018), and a subsequent study revealed sensory areas segmenting short events in movies and DMN structures segmenting long ones, with the latter also invoking hippocampal coding at the boundaries of long events (Baldassano et al., 2017).

In summary, event segmentation may be one of the most fundamental of human memory functions, and naturalistic stimuli are highly suited to studying its neural basis. Advances in data-analysis approaches, such as machine-learning algorithms, could, in the near future, offer keys to further insight surrounding this function. It is possible also that event segmentation and the stimulus-rate-related flexibility in the durations of some longer TRWs, such as those that match paragraph lengths, are supported by a common underlying neural mechanism. The observation that sensory areas process short events and exhibit short TRWs while DMN structures process long events and exhibit long TRWs (Baldassano et al., 2017) lends further support to such a hypothesis, which could be tested in future research.

Use of movies in neuroimaging studies of attention

Selective attention mechanisms during naturalistic stimulation

Attention and cognitive control have been extensively studied in well-controlled laboratory conditions (Corbetta and Shulman, 2002; Jaaskelainen et al., 2011). For example, in dichotic listening tasks, both ears are presented with stimulus trains and subjects are instructed to attend to sounds of slightly higher pitch presented to one ear while ignoring the sounds presented on the other side. Comparing responses to sounds between the attended and non-attended stream then reveals selective attention effects, such as augmented amplitude for responses to tones in the attended stream. With recent sophisticated experiment designs, selective attention has been linked to specific mechanisms that filter attended-sound features at the level of the sensory cortices (Ahveninen et al., 2011). Natural viewing studies have helped elucidate the neural basis of selective attention in conditions that resemble real life.

In one study, changes in visual salience and unexpected turning points in a movie activated posterior and inferior cerebellar regions (Nguyen et al., 2017). This is something not previously documented in neuroimaging studies using traditional task designs, and it attests to the cerebellum’s role in bottom-up attention, in addition to its part in motor control. Additionally, visual attention to certain object categories during movie-viewing was observed to dynamically “warp”the cortical semantic representations of object categories in both visual- and association-cortex areas (Cukur et al., 2013). This expansion in the range of attended object categories occurred independently of whether the movie featured objects belonging to those categories. One could posit tentatively that these results might be explained by dynamic changes in neuronal receptive fields (Jaaskelainen et al., 2011) that help filter objects in attended categories. Further research could test this hypothesis.

Naturalistic stimuli such as commercial movies are very powerful in building suspense. This makes it possible to study engagement of attention in more genuine and voluntary conditions than found in more traditional experiment paradigms (Bezdek et al., 2017; Lehne et al., 2015). Strong immersion in naturalistic stimuli has been observed to be accompanied by activity in the left inferior frontal gyrus (IFG), in the right lateral temporal cortex, and bilaterally in the pSTS (Metz-Lutz et al., 2010). In another study, strong immersion in a narrative resulted in dACC activity that was stronger during fear-eliciting than neutral narratives (Hsu et al., 2014). Given that enhanced suspense is often mixed with experienced emotions, it is important to further explore the interplay of emotions and attention during naturalistic stimulation in future studies. When subjects engaged in a demanding secondary task while watching a movie, increased ISC in the dorsal-posterior cingulate gyrus predicted weaker memorization of its content (Oren et al., 2016). Similarly, attention lapses during narratives and active reading demands during listening/reading modulated functional connectivity in the DMN and attentional networks of the brain (Smallwood et al., 2013; Wang and Holland, 2014).

In addition to fMRI studies, there are EEG studies that have investigated the role of inter-subject synchrony of brain activity in attentional functions under naturalistic viewing conditions. Whereas fMRI measures brain hemodynamic activity with great spatial accuracy, EEG measures voltage differences on the scalp generated by neuronal activity with millisecond temporal accuracy but is limited in spatial accuracy. ISC of EEG activity is typically calculated via signal decomposition methods that find a few maximally correlated components based on multiple EEG records (Dmochowski et al., 2012). In one such study, enhanced ISC of EEG stimulus-evoked activity during meaningful auditory and audiovisual narratives was noted to covary with subjects’ attentional state (Ki et al., 2016). In other research, higher ISC of EEG activity predicted subjects’ greater voluntary sustained attention to a movie (Cohen et al., 2017). The ISC of EEG activity can even predict movies’ box-office performance more than 20 times better than established self-reporting-based methods used in the movie industry (Christoforou et al., 2017). Interestingly, inspection of how EEG activity is modulated in theta, beta and alpha frequency bands within different components during periods of enhanced ISC suggested modulation of ISC due to both attentional and emotional factors (Dmochowski et al., 2012). In sum, these results suggest that stronger ISC of EEG activity reflects attentional and emotional immersion in a captivating naturalistic stimulus so that it could be a useful metric in various application areas (education the sciences, marketing, the cinematic arts, etc.)

The neural basis of perspective-taking

Research on the neural basis of perspective-taking constitutes a case wherein naturalistic stimuli are particularly useful. One example is a study in which viewing a movie clip from the viewpoint of a detective vs. an interior decorator modulated activity in the dorsal attention network (DAN) and posterior hippocampus, suggesting that these areas control perspective-specific attentional information-gathering in naturalistic settings (Lahnakoski et al., 2014). Importantly, these effects could not be accounted for by the variability in eye-movement patterns beyond the early visual areas. This is consistent with later studies showing that stable eye-movement-invariant representations of the visual world dominate as one steps upward in the cortical hierarchy, away from the early visual areas (Lu et al., 2016; Nishimoto et al., 2017).

In other studies, individuals primed to hear a story from either of two distinct social perspectives (a protagonist with unfounded vs. justified jealousy) showed ISC differences in their DAN, hippocampus, DMN, mirror neuron system, and language-processing networks (Yeshurun et al., 2017b). Similarly, subjects equipped with a priori information about a movie’s major characters being biological vs. adopted sisters exhibited robust differences in their patterns of ISC when then watching the movie (Bacha-Trams et al., 2017). In addition, other work found taking a spatial perspective (here, the text’s protagonist being “there”vs. “here”) to be linked to the TPJ and posterior cingulate cortex (pCC) (Mano et al., 2009). Broca’s area, the left anterior superior frontal gyrus, and the inferior parietal cortex too are involved in perspective-taking (Naci et al., 2014; Smirnov et al., 2014). In contrast to designs manipulating the participants’ social or even spatial perspective, presenting the same narrative from first- vs. third-person perspective failed to produce analogous differences in frontoparietal brain activity (Hartung et al., 2017); for related findings, see Nijhof and Willems (2015). Finally, results from a recent study suggest that adopting perspectives that require empathy vs. moral cognition differentially increase ISC in areas supporting each of the respective functions (Bacha-Trams et al., 2020).

These findings suggest that perspective-taking entails the brain flexibly recruiting regions in line with the cognitive-perceptual requirements of the perspective being adopted. In other words, the task demands during viewing, whether investigator-initiated via instructions to the subject, the movie director guiding the viewer via cinematic art techniques, or via influence of a priori information provided before viewing, determine the brain regions flexibly recruited during viewing. The results also demonstrate that, while it remains important to control for eye movements by measuring them during natural viewing studies, differential patterns in them fail to predict differential brain responses beyond the hierarchically early visual cortex areas.

Use of movies and narratives in language studies

Language-processing is among the first fields of neuroimaging research in which naturalistic stimuli were used (Crinion et al., 2003; Lechevalier et al., 1989; Yarkoni et al., 2008a; Zacks et al., 2001). In addition to neuroimaging methods such as fMRI, processing of natural speech has been examined via EEG (Becker et al., 2013; Brederoo et al., 2015; Broderick et al., 2018; Manfredi et al., 2017). For several research questions in this field, fundamental advances have been achieved beyond insights from experiments utilizing non-naturalistic stimuli, as described below.

Narrative-level semantics and the brain

Studies with naturalistic stimuli have helped elucidate neuronal mechanisms of narrative-level comprehension, which is something that would be difficult to study with presentation of single words vs. non-words or simple sentences as is typically the case in studies with non-naturalistic stimulus/task paradigms. Studies have shown that comprehension of narrative-level information that accumulates on larger timescales depends on DMN structures (AbdulSabur et al., 2014; Babajani-Feremi, 2017; Gordon et al., 2017; Nguyen et al., 2019; Regev et al., 2013; Simony et al., 2016; Tikka et al., 2018; Wilson et al., 2008; Xu et al., 2005; Yarkoni et al., 2008b; Yeshurun et al., 2017a). The DMN structures appear to be involved across the encoding, recall, and reconstruction of narrative events presented in a movie (Zadbood et al., 2017, but see Gilead et al., 2014), with these structures’ involvement in narrative comprehension being transferable across languages also (Dehghani et al., 2017). Together, these findings suggest that memory representations of higher-order events in the narratives, such as evolving plots and contexts, are supported by distributed patterns of brain activity in DMN structures, which is in line with findings from memory studies using naturalistic stimuli as described above.

Interestingly, reading a passage labeled as fiction activated the precuneus more strongly than did reading the same passage as factual (Altmann et al., 2012). This finding may have important implications since most neuroimaging studies with naturalistic stimuli have utilized fictitious movies/narratives. Finally, interpretation of structurally coherent dance phrases in dancers activated superior parietal and DMPFC areas, tentatively suggesting that there are similarities in neural mechanisms that support the processing of structure of dance and language (Bachrach et al., 2016). Both of these findings are very good examples of phenomena that can be best studied with naturalistic stimuli, since labeling artificial stimuli as fact vs. fiction would be cumbersome and dynamic stimuli are needed when presenting dance phrases.

Alongside DMN structures, other regions of the brain participate in processing of natural speech. Among them are the auditory and language areas of the lateral temporal and the inferior-lateral frontal cortex (Bautista and Wilson, 2016; de Heer et al., 2017; Spitsyna et al., 2006). During narrative-reading, the dorsal and ventral visual streams too are activated (Lin et al., 2019; Zhou et al., 2016), perhaps due to imagery elicited during reading. Some evidence suggests that the hierarchy of TRWs along the dorsal pathway supports processing of discourse-level cues. There is increased adaptation of one’s responses to a protagonist when higher likelihood of mentioning the protagonist has been cued earlier in the paragraph (Kandylaki et al., 2016). Lexical stress that depends on shorter temporal receptive windows was, in turn, observed to be processed within the classical language network during natural listening (Kandylaki et al., 2017).

In summary, these results suggest that DMN structures support the processing of narratives across media types: movies, audiobooks, and even dance. Discoveries of this nature would have been extremely difficult to come by without the aid of naturalistic stimuli, since narrative-level information is altogether absent from traditional experiment designs. Inconsistencies remain between individual studies with regard to which DMN regions in particular are involved, though the precuneus is implicated fairly consistently. Further studies with large participant samples to compare the narratives presented across media could address this question. Finally, it is easy to identify convergence between knowledge of the DMN’s involvement in narrative-level processing and results from investigation of the hierarchy of TRWs and event segmentation wherein specifically DMN structures were found to process information with long TRWs and segment long events. Future studies should address the interdependence of these highly interesting phenomena, each of which was uncovered in a separate branch of investigation informed by using naturalistic stimuli.

Bilateral brain activity during listening to natural speech

Semantic processing of natural speech in the brain is predominantly bilateral (de Heer et al., 2017; Huth et al., 2016; Saalasti et al., 2019). This stands in contrast against observations of left-hemisphere dominance of the semantic system that have been documented in studies with non-naturalistic paradigms (Binder et al., 2009). Whereas spectral features of natural narrative speech activate the early auditory cortices (de Heer et al., 2017), syntax-processing activates the lateral temporal cortex (Brennan et al., 2016), and articulatory features activate inferior-lateral frontal areas also (de Heer et al., 2017), studies have shown that semantic content activates extensive sets of brain regions higher in the hierarchy bilaterally (de Heer et al., 2017; Huth et al., 2016). Semantic categories as contained in movie clips, such as a movie’s cars or humans, also activate the brain bilaterally (Nishida and Nishimoto, 2018). Bilateral activity is seen, in addition, with sign-language stimuli having narrative-related elements (e.g., prosody and discourse cues) (Newman et al., 2010). It is possible that the rich semantics of narrated stories or the demand for processing over longer TRWs gives rise to the bilateral activations. These questions point to avenues for further study.

The speech motor system and speech perception

Speech motor areas’ involvement in speech perception was posited in early theoretical work (Liberman et al., 1967) and more recent neuroimaging studies (Hickok and Poeppel, 2007). Corroborating these accounts, speech motor areas were found to be activated during both speech production and listening to natural speech (AbdulSabur et al., 2014; Awad et al., 2007). Furthermore, natural speech seems to be handled in general auditory and speech-specific processing streams both, the latter overlapping speech motor regions’ somewhat (Boldt et al., 2013). Also, robust speaker–listener coupling was observed in speech motor areas (Silbert et al., 2014). Violations of action predictions exhibited posterior scalp topography similar to that with N400 responses to violations of semantic predictions in an EEG study (Sitnikova et al., (2008). All in all, neuroimaging studies with naturalistic stimuli have provided further evidence for articulatory-motor areas’ involvement in speech perception under more ecologically valid conditions than were previously possible.

Development of narrative comprehension in children

Developmental studies have revealed that portions of the lateral temporal lobe are more crucial for narrative comprehension in children than in adults (Szaflarski et al., 2012), even though the DMN does seem involved also (Emerson et al., 2015). Moreover, it has been demonstrated that functional connectivity between the inferior-lateral frontal and posterior-lateral temporal lobe areas increases with development (Karunanayaka et al., 2007; Schmithorst et al., 2007). Interestingly, the strengthening of these connections seems to be a mechanism that results in left-hemisphere dominance for language functions as the brain matures (Karunanayaka et al., 2007).

Also, factors in language development have been studied. Exposure to reading at home is connected with stronger activity in the temporal–parietal–occipital cortex areas during listening to a narrative (Hutton et al., 2015). Cerebellar activity and connectivity with the various cortex areas was, in turn, correlated with greater story-time engagement when a mother was reading to her child (Hutton et al., 2017). It has been shown, in addition, that increased activity in small children’s executive and auditory–visual networks during narratives’ comprehension predicts their later reading ability (Horowitz-Kraus et al., 2013). In parallel with this, illustrated stories seem to support language development in preschool-age children by reducing reliance on the brain’s language networks (Hutton et al., 2018). Finally, naturalistic studies have demonstrated that a second language acquired early in life is processed similarly to the first language, even if acquired passively (Bloch et al., (2009).

Naturalistic stimuli show great promise for informing developmental research. Children may find them more captivating and enjoyable than experiment setups with non-naturalistic stimuli. Furthermore, the studies conducted thus far indicate that, while commonalities exist with findings from adult subjects, there also exist differences that offer important insights into how, for example, narratives’ comprehension evolves over the course of development. One area of application that may be able to benefit from opportunities to use naturalistic stimuli in neuroimaging is the development of educational materials, through estimation of the extent to which specific materials captivate the user, engage language-processing, and interact with memory-encoding structures.

Feature films in non-invasive neuroimaging studies of emotions

Movies and narratives as powerful tools in emotion research

Movies and narratives can robustly elicit genuine emotional reactions far beyond those to, for example, “emotional” pictures (Westermann et al., 1996). Therefore, naturalistic stimuli represent a powerful tool for neuroimaging studies of emotions. Such stimuli were used to elicit emotions during recording of brain activity as early as 1954 (Gastaut and Bert, 1954). Today, several decades later, movies and narratives have seen use as naturalistic stimuli in a wealth of neuroimaging studies (Aftanas et al., 1998; Cahill et al., 1996; Decety and Chaminade, 2003; Dimpfel et al., 2003; Dougherty et al., 1999; Gilbert et al., 1989; Jones and Fox, 1992; Karama et al., 2002; Krause et al., 2000; Lane et al., 1997; Levesque et al., 2003; Redoute et al., 2000; Reiman et al., 1997). Even inducing anger, often considered challenging, has been possible with movie stimuli (Jacob et al., 2018). First-person perspective seems especially efficient in driving embodied and emotional perception (Eich et al., 2009; Said Yekta et al., 2009). In this research area, methodology advances have been vital (Dayan et al., 2018; Gaz et al., 2012; Jaaskelainen et al., 2008, 2016b; Nummenmaa et al., 2012, 2014a; Viinikainen et al., 2012; Wallentin et al., 2011). One good example is producing successful classification of emotional states on the basis of brain-activity patterns by developing machine-learning algorithms.

Classifying emotions by means of brain-activity distribution patterns

There are reports of successfully classifying specific emotions (e.g., anger, fear, sadness, and happiness) from fMRI data with machine-learning algorithms (Kragel and LaBar, 2015; Saarimäki et al., 2016, but see also Karama et al., 2011). In other words, by proceeding from patterns of brain-activity distributions across and within specific anatomical areas, one can ascertain with fair accuracy which basic emotion the subject was experiencing during neuroimaging. This suggests that each basic emotion is generated via a distinct brain mechanism (Kragel and LaBar, 2015; Saarimäki et al., 2016). Recent evidence indicates that it is possible to classify more specific emotions also, such as love and pride within the “happiness” basic-emotion category (Saarimäki et al., 2018). In addition, brain activity accompanying sadness differs between two types of sadness-inducing movies, one involving also sympathy and another involving hate (Raz et al., 2012). These findings lend support to the hypothesis that successful classification of patterns of activity for the basic-emotion categories reflects an average pattern across slightly differential within-category instances of the basic emotion rather than, e.g., sadness always involving exactly the same replicable activation pattern (Clark-Polner et al., 2017). This is an interesting theoretical question relating to the debate on discrete vs. constructivist emotion theories that could be addressed in future studies.

The ability to classify emotions opens new possibilities for investigating the neural events underlying experienced emotions. For the social sciences, this might provide a method by which to circumvent the problem of subjects responding in a socially acceptable manner in self-reporting-type questionnaires. In addition to fMRI, emotion classification can be accomplished from EEG data at accuracies as high as 80% (Özerdem and Polat, 2017; Shuang et al., 2016; Yano and Suyama, 2016), and EEG also suffices for estimating emotional valence – i.e., negative vs. positive emotional state (Costa et al., 2006; Zhao et al., 2018). Given that EEG can be implemented as wearable technology, it holds potential for studies in more naturalistic conditions than during fMRI, such as estimation of emotional reactions during social interactions. Estimation of differential emotional reactions to members of ingroup vs. outgroups provides one example of this, as self-reports are often biased in such settings by the need to look acceptable in the eyes of others.

Spatiotemporal orchestration of brain activity during emotional experiences

Classifying spatiotemporal patterns of activity is an important area of development in data analysis. Many studies have investigated inter-area connectivity and activity during emotional experiences (Jang et al., 2017; Leon-Carrion et al., 2006; Nummenmaa et al., 2012, 2014a, 2014b; Raz et al., 2016; Schlochtermeier et al., 2017; Straube et al., 2010); however, few have examined spatiotemporal evolution of brain activity during emotional experiences. An exceptional early study identified regions of the brain that are involved in the initial stages of emotion elicitation during listening to a narrative (Sabatinelli et al., 2006). More recently, movie clips of disgusting vs. appetizing food were found to elicit a cascade of brain events from the orbitofrontal cortex and visual cortices to the periaqueductal gray matter, amygdala, and insula, finally reaching the cerebellum and DMN structures (Pujol et al., 2018). These findings suggest that midline structures, which are central to emotion classification (Saarimäki et al., 2016), are involved in the final stages of emotions’ elicitation and maintenance. Furthermore, future studies could address to what extent such emotion-representations play a role in memory representations that appear to be coded as distributed patterns of activity across similar set of brain regions.

The neural basis of emotion regulation

Emotion regulation can both suppress emotional experiences (as with wanting to brush aside sadness) and augment it (e.g., in wanting to use anger to fuel one’s assertiveness before complaining about poor customer service). Emotion-regulation instructions can be readily given to experiment subjects watching movies or listening to narratives. Frontoparietal areas showed widespread activation when subjects were instructed to suppress their emotions while viewing negative-emotion-oriented movie clips during fMRI (Shimamura et al., 2013). The greater the VMPFC’s role in the emotion-regulation network during highly anger-inducing parts of the movie, the stronger the individual subject’s anger management and the lower the subject’s trait-anger level (Jacob et al., 2018). Similarly, stronger frontal EEG activity is associated with better emotion regulation during induction of emotions (Dennis and Solomon, 2010; see also Nitschke et al., 2004). These findings point to high importance of frontal cortical mechanisms in emotion regulation, thus validating previous findings obtained with non-naturalistic stimuli. Finally, there is also recent intracranial EEG evidence suggesting that temporal pole regulates amygdala responses during both presentation of emotional pictures and movie clips (Sonkusare et al., 2020).

A related line of research has investigated how acute stress modulates subsequent emotional responses. In one study, acute stress induced via aversive movie clips resulted in hypersensitivity and less specificity of amygdala responses while activations in sensory areas were augmented (van Marle et al., 2009). In a follow-up study, functional connectivity increased between the amygdala and a set of brain regions including the anterior cingulate cortex (ACC), the anterior insula (AI), and subcortical regions after acute stress (van Marle et al., 2010). A study a few years later found emotional movies to prime the amygdala so as to react more strongly to fearful faces after negative-valence emotional movies, with the opposite pattern observed after viewing of positive-valence ones (Pichon et al., 2015). Together, these findings shed light on the neural basis of lingering emotional aftereffects. The strength of naturalistic stimuli in inducing emotional states (in comparison with, e.g., still pictures) makes them very well suited for studying such emotional aftereffects. They also suggest that future neuroimaging-based studies of emotions should employ sufficiently long wash-out periods between emotion conditions.

A few studies have addressed the neurotransmitter basis of emotion regulation. In an early EEG study, anxiolytic effects of smoking occurred in conjunction with right-hemisphere activity in smokers, suggesting that acetylcholine plays a role in emotion regulation given that nicotine stimulates the acetylcholine system (Gilbert et al., 1989). In an fMRI study, a single dose of the antidepressant mirtazapine increased subjects’ functional connectivity associated with positive emotional valence in cortical midline and limbic structures while they listened to a narrative, suggesting that the histamine and serotonin systems are involved in emotion regulation (Komulainen et al., 2017). Also, prenatal cocaine exposure has been observed to modulate responses to food cues in the ventral striatum, ACC, and MPFC, with this effect correlated with self-reported food cravings (Yip et al., 2014). These findings offer some promising leads for further research on the neurotransmitter basis of emotion regulation.

Neuroimaging that addresses brain correlates of sexual arousal

Movie clips offer a valuable tool for investigating brain-activity correlates of sexual arousal. An early PET study wherein subject were shown sexually arousing movie clips suggested that its cognitive, emotional, motivational, and autonomic aspects are associated with ACC, orbitofrontal cortex, striatal structure, and hypothalamus activations (Redoute et al., 2000). In another study, the nucleus accumbens and MPFC were selectively activated during anticipation of erotic film content vs. aversive movie clips (Greenberg et al., 2015). Results of gender-effect studies have been mixed. Greater hypothalamus activation has been observed in males than females during watching of erotic movies, an effect that correlated with self-reported sexual arousal (Karama et al., 2002). In contrast, a recent study found stronger correlations between brain activations and physiological measurements of sexual arousal in females than males (Parada et al., 2018). In yet another study, researchers found the between-gender similarities to be greater, overall, than the differences (Parada et al., 2016). Such results seem to indicate that ample room remains for scholars to address the gender differences with respect to sexual arousal related brain activations.

Social emotions

Social emotions are often defined as emotional states and rections that depend on, or are modulated by, others’ actual or imagined emotions. Shame and pride are good examples of social emotions that differ from basic emotions such as fear that can be felt irrespective of others’ feelings, such as fear experienced when walking alone in a dark forest. Via machine-learning algorithms, social emotions can be classified on the basis of patterns in the distribution of brain activity, which suggests that individual social emotions have partly distinct neural bases (Saarimäki et al., 2018). Inter-individual differences play a role in social emotions, with differences in self-reported power vs. affiliation motivations being predictive of differential fMRI activity in the prefrontal cortex and striatum during viewing of power-related vs. love-related movie clips (Quirin et al., 2013). Affection for someone in the movie also makes a difference: in a PET study, watching one’s favorite people elicited a positive mood; elevated immune-system function; increased dopamine levels; and activated the MPFC, thalamus, hypothalamus, pCC, STG, and cerebellum (Matsunaga et al., 2008).

In a recent study, activity in core-affective and MPFC areas was enhanced when subjects were shown an arousal-indicated measurement purportedly of another subject watching the movie (Golland et al., 2017). This approach offers a way of assessing the modulatory effects of others’ emotional reactions on experiment subjects’ emotions, which plays a central role in study of social emotions. “Catching” and being influenced by other people’s emotions have a central role in the phenomenon of herding (i.e., how individual decisions are influenced by behavior of the group), so methods that permit studying the underlying neural mechanisms create new and exciting possibilities for future research.

In parallel with findings from the aforementioned studies of the aftereffects of emotion-eliciting stimuli, interesting studies have examined aftereffects of social-emotional narratives. Listening to a patient’s story about a dismissive parental relationship in childhood reduced supplementary motor areas’ connectivity with other areas of the brain during subsequent rest (Borchardt et al., 2015). In another study, reduced functional connectivity patterns were identified between the caudate nucleus, TPJ, DLPFC, and cingulate cortex after certain types of emotional attachment narrative (Krause et al., 2018), and EEG findings suggest that, while subjects remain aroused after hearing attachment narratives, they affectively disengage from content about dismissiveness soon after hearing it (Borchardt et al., 2018). Finally, negative emotions induced by movie clips inhibited lateral prefrontal cortical activation during a subsequent theory-of-mind (ToM) task in a functional near-infrared spectroscopy study (Himichi et al., 2015); due to the method used in this study only the activity of lateral prefrontal cortex was measured. These findings suggest, in sum, that negative-emotional narratives reduce functional connectivity (and activity) in areas that support social cognition and social approach. This type of modulation is possibly akin to how a priori information and explicit perspective-taking modulate processing of movie clips and narratives in the brain, as described above in connection with perspective-taking. Notably, the lingering effects of hearing about social rejection are something that would be highly difficult to study using non-naturalistic stimuli.

Naturalistic stimuli in neuroimaging studies of social cognition

The default-mode network and social cognition

In the area of social cognition, naturalistic stimuli have extended the possibilities of neuroimaging to the investigation of the neural mechanisms supporting observation of dynamically evolving social interactions, which is something that would be difficult to study with non-naturalistic stimuli. Neuroimaging studies with naturalistic stimuli have provided significant insights into what takes place when DMN structures are activated. Prior to these studies, researchers had observed that the DMN is activated when there is no task, yet concrete proof of what function this might serve was lacking. In a pioneering study, perception of social interactions robustly activated the DMN (Iacoboni et al., 2004). Importantly, the results suggested that subjects were “daydreaming” about their relationships during rest. Highlighting the DMN’s role in social cognition, these findings support the view that the DMN creates and plays through new scenarios of how one should act in various social situations, which get stored to long-term memory and help prepare the individual for later real-life social interactions (Bar, 2009). Even though the functions served by DMN are highly versatile beyond social cognition, naturalistic stimuli have been especially suited for studying the role of DMN in social cognition.

It is noteworthy with regard to the DMN regions that the pSTS/TPJ seems to play a central role in perception of social cues that occur over short time intervals (e.g., one person smiling at another). When responses to brief clips with various types of social content vs. non-social content were compared, the pSTS/TPJ emerged as the region most broadly responsive to social content (Lahnakoski et al., 2012). Corroborating this, short movie clips of facial expressions elicited activity across the STS, fusiform face area, and occipital face area (Reinl and Bartels, 2014). Providing causal evidence of the involvement of pSTS, transcranial magnetic stimulation over the pSTS reduced fMRI responses to three-second clips of moving faces in the pSTS (Pitcher et al., 2017). The involvement of pSTS/TPJ in perception of social cues has been further clarified in a study that contrasted activations caused by movie clips of unresolved interactions (e.g., two people wordlessly staring at each other), non-habitual interactions (e.g., a pedestrian patting a cyclist on the shoulder at a traffic light), and habitual interactions (e.g., paying for groceries at a store) (Wolf et al., 2018). Notably, movie clips are very well suited for contrasting fMRI activity between these different types of interactions, and it would have been difficult to study this with artificial stimuli. Watching the non-habitual interactions specifically activated the TPJ. In contrast, watching habitual interactions activated the hippocampus and lateral occipital cortex. Together these findings suggest that pSTS/TPJ is intimately involved in the perception of a wide range of social cues, and seems to be especially recruited during perception of non-habitual social interactions, suggesting that it participates in higher-order, less automated, processing of social cues.

Perception of goal-directed behavior in movies

Naturalistic stimulation studies have helped clarify the neural mechanisms underlying humans’ processing of goal-directed actions in movies (Cooper et al., 2011; Guclu and van Gerven, 2017; Hamilton and Grafton, 2006; Salmi et al., 2014; Shimada, 2009). Whereas still pictures can represent goal-directed actions, movies contain dynamic information that the brain must process in day-to-day life, which involves others’ goal-directed actions. Movies permit, for instance, comparing the effects of seeing mere stimulus motion with those of viewing goal-directed behaviors. In such comparison, posterior parietal cortical areas were observed to respond specifically to observed goal-directed actions of others (Salmi et al., 2014). These findings are extended by observations of posterior parietal regions specifically responding to semantically meaningful actions (Newman-Norlund et al., 2010). Furthermore, seeing a protagonist pass on an object to someone who then performs either an incongruent or a congruent action with it activated the right IFG, suggesting on the involvement of IFG as the goal-directed actions get more complex (Shibata et al., 2011). Supporting this interpretation, in another study, inferior frontal areas exhibited augmented hemodynamic responses during watching of cooperative vs. non-cooperative actions (Shibata et al., 2007). Again, here the use of naturalistic stimuli has opened up possibilities for research, as studying the processing of cooperative vs. non-cooperative behaviors and incongruent vs. non-incongruent actions with an object would be very challenging using artificial stimuli.

Brain regions associated with processing of objects (occipital), actions (posterior parietal and IFG), and others’ mental states (VMPFC) were context-dependently activated when experiment subjects watched another person search for hidden objects, thus providing insights beyond findings obtained using artificial stimuli by suggesting that action comprehension in social contexts activates mental state processing areas in addition to the mirror neuron system (Ramsey and Hamilton, 2012). The VMPFC was also activated in healthy subjects during perception of social actions with potentially negative consequences such as cutting in line (Grossman et al., 2010). Furthermore, this activation overlapped with damaged brain areas in neurological patients with reduced sensitivity to negative consequences of inappropriate actions (Grossman et al., (2010). Detecting errors committed by others in movie clips activated the striatum, TPJ, and IFG (Jaaskelainen et al., 2016a). When errors by others were observed under only a non-naturalistic paradigm, this activity was limited to the IFG, demonstrating that naturalistic stimuli can more fully reveal the brain regions involved in processing errors by others (Jaaskelainen et al., 2016a). The studies mentioned here hint at the wide range of goal-directed actions amenable to study through naturalistic stimuli.

Theory of mind and mentalizing

Naturalistic stimuli offer a rich array of possibilities for research on the neural basis of ToM, in that mentalizing about others often involves processing social cues that occur over longer timescales. There are studies attesting to considerable overlap between activity elicited by ToM events in naturalistic stimuli and areas activated by explicit ToM localizer tasks (Jacoby et al., 2016; Naughtin et al., 2017; Wolf et al., 2010). However, entirely new observations too have been provided by neuroimaging studies employing naturalistic stimuli. For instance, ToM was additionally associated with the left superior medial frontal gyrus and precuneus activity during listening to an audiobook (Kandylaki et al., 2015). Inference of the intention vs. consequences of protagonist actions showed involvement of the TPJ, visual areas, MPFC, IFG, and angular gyrus (Mason and Just, 2011). Observing virtual characters in host–guest business interactions where the subject was to subsequently judge whether the interactions had been between “friends” or “foes” activated the STS, lateral and medial prefrontal (MPFC) areas, and the amygdala (Sung et al., 2011). In another study, clips from live-action movies activated the MPFC more strongly than cartoons, perhaps through stronger mentalization (Han et al., 2005). Also, the anterior temporal lobes were recruited when social cognition demanded access to social conceptual knowledge (Ross and Olson, 2010). In developmental studies with naturalistic stimuli, a distinct ToM network was observed to develop gradually after three years of age without strict correspondence with the emergence of ability to pass the false belief task (Richardson et al., 2018), and in adolescents activity in areas associated with ToM correlated with development of moral reasoning (Sommer et al., 2014).

Emotional and cognitive empathy

Empathy is an aspect of social cognition that is thought to consist of two distinct processes, where emotional empathy involves being disposed to feel others’ emotions (and thus becomes intertwined with research on emotions as described above) and cognitive empathy is analogous to ToM (Shamay-Tsoory, 2011). Neuroimaging studies with naturalistic stimuli have both supported this model and yielded additional insights. Dissociation between independent components attributed to emotional and cognitive empathy during viewing of an emotional movie provided evidence for the model (Vemuri and Surampudi, 2015). Further, findings obtained in a large sample of subjects using “EmpaToM” task specifically developed to contrast between empathetic vs. ToM processing during viewing of short movie clips clearly indicated that emotional empathy and ToM are supported by distinct brain networks that include anterior insula and TPJ, respectively (Kanske et al., 2015). Interestingly, these networks were also separable during resting state.

Findings that have brought new insight include observing that empathy with others’ physical vs. emotional pain selectively activated the insula vs. dorsal MPFC, respectively (Bruneau et al., 2012). This type of dissociation between empathy with physical pain and emotional pain proves far easier to probe with naturalistic than with more traditional stimuli. In a combined fMRI–PET study, the strength of hemodynamic responses in the insula while one was seeing others in pain was predicted by lower mu-opioid receptor availability in the same structure, highlighting the opioidergic system’s role in feeling others’ pain (Karjalainen et al., 2017).

Movies in neuroimaging studies of humor

The neural basis of humor-processing is another area of investigation wherein naturalistic stimuli are highly useful. Many genres of humor are narrative- or film-based, and jokes in comedy movies often serve as more effective stimuli than humor presented by other means, thereby allowing for more robust stimulation of the relevant brain mechanisms during neuroimaging. In an early study, scholars observed that mechanisms of the inferior frontal and posterior temporal areas underlie detection of humor and that the insula and amygdala support appreciating humor (Moran et al., 2004). In another study, the neural basis of irony-processing was broken into components, associated with specific brain structures. The anterior STG was connected with social-conceptual knowledge, the MPFC and anterior ITG with context-appreciation, the amygdala and hippocampus with strength of irony perceptions, and the DLPFC with the degree of humorousness experienced (Akimoto et al., 2014). Other work presented observations that the right DLPFC and temporal-lobe areas seem to govern anticipation of humorous events (Sawahata et al., 2013). In a final noteworthy study, the frontal pole was activated during humor in comedy clips, possibly because the humor extended over a longer time in those clips (Iidaka, 2017; Jaaskelainen et al., 2016b).

Perception of rewards in movies

Observing reward-producing vs. loss-producing actions by others in movie clips resulted in a stronger motor-cortical mu rhythm in EEG traces than did viewing of neutral actions (Brown et al., 2013). Also, active smokers, in a contrast against successfully treated abstaining smokers, showed activity in mesolimbic reward circuitry and several cortical regions correlated with an urge to smoke, including the orbitofrontal cortex (Weinstein et al., 2010). Scholars conducting a related fMRI study found smokers, as compared with non-smokers, to exhibit “mirror-neuron”-type activity in the frontal and parietal regions while watching videos that depict smoking actions (Wagner et al., 2011). A heroin-use-oriented movie produced higher levels of brain activity in prefrontal cortical areas in heroin addicts than in controls, with the reverse observed for erotic movies (Jiang et al., 2014). Such studies as these demonstrate movies’ utility for investigation of the neural basis of reward-processing, which forms the basis for developing addictions.

Conclusions

Movies and narratives have been successfully used as naturalistic stimuli to address a plethora of significant outstanding research questions pertaining to how the sensory, attentional, memory, language, emotion, and social cognition systems function under naturalistic conditions, in cases of health and disorders alike. Using naturalistic stimuli has opened possibilities especially to address questions that would have been difficult to tackle well with a more traditional experimental approach. This article has highlighted such research findings in numerous key branches of research.

These advances include discovery of a cortical hierarchy of TRWs that enable the brain to process dynamic information that accumulates over different timescales in such contexts as conversation, social interactions, and goal-directed behavior. Fuller understanding of how the hippocampus helps segment events occurring in day-to-day life and encode them in memory (e.g., what happens at breakfast gets encoded as a separate event from a subsequent event such as driving to work) represents another significant advance. Our understanding of the nature of human memory has been fundamentally changed by naturalistic neuroimaging studies (Hasson et al., 2015). These mechanisms hold further promise such as great potential in work investigating how the TRW hierarchy and event segmentation interact during natural viewing (e.g., which neural mechanisms cause the various event boundaries to reset the TRWs).

A further insight is that DMN structures support processing of a narrative’s evolving plot. Future studies could investigate how the DMN supports processing across event boundaries. It is also robustly activated during viewing of social interactions. These findings together suggest that the classical observation that the DMN is active during rest results from subjects daydreaming about complex social interactions when they are not engaged in a task during neuroimaging. This could point to new ways of investigating resting state (e.g., aided by experience-sampling methods).

Naturalistic stimuli have also yielded insights into attentional brain mechanisms underlying ability to adopt specific perspectives during natural viewing. This research could be expanded by further linking specific aspects of perspective-related information-gathering to underlying neural mechanisms. For example, particular kinds of social perspective, as in watching an employee and supervisor interact, might be associated with distinct types of demands with regard to the social cues being given attention.

By eliciting genuine and robust emotions, naturalistic stimuli have helped elucidate the brain basis of both basic and social emotions, which seem manifested in highly overlapping yet distinguishable patterns of activity. Future studies could, for example, investigate the spatiotemporal cascades underlying emotional reactions. In the area of social cognition, naturalistic stimuli hold vast potential for examining the neural basis of such phenomena as ToM (e.g., while subjects mentalize about the minds of movie protagonists) and processing of humor (as it evolves on the screen).

It should also be acknowledged that there are a number of limitations and caveats in the use of naturalistic stimuli in neuroimaging studies. Overall, one always loses in control as the stimulus becomes more naturalistic, for example, it could hypothetically be possible that some brain responses attributed to emotions or social cognition could be in fact caused by there being more close-ups in the respective parts of the movie clips being used. These types of caveats can be avoided by utilization of artificial and naturalistic stimulus paradigms in parallel (e.g., by using a continuum of stimuli from artificial to naturalistic across experiments in the same subjects) and by modeling both factors of interest (e.g., self-reports of emotional experiences) and potential nuisance factors (e.g., shot sizes in the movie) and taking them into account in the analyses. Another limitation is that movies and narratives still differ from real life in many ways: the most notable of these is perhaps that the subject is a passive viewer/listener instead of being an active agent with goals that he/she pursues while making decisions. Virtual reality setups would allow circumventing this limitation that pertains to research on memory, attention, language, emotions and social cognition, and might indeed be the next step in this exciting area of research.

Overall, developments in neuroimaging and in complementary behavior- and data-analysis methods hold keys to advancing rapidly to even more robust use of naturalistic stimuli. Combining more traditional controlled experimental designs, such as ToM and other localizer tasks, with use of movies and narratives represents another highly promising research direction, as does expanding naturalistic stimulation from movie clips and narratives to virtual reality and computer-game/simulated environments.