Analysis of stimulus-induced brain dynamics during naturalistic paradigms.

Naturalistic stimuli offer promising avenues for investigating brain function across the rich, realistic spectrum of human experiences. Functional magnetic resonance imaging (fMRI) studies of brain activity during naturalistic paradigms have provided new information about dynamic neural processing in ecologically valid contexts. Yet, the complex, uncontrolled nature of such stimuli -- and the resulting mixture of neuronal and physiological responses embedded within the fMRI signals -- present challenges with respect to data analysis and interpretation. In this brief commentary, we discuss methods and open challenges in naturalistic fMRI investigations, with a focus on extracting and interpreting stimulus-induced fMRI signals.

Introduction

An important challenge in neuroscience is to understand how the brain processes information in dynamic, naturalistic settings. Most current knowledge about brain function comes from the use of classical task paradigms (Kanwisher et al., 1997; Raichle et al., 2001), and although these methods have provided tremendous insight, they are artificial by nature. In contrast, naturalistic stimuli – though highly complex and less well controlled – may be better suited to probe the inherent richness of human experiences. Indeed, one exciting advantage of naturalistic stimuli over classical paradigms is the ability to explore context-dependent neural processes across different timescales.

However, a key question is: how can we best map between these measured fMRI signals and the multiple cognitive and affective processes that emerge in response to these complex, naturalistic stimuli? Below, we discuss two main challenges in this end eavor: first, how can we best extract stimulus-related neural dynamics from other intrinsic and noise contributions to fMRI signals? And second, assuming one can successfully extract fMRI signals that are driven by naturalistic stimuli, how can we go about resolving the multiple ongoing neural (and non-neural) processes embedded in these fluctuations? We focus on the use of dynamic naturalistic stimuli – such as listening to a story or watching a movie – during fMRI.

Extracting stimulus-driven fMRI signals

As a narrative or movie unfolds, it engages the brain in a neural dynamic trajectory shaped by both low-level and higher-order features of the naturalistic stimulus. In addition to quantifying fMRI signal levels that correspond to certain derived stimulus features (Bartels and Zeki, 2004; Russ and Leopold, 2015), there is a wealth of dynamic analysis techniques that may be fruitfully applied to naturalistic paradigms. For example, time-varying correlations (Lindquist et al., 2014; Sakoglu et al., 2010; Allen et al., 2014), time-frequency analysis (Chang and Glover, 2010), autoregressive models (Liegeois et al., 2019), multivariate patterns (Chang et al., 2019), low-dimensional trajectories (Venkatesh et al., 2019), and co-activation patterns (Liu and Duyn, 2013) may be applied to examine the temporal evolution of regional or network properties. However, when applied to individual fMRI scans (as with resting-state data), the resulting ‘dynamics’ not only reflect stimulus-induced neural processes, but also intrinsic activity and any non-neural fluctuations that were not suppressed during pre-processing.

Analyses based on inter-subject or inter-scan paradigms (Glerean et al., 2012; Hasson et al., 2004; Haxby et al., 2011; Hejnar et al., 2007; Simony et al., 2016) may be well suited for addressing this issue. In contrast with the stimulus-free resting state, naturalistic stimuli can drive reliable neural responses across different subjects. Early fMRI studies of naturalistic stimuli introduced inter-subject correlation (ISC), which quantifies the across-subject reliability of stimulus-driven responses within regions of interest (Hasson et al., 2004). Building upon ISC, the idea of inter-subject functional correlation (ISFC) was recently introduced (Simony et al., 2016), where inter-regional correlations are calculated between fMRI signals of different subjects experiencing the same continuous naturalistic stimulus (Fig. 1A and B). Here, ISFC dramatically increased the specificity to stimulus-induced inter-regional correlation (Simony et al., 2016), revealing reproducible dynamics of the default-mode network during a continuous auditory narrative (Fig. 1C). As with ISC, the improvement arises from suppressing intrinsic, task-unrelated neural dynamics (e.g., attentional variations) as well as artifacts (e.g., respiratory rate; head movement (Van Dijk et al., 2012; Power et al., 2012)) that can influence network correlation patterns within a brain (Fig. 2A and B) but are less likely to be correlated across subjects (but see Fig. 2C–E). On the other hand, IS(F)C does not detect potentially interesting individual differences in neural activity.

Fig. 1.

Inter-subject functional correlation (ISFC) method to study brain dynamics during Naturalistic stimulation.

(A) During task processing, the measured BOLD signal can be decomposed into stimulus-induced signal (blue); intrinsic neural signals (spontaneous fluctuations), and non-neuronal signals (red). As some non-neuronal signals are stimulus-induced, ‘N’ here refers to those which occur independently of the stimulus. (B) Seed-based ISFC is the Pearson correlation between a time course extracted from one region in subject X and all other regions in subject Y. (C) Reliable dynamics of the mean ISFC within the DMN network across two independent groups of 18 subjects. ISFC is computed using a sliding window of 45 s (30 TRs), in steps of 1 TR. (D) Mean ISFC dynamics corresponded to behavioral/cognitive dynamics. Here, the mean ISFC in the DMN for each segment of the story was correlated with recall of that segment of the narrative (r = 0.6, q < 0.02). Panels adapted from Simony et al., 2016).

Fig. 2.

Physiological measures collected during naturalistic stimulation.

(A) Cross-correlation (r~ = 0.5) between the RV and BOLD signal in the precuneus for one individual subject during rest. The precuneus is a major hub of the default-mode network, which has been a focus of recent ISC/ISFC investigations. (B) The correlation between the RV and BOLD signal within the brain of an individual subject during rest is high. (C) Respiratory variation (RV) signals were uncorrelated across the 9 subjects during both the rest condition (r = −0.004, p > 0.2) and the story condition (r = −0.02, p > 0.18). (D) Interestingly, regressing out the RV signal from the precuneus time course resulted in a 20% increase in the inter-subject correlation (ISC). This suggests that when physiological effects are not synchronized across subjects, filtering them out during preprocessing can better expose stimulus-induced signal that is shared across subjects. (E) However, in some cases – particularly with emotional stimuli – peripheral physiological signals can be highly synchronized across subjects. Here, time courses of electrodermal activity (EDA; left) and heart rate (HR; right) were each compared across two subsamples of participants (n = 10 and n = 11) during free viewing of an emotional movie. Panels adapted from Simony et al., 2016) (A,B,C,D) and Golland et al., 2014 (E).

Interpreting stimulus-driven fMRI signals

Once stimulus-driven fMRI signals are extracted, how can we then uncover the functional role(s) of localized activity or network patterns along different time scales? Furthermore, as certain naturalistic stimuli can powerfully elicit autonomic responses, how can such dynamics be disentangled from those relating to cognition? We discuss these questions and potential avenues below.

Linking neural and cognitive dynamics

Fig. 1D demonstrates that measuring behavior can be crucial for uncovering the functional role of fMRI dynamics in naturalistic conditions, such as memory encoding (Simony et al., 2016). For example, our more recent work (Brandman et al., 2019) measures multiple facets of cognitive function (such as episodic memory, emotional intensity, valence and surprise), which are modelled as the movie unfolds. This allowed us to track complex cognitive states across time, which was then correlated with ISFC fluctuations to elucidate the functional role of the DMN in each of these naturally induced cognitive states.

In addition, using different types of naturalistic stimuli, and comparing multiple approaches for measuring cognitive dynamics (including off-line questionnaires, on-line ratings (Brandman et al., 2019; Furman et al., 2007; Chen et al., 2016; Nummenmaa et al., 2012; Chen et al., 2017)) may help to uncover commonalities in neural mechanisms underlying cognitive functions that would be hard to discover using reductionist paradigms. An interesting question here is to what extent cognitive dynamics can be explained by localized fMRI activity, or from network co-activation dynamics.

Further, modeling naturalistic stimulus features (e.g., faces, language) may allow for disentangling their respective fMRI responses. While stimulus modeling presents challenges, several groups have successfully addressed this topic and uncovered new insights into neural representations (Bartels and Zeki, 2004; Russ and Leopold, 2015; Bartels et al., 2008; Huth et al., 2016; Nishimoto et al., 2011). Finally, an emerging line of work involves selecting/designing naturalistic stimuli that may enhance detection of cognitive traits (e.g., paranoia (Finn et al., 2018)), link neural and social distance (Parkinson et al., 2018), and enhance compliance in pediatric studies (Vanderwal et al., 2015).

Temporal and spatial representation of naturalistic stimuli

Since natural stimuli are inherently embedded in time, the fMRI signal at any given moment may be shaped by varying amounts of past history and context. Timescales of meaningful neural dynamics can be probed by manipulating the temporal structure of naturalistic stimuli, and studies adopting such an approach have demonstrated that “temporal receptive windows” differ across the processing hierarchy (Hasson et al., 2008; Lerner et al., 2011; Honey et al., 2012; Murray et al., 2014), with sensory areas reflecting low-level stimulus features and high-order cognitive/perceptual areas having longer integration times. Such findings highlight one key advantage of naturalistic stimuli over resting-state fMRI – as the latter contains no reference to guide the selection of potentially relevant time scales – and also motivates the further development of data analysis methods that capture dynamic neural processing on multiple temporal scales.

Another exciting line of work, enabled by the growing availability of large datasets, is the use of deep neural networks (DNN) for modeling how naturalistic stimuli are translated into fMRI signals (encoding models), and for decoding cognitive states from fMRI signals. The layers of a trained DNN, for example, may capture a spectrum of lower-level to more abstract spatial stimulus representations (e.g., from low-level visual features to categories), which may be interrogated in terms of their relation to neuroimaging data and hence contribute toward our understanding of neural representations of natural stimuli (Wen et al., 2017; Grossman et al., 2018; Khaligh-Razavi et al., 2016). In that sense, DNN may be regarded as complementary to ISFC, with the latter perhaps more suitable for examining correlations between brain-activity patterns and stimulus features.

Currently, while DNN may succeed on specific tasks (e.g. face recognition) they are still far from exhibiting artificial general intelligence (AGI) (Ullman, 2019). A critical issue is that current DNN models do not incorporate known brain circuitry (e.g., lateral connections within a neural-network layer, large-scale connectivity across layers (Ullman, 2019)), and do not model the interplay between spontaneous and task-induced activity. We hope that future advances in DNN models of neural activity during naturalistic stimulation can benefit both AGI and neuroscience.

Disentangling cognitive processing, autonomic responses, and physiological “noise”

While inter-subject analyses can help to isolate stimulus-related fMRI dynamics, a further challenge arises in interpreting stimulus-related signals, since naturalistic stimuli can drive not only multiple forms of cognition but also synchronized autonomic responses. Emotional scenes, for example, can elicit local neural responses from regions involved in autonomic nervous system (ANS) regulation (Golland et al., 2014; Napadow et al., 2008) as well as hemodynamic fluctuations relating to changes in systemic physiology (Wise et al., 2004; Murphy et al., 2013; Ozbay et al., 2018).

Therefore, stimulus-induced fMRI signals during naturalistic stimuli may contain several components: (a) cognitive neural activity (e.g. memory traces),(b) autonomic neural activity (e.g. in insula and hypothalamus, which integrate emotional or cognitive information into the coordination of ANS responses), and (c) systemic modulation of brain hemodynamics (e.g. from changes in arterial CO2 arising from changes in respiration depth/rate (Wise et al., 2004). While the best ways of isolating these signal sources remain open questions, one avenue is based on continuously recording psychophysical or physiological markers throughout the scan, complementary to ratings of cognition. These measures may allow for tracking ongoing drifts in arousal (e.g., with pupil diameter) and systemic physiology (e.g. (Golland et al., 2014; Yellin et al., 2015; Chang et al., 2016; Falahpour et al., 2018), and Fig. 2A,E).

One valuable use of such measures is to quantify the degree to which time courses of such effects differentiate from those of other cognitive/behavioral features – indicating whether effects (a-c) may be separated for analysis. In addition, we found that when physiological responses are not synchronized across subjects, explicitly filtering such effects can improve the signal-to-noise ratio of ISFC (Fig. 2B-D (Simony et al., 2016)).

Indeed, a second potential use of peripheral signals is for removing non-neural effects from fMRI data (Murphy et al., 2013). Here, we note that (b) and (c) may differ somewhat in their spatial and temporal dynamics. Considering respiration, the primary impact of a deep breath on the BOLD signal appears to occur at least 10s after the breath itself (Birn et al., 2008; Chang et al., 2009), whereas neural activity causing the deep breath would elicit a BOLD response closer in time to the breath itself. Spatially, neural activity in (b) may localize to regions comprising a central autonomic network (Beissner et al., 2013), while the passive effects of heart rate and respiration fluctuations are more spatially widespread (Birn et al., 2006; Shmueli et al., 2007).

Finally, physiological metrics also provide a powerful means for investigating brain-body interactions; recently, for example, synchronized cardiac responses during emotionally salient films were harnessed to inform mechanisms by which the insula may integrate internal and exterior percepts (Nguyen et al., 2016).

Conclusions

Naturalistic stimuli with fMRI present rich opportunities for understanding brain function in real-world contexts, and carries a number of interesting future directions for signal analysis and interpretation. We believe that progress in this field will be expanded by further development of analytic tools for mapping between fMRI signals and cognitive dynamics, and by recording behavioral probes of cognitive state and peripheral physiological measures to strengthen investigation into neural processes, such as memory and emotion, from fMRI signal dynamics.