Levels of naturalism in social neuroscience research.

In order to understand ecologically meaningful social behaviors and their neural substrates in humans and other animals, researchers have been using a variety of social stimuli in the laboratory with a goal of extracting specific processes in real-life scenarios. However, certain stimuli may not be sufficiently effective at evoking typical social behaviors and neural responses. Here, we review empirical research employing different types of social stimuli by classifying them into five levels of naturalism. We describe the advantages and limitations while providing selected example studies for each level. We emphasize the important trade-off between experimental control and ecological validity across the five levels of naturalism. Taking advantage of newly emerging tools, such as real-time videos, virtual avatars, and wireless neural sampling techniques, researchers are now more than ever able to adopt social stimuli at a higher level of naturalism to better capture the dynamics and contingency of real-life social interaction.

Introduction

Social interaction is critical to the survival of humans, non-human primates, and other animals. Humans with better social skills are more likely to obtain access to resources and tend to be healthier (Cohen, 2004). Similarly, social relationships in non-human primates enhance longevity in many species, including baboons and macaques (Silk et al., 2010; Brent et al., 2017). Social behaviors such as face and gaze processing, observational learning, cooperation, and competition are essential for the survival and are importantly regulated by both cortical and subcortical brain regions (Chaverri et al., 2018; Chen and Hong, 2018; Freiwald, 2020; Gangopadhyay et al., 2021).

Social interaction is complex and can be modulated by many social factors, such as the identity, hierarchy, emotional state of others, as well as their action and decision. How humans and other animals perceive, process, and react to such information among noise in the everchanging environment is an intriguing yet challenging question. To answer that, researchers have been using a variety of social stimuli to examine social interaction and the underlying neural mechanisms. In this review, we discuss social stimuli used in empirical research by categorizing them into five levels of naturalism and describe the advantages and limitations of each level (Figure 1).

Figure 1

Illustration of five levels of naturalisms with respect to social stimuli commonly used in social neuroscience research and their positions along the tradeoff between ecological validity and experimental control

Traditional approaches in social neuroscience have largely focused on how individuals process social information. By presenting images of social agents and social scenes (Figure 1, Level 1) that are simple to generate and manipulate, researchers have examined a wide array of topics, such as face and gaze perception, action understanding, and social decision-making. To overcome the information-limiting nature of using static images, researchers have also capitalized on video playbacks (Figure 1, Level 2) to provide additional dynamic information to study more complex social behaviors while largely preserving experimental control. However, during real-life social interactions, individuals not only perceive sensory social cues but also actively respond to them and send social signals to others—that is, social interaction is inherently dynamic and contingent. Our behaviors have direct or indirect consequences on others' behaviors and change accordingly over time. Real-time videos of real agents and real-time animations of virtual avatars controlled by real agents or computer algorithms (Figure 1, Level 3) render relatively high level of both experimental control and ecological validity. Although virtual avatars have been used in images and pre-edited videos, one prominent benefit of using virtual avatars is that they can be easily modified to display contingent behaviors with the advancement of computer science and artificial intelligence. Therefore, despite of existing applications in images and videos, we include virtual avatars in Level 3 given their potential to convey both dynamic and contingent social information. Researchers have begun to embrace the idea of including dyads or triads of real subjects in the laboratory (Figure 1, Level 4) (Schilbach et al., 2013; Isoda et al., 2018). Studies using hyper-scanning via functional near-infrared spectroscopy (fNIRS), electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), as well as simultaneous neuronal recording from multiple agents have revealed novel supports for inter-brain synchronization. A similar approach can be applied to study real-life interactions in natural environments (Figure 1, Level 5), such as teacher-student interaction in classrooms, musician coordination in ensembles, a small group of animals in semi-natural environments, and a large population of animals in the wild.

Social interaction is thought to operate under an irreducibly collective “we-mode” that goes beyond individualism (Gallotti and Frith, 2013). Further, the neural representation of real-life multisensory social interaction cannot be easily predicted by studying unisensory social information in isolation (Schilbach et al., 2013; Krakauer et al., 2017; Shamay-Tsoory and Mendelsohn, 2019). Overall, in order to capture the key features of social interactions (e.g., dynamics and contingency), we encourage researchers to adopt stimuli with a higher level of naturalism by taking advantage of advanced techniques (e.g., wireless multi-channel recordings from single neurons as in Roy and Wang, 2012) and data analytics (e.g., markerless tracking of behaviors using deep learning network as in Mathis et al., 2018). The field of social neuroscience has already been evolving toward a non-reductionist direction, and researchers have advocated for naturalistic paradigms. For example, Schilbach et al. (2013) emphasized the importance of emotional engagement during social interaction compared to passive observation and reflection of others. Shamay-Tsoory and Mendelsohn (2019) discussed how active versus passive engagement modulates memory formation and retrieval in social contexts. In this review, rather than focusing on a specific social behavior or function, we discuss social stimuli used in empirical research in a wide range of topics by categorizing social stimuli into five levels of naturalism. We present these five levels of stimuli in an order based on their respective positions along the trade-off between ecological validity and experimental control. However, we by no means negate the importance of traditional approaches. We believe that what we learn about the brain's basic functions using well-controlled social stimuli can facilitate our understanding of real-life social interaction and vice versa (Matusz et al., 2019).

Level 1: Static social stimuli

Traditionally, researchers have presented static images of social agents and social scenes to investigate various aspects of social cognition. Such stimuli are easy to manipulate and highly controlled, offering convenient ways to correlate behavioral and neural variables with minimal confounds. Figure 2 (left column) illustrates some example paradigms using Level 1 social stimuli.

Figure 2

Example paradigms using social stimuli Level 1–2

Shown are selected static images (left column) and video playbacks (right column) used to examine a wide range of social behaviors.

(A–J) (A) Wang et al., 2017; (B) Shepherd et al., 2006; (C) Deuse et al., 2016; (D) Carter et al., 2012; (E) Arsalidou et al., 2011; (F) Mosher et al., 2011; (G) Shepherd and Freiwald, 2018; (H) Centelles et al., 2011; (I) Sliwa and Freiwald, 2017; (J) Gunhold et al., 2014a.

Reproduced with permissions or under Creative Commons CC BY license (https://creativecommons.org/licenses/by/4.0/).

Social information is often gathered from the face, and the ability to perceive, recognize, and interpret facial cues is regarded as one of the most important social skills. By simply observing a face, both humans and non-human primates can gain information about others' identity, social status, intention, emotion, as well as mental state (Adolphs, 2006; Tsao and Livingstone, 2008; Baron-Cohen et al., 2013). Classical experimental paradigms for studying face processing have used controlled two-dimensional facial images presented on a computer monitor in humans (Guastella et al., 2008), rhesus monkeys (Macaca mulatta) (Gothard et al., 2007; Dal Monte et al., 2014), as well as chimpanzees (Pan troglodytes) (Kret et al., 2018). These studies have played pioneering roles in our understanding of social cognitive processes with the power of manipulating images or image sequences in experimentally controlled manners (Whalen et al., 1998; Hasson et al., 2001; Kret et al., 2018).

By using static images, the field has obtained significant insights into the neural bases of face perception. It has been discovered that face processing is highly localized to several face-selective areas extending from the occipital lobe to the temporal lobe, containing “modules” with different functional specializations (Kanwisher et al., 1997; Allison et al., 2000; Tsao and Livingstone, 2008). Additionally, by using static facial images, functional neuroimaging studies in humans and macaques have established a key role of the amygdala in perceiving faces (Mende-Siedlecki et al., 2013) and facial expressions (Sabatinelli et al., 2011; Hadj-Bouziane et al., 2012). Neuronal recording studies using images have found that the amygdala indeed signals faces and their relevant variables in both humans and monkeys (Gothard et al., 2007; Wang et al., 2017).

Static images have also been broadly used to study social gaze. Both humans and non-human primates reflexively follow gaze from a young age (Moore and Corkum, 1998; Shepherd et al., 2006) and can deliberately direct attention to where another individual is looking by exhibiting joint attention (Tomasello, 1995). Gaze following allows access to the attentional focus of others, which is essential for understanding others' goals, intentions, as well as mental states. Paradigms used to test gaze following and joint attention typically present static images of faces or eyes with direct gaze or averted gaze. For example, in traditional gaze-following tasks, participants are asked to saccade to a peripheral target, with an image of a conspecific's eyes in the center displaying an averted gaze toward the target location greater than 70% of the time (congruent) or toward the opposite direction less than 30% of the time (incongruent). This paradigm allows researchers to use saccade latency to index gaze-following performance (shorter in congruent and longer in incongruent trials). In typical joint attention tasks, participants are often presented with an image of eyes in the center of a screen and need to saccade to one of two peripheral targets, with the correct choice indicated by the gaze direction depicted in the image. It has been found that the amygdala activations modulated visual areas involved in gaze detection, such as the superior temporal sulcus (STS), the temporo-parietal junction (TPJ), as well as the medial prefrontal cortex (mPFC) (Senju and Johnson, 2009). Similarly, using static facial images with different gaze angles, a recent work identified a novel gaze-following patch in the macaque posterior STS (Ramezanpour and Thier, 2020).

Another important element for successfully navigating our social world is the ability to understand others' actions. Notably, researchers found that even a brief display (73 ms) of naturalistic photographs depicting simple actions between two individuals, such as kicking or pushing, could induce conceptual representations of these social actions (Hafri et al., 2013) and activate the mirror neuron network including the premotor cortex and intraparietal sulcus (IPS) (Hafri et al., 2017). Moreover, when subjects were asked to evaluate complex social scenes with different valence and ambiguity, blood-oxygen-level-dependent (BOLD) signals tended to increase in the mentalizing network, including the TPJ, STS, precuneus, and mPFC, compared to when evaluating non-social images (Deuse et al., 2016).

A series of sequential images have also been successfully used to examine social decision-making. When a participant was shown a series of pictures depicting a simplified poker game with an opponent, neural signals in the TPJ uniquely predicted the opponent's upcoming decisions when playing with a social player but not a computer (Carter et al., 2012). Moreover, when humans observed a series of images showing options, choices, and outcomes related to confederate's decision, participants learned better from viewing both the action and outcome of the confederate compared to only viewing other's action or viewing neither, and neural activity in the ventral striatum and the ventromedial prefrontal cortex (vmPFC) represented the prediction errors of outcomes of self and other, respectively (Burke et al., 2010).

Overall, static images are still the most commonly used stimulus type in social neuroscience research with excellent control and manipulability. They are able to provide key information about social agents, and studies using images have facilitated our understanding of some basic functions of the brain during social interaction, such as face and gaze processing, action understanding, and simple decision-making. However, static images are simplistic and are less effective or unable to convey realistic dynamic information unfolding over time, one of the key components contributing to real-life social interaction.

Level 2: Dynamic social stimuli

Social cues in the environment change over time. To capture dynamic social information, many studies have used pre-recorded video clips. Both humans and non-human primates are usually more engaged when viewing videos of social agents and social scenes compared to viewing static images, and they frequently display different behaviors and neural responses between these stimulus types (Arsalidou et al., 2011; Furl et al., 2012; Gunhold et al., 2014a; Dal Monte et al., 2016; Gothard et al., 2018). Figure 2 (right column) illustrates some example paradigms using Level 2 social stimuli.

Similar to static images, video clips have been used widely to examine face processing. Viewing dynamic faces with different expressions has been found to result in better recognition of emotions and higher ratings of faces as emotional and further evokes greater facial muscle activity in observers, compared to viewing static facial images (Rymarczyk et al., 2016; Richoz et al., 2018). While images and videos of faces in general elicit similar responses in the core visual areas, dynamic stimuli with emotional expressions tend to evoke greater BOLD signals in the STS, middle temporal gyrus, fusiform gyrus, and amygdala compared to static stimuli (LaBar et al., 2003; Arsalidou et al., 2011; Pitcher et al., 2011). Similarly, STS areas in rhesus macaques have also been shown to be more activated by dynamic faces, an effect consistent for threat, neutral, and submissive expressions (Furl et al., 2012). In other words, although static stimuli might be sufficient to study the core visual regions, dynamic faces tend to activate extended face processing networks that are critical for contextualizing face perception. Furthermore, videos can be paired with auditory stimuli to provide multisensory information to better mimic real-life social interaction. A recent study presenting videos of monkeys together with vocalization revealed that the auditory component modulated face-selective single-neuron activity in a specific face patch (Khandhadia et al., 2021), supporting the idea that social stimuli with unisensory versus multisensory information engage the brain in different ways.

Additionally, video clips have been used to demonstrate remarkably similar social gaze behaviors between humans and long-tailed macaques (Macaca fascicularis), driven by a shared preference for looking at faces and eyes (Shepherd et al., 2010). Unlike static images, videos of conspecifics are able to elicit spontaneous gaze aversion and gaze-following behaviors, and mutual eye contact depicted in videos can evoke facial expressions in monkeys (Mosher et al., 2011). Further, when monkeys watched videos of conspecifics with direct face and gaze (i.e., second person view as if they were engaged in interaction), the mPFC, including the rostral anterior cingulate cortex (ACC), was specifically recruited compared to viewing conspecifics with averted face and gaze (i.e., third person view as if they watched conspecifics in the video communicating with someone else) (Shepherd and Freiwald, 2018).

Video clips are particularly useful for providing certain perspectives of actions, such as the dynamics and intention of a given action, as well as the interactive relationship among agents. Such social information is challenging or ambiguous to convey through static images that lack temporal dimension. Researchers have found that watching videos of repeated actions performed by others suppressed BOLD signals in the anterior IPS regardless of action trajectory (Hamilton and Grafton, 2006), suggesting a role of this area in understanding the intention behind an action. Moreover, salient video clips have been used to assess a causal contribution of the mPFC in anticipatory looking behaviors in relation to a process potentially linked to the theory of mind in monkeys (Hayashi et al., 2020). Further, studies using video stimuli have revealed multiple human brain areas involved in processing interactive actions among agents compared to non-interactive actions. For example, interactive point-light displays evoked greater activity in both the mirror neuron system and the mentalizing system (Centelles et al., 2011; Oberman et al., 2007). Similarly, when monkeys watched videos of conspecifics grooming or mounting, many neurons in the ventral lateral prefrontal cortex (vlPFC) showed distinct activity compared to when monkeys viewed videos of non-interacting monkeys (Tsunada and Sawaguchi, 2012). Notably, a whole-brain fMRI study in rhesus monkeys further identified a dedicated network for processing social interactions, including the mPFC, vlPFC, orbitofrontal cortex, and interior parietal lobule, that showed enhanced signals exclusively for watching videos of interactive conspecifics (Sliwa and Freiwald, 2017).

Compared to static images, video clips usually provide more critical social information for animals to acquire learning from conspecifics. For example, when marmosets (Callithrix jacchus) watched videos or images of unfamiliar conspecifics opening a tool, only those who viewed videos spent more time manipulating it, were more likely to succeed, and also more likely to apply the strategy they had learned (Gunhold et al., 2014a). Chimpanzees (Hopper et al., 2012) and colobus monkeys (Colobus guereza kikuyuensis) (Price and Caldwell, 2007) are also capable to learn new tasks from watching videos of conspecifics performing it. Similar experiments have been conducted in birds, but the results have been inconsistent. While some studies suggested that certain species of birds such as zebra finches (Taeniopygia guttata, Galoch and Bischof, 2007) and nutmeg mannikins (Lonchura punctulate, Rieucau and Giraldeau, 2009) could understand and react to video clips of conspecifics, other studies in blue tits (Cyanistes caeruleus, Hämäläinen et al., 2017) or California scrub jays (Aphelocoma californica, Brecht et al., 2018) demonstrated that these birds did not seem to use information in the videos for their own decisions.

In summary, video playbacks are similarly easy to generate and manipulate as static images. As discussed above, compared to images, videos are able to more effectively convey social information to elicit emotional and gaze responses (Mosher et al., 2011; Rymarczyk et al., 2016; Richoz et al., 2018) and facilitate action understanding and social learning (Price and Caldwell, 2007; Hopper et al., 2012; Gunhold et al., 2014a). More importantly, video stimuli tend to recruit a subset of different brain areas compared to image stimuli (LaBar et al., 2003; Arsalidou et al., 2011; Pitcher et al., 2011; Furl et al., 2012). Furthermore, videos paired with acoustic stimuli provide multisensory information that could enhance behaviors (Galoch and Bischof, 2007) and modulate neural activity (Khandhadia et al., 2021).

Level 3: Dynamic and contingent social stimuli

While pre-recorded videos are able to provide social information unfolding over time, they cannot effectively convey contingent behaviors among social agents. To overcome this limitation, researchers have used real-time videos of real agents and real-time animations of virtual avatars controlled by either real agents or by computer-based algorithms. This type of stimuli not only allows a high degree of experimental control but also provides contingent social information that images and pre-recorded videos fail to convey. Figure 3 illustrates some example paradigms using Level 3 social stimuli.

Figure 3

Examples paradigms using social stimuli with naturalism Level 3

Shown are selected stimuli that convey dynamic and contingent social information, such as real-time videos of real agents and real-time animations of virtual avatars.

(A–G) (A) Redcay et al., 2012; (B) Chu et al., 2020; (C) Murphy and Leopold, 2019; (D) Dumas et al., 2010; (E) Bevacqua et al., 2017; (F) Slater et al., 2013; (G) Cheetham et al., 2009.

Reproduced with permissions or under Creative Commons Attribution CC BY License (https://creativecommons.org/licenses/by/4.0/) and Creative Commons Attribution-NonCommercial CC BY-NC License (https://creativecommons.org/licenses/by-nc/4.0/).

Real-time videos of real agents are able to capture contingent social gaze behaviors. By using a dual-video setup that showed the face and gaze of an experimenter to a participant in a fMRI scanner in real time, researchers found that the vmPFC was selectively responsive to joint attention initiated by others, while the IPS and middle frontal gyrus were activated by joint attention initiated by self (Redcay et al., 2012). Real-time videos have also been applied to study social actions. Researchers used synchronized cameras to record the hand movements of pairs of participants and found region- and frequency-specific inter-brain synchronization in their EEG signals during coordinated actions (Dumas et al., 2010).

Similarly, virtual avatars controlled by real agents or computer algorithms are also able to display contingent behaviors and can be more easily embedded in complex social scenes compared to videos of real agents. For example, paired with real-time eye tracking, virtual characters have been used to study contingent gaze behaviors (Wilms et al., 2010). Impressively, a study showed that advanced computer algorithms were able to evoke similar subjective perception in participants based on their interactions with a virtual agent as when the virtual agent was controlled by an operator in a different room, suggesting the feasibility and reliability of virtual stimuli in studying contingent actions (Bevacqua et al., 2017). Although it is challenging to generate realistic avatars with contingent behaviors in complex social scenarios, some successful attempts have been made to study bystander effect (Slater et al., 2013) and obedience to authority in a virtual Milgram paradigm (Cheetham et al., 2009). Future research should focus on increasing the complexity of closed-loop contingent behaviors of virtual avatars.

Computer-generated 3D models have been recently applied in non-human primates as well. Based on anatomical scans, researchers have been able to generate natural-looking monkey avatars. When tested, both long-tailed and rhesus macaques displayed similar looking behaviors to monkey avatars as to images of real monkeys (Wilson et al., 2020). Notably, researchers can parametrically control various features, such as an avatar's head orientation, gaze direction, facial expression, gender, and age (Murphy and Leopold, 2019). Such stimuli can be further modified into animations to convey dynamic information (Murphy and Leopold, 2019; Siebert et al., 2020; Taubert et al., 2020) that can be paired with auditory stimuli (Khandhadia et al., 2021). All these techniques make it possible for future studies to design animal avatars that behave contingently in certain ways with real subjects during social interaction.

Virtual reality (VR) is an exciting novel tool with great potential for the field of social neuroscience (Cross et al., 2019). In order to elicit typical social behaviors, concerted efforts from multiple disciplines are necessary to create the most ideal avatars with naturalistic and engaging appearance, synchronous multisensory information, and realistic contingent behaviors. Failure to do so will likely result in an uncanny valley effect, first coined by Masahiro Mori, that describes the phenomenon that human-like robots could lead to unusual responses possibly due to their realistic appearance but abnormal behaviors (Mori, 1970). Although some studies have observed an uncanny valley effect in humans and monkeys (Saygin et al., 2012; Siebert et al., 2020; Steckenfinger and Ghazanfar, 2009), they do not necessarily negate the validity of using avatars as potential social stimuli. The dip of the uncanny valley curve depends on the stimuli and may be related to ambiguous categorization, less realistic appearance, or asynchronous emotional, motor, and audio cues (Burleigh et al., 2013; Cheetham et al., 2015; Skiba and Vuilleumier, 2020). For example, in macaques, the most naturalistic avatars have been found to elicit similar facial expressions as videos of real monkeys (Siebert et al., 2020). Excitingly, different labs across the world are developing real-time hyper-realistic human avatars. Researchers have been able to simultaneously record gaze and mouth movement by using cameras on a VR headset and render real-time animations (Lombardi et al., 2018; Chu et al., 2020). Accurate real-time avatars can even be generated by only using eye gaze recorded from a frontal view together with audio recording (Richard et al., 2020). Taken together, natural-looking virtual avatars with synchronous and contingent behaviors are state-of-the-art techniques that have high potential in the future research of social neuroscience.

In summary, real-time videos are able to capture the contingency of real-life social behaviors such as social gaze interaction (Redcay et al., 2012) and action coordination (Dumas et al., 2010) that static images and pre-recorded videos fail to convey. To serve similar functions, virtual avatars controlled by real agents or computer algorithms have also become highly effective social stimuli (Wilms et al., 2010; Bevacqua et al., 2017; Murphy and Leopold, 2019; Siebert et al., 2020; Taubert et al., 2020; Wilson et al., 2020). They not only allow parametric control over specific features,but also can be programmed to display contingent behaviors and reactions. Hyper-realistic avatars with natural looking, synchronous multi-modal cues, and contingent behaviors may be able to overcome uncanny valley effect and resemble realistic agents during naturalistic social interaction.

Level 4: Real-life social stimuli in lab environments

While real-time videos and virtual characters are able to simulate dynamic and contingent social behaviors, the lack of real-life interaction may fail to induce the same behavioral or neural states as in real scenarios. The field of social neuroscience has been moving toward applying a two-person experimental approach (Gallotti and Frith, 2013; Schilbach et al., 2013; Isoda et al., 2018; Redcay and Schilbach, 2019). Recruiting pairs of participants or animals in the laboratory has the potential to capture naturalistic and interactive social behaviors. Such paradigms, albeit still arbitrary compared to real-world settings, can offer unique avenues to examine inter-brain synchrony. Studies applying simultaneous neuronal recording and hyper-scanning have revealed specialized inter-brain coherence during social interaction (Hasson et al., 2012; Liu and Pelowski, 2014; Wang et al., 2018; Czeszumski et al., 2020). Figure 4 illustrates some example paradigms using Level 4 social stimuli.

Figure 4

Example paradigms using social stimuli with naturalism Level 4

Shown are selected real-life paradigms in laboratory settings involving pairs of individuals engaged in social interaction.

(A–M) (A) Hayward et al., 2017; (B) Dal Monte et al., 2016; (C) Kingsbury et al., 2020; (D) Ferretti et al., 2019; (E) Fujii et al., 2007; (F) Isoda et al., 2018; (G) Noritake et al., 2018; (H) Chang et al., 2013; (I) Grabenhorst et al., 2019; (J) Haroush and Williams, 2015; (K) Lee et al., 2012; (L) Ferrari-Toniolo et al., 2019; (M) Kingsbury et al., 2019.

Reproduced with permissions or under Creative Commons Attribution CC BY License (https://creativecommons.org/licenses/by/4.0/).

Real-life interactions likely evoke distinct communication goals and behaviors under specific behavioral and neural states, compared to interactions with artificial stimuli. For example, by comparing social attention during a natural conversation to a controlled gaze-cuing task, researchers found little correlation of gaze behaviors between the two conditions (Hayward et al., 2017). In addition, participants showed distinct neural activity when they viewed live faces with direct gaze, compared to averted gaze or closed eyes, and this effect was absent when viewing facial images (Pönkänen et al., 2011). Similarly, when comparing social gaze behaviors across different stimulus contexts, researchers found that rhesus macaques displayed distinct gaze behaviors when interacting with a real-life conspecific compared to looking at the same monkey in images or videos (Dal Monte et al., 2016). Monkeys tended to fixate more to the eyes with smaller dispersion around the eyes of a real partner compared to the other two conditions. Moreover, social gaze dynamics following a mutual eye contact were modulated by dominance and familiarity only in the live social condition (Dal Monte et al., 2016). These studies suggest that the neural mechanisms might differ when viewing gaze of real agents versus gaze depicted in images and emphasize the importance of applying real-life paradigms to examine social gaze interaction.

Face-to-face interactions also facilitate individuals to obtain specific social information about real-life social agent, such as their hierarchical rank, sex, and affective state. During a food-grab task where pairs of monkeys needed to reach a piece of reward located in either a shared or non-shared space, neural activity in many neurons in the lateral prefrontal cortex (lPFC) reflected a monkey's current social status such that lPFC activity was increased in dominant monkeys but suppressed in subordinate ones (Fujii et al., 2009). Interestingly, when a previously subordinate monkey was paired with a more subordinate new partner, activity in lPFC neurons switched to a dominant mode of modulation, suggesting that such dominance modulation is context dependent (Fujii et al., 2009). In fact, real-life interaction paradigms have been commonly applied in rodents to study their social behaviors. By examining social interactions in freely moving mice, researchers found that neurons in the dorsomedial prefrontal cortex (dmPFC) showed distinct activity for interacting with male versus female mice that was causally linked to establishing preference (Kingsbury et al., 2020). Moreover, several studies have recently focused on investigating the responses of mice to different affective states expressed by conspecifics. A newly developed real-life paradigm has allowed researchers to induce different affective states in rodents, such as fear by conditioned tone, stress by restraint tube test, pain by shock, and relief by giving water after water deprivation, and test how conspecifics perceive and react to the affective states of others (Carrillo et al., 2019; Ferretti et al., 2019; Scheggia et al., 2020). Although there were some constraints imposed on the rodents' behaviors, these studies reveal that rodents could distinguish others' emotional states and react to them. Indeed, emotional mirror neurons have been detected in the ACC whose activity was modulated similarly when a rat was in pain as when an observed conspecific was in pain (Carrillo et al., 2019). Similarly, when mice were placed close to conspecifics with specific induced emotional states, they were able to differentiate the expression of emotional states in others, which was related to oxytocin signaling in the central amygdala as well as somatostatin interneurons in the mPFC (Ferretti et al., 2019; Scheggia et al., 2020).

Real-life paradigms in lab environments are also well suited to study social actions. By using the food-grab task mentioned before, researchers found that IPS neurons mainly responded to self-action when the pair of monkeys could not interact to grab a reward but showed distinct and rather complex neural responses when they both could reach the reward located in a common space (Fujii et al., 2007). By involving pairs of monkeys facing each other and performing a role reversal task, researchers revealed that neurons in the dorsal medial frontal cortex, including the pre-supplementary motor area, exhibited agent-specific representations of actions performed by either self or other (Yoshida et al., 2011; Isoda et al., 2018) as well as errors made by self or other (Yoshida et al., 2012). Furthermore, when bats learned and followed a demonstrator's flight trajectory through observation, signals in their other-referenced place cells in the hippocampus CA1 represented the location of the demonstrator in an allocentric fashion, and such signals were specific to social stimuli as these cells responded differently to the movement of inanimate objects (Omer et al., 2018).

In real-life social interactions, it is especially important for individuals to track what happens to self and other. In an experiment requiring pairs of monkeys to take turns to touch stimuli associated with or without reward for self or a partner monkey, neurons in the striatum signaled reward inequity between the two monkeys (Báez-Mendoza et al., 2016). In a social Pavlovian conditioning task using cues associated with different reward probabilities for self and other, monkeys' anticipatory licking behavior was negatively correlated with other's reward probability even though the reward probability for self was fixed (Noritake et al., 2018). Notably, when the partner monkey was absent (or present but was unable to get reward), modulation of licking behaviors as well as neural activity in dmPFC and midbrain dopaminergic neurons related to social reward valuation was reduced (Noritake et al., 2018). Moreover, when monkeys were presented with different cues associated with reward to self alone or to both self and other, they were more willing to work for reward to self alone, suggesting that concurrent reward received by others devalued one's own reward (Azzi et al., 2012). However, such reward devaluation was specific to real-life social condition, as replacing the partner with an empty chair did not elicit the same effect (Azzi et al., 2012). Additionally, while monkeys prefer drinking alone over sharing, a series of studies found that monkeys displayed a consistent prosocial preference for donating juice reward to a conspecific over wasting it (Chang et al., 2011). Similarly, when the partner monkey was replaced by a bottle, the actor monkey no longer showed such prosocial preference (Chang et al., 2011). Subsequent neuronal recording studies found that such prosocial behavior was associated with other-referenced reward neurons in the gyrus of the ACC (ACCg) (Chang et al., 2013) and increased interareal coherence between the ACCg and the basolateral amygdala (BLA) (Dal Monte et al., 2020). Causally, lesions of the ACC have been recently shown to reduce learning from rewards delivered to conspecifics, preventing monkeys to acquire the preference for delivering juice to a conspecific over neither when learning from a new set of cues (Basile et al., 2020), supporting the role of the ACC in processing vicarious reward. Prosocial behaviors have been also observed in rats during real-life social interactions. Rats preferred sharing rewards with another rat but not with a toy rat, and this preference was dependent on the BLA (Hernandez-Lallement et al., 2016). Animals can also learn from real-life social partners based on reward or punishment. For example, when pairs of monkeys took turns to make reward-based choices, they were able to learn from observation and neurons in the amygdala derived learned value to predict others' choices (Grabenhorst et al., 2019). In addition, during an observational fear learning task, mice were able to learn the association between a cue and a shock delivered to a conspecific, and this learning was orchestrated by a population of ACC neurons projecting to the BLA (Allsop et al., 2018).

Studies using real-life interaction paradigms have also helped further examine nuanced social behaviors and the underlying neural bases in non-human animals. When vampire bats (Desmodus rotundus) are fasted, they usually receive food from others by regurgitation, which happen naturally despite high energy cost. This food-sharing behavior has been found correlated with mutual allo-grooming that may facilitate social bonding (Carter and Wilkinson, 2013). In addition, by using a real-life interaction paradigm, researchers found that both male and female rats avoided actions that led to harming conspecifics, and the ACC causally contributed to this harm aversion behavior (Hernandez-Lallement et al., 2020). Crucially, when victim rats randomly received the same number of shocks, actor rats did not show harm aversion, suggesting the importance of behavioral contingency (Hernandez-Lallement et al., 2020). In the same vein, researchers found that consoling behaviors in prairie voles (Microtus ochrogaster) displayed toward a familiar conspecific in distress depended on oxytocin processing in the ACC (Burkett et al., 2016).

Real-life paradigms are also ideal for studying collective social decisions that require interactive inputs. When monkeys played a competitive video shooting game, they were more motivated to compete with another monkey compared to a computer (Hosokawa and Watanabe, 2012). In this study, researchers also included a two-monkey non-competitive condition where the partner monkey was present, but the game was controlled by a computer. In this condition, the actor monkey behaved similarly as in competing with a computer, suggesting that the mere presence of a social partner without contingent behavior was not sufficient to elicit the same level of competition (Hosokawa and Watanabe, 2012). Furthermore, by using an iterative prisoner's dilemma game in monkeys, researchers found that a distinct group of neurons in the ACC predictively signaled partner's decisions to cooperate or defect based on interaction history (Haroush and Williams, 2015). Crucially, replacing the monkey partner with a computer opponent or placing the partner in a separate room reduced cooperative choices as well as the proportion of ACC neurons predicting partner's future decisions, again emphasizing the importance of interacting with a real social agent (Haroush and Williams, 2015).

As discussed above, real-life paradigms in the laboratory offer a unique opportunity to study face-to-face social interactions among real social agents in a well-controlled way. Furthermore, real-life paradigms provide a critical avenue to study two or multiple brains at the same time (Hasson et al., 2012; Liu and Pelowski, 2014; Wang et al., 2018; Czeszumski et al., 2020). In humans, studies using simultaneous recording via fNIRS and EEG have revealed inter-brain synchrony in relation to social gaze. For example, a study using fNIRS hyper-scanning showed that real-time mutual eye contact evoked greater inter-brain coherence in the frontal-temporal areas, compared to fixating to the eyes of a face in images (Hirsch et al., 2017). A simultaneous EEG recording study revealed temporal-parietal synchrony among romantic couples, but not strangers, during free conversation in relation to social gaze and positive affect (Kinreich et al., 2017). Similarly, direct gaze was shown to strengthen adult-infant directional neural synchronization indicated by EEG signals in both live and image conditions. However, infant-adult functional connectivity was observed only in the live condition (Leong et al., 2017). Further, mother-infant inter-brain coherence was shown to be modulated by emotional valence of the mother (Santamaria et al., 2020). Hyper-scanning has also been applied to study physical synchrony and more complex social behaviors in humans. When two people were asked to perform a cooperative task to press a button in synchrony, fNIRS activations showed greater inter-brain coherence in the prefrontal cortices for more synchronized performance (Funane et al., 2011). Further, when participants played a simplified poker game, researchers found an increase in inter-brain synchrony measured by fNIRS in the frontal-parietal network, including the TPJ and the dorsolateral prefrontal cortex (Piva et al., 2017).

Compared to fNIRS and EEG, fMRI has been applied less in face-to-face social interactions, mainly due to the spatial restraint of the scanner. However, attempts have been made to examine face-to-face interactions in the scanner. For example, by having pairs of participants lying in two scanners and instructing them to engage in real-time joint attention task, researchers showed inter-brain synchrony in participants' right inferior frontal gyrus, compared to non-paired participants (Saito et al., 2010). Researchers have also developed a dual-head coil pair that allows simultaneous scanning of two subjects in the same scanner (Lee et al., 2012) to study social gaze (Lee, 2015) and simple actions (Renvall et al., 2020).

Finally, simultaneous recording approaches have been applied in non-human animals. When two monkeys moved cursors to reach a common goal, neuronal activity from interacting animals could be used to decode the motion trajectory of their joint action, and such representations were reduced when monkeys played with a computer that was neither interactive nor cooperative (Ferrari-Toniolo et al., 2019). Furthermore, when pairs of bats (Rousettus aegyptiacus) were involved in natural social interactions, researchers found inter-brain coherence in a relatively high frequency band in the frontal cortices (Zhang and Yartsev, 2019). Similarly, simultaneous microendoscopic calcium imaging in pairs of mice revealed inter-brain coherence in the dmPFC during natural social interactions that was functionally related to social dominance behaviors (Kingsbury et al., 2019).

In summary, real-life paradigms in the laboratory involving multiple agents can capture naturalistic face-to-face social interactions that are much more ecologically valid than using artificial stimuli. Notably, studies using real-life paradigms have revealed distinct social behaviors as well as specific neural activity that are not observed when the social partners are absent or display non-contingent behaviors (Chang et al., 2011; Azzi et al., 2012; Hosokawa and Watanabe, 2012; Haroush and Williams, 2015; Noritake et al., 2018) or when the social partners are portrayed in images or videos (Pönkänen et al., 2011; Dal Monte et al., 2016; Hayward et al., 2017). While such paradigms often require more extensive data analyses to control for unconstrained behaviors of more than one agent, they also provide unique opportunities to study multiple brains simultaneously.

Level 5: Real-life social stimuli in natural environments

More recently, researchers have been working toward studying social behaviors and their neural underpinnings directly in natural environments. Such endeavors require advanced equipment to record behaviors and neural activity from multiple individuals simultaneously, as well as more complex data analytic tools. In humans, such attempts have been made to investigate inter-brain coherence among students and a teacher in classrooms and among musicians in ensembles. For non-human primate research, Cayo Santiago, an island in Puerto Rico with a large population of free-ranging rhesus macaques living in the natural environment, has provided a unique opportunity to study natural social behaviors. Similarly, parks and zoos allow naturalistic social interactions among animals. Figure 5 illustrates some example paradigms using Level 5 social stimuli.

Figure 5

Example paradigms using social stimuli with naturalism Level 5

Shown are selected real-life paradigms involving multiple individuals in semi-natural and natural environments, as well as paradigms using wireless recording systems.

(A–I) (A) Dikker et al., 2021; (B) Dikker et al., 2017; (C) D’Ausilio et al., 2015; (D) Babiloni et al., 2011; (E) Bigand et al., 2010; (F) Brendan Borrell; (G) Gunhold et al., 2014b; (H) Weissbrod et al., 2013; (I) Roy and Wang, 2012.

Reproduced with permissions.

Many studies have been conducted in the real-world environment. Beyond traditional eye tracking in the laboratory, researchers examined gaze patterns when participants naturally walked on street and compared their gaze behaviors to when participants watched first-person videos of their walk in the laboratory (Foulsham et al., 2011). While many things tended to capture people's attention similarly in both conditions, there were differences in how people allocated gaze spatially across conditions. In an impressive field study, researchers performed simultaneous EEG recordings from hundreds of pairs of museum visitors and found that inter-brain coupling tended to be modulated by people's social traits, social relationship, closeness, as well as specific social behaviors such as mutual eye contact and joint action (Dikker et al., 2021). Other existing studies have revealed inter-brain synchrony during teacher-learner interactions. For example, simultaneous fNIRS recordings between pairs of experienced instructor and non-expert learner revealed inter-brain coupling in the prefrontal areas during conceptual learning, specifically when the teacher used a scaffolding method such as asking guiding questions compared to the explanation method by providing definitions (Pan et al., 2020). In real classrooms, simultaneous recordings from 12 high school students over multiple lectures using portable EEG showed that student-to-group synchrony was able to predict class engagement as well as social dynamics (group affinity, empathy, and social closeness with the teacher) (Dikker et al., 2017). In this study, pairs of students who directly interacted before class showed greater inter-brain synchrony during the class, compared to other pairs (Dikker et al., 2017). However, it still remains unclear how student-teacher inter-brain coherence is related to learning outcomes. While a similar study (Bevilacqua et al., 2019) in a real-world classroom revealed a correlation between memory retention and student-teacher closeness, it did not find a relationship between retention and student-teacher inter-brain activity. By contrast, a study in a laboratory classroom setting found that student-teacher synchrony in the alpha frequency band, implicated in attention, could predict learning outcomes (Davidesco et al., 2019).

Real-life social interactions have also been studied in music groups, such as small ensembles like quartets that function as self-managed teams and big ensembles like orchestras with a conductor as the leader with many musicians as the followers (D'Ausilio et al., 2015). By studying a string quartet, researchers detected specific features about musicians' head movement and direction that were different across coordinated and perturbed situations (Glowinski et al., 2013), suggesting that monitoring co-players is critical for coherent performance. In another study, simultaneous EEG recordings from four saxophonists revealed frequency- and area-specific inter-brain synchrony across resting and performing states (Babiloni et al., 2011). Furthermore, a study in a real orchestra revealed that the conductor looked at the musical score much more than at the performers, with long anticipatory looks at the notes mostly during expressive parts of a music piece (Bigand et al., 2010). Overall, classrooms and music ensembles provide novel and rich environments for studying social interaction and coordination with high ecological validity that might be qualitatively different from what one can measure in the laboratory settings. Examination of human neural activity and inter-brain synchrony in natural environments will continue to require technical innovations.

Researchers have long been studying social interactions of non-human animals in the natural environments. For example, by observing groups of wild marmosets learning to solve a task with or without a real demonstrator, researchers showed that monkeys were able to learn from a skilled demonstrator (Gunhold et al., 2014b). Similarly, a study in chacma baboons (Papio ursinus) in a park revealed that personality modulated social learning such that bolder and more anxious baboons learned better (Carter et al., 2014). In Cayo Santiago, researchers have access to large groups of monkeys' identity, kinship, social relationship, and genetic information to better characterize their naturalistic social interaction. For example, data spanning 21 years with hundreds of female adult macaques showed that prime-aged female macaques with more close adult female relatives were more likely to survive (Brent et al., 2017). Similarly, another long-term study on more than 300 female monkeys revealed that females with strong connections with favored partners were more likely to survive (Ellis et al., 2019). Studies conducted in Cayo Santiago have also extensively examined theory of mind in monkeys (Drayton and Santos, 2016) and genetic variations related to social behaviors, such as aggression and affiliation (Charpentier et al., 2008).

In rodent research, semi-natural setups have helped to bridge the gap between classical laboratory experiments and field studies in the wild. These semi-natural environments are usually full of enrichments and contain a group of mice whose locations can be tracked over days (Shemesh et al., 2013; Weissbrod et al., 2013; Anpilov et al., 2020). This type of paradigms can be easily applied to wild rodents (Schweinfurth, 2020) and combined with wireless systems to study naturalistic social interaction. Researchers have developed wireless single-neuron recording systems (Fan et al., 2011; Hasegawa et al., 2015), wireless optogenetic devices (Anpilov et al., 2020), and even bidirectional devices for simultaneous recording and stimulation in rats (Melo-Thomas et al., 2017). Similar wireless neuronal recording systems have been applied in non-human primates, such as marmosets and rhesus macaques (Roy and Wang, 2012; Schwarz et al., 2014; Berger et al., 2020), as well as bats (Kothari et al., 2018; Omer et al., 2018). Although such wireless recording techniques have not yet been applied frequently in social neuroscience, they have opened new avenues for researchers to study social interaction in semi-natural or natural settings.

Compared to laboratory experiments, studies conducted in the natural environments can provide unique insights into naturalistic social behaviors that arguably cannot be truly captured otherwise. Classrooms and music ensembles offer exceptional opportunities to examine social interactions in the real world (Babiloni et al., 2011; Dikker et al., 2017; Pan et al., 2020). Non-human primate research conducted on field sites or in zoos captures natural dynamics among large population of animals and enables the investigation of their social dynamics, social network, and genetic underpinnings (Carter et al., 2014; Gunhold et al., 2014b; Brent et al., 2017; Ellis et al., 2019). As neural data acquisition and manipulation technologies advance (Roy and Wang, 2012; Hasegawa et al., 2015; Melo-Thomas et al., 2017; Omer et al., 2018; Anpilov et al., 2020), studies in real-life natural environments will likely provide an unprecedented database for social neuroscience.

Concluding remarks

The field of social neuroscience aims to understand complex social behaviors among individuals and elucidate the underlying neural mechanisms. In this review, we classified social stimuli commonly used in the field into five levels of naturalism. Different levels of stimuli usually evoke distinct social behaviors and associated neural activity, prompting the need to examine if our knowledge about the brain can be generalized across different experimental settings and different levels of stimulus complexity. We encourage researchers to adopt stimuli with a higher level of naturalism, such as Level 4 and Level 5, to better capture the dynamic and contingent nature of social interaction.

As an excellent exemplary case, face and gaze processing has been greatly investigated by researchers using different types of social stimuli. Traditional approaches using static stimuli (Level 1) have revealed important neural networks, such as the core visual areas as well as extended face-selective areas. Nevertheless, it is critical to note that by adding dynamic information, video stimuli (Level 2) can already elicit different social behaviors and neural activity during face and gaze processing (Mosher et al., 2011; Arsalidou et al., 2011; Rymarczyk et al., 2016; Richoz et al., 2018). However, social agents in pre-recorded videos cannot show contingent behaviors and reactions. By using real-time videos or applying state-of-the-art VR techniques (Level 3), researchers have been able to effectively study contingent social gaze behaviors (Wilms et al., 2010; Redcay et al., 2012). Taking a big leap from presenting artificial stimuli, real-life interaction paradigms (Level 4) provide unique opportunities to study realistic social behaviors among real interacting agents and can reveal specific gaze behaviors and neural activity that are potentially different from those elicited by images or videos (Pönkänen et al., 2011; Dal Monte et al., 2016). Moreover, inter-brain synchrony also tends to be different during live social gaze interaction versus when looking at artificial stimuli (Hirsch et al., 2017; Leong et al., 2017). Such real-life paradigms can be further extended to real-world settings (Level 5). With advanced hyper-scanning techniques and analytic methods, researchers have been able to record neural activity simultaneously from pairs of museum visitors and showed that inter-brain coupling tends to be modulated by many specific social factors and behaviors, such as mutual eye contact (Dikker et al., 2021).

How the brain processes social information during naturalistic social interaction is not likely to be fully predicted by how it processes social information in an artificial and controlled laboratory setting. Traditional social stimuli have their own strengths, as well-controlled stimuli are ideal for testing specific hypotheses and can facilitate our understanding of basic brain functions that are surely likely engaged during more complex social interaction. However, to better capture the dynamic and contingent nature of social interaction, we conclude that, moving forward, researchers in social neuroscience would benefit from adopting social stimuli with a higher level of naturalism by taking advantage of advanced techniques and data analytics.