Live agent preference and social action monitoring in the macaque mid-superior temporal sulcus region.

Mentalizing, the ability to infer the mental states of others, is a cornerstone of adaptive social intelligence. While functional brain mapping of human mentalizing has progressed considerably, its evolutionary signature in nonhuman primates remains debated. The discovery that the middle part of the macaque superior temporal sulcus (mid-STS) region has a connectional fingerprint most similar to the human temporoparietal junction (TPJ)-a crucial node in the mentalizing network-raises the possibility that these cortical areas may also share basic functional properties associated with mentalizing. Here, we show that this is the case in aspects of a preference for live social interactions and in a theoretical framework of predictive coding. Macaque monkeys were trained to perform a turn-taking choice task with another real monkey partner sitting directly face-to-face or a filmed partner appearing in prerecorded videos. We found that about three-fourths of task-related mid-STS neurons exhibited agent-dependent activity, most responding selectively or preferentially to the partner's action. At the population level, activities of these partner-type neurons were significantly greater under live-partner compared to video-recorded-partner task conditions. Furthermore, a subset of the partner-type neurons responded proactively when predictions about the partner's action were violated. This prediction error coding was specific to the action domain; almost none of the neurons signaled error in the prediction of reward. The present findings highlight unique roles of the macaque mid-STS at the single-neuron level and further delineate its functional parallels with the human TPJ in social cognitive processes associated with mentalizing.

Humans have a remarkable ability to infer the mental states of others by observing their overt behavior. This social cognitive function, referred to as mentalizing or theory of mind, is associated with activity in a specific set of brain areas including the temporoparietal junction (TPJ) at the cortical level (1–3). Although functional brain mapping of mentalizing has been carried out extensively in humans, often using verbal false-belief tasks, little is known about whether the same brain areas in nonhuman primates also have a mentalizing function. This is mainly because nonverbal tasks suitable to assess mentalizing function in nonhuman primates have been difficult to develop. However, a recent study employing anticipatory looking paradigms demonstrates that, albeit still hotly debated (4), even macaque monkeys can exhibit false-belief, attribution–like behavior (5), pointing to the possibility that the capacity to mentalize exists in the macaque, at least in a rudimentary form.

Rushworth and coworkers have taken a unique approach to address this issue using functional brain imaging. They looked at structure, rather than function, to identify areas in the macaque brain that are similar to those in the human mentalizing network in terms of patterns of connectivity with the rest of the brain (6, 7). This approach revealed that a region in the dorsal bank and fundus of the macaque middle superior temporal sulcus (hereafter referred to as the mid-STS) has a connectional “fingerprint” most similar to the human TPJ (6), suggesting their comparable roles in social cognition. Consistent with this view, single neurons in the macaque mid-STS, also known as the superior temporal polysensory area, respond to various body parts including faces (8, 9). Neuroimaging studies demonstrate that the macaque mid-STS increases with social network size (10) and is activated during observation of videos capturing social interactions (11). Importantly, the great majority of mid-STS neurons respond selectively to non–self-motion, such as movement of the experimenter’s hand (12), suggesting that this area plays a role in the distinction between self-actions and others’ actions. The self–other distinction is thought to be a prerequisite ability for mentalizing (1).

The possibility that the mid-STS is the macaque homolog of the human TPJ—a core region in the mentalizing network—invites two critical questions. The first concerns whether neurons in the macaque mid-STS show greater activity as social interactions become more realistic. Human neuroimaging documented that task performance with a live agent, as compared to that with an agent in a prerecorded video, induces enhanced activity in the mentalizing network including the TPJ (13–15). Thus, activity in the mentalizing network might be fundamentally different for live and recorded paradigms. To our knowledge, this issue has never been addressed in the macaque mid-STS at the single-neuron level.

The second question concerns a theoretical framework known as predictive coding, which is postulated to be a unifying framework that can explain many domains of brain functions including mentalizing (16). The predictive coding framework posits that a neural system, be it sensory, motor, or social, makes forward-looking predictions about incoming information (16). Accordingly, brain areas in the mentalizing network are expected to have neurons that encode predictions of others’ actions as well as those that encode deviations from these predictions (i.e., prediction error coding) (17). Consistent with this prediction error code, a subregion in the TPJ exhibits greater activity in response to observed actions when they violate the observer’s expectations compared to when they are made as predicted (18–20). However, these findings have been obtained only in human neuroimaging studies. Although expectation affects neuronal activity in the macaque mid-STS in the context of tactile responses (21), single-neuron mechanisms underlying prediction errors of others’ actions have yet to be examined.

To address these issues, we recorded from single neurons in the mid-STS during performance of a turn-taking task, in which two monkeys alternated the roles of action execution and action observation every three trials (Fig. 1) (22). We addressed the first issue by comparing neuronal activities in trials performed with and directly facing another real monkey (Fig. 1, Top; real agent [RA] condition), and trials performed with the same monkey replayed in a prerecorded video (Fig. 1, Bottom; filmed monkey [FM] condition). We addressed the second issue by introducing a block design in the task, in which the correct action was unchanged for 11 to 17 consecutive trials. This design enabled the observer monkey to predict the correct action of the partner for most of the trials, thereby providing a unique opportunity for it to observe the others’ actions and their consequences when they deviated from prediction.

Fig. 1.

Role-reversal choice task. (A, Top) Sequence of events in a single trial in which a neuronally recorded monkey (M1) was the actor and its partner (M2) was the observer. (Bottom) Temporal sequence of role alternation and block change. The actor and observer alternated their roles every three trials. The actor is indicated by background colors (pink, M1 and blue, M2). The correct target position (B1, B2, or B3) was changed every 11 to 17 trials with no prior notice. The green arrows indicate those trials in which the correct target position was switched (switch trials). (B) Two types of partners: real and filmed.

Here, we show that the activities of single neurons in the macaque mid-STS were significantly greater during live task performances with a real monkey than to task performances with a video-recorded monkey, although the observed actions were similar. We further show that, consistent with the predictive coding framework, a group of mid-STS neurons responded consistently when the expectations of others’ actions were not fulfilled. This prediction–error coding was specific to the action domain; almost none of these neurons responded to errors in the prediction of rewards.

Results

Two monkeys (Macaca fuscata, monkeys A and B; both designated as M1) were trained to perform a role-reversal choice task with two types of partners (collectively designated as M2) (Fig. 1). In the RA condition (Fig. 1, Top), M1 and M2 alternated the roles of “actor” and “observer” every three trials. In each trial, both agents were initially required to hold their start buttons for 0.7 to 1.3 s. Three target buttons on the actor’s side then turned on simultaneously, and the actor had to press one of them within 3 s. Both M1 and M2 were rewarded with a drop of water when the actor chose the target associated with a reward (“correct” target); neither was rewarded when the actor chose an incorrect target. The association between target position and reward remained the same for a block of 11 to 17 trials (nonswitch trials) and then was changed with no prior notice in the first trial of each block (switch trials; green arrows in Fig. 1, Bottom). In many cases, M1 required one or two trials to find the correct target after the switch trial (). In addition, M1 occasionally chose an incorrect target during the 11- to 17-trial block of consistent position–reward associations (). In the FM condition (Fig. 1, Bottom), M1 performed the same task facing a large monitor, in which prerecorded videos of another monkey performing the task were replayed; here, real-time social interactions were not possible. The overall performances of M1 across sessions were comparable for both the real and filmed partners (monkey A, RA 82%, n = 44, FM 81%, n = 39, RA versus FM, and P = 0.16; monkey B, RA 79%, n = 57, FM 81%, n = 67, RA versus FM, and P = 0.082; Welch’s t test, two sided; see also for details of performance accuracy and error).

We recorded from single neurons in the mid-STS during the RA condition using multicontact probes without sampling bias (Fig. 2). The recording site encompassed the dorsal bank and fundus of the STS, centered at the anterior end of the fundus in the rostrocaudal direction. Among a total of 531 neurons isolated, the activities of 328 neurons were significantly different during a peri-action period (between 400 ms before and 200 ms after the button press) than during a control period (600 to 0 ms before target onset). These neurons were classified into one of three types based on their actor coding: self, mirror, and partner types (Fig. 2; Materials and Methods). Self-type neurons responded selectively or preferentially to the self-action (Fig. 2; see for the population activity). Mirror-type neurons responded nondifferentially to the self-action and partner action (Fig. 2; see for the population activity). There were comparable numbers of these two neuronal types (Fig. 2; self, n = 73, and 22%; mirror, n = 78, and 24%), although neurons with decreased responses were more often mirror type (inhibited type; ; self, n = 30; mirror, and n = 50). Finally, partner-type neurons responded selectively or preferentially to the partner action (Fig. 2; see for the population activity). This type of neuron constituted the majority of task-related neurons (Fig. 2; n = 177 and 54%). Only a small number of partner-type neurons were the inhibited type (; n = 18). These findings demonstrate that isolatable neurons in the mid-STS consist of an abundance of partner-type neurons.

Fig. 2.

Actor coding and live agent preference in the mid-STS. (A) Recording site. (Center) Nissl-stained section (Right) for the area indicated by the red rectangle. (Left) The approximate anteroposterior level of the section is indicated in the lateral view of the brain. The red arrow indicates electrolytic microlesion made at the dorsal bank of the mid-STS; cs, central sulcus; ls, lateral sulcus; and sts, superior temporal sulcus (scale bar, 1 mm). (B) Proportions of various STS neuron types active during the RA condition. Bars outlined with solid lines, excited type; bars outlined with dotted lines, inhibited type; and open bar, partner error–type. (C–E) Raster displays and spike density functions of self- (C), mirror- (D), and partner-type (E) neurons, aligned to the time of choice (i.e., target button press; vertical line at time 0). The times of individual action potentials and target onset are indicated by colored dots and black triangles, respectively. The vertical lines at 1,300 ms indicate the time of reward feedback. (F) Spike density functions of a partner-type neuron during the RA (Top) and FM (Bottom) conditions. Same conventions as in C. (G) Distributions of differential firing rates for self- (black), mirror- (red), and partner-type (blue) neurons (from Top to Bottom) in the self-correct trials (Left) and partner correct trials (Right). The differential firing rates are defined as [(firing rate during the RA condition) − (firing rate during the FM condition)]. Thus, neurons preferring the RA condition have positive values in the abscissa, while those preferring the FM condition have negative values in the abscissa. The filled bars indicate neurons with significantly different firing rates in the two conditions (P < 0.01 and permutation test, two sided). The triangles indicate the medians. The fraction number in each histogram denotes the number of neurons with a significantly different firing rate relative to the total number of neurons tested. Only partner-type neurons showed a systematic response bias (yellow background).

We first asked whether mid-STS neurons have a preference for live social interactions. To answer this question, we compared the firing rates of neurons active during the peri-action period during the RA and FM conditions. A notable finding was that at the level of individual neurons, activities differed significantly between the two conditions in either of two ways (i.e., RA > FM or RA < FM). For example, the partner-type neuron shown in Fig. 2 exhibited a marked increase in activity during the partner action in the RA condition but a negligible response during the FM condition. Such differential responses between the two conditions were observed in all neuronal types (Fig. 2, filled bars), with some neurons preferring the RA condition (positive values in the abscissa), while others preferred the FM condition (negative values in the abscissa). At the population level, however, only partner-type neurons showed a systematic response bias (Fig. 2, yellow background). Specifically, activities of partner-type neurons in the partner correct trials were significantly greater during the RA condition than during the FM condition (P = 5.5 × 10−3, Wilcoxon signed-rank test with the Bonferroni correction [P < 0.05/2]). This systematic bias was absent in the self-correct trials (P = 0.16). These findings demonstrate that, in the mid-STS partner-type neurons, as a group, showed a preference for live task performance with a social agent instead of non–live task performance with the same agent presented in a prerecorded video. It should be noted that M1, as the observer, looked significantly longer at the M2’s face and M2’s choice in the FM condition than in the RA condition after start button onset (monkey A, P = 0.047; monkey B, P = 4.5 × 10−11; and Welch’s t test, two sided) and during making actions (monkey A, P = 0.038; monkey B, P = 1.2 × 10−5; and Welch’s t test, two sided), respectively (). These findings do not support the possibility that lower activity in the FM condition was caused by reduced or lack of attention to the filmed partner. Note also that consistent with previous studies (8, 12), mid-STS neurons were generally insensitive to the nature of the stimulus in video-recorded conditions. When neuronal activities were compared between the FM condition and another video-recorded condition, in which a wooden stick, rather than a monkey, performed the task on the monitor (filmed object [FO] condition; ), neurons exhibiting a significant activity difference were in the minority, and a population-level bias was not observed for any of the neuronal types ().

We next examined whether activities of partner-type neurons carry signals about errors in the prediction of others’ actions. In our role-reversal choice task, the correct target was fixed to one of the three target buttons for consecutive trials (Fig. 1, Bottom). We therefore hypothesized that M1 was capable of predicting M2’s choice once the correct target was identified (i.e., most of the nonswitch trials). In support of this, M1’s gaze during the period that M2 initiated a choice action was directed toward the correct target button that M2 should choose rather than the button that M2 was actually choosing in M2 “choice error” trials (Fig. 3). Specifically, in M2’s choice error trials, the duration of M1’s gaze in the period between 400 and 200 ms before target button press was directed significantly longer at the correct position than at the chosen position (Fig. 3; monkey A, P = 3.2 × 10−3, and n = 20; monkey B, P = 1.0 × 10−8, and n = 24; and paired Student's t test, two sided). The gaze duration at the correct position in M2’s choice error trials was not significantly different from that at the correct (and chosen) position in M2’s correct trials (Fig. 3; P > 0.05 and paired Student's t test, two sided). Similar gaze behavior was observed in the FM condition (). The initial gaze at the correct position was often followed immediately by a saccade to the position chosen by M2 rather than the remaining position (; monkey A, P = 8.5 × 10−3, and n = 20; monkey B, P = 0.035, n = 24; and Welch’s t test, two sided). M1, as the actor, looked at the chosen position equally often in correct choice trials and choice error trials (). Additionally, the monkeys accurately predicted the likely consequence of the partner’s choice. The licking movement in both M1 monkeys started to develop as soon as M2 chose the correct target (i.e., ∼1.3 s before the receipt of actual reward feedback; Fig. 3, black). The magnitude of this anticipatory licking was significantly smaller when M2 made a choice error (Fig. 3, red; monkey A, P = 7.2 × 10−3, and n = 25; monkey B, P = 3.0 × 10−4, and n = 29; and paired Student's t test, two sided; see for similar licking behavior in the FM condition). We found that when M2 made a choice error rather than a correct choice, a subset of the partner-type neurons exhibited a significantly greater response (Fig. 3 , blue arrows; excited, n = 21 and inhibited, n = 3). Furthermore, almost all of these “partner error–type neurons” (n = 23/24) were devoid of response to an unexpected reward omission in the “switch error” trials (Fig. 3 , green arrows), which occurred in switch trials. At the population level, the responses of partner error–type neurons to an expected reward gain and an unexpected reward omission were not significantly different from one another (Fig. 3; P = 0.10, n = 24 and paired Student's t test, two sided). These findings demonstrate that the prediction error of others’ actions, but not of rewards, is represented by partner error–type neurons in the mid-STS.

Fig. 3.

Behavioral and neuronal evidence for predictive coding. (A) Plot of M1’s gaze positions (red) in a single M2’s choice error trial during a time from 400 to 200 ms before the choice, in which the correct target was B3 (i.e., left button viewed from M1) but M2 chose B2 (i.e., center button). Blue square, ROI set for M2’s chosen position. (B) Proportion of gazes entered in ROIs set for the chosen and correct positions in M2’s choice error trials and for the chosen position in M2’s correct trials. Mean ± SEM; **P < 0.01; n = 20 for monkey A, n = 24 for monkey B; and paired Student's t test, two sided. n.s., not significant. (C) Difference in anticipatory licking behaviors in M2’s correct choice trials (black) and choice error trials (red). The vertical lines at 1,300 ms indicate the time of reward feedback. The shaded areas indicate SEM. Insets show comparisons of licking activities averaged from 800 to 1,300 ms after choice. Mean ± SEM; **P < 0.01; n = 25 for monkey A, n = 29 for monkey B; paired Student's t test, two sided. (D and E) Spike density functions for an excited partner error–type neuron (D) and for the population of such neurons (E). Note that this neuron type was activated by unexpected actions in M2’s choice error trials (traces and arrows in blue) but not by unexpected reward omission in M2’s switch error trials (traces and arrows in light green). The shaded areas indicate SEM. Other conventions are as in Fig. 2. (F) Comparison of reward-related activities (100 to 600 ms after reward feedback) for partner error–type neurons in the partner correct (red) and partner switch error (light green) trials (P = 0.10; n = 24; and paired Student's t test, two sided). n.s., not significant.

It might be puzzling that the activities of as many as 46% of the task-related neurons, either the self-type or mirror type, were significantly changed during self-actions (Fig. 2), because this region is thought to have no motor properties (23). Thus, the activities of mid-STS neurons during self-actions should be interpreted with caution. A plausible explanation is that such activity was evoked by visual input derived from the self-action and hence was not involved in action execution per se. To clarify this point, we introduced an occluded (OC) condition (Fig. 4), which was the same as the FM condition, except that the self-generated hand movements were OC from M1’s sight by the presence of an opaque panel. Only the far ends of the target buttons were visible for M1 to make a choice. We compared the peri-action–period activities of self- and mirror-type neurons in the OC and nonoccluded (nOC; RA or FM) conditions. In an example of a mirror-type neuron shown in Fig. 4, the phasic activity during the self-action in the FM condition was not present in the OC condition, while the activity during the partner action was largely the same for both conditions. In many of the self- and mirror-type neurons in our sample, activities during the self-action were significantly lower in the OC condition than in the nOC condition (Fig. 4, black bars; self, 83% and mirror, 60%). It should be mentioned that the remaining neurons were still activated without the self-hand movements having been seen (Fig. 4, gray and white bars), suggesting that visual input alone cannot explain the activities of a subset of neurons during action execution.

Fig. 4.

Control experiments. (A) Experimental setup for the OC condition. (B) Responses of a mirror-type neuron in the FM and OC conditions. Same conventions as in Fig. 2. (C) Summary histogram of response comparisons in the nOC (RA or FM) and OC conditions for individual self- and mirror-type neurons. n.s., not significant (i.e., lack of preference). (D) Stimulus conditions for visual property testing. (E) Summary histograms of a three-way ANOVA for individual partner-type neurons (P < 0.01, factors: distance, direction, and laterality and P < 0.01, post hoc Welch’s t test). n.s., not significant (i.e., lack of preference). (F) Spike density functions of a partner-type neuron. Same conventions as in Fig. 2. (G) Comparisons of responses to each visual feature for the partner-type neuron in F. The thick black lines depict the period of stimulus presentation.

The self-action and the partner action are the same in terms of a movement of the hand but different in terms of the agent of action. We hypothesize that the response of partner-type neurons, abundant in the mid-STS, encodes such agent information (i.e., other) and is useful to distinguish others’ actions from one’s own actions. It is also possible, however, that other basic visual features inherent in the two types of hand movements, such as the distance or direction relative to the subject, might have caused the response difference. To test this possibility, we conducted a visual property experiment in a single-monkey condition (Fig. 4). In this control experiment, each trial began when a white square (0.5°) appeared as a fixation point (FP) at one of four possible locations (left or right × near or far) on a display tilted 70° backward. A Gabor patch, centered at the FP, was then presented for 500 ms, during which the monkey had to maintain fixation within a 3° radius, and the Gabor patch drifted toward or away from the subject. In this way, we examined response dependency of individual partner-type neurons on distance, direction, and laterality of motion stimuli viewed from the subject. Because the partner action in our task was always generated in a “far” location “toward” the recorded monkey as opposed to the self-action made in a near-away manner, the visual feature account should predict stronger activity for the “far” and “toward” stimuli. The activities of 35 partner-type neurons recorded in the RA or FM condition were examined in this control experiment. We confirmed that most of this population was insensitive to these visual features (Fig. 4; P < 0.01; a three-way ANOVA with factors distance, direction, and laterality). Where a significant main effect was obtained, neurons preferring “near” and “right” (contralateral to the recording sites) outnumbered those preferring “far” and “left” (Fig. 4, Left and Right), as exemplified in Fig. 4 , whereas neurons preferring “away” and “toward” were equally common (Fig. 4, Center). We concluded that, for the great majority of partner-type neurons, their response selectivity was unlikely to be due to the basic visual features inherent in the partner action.

Discussion

Using a social turn-taking task, we have shown that more than 75% of the task-related mid-STS neurons exhibited agent-specific activity (self and partner types), with a clear predominance of partner-type neurons (54% of the task-related neurons). The visual property testing further revealed that the partner-selective activity was not caused, for the most part, by mere differences in low-level visual features associated with action execution and action observation. Moreover, the great majority (83%) of the self-type neurons exhibited decreased activities during the OC condition, in which the subject’s hand was OC from sight. These findings suggest that partner- and self-type neurons in the mid-STS encode agent-based information derived from visual input and play roles in distinguishing between the self and others in the action domain.

The majority (60%) of the mirror-type neurons also showed decreased responses during the OC condition, indicating that their activities during action execution might also be explained by visual input derived from self-hand motion. Similar neurons with mirror properties have also been reported in the frontal cortex, such as the ventral premotor cortex (24, 25) and the medial prefrontal cortex (MPFC), including the presupplementary motor area (22, 26–28). Given that electrical stimulation in these frontal areas can evoke upper-limb movements (29–32) and that activity during the self-action can remain in total darkness (33, 34), mirror-type neurons in the frontal cortical areas are considered to have motor properties. Thus, a major difference of mirror-type neurons in the temporal and frontal cortices may reside in their general absence and presence of motor properties, respectively. The majority of the mid-STS mirror-type neurons are likely to respond to visual aspects of moving forms including arm movements independent of agent information. Prolonged activity of mirror-type neurons (e.g., Fig. 2 and ) might reflect their property to respond to both the movement toward and away from the target. On the other hand, the remaining (40%) mirror-type neurons were still active in the OC condition. The activities of this subset of neurons, as well as of a small subset of the self-type neurons with similar properties, might be ascribable to tactile or auditory inputs caused by reaching, as the mid-STS is a polysensory zone (8, 21).

The human TPJ, a possible homolog of the macaque mid-STS (6), is thought to be a node in the mentalizing network (2, 35). Action coding in the MPFC, another node in the mentalizing network, has been investigated in macaques at the single-neuron level (22, 26–28). From a previous study using the same task, we reported that partner-type neurons are prevalent in the MPFC (22). While the mentalizing network is characterized by the predominance of other-action coding (36), the present results revealed notable differences in properties of partner-type neurons in the mid-STS and those in the MPFC.

At the population level, partner-type neurons in the mid-STS showed enhanced activity during live task performance with a social agent (RA condition) compared to non–live task performance with the same but filmed agent (FM condition), which is consistent with human brain imaging studies (13–15). However, population activities during action observation were not significantly different in the two video-recorded conditions, although the choice was made by a monkey in the FM condition and by a wooden stick in the FO condition. Whether these neurons are also insensitive to the nature of the stimulus during live social interactions is an interesting question for future work. By contrast, partner-type neurons in the MPFC differentiated among the three types of partners with different levels of activity (RA > FM > FO) (22).

It is important to discuss which sensory (e.g., visual, acoustic, and olfactory) or contextual factors (e.g., real-time behavioral contingency and physical space sharing between self and other) might have contributed to the differences in neural activity between the RA and FM conditions. Understanding such contributing factors is one of the ultimate goals in social neuroscience. In our experiment, monkey identity in the RA and FM conditions was the same in 86% of the total sessions. The monkey size in the FM condition was adjusted so that it was comparable to the size of the corresponding monkey in the RA condition. The depth information based on binocular disparity cues was lost, however, in the FM condition. The acoustic sounds generated during task performance were recorded and replayed in the FM condition. However, the smell of real monkeys was not replicated in the FM condition. Finally, M2’s response times from target onset to target button press were not significantly different between the RA and FM conditions (M2 of monkey A, 688 ± 17 ms in the RA condition [n = 44], 706 ± 2 ms in the FM condition [n = 39], P = 0.332; M2 of monkey B, 672 ± 11 ms in the RA condition [n = 57], 691 ± 2 ms in the FM condition [n = 67], P = 0.064; mean ± SEM; and Welch’s t test, two sided). We conjecture that, in these sensory factors, the olfactory component might be more important than the visual component. This is because activities of mid-STS neurons were not significantly different between the two filmed conditions (), although the visual appearance of the acting agent (i.e., monkey [FM] versus stick [FO]) was entirely different. To better understand visual impacts on neural activity, the development of alternative experimental procedures would be useful. One idea is to design a condition in which M1 views M2 through a monitor on which the image can be manipulated freely using a video-editing tool.

Regarding the contextual factors, the RA and FM conditions differed in terms of the presence (RA) or absence (FM) of real-time behavioral contingency and physical space sharing between two monkeys. We were unable to determine which of the two factors played a more important role. To solve this issue, the design of a modified FM condition, which is a mixture of the RA and FM conditions, would be beneficial. One idea is to develop an experimental condition in which two monkeys are placed in separate rooms and they view each other via a monitor connected online. Understanding what makes the difference in brain activity between the RA and FM conditions provides important insights into what really defines “social.”

The existence of partner error–type neurons suggests that the mid-STS is involved in the predictive coding of others’ actions. About 14% of STS partner-type neurons responded preferentially to the partner’s unexpected choice errors. Interestingly, virtually none of them were responsive to unexpected negative feedback (i.e., reward omission) in the partner’s switch error trials. This finding is in marked contrast to partner error–type neurons in the MPFC in that nearly one-half of them respond to negative feedback as well (37). This difference provides an important insight into the functional role of the mid-STS in social error monitoring. In practice, the commission of error in the partner’s choice error trials and the negative feedback in the partner’s switch error trials are both useful for improving task performance. Indeed, error-related negativity in the medial frontal cortex is observed regardless of the type of error (38–41). Our observation indicates that mid-STS partner error–type neurons signal errors in the prediction of others’ actions, demonstrating that the detection of error commission and the detection of negative feedback are distinct neural processes, at least in the social context. The mid-STS may send signals about deviations from the predictions of others’ actions directly to the MPFC (42). While the MPFC encodes both action (visible) and outcome (invisible) information for performance monitoring (22, 26, 43–46), the mid-STS, as part of the visual system in the temporal lobe, is involved in performance monitoring on the basis of information about ongoing visible actions. This role of monitoring others’ visible actions seems to be complemented by the role of the ventral bank of the mid-STS in signaling others’ predicted strategy at a more abstract level (47).

Taken together, our current findings suggest that the mid-STS preferentially processes action information derived from other live agents. Furthermore, this cortical region carries signals consistent with the predictive coding of others’ actions in a way that is distinct from that of the MPFC. The present findings highlight unique roles of the macaque mid-STS and further delineate functional similarities between the human TPJ and the macaque mid-STS in aspects of social cognition associated with mentalizing. The temporal cortex and the TPJ are among the areas that have particularly expanded in the human compared to the macaque brain (48). It has been proposed that such expansion is associated with increased social abilities in humans (49, 50). These studies and ours invite a key evolutionary question of whether anatomical and functional similarities between the two apparently distant cortical areas (i.e., mid-STS and TPJ) are the result of the rearrangement of existing areas or the emergence of completely new areas.

Materials and Methods

Animals.

Four male macaques (M. fuscata) were used in this study. Two of them (monkey A [age 6, 5.1 kg] and monkey B [age 6, 5.0 kg]) underwent neuronal recordings. The other two (monkey C [age 8, 8.1 kg] and monkey Q [age 5, 6.3 kg]) participated in this study solely as partners. All animal care and experimentation protocols were approved by the Institutional Animal Care and Use Committee of the National Institutes of Natural Sciences and were conducted in accordance with the guidelines described in the US NIH Guide for the Care and Use of Laboratory Animals (51).

Behavioral Procedures.

Role-reversal choice task.

In this task, monkey A or B (neuronally recorded monkey, referred to as M1) performed a social choice task (Fig. 1) with three different types of partners (referred to as M2). The behavioral data were obtained in the same sessions as those in our previous study (22), in which details of the task were also described.

In the RA condition, M1 sat in a primate chair facing an RA partner in a sound-attenuated room to perform the task. During data collection, monkey A was paired with monkey B or monkey C, and monkey B was paired with monkey A, monkey C, or monkey Q. None of these pairs were cage mates, and therefore their fixed dominance relationship could not be determined. Four buttons, placed on a square panel on the front side of the chair, were assigned to each participant: a circular one directly in front as a start button and three rectangular ones a bit further in front as target buttons. In this task, M1 and M2 alternated their roles as “actor” and “observer” every three trials. Each trial started when the start buttons for both M1 and M2 were illuminated. Both participants were required to press the buttons for 0.7 to 1.3 s. The three target buttons on the actor’s side then turned on, only one of which was correct. The actor was required to choose one of them in 3 s. When the actor pressed any of the targets, a high-pitch tone (1 kHz) sounded as feedback for the button press. Both participants were rewarded with a drop of water with a delay of 1,300 ms when the correct button was chosen. Neither monkey was rewarded when the choice was wrong. The observer was required to hold the start button down throughout the trial. The position of the correct target remained the same for a block of 11 to 17 trials and was changed with no prior notice. The percentage of optimal choices after switch error was comparable between the two monkeys (monkey A, 93.8 ± 1.3%, and n = 44; monkey B, 91.7 ± 1.7%, and n = 57; mean ± SEM; P = 0.36; and Welch’s t test, two sided). Note that no experimenter participated for the STS recording unlike our previous study (22), and therefore, only real monkeys were the partner in this condition. However, we used the term RA for the sake of consistency with our previous study.

We introduced two more conditions, i.e., FM and FO conditions, in which M2 was a filmed partner. In these conditions, a large liquid crystal display (LCD) monitor (W67.41 × H99.56 cm) was placed in front of M1 such that the target buttons for M1 and those for the filmed partner were aligned in the same way as in the RA condition from M1’s viewpoint. The partner was replayed on the monitor; the filmed partner was either a monkey in the FM condition or a wooden stick in the FO condition. The task structure in the two filmed conditions was fundamentally the same. When the filmed partner was the actor, the monkey or stick on the monitor pressed one of the three target buttons. When the filmed partner was the observer, the monkey or stick held the start button down throughout the trial. Trials were separated by a black screen. The trials of the filmed partners were chosen to make choices with overall correct rates comparable to those of the RA partners (79, 77, and 78% in the RA, FM, and FO condition, respectively, for the partner of monkey A, P = 0.46; 80, 78, and 78 in the RA, FM, and FO condition, respectively, for the partner of monkey B, P = 0.062; one-way ANOVA). During data collection, the first filmed condition performed each day was chosen randomly, and the two filmed conditions alternated every nine blocks until a total of at least 18 blocks were run for each filmed condition. Whenever possible, the RA condition was also performed before or after the filmed conditions. In most sessions (86%), M2’s identity was the same between the RA and FM conditions.

To generate visual stimuli for the filmed conditions, actions of the filmed partners were recorded in advance using a video camera (HDR-CX470, Sony; 30 frames/s, 1,920 × 1,080 pixels), while they performed the task with a human experimenter. The video sequences were then edited to make short video clips (7 to 8 s, 800 × 860 pixels) for individual trials in the filmed conditions. Each video clip started about 1 s before the onset of the start button and ended about 3 s after the target button press. A total of 8 to 10 different video clips were prepared for each target button choice (B1, B2, and B3), and on each trial, one from them was randomly selected. Note that the same sequence of trials was not repeated across multiple recording sessions.

Two types of M2’s errors (i.e., M2’s choices that ended in no rewards) were defined as described previously (22). Briefly, M2’s switch error occurred in switch trials by M2 choosing a target that had been associated with a reward in the preceding block. M2’s choice error occurred in nonswitch trials by M2 choosing a target that was not associated with a reward in the current block.

OC conditions.

A control condition was introduced to test whether the responses of self- and mirror-type neurons in self-correct trials were ascribable to motor properties or visual properties. In this condition, M1 performed the role-reversal choice task with the FM, as in the FM condition, but with an opaque panel placed above the chair panel (Fig. 4). Any self-generated hand movements were OC from M1’s sight, whereas the far end of the target buttons as well as the partner’s actions were visible.

Visual property testing.

Another control experiment was introduced to test if the response differences of partner-type neurons during the self-action and partner action were attributed to differences of basic visual features in those situations. In this experiment, an LCD monitor (W30.11 × H37.63 cm), tilted 70° backward, was placed at a viewing distance of 47 cm. A drifting Gabor patch (2 Hz, ∼0.6 cycles/degree, ∼7° in diameter) was presented with varying distances (near or far), directions (away or toward), or lateralities (left or right) relative to the subject (Fig. 4). Each trial began when a white square (0.5°) appeared as a FP at one of four possible locations on the monitor. The monkey was initially required to maintain fixation within a 3° radius window for 300 to 600 ms. A drifting Gabor patch centered on the FP was presented for 500 ms, while the monkey was required to hold fixation. A drop of water was delivered as a reward for the successful fixation 500 ms after the offset of the stimulus. A block of eight stimuli (2 distance × 2 direction × 2 laterality) was presented in a randomized sequence. The stimulus sequence, typically repeated 5 to 30 times, was reshuffled in each block.

Surgical Procedures.

General anesthesia was induced by intramuscular injections of ketamine HCl (10 mg/kg) and xylazine (1 to 2 mg/kg) and maintained with isoflurane (1 to 2%) throughout the procedure. A plastic headpost and a plastic recording chamber were implanted on the skull using acrylic screws and dental acrylic under aseptic conditions. The coordinates of the chamber were decided with the aid of MRI. Craniotomy was performed to have access to the mid-STS when the monkeys had been fully trained on the behavioral procedures. Antibiotics and analgesics were administered after surgery.

Recording Procedures.

Behavior.

Stimulus presentation, behavioral data collection, and reward delivery were controlled by a personal computer running the MonkeyLogic Matlab toolbox (52). Gaze direction of M1 was monitored using an infrared video tracking system (EyeLink II; SR Research) at a time resolution of 1,000 Hz and a spatial resolution of 0.1°. Licking movements were sampled at 1 kHz, filtered (100 to 200 Hz), and amplified using a vibration sensor (AE-9922; NF corporation) attached to the reward spout. The licking data were then fed to the computer. The water reward was delivered through the spout using a solenoid valve controlled by the computer. The reward system was placed outside the sound-attenuated room. Overt movements of the monkeys were continuously monitored using a video-capturing system.

Neuronal activity.

Each pair of monkeys were trained on the task until their correct rates reached a criterion (75%). Single-unit recordings were then carried out in the left hemisphere of the two M1 monkeys as described previously (22). Extracellular potentials were measured using a 16-channel linear microelectrode array (U- or S-probe; Plexon Inc.), with an interelectrode spacing of 200 µm. The range of impedance for all channels was 0.3 to 0.5 MΩ at 1 kHz. Signals were amplified and filtered (150 Hz to 8 kHz), and single-unit activity was isolated using an online template-matching spike discriminator (OmniPlex system; Plexon Inc.). The electrode was advanced using an oil-driven micromanipulator (MO-97A; Narishige) through a stainless guide tube. The guide tube was held in place by a grid attached to the recording chamber, which allowed recordings every 0.5 mm between penetrations.

After the electrode was placed at the desired depth, we waited at least 1 h for stabilization of the probe position in the cortex. This waiting period resulted in stable neuronal recording, as described previously (43, 53). Spiking activities were manually sorted offline using a spike sorting software (Offline Sorter, Plexon Inc.) to improve the isolation of each unit. Spike waveforms and interspike intervals (ISIs) were inspected to exclude any data that were likely contaminated by multiple units. To validate the quality of the single-unit data, spikes were divided into two groups for each unit: those recorded in the first half of the recording session and those in the second half. The spike waveforms were then averaged in each period, and a correlation coefficient of the two averaged spike waveforms was calculated. The correlation coefficients of all units recorded in the RA condition were distributed almost exclusively above 0.9 (), suggesting that the spike waveform was stable during unit recording. While this analysis is useful to examine the stability of spikes, it is theoretically possible that recorded spikes actually consist of multiple units with similar waveforms that are also stable over time. In an attempt to rule out this possibility, we further analyzed the distribution of ISIs for each presumed single unit. For this purpose, the ISIs were binned in 1-ms resolution, and the first time bin containing more than 1% of the total spike count was taken as the minimal ISI. The distribution of the minimal ISIs for all presumed single units was devoid of bins below 3 ms (). Finally, we applied the same analyses to spiking activities recorded continuously in the RA and FM conditions. We confirmed that our unit isolation was generally of high quality and stability throughout the two task conditions ().

Identification of Recording Sites.

We aimed to record activity of neurons in the dorsal bank and fundus of the mid-STS, where neurons are known to project to the MPFC (42). The anterior end of the inferior parietal sulcus (IPS), estimated by the MRI images, was used as a landmark in the anteroposterior axis for the initial mapping. The lateral sulcus and its ventral bank were confirmed by a long region (>3 mm) completely lacking neuronal activity, followed by an auditory-responsive area. When the tip of the electrode reached the dorsal bank of the STS, we “clinically” tested response properties of well-isolated neurons by showing several actions known to activate the target region (54–56). For example, an experimenter presented reaching actions toward the monkeys or oriented their head toward or away from the monkeys. Neuronal recordings were mainly performed from 4 mm anterior to 1 mm posterior to the anterior end of the IPS and from 0 to 4 mm lateral to the medial end of the STS fundus. The recording sites were histologically confirmed (Fig. 2) to cover ∼8 to 13 mm anterior to the interaural plane. This region presumably corresponds to the superior temporal polysensory area (42, 57).

Statistics.

No statistical approach was conducted to predetermine sample sizes, but our sample sizes were similar to those in previous studies ( e.g., refs. 22, 56). Data were assumed to be normally distributed, but this was not formally tested. Neuronal recordings were performed with no sampling bias; all well-isolated neurons were recorded. Data collection and analysis were not performed blind to the conditions in each experiment. All statistical procedures were assessed by two sided tests using commercial software (Matlab 2016a and 2017a; MathWorks Inc.).

Behavioral Data Analysis.

M1’s gaze and licking behaviors were compared during M2’s correct choice trials and choice error trials. Note that the correct choice trials as well as the choice error trials used in the following analyses were preceded by a correct choice, made either by M1 or M2, at least once in each block.

Gaze behavior.

M1’s gaze behavior was examined during a period between 400 and 200 ms before button press for M2’s correct trials and choice error trials. The regions of interest (ROIs) were set at the chosen position and at the correct target position. We then calculated the proportion of time that M1’s gaze entered each ROI during the 200-ms period. This procedure yielded three numerical values for comparison (“chosen” for choice error trials, “correct” for choice error trials, and “chosen and correct” for correct choice trials; Fig. 3). The proportion of gaze time in the “correct” ROI in M2’s choice error trials was compared with those in the other two (P < 0.01 and paired Student's t test, two sided).

Licking behavior.

For each session, M1’s licking activity was initially convolved with a Gaussian kernel (SD = 10 ms) and then normalized by activity during 500 to 0 ms before target onset using a z-score normalization procedure. The resulting licking activities in a period between 800 and 1,300 ms after button press (i.e., 500 to 0 ms before reward feedback) were then averaged and compared between M2’s correct choice trials and choice error trials (P < 0.01 and paired Student's t test, two sided).

Neuronal Data Analysis.

A total of 531 neurons were recorded in the mid-STS of both monkeys. For the following statistical test, trials in the RA condition were sorted into three groups: self-correct trials, partner-correct trials, and partner error trials. Note that the partner error trials included only M2’s choice error trials, and these trials had to be preceded by a correct choice, made by either M1 or M2, at least once in the corresponding block. To classify individual neurons, a series of statistical tests were applied to the firing rates during a control period (600 to 0 ms before target onset) and during a peri-action period (−400 to 200 ms after target button press) in the three trial groups, as described previously (22). First, a two-way ANOVA (P < 0.05) was performed for the peri-action period activity with two factors: agent (self or partner) and performance outcome (correct or incorrect). Neurons with a significant main effect of agent were then divided into self or partner type, each being further subclassified into excited or inhibited type. “Excited self-type” neurons were defined as those with peri-action–period activities significantly higher in the self-action trials than in the partner action trials (P < 0.05, post hoc Tukey–Kramer test) and peri-action–period activities in the self-action trials significantly higher than the control-period activity (P < 0.05, paired Student's t test). “Inhibited self-type” neurons were defined as those with peri-action–period activities significantly lower in the self-action trials than in the partner action trials (P < 0.05, post hoc Tukey–Kramer test) and peri-action–period activities in the self-action trials significantly lower than the control-period activity (P < 0.05, paired Student's t test). Similarly, “excited partner-type” neurons were defined as those with peri-action–period activities significantly higher in the partner action trials than in the self-action trials (P < 0.05, post hoc Tukey–Kramer test) and peri-action–period activities in the partner action trials significantly higher than the control period activity (P < 0.05, paired Student's t test). “Inhibited partner-type” neurons were defined as those with peri-action–period activities significantly lower in the partner action trials than in the self-action trials (P < 0.05, post hoc Tukey–Kramer test) and peri-action–period activities in the partner action trials significantly lower than the control-period activity (P < 0.05, paired Student's t test). “Partner-type” neurons with a significant main effect of performance outcome were further subdivided into partner error types. Specifically, excited partner-type neurons were defined as an “excited partner error type” if their peri-action–period activities were significantly higher in the partner error trials than in the partner-correct trials (P < 0.05, post hoc Tukey–Kramer test). Inhibited partner-type neurons were defined as an “inhibited partner error type” if their peri-action–period activities were significantly lower in the partner error trials than in the partner-correct trials (P < 0.05, post hoc Tukey–Kramer test). Lastly, neurons with no significant main effect were defined to be a “mirror” type if their peri-action–period activity was significantly higher (“excited”) or lower (“inhibited”) than the control-period activity in both self-correct and partner-correct trials (P < 0.05, paired Student's t test).

To examine whether partner error–type neurons were responsive to unexpected negative feedback (i.e., reward omission), neuronal activities during a reward feedback period (100 to 600 ms after the reward feedback) were compared during the partner’s correct trials and switch error trials (P < 0.05, paired Student's t test). For this analysis, we chose recording sessions in which M2’s switch error occurred more than five times.

Continuous spike density functions were constructed for each neuron by convolving individual spikes with a Gaussian kernel (SD = 30 ms). The resulting spike densities were normalized from zero to maximum and averaged across neurons in each type to obtain population activities.

The preference of individual neurons for live interactions was examined by comparing the firing rates in the peri-action periods of the RA and FM conditions (P < 0.01, permutation test with 1,000 iterations). Likewise, the preference of individual neurons for the nature of the filmed agent was examined by comparing the firing rates in the peri-action periods of the FM and FO conditions (P < 0.01, permutation test with 1,000 iterations). The preferences for live actors and real filmed actors of each neuron type at the population level was examined by plotting the distributions of differential firing rates for the self- and partner-correct trials. The differential firing rate of each neuron for the RA–FM comparison was obtained by subtracting the peri-action–period activity in the FM condition from that in the RA condition. Likewise, the differential firing rate of each neuron for the FM–FO comparison was obtained by subtracting the peri-action–period activity in the FO condition from that in the FM condition. The sign of the resulting value was inverted for inhibited neurons. It was then tested whether the median value of the distribution of the differential firing rates for each neuron type was significantly different from zero in either self-correct or partner correct trials (P < 0.05/2 and Wilcoxon signed-rank test with Bonferroni correction, two sided).

Histology.

After all experiments were completed for monkey B, electrolytic microlesions were made at the anterior end of the recording sites at locations where the electrode entered the dorsal bank of the mid-STS. Monkey B was then deeply anesthetized with sodium pentobarbital (70 mg/kg, intravenously) for perfusion fixation. The monkey was transcardially perfused with 0.1 M phosphate-buffered saline (pH 7.4), followed by 10% formalin in 0.1 M phosphate buffer (pH 7.4). The brain was removed from the skull, postfixed overnight, and saturated in 30% sucrose for 2 wk at 4 °C. The brain was sliced coronally at 50-µm thickness using a freezing microtome (REM-710 + Electro Freeze MC-802C; Yamato). A series of every 10th section was Nissl stained with 5% cresyl violet. Images of each section were acquired using brightfield microscopy with a 1 or 4× objective (Eclipse Ni-U; Nikon).