Differences in working memory coding of biological motion attributed to oneself and others.

The question how the brain distinguishes between information about self and others is of fundamental interest to both philosophy and neuroscience. In this functional magnetic resonance imaging (fMRI) study, we sought to distinguish the neural substrates of representing a full-body movement as one's movement and as someone else's movement. Participants performed a delayed match-to-sample working memory task where a retained full-body movement (displayed using point-light walkers) was arbitrarily labeled as one's own movement or as performed by someone else. By using arbitrary associations we aimed to address a limitation of previous studies, namely that our own movements are more familiar to us than movements of other people. A searchlight multivariate decoding analysis was used to test where information about types of movement and about self-association was coded. Movement specific activation patterns were found in a network of regions also involved in perceptual processing of movement stimuli, however not in early sensory regions. Information about whether a memorized movement was associated with the self or with another person was found to be coded by activity in the left middle frontal gyrus (MFG), left inferior frontal gyrus (IFG), bilateral supplementary motor area, and (at reduced threshold) in the left temporoparietal junction (TPJ). These areas are frequently reported as involved in action understanding (IFG, MFG) and domain-general self/other distinction (TPJ). Finally, in univariate analysis we found that selecting a self-associated movement for retention was related to increased activity in the ventral medial prefrontal cortex.

INTRODUCTION

The body is considered as the basis of our self—as Baumeister (1999) influentially remarked: “Everywhere in the world, self starts with body.” While the abstract, language‐mediated aspect of our self‐emerge only later in our lives, knowledge about our bodies is present from the onset of our existence. These two types of information have been argued to form two distinct self‐representations: the bodily self and the conceptual self (Christoff, Cosmelli, Legrand, & Thompson, 2011; Gallagher, 2000; Neisser, 1988). This difference also finds support in research on the brain mechanisms underpinning processing of bodily and abstract self‐related information. The bodily self is underpinned primarily by multisensory and sensorimotor brain systems, as shown by studies on rubber hand illusion, full body illusion, and recognition of one's face, body, and body movements (Blanke, 2012; Blanke, Slater, & Serino, 2015; Christoff et al., 2011; Limanowski & Blankenburg, 2013; Tsakiris, 2010, 2017). The conceptual self is mostly related to the activity of higher‐level brain areas, especially the cortical midline structures and has been investigated through evaluation of self‐related adjectives or formation of abstract, arbitrary self‐associations (Hu et al., 2016; Martinelli, Sperduti, & Piolino, 2013; Northoff et al., 2006; Qin & Northoff, 2011; Yin, Bi, Chen, & Egner, 2021). However, there is relatively little overlap between the brain areas involved in representing these two types of self‐related information (Hu et al., 2016; Qin, Wang, & Northoff, 2020).

Neurocognitive processing of information about one's bodily self is typically investigated through the use of images or videos of participants' face, body, body parts, or body movements. However, all of these stimuli are characterized not only by their self‐relatedness, but also by the fact that they are highly familiar to participants. Most previous studies have attempted to minimize the influence of familiarity by comparing processing of self‐related stimuli (own face, own movement pattern) with stimuli representing familiar or famous others (friends, family, celebrities). However, this kind of control conditions can never fully match with regard to familiarity. We spend all of our lives inside our bodies, while we devote substantially smaller fraction of lifetime paying attention to the bodies of other people, even famous ones. Therefore, while studies comparing processing of self‐related information with information related to familiar others provide an important contribution to our understanding of how the brain represents our bodily self, there is a need for validation of these results using designs which to a greater extent control for familiarity.

To control for familiarity in self‐related processing, recent studies have begun investigating how creating ad hoc self‐associations can modulate cognitive processing. In such studies participants are told to associate arbitrary stimuli (e.g., geometrical shapes) with various identities. Crucially, one of these identities is the participant themself. For example, a participant can be asked to associate a triangle with themself (through an instruction: “You are a triangle”), a square with a friend, and a circle with a stranger (Sui, He, & Humphreys, 2012; Sui, Rotshtein, & Humphreys, 2013). The results show that even an arbitrary association with the self leads to robust prioritization of self‐associated stimuli, as manifested by shorter reaction times and improved detection accuracy. In behavioral studies, the same procedure has been also used with bodily stimuli such as faces and avatar bodies, yielding similar effects (Mattan, Quinn, Apperly, Sui, & Rotshtein, 2015; Payne, Tsakiris, & Maister, 2017; Woźniak & Hohwy, 2020; Woźniak & Knoblich, 2019). At the neural level, several previous experiments investigated which brain areas are affected by creating such arbitrary self‐associations for abstract, symbolic stimuli (Lockwood et al., 2018; Sui et al., 2013; Sui, Liu, Mevorach, & Humphreys, 2015; Yankouskaya et al., 2017; Yankouskaya & Sui, 2021; Yin et al., 2021; Zhao, Uono, Li, Yoshimura, & Toichi, 2018) and found that midline cortical structures (especially the ventromedial prefrontal cortex) are critically involved in such tasks. However, to date, no neuroimaging study used this approach to investigate which brain areas represent bodily, rather than abstract, information associated with the self. This is a significant omission, as bodily self is arguably more basic than the conceptual self (Christoff et al., 2011; Gallagher, 2000; Gillihan & Farah, 2005; Woźniak, 2019), and also because, as mentioned before, previous research found that brain regions underpinning processing of bodily information rarely overlap with brain regions responsible for processing of abstract self‐related information (Gillihan & Farah, 2005; Hu et al., 2016; Qin et al., 2020).

The present study investigated, focusing on full‐body movements, which brain regions differentially represent self‐ versus other‐associated bodily information. Specifically, it was designed to address how the brain differentiates between seeing identical actions—that is, the same movements presented from a third‐person perspective—attributed to oneself or to other people. This stands in contrast to previous studies on self‐recognition of body movements, in which people were asked to recognize their real body movements (Bischoff et al., 2012; Jokisch, Daum, & Troje, 2006), and which hence targeted different cognitive processes. In the current study participants were presented with stimuli depicting different types of movements, presented as point‐light walkers (short animations presenting people performing an action, but devoid of any information about their physical body appearance). Critically, across different trials participants were told to attribute the same movements either to themselves, or to others, which serves to match familiarity between the conditions. We employed a working memory (WM) paradigm (delayed‐match‐to‐sample task) during which participants had to form and actively maintain the representation of a self or non‐self‐associated movement for several seconds. We used MVPA on functional magnetic resonance imaging (fMRI) data to detect brain areas that differently code memorized movement associated with the self or another person.

Previous research on self‐related processing suggest that it is mainly activity in three brain regions distinguishing whether a presented movement is associated with oneself or other identities. First, selective processing of self‐related information have been reported in frontal cortices that are also involved in action recognition and which are believed to be part of the human mirror‐neuron system, especially the inferior frontal gyrus (Molenberghs, Cunnington, & Mattingley, 2012). Second, the ventral medial prefrontal cortex has been found to play an important role in processing abstract self‐related information, such as one's name, self‐related adjectives, or arbitrary self‐associated stimuli (Hu et al., 2016; Sui et al., 2013; Yankouskaya et al., 2017). Finally, the temporal parietal junction and surrounding areas have been identified as being critically involved in self‐other distinction across a range of cognitive processes (Ganesh, van Schie, de Lange, Thompson, & Wigboldus, 2011; Hu et al., 2016; Quesque & Brass, 2019; Sui et al., 2013). Based on these reports, we were particularly interested in the involvement of these three regions in representing movements as self‐related. To test this, we included three experimental conditions, namely memorizing the movement of oneself, or of two different stranger identities. This design allowed us to test for distinguishable activation patterns evoked by the self‐associated and stranger‐associated movements, while we expected no differences between the two strangers.

METHODS

Participants

The inclusion criteria for participation were age over 18 and no psychiatric or neurological disorders. Twenty‐four volunteers participated in the study (age range: 21–40 years; M = 26.3, SD = 4.89; 13 females, 11 males). All participants were right‐handed. All participants had normal or corrected‐to‐normal vision. Informed consent was obtained from all participants before the start of the experiment according to procedures approved by the Freie Universität Berlin Ethics Committee.

Procedure

The experimental paradigm comprised a delayed match‐to‐sample working memory task employing retro‐cues associated with specific identities. Inside the MRI scanner, participants were first familiarized with the task while structural MRI data was acquired. Next, five experimental runs of 15 min 30 s each were conducted while functional MRI data were collected.

At the beginning of the experiment, participants learned which of three colors (red, green, blue) represents which one of three identities: You (the participant her‐ or him‐self), Pat (Patricia or Patrick—gender of the name was always matching with participant's gender), and Doc (a gender‐matched stranger who is a doctor). The labels “Pat” and “Doc” were chosen as indicators of two strangers' identities because they have the same length as the word “You,” they can be used to reflect both genders (the identity of the strangers were always instructed as of the same gender as the participant, to exclude potential effects of associations with a different gender‐identity than the own one), and each belongs to a different grammatical category ruling out potential confound effect of grammatical distinctiveness (Schäfer, Wentura, & Frings, 2017). The associations between colors and identities were presented in a written form on the instruction sheet which was given to each participant. The color‐identity associations were counterbalanced across participants using a Latin square scheme. Color‐identity associations were introduced to the task in order to avoid using labels as retro‐cues in the experimental task.

Figure 1 illustrates the procedure of the retro‐cue task used in the present study. In each trial participants had to memorize a biological motion stimulus, which was indicated to be either a movement of oneself or of another person. Each trial started with the presentation of two short animations of movements displayed as point‐light walkers (each 1,000 ms, separated by a 1,000 ms ISI, see section “Stimuli”). Before each stimulus the identity performing the motion was indicated by an identity‐cue: either “YOU” (self‐condition), “PAT”, or “DOC” (others‐conditions) for 500 ms (see Figure 1). For example, if the first cue was “PAT” then it indicated that the first movement was performed by Pat, and if the second cue was “Doc” then it conveyed the information that the second movement was performed by Doc. If one of the cues was “You” then participants were told to interpret the corresponding movement as if they were watching themselves from a third‐person perspective (for example on a video recording). After the presentation of stimuli, a retro‐cue instructed which of these two movements should be maintained for the working memory phase. The retro‐cue was either a red, blue, or green circle and indicated the identity (You, Pat, or Doc) whose movement had to be memorized, according to the previously learned color‐identity mappings. For example, if the associations were the same as presented in Figure 1a (upper right) then a blue retro‐cue told a participant to memorize Doc's movement, and if it was red then they were expected to memorize “Your” (participant's) movement. After a 7‐s retention period, a target movement was presented and participants' task was to decide if the target movement was identical (i.e., the same type of movement and presented from the same point of view) to the memorized movement. Reports were given via right‐hand middle and index finger button presses (yes/no–left/right balanced across participants).

FIGURE 1

Experimental paradigm and analysis. (a) Before the experiment started participants were told which of three colors represent which identity (example given in upper right panel, assignment of colors was counterbalanced across participants). During fMRI scanning, participants performed a retro‐cue delayed match‐to‐sample working memory task. In each trial, two full‐body movement stimuli were presented and a consecutive retro‐cue indicated which of the two movements had to be retained for a 7‐s retention period. Finally, a target movement stimulus was presented and had to be judged whether it was the same or different from the memorized movement. A distinctive feature of this task was that each of two presented movements was associated with one of the three identities: Either with the participant oneself or with one of two strangers. The color retro‐cue indicated which movement should be memorized by referring to the identity of a person performing a movement. (b) As fMRI data acquisition was time‐locked to the start of the working memory retention period, it was possible to conduct time‐resolved decoding analysis. In short, whole‐brain searchlight classification analyses were performed on the data of each timebin, corresponding to separate functional volumes. We used a group‐level t‐contrast across timebins t2–t5 to test for above‐chance decoding accuracies

Colors rather than labels were used as retro‐cues to avoid potential confounds related to label‐specific processing, as there exists evidence that self‐related labels are processed preferentially over other labels (Hu et al., 2016; Shi, Zhou, Han, & Liu, 2011; Woźniak & Knoblich, 2019; Woźniak, Kourtis, & Knoblich, 2018; Zhou et al., 2010). At the same time, familiar labels were used as cues indicating associations of each presented movement. Previous research found that the effect of self‐association is present only when associations between stimuli and identities are perceived as task‐relevant (Caughey et al., 2020; Woźniak & Knoblich, 2021). Therefore, we used labels as cues in our task to make remembering color‐identity associations necessary to complete the task and avoid the situation in which participants could use alternative strategies.

A target movement was only considered identical to the memorized movement if it was both the same type of movement (e.g., chopping, painting) and was presented from the same camera position, that is, it was identical in regard to low‐level visual features. Participants were required to make their responses within a time window of 1.5 s after the target movement offsets. Afterwards they were presented with feedback provided as “+ + +” (correct), “− − −” (incorrect), or “0 0 0” (missing response). The inter‐trial interval was either 1 or 3 s, balanced and randomly interleaved across trials.

Each run comprised 48 trials which were balanced with regards to: (a) which movement was retained in working memory, (b) whether the first or the second movement was retained, and (c) which identities were presented as cue 1 and cue 2. The target movement was identical to the memorized movement in 50% of the trials, a different type of movement in 25% (randomly selected) of the trials and in the remaining 25% it was the same type of movement, but presented from a different camera position. The remaining factors were counterbalanced across trials (which movement was presented as a target, which camera position was used for each movement, etc.).

Stimuli

The experiment was programmed and conducted using Matlab (version R2013 8.2). The Psychophysics Toolbox (version 3.0.10) was used for presentation of the experimental task, and Cogent 2000 Toolbox (developed by the Cogent 2000 team at the FIL and the ICN) was used to create log files.

Short movie clips presenting point‐light walkers illustrating eight different movements were chosen from the data set created by (Vanrie & Verfaillie, 2004). All stimuli portrayed full bodies performing movements with their hands while standing (chop, drink, mow, paint, peddle, spade, stir, sweep) from five different camera positions (frontal, 45° left, 90° left, 45° right, 90° right). Because original clips differed in length, only 1 s was selected from each of them. For each participant, four of these eight movement stimuli were randomly selected from our stimulus set, to minimize the potential of stimulus‐specific confounding effects. The assignment of movements to participants was pseudo‐random in a way that ensured that each movement type was used for an equal number of participants.

Visual stimuli were projected with an LCD projector (800 × 600, 60 Hz frame rate) onto a screen on the MR scanner's bore opening. Participants observed the visual stimuli via a mirror attached to the MR head coil from a distance of 110 ± 2 cm. Fixation cross and visual cues were presented in white on a grey background. All identity‐cues were presented with approx. 0.5° height and 1°; the radius of the retro‐cue was 0.2°. The size of point‐light walker stimuli was 3.2 × 4.3°.

Data acquisition and preprocessing

MRI data were acquired using a 3 T TIM Trio scanner (Siemens, Erlangen) at the Center of Cognitive Neuroscience Berlin at Freie Universität Berlin. A structural T1‐weighted MRI image was acquired (MPRAGE; 176 sagittal slices/3D acquisition; TR = 1,900 ms; TE = 2.52 ms; 1 × 1 × 1 mm3 voxel; flip angle = 90°; field of view = 256 mm) when participants were practicing the task during the practice run. Functional MRI data were acquired in five runs, each lasting 15 min 30 s. In each run, 463 images were acquired (T2*‐weighted gradient‐echo EPI; 37 slices; ascending order; 20% gap; whole brain; TR = 2,000 ms; TE = 30 ms; 3 × 3 × 3 mm3 voxel; flip angle = 70°; field of view = 192 mm; 64 × 64 matrix). Trial onsets as well as onsets of retro‐cues and retention periods were time‐locked to the onset of functional image acquisition allowing TR‐wise analysis (see Figure 1).

Data analysis

Preprocessing of the data and GLM analyses were conducted using SPM12 (Wellcome Trust Centre for Neuroimaging, Institute for Neurology, University College London, London, UK). Preprocessing of functional data for decoding analyses was limited to spatial realignment and temporal detrending (Macey, Macey, Kumar, & Harper, 2004), to preserve the spatiotemporal structure of data for the decoding analysis.

Time‐resolved searchlight decoding

We used a time‐resolved multivariate searchlight decoding approach as applied in previous working memory studies (Christophel, Hebart, & Haynes, 2012; Kriegeskorte, Goebel, & Bandettini, 2006; Schmidt, Wu, & Blankenburg, 2017). We performed two independent analyses on the data, in order to determine brain areas that (1) carry information about the memorized movement as such (Movement Decoding: MD) and (2) the identity of the person performing a memorized movement (Identity Decoding: ID).

The analysis was performed as in a series of several similar WM decoding studies to make the current results comparable in the best possible way (e.g., Christophel, Cichy, Hebart, & Haynes, 2015; Schmidt et al., 2017; Schmidt & Blankenburg, 2018; Uluç, Velenosi, Schmidt, & Blankenburg, 2020; Wu et al., 2018). For this purpose we first fitted two independent subject‐level GLMs comprised of finite‐impulse‐response (FIR) regressors (high‐pass filter cut‐off: 200 s) modeling the time period from 2 s before the onset of the retro‐cue to 2 s after the WM retention period, corresponding to six FIR regressors of 2 s each (see Figure 1b). The first GLM comprised regressors for the three identities (You, Pat, or Doc). The second GLM modeled the retention activity with regressors for the different movements irrespective of the identity. This was possible, as the factors movement type and identity were orthogonal in our experimental design. The FIR GLM conducted for Identity Decoding resulted in 95 beta estimates (3 conditions × 6 time bins × 5 runs + 5 constants), and the FIR GLM for Movement Decoding yielded 125 beta estimates (4 stimuli × 6 time bins × 5 runs + 5 constants). Beta estimates were forwarded to the subsequent multivariate pattern analysis.

We used The Decoding Toolbox (TDT) (Hebart, Görgen, & Haynes, 2015), which provides an interface for applying machine learning algorithms to neuroimaging data using LIBSVM library (Chang & Lin, 2011). We combined MVPA with a searchlight routine (Kriegeskorte et al., 2006), which allowed us to conduct data‐driven analysis without any prior assumptions on the localization of information. The spatially‐unbiased searchlight MVPA was performed for each of the six time bins. In the searchlight analysis, a spherical region‐of‐interest (radius = 4 voxels) was moved voxel‐by‐voxel through the measured brain volume. At each position, we used Support Vector Machine (SVM) to test for the decodability of WM contents in a pair‐wise manner (linear kernel with constant regularization parameter c = 1). Beta estimates for each time bin were z‐scaled (normalized) across the samples for each voxel as implemented in TDT. The decoding analyses were performed using a fivefold, leave‐one‐run‐out cross‐validation decoding scheme across the five experimental runs. This procedure yields whole‐brain decoding accuracy maps for each subject and each time bin. Decoding accuracy maps were normalized to MNI space, using unified segmentation and spatially smoothed using a 5 mm FWHM Gaussian kernel, using SPM 12 routines. For the Movement Decoding analysis accuracy maps for each comparison between four movements (six comparisons per participant) were averaged to obtain one decoding accuracy map per timebin. For the Identity Decoding analysis accuracy maps for comparisons between the participant and other identities (You vs. Pat and You vs. Doc) were averaged to obtain one map of You vs. Others comparison per timebin. The accuracy maps for comparison Pat vs. Doc for each timebin were used as a control condition contrasting two stranger identities.

We obtained 6 accuracy maps for Movement Decoding (reflecting decoding in six time bins), and 12 accuracy maps for Identity Decoding: six maps reflecting decoding of self versus others (Pat and Doc), and six maps reflecting decoding of Pat versus Doc. Above chance decoding accuracy maps were entered into flexible‐factorial design group‐level analyses, which corresponds to SPMs realization of group‐level ANOVA designs (Gläscher & Gitelman, 2008). We ran three independent group‐level models: (1) Movement Decoding, (2) Identity Decoding: self versus others, and (3) Identity Decoding: Pat versus Doc. Within these group‐level models we computed t‐contrasts across the time‐bins of interest (2, 3, 4, and 5) against zero, corresponding to testing the mean decoding‐accuracy minus chance across these time‐bins against zero (see Figure 1). Results are reported at p < .05, family‐wise error (FWE) corrected for multiple comparisons, with a minimal cluster extent threshold of 50 voxels.

Control analysis: Decoding of non‐memorized stimulus

In order to test for the specificity of our main decoding analyses, we conducted control analyses to test for above chance decoding of the stimulus which was not retained for working memory (uncued stimulus). New FIR models were estimated which, due to the balanced experimental design, had the same amount of regressors, as each stimulus was equally often memorized as they were presented and not memorized. We performed analogous decoding analyses for the non‐memorized (1) Movement Decoding, (2) Identity Decoding: self versus others, and (3) Identity Decoding: Pat versus Doc.

Univariate analysis

A univariate analysis was conducted in order to identify regions that exhibit increased activity when processing self‐ than other‐related movements. Preprocessing of functional MRI data comprised realignment, coregistration with the structural scan, normalization to MNI space using unified segmentation, smoothing with 5 mm FWHM Gaussian kernel, and temporal detrending (Macey et al., 2004). Data was entered into a standard SPM12 GLM analysis (High‐pass cut‐off: 128 s) including run‐constants to model potential data offsets between runs. Our first‐level design included 10 HRF convolved regressors per run to model for every trial: (1) the stimulus presentation phase before the retro‐cue; (2–4) the retro‐cues, for self‐, Pat‐, and Doc; (5–7) the WM retention period for self, Pat, and Doc; (8) the onset of the target stimulus; and (9,10) participants' responses for left‐ and right‐button presses. Subject‐level contrasts were computed for the self versus other retro‐cue and WM retention periods and tested against zero in a group‐level one‐sample t‐test. An analogous control analysis was conducted for the Pat versus Doc comparison. Results are reported at p < .05 FWE corrected on the cluster level.

RESULTS

Behavioral performance

Participants performed with a mean of 83.6% (SD = 5.7%) correct responses in the applied match‐to‐sample task (range: 70–94.2%). Testing for differences in performance levels (percentage of correct responses) with a one‐way repeated measured ANOVA with factor identity (self, Pat, Doc) revealed a significant main effect of identity (F(2,46) = 5.49, p = .007, partial η 2 = 0.19). Post hoc t‐tests showed that participants were significantly less accurate to recognize a self‐associated movement (M = 81.7%, SD = 5.4%) than a movement associated with Pat (M = 84.5%, SD = 6.3%, t(23) = 3.38, p = .008 Bonferroni‐corrected) or Doc (M = 84.4%, SD = 7.2%, t(23) = 2.68, p = .04 Bonferroni‐corrected), while the difference between Pat and Doc was not significant (t(23) = 0.1, p = 1 Bonferroni‐corrected).

In order to compare accuracies for different types of movements we conducted a linear mixed model (LMM) analysis. Conducting an LMM analysis was necessary in order to account for the crossed‐nested character of this data. The analysis did not reveal any significant differences between movements. We conducted an LMM analysis using Statsmodels Python API with dependent variable being accuracy (percentage of correct responses), each of eight movements used in the experiment as independent variables, and participant as a grouping factor. We did not find any significant differences between accuracies for memorizing different types of movements (all movements were compared against the movement with highest accuracy, i.e., chopping).

Movement decoding

To identify regions where brain activation relates to the movement presented in the stimulus, we computed a t‐contrast to test for above‐chance decoding accuracies during the WM retention. The results (at p < .05 FWE, min. cluster size = 50) are displayed in Table 1 and Figure 2. The largest cluster encompasses a big part of the parietal cortex, as well as occipital and temporal cortices (all bilaterally), with peaks in the left (posterior) middle temporal gyrus, left superior parietal lobe, and right inferior occipital gyrus. The second‐largest cluster spans the bilateral precentral gyrus as well as the supplementary motor area.

TABLE 1

Regions with above‐chance decoding accuracy for movement decoding (a) and identity decoding (b) during the retention period at p < .05, FWE corrected

		Peak MNI coordinates
Cluster size	Anatomical region	x	y	z	z‐score	T
(a) Movement decoding
19,015	Left middle temporal gyrus	−48	−60	4	7.82	9.03
	Left superior parietal lobule	−36	−46	56	7.77	8.95
	Right inferior occipital gyrus	50	−70	6	7.19	8.10
5,587	Left precentral gyrus	−34	−8	58	7.43	8.44
	Right precentral gyrus	30	−2	54	7.11	7.99
	Left supplementary motor cortex	−2	2	58	6.81	7.58
135	Right temporal pole	20	20	−38	6.40	7.04
305	Left inferior occipital gyrus	−30	−94	−22	5.54	5.95
313	R inferior frontal gyrus/white matter	36	16	20	5.03	5.34
185	Left posterior cingulate gyrus	−8	−42	−36	4.86	5.13
293	Right supramarginal gyrus	64	−28	36	4.79	5.05
61	Right exterior cerebellum	36	−62	−56	4.61	4.85
(b) Identity decoding
1,170	Left middle frontal gyrus	−40	4	40	7.46	9.73
	Left inferior frontal gyrus, pars opercularis	−54	24	20	5.33	5.99
68	Left supplementary motor area	−2	14	54	5.21	5.77

Note: Only clusters encompassing at least 50 voxels are displayed.

FIGURE 2

(a) Brain areas representing information about the type of movement held in working memory. Time‐courses of the descriptive evolution of decoding accuracies over the delay period are shown. The peak voxels were selected from the whole‐brain analysis, impeding further statistical assessment. Blue lines represent decoding accuracy from peak voxels for memorized movement. Grey lines represent decoding accuracy from the same areas for non‐memorized movement. The red line represents the 50% chance level of the decoding accuracy maps. (b) Brain areas representing information about whether the movement held in working memory is associated with the self versus others. Blue lines represent decoding accuracy from peak voxels between self and others as a factor of time. Black lines represent decoding accuracy from the same areas between two control (“other”) identities: Pat and Doc. All results displayed at p < .05, FWE corrected, error bars indicate SEM

Time courses of decoding accuracies for the identified regions are displayed in Figure 2. The delayed build‐up of decoding accuracies along with the leveling off until the end of the retention period indicate that the decoding results are specific to a WM representation and do not stem from mere perceptual processing (see also control analyses).

Identity decoding

In order to localize brain areas representing identity‐related information, we computed two decoding analyses: (1) Self versus Other identities, and (2) Pat versus Doc. Brain regions representing self‐related information should show above‐chance level decoding accuracies only in (1), where activation patterns of a self‐related condition are compared with activation pattern elicited by Others. In turn, regions coding self‐related information are assumed to not show above‐chance decoding accuracies when activation pattern of two Other persons are compared. Thereby regions revealed in (1) and not in (2) code not only the difference between two identities, but also the difference between movements encoded for oneself and movements encoded for other persons. A corresponding t‐contrast of decoding accuracies against chance‐level across the WM retention period (p < .05, FWE corrected, min. cluster size = 50) revealed two clusters for self versus others: One cluster spanning the left middle frontal gyrus and the left inferior frontal gyrus, pars opercularis, and a smaller cluster in the left supplementary motor area (Figure 2, Table 1).

Following our a priori hypothesis we inspected the data also at a lower threshold to check for above change decoding accuracy in the TPJ and vmPFC, which might have been missed due to conservative FWE correction threshold. Indeed, this revealed a cluster in the TPJ, more specifically angular gyrus (p < .001, FWE‐corrected on the cluster level; peak coordinates: x = −30, y = −64, z = 48, size = 643 voxels, z = 436). No above chance decoding was found in the vmPFC.

Testing for above‐chance decoding between the two Other‐identities Pat versus Doc yielded no significant clusters. This suggests that the regions revealed in the Self versus Other decoding are specific to the coding of self‐related information.

Control analyses: Decoding of non‐memorized content

We ran three control decoding analyses to test for the specificity of the main results: the (1) Non‐memorized Movement Decoding, (2) Non‐memorized Identity Decoding: self versus others, and (3) Non‐memorized Identity Decoding: Pat versus Doc. These analyses contained the same amount of data and the same processing steps as the three main analyses but tested if information can be decoded on the corresponding non‐memorized stimuli. No significant clusters of activation were found in either of the analyses.

Additionally, we tested for potential interaction effects between movement and identity decoding. For this purpose, we performed an additional analysis decoding for differences between the movements (Movement decoding). However, in contrast to the main Movement decoding analysis, here the trials of one movement type were split according to whether the movement was associated with self, Pat, or Doc. We then tested with a second level design for differences in decoding accuracies for Movement decoding between Self versus Pat, Self versus Doc, and Pat versus Doc. These analyses did not reveal any significant differences, even at a reduced threshold (p < .001, uncorrected) indicating no relevant interaction.

Univariate analysis

Testing in an assumption‐free manner throughout the whole brain for the self > other contrast of the retro‐cue revealed activation clusters in the left vmPFC (x = −6, y = 58, z = 4, cluster size = 113 voxels, z‐score = 3.88, T‐score = 4.67; Figure 3), the left superior frontal gyrus (x = −20, y = 34, z = 42, cluster size = 231 voxels, z‐score = 4.44, T‐score = 5.66) and in the right superior frontal gyrus (cluster1: x = 22, y = 42, z = 44, cluster size = 150 voxels, z‐score = 4.56, T‐score = 5.89), at p < .05 FWE‐corrected on the cluster level (clusters did not survive p < .05 FWE correction for the whole brain). Contrasting self versus other for the WM retention period did not reveal any significant clusters.

FIGURE 3

Presentation of a self‐associated retro‐cue led to increased activity in the ventral medial prefrontal cortex and bilaterally in the superior frontal gyri. The p < .05 FWE‐corrected on the cluster level

Testing for differences between the two stranger‐identity conditions (Pat and Doc, in both directions) did not reveal any significant clusters neither for the retro‐cue, nor throughout the working memory retention period (p < .001 uncorrected).

DISCUSSION

We conducted an fMRI study and used MVPA to test which brain areas code information about whether a memorized movement was represented in WM as performed by oneself or by someone else. In comparison to previous studies, we used MVPA to obtain information maps, reflecting where in the brain the patterns of activation allow to distinguish between the representation of one or another content of working memory. Thereby our study did not only investigate the general involvement of brain regions, but we are able to show that these regions exhibit codes that are specific to the mental representation of a specific working memory content. First, we tested for the brain regions in which activation patterns relate to the retention of specific movements (e.g., chopping, painting, etc.: Movement Decoding). We found regions, related to higher‐order perceptual and multimodal integration processing of movements, such as the superior temporal sulcus, middle temporal cortex, extrastriate body area, intraparietal sulcus, as well as supplementary and premotor/motor cortices. Testing for brain regions that exhibit distinct activation patterns for movements associated with oneself as compared with those associated with another identity (Identity Decoding), our analyses revealed the left middle frontal gyrus, inferior frontal gyrus, and supplementary motor area. Finally, we found increased activation in the vmPFC in response to the presentation of a self‐associated retro‐cue providing further support for previous findings from (Sui et al., 2013; Yankouskaya et al., 2017), as well as in bilateral SFG, however not during the WM retention period.

The introduction of machine learning techniques, especially multivoxel pattern analysis to neuroimaging, improved the sensitivity of analyses to detect fine differences between activation patterns. This has been particularly influential for WM research (Christophel, Klink, Spitzer, Roelfsema, & Haynes, 2017; Riggall & Postle, 2012). The conventional univariate analysis is limited to the detection of the overall mean activation change in a given brain region between two conditions. This approach is often not suitable to assess the neural coding of specific WM contents as the overall activation elicited by different WM contents are often indistinguishable. In contrast, MVPA allows to test if the retention of different WM contents (e.g., stimulus 1 or stimulus 2) lead to different activation pattern within a brain regions. Thereby the results of MVPA searchlight analyses can be interpreted as information maps rather than activation maps. This means that the regions revealed by our analysis relate to the actual content representation of WM. Thereby the contribution of these regions to realize a mental WM representation is considered to go beyond The differences in IFG, MFG, and SMA general support functions, such as attentional mechanisms or cognitive control contributions to WM. These regions are thought to realize the representation of a WM content. Multiple WM MVPA studies indicate that the brain regions that exhibit activity related to specific WM contents partly overlap with the brain regions involved in the perceptual processing of corresponding stimulus material (Lee & Baker, 2016). It has been suggested that the amount of overlap is determined by the degree of abstractness in which contents are retained, for example, in a rather sensory format, like imagining the visual appearance of a shape, or in a rather symbolic or language format, like used when rehearsing a telephone number (Christophel et al., 2017). Our movement decoding analysis extends this literature, by testing what brain regions exhibit codes for the retention of more complex and more naturalistic material, namely specific movements (Galvez‐Pol, Forster, & Calvo‐Merino, 2020). Indeed, the regions revealed by our analysis partly overlap with regions known for perceptual processing of biological motion and action observation (Caspers, Zilles, Laird, & Eickhoff, 2010; E. Grossman et al., 2000; E. D. Grossman & Blake, 2002; Lu et al., 2016; Pelphrey, Morris, Michelich, Allison, & McCarthy, 2005; Saygin, 2007a; Saygin, Wilson, Hagler, Bates, & Sereno, 2004; Vaina, Solomon, Chowdhury, Sinha, & Belliveau, 2001; van Kemenade, Muggleton, Walsh, & Saygin, 2012).

Finding only a small cluster in early visual areas indicates that the mere visual appearance of the stimulus material plays a minor role for the given WM representations, in contrast to the retention of static visual stimulus material (e.g., Albers, Kok, Toni, Dijkerman, & De Lange, 2013; Christophel et al., 2012). As expected, our analysis revealed bilateral clusters in the lateral occipital cortex (LOC), including the motion‐sensitive human middle temporal (hMT) region and the extrastriate body area (EBA), which have been shown to be involved in the visual processing of human body parts (Limanowski & Blankenburg, 2016). The identified cluster extends to the posterior superior temporal sulcus (pSTS), which has been found to be involved in social perception, including being sensitive to biological motion and goal‐directed actions (Pelphrey et al., 2005; Pitcher & Ungerleider, 2020; Saygin, 2007b; Shultz, Lee, Pelphrey, & McCarthy, 2011; Vaina et al., 2001). Superior‐ and intraparietal lobe regions (SPL/IPL) have been found in visual and tactile WM tasks where spatial information was retained, and are well known for their involvement in the remapping of spatial coordinate systems between allocentric and egocentric perspectives, particularly important during movements (Christophel et al., 2012; Heed, Buchholz, Engel, & Röder, 2015; Schmidt & Blankenburg, 2018). Finally, as expected, our analysis revealed pre‐ and supplementary motor areas. These regions have been found to contribute to different types of WM representations (Schmidt et al., 2017; Uluç, Schmidt, Wu, & Blankenburg, 2018), where they are thought to reflect attentional top‐down mechanisms of mental rehearsal processes (compare: Schmidt, Schröder, Reinhardt, & Blankenburg, 2020). These regions are also well known for their contributions to action planning and motor coordination. Therefore, they most likely reflect the mental simulation of the memorized movement (Rizzolatti & Sinigaglia, 2010, but see also: Cook, Bird, Catmur, Press, & Heyes, 2014).

Taken together, the identified regions partly overlaps with regions involved in perceptual processing of dot‐motion movement‐stimuli (Lu et al., 2016; Sokolov et al., 2018) and regions involved in mental simulation of movements, and replicates findings from previous studies (Galvez‐Pol, Forster, & Calvo‐Merino, 2020).

The main goal of our study was to test in which brain area's activation pattern allows us to distinguish whether a memorized full‐body movement was represented as a movement performed by oneself or performed by someone else. In contrast to previous research, we did not investigate self‐recognition or memorization of one's real movements (Bischoff et al., 2012; Jokisch et al., 2006), but coding of movements only attributed to oneself. It means that our study did not target the same processes of action recognition based on visual and motor familiarity as previous research. Our results demonstrate that even an arbitrary movement, if it is interpreted as one's own movement, induces different brain activation patterns than a movement not associated with oneself. We found above‐chance decoding accuracy in one cluster spanning the left middle frontal gyrus and the left inferior frontal gyrus pars opercularis, and a second cluster in the left supplementary motor area. All of these areas (especially the IFG pars opercularis) have previously been identified to be active during action observation and imitation (Caspers et al., 2010; Johnson‐Frey et al., 2003; Molnar‐Szakacs, Iacoboni, Koski, & Mazziotta, 2005), as well as retention in WM of bodily actions (Galvez‐Pol, Calvo‐Merino, & Forster, 2020). Furthermore, there is evidence that IFG pars opercularis and MFG constitute parts of the human mirror neuron system, as indicated by ALE‐meta‐analysis by (Molenberghs et al., 2012). The differences in IFG, MFG, and SMA were revealed by our MVPA, while the corresponding univariate comparison did not reveal differences. This shows that our results do not simply reflect that the activity was greater in these areas, when self‐ than other‐associated movements were retained in WM. Instead, they suggest that neural codes, or the activation pattern within these areas, differ when one observes a movement that is represented as “myself” versus “other”, which can be detected by the higher sensitivity of MVPA as compared with univariate analyses. Importantly, we found these differences in the motor areas, even though participants could have finished the task without representing the movements as self‐related and by just memorizing the labels. However, if that were the case then we would observe no differences between conditions. The fact that we found selective coding of self‐associated movements in motor areas shows that participants completed the task as required, what is further supported by their reports after finishing the study.

All movements used in our study were displayed from the third person (allocentric) perspective, that is, how we usually perceive other people. A situations in which we observe our bodies acting from that perspective is when we see ourselves in a mirror or on a video. However, in these two kinds of situations, we observe ourselves performing movements that we already know from experience—that is, we performed them when they were recorded (in case of a recently recorded video), or that we are performing right now (in case of a mirror). It has been found that activity of the mirror neurons network, including the IFG, is increased when observing actions that are familiar to us (Calvo‐Merino, Glaser, Grèzes, Passingham, & Haggard, 2004; Calvo‐Merino, Grèzes, Glaser, Passingham, & Haggard, 2006; Cross, Hamilton, & Grafton, 2006; Cross, Hamilton, Kraemer, Kelley, & Grafton, 2009). However, in our study, no such effect could directly account for the results, because our design excluded the effect of familiarity of the actions associated with each identity, because for every participant the same actions were equally often associated with all identities across trials. Therefore, our findings in the IFG do not simply reflect motor familiarity, instead they reflect in a well‐controlled manner whether a seen action is represented as self‐ or other‐related.

The second region we expected to be modulated by self‐association was the TPJ. With a more liberal significance threshold we found that activity in the left angular gyrus, which constitutes a dorsal posterior part of the TPJ, differentiated between maintaining representations of self‐ and other‐associated movements. Bilateral TPJ has been suggested as an area that is responsible for implementing domain‐general self‐other distinction, as shown by a large variety of tasks involving self‐body perception, self‐agency, and mindreading (Quesque & Brass, 2019). Some research points to the role of the left TPJ in third‐person self‐representation (Ganesh et al., 2011), but more research is needed to fully establish this link. Previous studies on perception of actions performed by self or others that have identified TPJ activity mainly used videos and animations of participants' real movements—stimuli that were portraying highly familiar movement patterns (e.g., Bischoff et al., 2012; Macuga & Frey, 2011; Sugiura et al., 2006). These studies typically argued that the role of TPJ in self‐other distinction relies on detecting motor familiarity, and specifically on comparing predicted and observed body movements. However, because all animations used in our study reflected kinematics that were equally unfamiliar to participants (none of them was based on their real kinematics), our results support the view that the TPJ's role in representing the distinction between self and others is more domain‐general as advocated by Quesque and Brass (2019), and is not limited to a mechanisms that rely on familiarity of observed movements.

In contrast to previous studies on processing of movements attributed to oneself and others, in our study we did not use participants' real movements. Nevertheless, we found the effects of self‐attribution in similar brain areas to the ones that have been reported in studies using familiar self‐related bodily stimuli (IFG, MFG, TPJ). However, our results showed predominantly left‐lateralized effects, while previous reports on self‐related processing usually show right‐lateralized effects (Devue et al., 2007; Morita, Asada, & Naito, 2020), with less frequent reports of left‐lateralized effects. This difference in lateralization might be caused by familiarity. Indeed, the right hemisphere is more strongly implicated in cognitive processing of familiarity (Cross et al., 2009; Gainotti, 2007), meaning that right‐lateralization effects from previous studies might be driven by higher familiarity of self‐related than other‐related stimuli. However, more work is needed and there might be alternative explanation of the lateralization effects. For example, Ganesh et al. (2011), reported on neural correlates of self‐identification with a computer game avatars, and found increased avatar‐referential activity selectively in the left angular gyrus (Ganesh et al., 2011). The authors interpret their finding by suggesting that the left hemisphere might be more involved in self‐identification from the third‐person perspective (like in our study), while the right hemisphere dominates identification from the first‐person perspective.

The fact that our study involved unfamiliar stimuli might have had one more consequence: we found that self‐attributed movements led to different pattern of neural responses as uncovered by MVPA, but not to an overall increase of activity in these areas. This might suggests that the univariate results from previous studies could have been to certain extent strengthened by increased familiarity of presented stimuli.

Surprisingly, in our behavioral data we found that participants performed slightly worse in the forced‐choice task at the end of the WM delay period, when memorizing self‐associated movements than when they were memorizing other‐associated movements. Previous work has shown that self‐related material is processed in a privileged way and one could therefore expect a higher performance for WM of self‐related material also in our study (Sui & Humphreys, 2015; Symons & Johnson, 1997). However, previous studies showing higher accuracy for processing of arbitrarily self‐associated information used mostly abstract stimuli. This shows that associating arbitrary abstract information with the self does not lead to interference with one's conceptual self that would cause decrease in task performance. One interpretation that could explain why we found an opposite effect would be that the unfamiliar movements that had to be actively associated with oneself might cause interference with sensorimotor knowledge about one's real body dynamics. Therefore, the WM representation might have been influenced by one's own sensorimotor body representation underpinning the bodily self (De Vignemont, 2010, 2018; Dewey & Knoblich, 2016; Gallagher, 2005; Kanayama & Hiromitsu, 2021; Schwoebel & Coslett, 2005), which would make it more difficult to match the memorized movement with the comparison stimulus. This would also explain why previous reports did not find similar interference when participants were told to self‐identify with avatar faces (Payne et al., 2017; Woźniak et al., 2018; Woźniak & Hohwy, 2020)—static images of faces do not map onto sensorimotor bodily self representation like the biological motion does. This difference in performance should not have affected the main neuroimaging results, as a control analysis modeling only the correct trials confirmed above chance decoding in the three reported regions.

We did not find distinctive representations of self, in terms of differentiable activation patterns detectable by MVPA, during the WM retention period in one area in which we expected to find them: in vmPFC. Previous studies found that aspects of the conceptual self (self‐related adjectives, autobiographical memories, etc.) are usually associated with increased activity in midline structures (Qin et al., 2020; Qin & Northoff, 2011): precuneus and vmPFC, the latter of which is often regarded as the central node for self‐related processing (for a discussion see: D'Argembeau, 2013; Martinelli et al., 2013; Northoff et al., 2006). However, studies which investigated aspects of the self related to one's body, rather than personality, rarely find modulation of activity in the vmPFC or other midline structures (Gillihan & Farah, 2005; Hu et al., 2016; Legrand & Ruby, 2009; Qin et al., 2020). A possible interpretation of these results is that midline structures, in particular vmPFC, are more relevant for abstract and conceptual (e.g., semantic) information about the self than for the less abstract and more naturalistic bodily information associated with the self used in the study at hand.

Univariate results

In order to test if we can replicate previous reports on activation differences in vmPFC, when self‐related or non‐self‐related cues are processed, we additionally computed a corresponding univariate contrast for our data. In contrast to the multivariate decoding analysis which reveals differences between patterns of activation, in univariate analyses, which detect overall activity increases or decreases in a region, we did find modulation of activation in the vmPFC. Stronger retro‐cue induced activation was found for self‐associated retro‐cues as compared with non‐self related cues. In our study, the retro‐cue displayed one of the colors that prior to the experiment were associated with the self or the other identities. As such, it served the function of an abstract symbol referring to the self or others, in the same way as geometrical shapes that have been used in previous fMRI research on processing of arbitrary self‐associated stimuli (self‐prioritization effect: Sui et al., 2013; Yankouskaya et al., 2017, see also: Lockwood et al., 2018; Yin et al., 2021). Moreover, the specific vmPFC cluster detected in our MVPA analysis directly overlapped with the cluster found in previous studies, investigating univariate activation differences for processing of self‐associated shapes (Sui et al., 2013). This pattern of results replicates previous findings and support the interpretation that vmPFC is primarily associated with self‐related processing of conceptual aspects of the self, and not necessarily bodily aspects. Moreover, the finding that self‐association is related to increased activation in vmPFC after presentation of the retro‐cue, but not throughout the whole retention period, suggests that vmPFC is involved in selecting self‐associations for retention in working memory (Myers, Stokes, & Nobre, 2017), but not in maintaining them. This interpretation is consistent with previous research on the influence of self‐relatedness on memory, which indicates that vmPFC is involved in encoding and retrieval, but not maintenance of self‐related information (Kalenzaga et al., 2015; Leshikar & Duarte, 2012; Philippi, Duff, Denburg, Tranel, & Rudrauf, 2012; Yaoi, Osaka, & Osaka, 2015).

CONCLUSIONS

We conducted a study in which we tested which brain areas display different multivoxel activation patterns depending on whether a memorized action (seen from the third‐person perspective) is represented as being performed by oneself or by someone else. Crucially and in contrast to previous studies investigating this topic, as the same movements were associated with the self and with other identities in different trials, our study design rendered confounding influence of motor and perceptual familiarity unlikely. Our findings indicate that self‐associated movements are differentially represented during the working memory retention period through left hemispheric activation patterns in frontal areas responsible for action recognition and motor mirroring (MFG, IFG, and SMA). When testing for univariate activation differences, we further found increased activity in the vmPFC directly after the presentation of the retro‐cue that instructed to recall a self‐associated movement but no difference across the working memory retention period. This suggests that the influence of the vmPFC in our task might be limited to selecting a self‐associated movement for working memory, and not for maintaining it in working memory.