The brain network in support of social semantic accumulation.

Some studies have indicated that a specific 'social semantic network' represents the social meanings of words. However, studies of the comprehension of complex materials, such as sentences and narratives, have indicated that the same network supports the online accumulation of connected semantic information. In this study, we examined the hypothesis that this network does not simply represent the social meanings of words but also accumulates connected social meanings from texts. We defined the social semantic network by conducting a meta-analysis of previous studies on social semantic processing and then examined the effects of social semantic accumulation using a functional Magnetic Resonance Imaging (fMRI) experiment. Two important findings were obtained. First, the social semantic network showed a stronger social semantic effect in sentence and narrative reading than in word list reading, indicating the amplitude of social semantic activation can be accumulated in the network. Second, the activation of the social semantic network in sentence and narrative reading can be better explained by the holistic social-semantic-richness rating scores of the stimuli than by those of the constitutive words, indicating the social semantic contents can be integrated in the network. These two findings convergently indicate that the social semantic network supports the accumulation of connected social meanings.

Introduction

An important finding of cognitive neuroscience is that the brain network supporting semantic representation is partially organized according to information types (Martin, 2007; Mahon and Caramazza, 2009; Binder ). Sensory-motor semantic information and social semantic information are the most salient information types that constrain the organization of the semantic system in brain, which are supported by two separate semantic subsystems (Huth ; Lin ). The semantic subsystem that selectively supports social semantic representation is referred to as the social semantic network, which includes the bilateral anterior temporal lobes (ATLs), temporoparietal junction (TPJ)/angular gyrus, dorsomedial prefrontal cortex (DMPFC) and posterior cingulate (PC)/precuneus (Lin , 2020). They show strong activation during the processing of words with rich social meanings (Lin , 2018a; Wang ), and their activities can be used to decode the social semantic contents being processed (Huth ; Thornton and Mitchell, 2018). It was proposed that these areas represent the social concepts underlying word meanings, which is a part of semantic memory (Binder ; Lin ).

The neuroimaging studies on the comprehension of complex materials, such as sentences and narratives, however, have indicated that the brain areas of the social sematic network may support the online accumulation of connected semantic information. The effect of semantic accumulation on brain activation has been primarily revealed by studies that manipulated the size of the semantically continuous structures embedded in the stimuli. In an early study, Xu compared the brain activations evoked by the narratives, unconnected sentences and word lists in a reading task and found that the ATL showed stronger activation to unconnected sentences than to word lists, and the DMPFC, precuneus and TPJ showed stronger activation to narratives than to unconnected sentences. Lerner used a design similar to Xu in a listening task, but focused on the intersubject correlation (ISC) of the BOLD response time courses instead of the strength of brain activity. They also found the effects of linguistic hierarchies in several brain areas: in the posterior superior temporal gyrus, a significant ISC was observed when listening to sentences, paragraphs and stories but not word lists; in the TPJ and precuneus, a significant ISC was observed when listening to paragraphs and stories, but not to sentences or word lists; and in the medial prefrontal cortex, a significant ISC was observed only when listening to complete stories. Pallier demonstrated a more fine-grained semantic accumulation effect by continuously manipulating the size of sentential constituents (1 word, 2 words, 3 words, 4 words, 6 words and 12 words) embedded in a stream comprising 12 written words. They found that the activation of the ATL and TPJ increased parametrically with the constituent size in both amplitude and phase. Importantly, the constituent-size effect in the ATL and TPJ disappeared when the content words were replaced with pseudowords of the same morphological endings, indicating that the effect reflected semantic rather than syntactic accumulation. Mellem replicated the finding of Pallier and found an overlap between the constituent-size effect and the effect of social–emotional semantic processing in the left ATL. In addition to linguistic hierarchy and constituent size, the brain areas associated with semantic accumulation are also sensitive to factors that influence the holistic comprehension of stimuli, such as the powerfulness of a political speech (Schmälzle ) and subtle word changes that alter the interpretation of a story (Yeshurun ).

In the two aforementioned lines of studies, the two features of the social semantic network, i.e. being sensitive to social semantic information and being sensitive to connected semantic information, were attributed to social concept representation and domain-general semantic accumulation, respectively. Here we propose that these two features may both be associated with a single cognitive function, i.e. the accumulation of connected social semantic information. We will refer to this function as social semantic accumulation for short. We assume that, during text comprehension, social semantic accumulation starts by representing the social meanings of the initial word and then accumulates and integrates the connected social meanings from the following texts. This hypothesis can explain the findings of both aforementioned lines of studies: because social semantic accumulation starts with representing the social meanings of words, it can explain the sensitivity of the network to social semantic information in word comprehension tasks; because the previous studies of semantic accumulation typically used stimuli containing rich social semantic information, the existing evidence for semantic accumulation can also be viewed as evidence for social semantic accumulation.

In this study, we examined two novel predictions of our hypothesis of social semantic accumulation: first, in the social semantic network, the amplitude of social semantic activation accumulates along with the processing of connected social meanings, exhibiting linguistic hierarchical differences (narrative > sentence > word); second, during text comprehension, the activation of the social semantic network can be better explained by the holistic social meanings of the stimulus than by the word-level social meanings. The confirmation of these predictions would indicate that the social semantic processing occurring in the social semantic network during text comprehension is not simply the retrieval of the social meanings of words but rather involves social semantic accumulation.

Methods

Participants

In total, 36 healthy undergraduate and graduate students (22 females) participated in the functional Magnetic Resonance Imaging (fMRI) experiment. The mean age of the participants was 21.2 years (SD = 2.5 years). All participants were right-handed and native Chinese speakers. None of the participants had suffered from psychiatric or neurological disorders or had ever sustained a head injury. All protocols and procedures were approved by the Institutional Review Board of the Magnetic Resonance Imaging Research Center of the Institute of Psychology of the Chinese Academy of Sciences, and each participant read and signed an informed consent form before the experiment. In the data analysis, the data of three subjects (two females) were discarded due to excessive head movement (>3.0 mm or 3.0° in any direction). Thus, the data analyses were based on the remaining 33 participants.

Design and materials

In the fMRI experiment, we manipulated the social semantic richness (high/low) and linguistic hierarchies (word/sentence/narrative) of the stimuli. Therefore, the experiment contained six conditions, namely, the high and low social-semantic-richness word-list conditions, sentence conditions and narrative conditions.

Both high and low social-semantic-richness narrative conditions contained 42 narratives, with each narrative consisting of four sentences. We obtained the social semantic richness scores of these materials at the narrative, sentence and word levels using three rating experiments (see Supplementary Materials for details). We carefully matched a series of variables between the high and low social-semantic-richness narratives, which include the sentence-level and narrative-level semantic plausibility, the coherence of narratives, the number of words per narrative and per sentence, the number of characters per narrative, per sentence, and per word, and the word frequency (Table 1). The high and low social-semantic-richness narrative stimuli were both randomized into three sets, with each set of stimuli containing 14 narratives. For each set of narrative stimuli, corresponding sets of sentence stimuli and word-list stimuli were constructed. Therefore, both the high and low social-semantic-richness stimuli were separated into three sets, with each set having three versions, i.e. the narrative version (14 narratives), the sentence version (14 sentence lists) and the word-list version (14 groups of word lists). In the fMRI experiment, only one version of each set of stimuli was presented to a participant, which corresponded to one of the six experimental conditions. The uses of the three different versions of the three sets of stimuli were counterbalanced across participants. In the Supplementary Materials, we detailed how the social semantic richness and control variables of the stimuli were manipulated and controlled and how the sentence and word-list stimuli were constructed based on the narrative stimuli.

Table 1.

Variables that were manipulated or controlled in the high and low social-semantic-richness narrative stimuli

	High social-	Low social-	High social-
	semantic-	semantic-	semantic-richness
	richness	richness	vs low social-
	narratives	narratives	semantic-richness
			t	P
Narrative-level variables
Social-semantic-richness	5.75 ± 0.51	1.42 ± 0.37	44.482	0.000
Coherence	6.68 ± 0.27	6.71 ± 0.25	0.393	0.695
Semantic plausibility	6.4 ± 0.27	6.4 ± 0.39	0.061	0.952
Number of sentences per narrative	4 ± 0	4 ± 0	–	–
Number of words per narrative	28.71 ± 1.2	28.79 ± 1.26	0.267	0.790
Number of characters per narrative	48.43 ± 1.7	48.31 ± 1.49	0.342	0.734
Sentence-level variables
Social-semantic-richness	4.79 ± 0.99	1.39 ± 0.33	42.252	0.000
Semantic plausibility	6.78 ± 0.2	6.77 ± 0.42	0.311	0.756
Number of words per sentence	7.18 ± 0.93	7.19 ± 0.94	0.058	0.953
Number of characters per sentence	12.11 ± 1.15	12.08 ± 1.13	0.239	0.811
Word-level variables
Social-semantic-richness	2.97 ± 1.69	1.69 ± 0.7	48.166	0.000
Number of characters per word	1.69 ± 0.52	1.68 ± 0.55	0.383	0.702
Log (word frequency + 1)	1.82 ± 1.33	1.83 ± 1.37	0.145	0.885

Notes: The word-level social-semantic-richness values of the high and low social-semantic-richness narratives shown in the table are the average social-semantic-richness rating scores of all constitutive words of the two types of narratives. Although both types of narratives contain a considerable proportion of low social-semantic-richness words (e.g. function words), the high social-semantic-richness narratives contain a much larger proportion of high social-semantic-richness words (social-semantic-richness rating score ≥5) than do the low social-semantic-richness narratives (proportion of high social-semantic-richness words: high social-semantic-richness narratives: 226/1206 (18.74%); low social-semantic-richness narratives: 2/1209 (0.17%)).

Procedures

The fMRI experiment employed a block design, containing three runs of 10 min and 26 s each. Each run included 28 blocks, with four or five blocks for each condition. In total, each condition includes 14 blocks in the experiment. The numbers and orders of the blocks for the six conditions were counterbalanced across runs and participants. In the first 10 s of each run, participants were shown a fixation. They then performed a silent reading task in which they were shown a narrative, a sentence list (four unconnected sentences) or a group of word lists (four word-lists) in each block. In each block, each sentence or word list appeared for 3 s. Each block lasted for 12 s, followed by a 10 s fixation.

To make sure the participants could pay attention to the stimuli during the scanning, they were told to complete a recognition test to evaluate their performance after scanning. The stimuli of the recognition test included all stimuli that the participants had seen in the scanner and an equal number of stimuli that were never used in the fMRI experiment. All stimuli were presented in blocks as in the fMRI experiment, except that the fixation between blocks was shortened to 0.5 s. Participants were asked to indicate whether they believed the block of stimuli they saw had been presented in the fMRI experiment by pressing buttons.

Image acquisition and preprocessing

Structural and functional data were collected using a GE Discovery MR750 3 T scanner at the Magnetic Resonance Imaging Research Center of the Institute of Psychology of the Chinese Academy of Sciences. T1-weighted structural images were obtained using a spoiled gradient-recalled pulse sequence in 176 sagittal slices with 1.0 mm isotropic voxels. Functional blood-oxygenation-level-dependent data were collected using a gradient-echo echo-planar imaging sequence in 42 near-axial slices with 3.0 mm isotropic voxels (matrix size = 64 × 64; repetition time = 2000 ms; echo time = 30 ms).

The fMRI data were preprocessed using Statistical Parametric Mapping software (SPM8; http://www.fil.ion.ucl.ac.uk/spm/). For the preprocessing of the task fMRI data, the first five volumes of each functional run were discarded to reach signal equilibrium. Slice timing and 3-D head motion correction were performed. After that, a mean functional image was obtained for each participant, and the structural image of each participant was coregistered to the mean functional image. Then, the structural image was segmented using the unified segmentation module (Ashburner and Friston, 2005). The parameters obtained during segmentation were used to normalize the functional images of each participant into the Montreal Neurological Institute space. Functional images were subsequently spatially smoothed using a 6 mm full-width-half-maximum Gaussian kernel.

Data analysis

Defining the social semantic network: a meta-analysis

We conducted an activation likelihood estimation (ALE) meta-analysis to define the social semantic network. A literature search was conducted on the Web of Knowledge (www.isiknowledge.com). The inclusion criteria are detailed in the Supplementary Materials. In total, we collected 95 activity peaks from the 10 included studies (Table 2). We then conducted ALE meta-analysis based on these data using GingerALE 3.0.2 (Eickhoff ). The coordinates reported in the Talairach space were transformed into the Montreal Neurological Institute (MNI) space using the Convert Foci function of the GingerALE. The results of this ALE meta-analysis then served as the regions of interests (ROIs) of our data analysis.

Table 2.

Studies and peak coordinates included in the ALE meta-analysis

Study	Subject number	Task	Contrast	MNI coordinates
				x	y	z
Binney et al. (2016)	19	Semantic relatedness judgment	Social > animal	−48	9	−39
				−57	9	−12
				−39	3	−48
				−15	−87	−9
				3	−84	−6
				−33	−72	−9
				−12	−78	24
				−27	−78	24
				−15	−87	30
				−54	−39	21
				−48	−24	18
				−60	−21	18
Contreras et al. (2012)	19	Categorical knowledge judgment	Social > non-social	−4	−58	28
				−8	56	34
				−4	48	−8
				−50	−10	−22
				60	−2	−22
				−56	−60	24
				56	−56	18
				−12	−96	−4
				−26	−74	−16
				−10	38	50
		Feature verification	Person > object	−2	58	22
				−56	−4	−24
				38	22	−22
				−56	−64	26
				54	−12	−32
				−40	20	−16
				−4	0	6
				44	30	−8
				62	18	16
Lin et al. (2015)	15	Semantic relatedness judgment	Social > private	−57	−10	−8
				50	23	24
				60	−31	−1
				−45	21	−22
				−5	55	31
				−11	−44	35
				4	47	−18
				−18	−83	−2
			Social > nonhuman	15	40	43
				59	6	−18
				8	−54	32
				−51	2	−23
				−41	−49	19
				−41	25	−19
Lin et al. (2018a)	19	Semantic relatedness judgment	High social-semantic-richness verb > low social-semantic-richness verb	−42	12	−36
				−9	51	36
				−48	−60	21
				45	21	−33
				51	−57	21
				−3	−51	21
Lin et al. (2019)	20	Semantic relatedness judgment	High social-semantic-richness noun > low social-semantic-richness noun	−57	−3	−24
			High social-semantic-richness verb > low social-semantic-richness verb	−57	0	−21
				54	0	−18
				−51	−66	21
				0	−57	21
				−12	60	33
				51	−54	15
Mason et al. (2004)	17	Feature verification	Person > dog	38	42	30
				30	23	43
				12	56	28
				8	27	37
Mitchell et al. (2002)	14	Feature verification	Person > object	1	61	13
				4	43	−9
				14	40	−9
				34	−53	−3
				70	−30	34
				66	−31	21
				53	−64	14
				−64	−5	−6
				−64	−13	−16
				−70	−26	−14
				−70	−19	−18
				−47	−21	69
				−30	−34	67
				−34	−30	76
				−30	−20	75
				−53	−76	27
				−14	−102	29
Ross and Olson (2010)	15	Semantic relatedness judgment	Social > animal	66	−10	−24
				−51	16	−28
				−32	−77	−15
Wang et al. (2019)	22	Semantic relatedness judgment	Social > non-social	−58	−4	−16
				−44	−72	28
Zahn et al. (2007)	26	Semantic relatedness judgment	Social > animal	48	21	−9
				57	12	0
				54	33	6
				−6	21	54
				−36	33	24
				−48	15	9
				−57	−45	30
				−63	−39	−12
				−42	−51	−30
				−33	−84	12
				−12	−15	−3

Note: The coordinates reported in the Talairach space were transformed into the MNI space using the Convert Foci function of the GingerALE.

One limitation of our ALE meta-analysis is that it included only a small number of studies. To verify the results of the ALE meta-analysis, we conducted a supplementary meta-analysis using Neurosynth (neurosynth.org; Yarkoni ), which is based on a much larger data set. First, we conducted two separate Neurosynth meta-analyses using the terms ‘social’ and ‘semantic’ (using the default settings of the Neurosynth: association test; false discovery rate criterion of 0.01). These two terms yielded 1302 and 1031 studies (47 083 and 40 030 activations), respectively. We then computed the overlap of the brain maps from the two results and using this overlap to reflect the distribution of the social semantic network. This overlapping analysis was based on two assumptions. First, the social semantic network should be activated in most social tasks because accessing social semantic knowledge is a fundamental component of social cognition. Second, the social semantic network should be activated in a considerable proportion of semantic studies because social knowledge is a basic and broad type of semantic information. However, this second assumption suffers from a risk that the dataset of semantic studies may possibly have a bias towards focusing on some non-social types of knowledge, such as object knowledge. Therefore, this overlapping analysis is not guaranteed to fully reveal the distributions of the social semantic network and was only used as a supplementary method.

Modelling the effects of social semantic accumulation

Statistical analyses of the fMRI data were performed according to 2-level, mixed-effects models implemented in SPM8, focusing on two predictions. First, in the social semantic network, the amplitude of social semantic activation accumulates along with the processing of connected social meanings, exhibiting linguistic hierarchical differences (narrative > sentence > word). Second, during text comprehension, the activation of the social semantic network can be better explained by the holistic social meanings of the stimulus than by the simple additivity of the word-level social meanings. These two predictions were examined using two different modelling methods.

The first prediction was examined using the classic contrast-based modelling analysis. In this analysis, we modelled the social semantic activation as the additional activation evoked by the high social-semantic-richness stimuli over that evoked by the low social-semantic-richness stimuli. The social semantic accumulation effect was reflected by the additional social semantic activation in sentence and narrative conditions over that in the word-list conditions and additional social semantic activation in the narrative conditions over that in the sentence conditions. The underlying logic of this method is derived from the previous studies using the same or similar paradigm to study the domain-general semantic accumulation effect, in which the effect of semantic accumulation was modelled as the additional activation evoked by sentences over word-lists and by narratives over sentences (Xu ; Pallier ; Mellem ).

Specifically, at the first level, a general linear model was built by including the six conditions as covariates of interest. Each block of stimuli was modelled with a boxcar waveform lasting 12 s. Six head motion parameters obtained by the head motion correction were included as nuisance regressors. A high-pass filter (128 s) was used to remove low-frequency signal drift. The results of the first level analysis were then entered into the second-level random-effects analysis. We primarily focused on the data within the ROIs. For each participant and condition, the voxel-based beta values obtained in the first-level analysis were averaged within each ROI. The social semantic activations in word, sentence and narrative reading were then modelled as the beta differences between the high and low social-semantic-richness conditions at each hierarchy. We examined the social semantic activations in word, sentence and narrative reading using a one-sample t-test against zero and examined the social semantic accumulation effect by comparing the social semantic activations in the word, sentence and narrative conditions using paired t-test.

The second prediction was examined using the parametric modulation approach implemented in SPM8. At the first level, we merged the high and low social-semantic-richness conditions at each linguistic hierarchy into a single condition. To better capture the continuous changes of the social semantic richness within each block, we modelled the BOLD response to the stimuli according to the presence of each sentence and word list. For each condition, the presence of each sentence or word list was modelled using a constant regressor lasting 3 s, and the social semantic effects were modelled as the interactions between the presence of a sentence/word list and a number of parametric social-semantic-richness modulators associated with it. The number of parametric social-semantic-richness modulators varied across conditions. For the narrative condition, three parametric social-semantic-richness modulators were set, which are computed based on the narrative-, sentence-, and word-level social-semantic-richness scores obtained in the aforementioned rating experiments. The narrative-level social-semantic-richness modulators of the four sentences of a narrative were all set using the narrative-level social-semantic-richness rating score of the narrative. The sentence-level social-semantic-richness modulator of each sentence was set using its sentence-level social-semantic-richness rating score. The word-level social-semantic-richness modulator of each sentence was set as the average word-level social-semantic-richness rating score of all its constitutive words. For the sentence condition, only the sentence- and word-level social-semantic-richness modulators were set. For the word-list condition, only the word-level social-semantic-richness modulator was set.

We then analysed the social semantic effects using two models: in Model 1, the high-level social-semantic-richness modulators were orthogonalized with respect to the low-level ones so that the shared variability of the regressors was assigned to the low-level social-semantic-richness modulators; in Model 2, the low-level social-semantic-richness modulators were orthogonalized with respect to the high-level ones so that the shared variability of the regressors was assigned to the high-level social-semantic-richness modulators. For both models, the modulation effect of each parametric modulator was examined using a one-sample t-test against zero in the second-level analysis. The results of the two models indicate whether the high and low levels of social-semantic-richness modulators can explain additional variability of the activation over each other (see Supplementary Materials for more details of this analysis).

One possible problem of the parametric modulation analysis is that the social semantic richness of words varies systematically across grammatical categories so that when processing the social meaning of words, people may selectively focus on particular grammatical categories and ignore others. In this case, averaging the social-semantic-richness scores of all constitutive words of a sentence may dilute the effect of word-level social semantic richness. In literature, the word-level social semantic effect has been observed in three grammatical categories of words, which include adjectives (Mitchell ; Zahn ), verbs (Lin , 2018a) and nouns (Lin ; Wang ). Therefore, we conducted a second parametric modulation analysis in which we only included the social-semantic-richness scores of these three categories of words in the computation of the word-level social-semantic-richness modulator, and ignored the other words. The classification of the grammatical categories of words was mainly based on the Language Corpus System of Modern Chinese Studies (Sun ). For the 159 low-frequency words that were not included in the corpus, three authors (Guangyao Zhang, Meimei Zhang, and Nan Lin) together decided their grammatical categories. In total, the social-semantic-richness scores of 819 of the original 962 non-repetitive words were included in the analysis.

In addition to the ROI-based analyses, we also conducted whole-brain activation analysis. The major aim of the whole-brain analysis was to enable us to compare the social semantic effect and the sentence and narrative effects observed in the present study with those observed in previous studies. We also conducted a psychophysiological interaction (PPI) analysis to explore the task-modulated connectivity between the areas of the social semantic network. The methods of the whole-brain activation analysis and the PPI analysis are detailed in the Supplementary Materials.

All brain maps of our results were visualized using the BrainNet Viewer software (Xia ).

Results

Behavioural results of the post-scan recognition test

The participants showed considerable recognition accuracy in the post-scan recognition test (narratives: 81.8%; sentences: 78.5%; words: 66.2%), indicating that they had paid attention to the reading task. The accuracy data showed a strong linguistic hierarchical effect: the differences between each two of the three linguistic hierarchies were all significant, with the narrative stimuli being recognized best and the word stimuli being recognized worst (narrative vs sentence: t[32] = 2.367, P = 0.024; sentence vs word: t[32] = 7.885, P < 0.001; narrative vs word: t[32] = 7.633, P < 0.001). Because we did not manipulate the social semantic richness of the unfamiliar stimuli, the analysis of the social semantic effect was conducted within the familiar trials. A significant difference between high and low social-semantic-richness conditions was found in the narrative recognition (high social-semantic-richness condition vs low social-semantic-richness condition: 85.9% vs 80.7%, t[32] = 2.512, P = 0.017) but not in sentence or word recognition (ts < 1). The reaction time data showed no significant difference in any analysis.

fMRI results

The results of the meta-analysis for defining the ROIs of the social semantic network

As shown in Figure 1 and Table 3, the ALE meta-analysis revealed six significant clusters (thresholded at whole-brain cluster-level permutation corrected P < 0.05, voxel-wise P < 0.001). The clusters were located at the bilateral ATL, TPJ, PC and the left DMPFC. These clusters were defined as the ROIs for the fMRI data analysis. The overlapping of the Neurosynth results of the social and semantic networks revealed surprisingly similar results, despite using highly different datasets and methods: five of the six regions (the bilateral ATLs, left TPJ, left SFC and PC) revealed by the ALE analysis were also revealed by the Neurosynth overlapping analysis, confirming the reliability of the ROIs and indicating that the social semantic network is located at the junction of the semantic and social networks, serving as a component of both of them.

Fig. 1.

Results of the ALE meta-analysis and the overlapping analysis. Panel A: the result of the ALE meta-analysis of 10 fMRI studies of social concept processing. Panel B: the overlap of the results of the Neurosynth meta-analyses using the terms ‘social’ and ‘semantic’.

Table 3.

Results of the ALE meta-analysis

Cluster	Volume (mm3)	ALE	Centre			Maximum ALE value			Anatomical label
			x	y	z	x	y	z
1	1600	0.018	−56.3	−2.8	−20.2	−56	−4	−24	LATL
2	1088	0.019	−8.4	55.5	33.5	−8	56	34	LDMPFC
3	976	0.017	52.9	−56.7	17.6	52	−56	18	RTPJ
4	832	0.013	−51.7	−63	22.4	−50	−62	22	LTPJ
5	600	0.012	57.5	0.8	−19.5	58	0	−20	RATL
6	576	0.012	−1.9	−55.4	22.8	−2	−54	22	PC

Note: Anatomical labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.

The social semantic accumulation effect as reflected by the contrast-based analysis of the ROI data

The results of the contrast-based analysis are shown in Figure 2, Tables 4 and 5. In sentence and narrative reading, social semantic activations (high social-semantic-richness > low social-semantic-richness) were found in all ROIs, whereas in word-list reading, social semantic activation was found only in the bilateral ATL and the left DMPFC. The social semantic activations in sentence reading were stronger than in word-list reading in all ROIs except the right TPJ, and the social semantic activations in narrative reading were stronger than those in word-list reading in all ROIs except the left DMPFC. These findings indicate that in the social semantic network, the amplitude of the social semantic effect accumulates along with sentence processing. No ROIs showed a significant difference between the social semantic activations in sentence reading and those in narrative reading. Therefore, the results of the contrast-based analysis provide no evidence that the social semantic network supports the narrative-level social semantic accumulation.

Fig. 2.

ROI results of the contrast-based analysis. The brain map shows the locations of the ROIs. The bar plot shows the social semantic effect at the three linguistic hierarchies for each ROI; error bars represent the standard errors. Condition labels: HSN, high social-semantic-richness narrative; HSS, high social-semantic-richness sentence; HSW, high social-semantic-richness word; LSN, low social-semantic-richness narrative; LSS, low social-semantic-richness sentence and LSW, low social-semantic-richness word. ROI labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.

The social semantic accumulation effect as reflected by the parametric modulation analysis of the ROI data

The parametric modelling analysis showed that, in sentence and narrative reading, the social-semantic-richness modulator at the holistic level performed better than at the constitutive levels in explaining the activation of the social semantic network. The results of the parametric modelling analysis that considered all words in calculating the word-level social-semantic-richness modulator are summarized in Table 6. In word-list reading, the word-level social-semantic-richness modulators explained the activation of four ROIs, which included the left ATL, the right ATL the left TPJ and the left DMPFC. In sentence reading, both word-level and sentence-level social-semantic-richness modulators alone explained the activation of all ROIs. The sentence-level social-semantic-richness modulator explained the activation of all ROIs, even after being orthogonalized with respect to the word-level social-semantic-richness modulator (Model 1), whereas the word-level social-semantic-richness modulator no longer explained the activation of any ROI after being orthogonalized with respect to the sentence-level social-semantic-richness modulator (Model 2). Similarly, in narrative reading, both word-level and narrative-level social-semantic-richness modulators alone explained the activation of all ROIs. The narrative-level social-semantic-richness modulator explained the activation of three ROIs (the bilateral TPJ and the right ATL), even after being orthogonalized with respect to the word-level and sentence-level social-semantic-richness modulators (Model 1), while the word-level social-semantic-richness modulator no longer explained the activation of any ROI after being orthogonalized with respect to the narrative-level and sentence-level social-semantic-richness modulators (Model 2). In addition, in narrative reading, the sentence-level social-semantic-richness modulator explained the activation of four ROIs (the bilateral ATL, the left TPJ, and the left DMPFC) after being orthogonalized with respect to the word-level social-semantic-richness modulator (Model 1) but no longer explained the activation of any ROI after being orthogonalized with respect to the narrative-level social-semantic-richness modulator (Model 2). The results of the parametric modelling analysis that only considered nouns, verbs and adjectives in calculating the word-level social-semantic-richness modulator are very similar to those of the first parametric modelling analysis, which are shown in Table 7.

Table 6.

Results of the parametric modulation analysis that considered all words in calculating the word-level social-semantic-richness modulator

Stimuli	ROI	Word-level social-			Sentence-level social-			Narrative-level social-
		semantic-richness			semantic-richness			semantic-richness
		modulator			modulator			modulator
		Beta	SE	t	Beta	SE	t	Beta	SE	t
Model 1: high-level social-semantic-richness modulators were orthogonalized with respect to the low-level ones
Word lists	LATL	0.346	0.072	4.836***+	–	–	–	–	–	–
	LTPJ	0.369	0.151	2.449*	–	–	–	–	–	–
	RATL	0.237	0.091	2.609*	–	–	–	–	–	–
	RTPJ	0.156	0.136	1.150	–	–	–	–	–	–
	PC	0.210	0.171	1.229	–	–	–	–	–	–
	LDMPFC	0.327	0.095	3.438**+	–	–	–	–	–	–
Unconnected sentences	LATL	0.515	0.085	6.046***+	0.282	0.062	4.529***+	–	–	–
	LTPJ	0.600	0.148	4.054***+	0.461	0.121	3.817***+	–	–	–
	RATL	0.413	0.083	4.958***+	0.260	0.071	3.642***+	–	–	–
	RTPJ	0.302	0.110	2.755**	0.284	0.107	2.664*	–	–	–
	PC	0.707	0.145	4.881***+	0.255	0.121	2.104*	–	–	–
	LDMPFC	0.445	0.105	4.250***+	0.332	0.074	4.472***+	–	–	–
Narratives	LATL	0.586	0.067	8.755***+	0.264	0.094	2.809**	0.084	0.055	1.525
	LTPJ	0.685	0.167	4.096***+	0.412	0.148	2.787**	0.229	0.096	2.393*
	RATL	0.549	0.085	6.469***+	0.244	0.104	2.354*	0.141	0.066	2.131*
	RTPJ	0.523	0.109	4.812***+	0.214	0.126	1.698	0.239	0.100	2.406*
	PC	0.553	0.124	4.452***+	0.058	0.138	0.421	0.133	0.110	1.217
	LDMPFC	0.350	0.093	3.743***+	0.332	0.117	2.836**+	0.023	0.086	0.269
Model 2: low-level social-semantic-richness modulators were orthogonalized with respect to the high-level ones
Word lists	LATL	0.346	0.072	4.836***+	–	–	–	–	–	–
	LTPJ	0.369	0.151	2.449*	–	–	–	–	–	–
	RATL	0.237	0.091	2.609*	–	–	–	–	–	–
	RTPJ	0.156	0.136	1.150	–	–	–	–	–	–
	PC	0.210	0.171	1.229	–	–	–	–	–	–
	LDMPFC	0.327	0.095	3.438**+	–	–	–	–	–	–
Unconnected sentences	LATL	−0.133	0.144	0.923	0.232	0.036	6.440***+	–	–	–
	LTPJ	−0.469	0.262	1.792	0.287	0.064	4.464***+	–	–	–
	RATL	−0.196	0.163	1.198	0.191	0.034	5.585***+	–	–	–
	RTPJ	−0.385	0.255	1.509	0.153	0.046	3.307**+	–	–	–
	PC	0.101	0.316	0.3200	0.294	0.055	5.362***+	–	–	–
	LDMPFC	−0.342	0.179	1.909	0.204	0.041	4.962***+	–	–	–
Narratives	LATL	0.024	0.232	0.104	0.129	0.065	1.990	0.200	0.021	9.504***+
	LTPJ	−0.150	0.372	0.402	−0.002	0.102	0.024	0.246	0.055	4.460***+
	RATL	0.073	0.288	0.255	0.059	0.077	0.762	0.192	0.026	7.503***+
	RTPJ	0.186	0.348	0.535	−0.069	0.087	0.787	0.192	0.037	5.170***+
	PC	0.573	0.371	1.543	0.107	0.125	0.859	0.182	0.043	4.241***+
	LDMPFC	−0.333	0.259	1.283	0.067	0.092	0.729	0.131	0.035	3.759***+

Note: *P < 0.05; **P < 0.01; ***P < 0.001; +t-values surviving the Bonferroni correction in which the significance level is divided by the number of ROIs (N = 6). ROI labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.

Table 7.

Results of the parametric modulation analysis that only considered nouns, verbs and adjectives in calculating the word-level social-semantic-richness modulator

Stimuli	ROI	Word-level social-			Sentence-level social-			Narrative-level social-
		semantic-richness			semantic-richness			semantic-richness
		modulator			modulator			modulator
		Beta	SE	t	Beta	SE	t	Beta	SE	t
Model 1: high-level social-semantic-richness modulators were orthogonalized with respect to the low-level ones
Word lists	LATL	0.269	0.056	4.766***+	–	–	–	–	–	–
	LTPJ	0.269	0.113	2.393*	–	–	–	–	–	–
	RATL	0.168	0.073	2.319*	–	–	–	–	–	–
	RTPJ	0.139	0.106	1.312	–	–	–	–	–	–
	PC	0.154	0.128	1.207	–	–	–	–	–	–
	LDMPFC	0.243	0.075	3.257**+	–	–	–	–	–	–
Unconnected sentences	LATL	0.411	0.067	6.134***+	0.233	0.065	3.574**+	–	–	–
	LTPJ	0.486	0.115	4.215***+	0.418	0.135	3.088**+	–	–	–
	RATL	0.328	0.067	4.911***+	0.260	0.063	4.095***+	–	–	–
	RTPJ	0.247	0.086	2.867**+	0.242	0.094	2.580*	–	–	–
	PC	0.497	0.108	4.624***+	0.400	0.131	3.047**+	–	–	–
	LDMPFC	0.352	0.079	4.469***+	0.256	0.083	3.091**+	–	–	–
Narratives	LATL	0.439	0.049	9.048***+	0.274	0.080	3.427**+	0.080	0.055	1.447
	LTPJ	0.535	0.121	4.407***+	0.296	0.133	2.227*	0.226	0.096	2.365*
	RATL	0.418	0.057	7.362***+	0.278	0.091	3.064**+	0.119	0.071	1.683
	RTPJ	0.405	0.077	5.289***+	0.216	0.122	1.777	0.246	0.098	2.519*
	PC	0.400	0.091	4.394***+	0.273	0.145	1.887	0.113	0.112	1.006
	LDMPFC	0.286	0.066	4.325***+	0.217	0.126	1.727	0.058	0.079	0.743
Model 2: low-level social-semantic-richness modulators were orthogonalized with respect to the high-level ones
Word lists	LATL	0.269	0.056	4.766***+	–	–	–	–	–	–
	LTPJ	0.269	0.113	2.393*	–	–	–	–	–	–
	RATL	0.168	0.073	2.319*	–	–	–	–	–	–
	RTPJ	0.139	0.106	1.312	–	–	–	–	–	–
	PC	0.154	0.128	1.207	–	–	–	–	–	–
	LDMPFC	0.243	0.075	3.257**+	–	–	–	–	–	–
Unconnected sentences	LATL	−0.014	0.119	0.118	0.228	0.036	6.36***+	–	–	–
	LTPJ	−0.289	0.233	1.239	0.283	0.064	4.425***+	–	–	–
	RATL	−0.155	0.113	1.372	0.187	0.034	5.436***+	–	–	–
	RTPJ	−0.213	0.182	1.168	0.148	0.047	3.188**+	–	–	–
	PC	−0.191	0.257	0.744	0.288	0.055	5.192***+	–	–	–
	LDMPFC	−0.098	0.144	0.678	0.204	0.042	4.845***+	–	–	–
Narratives	LATL	−0.008	0.148	0.053	0.119	0.062	1.924	0.201	0.021	9.611***+
	LTPJ	0.102	0.220	0.463	0.000	0.098	0.004	0.246	0.055	4.446***+
	RATL	−0.020	0.167	0.122	0.051	0.075	0.687	0.194	0.026	7.594***+
	RTPJ	0.035	0.222	0.158	−0.074	0.082	0.903	0.193	0.037	5.206***+
	PC	−0.035	0.284	0.123	0.109	0.121	0.897	0.184	0.043	4.289***+
	LDMPFC	−0.036	0.214	0.167	0.061	0.090	0.679	0.129	0.035	3.725***+

Note: *P < 0.05; **P < 0.01; ***P < 0.001; +t-values surviving the Bonferroni correction in which the significance level is divided by the number of ROIs (N = 6). ROI labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.

The results of the whole-brain activation analysis

The results of the whole-brain activation analysis are detailed in the Supplementary Materials. To briefly summarize, the results largely replicate the social-semantic-richness and linguistic hierarchical effects reported in the literature (see Supplementary Tables S1 and S3, Supplementary Figures S1, and S3) and indicate that these two effects interact with each other. The social-semantic-richness effect in the sentence and narrative conditions was observed in all areas of the social semantic network, whereas the social-semantic-richness effect in word-list conditions was only observed in the left ATL (Supplementary Table S2 and Supplementary Figure S2). The sentential effect (sentence > word-list) in the high social-semantic-richness conditions was observed in most classic areas of the sentence processing network (Fedorenko et al., 2010; Labache ), whereas the sentential effect in the low social-semantic-richness conditions was observed in very few brain areas (Supplementary Table S4 and Supplementary Figure S4). The statistical comparisons of the social-semantic-richness effects across different linguistic hierarchies revealed a significant cluster in the right precuneus, where the social semantic activation was stronger in the sentence conditions than in the word-list conditions (see Supplementary Figure S5 and Supplementary Table S5).

The results of the PPI analysis

The PPI analysis did not reveal any significant social-semantic-richness effect or interaction between social semantic richness and linguistic hierarchy (see Supplementary Figure S6 and Supplementary Table S6), possibly due to that the functional coupling in the social semantic network is modulated not only by social semantic processes but also by the intrinsic functional antagonism between the default mode network and the multiple demand network. The results are reported and discussed in the Supplementary Materials.

Discussion

We investigated the effects of social semantic accumulation using an fMRI experiment in which the social semantic richness and linguistic hierarchies of stimuli were both manipulated. The social semantic network showed two aspects of social semantic accumulation effects. In the contrast-based analysis, the social semantic network showed stronger social semantic activations in sentence and narrative reading than in word-list reading, indicating that the amplitude of social semantic activation accumulates along with sentence processing. In the parametric modelling analysis, the activation of the social semantic network in sentence and narrative reading can be better explained by the holistic social-semantic-richness rating scores of the stimuli than by the social-semantic-richness rating scores of the constitutive words, regardless of whether all words or only nouns, verbs and adjectives were considered, indicating the social semantic contents can be integrated in the network. These two findings convergently indicate that the social semantic network is involved in social semantic accumulation during language comprehension.

Our findings provide new insights into the function of the social semantic network. Most previous studies of social semantic processing focused on the representation of social concepts underlying word meanings (Zahn ; Lin , 2018a, 2019; Wang ). Some studies have emphasized the role of the ATL in social concept representation (Zahn ; Wang ). The present study provided the first evidence that all areas of the social semantic network, including the ATL, were involved in not only social concept representation but also in social semantic accumulation. This important function of the social semantic network should be considered in future studies, especially those investigating the social semantic processing of complex materials, such as sentences, narratives and movies.

Our findings also shed new light on how semantic accumulation may occur in the brain. In previous studies, the effect of semantic accumulation has only been associated with the size and processing time scale of the semantically connected units (Lerner ; Pallier ). Our finding indicates that the type of semantic information being processed also modulates the effect of semantic accumulation on brain activation. Therefore, future studies on semantic accumulation should consider not only the domain-general factors influencing semantic accumulation but also the types of semantic contents being processed.

One advantage of the current study is that the use of the parametric modulation analysis has compensated for the shortness of the traditional methods for analysing the semantic accumulation effect. Comparing sentence processing with word-list processing is a frequently used paradigm to reflect the neural correlates of semantic accumulation (Xu ; Humphries ; Lerner ). However, one may argue that the difference between sentence and word-list processing is confounded by the effect of processing depth (Craik and Lockhart, 1972): word-list processing is a relatively shallow type of processing, in which people may tend to encode the orthographical and phonological information of stimuli; in contrast, sentence processing enables the chunking of meanings, making semantic encoding dominant. Similarly, the effects of the constituent size (Pallier ; Mellem ) on the brain activation properties (locations, amplitudes and phases) may also be explained by processing depth due to the fact that the processing depth could parametrically vary along with the constituent size. The parametric modulation analysis used in the present study overcomes this problem by focusing only on sentence and narrative processing and using the regression approach to dissociate the sentence and narrative levels of social semantic effect from the word-level social semantic effect. This approach has revealed a new aspect of semantic accumulation effect that cannot be confounded by processing depth.

An important question that remains to be explored is whether the social semantic accumulation in the social semantic network occurs only at the sentence level or also at the narrative level. The results of the contrast-based analysis did not reveal any evidence for the narrative-level social semantic accumulation effect. However, it should be noted that the main effect of narrative processing (narrative > sentence) observed in our whole-brain analysis (see the Supplementary Materials) was also much weaker than that reported by the previous studies of narrative-level semantic accumulation (e.g. Xu ). This is possibly due to the fact that we used much shorter narratives than did the previous studies, aiming to better match the linguistic variables between the high and low social-semantic-richness materials. On the other hand, the results of our parametric modelling analysis did reveal a narrative-level effect in the bilateral TPJ (see Tables 6 and 7), where the narrative-level social-semantic-richness modulator showed modulation effects on brain activation, even after being orthogonalized with respect to the sentence-level social-semantic-richness modulator. This finding is consistent with the previous observations that the bilateral TPJs are involved in narrative-level social semantic processes. Lin compared the brain activation in the beginning and ending sentences of social and non-social narratives and found an interaction between the narrative topic (social/non-social) and narrative processing period (ending/beginning): during the reading of social narratives, the ending sentence evoked much stronger activation than the beginning sentence in the bilateral TPJ and middle temporal gyrus; however, during the reading of non-social narratives, such an effect was either not significant or much smaller. Kaplan found that the bilateral TPJ, posterior medial cortices and medial prefrontal cortex showed stronger activation to narratives containing protected values (core personal, national or religious values that are non-negotiable) than to control narratives, and the effect was most pronounced during the ending segment of the narrative. Therefore, the findings of our parametric modelling analysis provided a new and convergent piece of evidence that the bilateral TPJ may support narrative-level social semantic processing.

Another important question that should be investigated in future is how non-social semantic information, such as sensory-motor semantic information, is accumulated in language comprehension. The brain areas that integrate sensory-motor semantic information are mainly distributed in the parahippocampal gyrus, retrosplenial cortex and temporal-parietal-occipital junction (Fernandino ; Lin ). Although these areas were seldom reported in previous studies of sentence and discourse comprehension (Walenski ; Yang ), a recent study has reported their selective activation in reading vivid passages (Tamir ). In addition, these areas are also known to support scene construction (Hassabis and Maguire, 2009). Therefore, future studies may examine whether and how these areas accumulate sensory semantic information using texts that describe scenes or images.

Conclusion

We found that the social semantic network showed stronger social semantic activation in sentence and narrative reading than in word-list reading, and during sentence and narrative reading, the social semantic network showed higher sensitivity to the holistic social semantic richness of the stimuli than to the social semantic richness of the constitutive words. These two findings convergently indicate that the social semantic network is involved in social semantic accumulation during language comprehension.

Click here for additional data file.

Table 4.

ROI results of the contrast-based analysis: the social semantic activations in narrative reading, sentence reading and word-list reading

ROI	Social semantic activation in narrative			Social semantic activation in sentence			Social semantic activation in word-list
	reading: HSN and LSN			reading: HSS and LSS			reading: HSW and LSW
	Beta	SE	t	Beta	SE	t	Beta	SE	t
LATL	0.907	0.089	10.150***+	0.857	0.126	6.818***+	0.500	0.104	4.816***+
LTPJ	1.062	0.228	4.665***+	1.112	0.237	4.688***+	0.351	0.204	1.721
RATL	0.875	0.113	7.733***+	0.701	0.128	5.460***+	0.323	0.115	2.819**+
RTPJ	0.820	0.154	5.322***+	0.573	0.169	3.393**+	0.239	0.165	1.453
PC	0.870	0.179	4.873***+	1.032	0.191	5.392***+	0.248	0.222	1.119
LDMPFC	0.607	0.141	4.303***+	0.794	0.143	5.546***+	0.426	0.133	3.215**+

Note: *P < 0.05; **P < 0.01; ***P < 0.001; +t-values surviving the Bonferroni correction in which the significance level is divided by the number of ROIs (N = 6).

Condition labels: HSN, high social-semantic-richness narrative; HSS, high social-semantic-richness sentence; HSW, high social-semantic-richness word; LSN, low social-semantic-richness narrative; LSS, low social-semantic-richness sentence; LSW, low social-semantic-richness word. ROI labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.

Table 5.

ROI results of the contrast-based analysis: comparing the social semantic activations between different linguistic hierarchies

ROI	Social semantic activation in narrative			Social semantic activation in narrative			Social semantic activation in sentence
	reading vs social semantic activation			reading vs social semantic activation			reading vs social semantic activation
	in sentence reading:			in word reading:			in word reading:
	(HSN and LSN)—(HSS and LSS)			(HSN and LSN)—(HSW and LSW)			(HSS and LSS)—(HSW and LSW)
	Beta	SE	t	Beta	SE	t	Beta	SE	t
LATL	0.050	0.119	0.421	0.407	0.116	3.519**+	0.357	0.120	2.971**+
LTPJ	−0.050	0.212	0.236	0.712	0.259	2.745**	0.762	0.257	2.967**+
RATL	0.174	0.159	1.093	0.552	0.144	3.839***+	0.378	0.148	2.543*
RTPJ	0.247	0.191	1.295	0.580	0.215	2.703*	0.334	0.215	1.554
PC	−0.161	0.236	0.684	0.622	0.285	2.185*	0.783	0.246	3.179**+
LDMPFC	−0.187	0.186	1.009	0.181	0.181	0.997	0.368	0.140	2.624*

Note: *P < 0.05; **P < 0.01; ***P < 0.001; +t-values surviving the Bonferroni correction in which the significance level is divided by the number of ROIs (N = 6).

Condition labels: HSN, high social-semantic-richness narrative; HSS, high social-semantic-richness sentence; HSW, high social-semantic-richness word; LSN, low social-semantic-richness narrative; LSS, low social-semantic-richness sentence; LSW, low social-semantic-richness word. ROI labels: LATL, left anterior temporal lobe; LDMPFC, left dorsal medial prefrontal cortex; RTPJ, right temporoparietal junction; LTPJ, left temporoparietal junction; RATL, right anterior temporal lobe; PC, posterior cingulate.