FN400 and LPC Responses to Different Degrees of Sensory Involvement: A Study of Sentence Comprehension.

The current study tested the likely effect of sensory involvement on the FN400 and late positive complex (LPC) responses to semantic and pragmatic comprehension of English sentences. Fifteen English language learners took part in the event-related potential (ERP) experiment and determined the acceptability of 432 sentences under congruent, semantically incongruent, and pragmatically incongruent conditions. Prior to the ERP recording, the subjects received different sensory instructions for six vocabulary items about which they had no previous knowledge. No sensory instruction was given for three extra words, and these served as the control group. The behavioral data corroborated that integration of more senses in instruction improved learners' pragmatic comprehension. The ERP data revealed that full sensory involvement (involvement) reduced the FN400 amplitude, facilitating real world knowledge retrieval and pragmatic comprehension. The LPC responses to semantic comprehension showed that learners reanalyzed the sentences instructed through limited sensory involvement (exvolvement) more deeply. Copyright:

Introduction

Sentence comprehension depends upon a variety of cognitive processes for decoding word meanings (Salmon & Pratt, 2002). The semantic account of these processing steps describes the integration of semantic and pragmatic sources of knowledge whose core assumptions rely on the degree and quality of sensory world experiences (Hagoort et al., 2004;Pishghadam et al., 2013).

Conventionally, in much of the research done on language, response accuracy values were the primary means of evaluating comprehension. As early as 1980, Kutas and Hillyard shifted this approach and popularized the idea of using different components of time-locked electroencephalography (EEG), particularly the N400. Numerous following studies began to use brain electrical activity to explore the way semantic information was incorporated into language comprehension. Although there was much debate over what kind of processing is evoked from the N400, experts unanimously converged upon its sensitivity to semantic integration processes of language as its prevalent property (e.g., Kutas & Hillyard, 1983). A number of later studies drew a line between the N400 and the frontally distributed N400 (termed the FN400), postulating an association between FN400 and feelings of familiarity (e.g., Johnson Jr et al., 1998; Rugg & Curran, 2007). Other studies provided evidence that, rather than mere familiarity, FN400 is tied to episodic memory and facilitates conceptual processing (Paller et al., 2012; Voss & Paller, 2007). In a later study, Voss et al. (2012) tested meaningless but familiar stimuli and observed no FN400, concluding that FN400 is associated more with meaning than familiarity. Although numerous studies documented functional distinctions between the N400 and FN400 (e.g.,Bridger et al., 2012), others have approached the issue differently (e.g., Curran, 1999; Voss & Federmeier, 2011). They claim that, owing to the similar morphology and time course of the two effects, FN400 is a frontally distributed N400 with a focus on episodic recognition.

The ongoing debates suggest that different language-related elements may account for the post-FN400 positivities as well. In their early studies, Kutas and Hillyard (1980a, 1980b, 1980c) documented a broad positivity peaked at approximately 560 ms in response to semantically acceptable sentences. In similar studies, Salmon and Pratt (2002) and Juottonen et al. (1996) confirmed the emergence of a delayed positive deflection not only to the semantically congruent condition but also to its incongruent counterpart, with the semantic anomaly yielding a larger P500. In multiple experiments (e.g., Cansino & Téllez-Alanís, 2000), it has been deduced that the positive deflection may be a delayed expression of the P300 peaking at about 600 ms post stimulus. Overall, previous research rendered the late positive component (LPC) as a function of syntactic violation (Hagoort et al., 1993), plausibility judgment tasks (Kuperberg, 2007), sentential and contextual constraints (Federmeier et al., 2007), disconfirmed predictions (DeLong et al., 2011), recollection and recognition (Hintzman & Curran, 1994), and semantically anomalous words (Federmeier et al., 2007).

Among the elements which modulate the electrophysiological brain components as the indicators of comprehension, sensory experiences seem to be of primary significance due to the undeniable role of senses in understanding the world where language is delivered. As a further potential variable, sensory experiences not only stimulate performance and behavior but they are also assumed to relativize cognition (i.e., sensory relativism, Pishghadam et al., 2016). That is, based on our sensory experiences, we may construct different realities and have different conceptions of the world. Thus, Pishghadam (2016a) argues that words and concepts are stored in and retrieved from memory according to the senses through which they have been experienced in the real world. That is, depending on the number of senses involved and quality of the sensory experiences (whether you see a real object or its picture for example), comprehension may improve. Thus, we presumed that the cognitive facet of sentence comprehension can perhaps be modified by incorporating different senses. To test this claim and to control the sensory experiences of the learners, we picked nine unknown English vocabulary items to teach to a group of Persian speakers and a sensoryoriented instruction model based on a newly-developed model for the categorization of senses in educational contexts. According to the emotioncy-based language instruction (EBLI) model (see Figure 1), senses, either in isolation or in combination, can boost later retrieval to different degrees. The EBLI, with emotioncy as its core concept (for the differences between emotioncy and concepts such as familiarity, see Pishghadam et al., 2017, embodied cognition, see Pishghadam et al., 2019, and concreteness, see Pishghadam & Shayesteh, 2017) draws upon the role of senses in constructing idiosyncratic representations of the world. The clarity of these mental representations is calibrated by the number of senses involved, with involvement (a combination of all senses) capturing the most authentic representations of the world (Pishghadam et al., 2013). The prioritization and integration of the senses in this model are based on the main modalities teachers regularly use in their instructional practices, starting from mere lecturing (Larsen-Freeman, 2000). For initial stages the teacher may proceed toward the end of the continuum, combining all the senses. For intermediate stages, this progress may stop in the middle of the continuum, combining three senses (i.e., auditory, visual, & kinesthetic) only. For advanced stages, however, only hearing or a combination of hearing and sight could function as the main delivery mode of instruction. Therefore, taking the needs of the students and factors such as time and workload into account, a teacher decides on the combination of the senses. As Figure 1 depicts, exvolvement begins with the sense of hearing (Level 1, Auditory emotioncy) and then mixes with sight (Level 2, Visual emotioncy) and touch (Level 3, Kinesthetic emotioncy, to create indirect/other-directed sensory experiences), whereas involvement manifests itself in the integration of all the senses including taste and smell (Level 4, Inner emotioncy, to create direct/self-directed sensory experiences). In order for the individuals to deepen their direct experiences of the concepts, Pishghadam et al. (2017; Pishghadam et al., 2019; Pishghadam & Shayesteh, 2017) suggest independent research as the final level of involvement (Level 5, Arch emotioncy).

Figure 1

Emotioncy levels (Reprinted with permission from , by Pishghadam (2016b), Proceedings of the 5th International Conference on Language, Education, and Innovation. London, UK).

Drawing upon the wider categorization of the model (i.e., avolvement, exvolvement, & involvement) which deals with different integration of senses, we developed two levels of instruction encompassing a limited sensory involvement level (exvolvement) and a full sensory involvement level (involvement), as well as a no sensory involvement level (avolvement) to serve as the control group and allow for comparing the cognitive changes as a result of the sensory instructions. To substantiate the extent to which exvolvement and involvement modulate learners’ behavioral and neural (i.e., FN400 and LPC as the main neural indicators of semantic processing) responses underlying sentence comprehension, a task design used in studies by Hagoort et al. (2004), Hald et al. (2006), and Kos et al. (2012) was employed.

Overall, relying upon the previous studies of emotioncy (e.g., Karami et al., 2019; Makiabadi et al., 2019; Pishghadam et al., 2017) multisensory education (e.g., Auer, 2008; Baines, 2008; Birsh & Carreker, 2018; Katai, 2011; Lee et al., 2019), and sensory integration (e.g., Alvarado et al., 2007; Lin, 2004; Young et al., 2011) showing that senses facilitate learning and knowledge retrieval, we speculated that full sensory involvement would carry more sensory knowledge and modify brain components in a way that facilitates the cognitive processes underlying semantic and pragmatic comprehension.

Analysis into semantic and pragmatic knowledge gained momentum with the study of Hagoort et al. (2004), comparing the role of semantic and pragmatic knowledge during online semantic processing of sentences. They devised a specific acceptability judgement task and clearly showed that semantically appropriate words with world knowledge violations create similar N400 amplitude modulations to the ones with semantic violations. Since then, different ERP studies investigated the two concepts in the form of sentences with semantic and pragmatic violations. Hald et al. (2007) identified an interaction between pragmatic knowledge and contextual effect. Similarly, Filik and Leuthold (2008) demonstrated that the N400 effect to pragmatic anomalies vanished when the violated sentence was placed within a context. Using Hagoort et al.’s (2004) framework, Kos et al. (2012) concluded that among the contextual constraint, inter-individual variation, and working memory, only interindividual variation affects semantic comprehension and the magnitude of LPC. In another study, DeLong et al. (2014) reported that predictability and plausibility modulate access to semantic knowledge.

To our best knowledge, almost all the studies which verified semantic and pragmatic comprehension separately have focused on L1 processing and the factors which influence its neurocognitive effects, aiming to explore basic language-related brain functioning. However, we believe adding L2 processing features to this growing body of literature is pertinent. Thus, we checked the two comprehension processes in L2 and gauged to what extent senses, as general facilitators of L2 learning, give rise to any changes in the magnitude of language-related ERP components. Based on the reviewed literature, we hypothesized that the subjects’ behavioral reactions to the stimuli may improve as more senses are used in their instruction. Moreover, the negativity of the FN400 responses to semantic and pragmatic comprehension may decrease as a result of full sensory involvement. This could be due to enhanced sensory familiarity which boosts semantic processing. Yet the positivity of the LPC, depending on whether it may appear on all sentence conditions, is expected to increase in response to full sensory involvement and improved recognition.

Methodology

Subjects

Thirty-five native Persian-speaking subjects, with English as their foreign language, volunteered to take part in the pretests. Twenty-one of those subjects (14 women, 7 men) were recruited to participate in the ERP experiment. They were intermediate learners of English, since they were supposed to have a minimum level of proficiency for understanding the English-language instruction (i.e., the EBLI) and reading and comprehending the English sentences. Moreover, intermediate rather than advanced learners of English were a more accessible sample. The participants were aged between 20 and 30 (Mage = 22.6, SD = 2.82) with a working memory score ranging from 10 to 12 (M = 11.53, SD = 1.59). The subjects were all right-handed, with normal or corrected-to-normal vision, and no known language or neurological impairment. They received either a gift or course credit in return for their participation. Of the 21 subjects, six were excluded from the final analysis due to excessive eye movement and muscle artifact. Prior to initiating the study, the subjects gave written informed consent under a protocol approved by the Ferdowsi University of Mashhad Ethics Committee.

The task design.

Materials

Pretest materials. In order to assure the homogeneity of the subjects, a number of measures were administered in the pretest phase. Table 1 shows the descriptive statistics of the tests.

The Emotioncy Scale. To evaluate the subjects’ degree of familiarity with the nine target words, a Persian emotioncy scale (adapted from Borsipour, 2016) was constructed and thereafter validated a six-point Likert scale: 0 (not familiar), 1 (heard), 2 (heard and seen), 3 (heard, seen, and touched), 4 (heard, seen, touched, and used/tasted) and 5 (heard, seen, touched, used, and done research on). The subjects who had prior experiences with any of the 9 words were excluded in this phase.

The Oxford Quick Placement Test. To homogenize the subjects in terms of their English language proficiency level, the Oxford Quick Placement Test (OQPT) was utilized. According to the test’s leveling schemes, scores between 30 and 40 describe the intermediate level of English.

Descriptive Statistics for the Pretests

The Digit Span subtest of the Wechsler Adult Intelligence Scale III. To measure the subjects’ ability to attend to and retain information, the digit span subtest of the translated version of the Wechsler’s Adult Intelligence Scale (WAIS) III (1981), prepared by Azmoon Padid Institute (1993), was used.

The Edinburgh Inventory of Handedness. To assess handedness quantitatively, the Edinburgh inventory (Oldfield, 1971) was employed. The questions center around typical daily chores. The subjects who did more than two of the activities with their left hand were excluded from the study.

Stimulus Material.To come up with the nine unknown words, 30 concrete words were selected (see Figure 2). Eighty individuals marked their sensory experiences with the items on the emotioncy scale. Of the 30 words, 93% of the participants had no sensory experience for mangosteen, rambutan, caper, quinoa, physalis, sorrel, salak, longan, and cranberry.

Procedure

A sentence acceptability task was constructed using Psychophysics Toolbox Version 3 (PTB-3) for MATLAB (version 2015a, The MathWorks, MA, see Figure 3). Nine vocabulary items including the name of novel foods, vegetables, and tropical fruits (each containing 5 to 10 characters) for which the subjects had no previous knowledge were embedded in 324 sentences with three to eight words (12 sentence triplets per instructed word). The triplets were all the same with an exception of one word only. The effect of syntax was reduced by utilizing a fixed syntactic structure to the sentences. Cloze probabilities were checked by two native speakers. The cloze probability for the correct, semantically incongruent, and pragmatically incongruent sentences was 100%, 0%, and 0%, respectively.

Following Hagoort et al. (2004), Hald et al. (2006), and Kos et al. (2012), which targeted semantic and pragmatic comprehension, the sentences were marked acceptable or unacceptable under three different conditions: sentences with acceptable world knowledge (n = 108, e.g., “A mangosteen is purple.”), sentences with word knowledge violation (n = 108, e.g., “A mangosteen is rainy.”), and sentences with world knowledge violation (n = 108, e.g., “A mangosteen is orange.”). One hundred and eight unrelated correct sentences of similar length, complexity, and structure were inserted as filler trials which mainly contained the name of animals and objects (e.g., “A gorilla is hairy.”). The sentences were randomized into 6 experimental blocks of 72 trials, followed by a 5 min break. They were presented word by word, in a pseudo-random order. All words were written in black lower-case letters against a light gray background with 36 pt Times New Roman font. Words subtended an approximate visual angle of 3 ° horizontally and 0.5 ° vertically. The words appeared in the center of the computer screen randomly for 750 to 850 ms, followed by a 300 ms blank page. The final word of every sentence was followed by a 2800 ms blank page for the subjects to press the pre-defined keys. Then, an eye image was displayed and the subjects were allowed to blink for 3 s prior to the next trial.

The mentioned protocol was set according to the literature. Prior to the study, the clarity of the 432 sentences was checked through a pilot study (a paper and pencil test) on 15 participants. A number of sentences were revised according to their comments and some words were replaced with their synonyms to avoid possible ambiguities. The timings were also confirmed through a pilot study (a computer test with no ERP recording) on 11 participants who were not the subjects of the main study. The 2800 ms response time was deduced form the performance of the participants. Table 2 shows that 95% of the responses were below 2800 ms.

The instruction. The data collection was split into the pre-experimental and experimental phases. During the pre-experimental phase, each subject came to the lab a few days before the ERP recording to take the pretests. At the end of the session, the subjects received a list of vocabulary items to take home to read before coming for the experiment (the list did not include the nine target words and the fillers). The logic was to prevent the subjects from paying unnecessary attention to the non-target words due to delayed recall during the ERP task. The experimental phase (done individually) was initiated by a 20 min instruction of the selected words. Relying on the EBLI, each subject received different sensory instructions: limited sensory instruction (i.e., exvolvement) for three words, full sensory instruction (i.e., involvement) for three words, and no specific sensory instruction (i.e., avolvement) for three words, all counterbalanced across the subjects. The instruction time and the amount of information transferred to the subjects for each word were the same. Of the six words, the subjects received auditory instruction for one word. For the second word, a photo booklet was added to verbal explanations (i.e., auditory + visual). For the third word, the subjects could see the real object and touch it while looking at the pictures and listening to the explanations (i.e., auditory + visual + kinesthetic). For the fourth word, they were allowed to smell the object as well (i.e., auditory + visual + kinesthetic + smell). For the fifth word, the subjects could experience full sensory involvement through peeling, cutting, smelling, and tasting the object at the same time they were listening to the instructor’s explanations. For the sixth word, in addition to the previous sensory experiences, they were also given the opportunity to search it online, on a computer, for almost a minute. The information they encountered during online searching was not included in the ERP experiment. After the instruction, the subjects were prepared for the ERP recording. Table 3 presents the information transferred to the subjects through a sample instruction of the six words.

Percentiles for the Response Time

A sample event-related potential study.

The EEG recording. Subjects were tested in a sound-attenuated, dimly lit EEG room. They were instructed to press the right arrow key (middle finger) for correct sentences and down arrow key (index finger) for incorrect ones. They were also advised against guessing the correctness of the sentences they had no information for (not pressing either of the keys). In order to get acquainted with the task requirements, they went through a practice block of 20 trials.

The EEG activity was recorded at a sampling rate of 250 Hz using an elastic electrode cap embedded with 23 Ag/AgCl scalp electrodes arranged according to the international 10–20 system. The optimal arrangement was determined according to similar studies in the field (e.g., Hagoort et al., 2004; Hald et al., 2006; Salmon & Pratt, 2002; Van Berkum et al., 2005). Five electrodes were placed at midline sites (Fz, FCz, Cz, Pz, and Oz). Nine lateral pairs of electrodes were located at anterio-frontal (AF3 and AF4), frontal (F3, F4, F7, and F8), frontotemporal (FT7 and FT8), fronto-central (FC3 and FC4), central (C3 and C4), parietal (P3, P4, P7, and P8), and parieto-occipital (PO7 and PO8) positions. Three additional electrodes were placed above and below the left eye, and on the left outer canthus to record blinks and horizontal eye movements. Two more electrodes were located on each mastoid with the left one serving as the online reference. The EEG and EOG recordings were amplified using the 32-channel wireless g.Nautilus EEG system (gtec, Austria), with a band-pass filter of 0.1–70 Hz.

A Sample Instruction

Data Analysis

The EEG recordings from the 23 electrode sites were imported offline into the MATLAB computing software environment (version 2015a, The MathWorks, MA). All amplifiers had a 50 Hz notch filter to eliminate AC line noise. The continuous EEG data were band-pass filtered between 0.5 and 60 Hz. Afterwards, the data were rereferenced to the mean of the linked-mastoids. The rest of the analysis was done using the EEGLAB toolbox for MATLAB. Poor EEG channels were replaced with their interpolated versions applied to the remaining channels. No more than two channels were interpolated for each subject, with the majority of the interpolated channels positioned at the parietooccipital and occipital sites. High amplitude eye blinks and muscle artifacts were then removed using the artifact subspace reconstruction (ASR) algorithm from the EEGLAB. The remaining high frequencies were eliminated using a low pass filter with a cut off frequency of 25 Hz. Next, the EEG was epoched (from 200 ms before to 1100 ms after the onset of the sentence final words) and baseline corrected (using a −200 to 0 ms prestimulus interval). A linear detrend algorithm (using the 200 ms before the stimulus onset to 3 s after) was applied to the epoched data to further remove drifts. Noisy epochs with fluctuations below -70 µVs and above 70 µVs were rejected. In the end, waveforms from all remaining trials were averaged (see Table 4 for descriptive statistics).

For the final words of the sentences, the FN400 and LPC were analyzed. The components’ mean amplitudes for the final words followed by the response time window was qualified based on the findings of the previous literature (e.g., Danko et al., 2014; Molinaro et al., 2016; Yang et al., 2014; Volz  et al., 2019) as the mean voltage between 300 to 550 ms (for the FN400) and 600 to 850 (for the LPC), subsequent to the word onset relative to a 200 ms baseline. The mean amplitude measures were evaluated with the Kolmogorov-Smirnov test for normality and then included in within-subjects analyses of variance (ANOVAs) calculated in SPSS 24 with the Greenhouse–Geisser correction for nonsphericity and the Bonferroni correction during post hoc testing.

Results

Behavioral Data

To address the first research question, the subjects’ behavioral reactions to decide upon the truthfulness of the sentences were examined. The analysis of behavior data included the accuracy of the responses (RA) given to the sentences in addition to the corresponding response times (RTs). Table 5 summarizes the descriptive statistics.

Descriptive Statistics for the Number of Averaged Epochs

Response accuracy (RA). A two-way repeated-measures ANOVA with the within-subject factors of instruction type (avolvement [av.], exvolvement [ex.], and involvement [in.]) and linguistic condition (correct [corr.], semantic [sem.], and pragmatic [prag.]) was applied to the behavioral data. Significant differences were observed for instruction type, F(2, 28) = 143.08, p < .001, ηp 2 = .91, 1−β = 1.00, and linguistic condition, F(2, 28) = 44.49, p < .001, ηp 2 = .76, 1−β = 1.00, as well as their interaction, F(4, 56) = 18.99, p < .001, η2 = .57, 1−β = 1.00. The post hoc tests for the instruction type indicated that the subjects’ accurate responses significantly varied across the three types of instructions. They gave the most and the least correct answers to involved (M = 32.62) and avolved sentences (M = 10.17), respectively (i.e., av. < ex. < in.).

To gauge the interaction effect of instruction and condition, the values for the three instruction types were compared across the conditions. The ANOVA results showed significant differences in the mean values for congruent, F(2, 28) = 298.02, p < .001, ηp2 =.95, 1−β = 1.00, semantically incongruent, F(2, 28) = 23.77, p < .001, ηp2 = .62, 1−β = .99, and pragmatically incongruent sentences, F(2, 28) = 126.82, p < .001, ηp2 = .90, 1−β = 1.00.

The post hoc analysis revealed that the subjects’ responses to pragmatically anomalous sentences were significantly different (p < .00) across all the three types of instructions with involvement receiving the maximum (M = 32.40) and avolvement receiving the minimum (M = 7.00) number of correct responses (i.e., av. < ex. < in.). Their responses to correct and semantically anomalous sentences did not make a distinction between exvolvement and involvement (p = .40 for corr.; p= 1.00 for sem.). That is, in comparison with avolvement (Mcorr. = 2.66; Msem. = 20.86) they outdid the sentences pertinent to exvolvement (Mcorr. = 28.73, p < .001; Msem. = 34.53, p < .001) and involvement (Mcorr. = 30.66, p < .001; Msem. = 34.80, p < .001(i.e., av. < ex./in.). Thus, taking the linguistic conditions into account, the mere influence of the EBLI manifested itself more conspicuously in pragmatic comprehension.

Response time (RA). Given that, several subjects did not judge the truthfulness of the avolved sentences across conditions, we had missing values for their response times and were not able to run a two-way ANOVA. To account for the unbalanced data, the RT values of each subject for the three linguistic conditions were manually averaged and used as their RT of the corresponding instruction type. Followed by that, a single-factor repeated-measures ANOVA was used, with instruction type as the independent variable. The F test results were significant, F(2, 28) = 12.35, p < .001, ηp2 = .46, 1−β = .91.

The post hoc tests revealed that the subjects were faster to accurately comprehend and thus respond to involved sentences (M = .83 s) as compared with exvolved (M = .88 s, p < .01) and avolved ones (M = 1.19 s, p < .001; i.e., av. > ex. > in.). That is, subjects had the longest hesitations for the avolved sentences. The obtained results led us to confirm the main effect of the EBLI on the subjects’ RT to judge the correctness of sentences.

Similar analyses were performed to inspect the main effect of instruction type interacting with the incongruent conditions. Sentences with congruous endings were excluded from the analysis due to eight missing values for avolvement. Regarding semantic and pragmatic conditions of avolvement, one value was missing from each. The F test results revealed that significant differences exist between semantically incongruent, F(2, 26) = 10.645, p < .001, ηp2 = .45, 1−β = .87, and pragmatically incongruent sentences, F(2, 26) = 9.43, p < .001, ηp2 = .42, 1−β = .84.

Post hoc tests showed that subjects’ RTs to semantically violated sentences were significantly different between avolvement (M = .96 s), exvolvement (M = .65 s, p < .05), and between avolvement and involvement (M = .64 s, p < .01; i.e., av. > ex./in.). Furthermore, RTs for pragmatically violated sentences were found to be significantly different between avolvement (M = 1.24 s) and involvement (M = .89 s, p < .01), and between exvolvement (M = s) and involvement (p < .01). The values between avolvement and exvolvement were also marginally significant (p = .06; i.e., av. > ex. > in.). This implies that the EBLI has a stronger effect on the subjects’ mental agility to determine the truthfulness of pragmatically incongruent sentences than their semantically incongruent counterparts.

Means and SDs for Response Accuracy and Response Time

Panel A = Grand average (n = 15 subjects) ERPs at Fz for the main effect of instructions on the congruent, semantically incongruent, and pragmatically incongruent conditions. Panel B = The maps show different degrees of FN400 over the anterior locations. Panel C = The bar plots depict the significant differences between avolvement, exvolvement, and involvement.

ERP Data

In the time window of 300 to 550 ms, a negative going component was evident for both forms of violations distributed over frontal, frontocentral, and central areas, with a more frontally located maximum, F(5, 70) = 9.28, p < .001, ηp2 = .39, 1−β = .97, based on six defined ROIs (right frontal [F4], right central [C4], right parietal [P4], left frontal [F3], left central [C3], and left parietal [P3]), which resembled the FN400. The component was not observable for the congruent condition of exvolvement and involvement. In the time window of 600-850 ms, we noticed a broadly distributed positive effect for the semantically violated sentences, with a maximum at parietal cortices, F(5, 70) = 10.78, p < .001, ηp2 = .43, 1−β = .99.

The 300-550 ms time window. To address the second hypothesis and examine the main effect of instructions on the FN400 amplitude, a three-way repeated-measures ANOVA was applied to the mean amplitude measures of the mentioned electrodes. The negativity of the FN400 for avolvement (M = 1.02 µV) was larger than that of exvolvement (M = 2.03 µV, p < .03) and involvement (M = 2.31 µV, p < .01), F(2, 28) = 8.69, p < .001, ηp2 = .38, 1−β = .91.

Regarding the interaction between the EBLI and linguistic conditions, F(4, 56) = 3.91, p < .01, ηp2 = .21, 1−β = .82, the tests did not differentiate between the FN400 for exvolvement (M = 2.48 µV) and involvement (M = 2.85 µV, p = 1.00). The FN400 for semantically incongruent sentences required a larger sample size for an accurate interpretation, F(2, 28) = .50, p = .58, ηp 2 = .03, 1−β = .11. For pragmatically incongruent sentences, F(2, 28) = 6.39, p < .01, ηp2 = .31, 1−β = .80, avolvement (M = .81 µV) was larger than involvement (M = 2.94 µV, p < .001) but avolvement showed no difference over exvolvement (M = 2.00 µV, p = .37, see Figure 4).

The 600-850 ms time window. A positive-going deflection followed the FN400 response to semantically incongruent sentences (not congruent or pragmatically incongruent sentences) for exvolvement and involvement over AF3/AF4, Fz, F3/4, FCz, FC3/FC4, Cz, C3/4, Pz, P3/4, P7/8, PO7/PO8, and Oz.

To address the third hypothesis and investigate the main effect of the instructions on LPC, a two-way repeated-measures ANOVA was applied to the mean amplitude estimates. Based on the results, the LPC mean amplitude was larger for exvolvement (M = 3.35 µV) than for avolvement (M = 2.17 µV, p < .01), F(2, 28) = 3.70, p < .05, ηp2 = .20, 1−β = .43 (see Figure 5).

Panel A = Grand average (n = 15 subjects) ERPs at Pz for the main effect of instructions on semantically incongruent condition. Panel B = The maps show different degrees of LPC over the posterior locations. Panel C = The bar plots depict the significant difference between avolvement and exvolvement.

Discussion

Response Accuracy and Response Time

Based on the RA results, the first hypothesis was confirmed. This implies that gradual enhancement in the level of sensory involvement meaningfully increases the subjects’ RA. This is consistent with the findings of Shahian et al. (2017), who reported a positive relationship between emotioncy and reading comprehension.

In the comparison of differences between the three types of instructions in each single condition, only the sentences with pragmatically incongruent sentences were influenced by the EBLI (e.g., “Quinoa is bitter.”). Yet, the findings revealed no significant difference between exvolvement and involvement in congruent and semantically incongruent sentences. One line of explanation could be that the subjects could easily judge the semantically incongruent sentences due to their more obvious anomaly (e.g., “Quinoa is angry.”). They were even able to respond to the semantically violated avolvement sentences for which they had no information. This shows that only a minimum level of language proficiency was necessary to decide upon the lexical acceptability of the sentences. Studying the N400 effect, McLaughlin et al. (2004) concluded that, in the second language domain, the learners were able to recognize semantically irrelevant words after 63 hours of classroom instruction. Since pragmatic knowledge pertains to personal experiences retrieved from the episodic memory (Kiefer & Pulvermüller, 2012), it is possible to infer that, unlike semantic comprehension which may be independent of instruction, different integration of the senses seem to modify one’s knowledge of the world and pragmatic comprehension.

The results proved the reliable effect of the EBLI on the subject’s reaction time. In particular, they were able to react more quickly in response to the sentences with involved target words. As fewer senses were integrated in exvolvement, the subjects needed more time to discern the pertinent sentences. The subjects responded to a number of avolvement sentences which, according to their impressions, were easy to conjecture. However, even to guess the meanings, they spent more time than they did for exvolvement and involvement. Given that RT reflects the integrative process of comprehension (Rösler et al., 1993), we presume that involvement has a stronger effect on the facilitation of this operation. Furthermore, the subjects’ reaction time to the pragmatically incongruent sentences was considerably modified by the number of senses being involved (“A salak has a flower.”, standing against its correct version: “A salak has a seed.”). It is justifiable to deduce that the combination of senses boost pragmatic comprehension, hence reducing the RT.

The FN400

The status of this N400-like component was assessed through the statistical analysis of the ERPs along with the inspection of the topographies. The outcomes featured some sentential FN400 activities in the anterior scalp, with maximal amplitude found over the anterio-frontal and frontal sites. Despite its atypical scalp distribution, we assume that this negativity is a form of the N400, qualified as FN400, since the nature of the effect regarding its shape and timing characteristics resembles that of classical N400. There exists conflicting evidence regarding the functional features of the FN400. Grounding in a number of experimental studies, concrete stimuli generate a more frontally distributed N400 in comparison with their abstract counterparts (e.g., Kounios & Holcomb, 1994). Adopting an alternative approach, Stenberg et al. (2009) and Bridger et al. (2012) characterized the FN400 as a familiarity-based recognition marker, supporting episodic memory. According to Hintzman and Curran (1994), recognition constitutes a pair of independent processes, namely, familiarity and recollection of contextual details, rendering over midline frontal (FN400, between 300-500 ms) and left parietal (LPC, between 500–700 ms) regions, respectively.

Critically, our findings revealed a shift away from the belief that considered N400 and FN400 functionally unrelated. It is reasonable to assume that the FN400 is not a constituent of the biphasic negative-positive complex, since the late positive component following the FN400 was absent from the parietal areas of the scalp across the congruent and pragmatically incongruent conditions. However, the behavioral estimates point to the accurate retrieval of the target information by the subjects. Therefore, unlike Bridger et al.’s (2012) assumption grounded in the single-process theories, we can presume that recognition and familiarity inspection jointly occur in form of the FN400. Similar to Paller et al. (2007) and Voss and Federmeier (2011), we conclude that the FN400 may function as a marker of semantic processing influenced by the feelings of familiarity.

To check the extent to which the integration of different senses changes the amplitude of the FN400, a set of statistical procedures was used. As the analyses of the component revealed, among the congruent condition of the three instruction types, an FN400 component was visible to avolvement only. Such a finding gives rise to the FN400 as an indicator of semantic processing whose amplitude is manipulated by the degree of coherence of the sentence. The FN400 deflection was additionally seen in the incongruent conditions of the avolved sentences. Although the target words of these sentences seemed unknown to the subjects, some sort of prior knowledge of the topic existed. All the target words were edible foodstuffs and the subjects could overgeneralize it to the unknown items. Thus, they were able to superficially analyze the sentences by putting their schema or topical knowledge into practice.

Based on the F tests, the second hypothesis was partially confirmed. The main effect of instruction distinguished between the sentences with no sensory instruction (i.e., avolvement) and the ones with instructions (i.e., exvolvement and involvement). The findings do not match with the behavioral ones, suggesting that there might be other, later neurocognitive effects which account for the act of comprehension.

Comparing the interaction effect of instruction and condition, we observed that, for pragmatically incongruent sentences, the FN400 amplitude was more negative to avolvement than involvement. From physiological perspectives, the diminution of amplitude may be associated with “the activation of fewer neurons” (Kutas & Federmeier, 2011, p. 625). Therefore, when full sensory involvement is provided, there is less pressure on the brain and pragmatic comprehension is facilitated.

The Late Positive Component

In addition to the FN400 effect, semantically incongruent completions led to a late-emerging positivity with its maximum over posterior locations in the time window of 600 to 850 ms. The positive shift could not actually be regarded as a late recollection-like pattern (Hintzman & Curran, 1994) since the effect was absent in the pragmatically incongruent sentences. Therefore, the third hypothesis was not confirmed. Critically, however, the component was similar in morphology to the ones reported in the studies which associated the shift with the later stages of semantic integration processing. The patterns of the LPC changes conformed to the presumption that this component is significantly modified by linguistic conditions, indexing a different cognitive reaction to semantic violation (Salmon & Pratt, 2002). Relying upon the reduced awareness hypotheses, when the sentence-final word is a good fit to the global context, attention is directed toward other features of the sentence rather than reprocessing of the misfit word (Sanford et al., 2011). In comparison with pragmatically incongruent sentences (e.g., “A caper looks like an apricot.”), or what Sanford et al. referred to as borderline anomalies, semantically incongruent sentences (e.g., “A caper looks like an event.”) go through a more complex languagerelated cognitive processing since they are not primarily predicted. To detect the violation of semantic constraints, the meaning of the words is not retrieved fully, but only to the extent that is needed to uncover the irrelevance of the final word to the context (Sanford et al., 2011; Siochrú, 2018).

Some authors claim that (e.g., DeLong et al., 2011) late positive ERPs have their roots in the disconfirmation of the prediction formed during language comprehension in the sentences with semantically incongruent endings. Van Petten and Luka (2012) attributed the shift to “cognitive costs in recovering from an incorrect guess” (p. 187). Others (e.g., Kos et al., 2012), however, posited that since easy-to-detect anomalies look fairly odd, the incremental build-up of semantic meaning goes through a longer reanalysis phase in which the reader monitors his/her comprehension and rechecks the suitability of the misfit item to the sentential context (e.g., “A rambutan has a car.”). In other words, every endeavor made to revise problematic elements of a sentence triggers retrieval or reactivation of the rest of the words in the sentential context (Van Petten & Luka, 2012).

According to the results, the LPC was more positive to exvolvement than to avolvement. Given that the depth of processing influences the LPC amplitude (e.g., Richardson-Klavehn et al., 1996; Toth et al., 1994), presumably, due to insufficient sensory experiences, the sentences with exvolved words needed more in-depth analysis to reinterpret the sentential context. Since the major discrepancy between exvolvement and involvement has some connections with the degree and accessibility of world knowledge, incomplete world knowledge may impose additional mental effort to the subjects’ memory.

Overall, although the instruction was short in duration and large differences were not observed in the ERP components, the results revealed that different levels of sensory involvement influence semantic processing to different degrees. While full sensory involvement improves pragmatic comprehension, limited sensory involvement exerts more mental effort during semantic comprehension. Data from this study could further inform the debate over the functional characterization of the FN400. In addition, the results privileged the history of post-FN400 positivities, suggesting that like the FN400, the LPC, as a reanalysis marker, is rather sensitive to sensory involvement. As such, different combinations of senses may increase or decrease the depth of reanalysis. Moreover, we observed that the memory reaction to semantic and pragmatic comprehension is rather idiosyncratic. While good-fit conditions elicit a monophasic pattern (i.e., the FN400 only), poor-fit items evoke biphasic ERP responses with the FN400 proceeded by a broadly distributed LPC. Finally, the current study was not without limitations, the most important of which was the small sample size and the instruction time. To confirm the insignificant results and enhance the validity of the findings, further explorations are advised on greater number of subjects, using a series of instructions. Moreover, due to the paucity of research on the number and order of senses to combine, we made use of the emotioncy model to open up new directions for research in this area. Owing to the poor spatial resolution of the scalp EEG, complementary fMRI studies are called for to provide precise brain maps as a function of sensory involvement. Conducting the study on the first language as well as the abstract concepts will similarly provide us with valuable findings.