N400 amplitude reduction correlates with an increase in pupil size.

Pupil dilation is classically associated with increase in cognitive load in humans. Here, we studied the potential link between human pupil dilation and meaning integration effort as indexed by event-related brain potentials (ERPs). We recorded pupil size variation and ERPs simultaneously while participants were presented with matching or unrelated picture-word pairs. Whilst relatedness in meaning between spoken words and pictures typically modulated ERPs, pupil size was also affected quickly after picture onset. Moreover, during the time-window associated with meaning integration, greater pupil dilation correlated with less negative N400 amplitudes elicited by unrelated pictures. Since pupil dilation has been linked to activity of the locus coeruleus-norepinephrine (LC-NE) system, these findings may provide new insights into the suggested link between human high-level cognitive function and activity of the LC-NE system.

Introduction

For centuries, dilation of the pupil has been associated with ongoing cognitive processes in the human brain (Loewenfeld, 1958; Beatty, 1982b). Empirical evidence from the beginning of the twentieth century has shown that, in general, mental processing effort increases pupil size (Hess, 1975; Karatekin, 2007). For instance, the pupil dilates with increasing load on memory (Kahneman and Beatty, 1966; Peavler, 1974), problem solving (Hess and Polt, 1964), or perceptual processing (Hakerem and Sutton, 1966).

The human pupil response is hypothesized to be composed of (a) an early component accounting for most of the first dilation phase (i.e., within the first second) due to the inhibition of the parasympathetic system, in turn, leading to relaxation of the pupillary contraction muscles, and (b) a second component (involved beyond 900 ms) attributed to the activation of the sympathetic system, and corresponding to the contraction of pupillary dilator muscles (Steinhauer and Hakerem, 1992; Loewenfeld, 1993).

Recently, stimulus evoked dilation of the pupil has been suggested to correlate with phasic activity of the locus coeruleus (LC; Aston-Jones and Cohen, 2005), although the precise nature of the relation between LC neuronal activity and pupil dilation is still debated (Nieuwenhuis et al., 2011). The LC system is the sole source of the neuromodulator norepinephrine (NE) in the forebrain making it (in part) responsible for the regulation of cognitive, emotional, and motivational states (Samuels and Szabadi, 2008a; Sara, 2009). Indeed, as shown in studies involving in vivo stimulation or recording of LC neurons’ activity in monkeys or rodents performing sensory and memory tasks, increasing phasic LC–NE system activity is associated with increasing task performance and stimulus processing efficiency (Usher et al., 1999; Berridge and Waterhouse, 2003; Aston-Jones and Cohen, 2005; Samuels and Szabadi, 2008b). Consequently, an increasing number of studies have used stimulus evoked pupil dilation as an indirect measure of phasic activity of the human LC–NE system in the absence of intracranial recordings of human LC neural activity (e.g., Einhauser et al., 2008; Gilzenrat et al., 2010; Gabay et al., 2011; Jepma and Nieuwenhuis, 2011).

Here, we investigated the potential link between a phasic pupil dilation response and human high-level cognitive performance. We recorded pupil size measurements and event-related potentials (ERPs) simultaneously. ERPs are average electrical brain potentials recorded from the surface of the scalp and provide insight into human brain function with high-temporal resolution. By correlating pupil size with ERP amplitude from stimulus onset, we investigated whether pupil dilation is associated with the cognitive processes reflected in ERPs. A temporal association between the two measures would suggest that LC–NE system activity impacts particular cognitive processes.

The N400 ERP component was used here as a reliable index of a high-level cognitive process. The N400 is a negative going ERP wave that increases in amplitude with increasing mental effort involved in semantically integrating a stimulus in its context whether it is a (spoken or written) word or a picture (Chwilla et al., 1995; Kutas and Federmeier, 2000).

Participants were presented with picture–spoken word pairs while their pupil size and ERPs were recorded in parallel. The picture and word of each stimulus pair were either matched in meaning (a picture of a horse and the word “horse”) or unrelated (a picture of a horse and the word “flower”). In one experimental block, each trial featured a spoken word shortly followed by a picture presented at fixation (picture-target) and in another block, stimulus order was reversed (word-target; Figure 1). The participants were asked to pay attention to the picture and word pairs but were not overtly engaged in a task to prevent purely task-driven modulation of pupil dilation. Note that such a paradigm has been used before in ERP studies and shown successful N400 modulations (Relander et al., 2009).

Figure 1

Trial procedure in the picture-target and word-target blocks.

Although we anticipated a N400 modulation of ERP amplitude by the degree of semantic match between picture and word in stimulus pairs, we could not predict whether pupil size would be modulated in the same time range in which ERP modulations are typically observed. Maximal pupil size modulations are usually observed beyond the first second after stimulus onset (Beatty, 1982b), however, the first phase of a phasic pupil dilation response start 200 ms after stimulus onset and has a maximal amplitude at around 500–600 ms (Steinhauer and Hakerem, 1992) which is not dissimilar to the time course of N400 ERP response.

The order of picture and word stimuli was manipulated in two experimental blocks to reveal whether a possible modulation of pupil size reflects an interaction between visual processing and semantic integration (picture-target block; visual stimulus present) or whether it also generalizes to auditory semantic integration in the absence of a visual stimulus (word-target block; no visual stimulus present). If semantic match would modulate the phasic pupil dilation response in both blocks, this would suggest that semantic priming effects are reflected in the pupil dilation in general.

Materials and Methods

Participants

Twenty students of Bangor University (mean age 19 ± 1.2; 10 female; 17 right-handed) gave written consent to participate for course credits in the experiment that was approved by the local ethics committee. They were native speakers of English, had uncorrected normal vision, and no self-reported neurological disorders. The data of five participants were discarded due to technical problems or excessive levels of noise in the ERP signal.

Stimuli

Fifty-two color pictures of highly familiar objects or animals were selected from an in-house database. The basic-level English name of each picture was spoken by a woman and digitized at a sampling rate of 44.1 kHz, 16 bit encoding, mono. Average word duration was 687 ± 39 ms, mean familiarity and concreteness ratings were (on a scale from 100 to 700): 563 ± 19 and 596 ± 10, and mean Kucera–Francis frequency (Kucera and Francis, 1967) was 85 ± 35 (Coltheart, 1981). Stimuli were identical across matched and unrelated conditions but differed in their pairing, that is, there could be no difference in either ERP or pupil measurements resulting from low-level perceptual differences between conditions within each of the two experimental blocks.

Data acquisition

Event-related potentials were recorded at 1 kHz DC, band-pass filtered between 0.1 and 200 Hz from 32 Ag/AgCl electrodes placed according to the 10–20 convention (Fp1, Fp2, F7, F3, Fz, F4, F8, FT7, FC3, FCZ, FC4, FT8, T7, C3, Cz, C4, T8, TP7, CP3, CPZ, CP4, TP8, P7, P3, Pz, P4, P8, O1, OZ, O2, right mastoid) and referenced to the left mastoid. Eye movements were recorded via vertical and horizontal electrooculogram channels and impedances were kept below 5 kΩ. Continuous recordings were filtered off-line with a zero-phase shift low-pass filter of 30 Hz and then re-referenced to the average of the left and right mastoids. Eye-blinks were mathematically corrected using Scan 4.4 (NeuroScan, Inc.) and remaining artifacts were removed using a ±75 μV artifact rejection procedure. Epochs ranged from −100 to 850 ms relative to the onset of the target stimulus and were baseline corrected in reference to pre-stimulus activity. Pupil size was recorded with a remote eye-tracker (TOBII X60; 60 Hz sampling rate) placed approximately 70 cm from the participant's eyes. Each participant's average of left and right pupil size was epoched (−100 to 850 ms), baseline corrected in reference to a 100 ms pre-stimulus interval, and filtered low pass at 10 Hz also using Scan 4.4. The 10 Hz low-pass filter setting corresponds to filter settings commonly used for pupil dilation data (e.g., Hupe et al., 2009). Intervals in which the connection with at least one of the eyes was lost and values <1 and >5 mm were discarded (7.8 ± 2.4% in total).

Procedure

The experiment took place in a normally illuminated room (30 cd/m2 background luminance and 156 cd/m2 at the screen) to maintain tonic parasympathetic activity enabling detection of the first phase of pupil dilation (Steinhauer et al., 2004). Participants were seated at 1.8 m from a high resolution LED 40′′ screen (Samsung UE40B80000) and were instructed to passively attend the stimuli and fixate the center of the screen or the picture if presented. Pictures spanned a maximum of 9° of visual angle. Each trial started with presentation of a fixation cross and the spoken word “look” (duration 700 ms) followed by the prime stimulus (picture or word; duration 750 ms), a 100 ms inter stimulus interval, and the target (picture or word) presented for a duration of 1200 ms (Figure 1). Auditory stimuli were presented with 60–68 dB intensity via loudspeakers set 2 m in front of the participant. The picture–word pairs were constructed such that each word was paired with a matching picture (e.g., horse–horse; match condition) or an unrelated one (e.g., horse–flower; unrelated condition) avoiding semantic, phonological, and orthographic overlap. Overall, 104 word–picture pairs were presented twice in a random order in each block separated by a short break. Block order was counterbalanced across participants. In the event of participants failing to fixate the center of the screen or when the eye-tracker lost connection with both the eyes the experiment was automatically paused until connection was restored.

Statistical analysis

First we performed time-step t-tests on mean amplitude of 50 ms long bins on six electrodes traditionally associated with the N400 (C3, Cz, C4, CP3, CPz, CP4) and on pupil size in order to gain insight in the relative timing of any modulation by semantic relatedness. Next, to corroborate the findings of the ms-by-ms tests, we performed repeated measures analysis of variances (ANOVAs) on individual mean ERP amplitudes with the factor electrode (the same subset of six centroparietal electrodes) and the factor relatedness (semantic match vs. unrelated). Based on the pattern of the grand average ERP waveforms (Figure 2), N400 peak latency was detected at Cz between 280 and 400 ms in both blocks and the same time windows were used for mean amplitude analyses. As the pupil response was slower than the ERP response, we chose a longer time-window for the mean amplitude analysis of pupil size (300–500 ms), which nevertheless is still within the traditional N400 time range. Mean pupil size was subjected to ANOVAs for each block separately. Finally, to investigate the potential link between the two measures, we correlated pupil size (measurements at each 16 ms) with concurrent mean ERP amplitude (in 10 ms time windows around each 16 ms). Largest correlations were observed at nine frontocentral electrodes (F3, Fz, F4, FC3, FCz, FC4, C3, Cz, C4) and we used the individual average ERP signal of these channels for the correlation analyses.

Figure 2

Correspondence between ERP and pupil dilation measurements. (A) Grand average ERP waveforms of a linear derivation of 6 Centro-Parietal electrodes in the match and mismatch conditions in the picture block (picture-target) and the word block (word-target). Shaded areas show significant modulations (p <.05) by semantic match. (B) Grand average pupil dilation records in the match and unrelated conditions of the picture and word blocks.

Results

The time-step t-tests on 50 ms long bins were performed every millisecond for ERPs and every 16 ms for pupil dilation records according to their sampling rate. Following the Guthrie and Buchwald (1991) method, a minimum sequence of 12 consecutive significant t-tests for pupil dilation and ERP amplitude was used to compensate for Type I errors (both the ERP amplitude and the pupil dilation had an autocorrelation value larger than 0.9). In the picture-target block, the modulation of the ERP amplitude was significant from 100 to 189 ms and from 239 to 433 ms (Figure 2A). T-tests on the pupil size were significant from 366 to 800 ms (Figure 2B). In the word-target block, the modulation of the ERP amplitude was significant from 111 to 699 ms. Pupil size was not significantly modulated in the word-target block.

The analysis on ERP amplitudes revealed that the N400 was maximal at 332 ms in the picture block and at 348 ms in the word block. Peak latencies were not affected by relatedness (both p > 0.3). The mean N400 amplitude analysis in the picture block (Fp) and the word block (Fw) revealed significant effects of electrode (Fp5,70 = 12.2, p < 0.001, η2 = 0.47; Fw5,70 = 10.7, p < 0.001, η2 = 0.43) and relatedness (Fp1,14 = 24.6, p < 0.001, η2 = 0.64; Fw1,14 = 9.4, p < 0.01, η2 = 0.4) but no interaction (both p > 0.1).

In the picture-target block, mean pupil dilation was larger in the unrelated condition than the match condition (Fp1,14 = 5.0, p < 0.05, η2 = 0.26; Figure 2) while there was no significant modulation of pupil size in the word block (p > 0.2).

Next, we performed correlation analyses on ERP amplitude and pupil size from stimulus onset onward. In the word-target block, no reliable significant correlations were found between ERP amplitude and pupil size (a maximum of two consecutive significant tests). In the match condition of the picture-target block, significantly positive correlations were found from 100 to 150 ms and from 333 to 500 ms post stimulus onset (Figure 3). In the corresponding unrelated condition, correlations were also positive and significant from 250 to 416 ms. Finally, the semantic effects in ERP amplitude and pupil size were significantly negatively correlated between 283 and 350 ms.

Figure 3

Event-related potentials–pupil size correlations. Time-step (each 16 ms) monitoring of correlation coefficients between ERP amplitude and pupil size records in the picture-target block.

Discussion

To study the potential link between pupil dilation and ERP responses we recorded both measures simultaneously in a semantic priming context. As expected, N400 amplitudes were significantly more negative for target stimuli unrelated to their prime stimulus than related ones in both experimental blocks. Pupil size, on the other hand, was significantly increased for target pictures unrelated to prime words as compared to pictures matching their prime words, but no such modulation was found for target words in the word block.

To our knowledge, this is the first demonstration of pupil size modulation by semantic priming within the first second after stimulus presentation. These results suggest that the auditory prime words prompted access to corresponding visual representations. Indeed, connections between the auditory and visual cortex have been found in monkeys (Falchier et al., 2002) and humans (Eckert et al., 2008) and attended auditory prime words have been shown to activate areas in the primary visual cortex (Cate et al., 2009). We propose that due to the auditory priming of low-level visual features, the detection of unexpected visual features lead to a greater pupil accommodation response than when visual features were expected.

The absence of a semantic priming effect on pupil dilation when auditory words were targets shows that detection of unexpected phonemes did not directly elicit a pupil dilation response. Hence, although auditory neurons are known to respond to NE (released by the LC) in the monkey (Foote et al., 1975) and pupil dilation can be triggered by auditory input (Beatty, 1982a), it seems that there is no functional connection between the auditory orienting response and pupil dilation during the first second after stimulus onset. Indeed, in the word-target block, participants did not anticipate upcoming pictures and whereas a visual stimulus can be processed for the whole duration of its presentation, an auditory stimulus affords no sustained exploration beyond its end point. However, this does not exclude the possibility that auditory semantic incongruence could lead to an increase in pupil dilation beyond the time-window of interest here.

The time-step correlation analyses between pupil diameter and ERP amplitude revealed significant correlations between pupil size and ERP amplitude in the picture-target block. In the N400 time-window, pupil dilation and ERP amplitude were positively correlated in both the match and unrelated conditions. Hence, whether the picture's identity was expected or not, ERP amplitude and pupil size appeared to be functional connected. The significant correlation in the match condition may seem surprising, but it shows the variance in the extent to which the pictures presented were good examples of the prime words.

The fact that the correlation was positive implies that participants who displayed large negative ERP amplitudes in the time range of the N400 displayed small pupil dilation in the same time-window. Since more negative ERP amplitudes in the N400 time range are classically associated with more effortful semantic integration (Chwilla et al., 1995) and because less pupil dilation is thought to reflect less phasic activity of the LC–NE system (e.g., Aston-Jones and Cohen, 2005), these results suggest that more effortful semantic integration is associated with less LC–NE system activity, and thus a lower level of cortical arousal.

The above conclusion is corroborated by the fact that the correlation between the pupil size and the N400 effect was negative. Participants who displayed a large N400 effect showed a small semantic effect in pupil dilation. These results suggest that a stronger phasic LC–NE response goes hand-in-hand with more efficient semantic integration in the human brain, which is in line with the more general claim that phasic LC–NE activity increases stimulus processing efficiency (Aston-Jones and Cohen, 2005). It must be noted however, that our findings are of a correlative nature and that alternative (i.e., direct) measurements of LC–NE activity would be necessary to confirm our findings and their interpretation.

To conclude, our results show that visual exploration of a stimulus triggers a stronger pupil accommodation response when visual features are unexpected than when they were expected. Since stimulus related pupil dilation has been associated with phasic activity of the LC–NE system, our results suggest that phasic activity of the LC–NE system has a potential impact on the efficiency of visual semantic integration. In line with current research on pupil dilation, task performance, and LC–NE activity, we show that simultaneous pupil dilation measurement and ERP recording are proving useful in the study of human brain function underlying cognition given their high-temporal resolution, their non-invasiveness, and the complementary insights they provide.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.