Spontaneous eye-movements in neutral and emotional gaze-cuing: An eye-tracking investigation.

Our attention is spontaneously oriented in the direction where others are looking. This attention shift manifests as faster responses to peripheral targets when they are gazed at by a central face instead of gazed away from, and this effect is even more pronounced when the face expresses an emotion. This so called gaze-cuing effect, and its enhancement by emotion, is thought to reflect covert attention orienting. However, eye movements are typically not monitored in gaze-cuing paradigms, yet free viewing and saccadic reaction time research suggests individuals commonly and quickly look at gazed-at locations. Furthermore, in dynamic gaze-cuing studies, emotional faces differ from neutral faces in their affective content but also in their apparent facial motion, both of which could affect participants' eye-movements. We investigated the contribution of overt orienting to the gaze-cuing effect by monitoring eye-movements during emotional and neutral gaze-cuing trials. We found that eye-movements were infrequent, and when they occurred, they were directed toward the target, not toward the gazed-at location. Removing trials with eye-movements did not affect gaze-cuing much, confirming it reflects a covert attention process. However, participants were more likely to move their eyes during neutral trials, which lacked perceived face movement, than during emotion trials or neutral movement trials. Including these eye-movement contaminated trials in our analysis resulted in an impaired ability to detect the gaze-cuing variations with emotion. In contrast, removing trials with eye-movements, or including a neutral movement control such as a neutral tongue protrusion, revealed more subtle emotional modulation of gaze-cuing.

Introduction

Gaze direction and facial expressions are powerful nonverbal cues about the mental states of others (Baron-Cohen, 1995; Itier and Batty, 2009; Gobel et al., 2015; Hamilton, 2016). While gaze direction provides spatial information regarding the location of others' attention, facial expressions afford emotional information related to the gazed-at object (Ekman and Friesen, 1971). When individuals observe others shifting their gaze, their attention spontaneously shifts in the direction of the gazed-at location (Driver et al., 1999; Friesen and Kingstone, 1998; Langton et al., 2000). This phenomenon is known as gaze-cuing, and it has been shown to be modulated by facial expressions, such that gaze-cuing is typically enhanced for faces expressing an emotion compared to neutral faces (e.g. Putman et al., 2006; Tipples, 2006). Understanding how gaze and emotion cues interact to orient attention is important because gaze-cuing is believed to be a necessary precursor to joint attention, the ability to orient attention to what others are attending to (Scaife and Bruner, 1975). Joint attention is how we share experiences with others as infants, and is crucial for our social development (Baron-Cohen, 1995; Edwards et al., 2015; Brooks and Meltzoff, 2014; Moore et al., 2014). It is no surprise then that failure to integrate gaze and emotion cues has been reported in individuals with Autism Spectrum Disorder (Uono et al., 2009; De Jong et al., 2008) and is related to poor social functioning (Hayward and Ristic, 2017) and autistic like traits within the general population (McCrackin and Itier, 2018b).

Gaze-cuing is typically studied using a modified Posner cuing task (Posner, 1980), whereby a centrally-presented face looking to the left or to the right is followed by the presentation of a peripheral target. Attention orienting by gaze is indicated by faster response times to targets appearing on the gazed-at (or gaze-congruent) side than to targets appearing on the side opposite to gaze direction (gaze-incongruent). This gaze-cuing effect is seen reliably even when participants are told that the location of the target is not predicted by the direction of gaze (see Frischen et al., 2007 for a review). As participants are carefully instructed to keep their eyes fixated on the face for the whole trial duration, the currently held view is that gaze cues effectively orient covert attention, that is, attention deployed without the use of eye or head movements. However, whether overt attention orienting towards the gazed-at location through eye-movements could also impact target detection, and thus modulate the gaze-cuing effect, remains unclear as eye movements are rarely monitored. In fact, participants often anecdotally report difficulty in overriding an initial tendency to follow gaze shifts with eye-movements. The early work by Friesen and Kingstone (2003) suggested that the gaze-cuing effect was still present after removing trials contaminated by blinks and saccades. However, this investigation was performed in a small sample using schematic face stimuli and should be replicated with real face images. It should also be extended to include emotional faces to investigate the possibility that eye-movements contribute to the modulation of gaze-cuing by emotion, which to the best of our knowledge has never been investigated.

It is possible that eye-movements play a greater role in classic gaze-cuing studies than currently believed, especially in view of findings from both free-viewing and saccadic reaction time tasks that highlight participants' strong tendency to spontaneously look towards gazed-at locations. During free-viewing conditions, viewers actively follow magicians' gaze with their eyes during filmed magic trick performances (Kuhn and Land, 2006; Kuhn et al., 2009) and look towards where people displayed in natural scenes (Hermens and Walker, 2016) and paintings (Dukewich et al., 2008) are looking. In saccadic reaction time tasks, both adults and infants (Hood et al., 1998) are also faster to look from a central face to a gaze-congruent target than to a gaze-incongruent target, regardless of whether the stimuli used are real face sequences (Ricciardelli et al., 2002; Mansfield et al., 2003; Hood et al., 1998), schematic faces (Kuhn and Kingstone, 2009; Kuhn et al., 2010; Kuhn and Benson, 2007), or face videos (Hermens and Walker, 2012). Furthermore, gaze-congruent saccades occur quite often (e.g. Ricciardelli et al., 2002) and even when the gaze-cue negatively predicts where the target location will be 80% of the time (Kuhn and Kingstone, 2009), suggesting that there is a strong tendency to look in the gazed-at direction even when it is counterproductive to the main task. In contrast, a recent study suggests that eye movements in the direction of gaze-cues can be inhibited fairly well (Zeligman and Zivotofsky, 2018). The contribution of possible eye movements to the gaze-cuing effect thus deserves a re-examination.

It is also unknown whether overt attention could play a role in the emotional enhancement of gaze-cuing. Many studies have shown that the gaze-cuing effect is larger for fearful than for happy and neutral faces (Bayless et al., 2011; Graham et al., 2010; Lassalle and Itier, 2013, 2015a,b; Neath et al., 2013; Putman et al., 2006; Tipples, 2006; McCrackin and Itier, 2018a; McCrackin and Itier, 2018b), and more recent studies suggest that happy expressions can also enhance cuing effects relative to neutral faces (McCrackin and Itier, 2018a, 2018b). It is possible that overt attentional effects contribute to the emotional enhancement of gaze-cuing, especially if eye-movements toward the gazed-at location occur more frequently, or are faster, for emotional expressions than for neutral expressions. Some studies suggest that emotional enhancement of gaze-cuing is strongest at cue-to-target intervals longer than 300 ms (e.g. Graham et al., 2010), which could reflect that enough time to make a saccade is needed to produce emotion effects. While more recent findings suggest that enhancement of gaze-cuing by emotions can occur at SOAs as short as 200 ms (McCrackin and Itier, 2018a), saccadic reaction times can be as fast as 170 ms (Colonius and Diederich, 2004; Fischer and Breitmeyer, 1987), or even 120 ms (Kirchner and Thorpe, 2006), under the right conditions, making saccadic contribution to this gaze-cuing enhancement possible.

Preliminary findings from a visual search task suggest that participants are indeed faster to look toward threatening targets if the target occurs in the gazed-at location following the presentation of a fearful face (though not following a happy face; Kuhn and Tipples, 2011). In addition, research suggests a facilitation of both voluntary and involuntary eye-movements toward emotional stimuli. For instance, participants voluntarily orient their eyes faster to fearful than to neutral faces (Bannerman et al., 2009), and positive and negative pictures (including face and non-face stimuli) are fixated before neutral pictures during free-viewing sessions, even when participants are explicitly instructed to look at neutral pictures first (Nummenmaa et al., 2006). Although in these studies, the emotional or threatening stimuli are the targets (while in the gaze-cuing paradigm emotional faces are the cues informing about the nature of the target in the environment - possibly a threat), the emotional content of faces in gaze-cuing studies could elicit spontaneous eye movements that could impact the gaze-cuing effect.

Finally, in dynamic gaze-cuing sequences, perceived motion could also elicit a greater amount of overt orienting for emotion than for neutral trials. Lassalle and Itier (2015a) showed that gaze-cuing paradigms in which the gaze shift occurs before the expression of emotion produce the strongest gaze-cuing enhancement with emotion. Pictures of the same individual face are presented back to back in such a way that a person with direct gaze appears to avert gaze and then to either react emotionally (emotion condition) or not at all (neutral condition) to something in the environment. In this sequence, emotion trials differ from neutral trials not only in their emotional content, but also in their apparent facial motion content. It is thus possible that apparent motion, rather than affective content, elicits more eye-movements in emotional trials, in turn modulating the gaze-cuing effect to a greater extent for emotional than neutral trials. Indeed, there is support for the idea that processing biological motion requires less attention than stimuli lacking biological motion, which may free up attentional resources to focus or to look elsewhere (Thompson and Parasuraman, 2012), though motion that is predictive of a target location and the sudden onset of a target have also both been shown to draw attention during a free viewing task (Hillstrom and Yantis, 1994).

To the best of our knowledge, the present study is the first to systematically monitor and compare participants' eye-movements in a dynamic gaze-cuing paradigm involving faces with emotional expressions and neutral faces with apparent motion. In this study, the movement data that had been collected in McCrackin and Itier's Experiment 4 (2018a) were analysed. In this experiment, participants completed fearful and happy gaze-cuing trials, along with two types of neutral trials. The first was the classic neutral face condition typically used in the literature (hereafter referred to as the “classic neutral” condition), which lacks the apparent motion found in the emotion trials. The second was a previously validated neutral condition, where the neutral face protrudes its tongue so that face motion is still perceived but the face does not display any other changes that would be perceived as one of the six basic emotional expressions (hereafter “neutral tongue” condition; see the discussion on this point and the stimuli validation in McCrackin and Itier, 2018a). As in the typical gaze-cuing paradigm, participants were instructed not to move their eyes during each trial. The main goal was to re-examine the contribution of overt attention to the (neutral) gaze-cuing effect, and to assess its contribution to the enhancement of gaze-cuing with facial expressions for the first time. Second, we assessed the extent to which trials were contaminated by involuntary eye movements, whether certain emotions elicited more eye-movements than others, and whether those eye movements were driven by the face's apparent motion or by its affective content. Finally, we investigated whether these eye movements were made in the direction of the face's gaze, which would suggest obligatory orienting of overt attention, or whether they were made mostly toward the target location. We predicted that 1) both the gaze-cuing effect for neutral faces and the emotional enhancement of gaze-cuing would hold after removing trials contaminated with eye-movements, supporting the assumption that they are primarily driven by covert attention; 2) when participants made saccades, they would make them primarily in the direction of the face's gaze, reflecting a tendency to orient the eyes to a gazed-at location; 3) orienting of the eyes toward the gazed-at location would occur more for faces displaying an emotional expression than for neutral faces, due to the faces' affective content.

Methods

Participants

Forty-seven undergraduate students at the University of Waterloo (UW) received course credit in exchange for their participation. All participants were between the ages of 18–23, with normal or corrected-to-normal vision. Three participants were excluded due to poor accuracy, 9 were removed due to technical issues with the eye tracker and 3 were excluded because their proportion of trials with one or more eye-movements during either the face sequence and/or target sequence (see below) fell beyond 3 standard deviations from the group mean, leaving a final sample of 32 participants (17 males, 15 females, mean age = 20 years – SD = 1.16).

The UW Research Ethics Board approved the study in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki), and participants provided informed written consent upon arrival. Participants reported being free of neurological or psychiatric illness and reported no previous loss of consciousness for longer than 5 minutes. To ensure equivalent cultural experience and English proficiency, only those who reported living in Canada or the United States for the past 5 years were selected. Participants also rated their ability to identify faces and facial expressions on two scales ranging from 0 (extremely poor) to 10 (extremely good) and only those with self-ratings between 7 and 10 on both scales were selected to ensure the absence of any face-related impairments.

Experimental design and procedure

Eight face identities (four female) were selected from the NimStim database1 (Tottenham et al., 2009), each portraying neutral, happy and fearful expressions. The stimuli were edited to manipulate gaze direction and to create the tongue protrusion condition from the neutral expressions (see McCrackin and Itier, 2018a for details). A chinrest ensured a fixed distance of 55.5 cm from the computer screen and minimized head movements. Faces (16.17° tall and 10.41° wide) were displayed centrally on a white background in a specific sequence, as described below. Each trial (Fig. 1) began with a fixation cross (0.72° by 0.72°) centered horizontally 11.90° below the top of the screen, and randomly presented for 500, 600, 700, or 800 ms. The cross remained present for the duration of the experiment and was situated between the nose and nasion when faces were displayed.

Fig. 1

Sample trial. The neutral direct, neutral averted and neutral tongue/classic neutral/emotion frames were included in the face sequence analyses. The target frame was included in the target frame analyses (human image obtained from NimStim Face Stimulus Set, used with permissionhttps://www.macbrain.org/resources.htm).

The face sequence began with the presentation of a neutral direct-gaze face for 300 ms, followed by the same neutral face showing an averted-gaze for 100 ms, and ended with the same averted-gaze face displaying a happy expression, fearful expression, neutral tongue protrusion, or no expression at all (classic neutral condition) for 100, 150, 200, or 250 ms. Thus, the gaze-cue-to-target interval (or stimulus onset asynchrony – SOA) was 200, 250, 300, or 350 ms. This dynamic sequence was perceived as a person looking to one side and then reacting with a fearful or happy expression, simply protruding its tongue, or not reacting at all. Immediately following the offset of the final face frame, a target asterisk (0.92° by 0.92°) appeared on one side of the screen (centered vertically, 14.15° from the center) until the participant's response or for a maximum of 500 ms. Congruent trials included targets on the gazed-at side, whereas incongruent trials included targets presented on the side opposite to gaze direction. Half of the trials were congruent, and half were incongruent, with an equal number of left and right targets in each. In total, 12 blocks of 256 randomly presented trials were run. Across blocks, there were a total of 96 trials for each of the 32 main conditions (4 SOAs: 200, 250, 300, 350 ms X 4 Expressions: fearful, happy, classic neutral, neutral tongue X 2 Congruency conditions: congruent, incongruent).

Upon arrival, participants completed a demographic questionnaire. Then, they performed a target localization task by pressing the left or right arrow keys with the ring and index finger on their dominant hand, in accordance with the target location. Participants were asked to answer as rapidly as possible without compromising accuracy, and were notified that gaze direction did not predict target location. Prior to the first study block, participants completed 16 practice trials to ensure they were familiar with the task and could maintain fixation on the cross for the entire trial duration. Eye movements were recorded at a sampling rate of 1,000 Hz using a remote EyeLink 1,000 eye-tracker from SR Research. The eye-tracker was calibrated to the dominant eye, which was determined using the Miles test (Miles, 1930), but viewing was binocular. A nine-point calibration accuracy test was performed at the beginning of each block and calibration was repeated if the error at any point was more than 1°, or if the average error for all points was greater than 0.5°. SR Research Experiment Builder (SR Research, http://sr-research.com) was used to present the stimuli and record the eye-movements and responses. Experimental sessions lasted approximately 1.5 hours.

Data analysis

Reaction time data analysis

A trial was considered correct if the key response matched the side of the screen where the target was located, and if the response time was less than 2.5 standard deviations away from the participant's mean for that condition (Selst and Jolicoeur, 1994). Participants were very accurate (Mean accuracy = 94.75%, SE = 0.78%). Responses made after the 500 ms time limit were deemed a miss and discarded. Mean reaction times for correct responses were calculated for each of the 32 experimental conditions.

To investigate the impact of overt attention on the gaze-cuing effect and its modulation by facial expressions, RTs were computed twice for each condition, once with all correct trials, regardless of participants' eye-movements (“RT eye-movements” or RTem), and once after the trials contaminated by saccades or blinks were removed (“RT no eye-movements” or RTnem). A repeated measures ANOVA with the within-subject factors of Expression (4; fear, happy, classic neutral; neutral tongue), Congruency (2; congruent, incongruent), and SOA (4; 200, 250, 300, 350 ms) was then run separately on the mean RTem and RTnem. To compare with previous research, we also compared the gaze-cuing effect scores (RTincongruent – RTcongruent) between the various emotions for both types of reaction time data.

Eye-movement data analysis

Trials contaminated by blinks were first removed from all subsequent analyses. Of the remaining trials, saccadic eye-movements made during the face sequence (including the neutral direct frame, the neutral averted frame, and the neutral tongue/emotion/classic neutral frame, spanning a duration between 500 and 650 ms; see Fig. 1) and those made during target presentation (up to 500 ms) were analyzed separately. Saccades were defined as a movement of more than 0.1° of visual angle that had an acceleration of at least 8000°/s2 and a velocity of at least 30°/s.

To investigate whether participants were more likely to move their eyes during trials with certain facial expressions or stimulus-onset asynchronies, we analyzed the proportion of trials with one or more saccades using a 4 (Expression) × 4 (SOA) repeated measures ANOVA for each time period (face sequence and target presentation).

Next, we investigated whether participants had an overall bias towards making leftward or rightward eye-movements. For each participant, we computed the total number of leftward saccades and the total number of rightward saccades participants made during the face sequence and those made during the target presentation, across conditions. Paired sample t-tests compared these leftward and rightwards saccades across the group.

Beyond any bias in making leftward and rightward eye-movements, we also wanted to see whether participants were more likely to move their eyes in the direction of the face cue gaze shift, or in the direction of the target location. For each participant, the total number of leftward and rightward saccades made during the face sequence was calculated for each face gaze direction, emotion, and SOA condition. Separate repeated measures ANOVAs were performed on the total number of rightward and leftward saccades, with the factors of face gaze direction (2; left, right), emotion (4; fear, happy, classic neutral; neutral tongue), and SOA (4; 200, 250, 300, 350).

For the target presentation frame, total numbers of leftward and rightward saccades were also additionally quantified with respect to target location. Separate repeated measures ANOVAs were performed on the total number of rightward and leftward saccades, with factors of target side (2; left, right), face gaze direction (2; left, right), emotion (4; fear, happy, classic neutral; neutral tongue) and SOA (4; 200, 250, 300, 350).

SPSS Statistics 25 was used to carry out all statistical analyses. Greenhouse-Geisser corrected degrees of freedom were reported when the Mauchly's Test of sphericity was significant, and all follow-up pairwise comparisons were Bonferroni corrected.

Results

Analysis of the gaze-cuing effect computed with RTem and RTnem data

The statistical results of the 4 (Expression) × 4 (SOA) × 2 (congruency) ANOVAs run separately on the RTem and RTnem data can be found in Table 1. Interestingly, while the results are very similar between the two analyses, the direct comparison of the numbers (F, p and η values) suggest strongest effects for the RTnem data. In both analyses, the classic gaze-cuing effect was found (main effect of congruency), driven by faster responses to gazed-at (congruent) than non-gazed-at (incongruent) targets (Fig. 2a). Similarly, a foreperiod effect was found (main effect of SOA), driven by faster reaction times at longer SOAs (Fig. 2a). A significant interaction between congruency and SOA was also found for both analyses, reflecting larger gaze-cuing effects at longer SOAs.

Table 1

Results of the statistical analyses performed on RTs when trials contaminated by eye-movements were included in the mean RT calculation, and when those trials were removed from the mean RT calculation.

	Eye-movements included (RTEM)	Eye-movements removed (RTNEM)
Main effect of congruency (Gaze-cuing effect)	F(1,31) = 91.70, MSE = 1017.70, p < .001, ηp2 = .75	F(1,31) = 98.99, MSE = 802.23, p < .001, ηp2 = .76
Main effect of SOA	F(2.02,62.34) = 75.97, MSE = 603.84, p < .001, ηp2 = .71	F(1.69,52.25) = 161.34, MSE = 241.66, p < .001, ηp2 = .84
Congruency by SOA interaction	F(2.02,62.53) = 4.09, MSE = 317.14, p = .021, ηp2 = .12	F(3,93) = 5.52, MSE = 61.05, p = .002, ηp2 = .15
Main effect of emotion	F(2.36,73.25) = 9.80, MSE = 378.15, p < .001, ηp2 = .24	F(3,93) = 16.59, MSE = 104.95, p < .001, ηp2 = .35
Emotion by SOA Interaction	F(3.24,100.36) = 1.17, MSE = 550.36, p = .32, ηp2 = .036	F(5.33,165.18) = .56, MSE = 101.25, p = .73, ηp2 = .018
Congruency by emotion by SOA interaction	F(3.69, 114.46) = 1.36, MSE = 525.86, p = .26, ηp2 = .04	F(5.56, 172.30) = 1.51, MSE = 109.16, p = .18, ηp2 = .047
Emotion by congruency interaction (Emotional modulation of gaze-cuing)	F(2.22, 68.77) = 8.79, MSE = 252.78, p < .001, ηp2 = .22Congruent trials only:F(2.16, 67.08) = 7.33, MSE = 473.54, p = .001, ηp2 = .19Fear < Classic neutral (p = .039)Fear < Neutral tongue (p = .001)Happy < Neutral tongue (p = .001)Incongruent trials only:F(2.34, 72.60) = 14.38, MSE = 183.40, p < .001, ηp2 = .32Fear < Classic neutral (p = .017)Happy < Classic neutral (p = .001)Neutral tongue < Classic neutral (p < .001)Happy < Fear (p = .001)Gaze-cuing effect (incongruent – congruent):Fear > Happy (p = .002)Fear > Neutral tongue (p < .001)Classic Neutral > Neutral tongue (p = .046)	F(3, 93) = 25.79, MSE = 53.72, p < .001, ηp2 = .45Congruent trials only:F(3,93) = 17.76, MSE = 84.57, p < .001, ηp2 = .36Fear < Classic neutral (p = .001)Fear < Neutral tongue (p < .001)Happy < Neutral tongue (p < .001)Happy < Classic neutral (p = .024)Incongruent trials only:F(3, 93) = 21.93, MSE = 74.10, p < .001, ηp2 = .41Fear < Classic neutral (p = .004)Happy < Classic neutral (p < .001)Neutral tongue < Classic neutral (p < .001)Happy < Fear (p = .002)Gaze-cuing effect (incongruent – congruent):Fear > Happy (p < .001)Fear > Neutral tongue (p < .001)Classic Neutral > Neutral tongue (p < .001)Classic Neutral > Happy (p = .017)Happy > Neutral Tongue (p = .010)

Notes: All pairwise comparisons are Bonferroni corrected.

Fig. 2

Comparison of mean RTs computed with trials in which eye-movements were included (left panels) and removed (right panels). a) Congruent and incongruent reaction times (RT) for each emotion condition across the four stimulus-onset asynchronies (SOA). b) Congruent and incongruent RTs for each emotion (averaged across SOA). c) Gaze-cuing effect (RTincongruent – RTcongruent) for each emotion condition and SOA. d) Gaze-cuing effect for each emotion (averaged across SOA).

A main effect of emotion, qualified by an emotion by congruency interaction, was present for both RTem and RTnem (Fig. 2b). For both RT types, the separate analysis of incongruent trials yielded significant effects of emotion. Response times were slower for classic neutral incongruent trials than for fearful, happy or neutral tongue incongruent trials, and slower for fearful than happy trials. Note that the same pairwise comparisons were significant for both RTem and RTnem (Table 1).

Congruent trials analyzed separately also yielded main effects of emotion, but this time paired comparisons varied slightly between the RTem and RTnem analyses. For both RT types, responses were faster for fear trials than classic neutral and neutral tongue trials, and responses to happy trials were faster than to neutral tongue trials. However, in addition, significantly faster reaction times were found for happy than classic neutral trials for the RTnem only (Table 1, Fig. 2b).

As a result of these subtle variations in the congruent trials, the gaze cueing itself (RTinconruent – RTcongruent), was larger for classic neutral than happy faces, and larger for happy than neutral tongue faces, for RTnem only. In contrast, the larger cuing effect for fear than happy and neutral tongue conditions, and for the classic neutral than the neutral tongue condition, was found for both RT types (Table 1, Fig. 2c and d).

Analysis of eye-movements made during the face sequence

Proportion of trials with one or more saccades

On average, 10.59% of trials (SE = 1.52%) were contaminated by saccades during the face sequence. As seen in Fig. 3a, there was a main effect of emotion, F(3, 93) = 10.15, MSE = .001, p < .001, η = .25, driven by a larger proportion of classic neutral trials with saccades compared to fearful (p = .001), happy (p = .036) and neutral tongue (p = .004) trials. There was also a main effect of SOA, F(2.10, 64.94) = 12.45, MSE = .002, p < .001, η = .29, due to a larger proportion of trials with saccades at longer SOAs than at shorter SOAs, as seen in Fig. 3b.

Fig. 3

a) Proportion of trials with one or more saccades made during the face sequence for each emotion condition (averaged across stimulus-onset asynchrony – SOA). b) Proportion of trials with one or more eye-movements for each SOA (averaged across emotion).

Direction of eye-movements

The total number of leftward saccades that participants made across conditions (average across the group: M = 125.41, SE = 23.57) was not significantly different from the total number of rightward saccades (M = 105.38, SE = 20.73), t(31) = 1.83, p = .077, indicating no bias of eye-movements in a particular direction.

The influence of face gaze direction on saccade direction

For the leftward saccades, there was a main effect of SOA, as seen in Fig. 4a (left), F(1.95,60.30) = 9.92, MSE = 14.34, p < .001, η = .24, driven by an increased number of saccades at the longer SOAs. As shown in Fig. 4b (left), there was also a main effect of emotion, F(2.04, 63.17) = 4.87, MSE = 7.76, p = .010, η = .14, driven by an increased number of saccades made during classic neutral face sequences compared to fearful (p = .003) and happy (p = .046) sequences. However, there was no main effect of the face gaze direction (p = .495), and no interaction between face gaze direction and emotion (p = .654; Fig. 4c, left), nor between face gaze direction and SOA (p = .477).

Fig. 4

Total number of leftward (left panels) and rightward (right panels) saccades (averaged across the group) made during the face sequence, and displayed for a) each stimulus-onset asynchrony (SOA; averaged across the emotions), b) each emotion condition (averaged across all SOAs), and c) each emotion condition and face gaze direction. Note the overall larger number of leftward saccades for classic neutral compared to fearful and happy expressions and the same pattern for rightward saccades but only when the face looked to the right.

For rightward saccades, there was also a main effect of SOA, as seen in Fig. 4a (right), F(2.31,71.52) = 8.71, MSE = 9.05, p < .001, η = .22, with an increased number of saccades at longer SOAs. Unlike for the leftward saccades, there was no main effect of emotion (p = .540; Fig. 4b, right), but there was an interaction between face gaze direction and emotion, as seen in Fig. 4c (right), F(3,93) = 3.11, MSE = 2.67, p = .030, η = .09. Separate follow-up ANOVAs indicated no effect of emotion when faces looked to the left (p = .499), but an emotion effect when faces looked to the right, F(3,93) = 3.32, MSE = 2.22, p = .023, η = .097, driven by significantly more rightward saccades for classic neutral trials than fearful trials (p = .016). The main effect of face gaze direction (p = .60) and the interaction between face gaze direction and SOA were not significant (p = .50).

Analysis of target frame

On average, only 2.50% of trials (SE = 0.47%) contained saccades during the target frame. There was a main effect of SOA (Fig. 5a), F(3,93) = 9.68, MSE < .001, p < .001, η = .24, driven by a decreased proportion of trials with saccades at longer SOAs. There was also a main effect of emotion, F(3,93) = 3.11, MSE < .001, p = .030, η = .09, which interacted with SOA (Fig. 5b), F(3.86,119.64) = 2.56, MSE < .001, p = .044, η = .08. Separate ANOVAs run on each SOA indicated that there was a larger proportion of trials with saccades made during classic neutral trials compared to neutral tongue trials at the 300 ms SOA (p = .021), but no emotion effect at the other SOAs (200 ms: p = .55; 250 ms: p = .51 and 350 ms: p = .07).

Fig. 5

Proportion of trials with one or more saccades made during the target presentation, displayed a) for each SOA (averaged across emotional expression) and b) for each emotion and SOA condition.

There was no difference between the total number of leftward (average across the group: M = 49.41, SE = 12.17) and rightward (M = 42.84, SE = 9.86) saccades made during the target frame, t(31) = 1.24, p = .224.

The influence of gaze and target congruency on saccade direction

For leftward saccades, there was no effect of face gaze direction (p = .583), emotional expression (p = .339) or interaction between face gaze direction and target side (i.e. the congruency of the trial, p = .582). However, as seen in Fig. 6a (left), there was a main effect of target side, F(1,30) = 9.12, MSE = 22.42, p = .005, η = .23, with more leftward saccades for left target trials (M = 1.12, SE = .29) compared to right target trials (M = .47, SE = .12), indicating that participants were making more saccades towards the target direction, regardless of where the face had cued them and regardless of the face expression.

Fig. 6

Total number of leftward (left panels) and rightward (right panels) saccades (averaged across the group) made during the target presentation, displayed a) for each emotion condition and each face gaze direction, b) for each emotion condition (averaged across SOAs), c) for each SOA (averaged across emotion).

Similarly, for rightward saccades, there was no effect of face direction (p = .534) or interaction between face gaze direction and target side (p = .954), but there was a main effect of target side, F(1,31) = 10.77, MSE = 9.55, p = .003, η = .26, with more rightward saccades for right target trials (M = .89, SE = .21) than for left target trials (M = .45, SE = .12l Fig. 6a, right). There was also a main effect of emotion, F(3,93) = 3.56, MSE = 1.90, p = .017, η = .10, driven by significantly more saccades for classic neutral trials compared to neutral tongue trials (p = .009; Fig. 6b, right). Finally, there was a main effect of SOA (Fig. 6c, right), F(2.20,68.26) = 4.35, MSE = 2.18, p = .0.14, η = .12, due to less eye-movements at longer SOAs.

Discussion

The purpose of this study was to investigate saccadic eye-movements made in response to neutral and emotional faces during a dynamic version of the gaze-cuing paradigm, and their potential impact on the gaze-cuing effect. In the classic gaze-cuing paradigm, participants are instructed to maintain central fixation for the whole duration of each trial. Using the data from a subset of the participants tested in McCrackin & Itier's Experiment 4 (2018a, n = 44 versus n = 32 in the present study), we found that participants moved their eyes during the face sequence on less than 11% of trials, and during the target presentation on less than 3% of trials. These numbers are in line with the 8% of trials contaminated by eye-movements reported by Friesen and Kingstone (2003), though their number was not broken down into different sequence parts.

One of our main goals was to re-evaluate the impact of overt attention on the classic gaze-cuing effect, characterized by faster response times to gazed-at targets than non-gazed at targets. Unsurprisingly, we found that as SOA increased, there was a larger proportion of trials with saccades, and a larger number of rightward and leftward saccades during the face frame. However, we found a robust gaze-cuing effect after removing the trials contaminated by eye-blinks, providing strong support for the idea that the gaze-cuing effect reflects a covert orienting of attention to the gazed-at location. This replicates the original study by Friesen and Kingstone (2003) but using a sample three times as large and using real face photographs rather than schematic face stimuli.

Another main goal was to investigate how overt attention may contribute to the emotional enhancement of gaze-cuing. Previous research has shown that both fearful and happy faces elicit larger gaze-cuing effects than neutral faces (Bayless et al., 2011; Graham et al., 2010; Lassalle and Itier, 2013, 2015a,b; Neath et al., 2013; Putman et al., 2006; Tipples, 2006; McCrackin and Itier, 2018a,b), and one possibility was that overt attentional effects may contribute to the difference in gaze-cuing magnitude between emotional and neutral expressions. As in the original McCrackin & Itier study (2018a, Exp. 4), we found that emotional expression modulated the size of the gaze-cuing effect in a graded fashion. Fearful faces produced the largest gaze-cuing effect, followed by classic neutral faces, happy faces and neutral tongue faces respectively. While the overall pattern looked very similar when trials with eye-movements were included compared to when they were removed, emotion effects were strongest when the eye-movement trials were taken out, as supported by larger effect sizes and larger emotion differences. Importantly, subtle but important differences were found between happy face trials and the two neutral face trials.

Happy faces elicited a larger gaze-cuing effect than neutral tongue trials and a smaller gaze-cuing effect than classic neutral trials when eye-movements were removed, but not when they were included. Similarly, happy congruent trials elicited faster reaction times than classic neutral trials when eye-movements were removed, which was also found in the original paper with 12 more participants (McCrackin and Itier, 2018a), but was not found here when trials with eye-movements were included. These findings suggest that cleaning the data by removing eye-movement contaminated trials gave us more power to detect the subtle emotion differences present in the gaze-cuing effect. Emotion effects with happy expressions are thought to be harder to detect because they have a smaller effect size (McCrackin and Itier, 2018a,b). McCrackin and Itier (2018b) proposed that increasing the number of trials per condition and the number of participants may help increase chances of detecting a difference between happy and neutral expressions gaze-cuing, and the present study suggests that removing eye-movement contaminated trials may have a similar beneficial effect (even with a medium sample size of 32).

Another aim of the present study was to characterize which factors affect the direction of saccades in the dynamic gaze-cuing paradigm. In accordance with previous findings suggesting that gaze-cues aid in gaze-congruent saccades during free viewing (Kuhn et al., 2009; Hermens and Walker, 2016; Dukewich et al., 2008) and saccadic reaction time tasks (Mansfield et al., 2003; Kuhn and Kingstone, 2009; Kuhn et al., 2010; Hermens and Walker, 2012), we hypothesized that participants would make saccades in the direction of the gaze cue. We broke our trials down into two distinct time periods– the face sequence and the target presentation frame. Contrary to our predictions, we found that during both time periods, saccades were equally made towards the gazed-at and non-gazed-at locations, while during target presentation, saccades were significantly more likely to be made in the direction of the target. Thus, while the free-viewing literature suggests that people spontaneously follow gaze-cues with their eyes, the present study indicates that participants are actually quite good at supressing this tendency when asked to maintain face fixation. Our results are in line with the recent paper by Zeligman and Zivotofsky (2018), which reported that overt orienting in the direction of gaze-cues can be inhibited when it is counterproductive to the task, and are in contrast to some earlier work suggesting that people were not good at this inhibition (Kuhn and Kingstone, 2009). However, perhaps increasing the ecological validity of this task through the use of live actor or the incorporation of virtual reality headsets (which can track eye-movements quite effectively; Zhu et al., 2018) may change how participants respond to gaze cues.

Our analyses of the proportion of trials with one or more saccades, and our analyses of the number of rightward and leftward saccades revealed a common finding: participants made significantly more saccades during classic neutral trials than during the other trial types during both the face and target frames. This begs the question: what feature of the classic neutral trials elicit eye-movements? While it is possible that a key difference in affective content is driving the difference, classic neutral and neutral tongue trials are perceived to be similarly neutral (see McCrackin and Itier, 2018a for the neutral tongue face ratings and more discussion on this point). Thus, it seems more likely that the key distinguishing feature of the classic neutral condition is that it is the only condition in which the face appears to avert its gaze and remains still; it lacks the apparent motion from the neutral tongue, fearful and happy trials, which avert their gaze and then react with an expression or tongue protrusion (see Fig. 1). Motion is known to draw attention (Hillstrom and Yantis, 1994), and the present findings suggest that having motion on the face helps participants keep their eyes centrally fixated, presumably by capturing their attention to a larger extent. As the classic neutral trials are typically used as a baseline to compare emotion trials to, the present findings suggest that eye-movement differences should be taken into account before making a classic neutral-emotion comparison. As described above, removing eye-movements helped to reveal more subtle emotional modulation of gaze-cuing in the present study, potentially because eye-movements disproportionally affected the classic neutral condition. An alternate solution is to include a neutral movement control like the neutral tongue condition we used here, which produced no more eye-movements than the fear or happy conditions.

To summarize, our findings indicate that eye-movements in the gaze-cuing paradigm are generally rare, and that the gaze-cuing effect does reflect an orienting of covert attention. Contrary to what is typically found during free viewing and saccadic reaction time tasks, in dynamic gaze-cuing tasks such as the one employed here, there does not appear to be anything systematic about the direction of saccades made during the face frame, though during the target presentation, participants are prone to looking towards the target. Finally, during both the face and target frame, participants are more likely to move their eyes during classic neutral trials which lack movement, which affects our ability to detect the emotional modulation of gaze-cuing. Removing these eye-movement contaminated trials, or including a neutral movement control like the neutral tongue condition, can reveal more subtle emotional modulation of gaze-cuing.

Declarations

Author contribution statement

Sarah D. McCrackin: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Sarika K. Soomal: Analyzed and interpreted the data; Wrote the paper.

Payal Patel: Performed the experiments; Analyzed and interpreted the data.

Roxane J. Itier: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC Discovery Grant #418431), the Ontario government (Early Researcher Award, ER11-08-172), the Canada Foundation for Innovation (Grant #213322) and the Canada Research Chair (Grant #213322 and #230407) to RJI. SDM was supported by a Queen Elizabeth II Graduate Scholarship for Science and Technology (QEII-GSST). SKS was supported by a Natural Sciences and Engineering Research Council of Canada Undergraduate Student Research Award (NSERC USRA).

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.