Eye contact with neutral and smiling faces: effects on autonomic responses and frontal EEG asymmetry.

In our previous studies we have shown that seeing another person "live" with a direct vs. averted gaze results in enhanced skin conductance responses (SCRs) indicating autonomic arousal and in greater relative left-sided frontal activity in the electroencephalography (asymmetry in the alpha-band power), associated with approach motivation. In our studies, however, the stimulus persons had a neutral expression. In real-life social interaction, eye contact is often associated with a smile, which is another signal of the sender's approach-related motivation. A smile could, therefore, enhance the affective-motivational responses to eye contact. In the present study, we investigated whether the facial expression (neutral vs. social smile) would modulate autonomic arousal and frontal EEG alpha-band asymmetry to seeing a direct vs. an averted gaze in faces presented "live" through a liquid crystal (LC) shutter. The results showed that the SCRs were greater for the direct than the averted gaze and that the effect of gaze direction was more pronounced for a smiling than a neutral face. However, in this study, gaze direction and facial expression did not affect the frontal EEG asymmetry, although, for gaze direction, we found a marginally significant correlation between the degree of an overall bias for asymmetric frontal activity and the degree to which direct gaze elicited stronger left-sided frontal activity than did averted gaze.

Introduction

Gaze targeted toward an object can be seen as an expression of interest, either positive or negative. When someone turns his or her gaze to me, I may decide to approach or avoid this person. In making the final decision, it is useful to look at the gazer's facial expression. If the face expresses, for example, contempt while seeking eye contact, I may feel anxious and walk away, but if the face is happy, I am possibly inclined to approach the person looking at me. Facial expressions together with gaze direction thus convey information about the sender's emotions and personal goals. The perceptions of eye gaze and facial expression are partly processed by overlapping neural systems (Engell and Haxby, 2007), being independent at the early stages of neurocognitive processing but becoming integrated at the later stages (Rigato et al., 2010). However, the evidence concerning how gaze direction and facial expression interact during face processing is somewhat mixed.

When engaged in an eye contact with another person, we may end up thinking not only the impressions we get from the other (“What does he/she look like?”), but also the impressions given to the other (“What do I look like?”), implying that mutual gaze perhaps sensitizes us to the feelings of the encountered person and makes us more aware of ourselves. There is evidence that viewing another person with a direct gaze elicits greater autonomic arousal than viewing a person with an averted gaze (Nichols and Champness, 1971; Gale et al., 1975; Williams and Kleinke, 1993), even when the face is presented as an irrelevant stimulus during a demanding cognitive task (Conty et al., 2010). We have found enhanced skin conductance responses (SCRs), indicative of autonomic arousal, in response to eye contact with a “live” person and we have suggested that this effect may relate to increased self-awareness in the proximity of another person (Hietanen et al., 2008; Helminen et al., 2011; Pönkänen et al., 2011). But would a smile on a face of a person looking at the observer exert an additional effect on his or her autonomic responses? Looking at a smiling face has been shown to elicit feelings of warmth in an observer (Lau, 1982), and owing to this possible reward value of a smile, a smiling face might be a more salient stimulus than a neutral expression. It has been proposed that increased electrodermal activity reflects subjective salience of affective stimuli (Critchley, 2002). SCRs are shown to be intensified both when experiencing happiness (see Kreibig, 2010 for a review) and when seeing a happy face (Dimberg and Thunberg, 2007). Also, Martin and Gardner (1979) found greater arousal to a smiling than a neutral face in live interaction. They also varied the confederates' gaze direction but concluded that only facial expression had a significant effect on arousal.

It has been proposed that both facial expression and gaze direction can signal the sender's motivational tendencies of approach and avoidance, and if the motivational tendency signaled by gaze direction and facial expression match (e.g., both signal approach), face perception is enhanced (shared signal hypothesis, Adams and Kleck, 2003, 2005). For example, Adams and Kleck (2003) showed that facial expressions signaling approach (expressions of joy and anger) are perceived faster with direct than averted gaze, the pattern being reversed for expressions signaling avoidance (such as fearful and sad faces). However, Bindemann et al. (2008) have shown that these effects are task and stimulus bound. They provided, instead, evidence suggesting general impairments in facial expression recognition for faces with an averted gaze (Bindemann et al., 2008). Gaze direction has also an effect on perceived valence and intensity of facial expressions, and neutral expressions are interpreted as expressing approach-related emotions (such as joy) when combined with a direct gaze and avoidance-related emotions (such as fear) when combined with an averted gaze (Adams and Kleck, 2005). Sander et al. (2007) have interpreted these results within a framework of the appraisal theory. According to this theory, all facial cues are used to evaluate the meaning of these cues to one's own needs, intentions, and well-being. Because gaze is a critical cue for inferring the target of visual attention, it has a powerful influence on the appraisal of facial expressions and the self-relevance of the underlying emotion and motivation. According to the appraisal theory, the detection advantage of happy faces with a direct vs. an averted gaze is observed because a happy expression combined with a direct gaze implies a possible reward for the observer. In the context of happy faces, slightly averted gaze and face angles are prone to be misjudged to be looking at the observer (Lobmaier et al., 2008; Lobmaier and Perrett, 2011). The authors suggested that people prefer to think that they are the source of somebody else's happiness in order to improve their self-esteem.

In previous studies (Hietanen et al., 2008; Pönkänen et al., 2011), we provided psychophysiological evidence that seeing another person with a direct vs. an averted gaze differently activated the neural systems participating in the regulation of the approach-avoidance motivation. A large body of evidence supports the view that states of being prepared to approach or avoid targets are distinguishably represented in the brain and behavior. Studies of asymmetric EEG alpha-band activity measured over frontal scalp regions have played a central role in this research (Davidson, 1995, 2004; Allen et al., 2004). The alpha activity relates inversely to cortical activity (Davidson et al., 2000), and therefore, decrease in alpha power implicates increase in brain activity. A so-called motivational direction model has claimed that leftward frontal brain activity (based on the asymmetry in alpha-band power distribution; less alpha power on the left vs. right side) is involved in the experience and expression of approach-related emotions and motivation, whereas rightward activity is linked to avoidance-related emotions and motivation (Sutton and Davidson, 1997; van Honk and Schutter, 2006; Harmon-Jones et al., 2010). Compatibly with this theory, we reported enhanced relative left-sided frontal activity to seeing a direct rather than an averted gaze on a face of another person presented through a computer-controlled liquid crystal (LC) window (Hietanen et al., 2008; Pönkänen et al., 2011). Now, considering that in our previous studies the stimulus persons were having a neutral expression on their faces and, as noted above, that both the facial expression and gaze direction are used to infer the other person's motivational tendency, in this study, we wanted to investigate the effect of facial expression on the psychophysiological responses to direct and averted gaze. In the present study, we confined our investigation to the effects of a smile. There might be normative expectations for people who seek eye contact to smile. For instance, smiling people tend to be perceived as more sociable than people with neutral faces (Matsumoto and Kudoh, 1993). From this point of view, not smiling is an act in itself. One could ask whether the use of neutral faces in our previous studies exerted, in fact, a negative rather than neutral effect on the study participants. Eye contact with a person carrying a neutral expression may have resulted in the feeling that the person observed is indifferent, or even judgmental, toward the observer.

In the present study, we were also interested in investigating whether individual differences would modulate participants' responses to direct and averted gaze. There are individual differences in how people process facial information (see Calder et al., 2011), and, in general, social perception can differ depending on personal dispositions in approach-avoidance motivation (Elliot and Thrash, 2002; Strachman and Gable, 2006). Social anxiety, in particular, may alter ways of looking at faces and the motivation to approach or avoid them. Socially anxious individuals have been shown to avoid direct gaze, a result that was suggested to reflect the direct gaze being experienced as threatening (Horley et al., 2003; Wieser et al., 2009). Social anxiousness has been related to fear of social evaluation (Kocovski and Endler, 2000), also when the observed social signals appear positive (Weeks et al., 2008; Roelofs et al., 2010). Studies measuring the power in the alpha band EEG activity have shown that anxious arousal, characterized by somatic tension and physiological hyperarousal, is also associated with rightward brain activity (indicative of avoidance tendency) for both frontal and posterior regions (Engels et al., 2007). Carver and White (1994) introduced a self-report method (BIS/BAS scales) to measure the dispositional sensitivity of the neurobiologically based motivational systems regulating approach-avoidance behavior. This method is based on Gray's theory (see 1994, for review) about behavioral inhibition (BIS) and behavioral activation (BAS) systems. The BAS mediates approach behavior and is engaged by stimuli signaling reward. The BIS, in turn, is activated in a conflict situation, and serves to interrupt or inhibit ongoing goal-directed behavior (Carver and White, 1994). Recent brain imaging studies have found correlation between BIS/BAS activity and lateralized prefrontal cortex activity: right-sided activity was related to increases in BIS, whereas left-sided activity was associated to increases in BAS (Gray et al., 2002; Wacker et al., 2008, 2010; Balconi and Mazza, 2009; Berkman and Lieberman, 2010).

In this study, we measured SCRs and hemispheric asymmetry in the frontal EEG alpha-activity to seeing another person's live face. Both the gaze direction (direct and averted) and the facial expression (neutral and smiling) of the stimulus face were varied. Four main hypotheses were tested: (1) the SCR would be greater for a direct vs. an averted gaze, (2) perceiving a direct gaze would elicit relative left-sided frontal EEG asymmetry indicative of approach motivation, and (3) perceiving an averted gaze would elicit smaller relative left-sided asymmetry or even relative right-sided asymmetry indicative of avoidance, and (4) a smile in the stimulus face would enhance the differences in the SCR and frontal EEG asymmetry in response to a direct vs. an averted gaze. During physiological recordings, the participants evaluated the affective valence of the stimuli. This was done to ensure that the participants paid attention to the faces and also to investigate the effects of gaze direction and facial expressions on explicit face evaluations. We were also interested in examining whether individual differences in anxiety and BIS/BAS activity would modulate the pattern of physiological and behavioral responses to faces. Furthermore, previously we have suggested that a direct gaze elicits greater approach motivation than an averted gaze in response to real faces but not to face pictures because real faces are socially present and capable of interaction (Hietanen et al., 2008; Pönkänen et al., 2011). Now, we wanted to measure whether social presence, defined as a sense of awareness of the presence of an interaction partner (Sallnäs, 2005), would be related to the psychophysiological responses to real faces. In the present study, we decided to investigate only female dyads (participant and model). Recently, we showed that female participants exhibited differential motivational responses to male and female faces (Pönkänen et al., 2011), and therefore, in the current study, we did not want to confound the effects of gaze and expression by those of gender.

Materials and methods

Participants

The participants were 22 right-handed female undergraduates (mean age = 22.7 years, range = 19–39 years, SD = 5.1) who gained either course credits or two movie tickets for participation. All had normal or corrected-to-normal vision. Informed, written consent was obtained from each participant prior to the experiment in accordance with the declaration of Helsinki. Three participants were excluded from the final EEG and SCR analyses due to excessive artifacts, leaving 19 participants for the physiological data sample.

Stimuli and experimental procedure

The stimuli were the faces of two young females blind to the purpose of the experiment and unknown to the participants. They were instructed to present a neutral and a smiling face with a direct and an averted gaze (Figure 1). In the smiling face condition, the stimulus persons were trained to display non-enjoyment (or “social”) smiles by aiming at restricting the smile to the mouth area, thus lacking the activity of M. orbicularis oculi involved in enjoyment (or “Duchenne”) smiles (Ekman et al., 1990; Frank et al., 1993). It was considerably easier for the models to generate non-enjoyment than enjoyment smiles repeatedly, and our objective was to have as little variability as possible in the smiles throughout the experiment. The faces were presented through a 30 × 40 cm custom-built electronic shutter with a voltage sensitive LC window (NSG UMU Products Co., Ltd.) switching from the opaque to transparent state within less than 1 ms. The LC shutter was attached to a black frame between the stimulus person and the participant, the distance from the LC shutter being 30 cm for the stimulus person and 100 cm for the participant. The retinal size of the stimulus face was approximately 8.0° horizontally and 11° vertically. The averted gaze directions of the stimulus person were controlled by attaching fixation marks on the stimulus person's side of the LC shutter panel, one to the right side and another to the left side. The deviance from the direct gaze was 30°. The state of the LC shutter (transparent or opaque) was controlled by Neuroscan Stim software running on a desktop computer.

Figure 1

A stimulus model with direct and averted gaze having a smiling (above) and a neutral expression (below).

Each participant saw the faces of both stimulus persons. The faces were presented in four separate blocks: two for one stimulus person and two for the other. Within a block, there were a total of eight trials. After every block, there was a short break during which the stimulus person was changed. The presentation order of the stimulus persons was counterbalanced across the participants. There were 32 trials altogether: 16 smiling and 16 neutral faces. A half of the smiling and neutral faces were paired with a direct gaze and the other half with an averted gaze (half of them to the left, and the other half to the right). The presentation order of the stimulus types within a block was pseudo-random (no more than two consecutive trials of the same type). Each face was presented for 5 s, and the inter-stimulus-interval (ISI) was varied manually in each trial. The minimum ISI was fixed to 11 s (range of the mean intervals across participants: 17–24 s; grand mean: 19 s; maximum interval: 36 s). The experimenter was monitoring the skin conductance (SC) level on a computer screen throughout the experiment, and when the SC level seemed to have stabilized after the previous trial the experimenter initiated the next trial by pressing a start button. After pressing the button, there was a short audio signal intended to prepare the stimulus person to the opening of the shutter. The audio signal was presented via an earplug to the stimulus person. The earplug was not visible nor the signal audible to the participant. Four seconds after the audio signal, the LC shutter became transparent. During the ISIs and the breaks between the blocks, the shutter remained opaque. The mean duration of one block was 3.5 min.

After arriving to the laboratory, the participants were told that the experiment concerned face processing. They were instructed to assess the expression of the stimulus face on the positive—negative dimension immediately after each trial. The participants gave their response by sliding a lever with their dominant hand accordingly. Slight finger movements were enough to slide the lever that moved approximately 4 cm to the left or to the right from the central position, labeled as “neutral.” The left-end side of the potentiometer was labeled “extremely positive” and the right-end side was labeled “extremely negative.” The output voltage of the potentiometer was registered with the same equipment that was used to measure SC levels. The output voltages could vary between 2 (extremely positive) and −2 (extremely negative). The participants were told that the shutter would be opened several times for short periods of time, and that after a few openings there would be a short break. They were instructed to sit as motionless as possible, hold their gaze in the center of the LC shutter, and to view the face stimulus each time the shutter was open. During the experiment, one experimenter sat in the laboratory in such a way that she was invisible to the participant, but able to observe the monitor showing the SC levels. There was a digital video camera to record the stimulus person's facial expressions, eye-blinks, and other facial or bodily actions.

Immediately after the physiological recordings, the participants filled in the following questionnaires: a state-trait anxiety inventory (STAI) (Spielberger et al., 1983), a modified version of a social presence questionnaire (see Sallnäs, 2005), and a 20-item BIS/BAS questionnaire (Carver and White, 1994). The social presence questionnaire was used to measure the participants' feelings of social presence during the experiment. Our modified version of the questionnaire consisted of seven adjective pairs or statements that could be used to describe the face viewing condition (e.g., personal-impersonal) on a bipolar seven-point scale.

Acquisition of the physiological data

For the SC measurements, the electrodes (Ag/AgCl) were coated with electrode paste and attached to the palmar surface of the index and middle fingers on the participant's non-dominant hand. The signal was acquired with a SCR amplifier supplying constant-voltage AC excitation (22 mv) (ADInstruments). Power Lab 400 equipment was used to measure the SC. Data collection was controlled by LabChart Pro 7.1 programme running on a Dell Optiplex 760 computer. The sampling rate was 100/s. Continuous EEG was recorded with tin electrodes mounted in a stretch lycra electrode cap (Electro-Cap International, Eaton, OH) from F3, F4, F7, F8, C3, C4, P3, P4, and from both ears (A1, A2), all referenced online to Cz. Horizontal (HEOG) and vertical (VEOG) eye movements were monitored bipolarly from the sites beside the outer canthi of each eye (HEOG) and above and below the left eye (VEOG). Skin abrasion and electrode paste were used to reduce the electrode impedances below 5 kΩ. The EEG signal was amplified with SynAmps amplifiers with a gain of 5000 and a 1–200 Hz band-pass filter (50 Hz notch filter enabled). The continuous signal was digitized at 1000 Hz and stored for off-line analyses.

Data analysis

The face stimuli were videotaped and viewed off-line by two independent raters. They verified whether the models behaved according to the instructions in each trial and labeled each face having either a neutral expression or a polite smile. The raters agreed upon the facial expressions in 98.7% of the cases. The trials in which the facial expressions were not classified reliably were excluded from the final data analysis. Moreover, the records confirmed that the stimulus persons remained relatively motionless during the stimulus presentation. There were no observable differences between stimulus conditions in the stimulus persons' facial movements. The raters also detected the stimulus persons' blinks. On average, there were two blinks per one experimental session, and there were no differences in the number of blinks between the stimulus conditions. Trials containing stimulus person's blinks were excluded from the final data analyses.

The SCR was defined as a maximum change from the baseline level (at the stimulus onset) during a 4 s time period starting after 1 s from the stimulus onset till the end of the stimulus presentation. Responses contaminated by participant's body movements or technical problems during the measurement (16.9% of the trials) were eliminated from subsequent analysis. The statistical analyses were based on the mean values of SCRs computed for each participant across all trials within each stimulus category including those trials with maximum amplitude below 0.01 μS. This method of calculation results in the magnitude of the galvanic SCRs (cf., Dawson et al., 1990). A square root transformation [Sqrt (SCR)] was performed to normalize the data.

The continuous EEG signal was corrected off-line for the participants' blink artifact using a regression-based blink reduction algorithm (Semlitsch et al., 1986), and referred to both ears. Eye movements other than blinks and other visible artifacts were eliminated on the basis of visual inspection. Artifact-free EEG during the 5 s stimulus period was segmented to eight 1.024 ms epochs with 75% overlap between adjacent epochs. Spectral power was calculated for each epoch using Fast Fourier Transform (FFT) with a 10% Hanning taper. In epoching and overlapping the data segments, we followed the guidelines by Allen et al. (2004). According to these guidelines, short epochs are better to approximate the assumptions of the Fourier transform and they contain highly repeatable features throughout the waveform. Overlapping, in turn, diminishes the bias caused by the weighting functions in the windowing process which would result in the middle parts of each epoch receiving the most weight, and distal parts receiving negligible weight. The obtained power spectra were averaged over all artifact-free epochs within each trial and over separate trials within each experimental condition. Trials with less than 50% artifact-free epochs were excluded from averaging. Based on these criteria, 2.8% of the trials were eliminated. For average power spectra within each condition, power density values (μV2) within the alpha band (8–13 Hz) were calculated and natural ln-transformed to normalize the distributions. Asymmetry scores were calculated for electrode pairs at frontal (F8/F7, F4/F3), central (C4/C3), and parietal (P4/P3) scalp regions by subtracting the ln-transformed power density values for the left site from that for the right site (Allen et al., 2004). The main data analysis was confined to the data measured from the electrode pair F4/F3, since the affective and motivational effects on the frontal EEG asymmetry are frequently detected from these recording sites (Davidson, 1995). The other recording sites were also analyzed to detect the relative asymmetry in EEG activity. The effects were significant only for the frontal electrode pair F4/F3, and approached significance for electrode pair F8/F7.

Results

Skin conductance response

For the SCRs, an ANOVA with Gaze (direct, averted) and Expression (smiling, neutral) as within-subject factors resulted in the main effects of Gaze, F(1, 18) = 20.06, p < 0.001, η2 = 0.53, and Expression, F(1, 18) = 15.60, p = 0.001, η2 = 0.46, and a Gaze × Expression interaction, F(1, 18) = 5.26, p = 0.034, η2 = 0.23. Follow-up paired-samples t-tests indicated larger SCRs for the direct vs. averted gaze both in the neutral, t(18) = 3.04, p = 0.007, and in the smiling face, t(18) = 4.66, p < 0.001. Furthermore, SCRs were significantly greater for the direct gaze in the smiling vs. neutral face t(18) = 4.88, p < 0.001, but for the averted gaze, this difference was only marginally significant, t(18) = 2.08, p = 0.052. Figure 2 shows the mean SCRs as a function of gaze direction and facial expression.

Figure 2

Mean skin conductance responses (square root-transformed SCRs in μMho) as a function facial expression and gaze direction.

Overall, the SCRs did not correlate with the valence ratings. However, when analyzing the different Gaze × Expression combinations separately, we found a significant negative correlation between the SCRs and the valence ratings for the neutral expression with a direct gaze, r = −0.53, p = 0.02. None of the other behavioral measures did correlate significantly with the SCRs. The mean values and standard deviations of the STAI, BIS/BAS, and social presence scale scores are presented in Table 1.

Table 1

Means and standard deviations for STAI, BIS-BAS, and social presence questionnaire scores.

Measure	M	SD
STAI Total	67.19	11.26
STAI State	30.72	5.32
STAI Trait	36.47	8.52
BIS Total	24.25	5.80
BAS Total	47.05	6.48
BAS Reward	19.43	2.76
BAS Drive	14.00	2.54
BAS Fun	13.62	2.81
Social Presence	4.48	0.91

Frontal EEG asymmetry

A Two-Way ANOVA with Gaze (direct, averted) and Expression (smiling, neutral) as within-subject factors showed no significant main effects or interactions on the EEG asymmetry scores calculated for the frontal F4/F3 electrode pair, all Fs < 1. A similar ANOVA for the F8/F7 electrode pair showed no main effects or interactions either, Fs < 1.

However, we noticed that there was substantial variation among the participants in their overall asymmetry scores. Some participants showed negative asymmetry scores (indicative of avoidance) in response to all types of stimuli, and their asymmetry scores were more negative for the direct vs. averted gaze. Others, in contrast, had positive asymmetry scores (indicative of approach) to all stimulus types, and their asymmetry scores were more positive for the direct vs. averted gaze. We calculated two new asymmetry indices: (1) participant's asymmetry score averaged across all experimental conditions (asyscoreoverall) and (2) the difference in asymmetry scores for direct and averted gaze (asyscoredirect – asyscoreaverted). For the latter index, increasing negative values would indicate increasingly stronger avoidance-related brain activity for direct vs. averted gaze, whereas increasing positive values would indicate increasingly stronger approach-related brain activity for direct vs. averted gaze. When these indices were correlated, we found a marginally significant correlation between the overall asymmetry index and the gaze direction difference index, r = 0.45, p = 0.05 (see Figure 3). The results of the behavioral measures did not correlate with the asymmetry scores.

Figure 3

A scatter plot with a linear fit curve for the participant's overall asymmetry score in the EEG alpha power (in ln-transformed μV For X-axis scores, negative values indicate right-sided asymmetry (associated with avoidance) and positive values indicate left-sided asymmetry (associated with approach). For Y-axis scores, negative values indicate stronger avoidance-related brain activity for direct vs. averted gaze, whereas positive values indicate stronger approach-related brain activity for direct vs. averted gaze.

Valence evaluation

A similar ANOVA as above was run for the valence ratings. This analysis revealed the main effects of Gaze, F(1, 21) = 36.97, p < 0.001, η2 = 0.64, and Expression, F(1, 21) = 113.62, p < 0.001, η2 = 0.84, but not a significant interaction effect. The smiling faces (M = 0.76, SD = 0.32) were rated as being more positive than the neutral faces (M = −0.41, SD = 0.21), and the faces with a direct gaze (M = 0.29, SD = 0.19) were rated as being more positive than the faces with an averted gaze (M = 0.06, SD = 0.19).

Discussion

In this study, we measured autonomic arousal and frontal EEG asymmetry to faces presented “live” through an electronic shutter. Our main goal was to examine whether affective and motivational neural responses to seeing a direct vs. an averted gaze would be modulated by the expression (neutral/smiling) on the gazing face. We expected to observe greater arousal and greater relative left-sided frontal asymmetry to seeing a direct vs. an averted gaze, and we proposed that when a stimulus person was smiling instead of having a neutral expression, these effects would be enhanced.

We replicated our earlier findings (Hietanen et al., 2008; Helminen et al., 2011; Pönkänen et al., 2011) by showing greater SCRs to live faces with a direct vs. averted gaze. Here we also manipulated the facial expression of the stimuli and found that viewing of smiling faces elicited overall greater arousal than viewing of neutral faces, and, interestingly, that the effect of gaze direction was more pronounced in response to a smiling than a neutral face. The autonomic arousal was also greater to a smiling vs. a neutral face with an eye contact. This result reflects the emotional saliency of both the direct gaze and the smile, and shows that a combination of these signals results in a strong autonomic response. Indeed, both direct gaze and smiling face can be regarded as signals inviting to closer interaction. Increased intimacy, in turn, has been shown to elevate arousal (Patterson, 1976; Patterson et al., 1981).

Brain-imaging studies have shown that the amygdala is involved in the integration of emotional facial expressions and gaze direction (Sato et al., 2004, 2010). Sato et al. (2004) found greater amygdala activation to faces looking toward vs. away from the subject when the face was emotionally expressive rather than neutral. The amygdala response also increased together with self-reported arousal. As the amygdala is known to play a central role in regulating affective arousal (Mangina and Beuzeron-Mangina, 1996; LeDoux, 2000, Williams et al., 2005; Laine et al., 2009), these results are highly compatible with the present ones. However, it must be noted that the results from the brain imaging studies regarding the effects of gaze direction and facial expression on amygdala activation are not consistent (Adams et al., 2003; Hadjikhani et al., 2008; Straube et al., 2009). It is possible that these discrepant findings reflect differences in the functions of face processing. For instance, Adams et al. (in press) studied amygdala activity in response to direct and averted gaze on faces expressing fear as a function of stimulus presentation time. Subregions of the amygdala were distinctively tuned to short and long stimulus durations. The authors proposed that the shorter presentation time (300 ms) triggers reflexive attention to the faces, whereas the longer time (1000 ms) allows reflective processing of stimulus significance. Possibly relating to the reflective processing mode, previous research has suggested that stimulus ambiguity and unpredictability can modify amygdala activation (Hsu et al., 2005; Herry et al., 2007; Whalen, 2007). Recent research has suggested greater amygdala activity in response to highly self-relevant stimuli (Sato et al., 2004; Hadjikhani et al., 2008; N'Diaye et al., 2009; Boll et al., 2011). The differences in the relative relevance between direct vs. averted gaze may depend, perhaps, on the social context, personals dispositions, and task demands. In the present study, the relatively long stimulus duration (5 s) allowed a reflective processing mode. Perhaps it was more self-relevant and, hence, more arousing to reflect upon the meaning of an emotional vs. a neutral face. Looking at a smiling rather than a neutral face may have triggered the viewers to mentalize more effectively the observed person's emotional and cognitive state.

Interestingly, there is one previous study that investigated the SCRs to smiling and neutral faces with a direct vs. averted gaze. Martin and Gardner (1979) also used “live stimuli” by having the participant and a confederate sitting face-to face while the confederate presented combinations of direct/averted gaze directions and smiling/neutral expressions during 20 s trial periods. Similar to the present results, that study also reported higher autonomic arousal to a smiling than a neutral face. However, they did not find any effect of gaze direction. We can only speculate on the possible reasons for this discrepancy. In Martin and Gardner (1979) study, only male dyads were used, whereas there were only female dyads in our study. Compared to men, women tend to be more sensitive to facial communicative gestures (Gueguen and Jacob, 2002), and, especially, to feel more observed when interacting face-to-face with another person (Argyle and Williams, 1969). Furthermore, we employed a valence-rating task during viewing of the faces, but, in Martin and Gardner (1979) study, the other person was passively observed. It is possible that, in the present study, the evaluation of the faces enhanced the effect of gaze direction on autonomic arousal.

The current results showed that the smiling faces were rated as being positive, whereas the neutral faces were evaluated as being slightly negative. Moreover, faces with a direct gaze were rated as being more positive than faces with an averted gaze, regardless of whether the face was smiling or not. Interestingly, in our earlier studies in which the participants viewed neutral faces only (Hietanen et al., 2008; Pönkänen et al., 2011), looking at faces with averted gaze were rated to evoke higher level of pleasantness than faces with direct gaze. These overtly discrepant findings are likely to be explained by the fact that, in the current study, the participants evaluated how positive/negative the facial expression appeared, whereas in the previous studies they evaluated their own feelings in response to the faces. Now, we found that the participants made a clear distinction in the evaluations of smiling and neutral faces, and it was also relevant whether the stimulus person was looking at the participant or not. This is in line with the autonomic arousal results. Our results also showed a negative correlation between the SCRs and the valence ratings, but only for the neutral faces having a direct gaze. Those who were more inclined to evaluate a neutral, direct gazing face as being negative showed more pronounced autonomic arousal toward those faces. There was considerable variation in the valence ratings for the neutral faces; some participants rated them being slightly positive, whereas to others they appeared negative. Such a variation was not observed for the smiling faces which were consistently evaluated as positive. It can be speculated that a smiling face represents a salient communicative signal triggering autonomic arousal and positive evaluation in most individuals. A neutral face, instead, is a less salient signal, leaving more space for personal dispositions to influence the affective evaluations. A neutral face looking directly back to the viewer may be an especially effective signal in revealing individual differences in affective evaluations and in their association with autonomic responses. It is also important to note that the valence of the expression does not necessarily correspond to the intention of the person bearing the expression (e.g., Ambadar et al., 2009). Thus, a social smile can imply a sender's social motive to hide negative emotions, and, indeed, non-Duchenne smiles are less often used as a signal of social intent than Duchenne smiles (Hess and Bourgeois, 2010). Bindemann et al. (2008) have suggested that when information from gaze direction and facial expression is combined, the ambiguity of the expression might determine how important gaze direction is for determining the observed person's goals and emotions. In our study, the overall intensity of the smiles was low, and this expression was held still on a face for 5 s. If these smiles were regarded as contrived or otherwise ambiguous, it is possible that the elevated arousal to the smiling faces reflected these factors rather than the positivity or approachability of the expression itself. Apparently, this question warrants further studying to disambiguate the source of arousal in response to direct vs. averted gaze in expressive faces.

The results of the frontal alpha-band EEG asymmetry measures did not show effects of gaze direction or facial expression. This was contrary to our expectations. Particularly, the non-significant effect of gaze direction on frontal asymmetry was unpredicted since in our earlier studies with rather similar study designs we have observed greater relative left-sided activity for a direct vs. an averted gaze (Hietanen et al., 2008; Pönkänen et al., 2011). In these studies, the stimulus persons' identities have varied from experiment to experiment, and we cannot rule out the possibility that some factors related to their personal qualities could have had an effect on the results. However, one difference between the experimental settings of the present and the previous studies was that, in the present study, the stimulus persons were students who did not interact with the participants prior to the experiment at all, whereas in the previous studies the stimulus persons were also the experimenters of these studies (welcoming the participants, shaking hands with them, assisting in the preparation of the physiological recordings, etc.). It is an interesting possibility that some sort of social connectedness, or dominance factor due to the experimenter status, would have such a dramatic effect on the results. This is clearly an important question and needs further studying.

Although we did not find a gaze direction effect on the frontal alpha-band EEG asymmetry, the magnitude of the mean asymmetry score calculated across responses to all stimulus conditions exhibited a marginally significant positive correlation with the differential score expressing whether direct gaze evoked less (negative values) or more (positive values) left-sided brain activity (implying approach motivation) compared to averted gaze. In other words, the increase of the overall asymmetry scores was associated with a tendency that direct gaze elicited stronger approach-related activity than did the averted gaze. One interesting possibility is that this result could reflect differences in personal dispositions affecting the approach-avoidance—motivation and reactions to gaze directions. On the other hand, our behavioral data failed to give support to this possibility. Unfortunately, we did not measure baseline EEG alpha-band activity, which could have revealed some trait-based effects in the responses to gaze direction. It has been shown that there are trait-level individual differences in affective styles. Left-sided trait asymmetry has been associated with positive affect and approach behavior, whereas right-sided trait asymmetry has been related to negative affect, anxiety, and behavioral avoidance (Heller and Nitschke, 1998; Davidson and Irwin, 1999; Allen and Kline, 2004; Mathersul et al., 2008).

It also stands out that the facial expression had an effect on autonomic responses, but not on the neural activity related to approach-avoidance tendencies. According to the shared signal hypothesis (Adams and Kleck, 2003, 2005), a smile and direct gaze should both potentiate approach motivation, and Stins et al. (2011) recently found that it takes less time to initiate steps toward a smiling vs. an angry face, suggesting that a smile prompts approach behavior. Furthermore, by recording alpha-band EEG activity Davidson and Fox (1982) have shown that infants show leftward frontal activity in response to happy faces and rightward activity in response to angry faces. Our present results did not show an effect of gaze or facial expression on the frontal EEG asymmetry. Given that we had only faces presenting neutral and low-intensity smile expressions in this study, it remains an open question whether a face that communicates strong appeal or danger might play a more dominant role in influencing the frontal EEG asymmetry.

In the current study, we also investigated whether anxiety, the activation of the behavioral motivation systems BIS/BAS, and the felt social presence would correlate with the autonomic arousal and frontal EEG asymmetry scores in response to the face stimuli. We found that none of these measures were related to the psychophysiological responses. In future studies, it would be useful to measure individual differences in resting (baseline) frontal alpha-band EEG asymmetries, anxiety, and self-esteem when studying affective and motivational responses to eye gaze. These may be critical factors, especially when responses to faces with direct gaze (signaling social approval) and averted gaze (signaling rejection) are being studied. Since we tested only female dyads, the question of whether the observed results would be similar had we recruited also male or mixed dyads, remains. The ratings of social presence in response to the stimulus faces showed that the participants felt moderate levels of social presence during the experiment. However, we measured social presence only once, at the end of the experiment, and not separately for the different stimulus conditions. In the future, it would be worth studying how different types of evaluation tasks or active interaction between the participant and the stimulus person influence social presence and psychophysiological responses.

To sum up, the results of the current study showed that facial expression modified autonomic arousal elicited by sustained eye contact with another person. Being in an eye contact with another individual had an arousal-enhancing effect that was greater to seeing a smiling rather than a neutral face. We suggest that an eye contact has a function of “tuning” two persons to each other, and a positive facial expression, perhaps, boosts to greater sharing of affective information.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships with a potential conflict of interest.