Rapid processing of neutral and angry expressions within ongoing facial stimulus streams: Is it all about isolated facial features?

Our visual system extracts the emotional meaning of human facial expressions rapidly and automatically. Novel paradigms using fast periodic stimulations have provided insights into the electrophysiological processes underlying emotional content extraction: the regular occurrence of specific identities and/or emotional expressions alone can drive diagnostic brain responses. Consistent with a processing advantage for social cues of threat, we expected angry facial expressions to drive larger responses than neutral expressions. In a series of four EEG experiments, we studied the potential boundary conditions of such an effect: (i) we piloted emotional cue extraction using 9 facial identities and a fast presentation rate of 15 Hz (N = 16); (ii) we reduced the facial identities from 9 to 2, to assess whether (low or high) variability across emotional expressions would modulate brain responses (N = 16); (iii) we slowed the presentation rate from 15 Hz to 6 Hz (N = 31), the optimal presentation rate for facial feature extraction; (iv) we tested whether passive viewing instead of a concurrent task at fixation would play a role (N = 30). We consistently observed neural responses reflecting the rate of regularly presented emotional expressions (5 Hz and 2 Hz at presentation rates of 15 Hz and 6 Hz, respectively). Intriguingly, neutral expressions consistently produced stronger responses than angry expressions, contrary to the predicted processing advantage for threat-related stimuli. Our findings highlight the influence of physical differences across facial identities and emotional expressions.

Introduction

The human brain is capable of rapidly processing differences in facial expressions and identifying those that signal threat, presumably due to the survival advantage of such an ability [1,2]. From an evolutionary point of view, speed of processing, identification, and discrimination are essential for the survival of an individual within a group, because not all members will exhibit identical (facial) expressions: in a threatening situation, some might express anger—i.e., aggression and the will to fight—, whereas others might express anxiety and a tendency for flight. This communication of contradictory information must be rapidly processed to allow the individual to adapt behavior accordingly.

An elegant way to investigate the time required for emotion detection and discrimination is to show stimuli in a rapid serial visual presentation stream (RSVP) at a certain frequency. In RSVP paradigms, each cycle is defined by the onset of a new image; thus, each image serves as a forward mask for the subsequent image and as a backward mask for the preceding one [3]. Such frequency-tagged stimulation streams evoke the steady-state visual evoked potential (SSVEP). SSVEPs have the same fundamental frequency as the RSVP stream and may include higher harmonics [4,5]. Attending to RSVP streams results in a modulation of the SSVEP, either quantified as an increase in amplitude [6-8] or enhanced inter-trial phase consistency [9,10].

A number of studies demonstrated that the presentation of emotionally arousing stimuli resulted in enhanced SSVEPs compared with neutral stimuli [e.g., 11–13]. Manipulating the presentation frequency of the RSVP allows to test the minimal presentation times for emotional cue extraction by analyzing SSVEPs in the frequency domain. In a recent study by Bekhtereva and Müller [14], we showed that presenting complex images from the International Affective Picture System [IAPS; 15] at a rate of 15 Hz (i.e., about 67 msec per image) was too rapid to enable emotional cue extraction, resulting in no discernible changes in SSVEPs between emotional and neutral images. Conversely, a 6 Hz presentation (i.e., about 167 msec per image) resulted in a significant modulation of SSVEPs as a function of emotional content. This presentation time was close to what was previously reported by Alonso-Prieto et al. [16] using RSVP paradigms that discriminated between faces with different identities. In that study, SSVEPs increased when different faces were presented in an RSVP compared with presenting the same face for the entire stream, a finding consistent with repetition suppression effects [17,18]. Alonso-Prieto et al. [16] reasoned that a longer interstimulus interval allowed for the full development of the N170, an event-related potential (ERP) component classically linked to the identification and discrimination of faces [19-21]. This, in turn, would enable sufficient time for individual face identification and emotional cue extraction. Subsequently, Liu-Shuang et al. [22] published an extension of the aforementioned stimulation approach by introducing regularities after a fixed number of cycles (i.e., images). In other words, a certain exemplar (“oddball”) was presented after four filler faces, which resulted in an SSVEP at the base frequency (6 Hz) and a second peak in the frequency spectrum at a slower rate of the oddball presentation (1.2 Hz). Interestingly, the frequency spectrum lacked such a 1.2 Hz peak when faces were presented upside down, an experimental condition typically used to control for low-level visual features because inverted faces share the same first-order features (e.g., eyes, nose) but disrupt second-order configuration, i.e., the relation between features [e.g., 23].

Given that previous work mainly focused on face identification using RSVP protocols with either same/different faces [22] or faces among natural images [24], in 2014 we started a series of studies to extend RSVP stimulation using complex IAPS images and faces by including regular oddball stimuli in the RSVP. The present report is a comprehensive summary of the results we obtained in a research programme consisting of two pilot studies (N = 16/16) and two experiments with larger samples (N = 31/30), for a total of 93 recorded participants. Similar to our previous work using complex pictures [14,25-27], the purpose of this series of studies was to test whether—and under which conditions—the visual system is able to identify a regularly presented neutral or emotional face (angry in the present experiments) within an RSVP stream of different facial identities and expressions as “fillers”. Given the alleged motivational relevance of angry faces, we initially expected processing advantages—and, consequently, increased SSVEPs—for regularly presented angry as compared to neutral faces. By using different facial identities and expressions as “fillers”, our experimental protocol extended the fast periodic oddball stimulation introduced by Liu-Shuang et al. [22] by using a different face and/or facial expression for each cycle (except the regularly presented items). In doing so, we significantly increased stimulus variability and set the visual system under a “perceptual stress test” [27] to probe boundary conditions of rapid facial emotional cue extraction. By presenting RSVP streams with inverted faces of exactly the same sequences, we additionally tested whether any observed SSVEP modulations may be due to low-level visual features and/or second-order relationships between facial elements. We initially explored much shorter presentation times—well above 6 Hz—to probe the temporal boundary conditions established for complex images in previous work [14]. Specifically, we tested whether the modulation of the oddball frequency via fixed exemplars requires the full processing of an individual face or, instead, regular presentations within a longer RSVP stream may allow for a significantly shorter presentation time because the gist of the emotional expression can be integrated through regular presentation. To that end, similar to our first IAPS study [14], we piloted with a stimulation frequency of 15 Hz (i.e., cycle length ~67 msec). To ensure that participants attended the RSVP, we included an orthogonal attention task instructing participants to detect and respond to a colored dot that was unpredictably overlayed on the face stream. We started our experimental series with the initial hypotheses that, if the emotional facial expression could be extracted reliably from the stream of faces, SSVEPs at the regularity frequency should be measurable for the regular but not the irregular conditions. SSVEPs for angry faces should be higher than for neutral images and should be also higher for upright than for inverted faces.

In the meantime, Dzhelyova and colleagues [28] published a study based on a similar rationale and methodology. These authors presented an RSVP at about 6 Hz with one neutral face of one individual and inserted, once every 5th stimulus, a face with an emotional expression of that individual, respectively. In a second experiment, they increased the base presentation rate of neutral faces to 12 Hz, with an emotional face presented every 9th stimulus. They analysed SSVEPs to the oddball frequency (about 1.2 Hz in Experiment 1 and about 1.3 Hz in Experiment 2) and additionally explored all higher harmonics, i.e, integer multiples of the oddball frequency. Identical to our protocol, they presented streams with inverted faces at the same stimulation frequencies, to control for the influence of low-level features. SSVEPs were found to be above noise level at the sum of all oddball frequencies (i.e., including all higher harmonics) elicited by emotional faces. Interestingly, this was also true for inverted faces, although the effect was smaller in magnitude compared to upright faces when looking at more occipito-temporal electrodes.

In their experimental paradigm, Dzhelyova et al. [28] always switched from a neutral to an emotional expression. This methodological choice might have overlooked systematic low-level, physical differences between stimuli, which could also explain the reported SSVEPs above noise level in the inverted condition. In our studies, we used different emotional facial expressions and the regular expression was either an angry or a neutral face. In contrast to Dzhelyova et al. [28] and other studies with similar paradigms, our use of a range of expressions and identities also changed the status of the “oddball” to yet another facial expression in the RSVP, with the only difference that it was presented regularly among irregular presentations of filler faces. Throughout this manuscript, we will therefore refer to it as the regular rather than the oddball expression and term its presentation rate regularity rate and the corresponding neural response, if present, regularity-driven SSVEP.

Given the increased diversity of emotional facial expressions compared with only neutral faces as filler stimuli, we were able to better randomize physical stimulus differences which may arguably influence the neural response to the regular presentations. In addition, we swapped emotional expressions and either embedded angry or neutral faces regularly within the RSVP. This allowed us to test whether angry faces elicited a larger response compared to neutral expressions, theoretically expected under the assumption that threatening information is of greater behavioural significance and thus leads to prioritised neural processing. Furthermore, we included another important control condition: we presented the respective RSVP stream with different facial expressions without any regular repetitions. This important manipulation controls for an inherent problem of the stimulation protocol: the SSVEP driven by the regular presentation (i.e., the “oddball” in previous studies) is always a subharmonic of the RSVP frequency. Thus, only testing whether a regularity-driven SSVEP is above noise-level cannot exclude that it is, in fact, a subharmonic of the SSVEP to the RSVP rate and does not specifically indicate a functional processing of such regularity. With our control condition, presenting faces in irregular order only as a contrast, we were able to test for that possible confound.

Pilot 1

Materials and methods

Participants

Sixteen participants (10F/6M, median age 22.5 years, range 19–31, normal or corrected-to-normal vision, no self-reported history of neurological or psychiatric disorders) were recruited from the student population and the general public. Participants gave informed written consent prior to the experiment and were financially reimbursed €12 afterwards. All studies reported here were approved by the ethics committee of the University of Leipzig.

Stimuli

Stimuli were selected from NimStim, a validated database of facial expressions free for academic use [29]. This pilot experiment included identities #21, #22, #23, #25, #26, #33, #34, #36, and #37 (all males); the selected expressions were neutral, angry, and happy (all with closed mouth). All stimuli were resized to 152 x 190 pixels using Irfanview (https://www.irfanview.com/) and converted to grayscale in MATLAB R2015a (The MathWorks, Inc., Natick, MA) via the standard NTSC conversion formula used for calculating the effective luminance of a pixel: intensity = (0.2989 * red) + (0.5870 * green) + (0.1140 * blue) (see https://tinyurl.com/rgb2gray). Their luminance was matched using the SHINE toolbox [30]. To remove external facial features (e.g., hair and ears) and to standardize the spatial layout occupied by each face, stimuli were enclosed in an oval frame at presentation.

Procedure

After signing the informed consent, participants were seated comfortably in an acoustically dampened and electromagnetically shielded chamber and directed their gaze towards a central fixation cross (0.8° x 0.8° of visual angle) displayed on a 19-inch CRT screen (refresh rate: 60 frames per second; 640 x 480 pixel resolution) placed at a distance of 80 cm. The experimental stimulation consisted of a rapid serial visual presentation (RSVP) of face images showing each stimulus (size = 3.5°) in the center of the screen. The RSVP was presented at a rate of 15 faces per second (15 Hz), resulting in a presentation cycle of four frames (cycle length ~67 msec). Each face was shown during the first 50% of the frames of one cycle, producing a 50/50 on/off luminance flicker. Within the RSVP in each trial, faces were randomly drawn and organized in triplets. Depending on the experimental condition, the first image of each triplet was either an angry or a neutral face. For position two and three, images were pseudo-randomly drawn from the remaining expression categories so that emotional expressions were evenly distributed. Faces within one triplet were not allowed to re-occur in the following trial, to avoid short-term repetitions of identical faces. Happy faces were never presented regularly and only served as filler items.

In addition to the physical RSVP frequency (stimulation frequency), this presentation protocol introduced a second rhythm defined by the regular occurrence of faces of one emotional category with each third face. The category angry or neutral thus repeated at 5 Hz, i.e., at one-third of the RSVP rate (regularity frequency). We further added a third irregular condition, for which image sequences were created by randomly drawing from all emotional categories (i.e., no regularity at 5 Hz). As control conditions, we mirrored the set-up of upright-face RSVPs (regular angry, regular neutral, and irregular) but presented all stimuli upside-down, i.e. inverted (see ).

Exemplar image sequences used in Pilot 1.

(A) Sequence of six exemplar images for each experimental condition. For the regular conditions, every third image contained a repetition of the same emotional category (angry or neutral). For the irregular condition, image triplets contained all three emotional categories (neutral, angry, happy) in random order. Images were presented either upright or upside down. Note that, for illustration purposes, we here show images from the Face Research Lab London Set [31], additionally modified in GIMP 2.10.14 (https://www.gimp.org/) to display anger. A set of different (only male) face identities was presented during the experiment, but cannot be shown here due to copyright restrictions. Please refer to https://www.macbrain.org/resources.htm for examples of faces used in the experiment. (B) Exemplar visual presentation of stimuli within each trial. Single images were presented for 33 msec followed by a fixation cross-only image for 33 msec, leading to a presentation rate of 15 Hz for the image stream with a regularity frequency of 5 Hz during the regular conditions. Note: image not to scale.

At the beginning of each trial, participants were presented with a fixation cross for 1.2 sec. Subsequently, the RSVP was presented for 3.8 sec. Participants were instructed to press the spacebar on a standard QWERTZ USB keyboard any time they detected a turquoise dot (RGB: 128, 128, 196; diameter = 0.3° of visual angle) briefly displayed within the face area (2 consecutive frames = 33 msec). Targets occurred in 40% of trials and up to three times in one trial with a minimum interval of 600 msec between onsets. At the end of each trial, the fixation cross remained on screen for an extra 1 sec, allowing participants to blink before the next trial started.

We presented a total of 576 trials (96 trials per condition), divided into 8 blocks (~6 min 20 sec each). Before the start of the experiment, participants performed a few blocks of training. After each training and experimental block, they received feedback on their performance (average hit rate, reaction times, and number of false alarms within the block).

To ensure that our pre-selected facial expressions were processed in accordance with normative categorization, at the end of the main task we asked participants to judge the level of conveyed anger and happiness using a 9-point Likert scale (1: very low anger/happiness; 9 very high anger/happiness) (see results in the Supplementary Materials).

EEG recording and preprocessing

Electroencephalographic activity (EEG) was recorded with an ActiveTwo amplifier (BioSemi, Inc., The Netherlands) at a sampling rate of 256 Hz. Sixty-four Ag/AgCl electrodes were fitted into an elastic cap following the international 10/20 system [32]. Electrodes T7 and T8 of the standard BioSemi layout were moved in position I1 and I2 to increase spatial resolution at occipital sites. The common mode sense (CMS) active electrode and the driven right leg (DRL) passive electrode were used as reference and ground electrodes, respectively. Horizontal and vertical electrooculogram (EOG) were monitored using four facial bipolar electrodes placed on the outer canthi of each eye and in the inferior and superior areas of the left orbit.

EEG preprocessing was performed offline with custom MATLAB scripts and functions included in EEGLAB v14.1.1b [33] and FASTER v1.2.3b [34] toolboxes. The continuous EEG signal was referenced to the average activity of all electrodes. After subtracting the mean value of the waveform (DC offset), we assigned electrode coordinates and segmented the signal into epochs time-locked to the beginning of the flickering stimulation (0–3.8 sec). We discarded all trials with behavioral responses (N = 216), leaving 360 epochs (60 per condition). After re-referencing to electrode Cz, FASTER was used for artifact identification and rejection (see commented script at https://osf.io/au73y/) using the following settings: (i) over the whole normalized EEG signal, channels with variance, mean correlation, and Hurst exponent exceeding z = ±3 were interpolated via a spherical spline procedure [35]; (ii) the mean across channels was computed for each epoch and, if amplitude range, variance, and channel deviation exceeded z = ±3, the whole epoch was removed; (iii) within each epoch, channels with variance, median gradient, amplitude range, and channel deviation exceeding z = ±3 were interpolated; (iv) condition averages with amplitude range, variance, channel deviation, and maximum EOG value exceeding z = ±3 were removed; (v) epochs containing more than 12 interpolated channels were discarded. We also discarded epochs whose signal exceeded 5 standard deviations from single- and all-channel mean kurtosis value, i.e., displaying abrupt spikes or flat activity. In addition, we verified that the spectral estimates would not deviate from baseline by +/-50 dB in the 0–2 Hz frequency window (indicating blinks) or +25/-100 dB in the 20–40 Hz frequency window (indicating muscular activity) [36]. The number of interpolated channels was low (M = 4.06, SD = 1.25). For an overview of the mean percentage of rejected epochs per condition, see . Finally, the resulting epoched data were re-referenced to the average activity of all scalp electrodes.

Spectral decomposition of stimulus-driven EEG signals

Artifact-free epochs were truncated to 3 sec starting 0.5 sec after RSVP onset, to exclude the initial event-related potentials. In truncated and detrended (i.e., linear trend removed) epochs, we quantified SSVEPs by means of Fourier transforms (technical details below) at each EEG sensor and for each condition, separately. We first inspected power spectra for peaks at RSVP (15 Hz) and face-regularity rates (5 Hz). Provided these peaks were present (which was the case in all participants), we used a recently developed spatial filtering approach to represent SSVEPs as an optimally weighted sum of all EEG sensors [37]. Spatial filters were derived separately for each experimental condition. We defined the signal bandwidth as +/-0.5 Hz, centered on the SSVEP frequency of the regularity-driven signal (5 Hz). Noise was defined as the spectral components centred on frequencies -/+ 1 Hz surrounding frequencies of interest with a bandwidth of +/-0.5 Hz (FWHM), respectively. To reduce numerical instabilities in spatial-filter estimation that may arise due to low trial numbers, the noise covariance matrix was regularised by adding 1% of the mean of its eigenvalues to its diagonal [38]. We opted to derive spatial filters per condition because topographical maps of regularity-driven SSVEPs differed substantially between conditions and thus prevented the alternative common-filter approach (one filter for all conditions) as well as the traditional definition of regions (i.e., electrode clusters with largest amplitude identified on topographical maps).

Note that we applied the same approach to conditions with irregular stimulus presentation (in the absence of a regularly repeated emotional expression) for reasons of consistency, although we did not expect to find a regularity-driven SSVEP. This can lead to overfitting noise and produce a spectral peak in the absence of an SSVEP [37]. However, in comparing regular and irregular conditions, the former should always produce a greater response when driving an SSVEP (see Statistical Analysis section below).

Filter-projected single-trial EEG time series were then multiplied with a Tukey window and subjected to Fourier transforms using the Fieldtrip function ft_freqanalysis (method ‘mtmfft’) [39]. To this end, data were zero-padded to a length of 10 sec, allowing for a unified frequency resolution of 0.1 Hz across experiments. From complex spectral representations of single trials we then computed the Cosine Similarity Index [40]—a measure of inter-trial phase clustering that can be interpreted similarly to the classic inter-trial coherence, ITC [41,42], or its derivative ITCz [43,44] but is less sensitive to the number of trials n—according to: where cos denotes the cosine transform and θ is the angle of the complex Fourier coefficient at the frequency of interest in separate trials i and j. Essentially, CS is based on quantifying and then summing the cosine of the angle differences between any two pairs of trials.

An overview of the CS values for each condition can be found in .

Statistical analysis

CS values were analyzed with Bayesian multilevel regressions using brms [45], a user-friendly R package that interfaces with the probabilistic programming language STAN [46] to estimate posterior distributions of the parameters of interest via Markov Chain Monte Carlo (MCMC) algorithms [47]. All models were fitted using weakly informative priors, i.e., Normal(0,3) on beta coefficients and Student(3,0,2) on the standard deviation of varying effects (i.e., participants). As a response distribution function, a Gaussian distribution was chosen. Parameters were estimated using 8 MCMC chains with 8,000 iterations each, 4,000 warmup samples—to get the sequences closer to the mass of the posterior distributions and then discarded—, and a thinning interval of 2, to minimize sample autocorrelation. Thus, the total number of retained posterior samples per parameter was 16,000.

We verified model convergence by visually inspecting trace plots and graphical posterior predictive checks [48]. We also examined: (i) the ratio of effective numbers of samples—i.e., the effective number of samples divided by the total number of samples—, which we aimed to keep larger than 0.1 to avoid excessive dependency between samples; (ii) the Gelman-Rubin statistic [49]—comparing the between-chains variability (how much do chains differ from each other?) to the within-chain variability (how widely did a chain explore the parameter space?) [50]—which, as a rule of thumb, should not be larger than 1.05 or chains may not have successfully converged; (iii) the Monte Carlo standard error (MCSE)—the standard deviation of the chains divided by their effective sample size—a measure of sampling noise [51].

Differences between conditions were assessed by computing the mean and the 95% highest density interval (HDI) of the difference between posterior distributions of the relevant parameters [51] and calculating evidence ratios (ERs), i.e., the ratios between the proportion of posterior samples on each side of zero. ERs can be interpreted as the probability of a directional hypothesis (e.g., “condition A is larger than condition B”) against its alternative (e.g., “condition B is larger than condition A”). As a rule of thumb, we interpreted our results as providing “inconclusive” evidence for a specified directional hypothesis when 1 < ER < 3, “anecdotal” evidence when 3 < ER < 10, and “strong” evidence when ER > 10. When ER = Inf, the posterior distribution was completely on one side of zero, thus providing “very strong” evidence. Please note that contrasts between different conditions in the irregular stimulus presentations are excluded from the results because uninterpretable, due to the overfitting issue with the spatial filtering approach mentioned above (see section Spectral decomposition of stimulus-driven EEG signals).

Throughout the main text we report the results of the analysis carried out on the cosine-similarity values at the Fourier coefficients that correspond to the regularity frequency in respective stimulation conditions (also for irregular presentation conditions). The results of behavioral performance, post-experiment emotion ratings, and SSVEP activity at the stimulation frequency are described in their respective sections in the Supplementary Materials.

Software

Data visualization and statistical analyses were carried out in R v3.6.1 [52] via RStudio v1.2.1335 [53]. We used the following packages (and their respective dependencies):

data manipulation: tidyverse v1.2.1 [54], Rmisc v1.5 [55];

statistical analyses: brms v2.10 [56], rstan v2.19.2 [57];

v

isualization: ggplot2 v3.2.1 [58], ggpirate v0.1.1 [59], bayesplot v1.7.0 [48], tidybayes v1.1.0 [60], bayestestR v0.3.0 [61], BEST v0.5.1 [62], viridis v0.5.1 [63], cowplot v1.0.0 [64];

report generation: knitr v1.25 [65].

Results

Irrespective of face orientation, regular conditions elicited larger SSVEPs relative to irregular presentations (see ), indicating that our stimulation protocol produced the intended regularity-driven SSVEPs. This is further demonstrated by the prominent parieto-occipital topographies of SSVEP maxima in the regular conditions that are absent in the irregular conditions (see ). We also observed that angry and neutral conditions elicited comparable SSVEPs when upright and inverted. Interestingly, neutral regular conditions showed larger SSVEPs compared to angry, both when upright [ER = 162.27] and inverted [ER = 25.02].

Spectral characteristics of EEG responses to stimulation in Pilot 1.

(A) Topographical distributions of phase-locking, quantified as the cosine similarity (CS) index, at the regularity frequency of 5 Hz. Note the lack of phase-locking, i.e., consistent responses to the irregular stimulation conditions (maps use the same scale, in arbitrary units); (B) CS index of phase locking across the EEG spectrum (arbitrary scale) with the group average superimposed on single subject spectra, based on RESS spatial filter projections and collapsed across conditions featuring a regular presentation (i.e., excluding irregular stimulation conditions for their lack of signal; see panel A). For visualization only, CS has been converted to log(CS). (C) CS at the regularity frequency for each participant (single dots) and condition. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes. See for specific results of statistical comparisons.

Discussion

In this first pilot experiment (Pilot 1), we tested for enhanced SSVEPs driven by the regular presentation of angry over neutral faces at a rate of five similar emotional expressions per second (5 Hz) embedded in a stream of 15 facial identities per second (15 Hz). Based on previous studies [e.g., 25], we expected enhanced SSVEPs for regularly presented angry compared to neutral faces. Instead, our results showed the opposite pattern: neutral faces drove a more robust response. Further, upright and inverted regular faces showed comparable SSVEPs, in contrast with an expected effect of face inversion [i.e., smaller SSVEPs for inverted faces; see 22]. We speculated that the high variability of the stimulus material—nine different facial identities displaying three emotional expressions to various degrees of intensity—might have been a confounding factor. Specifically, angry expressions may differ more between identities than neutral expressions, and this greater dissimilarity may have led to a less consistent brain response.

To evaluate this hypothesis, in a follow-up pilot study we decided to use two instead of nine face identities. We chose a female identity with relatively low variability between angry and neutral expressions and one male identity with relatively high variability (see ). Trials were split evenly into only presenting either the female or the male identity. We expected that using fewer identities might facilitate the extraction of emotional information from the fast RSVP, thus leading to a face inversion effect (larger responses for upright relative to inverted faces). Furthermore, using one identity per trial should attenuate effects of variability in low-level visual features across facial expressions, thereby facilitating the actual extraction of emotional information leading to the expected gain effect for angry faces (i.e., enhanced SSVEP). Supporting this assumption, Vakli et al. [66] showed that the SSVEP adaptation effect to prolonged stimulation was invariant to changes in facial expression only when the same identity was presented.

Exemplar image sequences used in Pilot 2 and Experiment 1 and 2.

(A) Sequence of six exemplar images for each experimental condition. For the regular conditions, every third image contained a repetition of the same emotional category (angry or neutral). For the irregular condition, image triplets contained all four emotional categories (neutral, angry, happy, disgusted) in random order. Images were presented either upright or upside down. For each trial only one of two face identities was presented, either with low (see top row) or high (see row two) dissimilarity between emotional expressions. Note that, for illustration purposes, we here show images from the Face Research Lab London Set [31], additionally modified in GIMP 2.10.14 (https://www.gimp.org/) to display anger. A set of different (only male) face identities was presented during the experiment, but cannot be shown here due to copyright restrictions. Please refer to https://www.macbrain.org/resources.htm for examples of faces used in the experiment. (B) Exemplar visual presentation of stimuli within each trial. Single images were presented for 33 msec in Pilot 2 or 83 msec in Experiment 1 and 2, followed by a fixation cross-only image of similar duration, leading to a presentation rate of 15 Hz for the image stream in Pilot 2 and 6 Hz in Experiment 1 and 2. The resulting regularity frequency was 5 Hz for Pilot 2 and 2 Hz for Experiment 1 and 2 during the regular conditions. Note: image not to scale.

Pilot 2

Sixteen participants (12F/4M, median age 23.5 years, range 19–35) were recruited from the student population and the general public. Inclusion criteria, informed consent, reimbursement, and ethical statement are identical to Pilot 1.

From the NimStim database [29], we used identities #06 (female, low dissimilarity between emotional expressions) and #37 (male, high dissimilarity between emotional expressions). The selected expressions were neutral, angry, happy, and disgusted (all with closed mouth). Emotion dissimilarity was determined by calculating the structural similarity index [following the procedure in ref. 67] between emotional expressions within each identity and selecting the ones with highest and lowest values. Picture resizing, luminance matching, and oval placement were identical to Pilot 1. See for an illustration.

The procedure was similar to Pilot 1, with the following exceptions. For each trial, only one of two identities was shown with varying emotional expressions. Both identities varied in the degree of similarity between emotional expressions, i.e., female identity with low dissimilarity and male identity with high dissimilarity. An additional emotional expression (disgusted, closed mouth) was presented to compensate for the reduced variation in picture content due to the presentation of only one identity per trial. Disgusted faces were only used as fillers and never presented regularly during the trials.

EEG recording and preprocessing were similar to the previous experiment, except that epochs displaying female and male identities (i.e., with low and high emotion dissimilarity) were kept separate. The number of interpolated channels was M = 3.88, SD = 1.11. See for the mean percentage of rejected epochs.

The spectral decomposition of the preprocessed EEG signal was identical to Pilot 1. Again, the stimulation frequency was 15 Hz, consequently setting the face regularity frequency to 5 Hz.

The statistical analyses were identical to Pilot 1. However, given the inclusion of another predictor (i.e., emotion dissimilarity), the combinations of our condition levels were 12: 2 (face orientation: upright, inverted) x 3 (regularity: neutral, angry, irregular) x 2 (emotion dissimilarity: low, high).

Similarly to Pilot 1, our data indicated that the regular presentation of emotional expressions drove SSVEPs at the regular rate of 5 Hz, whereas the irregular presentation did not (see and ).

Spectral characteristics of EEG responses to stimulation in Pilot 2.

(A) Topographical distributions of phase-locking, quantified as cosine similarity (CS), at the regularity frequency of 5 Hz for high variance stimuli. Note the lack of phase-locking, i.e. consistent responses to the irregular stimulation conditions (maps same scale, in arbitrary units); (B) CS index of phase locking across the EEG spectrum (arbitrary scale) with the group average superimposed on single subject spectra, based on RESS spatial filter projections and collapsed across conditions featuring a regular presentation of high variance stimuli (i.e., excluding irregular stimulation conditions for their lack of signal; see A). For visualization only, CS has been converted to log(CS); (C & D) Same as in panels A & B (identical scale) but for 5 Hz SSVEPs driven by low variance stimulation; (E) CS at the regularity frequency for each participant (single dots) and condition. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes. See for specific results of statistical comparisons.

Further mirroring the results of the previous pilot study, face inversion did not influence brain responses, with similar SSVEPs for upright and inverted regular faces (see ).

Regarding contrasts between emotional expressions, regular neutral faces showed greater SSVEPs than angry faces only when inverted and when within-identity emotion variability was high [ER = 614.38].

This experimental design also allowed us to assess the impact of within-identity emotion variability on steady-state responses. Regular neutral faces of the highly variable (as compared with the less variable) identity elicited pronounced SSVEPs, both when upright [ER = 90.95] and inverted [ER = 409.26]. Angry, high variable faces elicited slightly enhanced SSVEPs only when upright [ER = 12.69].

We designed Pilot 2 to test whether we would find the expected gain effect of emotional (angry) over neutral expressions while using only one facial identity per trial (instead of nine) at a presentation rate of 15 Hz and the regular occurrence of one emotional expression of 5 Hz. While the results of Pilot 1 showed increased neural responses for regularly presented neutral faces, in Pilot 2 this effect could not be observed. Neutral and angry faces elicited SSVEPs of comparable magnitude, with the notable exception of a persisting reversed effect (neutral > angry) for the high-variability identity when inverted. Possibly, angry facial expressions differed more strongly between identities in Pilot 1, thus driving less consistent brain responses. Using only one facial identity per trial presumably mitigated this confound [66]. Nevertheless, the hypothesized processing advantage for angry faces was still absent.

Although our regularity frequency of 5 Hz seems close to the optimal frequency of 6 Hz for facial processing in an RSVP [e.g., 16], the actual RSVP was presented at 15 Hz which might have been too fast to extract the relevant emotional features while accentuating physical differences. Consistent with this potential caveat in our design, upright and inverted faces drove 5-Hz responses with comparable magnitudes [see also 28]. This absence of effect might indicate that the visual system was not properly able to extract the “facialness” of the stimuli, which typically produces enhanced neural responses for upright relative to inverted faces [23,68].

In the following Experiment 1 we thus opted for an RSVP rate of 6 Hz, which decreased the regularity frequency to 2 Hz. We assumed that slowing down presentation times would provide the visual system with enough time and perceptual information to routinely process faces within the RSVP and consequently extract emotional expressions. By using the same two identities as in Pilot 2 (thus controlling for low-level influences of facial variability), we expected to see the intuitive benefit of angry over neutral expressions as an increase in regularity-driven 2 Hz SSVEPs.

Experiment 1

Thirty-one participants (22F/9M, median age 25 years, range 18–48) were recruited from the student population and the general public. This sample size was chosen based on available time and economic resources (no statistical a priori power analysis was conducted). Inclusion criteria, informed consent, reimbursement, and ethical statement are identical to the previous experiments.

Stimuli were identical to Pilot 2 (see ).

The procedure was similar to Pilot 2 (see ), with the exception that the total presentation rate of faces was reduced to 6 per second (stimulation frequency = 6 Hz; cycle length ~167 msec) and regular faces occurred at a third of that rate (regularity frequency = 2 Hz). Due to the slower presentation rate, we extended the length of each trial to 7 sec (to ensure a number of cycles in the regularity frequency comparable with Pilot 1 and Pilot 2) and increased the minimum idle interval between subsequent target events from 600 to 800 msec. Longer trials also required to subdivide the experiment into 16 experimental blocks, each consisting of 36 trials and lasting ~5 min.

EEG recording and preprocessing were identical to Pilot 2. The interpolated channels were M = 3.26, SD = 1.37. See for the mean percentage of rejected epochs.

The spectral decomposition of the preprocessed EEG signal was identical to the previous experiments except that the extended trials, now including a visual stimulation of 7 sec, where truncated to EEG data epochs with a length of 6.5 sec, starting 0.5 sec after RSVP onset. Spatial filters were centered on the new regularity frequency (2 Hz).

The statistical analyses were identical to Pilot 2.

Spectra in demonstrate that slowing down the pace of the stimulation still elicited an RSVP-driven 6 Hz SSVEP and a regularity-driven 2 Hz SSVEP. Moreover, the 2 Hz SSVEP showed the same occipito-temporal topography as its 5 Hz counterpart in Pilot 1 and Pilot 2. No 2 Hz SSVEP was driven in the irregular presentation condition.

Spectral characteristics of EEG responses to stimulation in Experiment 1.

(A) Topographical distributions of phase-locking, quantified as cosine similarity (CS), at the regularity frequency of 2 Hz for high variance stimuli. Note the lack of phase-locking, i.e. consistent responses to the irregular stimulation conditions (maps same scale, in arbitrary units); (B) CS index of phase locking across the EEG spectrum (arbitrary scale) with the group average superimposed on single subject spectra, based on RESS spatial filter projections and collapsed across conditions featuring a regular presentation of high variance stimuli (i.e., excluding irregular stimulation conditions for their lack of signal; see A). For visualization only, CS has been converted to log(CS); (C & D) Same as in panels A & B (identical scale) but for 2 Hz SSVEPs driven by low variance stimulation; (E) CS at the regularity frequency for each participant (single dots) and condition. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes. See for specific results of statistical comparisons.

Face orientation had a small influence on brain activity, with stronger SSVEPs during upright relative to inverted presentation of angry but not neutral faces (see ). This effect was consistent for high- [ER = 17.16] and low-variability facial identities [ER = 10.15].

Regular neutral faces elicited stronger SSVEPs compared to angry faces in both orientations and emotion variabilities (see ). When presented upright and variability was low, this effect was weaker [ER = 6.57].

Finally, when comparing high vs. low variability, regular angry and neutral faces of the highly variable identity elicited stronger SSVEPs in both orientations.

In Experiment 1 we tested whether we could observe preferential attention allocation towards threatening faces using a slower presentation rate (6 Hz), with one facial identity per trial that displayed a particular emotion (angry or neutral) every third face (2 Hz). Results showed a difference in brain responses driven by the two orientations: upright faces drove a stronger response compared to their inverted counterparts, in line with the notion that naturally oriented faces experience a processing advantage in the visual system, likely due to their special relevance as a social cue display [23,68]. This effect also corroborated our choice of a slower optimal presentation frequency for faces, in line with earlier findings [14,16].

However, despite this progress in adjusting experimental parameters to reveal the hypothesized pattern of results, the expected processing advantage for emotional expressions was still not observed. In fact, the reversed effect—neutral expressions driving stronger responses, present in Pilot 1 yet not observed in Pilot 2—was re-introduced in Experiment 1 (with the exception of upright presented faces of the less variable variety). Additionally, note that the effect of variability (i.e., the physical difference between neutral and angry expressions of one facial identity) was highly consistent irrespective of the regular emotional expression or orientation of the stimuli. The latter effect indicated that physical differences could still be a confounding factor, also when comparing responses to emotional expressions [69].

We considered that Pilot 1, Pilot 2, and Experiment 1 all required participants to perform a visual detection task that was superimposed on the RSVP of faces but completely unrelated. This manipulation may have diverted attentional resources necessary to process the stream of faces more comprehensively. Our recent experiment, in which participants were not performing a task but passively viewing an RSVP of IAPS scenes [27], showed stronger SSVEPs (indicative of enhanced processing) for emotional content at the expected regularity frequencies. In Experiment 2 we therefore removed the dot detection task, changing the design to a passive viewing situation, and tested whether the findings observed with IAPS scenes would generalise to face stimuli.

Experiment 2

Thirty participants (27F/3M, median age 22 years, range 19–37) were recruited from the student population and the general public. Sample size rationale, inclusion criteria, informed consent, reimbursement, and ethical statement are identical to the previous experiments.

Stimuli were identical to the previous experiment (see ).

The procedure was similar to Experiment 1. However, in contrast to the detection task used in the previous experiments, here we employed a simple passive viewing task: participants were asked to fixate the cross and attentively view the picture stream. We presented only 360 trials (60 trials per condition) because, in contrast to Pilot 1, Pilot 2 and Experiment 1, here trials did not contain any targets and/or behavioral responses; therefore, all trials could be included in the analysis.

EEG recording and preprocessing were similar to the previous experiment. The interpolated channels were M = 3.50, SD = 1.09. See for the mean percentage of rejected epochs.

The spectral decomposition of the preprocessed EEG signal was identical to Experiment 1.

The statistical analyses were identical to the previous experiment.

As in Pilot 1, Pilot 2, and Experiment 1, presenting one emotional expression regularly every three faces elicited an SSVEP at one third of the RSVP rate, i.e., 2 Hz. Note, however, that in this instance the 2-Hz SSVEP for low-variability angry upright faces was almost indiscernible from noise [ER = 3.35] (see ). Regularity-driven SSVEP showed the same occipito-temporal topography and no 2 Hz SSVEP was driven in the irregular presentation condition (see ).

Spectral characteristics of EEG responses to stimulation in Experiment 2.

(A) Topographical distributions of phase-locking, quantified as cosine similarity (CS), at the regularity frequency of 2 Hz for high variance stimuli. Note the lack of phase-locking, i.e. consistent responses to the irregular stimulation conditions (maps same scale, in arbitrary units); (B) CS index of phase locking across the EEG spectrum (arbitrary scale) with the group average superimposed on single subject spectra, based on RESS spatial filter projections and collapsed across conditions featuring a regular presentation of high variance stimuli (i.e., excluding irregular stimulation conditions for their lack of signal; see A). For visualization only, CS has been converted to log(CS); (C & D) Same as in panels A & B (identical scale) but for 2 Hz SSVEPs driven by low variance stimulation; (E) CS at the regularity frequency for each participant (single dots) and condition. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes. See for specific results of statistical comparisons.

Face inversion modulated brain activity only for neutral expressions: irrespective of emotion variability, regular neutral conditions showed stronger SSVEPs when faces were upright (see ). SSVEPs driven by regular angry faces were comparable for upright and inverted presentations. Regular neutral faces elicited larger SSVEPs than angry faces in both orientations, but only when variability was high [upright: ER = 107.11; inverted: ER = 17.14]. Finally, all conditions elicited greater SSVEPs in high compared to low within-identity emotion variability.

In Experiment 2 we tested whether passive viewing changed SSVEPs during a 6Hz presentation rate (i.e., regular facial expressions at 2 Hz). Previous Pilots and Experiment 1 had used a dot detection task, effectively withdrawing attention from the picture stream (i.e., faces were distractors). Nonetheless, redirecting the attentional focus did not produce the expected increase in brain responses for angry over neutral faces. Instead, and in accordance with the studies presented above, neutral faces drove stronger responses than angry faces, at least for the highly variable identity displaying greater dissimilarity between angry and neutral expressions.

It is worth pointing out that passive viewing produced a different face inversion effect compared to Experiment 1. Here, neutral but not angry faces consistently drove stronger responses when presented upright, whereas in Experiment 1 we observed the opposite pattern of results. This might indicate that shifting the focus towards the face stream (instead of being a background distractor for an unrelated task) can alter the processing of the “facialness” of the stimuli [25]. Possibly, upright angry faces were stronger distractors in Experiment 1. Their threatening expressions may have increased the probability of them being recognized as human faces as opposed to other objects, presumably resulting in a stronger orientation effect. The passive viewing situation in Experiment 2 perhaps counteracted this effect because attention was already fully allocated to the faces. Here, neutral faces might have been processed more intensely because the visual system expected emotional content given the RSVP context (i.e., mostly emotional expressions). The increased effort to extract emotional content from neutral faces might have given rise to the observed orientation effect during passive viewing. However, this speculative, post-hoc explanation must be evaluated against the fact that neutral faces generally drove stronger brain responses when compared with angry faces, a finding that aligns with other recent reports for non-facial emotional stimuli [26,70]. Furthermore, SSVEP amplitude modulation due to task demands were not directly compared, thereby limiting our conclusions.

Finally, also in Experiment 2 the variability between emotional expressions of one facial identity was a strong contributor to the variance in the elicited brain response.

General discussion

The experiments reported here were designed to investigate under which circumstances the human brain is able to extract emotional facial expressions embedded regularly in an RSVP stream. The RSVP elicited periodic brain responses (SSVEPs) at its presentation rate of 15 Hz in two Pilot studies and at 6 Hz in two Experiments. Based on previous studies that introduced regular oddballs in continuous RSVP streams [e.g., 16,22,26], we expected to observe an SSVEP response at the regularity frequency (one-third of the presentation rate; 5 Hz in the Pilots, 2 Hz in the main Experiments). Further, if emotional expressions affords increased processing because motivationally salient, stronger regularity-driven brain responses ought to be expected for angry relative to neutral faces. We additionally implemented two control conditions: First, we presented RSVP streams with inverted faces, having identical physical properties compared with upright faces but resulting in delayed or disrupted identification processes [23,71,72] and expected diminished or absent electrophysiological responses at the regularity frequency [68]. Second, we presented RSVP streams consisting of the same face stimuli but without the regular repetition of emotional expressions, thereby assessing whether any observed responses at the regularity frequencies would indeed be related to processing the facial content or simply represent a subharmonic of the stimulation frequency.

Our analysis across all experiments revealed robust SSVEP responses at 15 Hz or 6 Hz, i.e. at the rate of respective RSVPs. In our Pilots (15 Hz RSVPs) we also observed a 5 Hz response for regularly presented stimuli, regardless of stimulus presentation (upright vs. inverted). For the two experiments with a 6 Hz RSVP, the 2 Hz SSVEPs were more prominent in streams with high within-expression variability compared to the streams with low variability. This was also observed with inverted faces, corroborating the results of our Pilots. In irregular streams, regularity-driven SSVEPs were absent, confirming that these brain responses cannot solely be ascribed to integer subharmonics of the driving frequencies of stimulus presentation but are indeed related to the processing of the regularly presented facial expression.

Regarding our main contrast of interest, we expected higher SSVEP responses for regular angry than for regular neutral expressions. This hypothesis stemmed from the results of a number of studies showing greater N170 amplitude for emotional compared to neutral faces [for a review, see ref. 73] as well as recent reports of enhanced SSVEPs for emotional oddball faces [e.g., 28]. Here, however, SSVEPs were enhanced for regularly presented neutral compared to angry faces in almost all our conditions, irrespective of face orientation. This effect cannot be explained by possible misinterpretations of the emotional cues within the faces: as outlined in the Supplementary Material, in all experiments angry faces were perceived as conveying more anger than neutral, happy, or disgusted faces. Therefore, it is unlikely that neutral faces were preferentially processed because they were less ambiguous compared to the other emotional expressions. Below we outline a few possible post-hoc interpretations of these results.

Superposition of emotion-sensitive ERP components

What we initially considered counterintuitive findings could be re-interpreted under a different light when integrating recent published results. A study using IAPS pictures, conducted in our laboratory in parallel to the experiments reported here, already showed enhanced SSVEPs for neutral compared to emotional (i.e., unpleasant) IAPS images [70]. We proposed that these unexpected results were due to a superposition effect of an occipito-temporal ERP component called early posterior negativity (EPN), whose amplitude is typically increased in response to emotional as opposed to neutral material [74-76]. The amplitude of the EPN is usually more positive for neutral compared to unpleasant images. We surmised that, at a 6-Hz stimulation rate, the EPN components serially elicited by each individual image might superimpose and consequently create larger SSVEPs for neutral RSVP streams. Follow-up studies showed that the increased SSVEP in response to neutral stimuli was limited to 6 Hz and not observed at 3 Hz, 4 Hz, and 8.57 Hz [26], providing additional indirect support that only stimulation frequencies long enough to allow the EPN to fully develop—but not too long to include subsequent ERPs—would elicit such an effect. In another study [77], we presented participants with a task-relevant random dot kinematogram superimposed onto a stream of neutral or emotional images and flickering at 4 or 6 Hz, respectively. Results showed prioritized processing of background (i.e., task-irrelevant) emotional compared to neutral stimuli at 4 Hz, but a reverse effect at 6 Hz. Another study [27] employed a paradigm very similar to Experiment 3 but using complex pictures instead of facial expressions. Results showed increased SSVEPs for pleasant compared to neutral and unpleasant pictures in the regularity frequency (i.e., 2 Hz). The difference between neutral and unpleasant conditions was not statistically significant, despite a trend for larger responses to neutral scenes.

Following a similar logic, enhanced SSVEPs in regular neutral conditions reported in the present paper might be a consequence of a smaller negativity (i.e., larger positivity) for neutral compared to angry faces within 200 msec after stimulus onset. In the case of facial stimuli, the corresponding ERP component would be the N170, the first component that more robustly differentiates between facial and non-facial objects [19,21] as well as neutral vs. emotional facial expressions [73,78-80]. According to the dominant view in the literature, however, a serial superposition of the N170 should lead to greater regularity responses for angry instead of neutral faces (i.e. our working hypothesis throughout the whole research program). More importantly, this interpretation is inconsistent with the results of Pilot 1 and Pilot 2, where the stimulation and regularity frequencies were much faster (15 Hz and 5 Hz, respectively).

The role of isolated facial features

In their seminal paper, Diamond and Carey [81] proposed that the visual system uses three types of information to recognize faces: (a) isolated features, such as eyes and mouth; (b) first-order relational/spatial properties (i.e., the eyes are above the nose); and (c) second-order relational information (i.e., the distance between the eyes, or from mouth to nose). More recently, Calvo et al. [82] suggested a functional model of face processing comprising of a serial sequence of steps, with individual features being processed first and very rapidly (< 170 ms) followed by the processing of configurational information, then followed by the analysis of affective cues (around 800 ms after stimulus onset). This is reminiscent of a classical neuroanatomical model of face perception [83] proposing a functional subdivision of a network of occipitotemporal brain areas into two systems: the “core system”, responsible for the initial processing of essential facial constituents (e.g., eyes, mouth) in the inferior occipital gyri, the analysis of dynamic features (e.g., eye gaze) in the superior temporal sulcus, and invariant aspects (e.g., unique identity) in lateral fusiform gyrus; and the “extended system”, comprising of various brain regions involved in higher-order cognitive functions such as attention (e.g., intraparietal sulcus), emotion extraction (e.g., amygdala), and speech prosody (e.g., auditory cortex). How may these serial models help us interpret the results reported here?

In our experiments, the rapid presentation of individual images within an RSVP stream would presumably allow only for the initial perceptual processes to complete (e.g., interpretation of eyes and mouth and relations among these elements) and little time to analyze more complex information, including emotional cues—especially if we assume strong forward/backward masking effects [3]. This interpretation is further consistent with the absence of a robust inversion effect, i.e. a processing advantage of upright presented faces across all stimulation frequencies. Specifically, inversion might have been ineffective because diagnostic information from each face was extracted from individual features (mainly eyes and mouth), whose processing may be less affected by face inversion [84,85, but see 86 for a different perspective].

Having said that, our participants’ brain activity was still differentially modulated by neutral and angry facial expressions. If not emotion, what kind of perceptual information was used by the visual system to diagnostically distinguish between these two conditions? We explored the possibility that within-identity emotion variability could play a role and found some support for this hypothesis: RSVP streams with low variability elicited a smaller regularity brain response at either 5 Hz or 2 Hz, and differences between neutral and angry faces were either not present (Pilot 2) or very small (Experiment 1 and Experiment 2) [see also 66].

A recent study by Poncet et al. [87] used a very similar protocol to the one employed here. These authors presented faces with varying emotional expressions at 6 Hz. Every 5th face (i.e., at 1.2 Hz), either a neutral regular oddball among emotional faces or a regular emotional oddball was presented. Similarly to our findings, their results showed overall larger electrophysiological responses for neutral compared to a range of emotional expressions (along with a selective neural response to fear). Different topographical distributions were also observed among the various emotional faces and neutral ones, suggesting the selective involvement of distinct neuronal networks [in accordance with ref. 83]. Poncet et al. [87] argue that contrasting emotional and non-emotional facial expressions may modulate core regions of the face processing network that differentiate between these two categories, in particular the superior temporal sulcus and the lateral fusiform gyrus which are particularly active in processing changeable features of the faces as well as emotional cues. Such an explanation might also accommodate our findings showing similar SSVEP responses for upright and inverted faces, and more robust results for RSVP streams with highly variable emotional expressions.

Finally, another possible explanation for our findings pertains to the fact that, in this experimental paradigm, neutral faces are the only “non-emotional” stimuli within the RSVP stream, thus serving as “oddball” throughout the whole stimulation. Such an interpretation would assign more weight on theories advocating for general arousal and/or valence mechanisms [88-90]—neutral would be less arousing, and its emotional connotation less prominent, than any other displayed emotion—rather than qualitatively discrete emotional categories [1,91]. This would be in contrast with recent studies interpreting their results as evidence for specific neural processes dedicated to each facial expression [85,90]. The RSVP techniques employed here and elsewhere [28,87,92] are promising tools to disentangle these alternative theoretical interpretations and are a fruitful avenue for future research.

Conclusions

In a series of experiments, we measured continuous brain responses to rapid serial presentations of faces with emotional expressions and tested whether the visual system prioritizes processing of angry over neutral faces. While manipulating a range of features and applying refinements and additional controls to existing paradigms, our collective results do not support such a processing advantage. Instead, we consistently observed enhanced electrical brain responses during the regular presentation of neutral as opposed to angry faces. We suggest that these findings might be due to the rapid extraction of featural information, particularly within-identity emotion variability: RSVP streams with low variability elicited a smaller regularity brain response, and differences between neutral and angry faces were either not present or very small. Further, we found that hampering face processing by inverting faces had only weak effects on brain responses. Conversely, using facial expressions with high variability yielded stronger brain responses than streams of low-variability expressions. Taken together, our results call for a cautious interpretation of results from experiments using face-stimulus RSVPs to study the processing of emotional expressions—confounds in low-level visual stimulus features might play an important role, and might be more difficult to control than previously acknowledged.

Supplementary materials.

(DOCX)

Click here for additional data file.

Face ratings.

Face ratings for each emotional expression, separately for each participant (single dots), question (how happy/angry/disgusted is this face?), and experiment. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes. Ratings range from 1 (very low emotional intensity) to 9 (very high emotional intensity).

(TIFF)

Click here for additional data file.

Reaction times.

Reaction times (in msec) in response to colored dots during the presentation of each face stream, separately for each participant (single dots), condition, and experiment. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes.

(TIFF)

Click here for additional data file.

Cosine similarity at stimulation frequency.

Cosine similarity (CS) calculated at the stimulation frequency for each participant (single dots), condition, and experiment. Mean values are marked by horizontal black lines and 95% confidence intervals represented as transparent boxes.

(TIFF)

Click here for additional data file.

Valence ratings: Descriptives.

Ratings of emotional valence (angry, happy, disgusted) of the different image sets, separately for each experiment, experimental condition, and question.

(XLSX)

Click here for additional data file.

Valence ratings: Analysis.

Statistical comparisons of valence ratings between all pairs of emotional image sets, separately for all experiments and experimental conditions. Mean and 95% HDI are related to differences in rated valence between the respective comparison. Comparisons showing strong evidence against the hypothesis of no difference are presented in bold.

(XLSX)

Click here for additional data file.

Reaction times: Descriptives.

Reaction times to targets presented during trials of the different experimental conditions, separately for each experiment with a behavioral task (i.e., excluding Experiment 2).

(XLSX)

Click here for additional data file.

Reaction times: Analysis.

Statistical comparisons of reaction times between all pairs of factor levels, separately for all experiments and experimental conditions.

(XLSX)

Click here for additional data file.

Cosine similarity at stimulation frequency: Descriptives.

Statistics of Cosine Similarity Index (CS) of the signals at the stimulation frequency, separately for the different experiments and experimental conditions.

(XLSX)

Click here for additional data file.

Cosine similarity at stimulation frequency: Analysis.

Statistical comparisons of Cosine Similarity values between all pairs of factor levels, separately for all experiments and experimental conditions. Mean and 95% HDI are related to differences in reaction times between the respective comparison. Comparisons showing strong evidence against the hypothesis of no difference are presented in bold.

(XLSX)

Click here for additional data file.

10 Jan 2020

PONE-D-19-32180

Rapid processing of neutral and angry expressions within ongoing facial stimulus streams: Is it all about isolated facial features?

PLOS ONE

Dear Dr. Schettino,

Thank you for submitting your manuscript to PLOS ONE. As you can see from their comments both reviewers were generally positive about your manuscript. My own reading of the manuscript is also positive. Therefore, i would like to invite you to resubmit a revised version of your work that carefully considers all these comments. Please, make sure to pay special attention to the reformulation of hypotheses and to the issue regarding the interpretation of your data in terms of task demands.

We would appreciate receiving your revised manuscript by Feb 24 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

José A Hinojosa, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.  Thank you for including your ethics statement:

"The experiments were conducted in accordance with the guidelines of the ethics committee of the University of Leipzig."

a) Please amend your current ethics statement to confirm that your named institutional review board or ethics committee specifically approved this study.

b) Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

3. We note that Figure(s) 1 and 3 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

1.    You may seek permission from the original copyright holder of Figure(s) 1 and 3 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

2.    If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

4. We note that Figures 1 and 3 include an image of a [patient / participant / in the study].

As per the PLOS ONE policy (http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research) on papers that include identifying, or potentially identifying, information, the individual(s) or parent(s)/guardian(s) must be informed of the terms of the PLOS open-access (CC-BY) license and provide specific permission for publication of these details under the terms of this license. Please download the Consent Form for Publication in a PLOS Journal (http://journals.plos.org/plosone/s/file?id=8ce6/plos-consent-form-english.pdf). The signed consent form should not be submitted with the manuscript, but should be securely filed in the individual's case notes. Please amend the methods section and ethics statement of the manuscript to explicitly state that the patient/participant has provided consent for publication: “The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details”.

If you are unable to obtain consent from the subject of the photograph, you will need to remove the figure and any other textual identifying information or case descriptions for this individual.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The studies by Schettino and colleagues investigated the electrophysiological correlates of facial expressions (i.e., neutral vs. angry) processing by means of a fast periodic stimulation (RSVP) approach. The Authors presented four different EEG studies in which several parameters/factors have been manipulated to assess the brain responses underlying the hypothesized processing advantage for emotional (vs. neutral) face expressions. These factors included the stimulus presentation rate (fast-15Hz vs. slow-6Hz), variability across facial expressions (high vs. low), and task (passive observation vs. secondary task). The main result reported was against the central hypothesis, but, more importantly, it was consistent throughout both the pilots and main experiments.

I think that the paper is well-written and technically sound, and it adheres to the standards of the Open Science framework.

Despite that, several aspects should be addressed and improved by the authors that would strengthen their manuscript.

Main comments:

1. Introduction. The introduction is well understandable, but it raises expectations (not respected) in the reader about results in line with the central hypothesis of the Authors. In the discussion section, the Authors reported several findings that are consistent with their findings (enhanced SSVEPs in response to neutral vs. emotional stimuli). In my humble opinion, the Authors could reorganize the Introduction by introducing and anticipating both hypotheses: the main one (enhanced SSVEPs to angry stimuli) and the possibility for the opposite one (supported by the literature in the General Discussion). The Bayesian approach, already used, could be suitable for testing the direction of such effect. Given that the Authors proposed four different studies to assess the role of different factors in modulating face processing, such a rigorous approach should be highlighted from the beginning. This would emphasize the consistency in the results and be beneficial to the entire manuscript.

2. Methods: Spectral decomposition of stimulus-driven EEG signals.

The authors employed spatial filtering (RESS) introduced by Cohen and Gulbinaite (2017). I wonder if a more “traditional” approach based on a single or group (ROI) electrode selection would lead to different results. Either a convergence or comparison between the two techniques of analysis would strengthen the manuscript. In this regard, what about possible hemispheric asymmetries previously reported during face (also emotional expression) processing? Did the Authors expect, or check, for any gender-related difference in brain responses, since their groups of participants included both male and female individuals?

3. Discussions and General Discussion. I think that each Discussion section would benefit from the inclusion of a few bibliographical references in support of the main results. This would help to link each section to the following experiment proposed.

In the General Discussion, I truly appreciated the statement about possible post-hoc interpretations of the results made by the Authors. However, under my comment about reorganizing the Introduction, I would suggest anticipating previous evidence reported here (consistent with that of the Authors) by reporting it in the earlier sections of the manuscript (i.e., Introduction and Interim Discussions). In this regard, the last paragraphs are dense with data and notions and seem to be written with a different style. Why not use previous studies to support, instead of justifying, the (stable) results from the present work?

Minor comments:

1. In both the introduction and General Discussion, the Authors referred to several studies without reporting the name of the principal author. This style of writing simplifies the text but could make it difficult to understand if the Authors are referring to their previous works or to those of other research groups. I suggest increasing clarity on this point.

2. In the plots regarding CS values (i.e., Figures 2, 4, 5, 6), it would be nice to mark (i.e., “*”) the significant differences between conditions that have been found. This would help to guide the reader through the results in the different studies. This suggestion also applies to supplementary figures reporting subjective ratings.

Reviewer #2: The series of experiments performed by Schettino and colleagues sought to investigate the degree to which the visual system can extract emotional information from faces during an RSVP paradigm. Specifically, the question was to determine what the ideal 'extraction speed' would be, hypothesizing that angry faces would be preferentially processed and therefore lead to an enhanced ssVEP over neutral. Surprisingly, neutral faces consistently, over 4 experiments, showed enhanced processing via the ssVEP.

The authors nicely explain this neutral face effect with various hypotheses in their discussion. Both hypotheses (superposition effect and isolated facial features) are viable. However, a third option could be that across identities, the expression of 'angry' is variable, whereas neutral is highly consistent. Although testing for within-identity variability in facial expression (how was dissimiliarity of facial expression determined?), the consideration of between-subjects variability for each emotion is worth debating. This would naturally apply more for the first pilot study using multiple identities, but may generally be decoded by the face network system as a reliable, consistent stimulus, that in a stream of highly variable expressions (constantly switching between 3 emotions) evokes a strong signal due to the more easily predictive nature of a consistent neutral face.

Overall, the study with its four experiments were well designed and described. It is reassuring to see topographies remain stable across experiments. The irregular control condition was particular nice, although authors should be careful in saying that "regularity frequencies were compared", as for the irregular condition there was no regularity frequency by design. Minor semantic comment. I would be interested to see the overall ssVEP signal to the RSVP for the irregular condition, however. Despite showing that the regularity frequency was not a subharmonic of the main driving frequency, would be useful to see if this regularity frequency modified the overall RSVP signal in any way, in a multiplicative or even destructive fashion. With growing conversation in the literature on superposition, the amplification of fundamental frequencies by non-physical harmonics, so to speak, could potentially add to this literature.

The interpretation of the emotional differences between experiments 1 and 2 should be described with more caution, as task demand interpretation could go either way (enhanced attention, withdrawn attention) and was not primarily tested for in this study.

I would recommend this paper for acceptance with minor revisions. Most of the revisions should be directed at a more streamlined and concise conceptual setup in the introduction. For example the reader begins by thinking the point of the paper is timing of emotional extraction, but then aside from two fundamental frequencies (15 and 6Hz), timing/frequency is not parametrically modulated. Instead, the story evolves into wanting to show that threat should evoke a greater response than neutral. The introduction reads more like an extended methods section than a conceptual walk through, with not enough ‘why we should care’. Overall, less conversational language should be used, which is particularly pronounced in the discussion. This work is timely and contributes to face processing literature as well as expanding our mechanistic understanding of ssVEP signals and their caveats.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

28 Feb 2020

Reviewers' comments

Reviewer #1

The studies by Schettino and colleagues investigated the electrophysiological correlates of facial expressions (i.e., neutral vs. angry) processing by means of a fast periodic stimulation (RSVP) approach. The Authors presented four different EEG studies in which several parameters/factors have been manipulated to assess the brain responses underlying the hypothesized processing advantage for emotional (vs. neutral) face expressions. These factors included the stimulus presentation rate (fast-15Hz vs. slow-6Hz), variability across facial expressions (high vs. low), and task (passive observation vs. secondary task). The main result reported was against the central hypothesis, but, more importantly, it was consistent throughout both the pilots and main experiments.

I think that the paper is well-written and technically sound, and it adheres to the standards of the Open Science framework.

We thank the Reviewer for the positive evaluation and constructive feedback.

Despite that, several aspects should be addressed and improved by the authors that would strengthen their manuscript.

Main comments:

1. Introduction. The introduction is well understandable, but it raises expectations (not respected) in the reader about results in line with the central hypothesis of the Authors. In the discussion section, the Authors reported several findings that are consistent with their findings (enhanced SSVEPs in response to neutral vs. emotional stimuli). In my humble opinion, the Authors could reorganize the Introduction by introducing and anticipating both hypotheses: the main one (enhanced SSVEPs to angry stimuli) and the possibility for the opposite one (supported by the literature in the General Discussion). The Bayesian approach, already used, could be suitable for testing the direction of such effect. Given that the Authors proposed four different studies to assess the role of different factors in modulating face processing, such a rigorous approach should be highlighted from the beginning. This would emphasize the consistency in the results and be beneficial to the entire manuscript.

Following the Reviewer’s suggestions, we integrated our Introduction and Abstract sections to more clearly emphasize our initial starting hypothesis: processing advantages for angry facial expressions should manifest in an increased SSVEP amplitude compared to neutral faces. The opposite pattern (attenuated SSVEPs for threatening stimuli) was initially unanticipated. In the literature, initial support for this opposite pattern only accrued in the course of this research program and was thus not previously available (also, to our knowledge, the majority of findings in the current literature on this topic still report larger SSVEP amplitude for emotion-laden as opposed to neutral information). We believe that, in its current form, our manuscript faithfully reflects this timeline. We prefer to keep this structure to avoid potential misinterpretations, i.e., finding support for a hypothesis we did not formulate a priori.

2. Methods: Spectral decomposition of stimulus-driven EEG signals.

The authors employed spatial filtering (RESS) introduced by Cohen and Gulbinaite (2017). I wonder if a more “traditional” approach based on a single or group (ROI) electrode selection would lead to different results. Either a convergence or comparison between the two techniques of analysis would strengthen the manuscript. In this regard, what about possible hemispheric asymmetries previously reported during face (also emotional expression) processing? Did the Authors expect, or check, for any gender-related difference in brain responses, since their groups of participants included both male and female individuals?

While we appreciate the reasoning behind the Reviewer’s request to provide more “traditional” analyses, we would argue against their inclusion in this manuscript, for several reasons. First, re-running all analyses and reporting their results would inflate an already voluminous paper, hence diluting the central findings. Second, Cohen and Gulbinaite (2017) already provide an in-principle comparison between the traditional and the RESS approach. Please note that we are not advocating the use of RESS as a superior analysis method for SSVEPs in general. However, in our series of experiments, we expected (and found) topographic differences in regularity-driven SSVEP between experimental conditions; therefore, defining amplitude-based ROIs would be suboptimal (we clarified this point in the revised manuscript, page 17). A traditional, ROI-based approach might produce slightly different findings -- although, we surmise, with no qualitative differences in the pattern of results reported in our manuscript. However, we would be inclined to give the RESS-derived findings more weight because they do not rest on assumptions imposed by ROI/electrode selections. Instead, RESS better adapts to topographic differences between experimental conditions: see, for instance, the scalp maps for upright vs. inverted faces in Fig. 2 or low vs. high expression variability in Figs. 4-5-6. The interested reader is encouraged to reuse and reanalyze our data (available at https://osf.io/uhczc/ under a Creative Commons Attribution 4.0 International Public License) with any approach they are interested in.

In a similar vein, we would also refer to the availability of the data for any follow-up exploratory analysis of hemispheric or gender differences. We believe it would not be appropriate to include them in the present manuscript not only for reasons of succinctness, but also because such exploratory questions were outside the scope of our investigation.

3. Discussions and General Discussion. I think that each Discussion section would benefit from the inclusion of a few bibliographical references in support of the main results. This would help to link each section to the following experiment proposed.

In the General Discussion, I truly appreciated the statement about possible post-hoc interpretations of the results made by the Authors. However, under my comment about reorganizing the Introduction, I would suggest anticipating previous evidence reported here (consistent with that of the Authors) by reporting it in the earlier sections of the manuscript (i.e., Introduction and Interim Discussions). In this regard, the last paragraphs are dense with data and notions and seem to be written with a different style. Why not use previous studies to support, instead of justifying, the (stable) results from the present work?

In line with the Reviewer’s comment, we have added supporting references to the Discussion sections of each experiment. Regarding the General Discussion, as explained above, we prefer to keep the current structure because it highlights the timeline of our research programme. Some of the cited studies, pivotal for subsequent design decisions and current interpretation of the results, were not published at the time we conceived and began the research program.

Minor comments:

1. In both the introduction and General Discussion, the Authors referred to several studies without reporting the name of the principal author. This style of writing simplifies the text but could make it difficult to understand if the Authors are referring to their previous works or to those of other research groups. I suggest increasing clarity on this point.

We thank the Reviewer for pointing this out. We now clarify the corresponding parts accordingly throughout the text.

2. In the plots regarding CS values (i.e., Figures 2, 4, 5, 6), it would be nice to mark (i.e., “*”) the significant differences between conditions that have been found. This would help to guide the reader through the results in the different studies. This suggestion also applies to supplementary figures reporting subjective ratings.

We would like to emphasize that, from a statistical perspective, “significance” is not defined in the present context. We provide rule-of-thumb thresholds for Evidence Ratios (ERs) for ease of readability; however, we encourage readers to appreciate the rich evidential value of continuous measures of Bayesian inference (for an introduction, see Kruschke & Liddell, 2017). For this reason, it would be inconsistent to mark comparisons in the Figures as suggested by the Reviewer. Furthermore, the large number of comparisons would result in many symbols in each figure, consequently impairing their readability.

The Figures are meant to illustrate the distribution of the data, for example by clearly showing no discernible regularity activity in the irregular conditions as well as differences between low and high within-identity emotion variability in upright faces (see Fig 2). Nonetheless, we appreciate the Reviewer’s suggestion to better guide readers through the results of the different studies. Therefore, we now ensure that, in each figure caption, the corresponding table with relevant statistical results are appropriately referenced.

Reviewer #2

The series of experiments performed by Schettino and colleagues sought to investigate the degree to which the visual system can extract emotional information from faces during an RSVP paradigm. Specifically, the question was to determine what the ideal 'extraction speed' would be, hypothesizing that angry faces would be preferentially processed and therefore lead to an enhanced ssVEP over neutral. Surprisingly, neutral faces consistently, over 4 experiments, showed enhanced processing via the ssVEP.

We thank the Reviewer for the careful consideration of our manuscript as well as the positive remarks throughout the review.

The authors nicely explain this neutral face effect with various hypotheses in their discussion. Both hypotheses (superposition effect and isolated facial features) are viable. However, a third option could be that across identities, the expression of 'angry' is variable, whereas neutral is highly consistent. Although testing for within-identity variability in facial expression (how was dissimiliarity of facial expression determined?), the consideration of between-subjects variability for each emotion is worth debating. This would naturally apply more for the first pilot study using multiple identities, but may generally be decoded by the face network system as a reliable, consistent stimulus, that in a stream of highly variable expressions (constantly switching between 3 emotions) evokes a strong signal due to the more easily predictive nature of a consistent neutral face.

We agree with the Reviewer. As noted on page. 23: “We speculated that the high variability of the stimulus material -- nine different facial identities displaying three emotional expressions to various degrees of intensity -- might have been a confounding factor. Specifically, angry expressions may differ more between identities than neutral expressions, and this greater dissimilarity may have led to a less consistent brain response.”. This prompted us to separately display facial identities with low and high within-identity emotion variability and assess the contribution of this factor in modulating the SSVEP signal. Another study reporting SSVEP adaptation invariant to changes in facial expression only when the same identity was presented (Vakli et al., 2014) is now cited (e.g., on page 24).

We took this opportunity to respond to another request of the Reviewer, i.e., clarify how we determined within-identity emotion variability in our experiments (see page 25).

One more point raised by the reviewer merits attention. It is plausible to hypothesize that regular neutral faces received prioritized attention allocation -- and, therefore, elicited larger SSVEP amplitude -- because they were the only non-emotional stimuli in the RSVP stream. We already briefly mentioned this alternative explanation on page 40: “[...] neutral faces might have been processed more intensely because the visual system expected emotional content given the RSVP context (i.e., mostly emotional expressions)”. We now emphasize this interesting alternative explanation in the General Discussion section, page 46-47.

Overall, the study with its four experiments were well designed and described. It is reassuring to see topographies remain stable across experiments. The irregular control condition was particular nice, although authors should be careful in saying that "regularity frequencies were compared", as for the irregular condition there was no regularity frequency by design. Minor semantic comment.

We have clarified the wording in line with the Reviewer’s comment. Specifically, on page 20, we now write: “Throughout the main text we report the results of the analysis carried out on the cosine-similarity values at the Fourier coefficients that correspond to the regularity frequency in respective stimulation conditions (also for irregular presentation conditions).”.

I would be interested to see the overall ssVEP signal to the RSVP for the irregular condition, however. Despite showing that the regularity frequency was not a subharmonic of the main driving frequency, would be useful to see if this regularity frequency modified the overall RSVP signal in any way, in a multiplicative or even destructive fashion. With growing conversation in the literature on superposition, the amplification of fundamental frequencies by non-physical harmonics, so to speak, could potentially add to this literature.

The Reviewer is invited to browse the Supplementary Materials included with our submission -- specifically, the Stimulation Frequency section --, where we report a full analysis of 15Hz (pilots) and 6Hz (experiments) SSVEPs driven by the RSVP. In brief, we observed increased attention allocation during highly variable (upright) face streams, particularly for regular angry and irregular conditions. Thus, there does not seem to be robust evidence that regularity specifically affected (in a multiplicative or destructive way) the SSVEP signal at the stimulation frequency. However, as highlighted in the Supplementary Materials, we advise caution in the interpretation of these findings given their heterogeneous pattern, and hope that future studies will shed light on this issue more systematically.

The interpretation of the emotional differences between experiments 1 and 2 should be described with more caution, as task demand interpretation could go either way (enhanced attention, withdrawn attention) and was not primarily tested for in this study.

We thank the Reviewer for pointing out this caveat. This information has now been added on page 41.

I would recommend this paper for acceptance with minor revisions. Most of the revisions should be directed at a more streamlined and concise conceptual setup in the introduction. For example the reader begins by thinking the point of the paper is timing of emotional extraction, but then aside from two fundamental frequencies (15 and 6Hz), timing/frequency is not parametrically modulated. Instead, the story evolves into wanting to show that threat should evoke a greater response than neutral. The introduction reads more like an extended methods section than a conceptual walk through, with not enough ‘why we should care’. Overall, less conversational language should be used, which is particularly pronounced in the discussion. This work is timely and contributes to face processing literature as well as expanding our mechanistic understanding of ssVEP signals and their caveats.

We thank the Reviewer for this observation, which mirrors Reviewer #1’s comments. We have now clarified and emphasized the main hypotheses in the Abstract and Introduction sections.

Submitted filename: PONE-D-19-32180_Response_Journal_Reviewers.pdf

Click here for additional data file.

6 Apr 2020

Rapid processing of neutral and angry expressions within ongoing facial stimulus streams: Is it all about isolated facial features?

PONE-D-19-32180R1

Dear Dr. Schettino,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

José A Hinojosa, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Consistent with reviewers' suggestions, the authors have made changes in their manuscript that increased the overall readability and understanding of the work. The hypotheses are better framed in the introduction, which led to enhanced consistency through the following sections.

Reviewer #2: The authors have nicely addressed all concerns that I raised and I am happy to see this work published. I particularly applaud the authors on the rigor of their 4-experimental setup.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

9 Apr 2020

PONE-D-19-32180R1

Rapid processing of neutral and angry expressions within ongoing facial stimulus streams: Is it all about isolated facial features?

Dear Dr. Schettino:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. José A Hinojosa

Academic Editor

PLOS ONE

Table 1

Percentage of rejected epochs after preprocessing.

experiment	variability	orientation	regularity	mean	st.dev.	min.	max.
Pilot 1		upright	angry	13.85	3.57	8.33	21.67
			neutral	14.27	4.45	10.00	23.33
			irregular	16.35	4.69	10.00	25.00
		inverted	angry	15.31	5.78	6.67	28.33
			neutral	15.73	4.64	6.67	26.67
			irregular	15.42	5.64	6.67	23.33
Pilot 2	high	upright	angry	11.25	4.06	3.33	16.67
			neutral	15.62	5.36	6.67	23.33
			irregular	15.42	5.76	6.67	30.00
		inverted	angry	13.54	6.18	3.33	26.67
			neutral	12.29	4.37	3.33	20.00
			irregular	15.62	6.64	3.33	30.00
	low	upright	angry	14.58	5.51	10.00	30.00
			neutral	13.75	5.64	3.33	26.67
			irregular	13.79	4.57	3.33	23.33
		inverted	angry	18.12	8.08	3.33	33.33
			neutral	15.83	7.41	3.33	26.67
			irregular	13.54	6.29	3.33	26.67
Experiment 1	high	upright	angry	15.73	6.87	3.45	33.33
			neutral	16.13	7.90	0.00	36.67
			irregular	18.45	8.05	3.33	40.00
		inverted	angry	17.26	8.39	0.00	40.00
			neutral	16.44	8.02	3.33	36.67
			irregular	15.12	6.74	3.33	33.33
	low	upright	angry	19.78	8.80	3.33	43.33
			neutral	15.63	7.49	3.33	36.67
			irregular	17.15	9.16	3.33	36.67
		inverted	angry	16.00	8.05	0.00	40.00
			neutral	15.96	6.04	3.33	30.00
			irregular	15.77	7.49	0.00	30.00
Experiment 2	high	upright	angry	14.11	8.20	3.33	33.33
			neutral	15.37	7.24	3.33	36.67
			irregular	12.20	5.78	0.00	24.14
		inverted	angry	14.13	6.19	0.00	30.00
			neutral	13.56	7.84	0.00	30.00
			irregular	14.33	6.62	3.33	36.67
	low	upright	angry	15.00	6.25	3.33	26.67
			neutral	12.81	7.07	0.00	23.33
			irregular	15.57	9.56	3.33	43.33
		inverted	angry	13.89	6.33	0.00	26.67
			neutral	14.56	9.08	3.33	43.33
			irregular	15.02	9.01	3.33	40.00

Descriptive statistics of the percentage of removed trials after preprocessing, separately for all experiments and experimental conditions.

Table 2

Cosine Similarity Index (CS), regularity frequency.

experiment	variability	orientation	regularity	CS
				mean [95% CI]
Pilot 1		upright	angry	0.03 [0.01, 0.04]
			neutral	0.05 [0.04, 0.06]
			irregular	0.00 [0.00, 0.01]
		inverted	angry	0.03 [0.01, 0.04]
			neutral	0.05 [0.04, 0.06]
			irregular	0.00 [-0.01, 0.01]
Pilot 2	high	upright	angry	0.06 [0.02, 0.09]
			neutral	0.09 [0.06, 0.12]
			irregular	-0.02 [-0.04, -0.01]
		inverted	angry	0.03 [0.02, 0.04]
			neutral	0.09 [0.06, 0.12]
			irregular	-0.01 [-0.02, 0.00]
	low	upright	angry	0.02 [0.00, 0.04]
			neutral	0.04 [0.03, 0.05]
			irregular	-0.02 [-0.03, -0.01]
		inverted	angry	0.03 [0.01, 0.05]
			neutral	0.03 [0.01, 0.05]
			irregular	-0.03 [-0.04, -0.01]
Experiment 1	high	upright	angry	0.04 [0.03, 0.05]
			neutral	0.06 [0.04, 0.07]
			irregular	0.01 [0.00, 0.02]
		inverted	angry	0.03 [0.02, 0.04]
			neutral	0.04 [0.03, 0.05]
			irregular	0.00 [0.00, 0.01]
	low	upright	angry	0.03 [0.02, 0.04]
			neutral	0.04 [0.02, 0.05]
			irregular	0.01 [0.00, 0.02]
		inverted	angry	0.02 [0.01, 0.03]
			neutral	0.03 [0.02, 0.04]
			irregular	0.00 [0.00, 0.01]
Experiment 2	high	upright	angry	0.04 [0.03, 0.05]
			neutral	0.06 [0.05, 0.08]
			irregular	0.00 [0.00, 0.01]
		inverted	angry	0.04 [0.02, 0.05]
			neutral	0.05 [0.04, 0.06]
			irregular	0.01 [0.00, 0.01]
	low	upright	angry	0.02 [0.01, 0.03]
			neutral	0.03 [0.02, 0.04]
			irregular	0.01 [0.00, 0.02]
		inverted	angry	0.02 [0.01, 0.02]
			neutral	0.02 [0.01, 0.03]
			irregular	0.00 [0.00, 0.01]

Statistics of Cosine Similarity Index (CS) of the signals at the regularity frequency, separately for the different experiments and experimental conditions. Regularity frequencies: 5 Hz in Pilot 1 and Pilot 2, 2 Hz in Experiment 1 and Experiment 2.

Table 3

Results statistical analyses Cosine Similarity Index (CS) at regularity frequency.

experiment	variability	orientation	regularity	comparison	mean [95% HDI]	evidence ratio
Pilot 1			angry	upright vs. inverted	0.00 [-0.02, 0.02]	1.38
			neutral		0.01 [-0.01, 0.02]	3.39
		upright		neutral vs. angry	-0.03 [-0.05, -0.01]	162.27
				irregular vs. angry	0.02 [0.01, 0.04]	234.29
				irregular vs. neutral	0.05 [0.04, 0.06]	Inf
		inverted		neutral vs. angry	-0.02 [-0.04, 0.00]	25.02
				irregular vs. angry	0.03 [0.01, 0.04]	1,141.86
				irregular vs. neutral	0.05 [0.03, 0.06]	Inf
Pilot 2	high		angry	upright vs. inverted	0.03 [-0.02, 0.07]	8.91
			neutral		0.00 [-0.06, 0.06]	1.03
	low		angry	upright vs. inverted	-0.01 [-0.04, 0.02]	2.54
			neutral		0.01 [-0.01, 0.04]	4.52
	high	upright		neutral vs. angry	-0.03 [-0.08, 0.02]	9.09
				irregular vs. angry	0.08 [0.04, 0.12]	1,776.78
				irregular vs. neutral	0.11 [0.07, 0.15]	Inf
		inverted		neutral vs. angry	-0.06 [-0.09, -0.02]	614.38
				irregular vs. angry	0.04 [0.02, 0.06]	1,332.33
				irregular vs. neutral	0.10 [0.06, 0.13]	Inf
	low	upright		neutral vs. angry	-0.02 [-0.04, 0.01]	8.19
				irregular vs. angry	0.04 [0.02, 0.07]	1,999.00
				irregular vs. neutral	0.06 [0.04, 0.08]	Inf
		inverted		neutral vs. angry	0.00 [-0.03, 0.04]	1.29
				irregular vs. angry	0.06 [0.03, 0.08]	15,999.00
				irregular vs. neutral	0.05 [0.03, 0.08]	1,776.78
		upright	angry	high vs. low	0.04 [-0.01, 0.08]	12.69
			neutral		0.05 [0.01, 0.09]	90.95
		inverted	angry	high vs. low	0.00 [-0.03, 0.03]	1.09
			neutral		0.06 [0.02, 0.10]	409.26
Experiment 1	high		angry	upright vs. inverted	0.01 [0.00, 0.02]	17.16
			neutral		0.01 [-0.01, 0.03]	7.62
	low		angry	upright vs. inverted	0.01 [0.00, 0.02]	10.15
			neutral		0.01 [-0.01, 0.02]	2.63
	high	upright		neutral vs. angry	-0.02 [-0.04, 0.01]	10.83
				irregular vs. angry	0.03 [0.02, 0.04]	Inf
				irregular vs. neutral	0.05 [0.03, 0.07]	15,999.00
		inverted		neutral vs. angry	-0.01 [-0.03, 0.00]	16.72
				irregular vs. angry	0.03 [0.02, 0.04]	Inf
				irregular vs. neutral	0.04 [0.03, 0.05]	Inf
	low	upright		neutral vs. angry	-0.01 [-0.02, 0.01]	6.57
				irregular vs. angry	0.02 [0.01, 0.03]	245.15
				irregular vs. neutral	0.03 [0.01, 0.05]	1,141.86
		inverted		neutral vs. angry	-0.01 [-0.02, 0.00]	102.90
				irregular vs. angry	0.01 [0.00, 0.02]	409.26
				irregular vs. neutral	0.03 [0.01, 0.04]	Inf
		upright	angry	high vs. low	0.01 [0.00, 0.03]	30.37
			neutral		0.02 [0.00, 0.04]	39.51
		inverted	angry	high vs. low	0.01 [0.00, 0.02]	49.63
			neutral		0.01 [0.00, 0.03]	23.46
Experiment 2	high		angry	upright vs. inverted	0.00 [-0.01, 0.02]	1.66
			neutral		0.01 [-0.01, 0.03]	11.44
	low		angry	upright vs. inverted	0.00 [-0.01, 0.02]	1.89
			neutral		0.01 [0.00, 0.02]	75.56
	high	upright		neutral vs. angry	-0.03 [-0.05, -0.01]	107.11
				irregular vs. angry	0.04 [0.02, 0.05]	Inf
				irregular vs. neutral	0.06 [0.05, 0.08]	Inf
		inverted		neutral vs. angry	-0.01 [-0.03, 0.00]	17.14
				irregular vs. angry	0.03 [0.01, 0.04]	15,999.00
				irregular vs. neutral	0.04 [0.03, 0.06]	Inf
	low	upright		neutral vs. angry	-0.01 [-0.02, 0.00]	9.51
				irregular vs. angry	0.0 [-0.01, 0.02]	3.35
				irregular vs. neutral	0.02 [0.00, 0.03]	189.48
		inverted		neutral vs. angry	0.00 [-0.01, 0.01]	1.03
				irregular vs. angry	0.01 [0.01, 0.02]	940.18
				irregular vs. neutral	0.01 [0.01, 0.02]	2,284.71
		upright	angry	high vs. low	0.02 [0.00, 0.04]	42.24
			neutral		0.04 [0.02, 0.05]	2,284.71
		inverted	angry	high vs. low	0.02 [0.01, 0.03]	362.64
			neutral		0.03 [0.02, 0.05]	3,199.00

Statistical comparisons of Cosine Similarity values between all pairs of factor levels, separately for all experiments and experimental conditions. Mean and 95% HDI are related to differences in CS value between the respective comparison. Comparisons showing strong evidence against the hypothesis of no difference are presented in bold.