Decoding Steady-State Visual Evoked Potentials From Electrocorticography.

We report on a unique electrocorticography (ECoG) experiment in which Steady-State Visual Evoked Potentials (SSVEPs) to frequency- and phase-tagged stimuli were recorded from a large subdural grid covering the entire right occipital cortex of a human subject. The paradigm is popular in EEG-based Brain Computer Interfacing where selectable targets are encoded by different frequency- and/or phase-tagged stimuli. We compare the performance of two state-of-the-art SSVEP decoders on both ECoG- and scalp-recorded EEG signals, and show that ECoG-based decoding is more accurate for very short stimulation lengths (i.e., less than 1 s). Furthermore, whereas the accuracy of scalp-EEG decoding benefits from a multi-electrode approach, to address interfering EEG responses and noise, ECoG decoding enjoys only a marginal improvement as even a single electrode, placed over the posterior part of the primary visual cortex, seems to suffice. This study shows, for the first time, that EEG-based SSVEP decoders can in principle be applied to ECoG, and can be expected to yield faster decoding speeds using less electrodes.

1. Introduction

Electrocorticography (ECoG) records electrophysiological signals from the cortical surface, without the electrodes penetrating the cortical tissue, unlike multi-electrode arrays and deep brain electrodes. This recording modality has led to new opportunities for Brain-Computer Interfacing (BCI). While scalp-EEG avoids surgical interventions, it comes with inferior signal quality and spatial resolution, due to volume conduction, which challenges the speed/accuracy ratio when judging decoding performance (Schalk, 2010; Schalk and Leuthardt, 2011). Additionally, success in EEG-based BCI applications depends on alleviating eye and head motion artifacts (Croft and Barry, 2000), including those causing minute electrode displacements, the drying out of the conductive gel that ensures low impedance contact with the subject's scalp, and the time-consuming electrode set-up and the ensuing retraining of the BCI prior to each use (for review, see Lacko et al., 2017). When relying on micro-electrode arrays such as the Utah array (Hochberg et al., 2006; Collinger et al., 2013) one is facing permanent damage to the cortical tissue and its vasculature leading to histological processes and fibrous scar tissue formation (Schalk and Leuthardt, 2011) and brain plasticity processes that deteriorate the signal quality and whence BCI performance (Shain et al., 2003). Compared to scalp-recorded EEG, ECoG signals are less contaminated by eye and motion artifacts (Ball et al., 2009), have higher amplitudes (Schalk and Leuthardt, 2011), a broader bandwidth (Staba et al., 2002), and higher spatial resolution (Miller et al., 2009b). Several studies were able to discriminate fine real (i.e., performed) movements from ECoG recordings such as hand extension and flexion (Jiang et al., 2017), hand movement direction (Leuthardt et al., 2004) and individual finger movements (Miller et al., 2009a), all of which are not be feasible from EEG recordings. ECoG-based BCI have furthermore shown great promise for long-term recording sessions (Chao et al., 2010; Wang et al., 2013) with reports of 262 days BCI-use without signal degradation (Vansteensel et al., 2016).

While motor paradigms are raising great hopes to eventually achieve intuitive control over prosthetic limbs, currently, instead of continuous, only discrete control can be achieved (i.e., discrimination between distinct imagined motor tasks) (Onose et al., 2012; Vansteensel et al., 2016) as the extraction of reliable limb trajectory information is still largely beyond reach (Korik et al., 2018). In the discrete setting, visual paradigms vastly outperform sensorimotor paradigms in terms of accuracy, number of selectable targets and information transfer rate (ITR) (Nicolas-Alonso and Gomez-Gil, 2012). Several studies have investigated the feasibility of ECoG for visual BCI paradigms, such as the P300 Event-Related Potential (Brunner et al., 2011; Krusienski and Shih, 2011; Speier et al., 2013) and the code-modulated Visual Evoked potential (Kapeller et al., 2013). The implanted electrodes were mainly located over frontal, temporal, parietal and occipital areas, albeit not reaching to the primary visual cortex (V1) and merely a few electrodes to the secondary visual cortex (V2) (Speier et al., 2013). However, to the best of our knowledge, the Steady-State Visual Evoked Potential (SSVEP), the most performant visual BCI paradigm (Chen et al., 2015b), has not yet been assessed in terms of ECoG (or even intracranially recorded signals in general). Subdural electrode implants primordially serve a clinical purpose such as to localize epileptogenic foci of drug-resistant epilepsy and the mapping of eloquent cortex prior to resective surgery. Of all epilepsy cases, only 5–10% have an occipital lobe origin (Sveinbjornsdottir and Duncan, 1993). Foci in the first visual layers (V1, V2) are, moreover, considered inoperable, as resection would leave the patient blinded, thereby normally not warranting electrode implantation. As, in general, only intractable cases of epilepsy are considered for implantation and given the above-mentioned precautions, patients with ECoG grids over primary visual areas are extremely rare. Furthermore, as the SSVEP paradigm relies on periodically flickering stimuli, photo-sensitive epilepsy patients are excluded from participation in an SSVEP study, further limiting the opportunities to study SSVEP responses over (primary) visual areas in human subjects. There are very few SSVEP-based studies with implanted electrodes in the occipital cortex, which are all limited to electrode strips (Kamp et al., 1960; Krolak-Salmon et al., 2003; Winawer et al., 2013), thereby not providing a wider view of the cortical activations, and none of these studies investigated SSVEP decoding performance. We report on a single case study where a large subdural grid was covering the entire right occipital cortex of a human subject.

When used in BCI, SSVEP-selectable targets are traditionally encoded with different flickering frequencies (Regan, 1979; Middendorf et al., 2000) or phases (Lee et al., 2010; Lopez-Gordo et al., 2010; Manyakov et al., 2012), but in order to maximize the number of selectable targets, a joint frequency-phase encoding has been suggested (Jia et al., 2011). Many SSVEP decoding algorithms have been described (for overviews, see Liu et al., 2013; Wittevrongel and Van Hulle, 2017; Lotte et al., 2018), but the more advanced ones are based on adapted versions of canonical correlation analysis (CCA) (Lin et al., 2006; Bin et al., 2009; Pan et al., 2011; Zhang et al., 2011, 2014; Chen et al., 2014; Nakanishi et al., 2014; Yin et al., 2015; Abu-Alqumsan and Peer, 2016; Vu et al., 2016), with the filterbank CCA (Chen et al., 2015a) yielding the highest decoding accuracy, and on a spatiotemporal extension of the beamforming algorithm (Wittevrongel and Van Hulle, 2017), successfully used for SSVEP (Wittevrongel and Van Hulle, 2016c) as well as several other visual BCI paradigms (Wittevrongel and Van Hulle, 2016a,b; Wittevrongel et al., 2017). While both filterbank CCA (fbCCA) and spatiotemporal beamformer (stBF) yield state-of-the-art decoding performance on scalp-recorded EEG, it is unclear how well they perform on ECoG as the signal properties are different (Buzsáki et al., 2012; Ritaccio et al., 2015). This study aims to investigate SSVEP decoding from the cortical surface and to compare the obtained accuracies to the ones from traditional scalp-recorded signals.

2. Methods

2.1. Participants

A 38-year old male patient (right-handed) with refractory, non-photosensitive epilepsy participated in the study. He was admitted to the hospital (UZ Gent) for monitoring seizure activity and functional mapping of eloquent cortex (visual, language, etc.). A subdural grid of 48 platinum electrodes embedded in silastic (Ad-Tech, USA) was implanted over the right occipital cortex. Additionally, a control group of eight healthy subjects (5 male, mean age 24.5 years, ranging from 19 to 30 years, 1 left-handed) were recruited for scalp-EEG recording. Previous studies have shown that BCI accuracy is not affected by age (Allison et al., 2010), thereby not requiring age-matched controls. All recruited participants, including the patient, had normal or corrected-to-normal vision and were informed, prior to participating, about the aim of the study, the experimental procedure, and what would be done with the recorded data, after which they signed the informed consent form. The study was conducted according to the latest version of the Declaration of Helsinki (2013) following prior approval from the ethical committee of Ghent University Hospital (ECoG study) and Leuven University Hospital (UZ Leuven) (EEG study).

2.2. Experimental procedure

The experimental interface consisted of six identical rectangular targets (8.8 × 5.8 cm) presented on a 60 Hz LCD monitor. A number (from 1 to 6) was displayed in the center of each target, serving as a fixation point for the corresponding rectangle. The targets adopted a green color to reduce the visual demand (Cao et al., 2012; Tello et al., 2015) and the risk of inducing an epileptic seizure (Kaiser, 1984). All subjects were seated approximately 60 cm from the screen. At this distance, the visual angle spanned by the rectangles was 8.4° × 5.5°, and the distance from the fixation point to the edge of the neighboring target was 5.5° horizontally and 4.1° vertically. The fixation numbers spanned a visual angle of approximately 0.35° × 0.70°.

In order to investigate the decoding performance for different stimulus settings, the experiment consisted of four sessions, in each of which all rectangles were assigned a unique combination of frequency and phase (Table 1). Each session consisted of 90 trials. At the beginning of each trial, one of the targets was cued by maintaining its green color while the other rectangles were shown in gray (Figure 1A). The subject was asked to direct his/her gaze at the cued target and maintain focus. When the subject pressed a key on the keyboard, all rectangles regained their green color, and 1 s later, the 4-s stimulation was initiated, during which all targets were flickering in accordance with their frequency-phase combination, achieved by sinusoidally modulating their luminosities (Manyakov et al., 2013). Between trials, the subject was allowed to take a short break, and a longer one (around 5 min) between sessions. In each session, all targets were cued 15 times in pseudorandom order. The experiment was implemented in Matlab, using the Psychophysics Toolbox extensions (Brainard and Vision, 1997; Pelli, 1997; Kleiner et al., 2007).

Table 1

Target frequency-phase combinations for each session, represented as [frequency (Hz)/phase (radians)].

Session	Target
	1	2	3	4	5	6
1	12/0	14/2π3	12/4π3	14/4π3	12/2π3	14/0
2	13/0	14/2π3	13/4π3	14/4π3	13/2π3	14/0
3	11/0	15/π	13/0	13/π	11/π	15/0
4	13/0	15/π	14/0	14/π	13/π	15/0

Figure 1

Experimental setup. (A) Time-course of one trial with characteristic displays. For the sake of exposition, target numbers in the first display are larger than those shown to our subjects. (B) Visualization of the patient's right hemisphere with location of implanted ECoG electrodes. Red and green areas correspond to V1 and V2, respectively, and were extracted with Freesurfer (Fischl, 2012). (C) Location of scalp-EEG electrodes for the control experiment.

2.3. Scalp-EEG and ECoG acquisition

For the patient, 48-channel ECoG and 27-channel scalp-EEG (following the international 10-20 system) data were recorded simultaneously at a sampling rate of 1,024 Hz using an SD LTM 64 Express (Micromed, Italy) medically certified device. Each ECoG electrode had a 4.0 mm diameter with 2.3 mm exposure and 10 mm inter-contact spacing. The subdural grid (6 × 8 contacts) was covering the right occipital cortex: convexity and mesial inter-hemispheric cortex (Figure 1B). Functional mapping confirmed that a large part of the grid was located over the primary and associative visual cortex. Due to insufficient quality of the simultaneously recorded scalp-EEG (dried conductive gel, as well as the influence of scarred and swollen tissue), the patient's scalp-EEG was excluded from further analysis.

The scalp recorded EEGs from the control subjects were acquired with a Synamps RT device (Neuroscan, Australia), using 32 active Ag/AgCl electrodes evenly distributed over the scalp (Figure 1C) and at a sampling rate of 2 kHz. The electrodes were placed using an electrode cap (EasyCap GmbH, Germany) with fixed holes according to the international 10/20 system. Conductive gel was applied at each electrode, and the impedance between skin and electrodes was kept below 2 kΩ. As SSVEPs are less susceptible to ocular artifacts (i.e., blinks and saccades) (Regan, 1989; Vialatte et al., 2010), no electro-oculogram (EOG) was recorded and no artifact correction was applied in this study.

2.4. Localization of ECoG electrodes

Based on the pre-implantation MRI scan of the patient, cortical reconstruction, and volumetric segmentation was performed with the Freesurfer image analysis suite (version 5.3.0) (Fischl, 2012). The Freesurfer output was then loaded into Brainstorm (Tadel et al., 2011) and co-registered with the post-implant CT. The coordinates of the implanted electrodes were then obtained by visual inspection and mapped onto the subject's cortical surface.

Figure 1B shows the locations of the implanted electrodes, with indication of the primary (V1) and associative (V2) visual cortices.

2.5. Processing

The processing of the ECoG and EEG data differs only in the used re-referencing. The raw ECoG data was offline re-referenced to the grid's average (Common Average Reference, CAR), while the EEG data was offline re-referenced to the average of both mastoid electrodes (TP9 & TP10).

Since each classifier (see next section for their description) requires different filtering ranges, all recordings were stored four times, but with different filtering ranges: a fourth-order Butterworth filter between 0.5 and 40 Hz for the naive classifier, between 4 and 20 Hz for the spatiotemporal beamformer (Wittevrongel and Van Hulle, 2016b), and between 8 and 70 Hz and between 16 and 70 Hz for the filterbank CCA algorithm (Chen et al., 2015a), respectively. The filtered EEG and ECoG recordings were then cut into 4-s trials time-locked to the onset of the stimulation, downsampled to 500 Hz, labeled with the corresponding cue, and stored for further analysis.

2.6. Classifiers

In this study, we assessed the performance of two state-of-the-art classifiers, filterbank CCA (fbCCA) and spatiotemporal beamformer (stBF), and a simple (called naive) classifier based on the Fourier transform in combination with a phase estimation method.

2.6.1. Naive classifier

This classifier is the most straightforward one to identify gazed targets. First, the most prominent frequency (among the used stimulation frequencies) in a trial was determined from the signal's power spectral density (PSD), obtained after applying the Fourier transform. Then, this frequency was used to estimate the phase of the trial. Finally, the predicted target was taken as the one whose frequency and phase best corresponded to the extracted frequency and phase. This is now explained in more detail.

Training the naive classifier involved the estimation of the phase Φ of each target i∈[1..6], obtained by applying the following equation to each training trial s of target i and averaging their values (Walker, 1988; Jia et al., 2011; Manyakov et al., 2013):

with t the sample in time units, f the frequency of target i, the discrete Fourier transform of s(t), and .

To classify a trial, the PSD was calculated using the Fourier transform with a Hamming window whose size was equal to the length of the trial. For each of the stimulation frequencies f, the power was calculated as the average in a narrow (0.5 Hz) band, centered at frequency f:

where x is the range of the PSD spectrum. The predicted frequency was given by f with the highest power, and further used for estimating the phase of the trial, using Equation 1. Finally, the phase difference δi between Φ and was calculated for all targets i that were encoded by frequency , using the CircStat toolbox (Berens et al., 2009). The gazed target was predicted as the one with the closest-to-zero phase difference.

2.6.2. Filterbank CCA

The CCA algorithm was originally used in the context of SSVEP-BCI for the identification of frequency-encoded SSVEP targets (Lin et al., 2006). The algorithm finds linear combinations and for multivariate sets X ∈ ℝ and Y ∈ ℝ, such that their linear combinations and are maximally correlated (Härdle and Simar, 2015).

In the context of SSVEP, X is the measured (multi-channel) EEG and Y is a set of sine and cosine signals with the stimulation frequency of target i and a number of harmonics (h = k/2).

Since the original algorithm does not contain any phase information, an extension was proposed (Nakanishi et al., 2014) to discriminate joint frequency-phase encoded SSVEP targets by incorporating individual calibration data. The filterbank CCA method (Chen et al., 2015a) combines the aforementioned CCA extension (Nakanishi et al., 2014) with a filterbank approach (Vetterli and Herley, 1992). By analyzing the signals in different sub-bands, information originating from the fundamental and harmonic components are examined more efficiently and aggregated at a later stage. In this study, the number of harmonics h for the predefined signals Y was set to 3 and the sub-bands used were [8-70] Hz and [16-70] Hz as in (Chen et al., 2015b).

2.6.3. Spatiotemporal beamforming

The spatiotemporal extension (van Vliet et al., 2015) (termed here stBF) of the original beamforming algorithm (Van Veen and Buckley, 1988) has first been proposed for the analysis of the N400 ERP in the context of semantic processing, and since then applied in the context of BCIs (Wittevrongel and Van Hulle, 2017) for target identification based on the P300 ERP (Wittevrongel and Van Hulle, 2016a), SSVEP (Wittevrongel and Van Hulle, 2016b,c) and cVEP (Wittevrongel et al., 2017) paradigms.

Spatiotemporal beamforming estimates the contribution of a predefined spatiotemporal pattern to new data. Each trial is cut into consecutive, non-overlapping segments whose length is equal to the period of the expected frequency (Luo and Sullivan, 2010; Manyakov et al., 2010). If the frequency is present in the signal, the averaged segments should resemble a sine-wave of a single period. If not, averaging the segments will cancel out any patterns and result in a “flat” response. As the segments are time-locked, the averaged pattern also accounts for the phase of target stimulation.

Since each target elicits a different brain response (cf., the unique frequency/phase combinations), each target requires a unique activation pattern, hence training the classifier involves calculating six spatiotemporal beamformers, one for each target.

The activation patterns and the beamformers were calculated from the training data as follows: let be the trials obtained during the training session in response to the cued target i∈[1..6], where m represents the number of channels, t the number of samples and r the number of trials, and f be the stimulation frequency of target i. Applying the time-domain approach to all trials in T with f the returned segments , where n is the number of samples corresponding to one period of frequency f and k the total amount of segments obtained from all trials in T. The spatiotemporal activation pattern for target i can now be obtained as the average of all k segments.

The spatiotemporal beamformer was calculated as a linearly-constrained minimum-variance (LCMV) beamformer for target i as follows: let be the matrix where each row was obtained by concatenating the rows of a corresponding segment S[*, *, k], the covariance matrix of E, and a⊺ ∈ ℝ1×( a vector containing the concatenated rows of A. The LCMV beamformer under constraint can now be calculated using the method of Langrage multipliers (Van Veen et al., 1997):

and applied to the data as a simple weighted sum: y = sw, where s ∈ ℝ1×( indicates the concatenated rows of an input segment .

The feature vector for a trial T was given by the beamformer outputs y for each target i. To obtain y, the trial was cut into segments of length one period of frequency f, the segments were averaged and filtered using the corresponding beamformer w.

Based on the feature vector, a prediction was made of the target gazed by the subject. As the feature vectors contain estimates of the degree to which the activation pattern of each target is present in the trial, the target having the highest score was marked as winner [i.e., prediction = max(y)].

Note that longer trials can be cut into more segments, leading to more reliable average segment responses that can be input to the beamformers. Prior to training and classification, trials were bandpass filtered between 4 and 20 Hz.

2.7. Performance evaluation

The decoding accuracy of all classifiers was estimated offline using a stratified 5-fold cross-validation strategy. We estimated performance for varying signal lengths, from 250 ms to 4 s in steps of 250 ms. Note that only the testing trials were shortened, while the training trials remained 4 s long. Single-electrode performance was assessed for all three classifiers, while multi-electrode performance was only estimated for the two state-of-the-art classifiers (filterbank CCA and spatiotemporal beamformer).

2.8. Statistical estimation of the patient's scalp-EEG performance

As the patient's scalp-EEG was not of sufficient quality for use in our analysis, we statistically estimated his scalp-EEG accuracy for the single-electrode case using an imputation analysis based on phase variability. For each trial, the phase was calculated using Equation 1. Albeit that the measured phase on the scalp might be different from the cortical phase measurement due to the mixing of different cortical activities, this should not affect its trial-to-trial variability. First, we verified that the phase variability (measured as the circular standard deviation; Berens et al., 2009) of the ECoG signal (at electrode 36) is not significantly different from the scalp-recorded phase variability at channel Oz of our control subjects. To this end, the Kuiper's test was used (CircStat toolbox; Berens et al., 2009). Next, for each session, an imputation analysis with 100 iterations was performed in which the patients scalp-EEG decoding accuracy was treated as missing value. The patient's trial-to-trial phase variability of the session under consideration (at electrode 36) was used, together with the phase variabilities and decoding accuracies of all sessions from the control group to estimate the patient's scalp-EEG decoding accuracy.

2.9. Electrode selection

While many approaches have been described to recruit electrodes for multi-electrode analysis (Lal et al., 2004; Schröder et al., 2005; Lv and Liu, 2008; Arvaneh et al., 2011; Barachant and Bonnet, 2011), we used greedy forward selection: we iteratively added the electrode that resulted in the largest accuracy increase until no further improvement could be achieved or 100% accuracy was reached. As an optimization criterion, we used the average classification accuracy across stimulation lengths (from 250 ms to 4 s in steps of 250 ms, see above). This procedure was repeated for each classifier separately, in order to maximize their individual accuracies.

While the greedy approach does not guarantee optimal performance (as it does not backtrack on the choices made), it is universal, simple and straightforward to implement compared to other selection strategies. As the main aim of this study was to compare ECoG- and EEG-based SSVEP target decoding accuracy, the selection of the optimal electrode set was not deemed important in this study. Furthermore, the greedy selection approach takes into account the differences inherent to the classifiers (e.g., fbCCA solely aims at detecting the frequency-phase combinations while stBF estimates the contribution of a given activation pattern and takes into account information about noise sources and non-targeted patterns), and is often described as a wrapper approach (Guyon and Elisseeff, 2003).

2.10. Statistics

For the EEG subjects, the significance of the differences in classifier accuracies was assessed with the (two-tailed) Wilcoxon signed rank test. We adopted this non-parametric test since the distributions did not consistently follow a Gaussian distribution. The significance threshold was set to 0.05 for the multi-electrode case, and was corrected to 0.0167 () for the single-electrode case, using the Bonferroni correction for multiple comparisons.

3. Results

Our subjects were shown a visual interface constituting of six rectangular targets, each one periodically flickering at a unique frequency and phase (Table 1). All subjects performed four sessions of 90 trials. In each trial, the subject was asked to direct his/her gaze to a cued target and to maintain focus during the ensuing 4-s flickering stimulation (Figure 1). In each session, all targets were cued 15 times in pseudorandom order.

3.1. Single-electrode performance

As this is the first study on ECoG-based SSVEP decoding, we started with an exploratory analysis of target detection accuracy of individual electrodes when using fbCCA (Chen et al., 2015a) and stBF (Wittevrongel and Van Hulle, 2016b), and a naive classifier based on spectral analysis but extended with phase modeling. The naive classifier has been applied to scalp-EEG recordings of SSVEP responses (for review, see Liu et al., 2013), but now considered outdated in terms of accuracy, hence, it is interesting to see to what extend that classifier could benefit from the improved ECoG signal quality.

Figure 2A shows the highest single-electrode decoding accuracies obtained from the three classifiers for increasing stimulation lengths during the second session. The results of the other sessions are equivalent and are shown in Figures S1–S3 in the Supplementary Material. For ECoG, the two state-of-the-art classifiers, as well as the naive classifier, exhibit a rapid increase in accuracy with increasing stimulation lengths up to 0.75 s of stimulation, after which the accuracy only improves marginally or remains constant. fbCCA is the most accurate classifier with an accuracy of 96.7% for 0.75-s stimulation, compared to 91.1 and 92.2% for stBF and the naive classifier, respectively. Using long stimulation (i.e., 4 s), the accuracy converges to 97.8% for the fbCCA and naive classifiers, and 96.7% for stBF. For scalp-EEG, the accuracy at the best electrode of each subject steadily increases with increasing signal lengths, but remains considerably lower than the ECoG-based accuracy (e.g., 79.6±13.5% for EEG compared to 97.8% for ECoG with fbCCA at a stimulation length of 2 s). Here, the accuracy of the naive classifier is consistently lower than that of the state-of-the-art classifiers, and is significantly different for all stimulation lengths above 0.25 s. At least 1.5 s of stimulation is needed for scalp-EEG to surpass the 70% accuracy threshold deemed necessary to establish reliable communication (Kübler et al., 2004; Kübler and Birbaumer, 2008; Brunner et al., 2011; Combaz et al., 2013). In comparison, using ECoG, all classifiers surpass this threshold within 0.5 s of stimulation.

Figure 2

Single-electrode performance. (A) Best single-electrode accuracies obtained with the three classifiers plotted as a function of stimulation lengths during the second session. The full line indicates accuracies based on cortical recordings and the boxplots summarize accuracies based on scalp-EEG (control subjects). (B-D) Overview of accuracies (averaged across all stimulation lengths) for all cortical electrodes (left) and scalp channels (right) for (B) fbCCA, (C) stBF, and (D) naive classifier. Subdural electrodes indicated in red and green indicate V1 and V2, respectively. Iso-accuracy lines on the subdural grids (left panels) start at 75% accuracy and increase in steps of 5% and on the scalp plots (right panels) start at 50% accuracy and increase in steps of 10%.

Figures 2B–D summarize the average decoding accuracy across all stimulation lengths for each individual electrode during the second session. The EEG electrodes over the middle occipital area (electrode Oz) yield the highest accuracies, as expected. The subdural electrodes located over the posterior part of the primary visual cortex (electrodes 36 and 44) most accurately decode the intended target. These results are consistent across classifiers and sessions (see Supplementary Material for the other sessions).

As the quality of the scalp-EEG of the patient was not sufficient, we statistically estimated his scalp-EEG decoding performance based on the measured phase of the ECoG signal. Figure 3A shows the distribution of the phase deviations across all targets for each session for both the ECoG (electrode 36) and EEG control group (electrode Oz). The distributions of ECoG and scalp-EEG are not statistically significant for all four sessions (p > 0.05, Kuiper's test), indicating that both ECoG and scalp-EEG experience similar phase deviations across trials, and that the higher classification accuracy of ECoG does not originate from more stable phase responses. Using the phase and accuracy measurements of the control group, the accuracy of the patient's scalp-EEG was statistically estimated using an imputation-analysis (Rubin, 1976; Carpenter and Kenward, 2012). Figure 3B shows a regression analysis of the accuracy (averaged over all stimulation lengths) with respect to the phase deviation based on the control group for all classifiers. Additionally, the accuracies and phase deviations of the ECoG signal at electrode 36 are added, as well as the estimations of the patient's scalp-EEG accuracy, given the observed ECoG phase deviations. While all three classifiers exhibit a decrease in accuracy for decreasing phase stability, the naive classifier is most strongly affected, probably as it does not adopt more advanced signal processing procedures. For all classifiers, the ECoG-based accuracies clearly outperform the scalp-EEG counterparts with similar phase deviations.

Figure 3

Estimation of the patient's scalp-EEG accuracy. (A) Distribution of the phase deviation at the stimulation frequency across trials of the ECoG (left column) and scalp-EEG (right column) for the four sessions (rows). Kuiper's test did not reveal any significant differences between the ECoG and scalp-EEG phase deviation distributions for any of the four sessions. (B) Regression of accuracy on phase deviation. Each dot represents one session of one subject. Blue dots indicate the control subjects, red filled dots indicate the patient, and red open dots indicate the statistically estimated scalp-EEG accuracy of the patient with the vertical red line the corresponding standard deviation for 100 imputations.

3.2. Multi-electrode performance

Since several EEG studies showed that the decoding accuracy of scalp-EEG improves when using a multi-electrode approach (Liu et al., 2013), we adopted a greedy forward electrode selection strategy to determine the electrode set for each classifier and subject individually.

Figure 4A shows the resulting classification accuracies for the second session. While the accuracies from ECoG largely remain the same as in the single-electrode case, for EEG, the accuracies are considerably improved. The stimulation length required to reach the 70% threshold now decreases to 0.5 and 0.75 s for stBF and fbCCA, respectively. The spatiotemporal beamformer significantly outperforms the filterbank CCA algorithm for signal lengths from 0.5 to 3.25 s, but significantly underperforms for the shortest stimulation length, albeit that the accuracies of both classifiers are not satisfactory (i.e., below 70%).

Figure 4

Multi-electrode performance. (A) Multi-electrode analysis of classification accuracies during the second session. (B) Number of electrodes selected by the greedy algorithm, for each session. Boxplots and solid line indicate results for EEG (control subjects) and ECoG (implanted patient), respectively. (C) Increase in accuracy (averaged over all stimulation lengths) for all sessions. Boxplots and solid line indicate results for EEG (control subjects) and ECoG (implanted patient), respectively.

Figure 4B shows the number of electrodes selected for each session for both classifiers, and Figure 4C shows the general increase in accuracy (i.e., averaged accuracy across all stimulation lengths) for the four sessions. While the accuracies with EEG increase considerably (median improvement between 14 and 23%), those with ECoG increase only marginally (~ 5% for the stBF and ≤ 1% for fbCCA). For sessions 2 to 4, greedy electrode selection did not come up with additional electrodes for the fbCCA method (Figure 4B), resulting in identical accuracies as for the single-electrode case. Unlike for the fbCCA classifier, between 6 and 11 electrodes were included for the stBF in the ECoG modality, but this led to only minor improvements in decoding accuracy (Figure 4C).

4. Discussion

We investigated the accuracy with which SSVEP-selectable targets can be decoded from ECoG recordings, obtained from a subdural electrode grid spanning the entire right occipital cortex, when using classifiers that were originally designed for scalp-EEG recordings. We considered two state-of-the-art classifiers and one naive approach, based on spectral analysis and phase detection, and assessed their accuracies for both ECoG and scalp-EEG recordings in terms of stimulation length in single- and multi-electrode settings.

4.1. High-quality ECoG signals allow for accurate single-electrode SSVEP decoding

The superior decoding accuracy of ECoG was most striking for the single-electrode analysis, as even the naive classifier surpassed 90% classification accuracy for stimulation lengths as short as 0.75 s. However, when adopting even shorter stimulation length (≤ 0.5 s), both state-of-the-art classifiers, and fbCCA in particular, yielded noticeably higher accuracies compared to the naive classifier, e.g., for the second session, we obtained 92.2 and 85.6% for fbCCA and stBF, respectively, compared to 72.2% for the naive classifier at a stimulation length of 0.5 s. In comparison, when using scalp-EEG, both state-of-the-art classifiers outperformed the naive classifier for all stimulation lengths, but required much longer stimulation times (more than 3 s) to reach 90% accuracy. The fact that the naive classifier attains high accuracy levels on ECoG supports the claim that subdural signals are of better quality. Figure 5 confirms this as the single-trial signal-to-noise ratio (SNR) of the subdural signal (at electrode 36) is considerably higher than that of its scalp-recorded counterpart (at electrode Oz).

Figure 5

Signal-to-noise ratio's for all considered frequencies (A) scalp-recorded EEG and (B) ECoG.

In order not to interfere with the patient's clinical workup, we were not able to obtain simultaneous scalp-EEG from the patient. However, even in case simultaneous scalp-EEG was obtained, it is unclear whether the decoding accuracy would be useful for comparison, as several studies have presented initial evidence that the implanted silastic layer containing the subdural electrodes considerably attenuates the scalp-recorded potentials (Tao et al., 2005; Lanfer et al., 2013; Ramantani et al., 2014): a 2-3 times attenuation for a 4 × 8 grid and up to 8 times for an 8 × 8 grid (von Ellenrieder et al., 2014). As an alternative, we estimated the patient's scalp-EEG decoding accuracy from the ECoG phase variability using a data imputation exercise based on the fact that volume conduction affects the EEG signal's amplitude, not its phase. In the same vein, the head-models used in EEG source localization studies are resistive and therefore do not account for phase information (Sarvas, 1987; Hallez et al., 2007; Wolters and de Munck, 2007).

The high regression value for the naive classifier with respect to the scalp-EEG phase deviation (Figure 3) indicates that this classifier is mainly influenced by the consistency of the phase responses. However, despite that the variability of the subdural phase responses are not significantly different to those of the control group, the subdural decoding accuracy is considerably higher. This discrepancy in accuracy also shows up in the imputation analysis (Figure 3). We suggest that the higher decoding accuracy is due to the higher SNR of ECoG signals (Figure 5).

As for the naive classifier, the decoding accuracies of stBF and fbCCA are reduced when the phase deviation increases (Figure 3B). However, as the latter are more advanced classifiers, their regression coefficients are lower, indicating they are able to partially account for the trial-by-trial phase variability.

4.2. ECoG-based SSVEP decoding does not benefit from a multi-electrode approach

Assuming that information from other channels improves the classification accuracy, many researchers (for review Liu et al., 2013) developed spatial filtering and multi-electrode decoding strategies, including the ones considered in this study. When adopting a multi-electrode approach, the scalp-EEG decoding accuracies improve considerably for both classifiers, with an increased accuracy (averaged over all stimulation lengths) between 13 and 23%, depending on the considered session. In contrast, for ECoG, the decoding accuracy increased only marginally (~ 5%) with stBF, and even less for fbCCA, for which the greedy electrode selection did not include any additional electrode, as the accuracy could not be improved (with exception of the first session where 1 additional electrode was included). Overall, when opting for a multi-electrode approach, the accuracies obtained from ECoG and EEG were similar as soon as the stimulation length exceeded 1 s, demonstrating the effectiveness of current state-of-the-art SSVEP decoding algorithms.

In our previous work (Wittevrongel et al., 2018), we showed that the representation of the foveal flickering was highly localized at the posterior part of the primary visual cortex (electrodes 35 and 36), with little variation across the stimulation frequencies used. Also in the single-electrode analysis (Figure 2), the highest accuracies were obtained with the electrodes covering the posterior part of the primary visual cortex. The inclusion of additional subdural electrodes in the analysis did not improve decoding accuracy, indicating that the electrodes over the posterior part of the primary visual cortex carry the information needed for target decoding, and that there is little overlap with the information provided by the surrounding electrodes. This is in line with the literature that, compared to scalp-recorded potentials, cortical activations (both induced and noise) are considerably more localized (Buzsáki et al., 2012). As volume conduction attenuates and disperses the cortical activity over a larger scalp area, multiple scalp-electrodes carry information from the activity-of-interest. A multi-electrode analysis is therefore beneficial for decoding SSVEPs from the scalp, as it includes multiple weaker sources of information about the activity-of-interest.

4.3. Comparison with other visual BCI paradigms

ECoG has been hailed for BCI purposes for its long-term stability, superior signal quality and permanent availability of the interface (no electrode mounting required) (Wang et al., 2013; Vansteensel et al., 2016), but the reported applications rely on multiple electrodes, implanted as grids or strips, requiring a major surgical intervention. For instance, when pursuing a motor imagery application, one typically relies on a multi-electrode implants covering over the sensorimotor cortex to decode actions of different limbs, the activity of which is highly localized (Miller et al., 2010). Visual Event-Related Potential (ERP) paradigms have shown high accuracies in multi-electrode ECoG settings, but the accuracy considerably drops when adopting a single-electrode analysis (100% with 4 electrodes and 3 stimulus repetitions compared to 78% with a single electrode; Brunner et al., 2011). As to the cVEP paradigm, a multi-electrode analysis using CCA as spatial filter showed an accuracy of 86.8% with a stimulation time of 1.05 s (Kapeller et al., 2013). Compared to the paradigms mentioned above, as we have shown in this contribution, SSVEP stands out due to its high single-electrode decoding accuracy with short stimulation times (96.7% with a stimulation time of only 0.75 s). This could even be reached by using naive classifier based on the Fourier transform, without tailoring the frequency or phase characteristics to the subject. If the stimulus characteristics were optimized, the decoding performance could likely be further increased (Chen et al., 2015b).

4.4. Practical considerations and future work

Another important point in the comparison of scalp-EEG and ECoG accuracies is the conditions in which the recording was performed. Despite the impressive stimulation length/accuracy ratio we obtained with ECoG, it is worth mentioning that in most sessions the patient was unable to reach maximal accuracy (i.e., 100%), even with longer stimulation lengths. While some EEG subjects did reach maximal accuracy, it should be noted that our control subjects performed the experiment in a dedicated experimental room without distractions (sound-attenuated, air-conditioned, black walls), while the patient performed the experiment in a hospital room where many distractions were present (e.g., the experiment was interrupted several times to ensure normal continued medical treatment, surrounding relatives, and so on). Despite these suboptimal recording conditions, our results clearly demonstrate the superiority of ECoG compared to scalp-EEG for the SSVEP paradigm, both in terms of number of electrodes and stimulation lengths.

While the patient did not have a seizure during the experimental session, interictal activity might influence the recorded signals. However, the clinical functional mapping showed that none of the electrodes over V1 overlapped with the clinically determined ictal onset zone, which was located lateral from V2, indicating that the primary visual cortex of this patient can be considered healthy, normal tissue. The fact that the variability in the measured phase responses of ECoG and scalp-EEG was not significantly different and that the imputation analysis returned accuracies in the expected range provide further confidence that our results would generalize to other subjects and that there was no influence of the epileptic tissue on our results.

Given these encouraging findings, although based on offline analysis, we believe that our results could also pertain to an online implementation: the used state-of-the-art decoding algorithms have already been shown to also yield superior online decoding accuracy when based on scalp-EEG (Chen et al., 2015b; Wittevrongel and Van Hulle, 2016c).

4.5. Limitations

For this study, we recorded one subject implanted with a subdural grid over the occipital cortex. While it is generally unclear whether or not the results from a single subject generalize to the population, the current study provided an unique opportunity to study the cortical activations of the widely adopted SSVEP paradigm. As mentioned before, the prevalence of occipital lobe epilepsy is low. Only 5–10% of epilepsy cases have an epileptic generator in the occipital cortex (Sveinbjornsdottir and Duncan, 1993). Of this small group, only cases of intractable (and operable) epilepsy are candidate for intracranial implantation. Furthermore, one has to exclude photosensitive epilepsy patients, as SSVEPs are elicited by periodically flickering stimulation. Finally, as an alternative to electrode implantation, more and more surgeons currently prefer awake surgery during which they can perform functional mapping, thereby removing the need for electrode implantation. Given all these restrictions, it is very rare to study cortical activations in response to SSVEP stimulation in-vivo in a human subject, even more when the primary visual cortex is completely covered, including the interhemispheric fissure.

Unfortunately, as the hospital's EEG system was not set up for communicating real-time recordings to an external system, a closed-loop SSVEP experiment could not be performed.

5. Conclusion

The recording of brain signals directly from the cortical surface using electrocorticography (ECoG) has great benefits for BCI. However, the most performant visual BCI paradigm, based on the SSVEP, has never been assessed with ECoG, primarily due to the highly exceptional occurrence of subdural electrode implantation over the occipital cortex. In this work, we quantify the superior SSVEP decoding accuracies of ECoG compared to those of a control group with scalp-recorded electroencephalography (EEG), and show that, unlike with scalp-EEG, ECoG does not benefit from a multi-electrode analysis.

Data availability statement

The datasets for this manuscript are not publicly available as it contains sensitive information about the patient, and therefore cannot be transferred using online repositories.

Author contributions

EC, LD, AM, PB, and DV recruited the patient. BW and EK designed and implemented the experiment. BW, EK, LD, and EC collected the data. BW performed the analysis. All authors wrote and reviewed the manuscript.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.