Comparison of Visual Stimuli for Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces in Virtual Reality Environment in terms of Classification Accuracy and Visual Comfort.

Recent studies on brain-computer interfaces (BCIs) based on the steady-state visual evoked potential (SSVEP) have demonstrated their use to control objects or generate commands in virtual reality (VR) environments. However, most SSVEP-based BCI studies performed in VR environments have adopted visual stimuli that are typically used in conventional LCD environments without considering the differences in the rendering devices (head-mounted displays (HMDs) used in the VR environments). The proximity between the visual stimuli and the eyes in HMDs can readily cause eyestrain, degrading the overall performance of SSVEP-based BCIs. Therefore, in the present study, we have tested two different types of visual stimuli-pattern-reversal checkerboard stimulus (PRCS) and grow/shrink stimulus (GSS)-on young healthy participants wearing HMDs. Preliminary experiments were conducted to investigate the visual comfort of each participant during the presentation of the visual stimuli. In subsequent online avatar control experiments, we observed considerable differences in the classification accuracy of individual participants based on the type of visual stimuli used to elicit SSVEP. Interestingly, there was a close relationship between the subjective visual comfort score and the online performance of the SSVEP-based BCI: most participants showed better classification accuracy under visual stimulus they were more comfortable with. Our experimental results suggest the importance of an appropriate visual stimulus to enhance the overall performance of the SSVEP-based BCIs in VR environments. In addition, it is expected that the appropriate visual stimulus for a certain user might be readily selected by surveying the user's visual comfort for different visual stimuli, without the need for the actual BCI experiments.

1. Introduction

Electroencephalography (EEG) has been the most widely used neural signal for brain-computer interfaces (BCIs), whose main aim is to provide the paralyzed or disabled with new means of communication with the external environment [1]. Typical paradigms for EEG-based BCIs include motor imagery (MI), P300, and steady-state visual evoked potential (SSVEP) [2]. Among these, an SSVEP-based BCI paradigm has been widely employed because of its robustness to external noises and very little training requirement [3]. Owing to its advantages over the other paradigms and recent development of advanced analysis methods [4, 5], the SSVEP-based BCIs have been implemented for a variety of applications including assistive and rehabilitation tools for the disabled [6] and practical applications for the healthy, such as car navigation [7] and entertainment [8]. Furthermore, with the rapid advancements in the virtual reality (VR) technology, the SSVEP-based BCIs have been successfully applied to VR applications with hand-free control of the VR objects or speechless communications [9-11].

Although most VR devices currently employ head-mounted displays (HMDs), no previous SSVEP-based BCI study has considered the environmental differences between the VR-HMDs and conventional LCD monitors. Since the traditional SSVEP-based BCIs have used an LCD monitor as the rendering device to present visual stimuli for the past decades, a number of studies have already been conducted on the influence of the various parameters of this visual stimulus on the performance of the BCIs; these parameters include spatial frequency [12], temporal frequencies [13], colors [14], data recording channels [15], and time window sizes [16, 17]. On the contrary, the SSVEP-based BCIs implemented in VR environments have employed visual stimuli identical to those used in conventional LCD monitor environments, without any major modification. In other words, all SSVEP-based BCI studies performed in VR environments assumed that the presentation of visual stimuli on HMD is not significantly different from that on an LCD monitor. For example, MindBalance game [9], a 3D video game using SSVEP-based BCIs in VR environments, employed pattern-reversal checkerboard stimulus (PRCS) to elicit SSVEP response. A recently developed neuro-optical diagnostic tool using the VR headset [18] also employed the conventional PRCS. However, it is well known that an experiment in the VR environment is highly vulnerable to visual fatigue than that in the LCD environment; this is mainly due to the image distortion, or crosstalk, in the stereoscopic viewing [19] as well as the proximity between the source of illumination and the eyes [20].

In the present study, we have used two different types of visual stimuli—PRCS and grow/shrink stimulus (GSS)—both of which are known to effectively elicit SSVEP responses in the LCD monitor environment, on 14 participants wearing HMDs. The performance of the two representative visual stimuli was then investigated in terms of individual classification accuracy and subjective visual comfort scores. After the survey of the visual comfort of the participants in the preliminary offline experiments, the performance of SSVEP-based BCIs was investigated through online avatar control experiments in a VR environment.

2. Materials and Methods

2.1. Participants

Sixteen young, healthy people (10 males and 6 females, aged 20.5 ± 1.6 years) with normal or corrected-to-normal vision participated in our experiment. All participants were informed of the details of the experiments and had given their written consent. The data of two participants were excluded in further analyses: the first was excluded owing to the frequent blinking of the eyes during the presentation of the visual stimuli (eye blinks contaminated 14 out of the total 40 trials) and the second owing to the nonexistence of spectral peaks in the recorded EEG. The eye blinks were identified by visually inspecting vertical electrooculogram (EOG) recorded during the offline experiment. This so-called “BCI-illiteracy” is a well-known issue in EEG-based BCIs [21]. This experiment was approved by the institutional review board of Hanyang University, Republic of Korea (IRB HYI-14-167-11).

2.2. Visual Stimuli

Two different types of visual stimuli were employed to elicit SSVEP responses: a PRCS and a GSS. The PRCS is a traditional visual stimulus, which is used most frequently to elicit SSVEP responses in LCD monitor environments; this stimulus alternately presents two checkerboard patterns with 180° phase difference [7] (Figure 1(a)). The GSS is a new visual stimulus that changes both luminance and size to elicit SSVEP responses. This stimulus was based on previous studies, which reported that motional changes can also elicit periodic VEP responses (often referred to as steady-state motion visual evoked potential or SSMVEP) [22, 23] (Figure 1(a)). These stimuli were presented in a VR environment using an HMD of the HTC VIVE™ VR system (HTC Co., Ltd., Xindian District, New Taipei City, Taiwan). Both visual stimuli were modulated to elicit SSVEP responses corresponding to four frequencies, namely, 6, 7.5, 9, and 10 Hz. These frequencies were determined by considering the refresh rate of the rendering device (90 Hz), which is an integer multiple of each of the four target frequencies. In the offline experiments, the visual angle of the PRCS was fixed at 14°, while that of the GSS was varied between 8° and 16°. In the online experiments, the visual angle of the PRCS was reduced to 6° and that of GSS was varied between 4° and 8° in order to validate the feasibility and usability of the visual stimuli in a realistic VR environment in which large-sized stimuli cannot be generally employed. Note that according to previous reports, visual stimuli with visual angles greater than 3.8° would produce similar levels of SSVEP responses [24].

Figure 1

(a) Overall timeline for a single trial in offline experiments. In each trial, after a 2 s instruction period to inform participants of the location of a target visual stimulus, four visual stimuli, each of which was either PRCS or GSS, were presented for 4 s. (b) Screenshots of online experiments. The left picture was taken when the PRCS was employed, while the right picture was taken when the GSS was employed. The human avatar needed to move along a designated path (e.g., in the left direction in both figures).

2.3. Experimental Paradigm

In the preliminary offline experiments, each stimulus type was presented in a randomly shuffled order to each participant. In each trial, four visual stimuli with different frequencies were presented for 4 s, as shown in Figure 1(a). The interstimulus interval (ISI) was set to 2 s, during which one of the numbers presented on the screen was colored green and flickered at 1 Hz to indicate the stimulus that the participant should focus on during the next stimulus interval. Each visual stimulus in each stimulus type appeared 20 times (five times for each frequency), and thus, the total number of trials was 40. The EEG signals were recorded; however, no immediate feedback was delivered to the participants during the experiment. At the end of the preliminary offline experiment, the participants were asked to subjectively rate their visual comfort with the two stimulus types on a scale ranging from 0 (very uncomfortable) to 10 (very comfortable).

In the online experiments, the participants who also participated in the preliminary offline experiments were asked to control a human full body avatar standing on a virtual road in a VR environment. The avatar could move in four directions: top, bottom, left, and right. Four visual stimuli with the frequencies used in the offline experiment were presented at the top, bottom, left, and right of the avatar to indicate the possible movement directions of the avatar (Figure 1(b)). Each participant was asked to sequentially move the avatar in a correct direction following the given path. A total of three different paths, each consisting of 20 movement steps, were created. For all 60 movement steps, the numbers of each directional step were counterbalanced. For each path, the same paradigm was repeated twice with either PRCS or GSS, when the presentation order of the visual stimuli was randomly determined for each participant. The avatar could move a step forward only when the classification result (direction) coincided with the correct direction of the path. Consequently, the minimum number of trials required to complete each session was 20, when the classification accuracy was 100%. Each trial lasted for 5 s, including 2 s for the presentation of the visual stimuli, 1 s for avatar's movement, and 2 s for ISI to give participants the time to shift their gaze for the next movement. A video clip showing the online experiment of a participant is attached to this manuscript as a Supplementary Movie, and its high resolution version can be found at YouTube™ (https://youtu.be/TC4QMPhW6y8).

2.4. Biosignal Acquisition and Preprocessing

The EEG data were recorded from seven electrodes (Cz, PO3, POz, PO4, O1, Oz, and O2) using a commercial biosignal recording system (ActiveTwo, BioSemi, Amsterdam, and the Netherlands). In addition, a pair of electrodes was attached above and below the right eye to acquire the vertical EOG data. The sampling rate was set at 2,048 Hz. The recorded EEG data were re-referenced to Cz [4, 25] and then band-pass filtered at 6 and 50 Hz using a zero-phase Chebyshev type I infinite impulse response filter implemented in MATLAB (MathWorks, Inc., Natick, MA, USA). The program to analyze data in real time was developed using the FieldTrip toolbox [26].

2.5. Data Analysis and Statistical Analysis

For the classification of the SSVEP responses, we adopted a recently introduced algorithm called the extension of the multivariate synchronization index (EMSI) [5], which has exhibited outstanding performance compared to the conventional classification methods [27].

The Wilcoxon signed-rank test was employed for the statistical analysis because the classification accuracies with respect to the two visual stimulus types did not follow normal distribution as assessed by the Kolmogorov–Smirnov test.

3. Results

In the offline experiment, the GSS outperformed the PRCS in both classification accuracy and information transfer rate (ITR) for all window sizes (Figures 2 and 3); ITR was calculated as follows:where N denotes the number of stimuli, P denotes the classification accuracy ranging from 0 to 1, and C denotes the time needed to classify a single trial [28]. Statistical analysis using the Wilcoxon signed-rank test also showed statistically significant difference in the performance of the GSS and PRCS (Bonferroni-corrected p < 0.005 for both classification accuracy and ITR for all window sizes). Although a window size of 1.5 s showed the highest ITR (Figure 3), 2 s epochs were used for the classification in the online experiments. This was because the difference between the ITRs for the 1.5 s and 2 s epochs was not big, but the improvement in the classification accuracy was relatively distinct for the 2 s epoch compared with the 1.5 s epoch.

Figure 2

Comparison of offline experimental results between the PRCS and GSS in terms of the average classification accuracy across participants. The error bars indicate the standard errors. p < 0.005.

Figure 3

Comparison of offline experimental results between the PRCS and GSS in terms of the average ITR across participants. The error bars indicate the standard errors. p < 0.005.

Table 1 shows the classification accuracy of each participant in the online experiment. Unlike the preliminary offline experiment, no statistical significance was observed between the classification accuracies for the PRCS and GSS (p=0.424; Wilcoxon signed-rank test) in the online experiment although the average classification accuracy for the GSS was higher than that for the PRCS. The possible reasons for the difference between the two cases, i.e., the offline and online experiments, will be discussed in Discussion.

Table 1

Comparison of online classification accuracies between the PRCS and GSS.

Group	Participant	PRCS accuracy (%)	GSS accuracy (%)
Group 1	P6	84.5	58.8
	P8	92.3	85.7
	P10	98.4	92.3
Group 2	P2	100	100
	P3	66.7	98.4
	P5	96.8	76.9
	P7	100	98.4
	P11	89.6	98.4
	P12	80.0	80.0
	P13	84.5	95.2
Group 3	P1	70.6	89.6
	P4	84.5	90.9
	P9	75.9	92.3
	P14	83.3	96.8
Average ± std.		86.2 ± 10.7	89.6 ± 11.3

Group 1 includes participants who rated PRCS as more comfortable to their eyes than GSS. Group 2 includes participants who rated GSS as more comfortable than PRCS. The remaining participants who gave the same score to both stimuli are categorized as Group 3.

For further analyses, all participants were divided into three groups based on the subjective visual comfort ratings for the two visual stimulus types that were obtained right after the preliminary offline experiment. The participants who were more comfortable with the PRCS were categorized as Group 1, and those who were more comfortable with the GSS were categorized as Group 2. The participants who rated both stimuli equally were categorized as Group 3 and excluded from further analyses. Interestingly, all three participants (i.e., P6, P8, and P10) in Group 1 exhibited higher classification accuracies for the PRCS than for the GSS, while most participants in Group 2, with the exception of only one participant (i.e., P5), exhibited higher or equivalent classification accuracies for the GSS than for the PRCS. These results suggest that the performance of the SSVEP-based BCIs in VR environments might be potentially improved by selecting the best stimulus type for each individual, which would be readily chosen by inspecting the individual's subjective visual comfort for different visual stimuli types.

4. Discussion

The performances of the reactive BCI systems are highly dependent on the types of stimuli used to elicit specific EEG responses. Although a series of studies has been performed to find an optimal visual stimulus for the conventional SSVEP-based BCIs in the LCD monitor environment, no study has yet been reported on the influence of visual stimuli on the performance of the SSVEP-based BCIs in VR-HMD environments. We hypothesized that the PRCS, which are widely used in the SSVEP-based BCIs, might not be the optimal visual stimulus in a VR-HMD environment because the images displayed on the HMDs are closer to the eyes than those on the LCD monitors, and thus, the PRCS might be too intense for the eyes. Therefore, in this study, we tested another type of visual stimulus called the GSS that changes both size and luminance in VR environments and compared the BCI performances with the PRCS.

In the offline experimental results, the GSS outperformed the PRCS in terms of classification accuracy; however, the difference in the performance was considerably reduced in the online experiments. This phenomenon is thought to originate from several factors: first, the spatial frequency of the PRCS in the offline experiment was different from that in the online experiment. The spatial frequency changed from 0.25 cycle/deg in the offline experiment to 0.5 cycle/deg in the online experiment. According to a previous report [12], spatial frequency of PRCS has close relationship with the performance of SSVEP-based BCIs. The second reason might be the difference in the background; for instance, in the offline experiment, a monotonous dark grey background was used, while in the online experiment, a relatively complicated background with many distractors was employed (Figure 1(b)). This complicated background might have distracted the elicitation of the SSMVEP because the border of the GSS sometimes becomes obscure owing to the background images. On the contrary, the PRCS would be less affected by the background because this stimulus maintains its size during the presentation.

Our online experiments demonstrated that the SSVEP-based BCI with a visual stimulus that was more comfortable for the user generally outperformed that with the other stimulus in VR environment. This finding is not in line with previous reports showing that a visual stimulus evoking stronger SSVEP responses induced the severer visual fatigue [29-31] when an LCD monitor was used for presenting visual stimuli. However, there are also some evidences showing that the relationship between visual comfort and BCI performance is dependent upon the stimulation rendering device (e.g., light emitting diodes: LEDs) or stimulus types (e.g., SSMVEP) [32, 33]. Our results also suggest that a user's optimal visual stimulus in VR environments might be readily determined by rating the subjective visual comfort of the user even before the main BCI experiment. This strategy might considerably alleviate the necessity of a series of offline BCI experiments to determine an optimal visual stimulus for the user in the VR environment.

In the offline experiment, four participants rated the same visual comfort score for both PRCS and GSS. Interestingly, they commonly achieved better classification accuracies in the GSS than in the PRCS. Although the limited sample size makes it hard to generalize, selecting GSS rather than PRCS might yield better classification accuracies in cases when there is no difference in the subjective visual comfort ratings. However, further investigations are required to formulate a more generalized rule for selecting the optimal visual stimulus for the SSVEP-based BCIs in VR environments. In addition, in the present study, we tested only two types of visual stimuli; however, more types of visual stimuli need to be developed and tested in VR environments in future studies.

5. Conclusions

To the best of our knowledge, this is the first study that has compared different types of visual stimuli for the SSVEP-based BCIs in VR environments. Our study demonstrated that selection of an optimal visual stimulus for an individual could improve the overall performance of the SSVEP-based BCIs and reduce visual fatigue in VR environment. A close association between the performance of the SSVEP-based BCIs and subjective visual comfort was observed, suggesting that the selection of an appropriate visual stimulus via a simple pre-experimental inspection of the individual's preference toward the visual stimuli might help to enhance the performance of the SSVEP-based BCIs in VR environments.