Development of a flickering action video based steady state visual evoked potential triggered brain computer interface-functional electrical stimulation for a rehabilitative action observation game.

BACKGROUND: This study focused on developing an upper limb rehabilitation program. In this regard, a steady state visual evoked potential (SSVEP) triggered brain computer interface (BCI)-functional electrical stimulation (FES) based action observation game featuring a flickering action video was designed.
OBJECTIVE: In particular, the synergetic effect of the game was investigated by combining the action observation paradigm with BCI based FES.
METHODS: The BCI-FES system was contrasted under two conditions: with flickering action video and flickering noise video. In this regard, 11 right-handed subjects aged between 22-27 years were recruited. The differences in brain activation in response to the two conditions were examined.
RESULTS: The results indicate that T3 and P3 channels exhibited greater Mu suppression in 8-13 Hz for the action video than the noise video. Furthermore, T4, C4, and P4 channels indicated augmented high beta (21-30 Hz) for the action in contrast to the noise video. Finally, T4 indicated suppressed low beta (14-20 Hz) for the action video in contrast to the noise video.
CONCLUSION: The flickering action video based BCI-FES system induced a more synergetic effect on cortical activation than the flickering noise based system.

Introduction

Stroke is the main cause of neurological disability and restricted body movements [1]. Previous studies on motor recovery have indicated that the most rapid recovery occurs during the first few weeks after a stroke [2]. Several clinical experiments have indicated that repetitive rehabilitation exercises such as ‘sitting down and up on a chair’ and ‘arm exercise using a ball’ are crucial for motor learning and recovery of stroke patients [3, 4, 5]. Further, rehabilitation treatments tend to focus on restoration of lost motor functions. However, patients with severe motor impairment have difficulty to perform repetitive rehabilitation exercises which may lead to successful recovery.

Treatments based on activation of the mirror neuron system (MNS) can facilitate brain plasticity in patients who have severe disability [6, 7, 8, 9]. The MNS is a neural mechanism related to the ability of understanding and imitating the actions of another person [10, 11, 12, 13, 14]. Specifically, mirror neuron activation appears to induce a response similar to actual exercising in the motor cortex when an action is observed [9, 15]. Activation of the MNS is indicated by suppression of the Mu band (8 to 13 Hz) in the sensorimotor cortex [16, 17, 18, 19, 20, 21]. Mu suppression is generated by desynchronization of the cells in the sensorimotor cortex during action observing, exercising, and imaging [16, 17, 18, 19, 20]. Therefore, patients unable to perform movement can experience stimulation of the motor cortex and learn the motions by watching an action video.

In this study, we developed a steady state visual evoked potential (SSVEP) triggered brain computer interface (BCI)-functional electrical stimulation (FES) based action observation game featuring a flickering action video for stroke patients who were unable to move their upper limbs. Although it may cause tiredness due to flickering stimuli, a SSVEP-based BCI system has advantages such as no need training and higher classification accuracy compared to a motor-imagery based BCI system [22]. In stroke patients who have difficulties performing motor-imagery, the SSVEP triggering system can be more useful to induce repetitive movement through FES. The FES is an effective method for assisting and improving motor functions in rehabilitation [23, 24]. It has been used as a neuroprosthetic, which generates functional movement of paralyzed muscle such as grasping, standing, and walking [25, 26]. The therapeutic application of FES has benefits of increasing muscle strength and endurance, and reducing muscle spasticity [23, 27, 28, 29]. Furthermore, it induces motor cortex activation through repetitive movement exercise [30, 31, 32, 33, 34]. In previous studies, the effect of FES was enhanced when applied in combination with other therapies than when applied alone [35, 36, 37].

Utilization of FES while watching the action video can be expected to stimulate the MNS synergistically to consequently improve rehabilitation effects in patients who cannot control the upper limbs. Synchronization between arm movement in the action video and real arm movement by the FES can significantly activate the motor cortex to enhance rehabilitation effects [38]. Therefore, synergetic effects of the flickering action video-based BCI-FES system need further investigation. In this regard, this study conducted experiments to compare an action video which activated the MNS and a noise video which could not. Subsequently, differences in brain activation in response to the two conditions were examined. It should be noted that this experiment recruited healthy subjects to evaluate the feasibility of the program. The results are likely to advance understanding of faster and improved rehabilitation.

Methods

Subjects

The experiment was conducted on 11 right-handed healthy subjects aged between 22 and 27 years. Subjects who consumed coffee and cold medication were excluded. All subjects were first provided with the instructions regarding the experiment, providing information about the experiment, the purpose of the study, and the data protection policy and signed the written consent form for this experiment (IRB no. DSMC2019-03-018-002).

Instruments used in the experiment

In this study, an electroencephalogram (EEG) acquisition device (DSI-24; WEARABLE Sensing, San Diego, USA) was utilized to obtain EEG signals and an FES device (Microstim; Medel GmbH Inc, Hamburg, Germany) was utilized to electrically stimulate the wrist extensor muscles. A personal computer analyzed the EEG signal and ran the game contents in real time. The DSI-24 comprised 21 electrodes which were placed according to the 10–20 system (Fig. 1). The classification and analysis of EEG signals were based on 19 channels (Fp1, Fp2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, and O2). Further, the Microstim had two electrodes to provide electrical stimulation pulses and induce muscle contractions (Fig. 1). The stimulation level was adjusted to avoid strain on the muscles.

Figure 1.

The experiment utilized an EEG acquisition device (DSI-24) and a functional electrical stimulation (FES; Microstim) device. The game program indicated a flickering action video at 15 Hz. The DSI-24 received steady state visual evoked potential (SSVEP) values which were transferred to the computer. If SSVEP was greater than the standard, the FES turned on and transmitted electrical stimulation to the electrodes; else, the FES turned off and did not trasmit electrical stimulation to the electrodes.

Experimental conditions

Two conditions were established in this experiment: (i) flickering action video condition (Fig. 2c) where the upper limb of the subject was periodically raised and lowered in the video area of the game and (ii) flickering noise video condition (Fig. 2d) where the brightness and contrast of the video matched those of the action video. It should be noted that the game was played in the same manner for the two conditions.

Figure 2.

(a) Training program screen and (b) game program screen during the experiment. Two types of videos were shown, i.e., (c) arm exercise scene and (d) noise scene.

Classifier training session

The training session involved subjects watching the flickering action video (Fig. 2c) or the flickering noise video (Fig. 2d) for a duration of 3 minutes to subsequently classify the SSVEP. The SSVEP would peak dominantly for the O1 and O2 channels of the occipital lobe while watching the flickering videos. It should be noted that peak SSVEP was observed at 15 Hz if the subjects watched the flickering action video at 15 Hz. Therefore, the presence of SSVEP was utilized to ascertain if the subjects were watching the video or not.

Each training session had seven trials and each trial comprised a block characterized by watching a white crosshair image with black background for 10 sec and subsequently watching the noise or action video for 15 sec (Fig. 3a). Finally, the measured EEG signal was transmitted to design an SSVEP distinguishing classifier. There was no FES stimulation during the training session.

Figure 3.

Training and experimental process, i.e., watching the video for 15 sec after watching the crosshair-screen for 10 sec. This was repeated for (a) 7 trials during the training process and (b) 12 trials during the experiment. (c) The entire experimental process.

Classifier design and classification process

Individual classifiers were designed with OpenVIBE software (ver. 1.2.0) (INRIA, Rennes, France) to distinguish between ‘watching the flickering video’ and ‘rest state’ of subjects. Common spatial pattern (CSP) filters and classifiers were designed using training data obtained for the 19 channels (Fp1, Fp2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, and O2). For this, the EEG data for two groups (1: action video or noise video; 2: rest) were used to design CSP filters and classifier for each group. The signal obtained for the 19 channels and transmitted through a temporal filter with SSVEP and mu ranging from 14.5–15.5 and 8–13 Hz, respectively. Subsequently, two CSP filters were created using the filtered signal so that it could maximize the variance of the signal between two states. As a next step, the original signal pass thorough again the temporal filter and the CSP filters followed by estimation of the power average for the two frequency bands to generate feature vectors. Using these feature vectors, the classifier was designed to distinguish the ‘watching video state’ from the ‘rest state’. After designing the CSP filters and the classifier, the CSP filters and classifiers was applied to thr real-time system. The real time data pass through the signal processing pipeline which was composed of the temporal filtering and the CSP filters, and finally the classifier gave command to control FES activation and game playing.

Game program design

The game program featured a farmer plucking apples by raising the arm and a flickering video area adjacent to the farmer (Fig. 2b). The video play and raising of the farmer’s arm occurred synchronously if a detectable SSVEP was evoked while watching the flickering video. Subsequently, the FES device provided electrical stimulation to induce wrist extension movements in the subjects. In particular, the upper limb in the video was raised from 0–2.25 sec (out of 3.75 sec) when SSVEP was detected for the entire duration of the action video. Subsequently, the farmer ‘plucked’ the golden apple and rested the arm for the remainder experiment at which point the subject’s wrist returned to resting position. It should be noted that if no SSVEP was evoked during the game play, the farmer’s arm would be lowered and the FES would be switched off. Consequently, the subject’s wrist would remain in the resting position and the farmer would not obtain the apple. The same process was followed for the noise video.

Experimental procedure

In this study, the subjects were instructed to sit on a chair in front of a desk while resting their arms on the table. One set of surface electrodes positioned on extensor carpi radialis (ECR) muscle of the dominant arm. The cathode was placed over the motor point of ECR muscle, and the anode on the surface of the forearm distal to cathode. The amplitude was set to produce wrist extension movement, and was adjusted to minimize discomfort during stimulation. They were further instructed to not move their hands, feet, and heads during all processes except the rest time. As indicated in Fig. 3c, first, the resting state EEG was obtained over a duration of 3 min while the subjects sat in a relaxed manner with their eyes open. Subsequently, the classifier training session was conducted with the action and noise video conditions (3 minutes each). A classifier was generated after concluding the training session. The experiment comprised running one of the action or noise video condition for 5 min followed by rest for 5 min without measuring the EEG signal. Finally, the game program was conducted by playing the other video for 5 min. The order of providing the action and noise videos was counter balanced. The training and experimental sessions had 7 and 12 trials, respectively. Each trial comprised watching a white crosshair image with black background for 10 seconds followed by watching the noise or action video for 15 seconds. There was no FES stimulation during the training session. The total experimental time was approximately 30 minutes. Great care was taken to ensure that subjects were participating in the experiments. The subjects were asked if they had any adverse effects including eye discomfort, dizziness, and headache during each experiment session.

EEG analysis

In this study, the EEG signal were obtained from 19 channels (Fp1, Fp2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, and O2) . However, only T3, T4, C3, C4, P3, and P4 were analyzed to investigate the effects on motor function. Data analysis was performed using MATLAB (MathWorks, Natick, USA). The EEG data were sampled at a rate of 256 Hz. Further, EEG data filtered with band pass filtering ranged from 5 to 30 Hz and electrooculography (EOG) was removed using the automatic artifact removal algorithm (Mathematics Department, Macquarie University, Sydney, Australia). The mean of epoch data was made zero by demean process which is subtracting the average value of the raw data of the epoch. Subsequently, the analysis utilized epochs ranging from 70 to 70 uv to avoid transfer to epochs characterized by high peaks. The epoch for the short-time Fourier transform (STFT) ranged from 1 sec before the video starts to the end of the video. A single trial comprised playing the white crosshair with black background for 10 sec followed by playing the action video for 3.75 sec. This block was repeated four times. Further, 1 to 0 sec was always brought at 9 to 10 sec of white crosshair with black background. Subsequently, the STFT was performed with Hamming window to obtain the magnitude of temporal frequency. The STFT values were averaged from 1 sec before the start image to the start image. Further, the average values were divided by STFT values obtained at 3.75 sec. Finally, the values were log transformed.

Statistics

Paired -test was conducted to compare spectrum of the measured EEG data between action video and noise video condition using function test in MATLAB (Mathworks, Natick, USA).

Results

All subjects completed the experiment with no adverse events. The classifying accuracy for our classifier for flickering action video and noise video was 93.51 6.96% and 91.08 7.36% respectively when a 10-fold cross validation test was conducted. Figure 4 presents the results of the frequency analysis for action and noise videos played during the game program session. The graphs indicate log values of the EEG signal with frequency ranging from 5 to 30 Hz. Red represents the action video while blue represents the noise video. Paired-t test was performed for identifying significant differences. Further, ranges less than the significant threshold ( 0.05) have been represented as blue lines under the graphs. The results indicated a suppressed Mu band for the action video than the noise video for T3 and P3. Further, T4, C4, and P4 indicated an augmented high beta band for the action video than the noise video. In contrast, T4 indicated a suppressed low beta band for the action video than the noise video.

Figure 4.

Frequency components obtained from channels (a) T3, (b) T4, (c) C3, (d) C4, (e) P3, and (f) P4 while watching action video (red) and noise videos (blue) of the game program. Paired-t test was performed for identifying significant differences. Significant ranges ( 0.05) have been indicated as blue lines under the graphs.

Discussion

This study investigated the effects of a rehabilitation game based on action observation and FES. Flickering action video could evoke the activations of the MNS and provide highly detectable SSVEP responses simultaneously. It could be contrasted to simple motor imagery paradigm which could reveal lower detection rate and more individual variability than SSVEP based control paradigm, even though motor imagery could evoke the activations of the MNS. In addition, the flickering action video combined with FES could be expected to reveal synergetic effects on the motor learning mechanism. In this regard, differences in brain activation due to flickering action and flickering noise videos were determined using BCI based FES.

The T3 and T4 electrodes were located over the temporal lobe which processes information from memory, sound, smell, and taste [39]. The primary function of the temporal lobe is to process auditory signals to aid formation of long-term memories and processing of new information [39]. In this study, T3 indicated a suppressed Mu band for the action video than the noise video (Fig. 4). The neural substrates responsible for the elaboration of new motor patterns includes the key centers of the mirror neuron system [9]. Therefore, action observation induced Mu suppression can activate the MNS to play an important role in rehabilitation therapy of patients [40, 41].

Furthermore, T4 indicated a suppressed low beta band (14 to 20 Hz) and an augmented high beta band (21 to 30 Hz) for the action video. Beta band indicates an active state of awakening or consciousness and logical thinking [39, 42]. Beta band can be divided into two bandwidths attributed to reactivity differences in tasks [42]. Low beta is generally associated with quiet, focused, and introverted concentration while high beta is related to significant stress, anxiety, high energy, and high arousal [39]. As indicated by the results, T4 was characterized by low beta suppression and high beta increase during the action video condition, thus suggesting that the subjects were more immersed in and focused on the game. These results indicate that action videos can aid active participation during BCI-FES programs. Additionally, the action video indicated high beta suppression (27–28 Hz) in T4. One can argue that the action video might lead to severe stress or excessive immersion. However, the range of significant difference between tow conditions was too narrow.

The C3 and C4 electrodes were located on the motor cortex of the lateral and medial surfaces of the cerebral hemisphere [43]. These are important areas for motor recovery. Therefore, the upper limb rehabilitation effect was mainly observed in the supplementary motor area (SMA), the premotor cortex, and the primary sensorimotor cortex [44, 45]. The C3 channel indicated Mu suppression under both conditions; however, the differences were not significant. This result could be attributed to a larger motor cortex activation by the FES than the difference between the action and noise videos. It should be noted that the non-significant differences between the two conditions (Fig. 4) could be attributed to the severe deviation caused by the FES [46]. However, T3 and P3 indicated significant differences since they were located relatively far from the center of the motor cortex and were less likely to be affected by the FES. The FES likely resulted in Mu suppression in the left hemisphere and high beta augmentation in the right hemisphere. The subjects were stimulated by attaching the FES device on their right arms, thus suggesting that the effect of the FES was likely stronger in the left hemisphere. Further, C4 indicated augmented high beta for the action video than the noise video. Therefore, the subjects were more focused while watching the game program.

The P3 and P4 channels were located on the parietal lobe which is involved in several complex tasks, sensory control, visual perception, movement, cognition, and information processing [39, 47, 48]. Further, the parietal lobe is closely linked to the occipital lobe [49]. Previous clinical studies indicated activation of the SMA and posterior parietal regions when a fake hand was threatened with a needle [50, 51, 52]. In this study, P3 indicated a suppressed Mu band for the action video. This could be attributed to the response of the parietal lobe which recognizes objects and accepts sensory information from observations.

Conclusions

We explored electrical brain activity during performance of flickering action video based BCI-FES system, how electrical activity may different compared with noise video condition. The results indicated larger cortical activation and high beta attenuation of action video based BCI-FES system compared with noise video. The results provide preliminary evidence of possibility of rehabilitation game based on utilizing the action video with FES in neurorehabilitation field in the future. Further clinical trial and application for patient are needed.