Quantitative EEG Signatures through Amplitude and Phase Modulation Patterns.

Cortical spatiotemporal signal patterns based on object recognition can be discerned from visual stimulation. These are in the form of amplitude modulation (AM) and phase modulation (PM) patterns, which contain perceptual information gathered from sensory input. A high-density Electroencephalograph (EEG) device consisting of 48 electrodes with a spacing of 5 mm was utilized to measure frontal lobe activity in order to capture event-related potentials from visual stimuli. Four randomized stimuli representing different levels of salient responsiveness were measured to determine if mild stimuli can be discerned from more extreme stimuli. AM/PM response patterns were detected between mild and more salient stimuli across participants. AM patterns presented distinct signatures for each stimulus. AM patterns had the highest number of incidents detected in the middle of the frontal lobe. Through this work, we can expand our encyclopedia of neural signatures to object recognition, and provide a broader understanding of quantitative neural responses to external stimuli. The results provide a quantitative approach utilizing spatiotemporal patterns to analyze where distinct AM patterns can be linked to object perception.

Introduction

Object saliency has been a crucial aspect of our survival and evolutionary process. The weighted values the brain places on objects have provided various degrees of neural responses, which eventually correlate to emotional, behavioral, and learned responses.[12] Appetitive (positive) or aversive (negative) responses to stimuli are learned responses, where conditioned stimulus is associated to our perception of the world.

Studies in positive/negative task experiments elicited different states of neural activity, where negative incentive-based tasks produced higher neural activity vs. positive incentive based tasks. The latter neural activity (positive based tasks) was influenced by perceptual processing stages, i.e. P300 vs P100 in V1.[3] Other studies featuring fear-related object saliency focused on the neural activity between visual cortical pathways and fast subcortical pathways. Functional magnetic resonance imaging was performed on patients with spider phobia and found strong aberrant functional connectivity of fast subcortical pathways when they are exposed to phobia-related stimuli.[4]

Previous work has focused on several methodologies to understand feature selection and object saliency. The concept of the saliency map proposed by Itti et al.[5] discusses how image features are decomposed into a series of topographical features. The location of these features is compared to the other features of the image to determine which area on the image presents a starker contrast to the rest of the image. This type of approach presents the idea that there are aspects of an image that generate higher attentional responses. This work leads to feature selection techniques by Hou and Zhang[6] which have demonstrated that the individual does not need previous dispositions of an object to perform object recognition. Their approach utilized log-spectrum of an input image to determine if there are features in an object that are starkly different than its background. Both approaches leads to fast detection of an object, but not necessarily object saliency. Additionally, both approaches do not rely on previous object learning to produce object recognition. Additional work in neural saliency processing has found attention-related visual pathways based on neurophysiology imaging. Saliency processing involves two partly segregated pathways for salience computation, namely the cortical and subcortical pathways. The subcortical pathways represent a visual saliency map where interactions in the subcortical circuit influence active populations of neurons.[7] Contrast changes and embedded noise in images influence neural activity and object perception. This type of interference causes differing levels of neural response saliency.[8]

Specific areas of the brain have been implicated in object recognition. Frontal lobe processing focuses on integrating spatial-object information from working memory.[910] Electrophysiological studies have found frontal lobe activation while an individual was focusing on an array of input items, that is, letters, numbers, objects, locations, and facial recognition. The medial prefrontal cortex has been shown to be active during emotion-related cortical processing. This is specifically the case found in emotionally salient stimuli, which includes induced emotions and memory retrieval.[11]

Classic neurological analysis has featured the averaging of event-related potentials to multiple trials of object stimuli. Through signal processing techniques, we can further isolate and find neural response incidents related to visual stimuli. Previous work by Freeman[12] isolated the amplitude and phase component of the Electroencephalograph (EEG) signal utilizing Hilbert transformations. According to Freeman, the analytic amplitude (AA) provides temporal patterns after sensory stimulation. Neuron populations can be expressed as a “wave packet” of synchronized activity after a stimulus is applied. The wave packet has an amplitude modulation (AM) pattern that relates to the axonal firing intensities of mesoscopic neural populations. When sensory input enters the cortex, neural activation rises above the basal noisy state. Neural intensity achieves a threshold that induces a phase transition denoting a new cortical state.[1314] Action potentials, as depicted as the peaks of Event-related potential (ERP) waveforms, can be quantified as spatial patterns of AM and phase modulation (PM). Classification of AM patterns can enable the discrimination of conditioned stimuli and perceptual processing.[151617]

We intend to provide a quantitative approach utilizing spatiotemporal patterns to analyze where distinct AM patterns can be linked to object perception. Our focus is on the frontal cortex due to its feature selection capabilities which have been tied to emotion-related cortical processing. In this manner, we accurately capture the neural responses to emotionally based object saliency.

Materials and Methods

The experiments for this project were held in the Computational Neurodynamics Lab at the FedEx institute of Technology at the University of Memphis. This study has been approved by the University of Memphis Institutional Review Board (IRB-071411-790). We utilized the electrode array prototype MINDO-48S-001AFF0900A7, produced by BRC/NCTU, Hsinchu, Taiwan.[18] MINDO-48S oversamples EEG + Electromyograph (EMG) by 48 closely spaced spring-loaded electrodes in a flexible curvilinear array that could be quickly fixed on the scalp of a volunteer in any orientation. The electrodes have a length of 3 mm and a diameter of 1 mm, and they have been developed based on MINDO's patented technology using a special golden alloy. MINDO-48S uses wireless transmission through Bluetooth to communicate with a laptop computer, which serves as portal with a GUI and saves the measured data. The sampling frequency was 512 Hz.

Stimuli

Two females and one male whose mean age was 30 participated in this study. After an initial 30-s period, four randomized stimuli were presented over five trials separated by 5-s intervals [Figure 1]. The stimuli consisted of an apple, orange, spider, and angler fish.

Figure 1

Randomized stimuli presented as low/high salient objects consisting of spiders/angular fish and apples/oranges

In order to measure neural responses to object saliency, we have the subjects focus on the randomized objects and measure their event-related potentials via the MINDO EEG acquisition device. The complete set includes three motherboards, where only the top board is visible. Each motherboard processes 16 channels and transmits the data via wireless Bluetooth communication to a laptop personal computer (PC) equipped with the MINDO GUI software. Every cable from each electrode to the motherboard is labeled and color-coded from 1 to 48 for channel identification [Figure 2].

Figure 2

The MINDO-48S has a dense array of 48 dry electrodes, with an approximate length of a linear array of 250 mm. The spring-loaded electrodes are made of a golden alloy; with a diameter of 1 mm and length of 3 mm. One motherboard is shown as well as ground and reference connections

The stimulus was presented via a 17-inch liquid crystal display (LCD) monitor with 1280 × 800 resolutions. The images were presented as a movie using Windows Movie Maker. Each image was presented for 5 s, and the rest time or blank screen interval between two images was 5 s. All images were centrally shown on a CRT LCD monitor. Image presentation was controlled via a computer Pentium 4 with 512 MB RAM and 40 GB HDD PC. [Figure 3] features a participant wearing the MINDO device as object stimuli are presented.

Figure 3

Visual stimuli presented to participant wearing Mindo device. The participant views a series of various objects while frontal lobe activity is captured via device

Location of the amplitude modulation patterns

The EEG signals were decomposed into amplitude and phase values. Before the application of the Hilbert transformation, a band-pass filter is desirable to select the optimum frequency under this study.[2021] In this work, the beta-gamma band-pass range is selected, which has enabled AM patterns found in EEGs to have a high degree of coherence, stability, and intensity. This band-pass range enabled the optimization of quantifying amplitude and phase thresholds of the EEG signal.

The dataset is broken into 10-s windows, that is, 5-s image stimuli, followed by 5-s rest periods. For each window, the following six steps were required to localize temporal frames so that amplitude feature vectors and phase values could be calculated for EEG quantification:

Step 1: EEG data were pre-processed using a notch filter to remove 60 Hz ambient machine noise, and low-pass filtered to remove unnecessary frequencies above 100 Hz since lower frequencies will be focused on in this study.

Step 2: EEG signals were band-pass filtered by convolution in the time domain with a Remez filter. A beta-gamma band pass filter was applied to find the optimal AA and analytic phase (AP) threshold values as they relate to active states in the cortex.

Step 3: The Hilbert transform was applied to the filtered signal to obtain the real and imaginary parts of the analytic signal.[2223] Analytic power was the squared sum of the real and imaginary parts of the analytic signal. AA was the square root of analytic power. AP was the ratio of the arctangent of the imaginary part of the analytic signal to the arctangent of the real part of the analytic signal. The AP was unwrapped using the MATLAB “unwrap” function. For an arbitrary signal v, the Hilbert transform is defined as follows:

where PV corresponds to the Cauchy principal value. Using v′(t), the complex analytic signal V(t) is defined as: V(t) = v(t) + iv′(t), where i has the usual meaning of . The AA is given by:

AA(t) = [v2(t)+v′2(t)]0.5      (2)

Step 4: Instantaneous frequency, the rate of change in phase with time (Hz), was estimated as the successive differences of the mean unwrapped AP divided by the digitizing step and 2π.[24] The standard deviation of phase differences was calculated to find the instances where phase differences demarcated the instances of AA to show cinematic frames of cortical information.

Step 5: Threshold values were applied to log(AA), unwrapped phase differences (UD), and the standard deviation of phase differences (US). US values segmented the EEG into temporal frames and the AA, which expressed the pattern in each frame, as a feature vector (log(AA)).

Step 6: AA, UD, and US values were analyzed to determine which output of these threshold values enabled the clear delineation of temporal grouping as it relates to cognitive tasking and resting areas.

Data processing

For each 10-s window, AM patterns are found through neural amplitude spike thresholds. Threshold selection separates neural responses to stimuli from background neural activity. Four sets of stimuli were presented five times per trial for three trials per participant. For each window, AM patterns were averaged into 10 point segments, and each segment was counted if it reached above an AM threshold. There may be multiple spikes per window for each object stimulus.

Additionally, all segments were grouped per object stimuli. For each segment, there were five instances per object within a group. For each segment grouping, if 3/5 (60%) of the AM patterns were greater than the threshold, that group was counted. In this manner, similar neural spiking patterns for each object were compared across a given trial. The statistical significance of object saliency between the different stimuli was measured through chi-square (X2) analysis.

Results

Our goal was to find quantitative neurological signals that were produced from different levels of visual stimuli. The analytic amplitude and phase values were found to capture distinctive signal responses to stimuli as seen in [Figure 4]. Figure 4(a) “orange” stimuli produces an early AA/AP signature as opposed to late responses found in Figure 4(b) and (d) which correspond to “apple” and “angler fish.” Figure 4(c) “spider” produces a response midway through the time window.

Figure 4

The analytic amplitude of the EEG signal provides distinct quantitative responses to: (a) orange, (b) apple, (c) spider, and (d) angler fish

Across each participant, each AA response was summed and averaged to determine which stimuli produced the most salient response. For participant 2201, the “orange” stimulus produced the highest average of AA response with the smallest standard error of the mean (SEM), followed by the “apple” stimuli (X2 = 13.92, P = 0.003). However, for participant 4942, the “angler fish” and “spider” stimuli produced the highest averages of AA responses (X2 = 15.06, P = 0.002), yet with relatively high SEM. Again, for participant 3250, the “spider” stimulus presented the highest averages with a low SEM, along with “apple” having a high count but with a relatively high SEM (X2 = 5.27, P = 0.152) [Figure 5].

Figure 5

Averages and standard error of the mean of analytic amplitudes of the EEG waveform per participant per stimuli

Overall AA averages [Figure 6] demonstrate that the most salient stimuli response was predominantly “oranges” followed by “spider” stimuli. It is worth noting that, while the “orange” stimulus produced an overwhelming number of AA responses for one participant, the “spider” stimulus maintained a consistent count throughout all the trials across all participants (X2 = 30.99, P = 0.001).

Figure 6

Total number of analytic amplitudes of the EEG waveform per participant per stimuli

Event-related potentials were acquired in three sections of the frontal lobe, consisting of 16 electrodes each. Each of these sections corresponded to left, center, and right areas of the MINDO device. We wanted to determine where the most dominant areas of the frontal cortex produced the responses to stimuli. The midrange frontal cortex produced the highest number of responses to stimuli presented to the participants versus the left and right areas. “spider” and “orange” stimuli were, once again, the most salient types of stimuli, which produced midrange frontal activation (X2 = 4.19, P = 0.004) [Figure 7].

Figure 7

Total number of analytic amplitude values associated with the EEG waveform per participant per stimuli from the left, right, and middle areas of the frontal cortex

Discussion

Our approach can differentiate between high and low salient objects. The AA/AP calculations have distinct and repeatable neuro-markers per participants that relate to object recognition. Although our analysis did not display reproducible neural responses between less salient objects such as apples and oranges, it did achieve differentiating between two types of object saliency, that is, low and high salient object such as spiders vs. oranges [Figures 5 and 6]. Our methodology randomized objects so that there were no repeatable images across the trials. Therefore, the participant did not become acclimated to viewing similar objects.

High-resolution or high-contrast images did not seem to play an important role in perceptual object responses.[25] One important aspect of visual neural processing to consider is non-classical receptive field inhibition,[26] whereas if all the attributes of an object are non-homogeneous, that is, the parts of the object are not the same in the focal area as they are in the neighboring areas of the central part of the object, then neural responses are less modulated or inhibited. This idea may correspond to higher neural activity found in the spider stimuli, since its appendages differ from the roundness of its body. Additionally, object association seems to play a greater part in neural responses, such that a low-resolution image of a threatening object may produce a higher neural response than a high-resolution image of fresh fruit.

The middle frontal cortex presented the highest responses to object saliency [Figure 7], specifically in higher saliency objects such as the image of the spider. Our results agree with previous work that entails the frontal cortex in integrating object recognition with working memory. This result confirms that working memory may be implicated in greater neural responses due to learned behavior. Previous studies have also found that emotional binding to object recognition has also been found in the frontal cortex.[27] Additionally, several regions of the brain appear to mediate responses to salient stimuli that lead to appetitive and aversive behaviors, areas implicated in object recognition. Neural responses to aversive stimuli have been found to be greater than nonaversive stimuli.[28] We see a higher degree of neural responses to aversive stimuli when the participant is subjected to the spider stimuli, whereas the presentation of the orange stimuli has slightly lesser appetitive responses.

Dendrites convert the axonal pulse they receive into a wave, which is later converted back into a pulse by the axon. This pulse-wave and wave-pulse is an essential function of the neurons and neuron populations. Neural activities produce a “wave” of synchronous activity across dense, larger neural populations. This activity is formed from a neural state, which can be associated to various visual sensory inputs. Freeman[12] has found that cortical responses to stimuli can be discerned via the AA of the signal, which is associated to the high or low intensities of the dendritic populations as well as the firing rates of neural neighborhood clusters. The Hilbert transform enables the calculation of the analytical amplitude, which has been found to provide the intensity of neural populations based on the saliency of the visual input. As the brain processes various visual inputs, a phase transition occurs where neural signal modulations capture the activity of perceptual information processing from sensory inputs.[23] State transitions involve both the amplitude component and the phase component of the signal, in the form of phase transitions. For each presented object, neurological signals initially undergo a resetting or initialization of the phase component of the signal as predominant beta-gamma activity. Afterwards, a re-synchronization then stabilization of amplitude modulation, followed by an increase in amplitude of the AM pattern, all within 24–34 ms.[14] We see these effects for each AM activity in the cortical dynamics of object representation. The interaction between the amplitude and phase component of the EEG signal presents a type of cinematographic phenomena where the amplitude component represents the picture or content of the film and the phase component becomes the shutter of the camera.[24] Successive AM/PM activity resembles the frames in a black/white cinema with successive spatial patterns held briefly. We see this AM/PM interchange per object stimuli, with each object having its own neural signature depending on appetitive or aversive stimuli.

Conclusion

Our work has demonstrated that visual object stimuli have different levels of neural responses that can be consistently measured through the frontal area of the cortex. The interpretation of sensory stimuli is in the form of AM/PM patterns, which exhibit impulse and timing depending on the stimuli. Additionally, learned responses to the stimuli have a large bearing to neural activity. Those objects that have traditionally either instilled fear or are appealing to the viewer will present a different type of neural response than those objects deemed “neutral” to the observer. The analysis presented in this work have shown significant differences in neural responses to aversive and appetitive stimuli, which may support the idea that objects that present a higher emotional response may bypass any habituation to neural activity. It is through the fine-grained signal processing approach in this work that support these types of neural activity.

Object perception involves the brain to be in a constant state of expectancy, where the external world presents stimuli that are converted to action potentials. Action potentials become pulse trains throughout the cortex that innervate pathways throughout the cortex to the frontal cortex, which enables object recognition with working memory. Sensory input has no meaning by itself − a type of “binding” has to occur that translates stimuli to meaningful information, such as through the brain's capability to determine if the objective is appetitive or aversive to the viewer. The conversion of external stimuli to recognition involves the mechanism of learned behaviors that associates objects to neural signatures, which can be deciphered as frames of AM/PM patterns. This mechanism becomes the language of the brain. Through this work, we can expand our encyclopedia of neural signatures to object recognition, and provide a broader understanding of quantitative neural responses to the outside world.

Financial support and sponsorship

None.

Conflicts of interest

There are no conflicts of interest.