Probing of Brain States in Real-Time: Introducing the ConSole Environment.

Recent years have seen huge advancements in the methods available and used in neuroscience employing EEG or MEG. However, the standard approach is to average a large number of trials for experimentally defined conditions in order to reduce intertrial-variability, i.e., treating it as a source of "noise." Yet it is now more and more accepted that trial-to-trial fluctuations bear functional significance, reflecting fluctuations of "brain states" that predispose perception and action. Such effects are often revealed in a pre-stimulus period, when comparing response variability to an invariant stimulus. However such offline analyses are disadvantageous as they are correlational by drawing conclusions in a post hoc-manner and stimulus presentation is random with respect to the feature of interest. A more direct test is to trigger stimulus presentation when the relevant feature is present. The current paper introduces Constance System for Online EEG (ConSole), a software package capable of analyzing ongoing EEG/MEG in real-time and presenting auditory and visual stimuli via internal routines. Stimulation via external devices (e.g., transcranial magnetic stimulation) or third-party software (e.g., PsyScope X) is possible by sending TTL-triggers. With ConSole it is thus possible to target the stimulation at specific brain states. In contrast to many available applications, ConSole is open-source. Its modular design enhances the power of the software as it can be easily adapted to new challenges and writing new experiments is an easy task. ConSole is already pre-equipped with modules performing standard signal processing steps. The software is also independent from the EEG/MEG system, as long as a driver can be written (currently two EEG systems are supported). Besides a general introduction, we present benchmark data regarding performance and validity of the calculations used, as well as three example applications of ConSole in different settings. ConSole can be downloaded at: http://console-kn.sf.net.

Introduction

Rationale

An increasing amount of electrophysiological (EEG/MEG) studies have recently shed new light on our understanding of how the brain processes and represents internally and externally generated input. The still-dominant approach of stimulus averaging across several trials implicitly or explicitly assumes an invariance of a neuronal response toward a certain stimulus and treats trial-by-trial fluctuations as noise. This notion is particularly pronounced in ERP studies in which even temporally fluctuating neuronal responses (“induced activity”; Tallon-Baudry and Bertrand, 1999) are removed as a consequence of averaging. However, it is becoming increasingly obvious that trial-by-trial fluctuations bear functionally meaningful information and explain a significant amount of the trial-to-trial variability observed in overt behavior. Evidence for this view originates in a variety of different approaches that encompass different levels of neuronal activity. For example, Romei et al. (2008b) showed that the level of ongoing alpha power in visual areas interindividually predicts the intensity needed to elicit phosphenes at chance level. Trial-by-trial fluctuations of pre-stimulus alpha power also predicted whether or not a phosphene would be perceived intraindividually, when subjects were stimulated at threshold intensity (Romei et al., 2008a). Other studies have shown better performance in a visual detection task in trials preceded by low power in the alpha band prior to stimulus onset (Ergenoglu et al., 2004; Hanslmayr et al., 2007; van Dijk et al., 2008). This is also the case for visual discrimination tasks (Hanslmayr et al., 2005a) and the performance can even be modulated by neurofeedback (Hanslmayr et al., 2005b). Although they are the most frequently reported phenomenon, pre-stimulus effects are not restricted to the alpha band. Similar results have been found for beta (Schubert et al., 2009) and gamma (Wyart and Tallon-Baudry, 2009). Taken together, these results suggest that the fluctuations of ongoing cortical oscillations represent certain brain states that determine the “fate” of how an incoming stimulus will be further processed. Of interest is that the patterns of these results found on a neurophysiological as well as behavioral level strongly resemble the patterns found in studies in which ongoing oscillatory activity is modulated in a top-down manner by differential experimental conditions. Prominent examples include attention (Klimesch et al., 1998; Worden et al., 2000; Bastiaansen et al., 2001; Romei et al., 2008a) as well as working memory (Jensen and Jokisch, 2007; Tuladhar et al., 2007). Evidence for the significance of these fluctuations is also provided by studies that show that ongoing brain activity produces highly structured patterns – similar to those evoked by an actual stimulus – on the level of single units (Kenet et al., 2003) as well as on the system level (Fox and Raichle, 2007). In functional terms, these findings support the aforementioned notion that fluctuations of ongoing brain activity represent fluctuations of brain states that are associated with differential predispositions for a certain cognitive performance. Instead of averaging variability away, a growing community of neuroscientists is realizing that understanding trial-to-trial variability may hold one key to a deeper understanding of brain functions. However, this research path requires the challenging analysis of data on a single-trial level.

A common approach to analyzing the contributions of trial-by-trial variability in MEG/EEG data involves first transforming the data from the time-domain to the time–frequency-domain. The resulting amplitude and phase values can then be averaged over trials (e.g., “seen” versus “not seen” comparison; Romei et al., 2008a) to increase the signal-to-noise ratio or analyzed on a single-trial level [e.g., correlation of EEG with motor-evoked potential (MEP) following transcranial magnetic stimulation (TMS); Sauseng et al., 2009]. It is crucial to keep in mind that this approach assumes that oscillations are the lingua franca for establishing communication within and between neuronal assemblies. The simplest approach is to analyze the change in power of certain frequency bands while reacting to a stimulus. While the power of an oscillation represents the amount of local synchronization (e.g., a brain region or a fraction thereof), other methods can be employed to explore connectivity between distant brain regions (Friston, 2002). These methods either assess the statistical dependency between two time series of two sensors or sources on a single-trial level (functional connectivity; e.g., coherence or phase synchrony; for an overview see e.g., Varela et al., 2001) or measure the “causal” association between two signals that yield information about the directionality of the information flow (effective connectivity; e.g., Partial Directed Coherence; Baccalá and Sameshima, 2001).

All these studies nonetheless face a major limitation: although they aim to show a direct relationship between certain features of cortical oscillations and an assumed functional state on a trial-by-trial basis, their conclusions are drawn ex post facto. Since the direct (real-time) control of cortical oscillations is difficult in vivo (an interesting avenue may be the recently reported “entrainment” paradigms; e.g., Mathewson et al., 2010; Romei et al., 2010), even though it is not resolved how “entrained” oscillations are really associated to genuinely spontaneously produced oscillations), the experimental setup that comes closest to allowing causal inferences is to temporally trigger events as close as possible to the hypothetically relevant brain activity feature, which fluctuates over the course of an experiment. A real-time framework even holds advantages for the more conventional offline analysis approach, including a clearer distinction between high and low alpha trials that enables a better contrast between the hypothesized brain state and its assumed behavioral impact (e.g., on reaction time). Currently, the presentation of stimuli is entirely random with respect to features of ongoing brain oscillations and the conclusions that can be drawn from it are correlational (see Figure 1).

Figure 1

Comparison between the classic offline and the proposed online approach. By targeting the presentation of events and/or stimuli to hypothesized brain states, the hypothesis can be more easily verified or falsified.

Requirements of a real-time application

In order to pursue the hypothesis-driven research approach outlined above, a system is needed that monitors in real-time the relevant feature of ongoing oscillatory activity and controls the course of the experiment dependent on certain criteria (e.g., level of power in a particular frequency band, level of synchrony between distinct sources) defined in advance by the experimenter. Such an approach would ideally complement explorative ex post facto studies in order to allow for stronger inference regarding the functional relevance of certain brain states. In this paper, we present a user-friendly and open-source software environment called Constance System for Online EEG (ConSole), which allows EEG/MEG researchers to pursue such a hypothesis-driven approach. Although some commercial and non-commercial (not necessarily open-source) programs exist that enable the implementation of some aspects of the described research approach, they are either targeted at clinical neurofeedback or brain–computer-interfaces (BCI). Even though this kind of research is also feasible within the ConSole environment, the main intention is its use as cognitive neuroscientific tool – that is, to test hypotheses regarding the functional relevance of brain oscillations in humans.

A few applications currently exist that make experimental setups like these possible. These, however, display shortcomings that cannot be neglected. Many EEG system vendors ship real-time software along with their products; yet these applications are closely tied to a specific brand or even type of amplifier and most are designed for neurofeedback. Thus, they do not provide the features essential for controlling the course of the experiment (e.g., a TMS or experimental computer). Switching to a different hardware system is virtually impossible – a serious limitation in cases when laboratories with different hardware have decided to collaborate. Moreover, these systems are mostly proprietary, meaning that one has to rely on the features provided by the manufacturer with no possibility to alter or add functionality, let alone distribute these changes so that other scientists can profit from them. Another option would be to use one of the two available systems targeted at BCI development – BCI2000 (Schalk et al., 2004) and OpenVIBE (Renard et al., 2010). While only the first system suffers from the licensing issues described above, both options are primarily designed for BCI and neurofeedback research and not for conducting real-time cognitive neuroscience experiments. The FieldTrip community has developed another promising approach by integrating their extensive Matlab routines with a real-time acquisition system (Oostenveld et al., 2011). As the whole toolbox is released under an open-source license, it also offers great flexibility. It lacks however the modularity of ConSole (see below) and suffers from lower processing speed since it requires the Matlab environment. The modules of ConSole are compiled binaries written in C++, a programming language that can be greatly optimized for speed with modern compilers and also provides an interface to Matlab, thus offering the best of both worlds.

Another important aspect sometimes neglected in software designs in this area is the distinction between developer, scientist and investigator, who all have different requirements for such an application. The available FieldTrip/EEGLAB toolboxes, for example, require a certain degree of proficiency in Matlab programming. However, the scientist who plans and sets up the experiment often has little or no programming experience. It is thus vital to provide an interface flexible enough to exploit all features of the program but still easy and intuitive enough to achieve this in a short time. Whenever the scientist needs a feature that is not yet available, the developer needs a clear and well documented framework for its rapid development. It is essential that this can be carried out without in-depth knowledge of the whole system as this makes the creation of new features easier, faster, and less error-prone. A modular system consisting of independent components that use a standardized means of communication is the ideal solution to this problem. This approach leaves a consistent interface for the scientist, thus further facilitating the setup of experiments. The investigator's (not necessarily the scientist; frequently a Masters student, student intern, etc.) needs in such a program are different, as he or she is the one using the application in the actual experiments. The constrained availability of laboratory time and the need for a professional setting for the participant place further demands on the software. First and most importantly, the software must be simple, fast, and intuitive to use. All unnecessary aspects of the experiment should be hidden, as this reduces the demand on the investigator and thus minimizes errors. Secondly, the application must be mature enough to run stably and must provide a means of recovering a started session after a crash without intervention from the investigator.

Constance System for Online EEG is an innovative system for real-time experiments in cognitive neuroscience, designed to meet the aforementioned demands of the software architecture. Importantly, it can be downloaded for free and is released under the GPL (Free Software Foundation, 1991), making it possible to share modifications or amendments made to the code. Furthermore, ConSole clearly separates between the different tasks mentioned above by using independent modules that can be combined to build the actual experiment. These modules are written in C++, which we consider to be the best compromise between high-execution speed and the demand for high-level programming. C++ is also a standard programming language and is thereby familiar to most developers. ConSole provides a set of built-in modules for virtually any task related to conducting real-time EEG, including different amplifiers, filters, FFT, source projection, and stimulation (see Methods section for further details). A Matlab interface allows for faster proof-of-principle checks in case a method is not available yet in C++ but can be found in one of the MEG/EEG Matlab toolboxes. Moreover, ConSole is able to control external devices via TTL pulses (e.g., TMS device, PsyScope X; Macwhinney et al., 1997). To set up an experiment, an XML file is written, specifying which modules to use, and how these are connected. The XML-dialect used for this purpose is easy to understand, well documented, and does not require prior knowledge of any programming language in order to prepare even moderately complex experiments. In order to make the actual runs of the experiment as easy as possible, ConSole provides the user with a simple and intuitive graphical user interface (GUI). No knowledge about the internal details of ConSole or the specific experiment is needed to run this.

In the current article, we present details on the techniques used with ConSole, including benchmark results. To further demonstrate the capabilities of ConSole, we provide three examples of possible applications. The first application was inspired by a recent paper by Romei et al. (2008a), which showed that phosphene detection not only depends on the correct site of stimulation and intensity but also on the current brain state represented by occipital alpha oscillations. Using the possibilities opened up by ConSole, we directly tested online whether a phosphene was more or less likely to be perceived when alpha was respectively low or high. This example serves as a proof-of-principle that the idea of online EEG experiments (see above) is feasible in practice.

The second example is based on findings of our workgroup (Dohrmann et al., 2007b) and illustrates how neurofeedback experiments can be implemented in ConSole. Patients suffering from chronic tinnitus (a sound lacking a physical source, usually described as a tone or a hissing) learned to normalize their alpha oscillations and thereby putatively increased inhibitory drive in auditory regions. While our first training approach was conducted using proprietary software supplied by the manufacturer, the training outlined in this article benefited to a great extent from the flexibility of ConSole.

To further emphasize the flexibility of ConSole and to demonstrate the Matlab interface, we also present an application for measuring single-trial MEP elicited by a TMS pulse.

Software Description

Constance System for Online EEG was designed with two main goals: ease of use and flexibility. As a monolithic architecture is unable to provide the flexibility needed for such an application, we decided to adopt a modular approach, dividing the application into three distinctive parts: the GUI, the framework, providing internal management functions, and the actual signal processing routines. While the first two parts are included in the main application, the signal processing (as well as drivers for amplifiers, data visualization, etc.) is implemented using totally independent modules – so-called plug-ins. These plug-ins are developed independent of ConSole, which only provides the framework and basic algorithms. This approach eases the development of new modules, as it is only necessary to understand the plug-in interface and not the whole structure of ConSole. The actual experiment is written in an XML-dialect, describing the modules used and the connections between these. This further abstraction from the internal structure of ConSole hides irrelevant details from the designer of the experiment, thus making it possible to design a paradigm without programming skills. For an overview of the structure of an experiment designed using ConSole, see Figure 2.

Figure 2

Coarse overview over the structure of ConSole. The general purpose modules implemented in C++ are used in conjunction with the actual real-time paradigm. Both are joined by the framework that ConSole provides. ConSole then displays all necessary information on a computer screen via its GUI.

The description of an experiment for ConSole follows a hierarchical approach: the highest element is the paradigm, which comprises the whole experiment. The definition of the paradigm includes general information about the experiment, such as the patient-specific information that needs to be acquired. A paradigm includes one or more Setups which define the signal processing and thus the logic of the experiment.

About modules

To achieve our goal of writing an easy-to-use and flexible application for real-time EEG experiments, we decided to restrict each module to one specific task independent from other modules. In this regard, modules are the basic entity of ConSole. This approach is beneficial to both module developers and experiment designers. The experiment designer is provided with modules, each specialized in one specific task that mimics steps from offline analysis. These modules and their connections have to be specified in order to design an experiment (see Figure A1 in Appendix). As all modules function independent of one another and only serve one specific purpose, the module developer only needs to focus on the specific function that the module is supposed to provide. Furthermore, all modules run in separate threads, thus making the application highly scalable on the multi-core PCs that have become a standard in recent years.

Figure A1

Schematic of a typical ConSole paradigm. Data are acquired from the source and propagated from module to module, each further processing the data. The data can also be sent to multiple modules. Eventually, ConSole uses the final results to trigger an event like a TMS pulse or the display of a stimulus.

The module is used as a simple concept in ConSole. Each module can have one or more input and/or output ports. Incoming data (EEG data or any kind of other information-like events) enter the module via one of the input ports (or through an external source, e.g., modules receiving data from amplifiers). The internal logic processes the data and sends the results to the next module via one of the output ports. It must be emphasized that the individual module is completely agnostic of the module it receives data from or sends data to. Moreover, each module can provide settings such as cutoff frequency and filter order for a module that implements frequency filters. To use a module in an experiment, the designer first declares which module will be used, then adjusts the relevant settings and finally connects the ports.

The following example further clarifies the use of modules in a real-time ConSole experiment. For simplicity, only two modules are considered: the source of the EEG data (i.e., the module acquiring the data from the amplifier and feeding it into the system) and a highpass filter. As the source is mandatory for an experimental paradigm, it must not be declared (unlike, e.g., the filter module). The filter module also takes settings defining the cutoff frequency and the order of the filter. In this example, we use a recursive Butterworth filter with a cutoff frequency of 2 Hz and an order of 3.

Afterward, the source module is connected to the filter module. The source module provides an output port called “output,” while the filter module provides both an input port called “input” and an output port called “output.” In this fashion, data flows from the output port of the source to the input port of the filter. After the data is processed in the filter it is sent to the next module via the filter's output port.

On top of providing a plug-in framework that enables the implementation of any module, ConSole already ships with well-tested standard modules that use methods commonly employed in offline analysis tools. These include:

Signal acquisition for a variety of devices

Various signal processing modules. See Table 1 for details.

Table 1

Signal processing modules implemented in ConSole including details and references where applicable.

Module	Details	References
Average reference	Re-reference data to average reference. Subtract the mean over all channels at each sample.
Check peak	Calculates the FFT on the block of data and rejects blocks that do not show a peak in a specified frequency range.
Combine orientations	Combines the orientations resulting from source projection by either rotating the components using a PCA and choosing the one with the highest eigen value or by calculating the total energy.
Complex demodulation	Complex demodulation of the incoming signal.
Distribution	Calculates the percentile of the data based on a distribution acquired in a calibration run.
FFT	Fast Fourier transform using the fftw library.	Frigo and Johnson (2005)
FIR filter (lowpass and highpass)	Finite impulse response filter calculation using Windowed-Sinc algorithm with Blackman-Window.	Octave-Forge (2010)
Hilbert	Calculates the Hilbert transform.
ICA artifact correction	Corrects the data for artifacts using filters calculated by PCA or ICA (currently only JADE is implemented)	Cardoso and Souloumiac (2002)
IIR filter (lowpass, highpass, and bandpass)	Infinite impulse response filter calculation using the Butterworth algorithm.	Octave-Forge (2010)
Interpolator	Interpolates the signal of all channels in a block of data that are identified of including artifacts based on variance and maximum amplitude using spline interpolation.	Perrin et al. (1989, 1990)
Matlab	Sends the data to Matlab and runs a script on the data. The result is fed back to ConSole.
Normalizer	Compute z-values of the data based on a calibration run.
RejectVarMax	Rejects blocks of data that show high variance or amplitude specified in the paradigm.
Source projection	Dipole source projection using a four-shell concentric sphere model. Adapted from Fieldtrip (Oostenveld et al., 2011).	Cuffin and Cohen (1979), Scherg (1990)
Spatial filter	Applies an externally calculated spatial filter to the data by matrix multiplication.

Acquisition, processing, and output of triggers

Graphic and sound output

Raw data displays

Neurofeedback display

Additional features of the console framework

Apart from providing modules for the most important signal processing tasks as well as an easy plug-in framework for extending the functionality of ConSole, the application further provides the experiment designer with important features that enable simple as well as complex paradigms.

Simulating a data source

As with non-real-time experiments, developing an experimental design with ConSole includes testing the code. ConSole provides a special input source that reads data from a file. It is thus possible to test an experiment as well as individual modules using real and simulated data. Two data formats are presently supported: Simple Binary Matrix, a format used for example by BESA and BDF, as used by BioSemi. Future releases will feature a variety of supported data formats for simulating as well as for saving acquired data to disk.

Subject and session management

A very important feature of an integrated experimental framework like ConSole is a proper and easy-to-use subject and session management. The purpose of such a system is to provide standardization in terms of where to store data acquired within the experiment (including events and externally generated information such as, for instance, key presses), general subject data (e.g., subject-id, age, and other information related to the experiment), and log files.

Crash management

Most, if not all, real-time experiments feature several runs that must be completed in one experimental session. It is thus vital for an efficient and unobstructed work flow that potential application and operating system crashes or technical issues like power failure cause the least possible impact on the current session. ConSole is therefore equipped with an efficient crash management system. The system automatically analyzes the chosen subject's dataset for incomplete sessions and is able to resume the program at the point of the crash, thus minimizing the loss of data and time.

Matlab interface

To facilitate the rapid implementation of innovative ideas, ConSole is equipped with an interface to Matlab. Module development can therefore be carried out using a widely accepted programming language familiar to many neuroscientists and innovative ideas can be implemented and tested very quickly. This approach, however, also faces downsides. Transferring data to Matlab and back produces a considerable overhead. Moreover, calculations in Matlab tend to be much slower compared to calculations using C++, although this greatly depends on the amount of optimization in the Matlab code. On the other hand, modules written in Matlab have access to a vast amount of functions and toolboxes not available to C++, such as EEGLab, FieldTrip, and NutMEG. Although it is of course possible to port those functions to C++, this is not always feasible due to time constraints, thus further increasing the value of the Matlab Interface for initial testing purposes.

Artifact correction

Depending on the regions of interest and the setup of a particular experiment, a powerful artifact correction or artifact rejection algorithm is essential to a real-time experiment. The impact of artifact-contaminated data on the outcome of an experiment is much greater for real-time, brain state-driven studies, as an artifact is not only a period of unusable data but might also lead to a false calculation of the current brain state and could therefore, for example, trigger a stimulus that should not have been triggered. The impact of certain artifacts of course depends to a great extent on the region of interest for the calculation of the brain states and the reference used. So the choice of whether to employ artifact correction, rejection of contaminated trials or no correction at all in an experiment should be carefully considered. Artifact correction algorithms alter the data and reduce independence between channels. Rejecting contaminated trials is possible as long as the experiment does not depend on ongoing, uninterrupted output. Identifying contaminated trials is not a trivial task in a real-time experiment. In contrast to offline analysis, visual inspection of the data is not possible. Only unsupervised, automatic algorithms, which commonly employ simple thresholds for the absolute maximum or the variance in the data, can be used. This method is, of course, far from perfect and results in false positives and false negatives. Proper tuning of the relevant parameters is therefore crucial.

Presently, the best method for artifact correction suitable for online approaches is using spatial filters calculated that use one of the many ICA algorithms available. In general, ICA algorithms calculate components of the data that are as independent as possible from one another. This property renders the approach very suitable for online EEG analysis, as (1) common artifacts like eye movements, blinks, and movements of the head or body are completely independent from any cortical signal, and (2) as opposed to PCA approaches, the contamination of artifact components with cortical data is kept to a minimum as the ICA algorithms ensure maximum independence between the components.

Constance System for Online EEG implements ICA-based artifact correction using a modular framework, essentially allowing any spatial filter-based algorithm to be included in the application. To calculate the weights, a calibration run is performed before the actual experiments on which the weights are calculated. ConSole provides the user with a very convenient way of choosing artifact components by offering a three-split window (see Figure A2 in Appendix). The upper panel shows the original data, the middle panel the calculated components, and the lower panel the resulting data after correction. Artifact components can be marked in the middle panel with an immediate effect on the lower panel. Together with the possibility of displaying the topography of the component on a 3D head, choosing the correct components to reject is a fast and easy process. To use the calculated weights and chosen components in the experiment, the “ICA Artifact Correction” module has to be placed between the “Average Reference” module and the rest of the processing queue.

Figure A2

This figure shows the artifact component selection process of ConSole's built-in artifact correction. An ICA algorithm was applied to calibration data. The upper panel shows the original data to identify the time points when artifacts were present. The middle panel shows the components calculated by the ICA algorithm. The user selects components representing artifacts by clicking on the number to the left. The lower panel shows the resulting data after correction to verify the outcome.

Two ConSole modules provide rejection of contaminated trials. The “Reject VarMax” module implements thresholding of the incoming data. It first identifies channels exceeding a certain variance or absolute maximum. These channels are then set to zero. The remaining channels are then tested for variance and absolute maximum. If one of the defined thresholds is exceeded, the trial is not forwarded to the next module.

Another approach for separating good and bad trials in an experiment investigating oscillatory activity is checking for peaks in the frequency spectrum of the trial. For instance, in an experiment studying the impact of high/low alpha on some cognitive measure, each trial can be checked for a peak in the alpha band using the “Check Peak” module. Trials containing only noise or some other artifact interfering with the normal spectrum are thus rejected.

Measuring timing accuracy

A crucial property of a real-time experiment system is the precise timing of the triggered events. Events should be detected and propagated with as little delay and jitter as possible. Both parameters have many influencing factors. It is thus important to separate these factors in order to optimize the system.

Factors influencing delay and jitter

As pure electrical transmission occurs almost instantly, the first factor to be taken into account is the amplifier, including the transmission to the PC (in most cases today via USB). The delay and jitter introduced until this point cannot be reliably estimated, and for the amplifier used in our setup (ANT, 128 channels), no specifications exist. We thus used a custom-built button box capable of generating TTL pulses to trigger the amplifier and a function generator (WAVETEK 10 MHz DDS model 29). The function generator produced a negative pulse at one of the amplifier's channels. When this negative pulse was detected by ConSole, it sent a second TTL pulse to the amplifier that could be differentiated from the pulse generated by the button box (see Figure A3 in Appendix). It is thus possible to calculate the delay between the button press and the TTL pulse generated by ConSole. We used 398 pulses to measure the minimum delay and jitter of the system.

Figure A3

Schematic overview of the setup to measure the delay and jitter of ConSole. The Button-Box sends a TTL pulse to the Function Generator and the amplifier. The Function Generator immediately sends the analog signal (negative pulse or 10 Hz sine wave) to the amplifier. Both the TTL pulse and the analog signal are recorded by ConSole. ConSole then sends a TTL pulse to the amplifier as soon as it reacts on the incoming analog signal generated by the Function Generator. This second TTL pulse is also acquired by the amplifier and recorded by ConSole. The difference between the two TTL pulses is the delay between signal generation and ConSole's reaction.

These are, of course, ideal conditions. In a normal experiment, delay and jitter are necessarily greater since the higher amount of running analysis modules will add to both. Furthermore, certain signal processing methods such as digital filters inherently add delay and jitter. In addition, many methods require blocks of data to operate on. In those cases, a decision between better and more valid signal processing against faster signal processing has to be made. These parameters largely depend on data quality and the features to be extracted and must be balanced with timing constraints. To measure a more realistic setup, we adapted the design of the second experiment to measure timing accuracy. Incoming data was filtered (IIR Highpass, order 2, cutoff 2 Hz; IIR bandpass, order 3, passband 3–40 Hz) and then grouped into 1-s blocks (512 samples) every 125 ms. These blocks were then prepared for Hilbert transformation by applying an IIR bandpass filter (order 3, passband 8–12 Hz). To avoid filter artifacts at the beginning of the blocks, the first 256 samples (500 ms) of the blocks were discarded before the Hilbert transform. The instantaneous amplitude of the remaining data was then computed by calculating the square root of the sum of squares of the real and imaginary part of the analytic signal. A preparatory run was used to calibrate the system to only react on high levels of the 10-Hz oscillation. As soon as the calibrated level was exceeded, a TTL pulse was sent to the amplifier.

We used the same function generator as in the first timing test to generate one hundred ten 10-Hz oscillations. The oscillation was triggered by pressing a button on the custom device and stopped after the button was released. The button press also sent a TTL pulse to the amplifier. As soon as ConSole detected the oscillation, it sent a TTL pulse to the amplifier. We used the difference between the two TTL pulses to estimate the delay and jitter of the setup.

As we had expected, the results differed greatly between the simple setup that only included the detection of a rectangular pulse and the more elaborate one in which a 10-Hz oscillation had to be detected. The first setup showed a mean delay of 17.5 ms with a SD of 0.5 ms. The delay was in the range between 0.8 and 27.3 ms and the distribution was approximately uniform (see Figure 3A). In the oscillation detection task, the delay increased to 477.4 ms on average with a SD of 46.3 ms and the distribution was approximately normal (see Figure 3B). This increase can be very well explained by the use of blocks of 500 ms and the inter-block interval of 125 ms.

Figure 3

Distribution of the delays between signal generation and reaction of ConSole. (A) Shows the histogram for the detection of a pulse, (B) shows the histogram for the detection of a 10-Hz oscillation.

Consequences and possible optimizations

The aforementioned benchmark results show that the delay and the jitter introduced by ConSole are reasonably low to conduct real-time experiments that analyze the power of oscillations. To further optimize for the delay and jitter between the input and the system's reaction the amount of data averaged in one block could be decreased as well as the inter-block interval. While the latter would only increase the load on the computer system used to run the software on, decreasing the amount of data analyzed in each block would also increase the susceptibility to random fluctuations (noise) of the analysis. For the benchmarking and the first example in this paper, we chose a window of 5 cycles of the center frequency. This corresponds to window-lengths commonly used in offline analysis settings that range from about 1.5–7 cycles (Tallon-Baudry et al., 1996; Rizzuto et al., 2003; Fujioka and Ross, 2008; Romei et al., 2008b; Schubert et al., 2009). As we did not average over trials, we decided to use a window-size from the upper range. Therefore with regards to online power analysis, there is an inherent trade-off that the researcher is faced with in setting the optimal parameters. A good strategy in general would be to run some offline experiments first and to use ConSole's possibility to simulate an online experiment using previously recorded data (see Simulating a Data Source). By this means the researcher has the possibility to optimize the code in order to be temporally as close to the events of interest as possible. In general “brain states” marked by an extended period (on the range of hundreds of milliseconds) of an increase/decrease of power are optimal for investigation using ConSole. Triggering on short-lived “bursts” of an oscillation is not recommended, at least with the standard analysis methods at hand.

The mentioned temporal limitations – i.e., to initiate events (e.g., TMS) as closely to the neuronal event of interest as possible – applies also to the phase of an oscillation which has received great interest recently (e.g., Hanslmayr et al., 2005a; Busch et al., 2009; Mathewson et al., 2009). Theoretically it would be interesting to directly stimulate at peaks and troughs of an oscillation, however the current delay using standard methods is not sufficient for this endeavor. An exception to this may be very slow oscillations below 1 Hz which have also gained increased interest recently.

Example 1: Phosphenes and Real-Time Alpha

Introduction

In recent years, an increasing number of articles on the functional role of spontaneous alpha oscillations have been published. Most of these studies use a task involving covert attention to a region left or right of a fixation cross, which detects a target in these areas. The great majority of these studies come to the conclusion that ongoing alpha oscillations are modulated by drawing attention to one area, either by an increase of power in ipsilateral cortical regions of the visual cortex or by a decrease contralateral to the attended visual hemifield (e.g., Thut et al., 2006). There is even evidence for a retinotopic organization of the modulation of ongoing alpha oscillations (Rihs et al., 2007). Overall, these studies give strong evidence that alpha represents a mechanism of active inhibition, pointing to a top-down mechanism modulating alpha oscillations. Although this assumption is most likely correct in regards to the aforementioned evidence, these studies do not answer the question of whether spontaneous fluctuations of these oscillations or even the different phases of the oscillation themselves have an impact on the processing of stimuli. The answer to the second question can be found in studies that account for the phase of the ongoing alpha oscillation immediately before stimulus presentation (Mathewson et al., 2009) or even earlier in the pre-stimulus period (Busch et al., 2009; Busch and VanRullen, 2010). Romei and Thut took an interesting approach to the first question and showed not only that the tonic level of occipital alpha predicts the interindividual phosphene threshold (Romei et al., 2008b), but also observed a dependency between fluctuations of ongoing alpha and the probability of phosphene perception when subjects where stimulated at phosphene threshold.

Although the dependency between ongoing alpha oscillations and phosphene perception could be causal, as mentioned above, offline studies only provide correlational evidence. In order to test the hypothesis that the likelihood of phosphene perception causally depends on the energy of spontaneous alpha oscillations, a real-time experiment is needed that can specifically stimulate the primary visual cortex at those points in time when alpha is either low or high.

Methods

Six subjects (four female, mean age ± SD: 24 ± 3.9 years) were screened and trained according to procedures previously used in similar studies (Romei et al., 2008a,b) after giving written informed consent. The procedure was approved by the local ethics committee. Screening and training was necessary in order to select only those participants who reported seeing phosphenes and to train the reliability of their answers. We blindfolded the participants and used single-pulse TMS to elicit the perception. Neuronavigated TMS (Magstim Rapid2) was used to reproduce stimulation sites between sessions. In the last session, TMS pulses were triggered depending on the current alpha power in primary and secondary visual areas, which was computed using a minimum-norm estimation (Hämäläinen and Ilmoniemi, 1994) relative to a 5-min baseline measurement. The participants performed four runs of 5 min each. A TMS pulse was triggered when the current alpha power fell into the upper or lower quartile of the distribution acquired in the baseline run. To achieve a balance between low and high alpha power trials, the system did not react on high alpha if the total amount of high alpha trials exceeded the total amount of low alpha trials by two. The same strategy was used to limit the amount of low alpha trials with respect to high alpha trials. The participants indicated with a mouse click whether or not they had perceived a phosphene.

In order to verify the effectiveness of ConSole and the applied algorithms as well as to test the hypothesis that phosphenes are more likely to be seen when occipital alpha is low, we cut the data into trials 3000 ms before to 500 ms after the TMS pulse. The epochs were then subjected to a detrending algorithm in order to remove linear trends from the data. Afterward, the data were bandstop filtered (fourth order forward Butterworth filter; frequency range 49–51 Hz) in order to eliminate possible power line noise from the data. As the experiment relied on stimulating the participants at their individual threshold intensity, we discarded all runs that, according to a binomial test, did not show a balance between seen and unseen trials. Data from the remaining runs were then grouped into four categories (Seen/Not Seen × High/Low Alpha) and visually inspected for artifacts in the time range between 1000- and 50-ms pre-stimulus. As we were interested in effects relative to the stimulation side, we mirrored data from the left hemisphere for those participants who were stimulated at the right hemisphere and vice versa. We then applied a time–frequency analysis using variable window-sizes of four times the wavelength of the respective frequency tapered with a Hanning window. We used a non-parametric, cluster-based statistic with 1000 randomizations to control for type 1 errors (Maris and Oostenveld, 2007). In this paper we show the preliminary results of six participants.

Results

On average, ConSole triggered 71 ± 20 TMS pulses for each subject. Fifty-two percent of those trials were identified as low alpha.

To test the validity of the separation between low and high alpha trials, we applied a cluster-based statistic on the time–frequency representation of the data from trials identified as high versus low alpha by ConSole. The algorithm found one significant cluster (p < 0.001) that confirmed higher alpha from 700 to 125 ms before the stimulation (see Figure 4). Alpha was increased rather globally, however the maximum increase was found at the site of stimulation as targeted by ConSole. The data also show a good timing accuracy. The window for the Hilbert transform of the paradigm was 500 ms. According to the aforementioned results, we would have expected delays of about 400–500 ms. The data, however, show that the maximum difference is between 350- and 175-ms pre-stimulation. Interestingly, higher synchronization in trials identified as containing high alpha is not restricted to the alpha band. Higher synchronization also extends into the beta range.

Figure 4

Results of example 1. (A,B) Show the comparison between trials identified as high alpha versus low alpha. (A) Time course at one representative occipital electrode. The maximum difference is between 400 and 300 ms before ConSole sent the trigger. (B) Topography at the point of maximum difference. Higher alpha is not restricted to the area analyzed but extends to the other hemifield as well as to frontal areas. The stimulated area is depicted by the TMS coil sketch. (C) Box and whiskers plot showing the distribution of alpha power for all four conditions. The red lines represent the medians, the edges of the boxes mark the first and third quartile. Whiskers extend to the most extreme value not considered an outlier. Outliers are values that exceed the first or third quartile by 1.5 times the total range between the first and third quartile.

Although our results are similar to those reported by Romei et al. (2008a) in that, on average, trials in which no phosphene was perceived were preceded by higher alpha, we were not able to show that pre-stimulus alpha power predicted the probability of the participant seeing a phosphene by comparing the responses to high and low alpha trials. At first glance, these results do not seem to fit together. This could be due to the small number of participants, not optimally defined thresholds or a more complex relationship between alpha and phosphene perception that a “simple” offline contrast of “seen” versus “unseen” would suggest. Interestingly, however, when comparing only the high alpha trials in which the phosphene was seen versus those in which it was not seen, a strong trend in frontal areas is revealed. If alpha was high under the stimulated site, alpha had to also be high in frontal areas in order to produce a phosphene. This could serve as a starting point for a more in-depth analysis, as this result could imply that the connectivity between these regions is relevant, but only if alpha is high in occipital areas.

Discussion

One purpose of this study was to test whether ConSole was capable of separating low and high alpha trials correctly and with reasonable delay in a real online setting. Our results show that this separation was perfect but could be optimized to minimize the delay between classification (high versus low alpha) and stimulus onset. One concern for real-time experiments is that the amount of trials cannot be predicted. The example however shows that on average 71 trials out of the maximum number of 120 were triggered. The balance between high and low identified trials was guaranteed by limiting the amount of imbalance by definition in the paradigm file.

Another purpose of the study was to test the hypothesis of Romei et al. (2008a) that the perception of phosphenes depends on the current level of alpha synchronization over the occipital cortex at the stimulated hemisphere. Although the offline analysis of the data points in the same direction, the online analysis seems at odds with a simple “high” versus “low” alpha functional distinction since we did not find any difference in the probability of seeing a phosphene in trials with high or low alpha as we had originally hypothesized. This might be due to the small number of participants, as the results shown are only preliminary. More interestingly, however, we observed that if alpha was high under the stimulated area, frontal areas had to be synchronized in the alpha band as well in order to produce the perception of a phosphene, thereby suggesting the influence of a connection between these two regions. Overall, the data suggest that the “high” alpha category is functionally more diverse than simply reflecting an inhibitory state and that under certain conditions it may even favor a perception near-threshold. In future approaches we will scrutinize the actual distribution of the underlying electrophysiological marker in more depth, as one further possibility for not seeing the hypothesized results might also be that “high alpha” does not only cover the uppermost 25% of the distribution but extends into the low alpha range far deeper than we had expected. This might also indicate that the relationship between alpha power and behavior is not linear, as assumed in the study, but, for instance, logarithmic.

Furthermore, we plan to extend this approach to other modalities such as the visual and motor system, as alpha is believed to have a common functional meaning (Weisz et al., submitted) – at least in primary and secondary sensory and motor cortical areas.

Example 2: Treating Tinnitus with Neurofeedback

Neurofeedback is generally defined as the operant conditioning of signals acquired from the brain via various methods such as EEG, MEG, fMRI. For almost 40 years, this approach has been used to teach patients how to normalize abnormal brain states (Sterman and Friar, 1972; Lubar and Shouse, 1976; Rockstroh et al., 1993; Masterpasqua and Healey, 2003; Dohrmann et al., 2007a,b). Recent studies by our group have shown significant improvements of tinnitus loudness and distress in those patients who were able to learn to normalize temporal alpha and/or delta activity but not in those who failed to learn the task (Dohrmann et al., 2007a,b). To increase the number of patients who are able to learn the task and thus benefit from the training, we used the fact that sensory alpha rhythms desynchronize on sensory input – the extent to which albeit depending strongly on top-down factors such as attention (Bastiaansen et al., 2001; Müller et al., submitted; Hartmann et al., submitted).

Nine otherwise healthy patients (one female, mean age ± SD: 57 ± 8.8 years) suffering from chronic tinnitus were recruited via advertisements in the local newspaper. All patients were informed about the training and gave written consent. The procedure was approved by the local ethics committee. The patients took part in 10 session within 3 weeks. Each session consisted of one baseline measurement to calibrate the neurofeedback system, four training runs, and another baseline measurement after the training to assess changes in cortical activity within each session. In the training runs, patients were shown a feedback on a screen for 5 s without hearing a tone (see Figure 5A). They were instructed to consider this period as a baseline that showed how auditory areas of their brain behaved without any input. Afterward, patients were stimulated with a sound that was filtered to match their tinnitus sound as closely as possible. Because of the aforementioned effect, patients saw a decrease in alpha activity via the feedback. They were instructed that one possible strategy for increasing alpha activity was to ignore the sound. Baseline measurements before and after the training runs differed from those only by not providing feedback to the patients. The patients were instructed to passively listen to the sounds with eyes open.

Figure 5

(A) Screenshot of the patient's training screen. The fish takes 10 s to move from the left to the right of the screen. The first 5 s are the “baseline” period for the patient without any stimulation. In the second half, the patients were stimulated with a sound that resulted in an alpha desynchronization. The patient's task was then to increase temporal alpha power which was indicated by the height of the displayed fish. The patient was rewarded after the trial if the fish stayed above the target line for a sufficient amount of time. (B) Normalized alpha power of all subjects over all 10 sessions before and after neurofeedback training. Alpha power increased significantly within and between sessions. Error bars denote SE. (C) Distress rating of all subjects over all 10 sessions before and after neurofeedback training. Distress was reduced significantly within and between sessions. Error bars denote SE.

For the training, data acquired from 29 electrodes on the scalp and 2 electrodes beside and above the right eye to facilitate artifact correction were sent to ConSole, highpass filtered, average referenced, and artifact corrected (via ICA). The data were then projected onto eight regional sources. The data of the two temporal sources were subjected to a frequency analysis and the relative energy at the individual's alpha frequency was fed back onto a computer screen. We analyzed the power at the individual's alpha frequency before and after each training.

To first test whether the participants in the neurofeedback study were able to learn how to modulate alpha power, we used a linear mixed-effects model with normalized alpha power at both temporal sources as the dependent variable. The individual session and the time of measurement (pre or post-training) were the independent variables. The advantage of a linear mixed-effects model is the possibility to add so-called random variables that control for variance between individuals. “Participant” was thereby chosen as the random variable. Our results show that, on average, alpha increased within each session (p < 0.001) by 19% (SD: 26%) as well as between the sessions (p < 0.05) as shown by an increase from the first to the last session by 38% (SD: 68%; see Figure 5B). A similar result was drawn by the behavioral results. We used a custom-made questionnaire with six Likert scales about different features of the tinnitus, like loudness, annoyance. On average, the total score decreased by 13% (SD: 17%) within the sessions (p < 0.001) as well as between them (p < 0.05) as shown by a decrease from the first to the last session by 11% (SD: 7%; see Figure 5C).

The increasing amount of studies into the effect of neurofeedback on tinnitus show that, though patients benefit from the treatment, these effects are moderate and subject to strong interindividual variability. This might be owing to a number of reasons that remain unclear. In general, three factors are important for a successful neurofeedback training: contingent feedback about the correct features acquired from the brain, good adjustment to the demands of the task, and keeping the patient motivated. The approach we have presented in this example shows another way of neurofeedback training. The results are comparable to those of recent studies in terms of how much the patients improved. What is however more important is that we have demonstrated that ConSole provides scientists with a tool for rapidly developing and testing new approaches to the treatment of conditions including (but not limited to) tinnitus using neurofeedback. New trainings might include more tailor-made designs like QEEG, different frequency bands, interareal functional connectivity or other ways of giving feedback to the patient, just to name a few possibilities. ConSole not only facilitates the development of these new approaches but also makes testing at multiple sites with different equipment easier due to its independence from the acquisition device used.

Example 3: Monitoring TMS-Induced MEPs

If a strong and brief magnetic pulse is applied to brain tissue associated with motor functioning via TMS, changes in corticospinal excitability can be observed (e.g., Van Der Werf and Paus, 2006). In suprathreshold stimulations, these changes result in a typical pattern of periphysiological muscle activity. This summation of electromyographic, response is commonly called MEP (for further details refer to Barker et al., 1985; Rothwell et al., 1999; Di Lazzaro et al., 2004).

Motor-evoked potential amplitude and shape depend on several factors, such as TMS pulse duration (Rothkegel et al., 2010), coil orientation (e.g., Mills et al., 1992), distance from the coil to the underlying cortex (Kozel et al., 2000), or individual biological differences (Wassermann, 2002). We ran a number of combined EEG–TMS studies with the aim of investigating oscillatory brain activity and its influence on MEPs. In order to achieve a comparable reference value between subjects, we determined resting motor thresholds using a Matlab-based decision-making algorithm within the ConSole environment. Data were acquired and preprocessed using the ConSole framework and the built-in modules. Although this could have also been carried out entirely in Matlab (apart from the acquisition), using the modules provided by ConSole is more convenient and much faster. The part of the paradigm that involved the special analysis of the MEP was confined to a block of data specific to the time range when a MEP was expected to occur. The analysis did therefore not have to run constantly and was not time-critical. This fact and the greater flexibility in having a Matlab script qualified this part of the online analysis for execution in Matlab.

Surface electromyography was recorded from the right first interosseous muscle (FDI) and was sampled at 2048 Hz. ConSole acquired the voltage difference between two surface electrodes attached in a belly tendon montage with a ground electrode taped to the right ear contralateral to TMS Stimulation (Magstim Rapid2). The data were highpass filtered (third order, cutoff: 4 Hz). Upon receiving the trigger representing the TMS pulse, ConSole sent a corresponding block of data via its Matlab interface to a routine that first rectified the epoch and then searched for a peak of >50 mV. The result was fed back to ConSole and displayed on the screen as a text message.

Neuronavigated high intensity single-pulse TMS (on average 60% stimulator output) was applied to the left hand knob area with the handle of the TMS coil pointing backward approximately 45° to the midsagittal line. Coil position was adjusted until absolute FDI MEP amplitude was maximal in three consecutive trials and was marked as so-called “hot spot.” Afterward the resting motor threshold was determined by a maximum likelihood paradigm using the software MTAT 2.0 (Awiszus, 2003).

Results and discussion

By using the aforementioned setup demonstrating the capabilities of interactions between ConSole and Matlab, we were able to receive reliable feedback about whether a TMS pulse had triggered a MEP (see Figure 6 for a schematic example of the feedback in case a MEP was detected and in case, the TMS failed to elicit an MEP). This example demonstrates the use of ConSole to analyze peripheral data in response to a stimulus. Future applications are not limited to TMS and MEPs but may include, for instance, reactions of peripheral signals like skin conductance on electrical or tactile stimulation.

Figure 6

Schematic drawing of the setup to automatically identify TMS-induced MEPs.

Discussion

The analysis of brain activity acquired by EEG or MEG on a single trial level is becoming an increasingly important topic. Treating the variability between trials as a valuable source of information and not as noise that is to be discarded will lead to great advances in our understanding of how brain activity relates to behavior. However, the analysis of data on a trial-by-trial basis raises the importance of good data quality during acquisition as well as advanced analysis strategies. Nevertheless, the offline analysis of the data does not exploit the full potential of the methods available as stimulation is done at random points in time and thus completely independent of the current brain state. Such a correlation approach makes sense as long as prior knowledge about the effect is small and the underlying hypothesis cannot be strictly defined. However, as soon as a clear hypothesis about the functional significance of distinct brain states exists, a more causal test of the hypothesis is favorable. Analyzing EEG or MEG in real-time is an approach that takes advantage of prior knowledge by controlling the experiment depending on those points in time that putatively mark a brain state of interest. This approach not only leads to a stronger test of the hypothesis but can also provide more in-depth insights in phenomena related to the primary effect (e.g., whether alpha is a correlate of active inhibition). In contrast to the offline correlational approach, differences between physiologically clearer defined brain states can be analyzed. For example, in the study comparing the effect of high and low alpha power on the perception of a phosphene, our online approach, despite superb separation of high and low posterior alpha, indicates that high levels of alpha may be functionally not a unitary phenomenon (e.g., inhibition). The contrast within the high alpha trials points to a critical role of frontal regions and suggests that, in some cases, high alpha activity may reflect a more efficient communication between sensory and frontal regions – potentially implemented via specific phase relationships (Busch et al., 2009).

In the current paper, we present a flexible software environment that helps scientists in conducting many kinds of real-time experiments. Its modular approach and clear file format makes setting up experiments easy, even without any knowledge of computer programming.

One of the most important requirements of a real-time EEG/MEG application is the exact and fast timing of the triggers that are used to stimulate the participant. Current computer systems with multi-core processors and gigabytes of RAM are generally fast enough to deal with the acquisition and processing of 128+ channels at high sampling rates. Our system with an Intel Core I7-970 and 4 GB of RAM is easily capable of handling the above presented experimental setups at very low processor load and with acceptable delay and jitter.

Three current examples from our laboratory were given to illustrate that ConSole is a flexible tool that can be universally applied in numerous real-time paradigms and that extends the possibilities in these areas. Apart from the envisaged real-time cognitive neuroscience experiments (Example 1), ConSole can be used, for instance, in clinical settings for neurofeedback (Example 2) or to obtain online feedback about the presence or absence of a peripheral physiological response (Example 3). The latter example was mainly intended to demonstrate the ConSole/Matlab interface, further facilitating the process of creating a real-time analysis-flow, even for scientists not experienced in programming languages like C++.

In conclusion we wish to promote the idea that, apart from conducting offline data analysis, cognitive neuroscience-specific hypotheses should in the future be tested in real-time experiments. This puts high demands on data quality and online signal processing. Furthermore, in order to enable as many neuroscientists as possible to pursue this research strategy, user-friendly, and open-source software frameworks are needed. With ConSole, we attempt to make such a framework publically available and have given first proof-of-concept evidence with regards to its functionality.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.