Closed-Loop Neuroscience and Non-Invasive Brain Stimulation: A Tale of Two Loops.

Closed-loop neuroscience is receiving increasing attention with recent technological advances that enable complex feedback loops to be implemented with millisecond resolution on commodity hardware. We summarize emerging conceptual and methodological frameworks that are available to experimenters investigating a "brain in the loop" using non-invasive brain stimulation and briefly review the experimental and therapeutic implications. We take the view that closed-loop neuroscience in fact deals with two conceptually quite different loops: a "brain-state dynamics" loop, used to couple with and modulate the trajectory of neuronal activity patterns, and a "task dynamics" loop, that is the bidirectional motor-sensory interaction between brain and (simulated) environment, and which enables goal-directed behavioral tasks to be incorporated. Both loops need to be considered and combined to realize the full experimental and therapeutic potential of closed-loop neuroscience.

Introduction

Much has been learned about the mechanics of the brain by treating it as a “black box”, placed in a controlled laboratory environment and stimulated by the experimenter in an open-loop fashion using a pre-defined stimulus protocol to determine its input-output characteristics and how these may be modulated. This approach has been fruitful also in the field of non-invasive brain stimulation (NIBS) and transcranial magnetic stimulation (TMS) in particular, enabling significant advances in the understanding of the functional and pharmacological basis of cortical dynamics (Rothwell et al., 1991; Hallett, 2000; Ziemann et al., 2015).

However, in spite of the significant advances of the past 30 years, TMS has yet to realize its full potential, especially with regard to therapy (Lefaucheur et al., 2014): it is a field plagued with ill-understood intra- and inter-individual variability, effects that hold on average but not reliably for any one individual subject or patient (Hamada et al., 2013; López-Alonso et al., 2014, 2015; Wiethoff et al., 2014; Horvath et al., 2015). One possible origin of the variability in the response to brain stimulation is the variability of instantaneous brain state at the time of stimulation (Ridding and Ziemann, 2010; Li et al., 2015). Recent advances in combining TMS with electroencephalogram (EEG) are enabling us to address this issue by designing stimulation protocols that are controlled by the EEG signal and thus “close the loop” around the brain in a very direct way, short-circuiting the motor-sensory loop (Bergmann et al., 2012).

Moreover, there are compelling conceptual considerations that motivate a “closed-loop” approach: the brain is not a black box; it is not a mere transducer of input to output signals but a generator of behavior—and behavior requires an environment, that is, an entailment between an output of the brain and a (feedback) input to the brain.

In the following, we discuss the conceptual context of closed-loop methods in general, the possibilities and technical challenges of using EEG and TMS to implement a closed-loop set-up, and the implications of this approach to neurophysiology. We review some of the current research and present preliminary technical results from our own lab supporting the expectation that brain-state dependent brain-stimulation will be significantly more effective at modulating neural pathways than current open-loop protocols.

Closed-Loop Method

Conceptual Considerations

An experiment may be considered a “closed-loop” experiment when actions (output from the brain) have consequences (future input to the brain). This is the natural state of affairs for an organism roaming through the environment; indeed, the output of the brain is relevant only insomuch as it has the ability to impact the future and hence the input the brain receives. However, this is typically not the case in the neurophysiology lab: in a controlled “black-box” experiment, the input to the brain is a stimulus protocol arbitrarily defined by the experimenter and the output from the brain is the experimenter’s relevant measure, the dependent variable. In such an “open-loop” experiment, it is only the input to the brain that has a consequence (Figure 1A), the output has no effect on the future and, from the point of view of the brain, is therefore wholly irrelevant.

Figure 1

A family of different closed-loop designs to couple with brain state dynamics. (A) Traditional “black box” experiment where there is no environment that the brain can act on: the stimulation is predetermined by the experimenter, the experimental observable is the output from the brain. (B) The feedback loop triggers a stimulus based on a spontaneously occurring instantaneous brain state; here, the environment is static and without state. (C) This loop represents an agent-environment interaction where the environment has its own internal dynamics, consisting of state (which is fully observable) and the equations of motion (the laws that govern the behavior of the environment). The combined result is an interacting complex system. Areas shaded red indicate parts of the set-up under experimental control, white areas show the “observable behavior”. (D) Shows the relationship between “true” brain state trajectories and the projection onto the measured electroencephalogram (EEG) signal as well as the effect that a transcranial magnetic stimulation (TMS) pulse has shifting state to a new position (figure taken from Mutanen et al., 2013) (E). An experimental closed-loop EEG-TMS set-up configured to couple with cortical dynamics during a simultaneously executed motor-sensory task. Two conceptually different feedback loops can be distinguished, a “brain-state dynamics” loop that is designed to influence the trajectory of the brain state and a “task dynamics” loop that enables active interaction with a (real or computer simulated) environment through the motor and sensory system and the respective encoding stages.

It is of course possible to “close the loop” in the laboratory by establishing a causal relationship between the measured output of the brain and the stimulus generator. Indeed, there are two quite distinct ways in which this can be done: In the simpler case, the stimulus is applied as a function of the simultaneously measured instantaneous brain state (Figure 1B). In this scenario, the neuronal “output” of the brain influences the input to the brain and in this sense the loop is closed; however, the loop is “stateless” and the measured activity of the brain fully determines the input stimulus at any given instant. This kind of a loop does not establish a dynamically changing “outside environment”, there is nothing outside of the brain that has any influence on the input signal.

The more complex closed-loop scenario (Figure 1C) is that of a nervous system experimentally coupled with a physical or simulated outside environment that, similarly to the brain, has its own dynamically changing state and a set of laws that define the evolution of the system (the laws of physics). Here, as in the interaction with the real-world through the motor and sensory systems, neither the brain, nor the environment (and the other brains within it) individually determine the time-course of the stimulus; the input to the brain arises from the complex interactions within the system, which results in the “behavior-in-the-loop” of an “embodied” system.

This third scenario is qualitatively different from the second scenario in that any causal explanation of the dynamic interaction between brain and environment can no longer be limited to the state and transition dynamics of the brain alone, but must take into account the state of the environment, too. Indeed, whereas the complete state of the brain is necessarily not accessible by the experimenter, we can present the subject with a simple constructed or simulated environment to which we do have comprehensive access and which can be experimentally controlled.

This is also the key difference to the ethological approach of just observing an animal in its natural environment: in the fully closed-loop setting as described above, the experimenter is able to access and interfere with the flow of information between neural system and environment on the one hand and the state and transition dynamics of the environment on the other. Instead of observing the output in response to different inputs, the “behavior” of the neural system now includes both the brain and the environment and can be studied in environments following different equations of motion (state transition laws). Whereas in the open-loop stimulus-response setting, the scope of what may be experimentally observed is limited to a single variable (the “response”), in the closed-loop setting the possible state space trajectories that the neural system may take in conjunction with the environment are not so constrained.

There is large body of work creating an artificial connection between the brain and the natural environment through a brain-computer interface (Buch et al., 2008; Bundy et al., 2012; Ramos-Murguialday et al., 2013), bypassing the (damaged) motor system and receiving proprioceptive and visual feedback through the intact sensory system. However, with the ability to stimulate the brain directly using TMS, the environment that is coupled to the neural system can be a simulation running on a computer. By providing “environment-like” closed-loop feedback protocols, a bi-directionally coupled device can implement therapeutic stimulation paradigms on a microchip that are designed to remodel dysfunctional networks by providing the kind of output-input relationships that the network would experience during “natural” interactions with the environment.

Closed-Loop Neuroscience Through EEG-Triggered TMS

The insight that a closed-loop experimental approach may enable us to learn new things about the brain that we cannot learn from a “black box” approach is of course not a new one, as both experimental work in vitro (Huerta and Lisman, 1993, 1995) and in vivo (Siegle and Wilson, 2014) as well as theoretical results (Friston, 2010) attest to. Here we wish to draw attention to the unique potential of non-invasive closed-loop brain stimulation, enabled by the combination of EEG and TMS (Ilmoniemi et al., 1997; Ilmoniemi and Kicić, 2010), and the recent availability of low-cost real-time processor solutions.

TMS is an old technique but it remains the only way to non-invasively excite a specific population of cortical neurons with a spatial resolution of millimeters and a temporal resolution of microseconds (Barker et al., 1985; Hallett, 2000; Müller-Dahlhaus and Vlachos, 2013). Considering the EEG signal conceptually as a lower-dimensional projection of instantaneous brain state, application of a TMS pulse may be viewed as a vector that shifts a spontaneously occurring brain state to a new trajectory (Mutanen et al., 2013, Figure 1D). Importantly, the new state that is reached by the TMS pulse depends on the precise state at the time of stimulation and this is what motivates the development of closed-loop brain-state dependent stimulation paradigms.

The “true” brain state is not accessible, it would constitute a long vector describing the state and activity of each nerve cell and synapse; in practice, the dimensionality is already reduced (information is lost) by the EEG or MEG recording and then further as this is projected onto some quantifiable correlate of a physiological relevant process. In the following description, we focus on what is perhaps the most salient feature of brain state as measured by EEG, i.e., spontaneous oscillatory activity of neuronal populations (Buzsáki and Draguhn, 2004), which takes place on different spatial and temporal scales and can be quantified by several measures, in particular phase.

Spatially, brain states can be observed locally, i.e., in the activity of a specific brain area (Gharabaghi et al., 2014) or in the ensemble activity across large-scale brain networks (Bundy et al., 2012). Temporally, the state of interest can be determined by a certain phase-state of an oscillation cycle (with a timescale of milliseconds) or by changes in spectral power in a specific frequency band (e.g., event-related desynchronization (ERD)), which has a slower time-scale of seconds. The latter has been successfully utilized for BMI-based robot-assisted motor tasks in stroke patients, both in the alpha spectrum of 8–12 Hz (Pfurtscheller and Neuper, 2009) and in the beta spectrum of 16–22 Hz (Kraus et al., 2016).

Closing the loop not only on spectral power but also on instantaneous phase is methodologically more challenging because it requires a real-time signal processing stage with a time-resolution of milliseconds, however, this has recently become possible (see the next section and Figure 2). Finally, in order to go beyond brain states that occur at rest, but instead correspond to a particular cognitive behavioral process, in addition to the “brain dynamics feedback loop”, a “task dynamics” loop will be needed (Figure 1E). EEG-TMS set-ups are already enabling a coupling with cortical dynamics during the performance of a behavioral task in awake human subjects (Thabit et al., 2010; Kraus et al., 2016), and performance dependent feedback has also been shown to support neurorehabilitation in stroke patients (Buetefisch et al., 2011; Ramos-Murguialday et al., 2013).

Figure 2

Preliminary results from a millisecond latency EEG-TMS set-up. (A) Simplified implementation of a closed-loop brain-state dependent brain-stimulation set-up consisting of EEG stage, real-time digital signal processing stage, and a triggered stimulation stage. (B) Raw EEG traces recorded from electrode C3 in the period before a TMS 100 Hz triplet pulse that is triggered by a real-time system based on the preceding 300 ms of EEG data. The system can be configured to trigger the stimulation either at the trough (top trace) or at the peak (bottom trace) of spontaneous alpha activity recorded by EEG over motor cortex.

Technical Challenges

The implementation of a closed-loop set-up requires several different stages: a brain output measurement stage, a signal processing stage, and a stimulus modulation stage (Figure 2A). This has become feasible by recent advances in information technology, which enable complex computations to be performed in real-time even with low-cost standard hardware.

The performance of the closed-loop system can be characterized by the fundamental sample-time of the processor, the bandwidth of data acquired, the complexity of computations that can be performed within a single time-step, the overall feedback latency through the loop from signal to stimulus, as well as any jitter in that latency. Importantly, a loop latency and jitter in the order of milliseconds is required for phase-dependent stimulation of endogenous brain activity in higher frequency bands (beta, gamma).

Sources of latency include the data processing and transfer buffers at each stage, but also the phase shift inherent to signal filters. Generic general real-time signal processing systems (such as Mathworks Simulink Real-Time or National Instruments LabView) can easily process data in sample time steps <1 ms (Zrenner et al., 2010), however, a specialized biosignal recording device is required that makes a non-buffered stream of data available with a small fixed delay of a few milliseconds.

Until recently, artifacts introduced by the TMS pulse in the simultaneously recorded EEG signal were a major obstacle to closed-loop EEG-TMS because of amplifier saturation. However, with the availability of high dynamic range 24 bit analog-to-digital converters, the stimulus artifact is simply captured by the amplifier and the complexity and discontinuities introduced by previously required sample-and-hold or blanking circuits is obviated. In combination with TMS compatible sintered ring electrodes the TMS artifact duration can be reduced to <10 ms (Virtanen et al., 1999; Bonato et al., 2006; Litvak et al., 2007; Veniero et al., 2009). This leaves muscle artifacts and somatosensory as well as auditory evoked potentials that overlap with the TMS-evoked potentials; the different components are then typically segregated using independent component analysis (ICA).

The most significant limitation concerns real-time analysis of the biosignal data, as it is streamed to the real-time processor. Only a sliding window of data preceding the current time point can be considered and this means that any kind of filtering will cause phase shifts or edge effects that need to be compensated for. Furthermore, since we are only considering an individual epoch in each trial, none of the standard methods for averaged data with a window extending both directions around the event of interest are available.

Implications of Closed-Loop Approaches for Neurophysiology

Closed-loop non-invasive brain stimulation with millisecond precision enables selective interference with ongoing brain activity and thereby can help to clarify important open questions in neurophysiology: How does the TMS pulse evoke cortico-spinal activity (Rossini et al., 2015)? Which oscillatory brain activity serves which functional role? Such a functional dissection of brain networks “by instantaneous state” adds an additional dimension along which the in vivo system can be separated into functional subsystems, similar to what has been achieved by pharmacological (Ziemann et al., 2015) and optogenetic interventions (for a review, see Grosenick et al., 2015).

A pioneering application of closed-loop brain-state triggered stimulation in a “behavior-in-the-loop” setting can be found in an animal experiment by Siegle and Wilson (2014) where freely behaving mice received a stimulus that was triggered by the phase of ongoing hippocampal theta oscillations during the performance of a memory task. Using this closed-loop design the authors could dissect both the role of phase and task segment (encoding vs. retrieval) for task-dependent information processing.

Nevertheless, the implicit physical environment of a freely behaving animal is difficult to capture, the sensory and motor interaction difficult to quantify, and the experimental control and opportunities for intervention limited; we therefore expect that an explicit “task dynamics” loop, in the form of an environment simulated on a computer, will be an increasingly relevant feature of closed-loop designs. Neuronal cell cultures have long been studied as an experimental system with such a view in mind (Shahaf and Marom, 2001; Chao et al., 2008; Keren and Marom, 2014; Potter et al., 2014) and the role of the agent-environment interaction (“relational dynamics”) has also been explored in human behavioral experiments (Marom and Wallach, 2011). These results provide suggestive avenues for the design of closed-loop non-invasive brain stimulation experiments to better understand the brain networks underlying the generation of behavior in human subjects.

Implications of Closed-Loop Approaches for Therapy

With regard to the therapy of neurological disorders due to “network dysregulation”, closed-loop paradigms can be applied in two different ways: firstly, a control-system theory approach uses the feedback stimulus as a “regulator” to re-“set” the local excitability and activity of a network (see Wallach, 2014 for an in vitro example). Secondly, a neuromodulatory approach uses a stimulus train to induce specific long-term plastic changes and thereby re-“wire” the connectivity of the network.

For instance, in a rodent model of generalized epilepsy closed-loop transcranial electrical stimulation based on a threshold mechanism of cortical local field potentials and unitary activity was shown to reduce epileptiform spike-and-wave episodes (Berényi et al., 2012). In humans, closed-loop deep brain stimulation (DBS) paradigms have recently been implemented for the treatment of advanced Parkinson’s disease by applying adaptive DBS whenever recording of local field potentials directly via the DBS electrodes exceeded a threshold in beta power, since exaggerated beta oscillations have been identified as a marker of akinesia and rigidity (Little and Brown, 2012; Little et al., 2013). Adaptive DBS was more effective than conventional open-loop DBS in improving Parkinsonian symptoms, at more than 50% reduction of stimulation time (Little et al., 2013). Likewise, surface EEG ERD over the motor cortex has been used to control DBS for the treatment of essential tremor (Herron et al., 2015).

Another important example for closed-loop stimulation is auditory stimulation during <1 Hz slow-wave sleep, a phase of sleep that is critical to declarative memory consolidation. It was demonstrated in healthy sleeping humans that auditory stimulation in phase with the ongoing rhythmic occurrence of slow oscillation up states measured in the EEG significantly enhanced the slow oscillation rhythm, phase-coupled sleep spindle activity, and in turn, the consolidation of declarative memory (Ngo et al., 2013). Stimulation out of phase with the ongoing slow oscillation rhythm remained ineffective with respect to all readouts.

In summary, EEG brain-state triggered NIBS or behavior-in-the-loop set-ups will enable physicians to interfere with their patients’ ongoing brain activity with high temporal, spatial and spectral precision. NIBS or behavioral neurofeedback can thus be coupled to endogenous brain activity in functionally defined brain networks in real time. This approach has several important advantages. Firstly, neuromodulation can be personalized to individual network function, that is, inter-individual differences in the excitability and connectivity of brain networks can be taken into account. Secondly, the time-course of dynamic changes during network reorganization such as during stroke rehabilitation (Grefkes and Ward, 2013) can be taken into account. Thirdly, since the modifiability of neurons and networks is a function of their recent activity (metaplasticity), which critically determines direction, extent and duration of plasticity in neural networks (Müller-Dahlhaus and Ziemann, 2015), and which can be used to time the stimulation appropriately using a closed-loop method. Implementation of closed-loop neuromodulation in a therapeutic setting will thus critically depend on characterization of informative EEG or MEG parameters for network function and plasticity.

Finally, behavior-in-the-loop set-ups can additionally capitalize on agency and subjectivity of brain-environment interactions. For instance, NIBS can be applied through the “brain dynamics loop” in synchrony to motor-sensory feedback from the “task dynamics loop” (see Figure 1E), reinforcing or interfering with the brain’s wired input-output processing in an agent-environment interaction. In this context, methodological advances of latest closed-loops systems as described above, i.e., low latency and low jitter, will prove essential to match the multiple natural time-scales of the environment that the brain is evolutionary tuned for. It appears promising to further develop this type of comprehensive combined closed-loop intervention in the treatment of neurological and psychiatric diseases.

Conclusion

Two different closed-loop interactions can be differentiated: a direct coupling to instantaneous brain states through non-invasive brain stimulation (“brain dynamics” loop), and a coupling to an environmental system that presents the brain with the opportunity to generate goal-directed behavior through the motor-sensory loop (“task dynamics” loop). These two approaches to closed-loop neuroscience are conceptually quite different but they are complementary in that they serve to induce and then interfere with specific brain states. Our assertion is that there is significant experimental and therapeutic potential in the application of brain-state dependent brain stimulation while the subject or patient is simultaneously performing a task, where both loops are interactively optimized for neuromodulatory efficacy.

Author Contributions

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.