Closed-loop neuro-robotic experiments to test computational properties of neuronal networks.

Information coding in the Central Nervous System (CNS) remains unexplored. There is mounting evidence that, even at a very low level, the representation of a given stimulus might be dependent on context and history. If this is actually the case, bi-directional interactions between the brain (or if need be a reduced model of it) and sensory-motor system can shed a light on how encoding and decoding of information is performed. Here an experimental system is introduced and described in which the activity of a neuronal element (i.e., a network of neurons extracted from embryonic mammalian hippocampi) is given context and used to control the movement of an artificial agent, while environmental information is fed back to the culture as a sequence of electrical stimuli. This architecture allows a quick selection of diverse encoding, decoding, and learning algorithms to test different hypotheses on the computational properties of neuronal networks.

Introduction

Many features of brain function are currently impossible to replicate in an artificial system. The brain’s ability to quickly process complex sensory information and to generate, in response, precise motor commands is by itself already beyond the current state-of-the-art. But its ability to adapt to different conditions by learning from past experience makes it so vastly superior to human-developed control systems. So far, attempts to replicate or exploit this plasticity have met little success, and the comprehension of the inner workings of the brain has eluded the grasp of researchers. One of the main issues while investigating the relationship between brain and behavior is the inability to properly access all of the variables in the system: ideally, an optimal experimental setup would allow simultaneous recording and stimulation to a large numbers of neurons, long-term stability, monitoring of synapses positions and weights, and controllable bi-directional interaction with the environment. The difficulty in tracking all those variables simultaneously led to the study of the brain-behavior relationship at two very different scales: either with behaving animals, with no fine control over experimental conditions 1-7 or with small, isolated parts, such as portions of neuronal tissue, with no overall view of the system 8. In the latter case, while no devised experimental setup allows the complete monitoring of all the parameters involved in the workings of even a simple neural network, a good trade-off is provided by dissociated neurons grown over Micro-Electrode Arrays (MEAs) 9. Those devices, born at the end of the 70's 10, have several advantages over traditional electrophysiology techniques: firstly, the possibility of recording and stimulating a neural network in many different locations at once (usually 60 electrodes). Furthermore, the coupling of MEAs with cells is almost non-invasive, allowing the observation of the same network for long periods of time, up to several months 11. The physiological effects of electrical stimulation on dissociated cultures have been extensively studied thanks to those devices, revealing that many properties observed at higher scales (such as, for example, plasticity and simple memory processes 12-14) are conserved despite the loss of architecture. During culture growth, those networks start showing spontaneous activity at about 7 days in vitro (DIV) 15,16. Network activity tends to change radically with further growth; first as single spikes gather into bursts (towards the end of the second week) 17, later as it changes into a highly complex pattern of synchronized, non-periodic network bursts 18, which represents the mature state of a network. It has been suggested 19 that this synchronous behavior, somewhat similar to that observed in in vivo recordings on sleeping animals, is caused by the lack of sensory input.

A different approach attempted to gain a better understanding of information coding has been taken by performing closed-loop experiments, in which different types of signals were used to control the stimulation of the neuronal network itself 11,20-23. In these experiments, an external agent capable of interaction with the environment has been used to generate sensory information fed to the neural network, which, in turn, produced motor commands for an effector mechanism. This allowed observations of how dynamic and adaptive properties of neural systems evolved in response to induced changes in the environment.

A setup to perform ‘embodied neurophysiology’ experiments was developed, where a wheeled sensor platform (a physical robot or its virtual model) moves about in an arena and its speed profiles are determined by the activity of a neuronal system (i.e., a population of rat neurons cultured over a MEA). The robot is characterized by the speed profiles of its two independently-controlled wheels and by the current readings of the distance sensors. The exact nature of the distance sensors is not relevant; they may be active or passive optical sensors or ultrasound sensors. Clearly, this issue does not apply in the case of virtual robots, in which sensors may be designed with any desired feature.

In the experiments herein described, the robot used is always the virtual implementation, with 6 distance sensors pointing at 30°, 60° and 90° from the robot heading in both directions. The activity of the three left and right sensors is averaged and the activity of the biological culture is driven by the information collected by such ‘super-sensors’ (which will just be referred to as ‘left’ and ‘right’ sensors in the rest of this work). The protocol described may actually be applied to the physical robot with fairly minor adjustments. The information collected by the robot (either physical or virtual) is encoded in a series of stimuli that are used to manipulate the activity of the biological neural network, which is physically separated by the robot. The stimuli themselves are all identical and thus do not code any information. What is relevant is their frequency: stimulation rate increases when the robot approaches an obstacle, with different delivery sites coding sensory information from the left and right ‘eyes’ of the robot. The neural network will present different responses to the incoming train of stimulations: the task of the decoding algorithm is to translate the resulting network activity into commands used to control the wheels of the robot. Given a ‘perfect’ network behavior (i.e., with reliable and totally separated responses to stimuli from different electrodes), this would result in the robot driving in its arena without hitting any obstacle. Most networks present a behavior very different from ideality, therefore a simple learning protocol is introduced: when activated, tetanic stimulation (brief spells of high-frequency stimulation, 20 Hz stimulation for 2 sec, inspired by protocols described in 24,25) following a collision with an obstacle is delivered. If the tetanic stimulation results in a local strengthening of network connectivity, this will result in a progressive increase in the navigational capabilities of the robot.

HyBrainWare2, an improved version of the custom software published in 26, is the core architecture developed to handle the control of the different devices of the system (stimulator, data acquisition, processing and visualization, robot communication or simulation). This software has been developed at our lab and is freely available on request. This software provides the interface with the data acquisition board: once the user starts data acquisition from the GUI, the software controls the acquisition board to start the sampling and A/D conversion of data coming from the recording electrodes. This data can then be recorded, displayed to screen or analyzed in real-time to detect spikes, according to the options set by the user (see Procedure section for details). Furthermore, within the software, the definition of coding (translation of sensory information into an electrical stimulation) and decoding (translation of recorded activity into motor commands for the robot) algorithms must be specified. In particular, our setup is relatively user-friendly compared to similar systems designed in the past 27, since almost all variables can be accessed by the user right before starting the actual experiment, while all the recorded information is automatically saved in a format compatible with a neural data analysis toolbox 28.

The following Procedure section describes a learning experiment on dissociated rat hippocampal cultures: all the culturing and experimental parameters are provided for this particular preparation and may need to be modified if a different biological substrate is to be used. Similarly, the described experiment takes advantage of the closed-loop architecture to investigate the learning effect of tetanic stimulation, but the architecture itself is flexible enough to be used in the study of different features of dissociated neural networks. Major variants of the proposed experiment are further explained in the Discussion section.

Plate neuronal cultures on MEA chips, as described 29. The description of a similar procedure is also provided 9 and in the Discussion section.

Turn the MEA heating system on 5-10 min before starting the recording to minimize the thermal stress experienced by the cells: set temperature target of the temperature controller to 37 °C and switch on the heating plate below the MEA itself (provided in most commercial MEA systems) and, if available, a heated cover in order to significantly reduce evaporation.

Sterilize gas-permeable caps before each use in an autoclave with humid-heat (30 min, 130 °C). To avoid evaporation and prevent changes in osmolarity, keep the cultures covered with caps during recording.

Place the neuronal culture in the amplifier 30 min before actually starting the recording. NOTE: Simply moving the cultures from the incubator to the amplifier will significantly disrupt spontaneous activity for about half an hour. During this time, culture medium temperature will stabilize.

If a carbair circuit is available, start circulation of carbair (a mixture of 5% CO2 and 95% O2 + N2) for experiments longer than a couple of hours: cultures will require it in order to maintain both oxygen and pH levels stable.

Select software filter bandwidth to detect spikes (i.e., Multi-Unit Activity – MUA) 30: in the RawDataDisplay form mark the ‘300 Hz-3 kHz’ checkbox .

Start the acquisition of data: press the ‘Start’ button in the RawDataDisplay form.

Set Threshold Gain for spike detection in the RawDataDisplay to 7. NOTE: Depending on the desired specificity/selectivity tradeoff and on the adopted detection algorithm, this threshold can be set between 6 and 10 times the computed standard deviation .

Set Threshold Memory to 2 sec and press the ‘Lock’ button when no spiking activity is visible on the display (both commands are in the RawDataDisplay form). Mark the ‘Estimate SD from median’ to compute noise standard deviation from the median of the absolute value of the observed signal 31,32, if it is difficult to provide even short time windows without spiking activity. Remove the checkmark from this option after pressing the ‘Lock’ button, as the underlying algorithm is computationally intensive and might cause the PC to lag.

Turn on spike detection routine (‘Spike Detection’ checkbox in the RawDataDisplay form). If spike detection is already running (i.e., the checkbox is already marked), discard spikes detected up to this moment by clicking the ‘Reset’ button in the Data Recording form.

Record spontaneous activity of the neuronal cells cultured over the MEA for 30 min: save data to file by clicking the ‘Record’ button, in the ‘Spikes’ box of the DataRecording form, after the desired amount of time has elapsed (30 min, in this case).

Identify the 10 most active channels (i.e., 10 channels with highest spike count), then select those channels in any of the MEA layouts (either in the Coding, Decoding or Connection Map forms) by dragging the mouse cursor over the desired areas. Once the channels are selected, right click anywhere on the MEA layout and select ‘Add to left sensory area’ in the pop-up menu: these electrodes will be used to deliver electrical stimulation in step 3.5.

Verify that stimulator and MEA amplifier are correctly connected: all configurations require two wires per desired stimulation channel, while an extra coaxial cable will be required to carry the synchrony signal (please refer to the manuals of the specifics instruments for connection diagrams). Proceed then to turn the stimulator on.

Define stimulus parameters in the Connection Map form. All the stimulations delivered to the culture are biphasic square voltage waves. Set half-duration to 300 µsec and amplitude to 1.5 Vpp33 . NOTE: Stimuli large enough to reliably evoke neural responses will very likely compromise the recording of activity from the same sites. In the rest of the paper, electrodes used for stimulation delivery should not be selected for recording any relevant information.

Record response to stimulation: press the Start button in the ConnectionMap form. A series of 30 stimuli, with an interval of 5 sec is, in turn, automatically delivered from each one of the selected electrodes, while responses are recorded from the remaining 59 electrodes.

Compute a connection map (i.e., the probability of observing a response from an electrode within a time slot after delivery of stimulus from a different electrode) for each stimulating channel through any math software or SpyCode, an application developed in the past (and freely available upon request) to perform computations on neural data 28.

From the connection maps, select best electrodes: discard all stimulating electrodes that did not evoke responses (i.e. in the time window following stimulation, firing rates are not significantly higher than during spontaneous firing).

Select, among the remaining electrodes, the pair with the least overleap in responses. Specifically for each stimulating electrode, compute the average spike count in each recording electrode, then compute the difference between corresponding electrodes for all stimulating electrodes pairs. Select the couple for which the sum of absolute values of response differences over all recording channels is highest.

Select one of these electrodes to code sensory information from the left side of the robot and the other to code readings from the right side: in order to do this, drag the mouse cursor over one electrode, right-click on the MEA layout then select ‘Add to left sensory area’ (or ‘Add to right sensory area’).

Set ‘Coding Type’ in the Coding form to Linear.

Define minimum and maximum stimulation rates in the Coding form. Use the default range of 0.5-2 Hz.

Set the ‘Jitter’ parameter in the Coding form to 0.

Set the decoding algorithm parameters in the Decoding Form (weight and extinction coefficient) to 1, for a moderately active culture (~1 spike/sec per channel). Define a new parameter pair if firing rates strongly and constantly deviate from this value. See discussion for the exact function of decoding parameters.

Set the decoding algorithm burst parameters in the Decoding Form. Set weight to 0 (decay time is then irrelevant), unless distinction between spikes and bursts is the subject of study. NOTE: The values suggested in step 4.4 will produce smooth robot movements and speeds compatible with robot reaction times for moderately active rat primary cultures. The burst parameters have exactly the same function as those described in the step 4.4, but the triggering event is the detection of a burst instead of a spike: the actual speed of each wheel is simply the sum of the contributions of detected spikes and bursts.

In the Virtual Arena Designer form, choose between bounded (the boundaries of the arena are visible to the robot and impassable) or boundless (if the robot exits from one side of the arena, it will instantly re-enter from the opposite one) arena and set the arena size in pixels.

Do not use bounded arenas smaller than 100 x 100 pixels, in order to allow significant movement. Do not use obstacles with a radius smaller than 5 pixels as they can simply fall between the lines of sight of the robot.

Keep in mind that very large arena sizes might cause computer performance to degrade: if a large arena is required, test software performance with the desired arena before starting hard-to-repeat experiments.

Set robot starting position either manually (click the ‘Manual Selection’ button in the Virtual Arena Designer form, then the desired location) or by specifying the coordinates of the robot at experiment start in the ‘Robot Starting Position’ fields.

Add any number of impassable obstacles within the arena. Either place them manually in the arena after clicking the ‘Add Obstacle Manually’ button or set number and size range.

Click the ‘Generate Arena’ button to generate the arena with the selected features. No changes will occur until this button is pressed.

Save the designed arena and load the relative file, before use in an experiment, with the command buttons in the lower part of the Virtual Arena Designer form.

Select as temporary recording electrodes those on which different responses, in step 3.8, have been observed: electrodes that showed larger responses following stimulation from the ‘left’ (or ‘right’) electrode will constitute the ‘left’ (or ‘right’) recording electrodes. Do not use electrodes that did not show significant responses to either stimulating electrode to control the robot. From the MEA layout pop-up menu, select ‘Add to left motor area’ (or ‘Add to right motor area’) to define recording electrodes.

Select features to record in the Experiment Manager form. At this stage, spikes and stimulation time stamps are the only relevant information.

Perform a 10 min test drive: start the robot run by clicking the Start Experiment button in the Experiment Manager form, click it again after 10 min have elapsed. NOTE: Once all the parameters have been set (coding and decoding, robot arena, stimulating and recording electrodes, features to record), a file name selection will ask for recorded files name and destination, then the robot will start moving in its arena, following the defined rules. All the selected features are automatically saved in real time.

Repeat steps 3.6 and 6.1 on data acquired in step 6.3, in order to select recording electrodes from data collected during an actual robot run (see Discussion for the rationale of this two-step approach).

Select in the Experiment Manager form the data to be recorded: mark the Spike, Robot and Stimuli Data checkboxes.

Launch a pre-learning robot run: click the ‘Start Experiment’ button in the Experiment Manager form. Select new file names for data files when prompted. When 30 minutes have elapsed, click the ‘Start Experiment’ button again to stop the robot run.

Switch on the learning protocol (mark the ‘Deliver Tetanic Stimulation After Hit’ check box in the Experiment Manager form) and perform the training robot run, of the same length as the pre-training phase (i.e., 30 min). Remember to input different file names when prompted to avoid overwriting data from step 7.2.

Switch off the learning protocol again (unmark the ‘Deliver Tetanic Stimulation After Hit’ check box) and perform the post-learning robot run. Once again, remember to change file names to prevent overwriting.

Repeat steps 3.1-3.6. Use data from these recordings to test if any changes have been induced either in the spontaneous or evoked activity patterns by the learning protocol.

Representative Results

The developed experimental framework allowed the testing of whether an exchange of information between a culture of neurons and the virtual realization of a physical robot is possible 34. Figure 1 shows several samples of obtained paths travelled by a virtual robot in 20 min experiments in different conditions: from left to right a proper closed-loop experiment, an ‘empty MEA’ robot experiment (no cells are plated on the MEA for this control experiment) and a open-loop robot experiment (stimulation rate is constant instead of coding sensor information) are represented. The representative traces confirm that a bi-directional interaction between the neuronal and artificial elements is necessary in order to obtain good navigation performances of the robot. Nevertheless, the robot experiences several hits against the obstacles.

In Figure 2, the navigation performance of the robot, expressed as pixels travelled between subsequent hits, is shown in different conditions. The first two columns display the distribution of distances travelled in the control experiments mentioned above (‘empty MEA’ and open-loop configurations), while the third and fourth columns display the performance without and with, respectively, the delivery of tetanic stimulation following each hit against an obstacle. The introduction of the tetanic stimulation (Cf. Procedure, point 7.2) improves significantly the distance travelled between two consecutive hits, thus improving the navigation performances of the robot 35.

In Figure 3, the navigation performance of a robot with different decoding conditions is presented. To this end, a different arena configuration has been adopted. This helped to quantify the successful navigation of the robot within the arena 36: as described in step 4.1 of the procedure, the robot is presented a series of short tracks. The success rate is simply the ratio of the number of successfully crossed tracks over the number of presented tracks. In particular, during the experiments a real-time identification of burst and isolated spikes was performed. The decoding paradigms differ from one another because of the relative weights of bursts and isolated spikes (Cf. Procedure, points 3.5-3.6 and Discussion).

Figure 1: A neuronal network and a robot have a bi-directional exchange of information. This figure shows three representative paths travelled by the robot over 20 minutes of experiment. In particular, light green areas are free for the robot to move in, whereas dark green pixels represent impassable obstacles that the robot can perceive through its distance sensors. In each trial, the robot starts in the upper left section of the arena and travels to its final position, depicted as a large pink dot. The smaller black dots represent hits against an obstacle, while the color-coded path provides an indication of the time evolution of robot movement. The color bar on the right expresses time elapsed from the beginning of the experiment in minutes. (A) Path followed during a closed-loop experiment. (B) Path followed during an ‘empty’ MEA experiment (no cells are cultured on the MEA surface). (C) Path followed during an open-loop experiment (stimulation rate is constant throughout the experiment). Please click here to view a larger version of the figure.

Figure 2: Performance results are affected by closed-loop and tetanic stimulation. This graph reports the distributions of distances covered by the robot between subsequent hits in different conditions. In particular, the first two distributions represent control experiments (‘Empty’ for recordings with no neurons plated on the MEA, ‘Open loop’ (OL) for experiment with a ‘blind’ robot). The last two columns represent the ‘Closed loop’ condition without (CL) and with (CL + TS) the learning protocol implemented through Tetanic Stimulation (TS), being delivered to the culture following hits against an obstacle. In each box, the central horizontal segment represents the median of the distribution, the empty square the mean value, the central bar extends to the first and third quartile and the whiskers extend to the 5th and 95th percentiles. Outliers are represented as diamonds. Statistics have been performed using the Kruskall-Wallis one way analysis of variance on ranks: the Student-Newman-Keuls pairwise comparison reveals that all median values are significantly different with p < 0.05.

Figure 3: Decoding influences robotic performance. The graph above represents the probability, for a given decoding algorithm, of the robot to navigate successfully through a short track in a limited amount of time. Identification of bursts and isolated spikes is performed in real time during the experiment itself. In the first case (‘Spikes’) all detected spikes present the same relative weight, for the second and third distributions a weight of 0 was set for, respectively, isolated spikes and spikes belonging to a burst. The last two columns represent the results obtained in the case where all the spikes are accounted for, but with a different relative weight according to their position. In particular in the fourth column isolated spikes were given higher relative weight than burst events, while the weighting is inverted for the decoding relative to the fifth column data. In each graph, the central line represents the median of the distribution, the empty square the mean value, while the central bar extends to the first and third quartile and the whiskers extend to the 5th and 95th percentiles. Statistics have been performed using the Kruskall-Wallis one way analysis of variance on ranks. The pairwise comparison (Student-Newman-Keuls method,) reveals that significant differences (p <0.05) can be observed between columns 1 and 4, 2 and 4, 2 and 5, 2 and 3.

Discussion

In this paper a neuro-robotic architecture based on a neural controller (i.e., a network of neurons coming from the cortex or the hippocampus of embryonic rats), bi-directionally connected to a virtual robot, is presented. The robot, which has sensors and wheels, is forced to move in a static arena with obstacles and its task consists of avoiding collisions.

The first and possibly most critical aspect of the described procedure is the preparation of the cultures themselves, as the failure rate will tend to be significant even in the best of technical conditions. A detailed description of culturing techniques is, however, outside of the scope of the present work. As a general guideline, recordings should occur when the network firing rate reaches a stable level, usually after 3 weeks in vitro. A rough indication of healthy cultures is the presence of spontaneous electrophysiological activity in several recording channels (at least 20 channels over 60 available). Such cultures are characterized by a high degree of neuronal inter-connectivity. Under such conditions, neural activity usually becomes strongly synchronized and, at times, it displays epileptiform events, with intense spiking followed by minutes-long silent periods 37. Both of these features present a problem: excessive synchronization will make it impossible to distinguish responses to stimuli from different electrodes, while cultures exhibiting epileptiform activity will respond to the first stimulus presented with a long outburst of activity, followed by a silent period, regardless of any successive stimuli delivered. Both these issues can be strongly ameliorated by the employment of patterned cultures 35, in which the neural population is split into two or more weakly interconnected sub-groups.

Another issue is that neural responses strongly depend on the distribution of incoming stimuli 38,39. In a closed-loop experiment, the delivered stimulation is a function of the sensor readings, which, in turn, are a consequence of the robot movement and therefore of the neural responses themselves. This implies that there is no easy way of establishing beforehand what responses will be observed during the actual experiment. As a result, the selection of input-output electrodes must rely on successive approximations. In the described protocol, a two-step process (namely, steps 5.5 and 6.4) is implemented to try to determine a connection map. In the first step, a regular sequence of stimulations is delivered and responses to such stimuli are used to derive a first connection map and a temporary set of recording channels. This configuration is then used to perform the test drive described in step 6.4 and select the recording channels that will be monitored during the actual experiment.

In the Results section, a representative navigation result and the immediate improvements brought about by the introduction of a learning paradigm are presented. Throughout the protocol description, several other possible experimental variants are mentioned. For instance, the two major coding systems implemented in HyBrainWare2 (linear and stochastic) where developed to investigate the exact role of temporal input variability on the neural code 38. In the linear case, instantaneous stimulation rate is a function of user-defined parameters and robot sensors recordings. In the stochastic case, each time instance has a given probability to be chosen to deliver a stimulus. Such probability is automatically computed by HyBrainWare2 so that the expected stimulation rates match that of the former case. The possibility to add jitter to linear coding provides a smooth transition between the two cases described above. In the same way, exploring different combinations of parameters in the decoding section might help shed light on the precise role of bursting in in vitro neural networks. The speed of each wheel of the robot increases proportionally to the weight parameter each time an event is detected in the corresponding output area, while the extinction coefficient indicates how much time, in seconds, one of the contributions takes to lose 50% of its value. The decay is a simple exponential. Those variants have already been taken into account into the current design of HyBrainWare2, but many more research possibilities are open if additional modifications in the software or experimental setup can be introduced.

A rather significant limitation of the protocol here described is the requirement of the developed custom software, HyBrainWare2 (freely available upon request to all interested users). This software has been designed for a set of devices (stimulator, acquisition board, MEA amplifier) of a specific model and manufacturer. While it is indeed possible to adapt it to work on different setups, the conversion will take some programming skill. Similarly, the included options cover only a limited set of all the experimental questions that could be investigated through such a setup. For example, the spike detection algorithm implemented in the presented architecture (precise time spike detection40) is completely defined by several hard-coded parameters that need to be changed if the subject of registration is very different from dissociated neural networks (e.g., cardiac cells or slices). Finally, the learning protocol consists of the delivery of a short, high frequency (2 sec of 20 Hz stimulation, each stimulus pulse is the same as those used to code sensory information) stimulation following each obstacle hit. If the robot hits an obstacle with its right side, the tetanic stimulation is delivered to the electrode that usually codes for right side information and the same holds true for left side hits. This protocol is hard-coded and cannot be changed by the user without modifying exiting software.

While the setup presented here is not the first embodied system for hybrid, closed-loop experiments 20,23,27,41,42, the ones designed in the past were focused on a single thesis supported by data from a limited number of analogous preparations. On the other hand, the described setup has been used for a large number of experiments (more than 100 cultures have been recorded since 2012) with preparations differing for modularity and origin, while the experiments themselves addressed different issues (e.g. impact of tetanic stimulation and relevance of bursting, as presented in the results). In the near future, new experimental sessions are foreseen to verify the lasting effects of both tetanic and closed-loop stimulation and the relationship between stimulation regularity and observed responses. Another issue to be addressed is the link between spontaneous and observed activity, with the idea of adding a new decoding algorithm that takes into account the history of past activity and stimulation 43.

Disclosures

The authors declare they have no competing financial interests.