Active tactile exploration using a brain-machine-brain interface.

Brain-machine interfaces use neuronal activity recorded from the brain to establish direct communication with external actuators, such as prosthetic arms. It is hoped that brain-machine interfaces can be used to restore the normal sensorimotor functions of the limbs, but so far they have lacked tactile sensation. Here we report the operation of a brain-machine-brain interface (BMBI) that both controls the exploratory reaching movements of an actuator and allows signalling of artificial tactile feedback through intracortical microstimulation (ICMS) of the primary somatosensory cortex. Monkeys performed an active exploration task in which an actuator (a computer cursor or a virtual-reality arm) was moved using a BMBI that derived motor commands from neuronal ensemble activity recorded in the primary motor cortex. ICMS feedback occurred whenever the actuator touched virtual objects. Temporal patterns of ICMS encoded the artificial tactile properties of each object. Neuronal recordings and ICMS epochs were temporally multiplexed to avoid interference. Two monkeys operated this BMBI to search for and distinguish one of three visually identical objects, using the virtual-reality arm to identify the unique artificial texture associated with each. These results suggest that clinical motor neuroprostheses might benefit from the addition of ICMS feedback to generate artificial somatic perceptions associated with mechanical, robotic or even virtual prostheses.

BMIs have evolved from one degree-of-freedom systems [3] to many-degree-of-freedom robotic arms [4] and muscle stimulators [5] that enact complex limb movements, such as reaching [6-8] and grasping [9]. However, somatosensory feedback, which is essential for dexterous control [10-12], remains underdeveloped in BMIs. With the exception of a few studies combining BMIs with tactile stimuli applied to the body [13], existing systems rely almost exclusively on visual feedback. Prosthetic sensation has been studied in the context of sensory substitution [14] and targeted reinnervation [15], however these approaches have limited application range and channel capacity. To provide a proof-of-concept method for sensorizing neuroprostheses, we implemented a BMBI that extracts movement commands from the motor areas of the brain while delivering ICMS feedback in somatosensory areas [1,2,16] to evoke discriminable percepts [17-20]. This idea received support from our pilot study [16] in which a monkey responded to ICMS cues with the movements of a BMI-controlled cursor. However the ICMS cue did not provide feedback of object-actuator interactions in this previous demonstration.

The BMBI developed here enabled active tactile exploration [21] during BMI control (Fig. 1a). Two monkeys (M and N) received multielectrode implants in M1 and S1 (Fig 1b). They explored virtual objects with either a computer cursor or a virtual image of an arm (Supplementary Fig. 1a,b). In hand control (HC), the monkeys moved a joystick with their left hands to position the actuator. They searched through a set of virtual objects, selected one with a particular artificial texture (AT) conveyed by ICMS, and held the actuator over that object to obtain reward (Fig. 1a; Supplementary Fig. 1c,d). During brain control (BC), the joystick was disconnected and the actuator was controlled by the activity of right-hemisphere M1 neurons [9,22,23]. The behavioural tasks varied in the number of objects on the screen, ATs employed, and the actuator type (Fig. 2a) and were more difficult than previously reported BMI tasks because of the presence of multiple objects in the workspace, a prolonged object selection period, and the necessity of interpreting ICMS feedback.

Figure 1

The brain-machine-brain interface

a, Movement intentions are decoded from M1; artificial tactile feedback is delivered to S1. b, Microwires were implanted in M1 and S1. c, Microwires used for ICMS in monkey M are accented in red. d, Actuator movements for a trial where monkey M explores the UAT object but ultimately selects the RAT object. Gray bars indicate stimulation patterns; insets indicate the ICMS frequency. e, Rastergram of M1 neurons recorded during the trial shown in panel d.

Figure 2

Learning to use ICMS feedback

a, Behavioural tasks. b, c, Performance of monkey M (72 sessions). Circles (b) depict fraction of correctly performed trials. Open circles indicate chance performance. Curves are lines of best fit. Asterisk indicates session used for psychometric measurements. Squares, triangles and crosses (c) represent mean times spent in RAT, UAT, and NAT, respectively. Black symbols indicate HC; red, BC. d, Intrasession performance for monkey M. Curves represent averages for BCWOH (main panel, 3 sessions), HC (inset, 12 sessions) and BCWH (inset, 12 sessions). Lines are best linear fits.

ICMS was delivered through two pairs of microwires to the hand representation area of S1 in monkey M, (Fig. 1c) and through one pair of microwires to the leg representation in monkey N. Each AT consisted of a high-frequency pulse train presented in packets at a lower secondary frequency (Fig. 1d; Supplementary Fig. 2a). The rewarded AT (RAT) consisted of 200 Hz pulse trains delivered in 10 Hz packets. The comparison ATs were represented by 400 Hz pulse trains delivered in 5 Hz packets (unrewarded artificial texture, UAT) or by an absence of ICMS (null artificial texture, NAT).

The major challenge solved here was the real-time coupling of ICMS feedback with the BMI decoder. As ICMS artefacts masked neuronal activity for 5–10 ms after each pulse (Fig. 1d, e), we multiplexed neuronal recordings and ICMS with a 20 Hz clock rate (Supplementary Fig. 2a). The interleaved intervals proved adequate for online motor control and artificial sensation—a result that was not clear a priori since S1 stimulation could have affected M1 processing through the connections between these areas.

BMBI performance improved with training. In Task I (Fig. 2a–I), monkey M surpassed chance performance after nine sessions; monkey N after four (P < 0.001, one-sided binomial test). Improvement continued with more difficult tasks (Tasks II–V) (Fig 2a,b; Supplementary Fig. 3a). In particular, the time spent exploring unrewarded ATs decreased (Fig. 2c; Supplementary Fig. 3b). Additionally, performance improved within daily experimental sessions (Fig. 2d). Psychometric analysis of RAT stimulation amplitudes indicated that at least 8 nC per ICMS waveform phase (100 μs wide current pulses of 80 μA) was needed for the AT discrimination (P < 0.001, one-sided binomial test). Performance was at chance level for catch trials (in Task II), where ICMS was not delivered (P = 0.90, one-sided binomial test).

The statistics of object exploration intervals (total time spent over a particular object on a given trial) indicated that the monkeys uniquely discriminated each AT type (Fig. 2c; 3a,c) and interpreted ICMS within hundreds of milliseconds—a timescale comparable to the discrimination of peripheral tactile stimuli [24,25]. Early in Task I, exploration intervals were equal for RAT and NAT (P > 0.5, Wilcoxon signed-rank test) and, with training, became longer for RAT and shorter for NAT (Tasks I–II) and UAT (Tasks III–V). During HC, the mean interval was longest for RAT (monkey M: 1,396 ± 21 ms; monkey N: 1,165 ± 15 ms; mean ± s.e.m.), shortest for NAT (304 ± 8 ms; 300 ± 10 ms), and intermediate for UAT (452 ± 13 ms; 402 ± 14 ms) (P < 0.01, ANOVA). During BC, intervals spent exploring NAT (498 ± 15 ms; 587 ± 25 ms) and UAT (685 ± 20 ms; 764 ± 32 ms) were longer as compared to HC, but still shorter as compared to RAT (1,420 ± 28 ms; 1,398 ± 55 ms) (P<0.01, ANOVA).

Figure 3

Statistics of object exploration

a, Object exploration intervals during HC and BC (inset) for monkey M (HC: n=1809 trials, BC: n=1355 trials). b, State transition diagrams for monkey M indicating the probability of reaches between RAT, UAT, and NAT after the first (left sub-panel) or second (right sub-panel) reach. Black labels indicate HC; red, BC. Line thickness is proportional to transition probability. c, Same as panel a, but for monkey N (HC: n=808 trials, BC: n=729 trials). d, Same as panel b, but for monkey N.

Additional hallmarks of active exploration were seen in the conditional probabilities of selecting different ATs (Figure 3b,d). During HC trials, the monkeys stayed over the first encountered AT (arrows that loop back to the same AT in Fig. 3b,d) with high probability if it was RAT (P=0.70 for monkey M and 0.76 for monkey N), but with low probability if it was UAT (0.05 and 0.01) or NAT (0.0 and 0.0) (Fig. 3b,d, left). After examining the second AT, the monkeys could identify the correct AT either by apprehending it directly or through a process of elimination. This follows from the increase in the probability of moving to RAT from NAT or UAT from chance to approximately 0.7 and the decrease in the probability of revisiting UAT or NAT to approximately 0.2 (Fig. 3b,d right). Similar effects were observed for BC (Fig. 3e, red text).

BC started in Task IV. During BC with hand movements (BCWH), the monkeys continued to hold the joystick although it was disconnected [16,22]. During brain control without hand (BCWOH) [9,22], the joystick was removed. In monkey M, with more than 200-recorded neurons, performance was less accurate during BCWH (73.75 ± 3.00%, mean ± s.e.m.) than during HC (91.48 ± 1.20%). In Monkey N with 50-recorded neurons, performance dropped further (50.37 ± 3.74 % versus 91.45 ± 1.91%), but still significantly exceeded the 33% chance level. M1 neurons exhibited directionally tuned modulations (Supplementary Fig. 5; 6) that were retained across different interfering ICMS patterns both during HC (Supplementary Fig. 4a,b) and BC (Fig. 4a,b).

Figure 4

M1 modulations during active control versus passive observation

a, b, Average BC movements for monkey M (n=294 trials) towards the RAT, UAT and NAT objects appearing on the left side off the screen (a) and corresponding neuronal modulations (b). Red trace is actuator; dashed black, joystick. Only trials with subsequent counter-clockwise reaches are included in the middle and right sub-panels. Red vertical lines indicate object onset. Firing rate normalized by s.d. c, d, Position of the actuator actively controlled (c) and passive observed (d) by monkey M. Black trace is joystick; red, offline predictions.

In BCWOH, task requirements were eased: the object selection period was reduced to 300–500 ms and monkeys were allowed to overstay at an incorrect object. Performance for monkey M, measured as the number of rewards per minute, steadily improved from 1.021 ± 0.007 to 2.962 ± 0.005 (mean ± s.e.m.) (Fig. 2d). Similar improvements were observed for HC and BCWH (inset in Fig. 2d). The average frequency of actuator displacements, calculated from power spectra, was correlated with the improvement in performance during BCWOH (R2=0.16 for the X-coordinate and R2=0.26 for the Y-coordinate, P<0.001, F-test), which indicated that the monkey modulated its brain activity to scan the targets faster. This behaviour was not random, as the exploration interval for NAT (3,620 ± 350 ms, mean ± s.e.m.) was significantly shorter (P<0.02, Wilcoxon rank sum test) than for UAT (4,270 ± 310 ms). The exploration of RAT (2,255 ± 94 ms) was the shortest due to the reduced selection period. For monkey N, BCWOH performance (2.084 ± 0.085 rewards per minute) did not change within sessions, and the differences in exploration intervals were not significant.

In agreement with others [26-30], we observed that M1 neurons represented the movements of the actuator even when it was passively observed by the monkey (Supplementary Fig. 7). Actuator movements (task V) replayed for the monkeys could be reconstructed from M1 activity, using a separately trained decoder (Fig. 4d), with similar accuracy to reconstructions made for HC (Fig. 4c). M1 representation of the passively viewed actuator is consistent with our suggestion that a neuroprosthetic limb might become incorporated in brain circuitry [1].

Our BMBI demonstrated direct bidirectional communication between a primate brain and an external actuator. As both the afferent and efferent channels bypassed the subject’s body, we propose that BMBIs can effectively liberate a brain from the physical constraints of the body. Accordingly, future BMBIs may not be limited to limb prostheses, but may include devices designed for reciprocal communication between and among neural structures and with a variety of external actuators.

METHODS SUMMARY

All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. Two rhesus monkeys were implanted with micro-wire arrays in both hemispheres. These implants were used for both recordings and ICMS (symmetric, biphasic, charge-balanced pulse trains; 100–200 μs, 120–200 μA). Monkeys manipulated a joystick to produce reaches with an actuator (computer cursor or a virtual reality arm) towards up to three objects displayed on a computer monitor. The task required searching for the object with particular artificial tactile properties. Objects consisted of a central response zone and a peripheral feedback zone. Artificial tactile feedback was delivered when the actuator entered the feedback zone and continued in the response zone. Monkeys held the actuator over the correct object for 0.8–1.3 s to receive a fruit juice reward. Holding over an incorrect object cancelled the trial. In brain control trials, the actuator was controlled by cortical ensemble activity decoded using an Unscented Kalman Filter [23]. An interleaved scheme of alternating recording and stimulation subintervals (50 ms each, 50% duty cycle) was implemented to achieve concurrent afferent and efferent operations. In all offline analyses, ICMS periods were excluded from calculations of neuronal firing rates. The virtual reality arm was animated using Motion Builder (Autodesk, Inc., San Rafael, CA).

METHODS

Subjects and implants

Two adult rhesus macaque monkeys (Macaca mulatta) participated in this study. Each monkey was implanted with four 96-micro-wire arrays constructed of stainless steel 304. Each hemisphere received two arrays: one in the upper and one in the lower limb representation areas. These array sampled neurons in both primary motor (M1) and primary sensory (S1) cortex. We used recordings from the right hemisphere arm arrays in each monkey, since both manipulated the joystick with their left hands. Within each array, micro-wires were grouped in two 4-by-4 uniformly-spaced grids of 16-triplets of electrodes. The separation between triplets of electrodes was 1 mm. The electrodes of each triplet had three different lengths, staggered at 300 μm intervals. The penetration depth of each triplet was adjusted with a miniature screw. After adjustments during the month following the implantation surgery, the depth of the triplets was fixed. The longest electrodes in all triplets protruded to 2 mm in length measured from the cortical surface.

Tasks

The monkeys were trained to manipulate a computer cursor or a virtual reality arm and to make reaches towards objects displayed on a computer monitor. The objects were visually identical, but exhibited different tactile properties as conveyed by ICMS of S1. In manual control, each trial commenced when the monkey held the joystick with their working hand. Then, a target appeared in the centre of the screen. The monkey had to hold the actuator (cursor or virtual reality monkey arm) within that centre target for a random hold time uniformly drawn from the interval 0 to 2 s. After this, the central target disappeared and was replaced by a set of virtual objects radially arranged about the centre of the screen. Each of these consisted of a central response zone and a peripheral feedback zone, distinguished by their shading (Supplementary Fig. 1c). Tactile feedback was delivered in the feedback zone or corresponding response zone. For monkey M, the radius of the response zone varied from 1.5 to 4.0 cm and the radius of the feedback zone varied from 4.5 to 7.25 cm, across all tasks and sessions. For monkey N, the radius of the response zone varied from 1.5 to 4.5 cm and the radius of the feedback zone varied from 4.75 to 9.5 cm, across all tasks and sessions. A trial was concluded when the monkey placed the actuator within the response zone for a hold interval (800 to 1300 ms for HC, depending on the session; 300 to 500 ms for BC) or the monkey released the joystick handle (in manual control trials). The next trial could commence after an inter-trial interval (500 ms). The sequence of events was the same during brain control trials. In some brain control sessions, the joystick was removed from the behavioural setup. For these, each new trial commenced following the previous inter-trial interval without the requirement for the monkey to hold the joystick. For Tasks I through III, monkeys chose from a set of two objects. For Task I, the monkeys had to choose between RAT and NAT for fixed object locations. For Task II, RAT and NAT were presented on the screen at different angular locations on each trial. For Task III, object number and spatial arrangement were the same as in Task II, but RAT and UAT were used. For Task IV, three objects were used (RAT, UAT and NAT), whose arrangement on the screen varied from trial to trial. Finally, for Task V, the virtual reality monkey arm replaced the computer cursor.

Psychometric measurements

Psychometric measurements determined the minimum ICMS amplitude that the monkeys could discriminate. In these measurements, the ICMS amplitude was different on every trial. In each psychometric session a range of amplitudes was selected so that about half were in a range clearly above the monkeys’ threshold for discrimination and half were in a range of unknown discriminability.

Catch trials

In some sessions, a small percentage of trials (typically 1%) were designated for catch trials. For these trials, the microstimulator delivered pulse trains with zero amplitude, however all other aspects of the behavioural task remained the same. This allowed us to confirm that there were no unintentional sources of information that the monkeys could use to perform the tasks.

Algorithms

An Nth-order Unscented Kalman filter (UKF) [23] used for BC predictions. Up to a 10th-order UKF was used in some sessions, but in most sessions we found that the 3rd-order UKF was sufficient. The filter parameters were fit based on the hand movements of the monkeys while they performed the task using a joystick or based on passive observation of actuator movements while the monkeys’ arms were restrained.

ICMS

Symmetric, biphasic, charge-balanced pulse trains were delivered in a bipolar fashion across pairs of microwires. The channels selected had clear sensory receptive fields in the upper limb (monkey M, two pairs of microwires with synchronous pulse trains) or lower limb (monkey N, one pair of microwires). For monkey M, the anodic and cathodic phases of stimulation had a pulse width of 105 μs; for monkey N the pulse width was 200 μs. The anodic and cathodic phases of the stimulation waveforms were separated by 25 μs.

Interleaved ICMS and recordings

We implemented an interleaved scheme of alternating recording and stimulation intervals (Supplementary Fig. 2a). Our BMI had a 10 Hz update rate. That is, 100 ms of past neural data are used to make predictions about the desired state of the actuator. We broke up each 100 ms interval into two 50 ms sub-intervals. In the first sub-interval (Rec), neural activity was recorded as usual and the measured spike count was used to estimate the firing rate for the whole 100 ms interval. The second sub-interval (Stim) was reserved exclusively for delivering ICMS; all spiking activity occurring in this subinterval was discarded. Whenever the actuator was in contact with a virtual object at the start of a Stim interval, an ICMS pulse train was delivered. For RAT, nine pulses of ICMS were delivered; For UAT, 18 pulses of ICMS were delivered; for NAT, no pulses of ICMS were delivered. The neural activity in the Stim interval was discarded even in the case of NAT, so that there would be no bias induced by ICMS-occluded neural data.

Virtual reality monkey arm

In Task V, we introduced a novel brain-controlled virtual reality arm with realistic kinematic movements and spatial interactions. The control loop rate was 50 Hz, with visual refreshing at 30 Hz. The arm model was designed to depict a rhesus conspecific. We presented a first person perspective of the virtual reality arm to the monkey, who controlled the position of the hand. Arm posture was controlled using a mixture of direct control of end effectors and inverse kinematics, constrained by the physical interdependencies of the joints.