Perceiving invisible light through a somatosensory cortical prosthesis.

Sensory neuroprostheses show great potential for alleviating major sensory deficits. It is not known, however, whether such devices can augment the subject's normal perceptual range. Here we show that adult rats can learn to perceive otherwise invisible infrared light through a neuroprosthesis that couples the output of a head-mounted infrared sensor to their somatosensory cortex (S1) via intracortical microstimulation. Rats readily learn to use this new information source, and generate active exploratory strategies to discriminate among infrared signals in their environment. S1 neurons in these infrared-perceiving rats respond to both whisker deflection and intracortical microstimulation, suggesting that the infrared representation does not displace the original tactile representation. Hence, sensory cortical prostheses, in addition to restoring normal neurological functions, may serve to expand natural perceptual capabilities in mammals.

Most sensory neuroprosthetic systems aim to restore function within the same modality as a pre-existing sensory deficit. For instance, cochlear implants restore auditory function[1], and stimulation along the visual pathway can restore visual function[2,3].

In some cases, however, it might be necessary to augment, rather than simply restore, the usual function of a sensory area[4,5]. For instance, in people that suffer permanent damage to the visual cortex, we might ask the somatosensory cortex to take over some of the roles of the visual system. Such cross-modal plasticity has been demonstrated in juvenile animals.[6,7,8] For example, rewiring experiments in newborn ferrets have shown that when visual inputs are rerouted to the auditory cortex, the auditory cortex acquires many anatomical and functional properties of the visual cortex,[7,8] and even mediates visually-guided behavior.[7,8]

Such results suggest that the function of a primary sensory area can dramatically change depending on the type of input it receives from the environment[9,10]. To date, however, the literature lacks a clear demonstration of such functional plasticity in normal adult mammals. This would require that the adult brain be plastic enough to extract novel information embedded within pre-existing representations, and use this information to generate appropriate behaviors.

In the present study, we test whether adult rats can incorporate a novel sensory modality into their perceptual repertoire. Specifically, we examined whether adult rats could learn to discriminate among IR sources after we coupled the output of a head-mounted IR-detector to electrical microstimulators in the whisker representation of S1. We discovered that, after training with this sensory prosthetic device, adult rats learned to navigate their IR world as if they had acquired a novel distal sensory modality.

Results

Behavioral Performance

We initially trained six rats on a simple visual discrimination task. Rats were placed in a circular chamber that included three reward ports (Figure 1a, 1b). On each trial, a visible LED was activated in one of the ports, and rats were rewarded with water for poking their nose in that port. Once they reached criterion on this task (70% correct, after 25±5 days (mean±sem)), we surgically affixed an infrared detector to the rat’s head, and implanted stimulating microelectrodes in the whisker region of S1 cortex (Figures 1c, 1d).

Figure 1

Methods

a. Setup of the IR behavior chamber. Three reward ports line the walls of the circular chamber. The proximity of the ports to one another is indicated by the angle θ. The rat has an IR detector (red) affixed to its head, and the cone emanating from the detector represents the area within which it will respond to IR stimuli. The red lines emanating from port 3 represent the IR signal emitted from that IR source. b. Arrangement of each reward port, which includes a recess with a water spout, an infrared LED above, and a broad-spectrum visible LED below. c. Design of stimulating electrodes (see Methods for details). The inset shows how each stimulating electrode pair is configured in the array (scale bar 300 µm). d. Example of electrode placement. Cytochrome oxidase-stained barrel field shows the location of four stimulating electrodes. Each penetration is indicated with a red asterisk. Reference line: 500 um. e. Coupling IR-levels with ICMS. On each trial, the IR light turns on, which activates the infrared detector that is mounted on the rat’s head. Processing converts the detected infrared level into a stimulation frequency. This value is sent to the microstimulator which produces the desired current pulses. The inset on the right-hand-side illustrates the structure of each biphasic waveform in the pulse train.

After this surgical procedure, we returned the animals to the same chamber, where they had to learn to perform the same task using IR light, which is invisible to rats (see Methods for details on the task and training)[11]. To allow our instrumented rats to perceive the IR light levels, the value of the IR detector output was converted into a pattern of electrical stimulation of S1, with stimulation frequency updated every 50ms depending on the IR intensity detected (Figure 1d). Electrical stimulation frequency increased as rats approached, or oriented their heads toward, an IR source (see Supplementary Movie 1). Note that such electrical stimulation in S1 is known to induce some type of tactile sensation in humans[12] and monkeys[13].

It took 26± 6 days for all six implanted rats to learn to discriminate among the IR sources at or above the criterion used in the initial visual task (>70% correct). While training with their new IR gear, rats underwent clear changes in behavioral strategy. At first, they did not associate ICMS with the task, and would poke randomly in the reward ports, occasionally scratching their faces in response to microstimulation. Eventually they learned to actively forage through the behavior chamber, sweeping the IR sensor on their heads back and forth to sample their infrared world (see Supplementary Movies 2–4).

Quantitatively, the rats’ performance increased from 41±6 to 93±2% correct during learning (Figure 2a), and their best single-session performance averaged 95±3% correct. Their behavioral latency on correct trials (the time between stimulation onset and poking in the correct reward port) decreased significantly as they improved at the task (latency dropped from 2.3±0.01 to 1.3±0.03 seconds; Figure 2b; r=−0.71; p=1.9017×10−11 (t-test)).

Figure 2

Rats discriminate among infrared sources using graded stimulation frequencies

a. Learning curve for IR-only trials. Graph shows percentage of correct trials as a function of session number (130 sessions in four rats). Black circles/lines indicate mean/sem for blocks of three sessions. b. Latency decreases as rats learn the task. Scatter plot of latency on IR-only trials, using same conventions as panel A (data are from 66 sessions in two rats). c. Discrimination performance varies with angle between ports. Plot shows percentage of correct trials versus task difficulty (angle between ports). Data are from 100 sessions in two rats. d. Performance degrades when stimulation frequency is constant rather than variable. Plot on the right shows performance on trials in which the stimulation frequency is variable (top left inset) versus held constant (bottom left inset). Data are from 34 sessions in two rats. e. When stimulation is turned off, performance is abolished. Left column shows performance on trials with IR-light turned on, but no stimulation. The right column shows the performance from the same sessions for the trials in which we stimulated as usual. Error bars are sem. Data are from 11 sessions in two rats.

We performed a series of additional psychophysical tests on the new modality. First, we varied task difficulty by changing the angle between the ports. Moving the ports closer together increased uncertainty about the stimulus source, as the light from each IR source had a broad wavefront (Figure 1a), and the infrared detector had a relatively wide ‘receptive field.’ While animals consistently performed above 90% of the trials correctly when the ports were 90° apart, this performance dropped off quickly as the angles between the IR sources were reduced below 60° (Figure 2c; Supplementary Movie 3). Behavioral performance was significantly dependent upon task difficulty (p=1.1×10−19; ANOVA).

Based on the way they swept their heads about to sample the IR levels, the rats seemed sensitive to the intensity of IR light, not just its presence or absence. To test this, in some sessions we randomly interspersed trials in which stimulation frequency was held constant, so the animal only received binary information about IR presence (Figure 2d, left). Performance was significantly degraded in such constant-frequency trials, demonstrating the importance of coding infrared intensity in a graded manner (Figure 2d, right, shows the results of 34 sessions in two animals; p=0.001; two-way ANOVA).

Rats are typically considered blind to the IR spectrum[11]: their cone spectral sensitivity is negligible above 650 nm[14], which is well outside the spectral emission of our IR source (940 nm peak emittance, with a range of nonzero emission between 825 and 1000 nm; see Methods). However, in the sensory prosthetics literature there has been some concern about the true range of spectral visual sensitivity of the rat[15], so we also experimentally checked whether our animals succeeded by using their visual system to discriminate the IR cues. In two animals we added random ‘no stimulation’ trials in which the IR light turned on, but there was no stimulation delivered to the cortex. Performance on the task was abolished in all 11 sessions, dropping from 69±3.2 to 8±5.2% correct, a significant change (p=7.6e-6; paired t-test) (Figure 2e). Percent correct was below 33% on the trials without stimulation because in the majority of trials the animals did not poke in any of the ports, and simply let the trial time out (see online Supplementary Movie 4).

Analysis of Activity in S1

Next, we investigated the effects, within S1, of acquiring the new sensory modality. For instance, are some neurons “hijacked” by IR inputs,[16] such that they no longer respond to whisker stimulation? To address this question, we recorded from multi-electrode arrays chronically implanted in S1 infragranular cortex in two well-trained rats (N=76 single units over five sessions). We presented three different stimuli, each lasting 200 ms: unilateral whisker deflection (via air puffs), ICMS (using current magnitudes that were applied during the task), and both stimuli delivered simultaneously.

S1 neurons were clearly not hijacked by ICMS, even after many months of microstimulation. In fact, most (83%) S1 neurons showed quite robust responses to whisker deflection (Figure 3a). The breakdown of all response types is detailed in Figure 3c. Briefly, of 76 units, 84% (64/76) responded to one or the other stimulus, while 16% (12/76) showed no significant response to either stimulus. The majority (83% (63/76)) showed significant responses to whisker deflection. Of these, 83% (52/63) were multimodal neurons, exhibiting significant responses to ICMS as well. Only one neuron in our sample showed a response to ICMS but not whisker deflection. Among the multimodal S1 neurons, the response to both stimuli delivered simultaneously was highly sublinear (Figure 3d; p=4.6×10−17; two-tailed paired t-test).

Figure 3

Neurons in S1 respond to both whisker deflection and ICMS

a. Sample PSTHs from neurons (rows) recorded simultaneously in an anesthetized animal during whisker deflections. The gray strip indicates time of whisker deflection (0 to 200 ms). Blue/red lines indicate mean± three standard deviations from the mean firing rate during the baseline period before stimulation. Rows 1–5 are single-unit responses, while row 6 is a representative multiunit response. b. Sample PSTHs from the same neurons in panel a under three stimulus conditions. Left column: response to whisker deflection (38 psi air puff), but with the response during stimulation clipped out to enable comparison with the other two columns. Middle column: response to ICMS only (250Hz). Right panel: response to simultaneous whisker deflection and ICMS. The light yellow strip indicates the time, after stimulation, we used to compare response magnitudes in the three cases. Note there are no responses during electrical stimulation because of electrical artifacts, so the firing rates drop to zero during those epochs. c. Overall distribution of response types. Columns are segregated by whether neurons responded significantly to both stimuli, only one stimulus, or neither. d. Mean (±sem) response of multimodal neurons to each of the three stimuli. The observed response to both stimuli is significantly sublinear (fourth bar indicates expected response under assumption of linearity). Asterisks indicate significance with p<1.0×10−6 (paired, two-tailed t-test) e. Multiunit response magnitude (mean firing rate±sem) as a function of whisker deflection magnitude (intensity of air pressure applied to whiskers) for 16 channels in two animals. Zero is baseline (mean rate before stimulus onset), and responses for each channel were normalized to the maximum mean response over all three stimulus conditions. f. Multiunit response magnitude versus ICMS frequency, with the same conventions as in panel e, with 13 channels in two animals.

Population responses in S1 increased according to a saturating function of whisker deflection magnitude (Figure 3e). Response magnitudes also increased with ICMS frequency up to 350Hz, but dropped off at 400Hz, likely due to adaptation (Figure 3f). This neuronal response-profile is akin to an IR receptive-field for the range of IR stimulation frequencies employed in the task.

Discussion

In 1969, Bach-y-Rita performed the classic ‘sensory substitution’ experiments, in which visual stimuli were projected onto the skin via mechanical actuators, that allowed congenitally blind patients to experience a visual world for the first time[4]. Here, we have applied the logic of sensory substitution directly to the somatosensory cortex, bypassing the body’s periphery, with the goal of building a cortical sensory prosthesis capable of augmenting the subject’s perceptual capability[5]. Instead of giving a binary, top-down signal instructing the rats where to move[17], we connected S1 to a new graded sensory cue available in their environment, and they spontaneously adopted novel foraging behaviors in response. Note that, in principle, this experimental paradigm could use any novel stimulus (e.g. magnetic or radio waves) to be represented in S1.

We observed that neurons in the stimulated regions of S1 maintained their ability to respond to whisker deflection. This suggests that two different cortical representations (e.g. one tactile and one IR) became superimposed on the animal’s S1 cortex, creating a novel bimodal processing region. However, it is important to emphasize that behavioral studies will be needed to determine the consequences, for whisker-based tactile discrimination, of adding this new information to S1.

The mechanisms by which animals learns to use this new information source will also be an interesting avenue for future research, as such research should suggest how to accelerate sensory prosthetic acquisition. To investigate the mechanisms of plasticity, including those theories implicating glial cells[18], it will be helpful to examine functional and structural changes in cortical and subcortical tactile processing centers as rats learn to discriminate among IR sources.

Overall, our behavioral results suggest that animals initially treated S1 electrical stimulation as an unexpected whisker deflection, and later they learned to treat it as a stimulus originating away from the body in the surrounding environment. However, we are unable, using the methods in this paper, to determine whether the fully trained rats consciously experienced microstimulation as a new distal sensory modality, or simply learned to associate a tactile sensation with an otherwise imperceptible distal sensory cue. This is a question that could presently be addressed with sensory substitution experiments in humans. Indeed, one such study suggests that some subjects experienced tactile stimuli as visual in nature after training with a visual-to-tactile peripheral substitution device[19].

The next generation of sensory neuroprosthetics research in humans will likely continue to rely on ICMS rather than optogenetics[20,21]. Indeed, our experimental strategy has been quite different from optogenetic approaches[22-25]: instead of stimulating a specific cell-type population, we indiscriminately stimulated all S1 neurons in the vicinity of the stimulating electrodes’ tips. Despite the fact that this unusual signal was delivered at extremely high frequencies, and likely spread through most of S1[26-28], these animals readily learned to exploit this new sensory channel.

One potential application of the technology used in this study is in the design of a new generation of motor neuroprostheses that, instead of simply sending a brain-derived motor control signal to move a prosthetic limb, would provide continuous sensory feedback to the user’s brain from the prosthesis[29-31]. Such closed-loop bidirectional brain-machine-brain interaction could significantly improve reaction time, behavioral accuracy, and likely aid the integration of the limb into the user’s internal body image[9,30,32,33].

But there is another aspect of the present work that has rarely been explored in the neuroprosthetics literature: the potential to expand or augment a species’ normal perceptual range. In that pursuit, we have implemented, as far as we can tell, the first cortical neuroprosthesis capable of expanding a species’ perceptual repertoire to include the near infrared electromagnetic spectrum, which is well outside the rat photoreceptors’ spectral sensitivity. Thus, by taking advantage of this novel paradigm, our rats were able to transcend the limitation of perceiving only those stimuli that can activate their bodies’ native sensory transducers.

Behavioral task and training

All experiments were performed on female Long Evans rats approximately 14 weeks old (~250 g; Harlan Laboratories). Rats were trained in a chamber with three reward ports that were situated 90° apart (Figure 1a). Each port was fit with a visible broad-spectrum LED, and an infrared (IR) LED (Opto Semiconductors Inc., 940 nm peak emittance (range of nonzero emissions was between 825–1000 nm), and IR intensity dropped to half-max at 120°) (Figure 1a). We first trained water-deprived rats to poke in the port whose visible LED was activated. Each trial began when a visible light in one randomly selected port was activated, and the animal received water reward if they broke the photobeam in that port (correct trials). On incorrect trials (when they poked in the wrong port) they received no water, an error tone, and a longer delay to the next trial. For some rats, we delivered air puffs from the port when they poked in error.

Once animals performed above a criterion value (>70% correct) more than 4 days in a row, we implanted an array of stimulating electrodes into S1 (Figure 1d). The array includes an IR detector attached to the connector (see Figure 1c and Surgery below). After the animals recovered from surgery, we determined the minimal currents required to evoke a behavioral response in at least two electrode pairs (thresholds were between 1 and 200 µA). We then trained them in the same behavioral chamber, but incrementally replacing visible light with ICMS linked to IR levels from their detector (Figure 1e).

Initially, each trial was the same as the basic behavioral task described above, except the onset of the visible light was preceded by infrared-level dependent stimulation (up to 400 Hz) that lasted 0.6–1.5 seconds. In four of six rats, we began with brief durations of stimulation (600–700 ms) to acclimate them to stimulation. In two of the rats, we started with longer stimulation durations to get estimates of behavioral latency as they learned the task (Figure 2b). Thus the animals learned that stimulation indicated the presence of the visible light. The stimulation frequency was exponentially proportional to the IR levels (Figure 1e). We used an exponential function because IR-intensity dropped logarithmically from the IR source. We stimulated at high frequencies for two reasons: one, in preliminary experiments we found that using a lower frequency range (i.e., 0 to 100Hz) yielded less reliable performance; two, we wished to avoid evoking kindling seizures[34]. Stimulation frequencies were updated every 50 ms based on IR levels, and it took approximately 5 ms for the stimulation frequency to actually change in the rat once the command was sent. Pulse frequency was the only variable that tracked IR levels: current amplitudes were kept the same for each pulse.

Once it became clear, based on visual inspection, that the animals were comfortable with stimulation, we added ‘IR-only’ trials in which the IR-dependent stimulation would appear without any accompanying visible light. That is, on these IR-only trials they could only use the signals from ICMS to get to the correct port. They trained on this until their performance on the IR-only trials reached a criterion value of 70% correct. For some rats that stayed above criterion for four sessions in a row, we then varied task difficulty in which across sessions, we pseudo-randomly chose a new angle between the ports (either 90, 60, 45, or 30 degrees; Figure 2c).

On some sessions, we added a small percentage of trials (~15%) in which the stimulation frequency was held constant, regardless of the infrared intensity. Such constant-frequency trials allowed us to compare performance when the stimulation frequency depended on IR levels with those trials when stimulation frequency was a constant function of IR intensity (Figure 2d, left hand side).

All behavior control was run in custom Matlab scripts using the data acquisition toolbox (run with NIDAQ PCI-7742 card, National Instruments).

Stimulating electrodes and stimulator

In preliminary studies, we found that the thresholds for evoking behavioral responses were lower with electrode pairs oriented along the same penetration of the cortex, versus two electrodes parallel to the cortical surface. Hence, for each biphasic stimulating electrode, we joined pairs of 30 µm stainless steel microwires to one another, separated by 300 µm (Figure 1c inset). We attached a single infrared detector (Lite-On Inc) to the connector (Omnetics), and powered the IR detector through the two extra reference pins on the connector. The phototransistor in the detector had a peak spectral sensitivity at a wavelength of 940 nm. The range of sensitivity was from 860 to 1020 nm, and its ‘receptive field’ (i.e., the angular range within which a 940 nm test stimuli would evoke a response from the detector) was 20° in diameter at half max (Figure 1a).

We used bipolar stimulation with charge-balanced, biphasic pulse trains, using a custom-controlled stimulator, as described elsewhere[35]. Pulses were 100 µs in duration, with 50 µs between the cathodic and anodic phases of the pulses (Figure 1e). Current magnitudes varied between 1µA and 300 µA. Before training rats on the IR-version of the task, we determined current thresholds by placing them in the empty behavioral chamber (Figure 1A), and stimulating with 1uA at 200Hz for 500 ms. If an animal noticeably moved in response to stimulation (this typically involved locomoting, moving their heads to the side, or scratching at their faces), that current level was taken as the threshold current. If not, we increased the current amplitude by 50%, keeping the frequency and duration the same, until we noted such a response. We used electrical microstimulation in lieu of optogenetics in this study, as one goal is to integrate this technology into human studies in the near future (see Discussion).

Surgery

Detailed surgical procedures are described elsewhere[36]. Briefly, we implanted the stimulating electrodes into S1 (−2.5mm posterior and 5.5 mm relative to bregma, 1.5 mm deep) under pentobarbital anesthesia (0.065 mg/g), and the rats were given at least a week to recover from the surgery before being deprived again. The Duke University Institutional Animal Use Committee approved all surgical and behavioral methods.

Neural recording and sorting

The basic recording setup is described in detail elsewhere[36]. Namely, on those channels which we were not using for stimulation, we recorded neural activity using the (Multichannel Acquisition Processor (MAP; Plexon, Inc., Dallas, Texas). To stimulate and record simultaneously without damaging the head stage, we peeled off the wires connected to the stimulator from the connector to bypass the head stage, while the other channels went directly to the head stage as usual[36]. Sorting of neural data is also described elsewhere[36]. Briefly, in addition to template-based online sorting, all voltage traces around a threshold-crossing event were saved for offline sorting. For offline sorting, we used clustering in PCA space, signal to noise ratio, and autocorrelation functions that showed a clear absolute and relative refractory period to assigned data as either single units (SUs) or multi-unit (MUs).

In our recordings, we lightly anesthetized head-fixed animals under isofluorane anesthesia (0.8–3% isofluorane mixed with pure oxygen). We controlled the duration and magnitude (psi) of air via a modified fluid dispenser (Oki DX-250, Garden Grove, CA) which was controlled with TTL pulses sent from Matlab. Before recording, we positioned the air dispenser to stimulate the majority of the large whiskers, approximately 10 mm from the whisker pad. We delivered three different stimuli, in pseudo-random order: air puff to whiskers, electrical stimulation, and both stimuli delivered simultaneously (see Figure 3).

Statistical methods

We calculated significant responses in PSTHs using a bootstrap cumulative sum algorithm described in more detail elsewhere[37]. This involved taking bootstrap samples of the cumulative response during a baseline period (period before time zero in Figure 3a), and using these to generate a cumulative sum 95% confidence interval (with Bonferonni correction for multiple comparisons), and comparing this to the actual cumulative sum during the period of stimulation. A significant response was when the actual cumulative sum left the 95% confidence interval, and ended when the response dropped below a 95% confidence interval estimated from bootstrapping the original data.

During electrical stimulation, recordings were saturated by stimulus artifact, so to directly compare responses between whisker deflection and ICMS, we compared mean firing rates during the 130 ms period following the offset of both stimuli. To compare relative response magnitudes among multiple neurons, we normalized the mean response to each stimulus by the maximum mean response over all three stimulus conditions, so the maximum response was assigned a value of one.

To generate the predicted responses to both stimuli delivered simultaneously (Figure 3c), we used the normalized responses described in the previous paragraph. For a linear system, the predicted response to both stimuli would simply be the sum of the responses to each stimulus taken individually. That is, R(E and W)=R(E)+R(W), where E is electrical stimulation, W is whisker deflection, and R(x) signifies the response to arbitrary input x.