Debarun Sengupta1, Michele Mastella2,3, Elisabetta Chicca2,3, Ajay Giri Prakash Kottapalli1,4. 1. Department of Advanced Production Engineering (APE), Engineering and Technology Institute Groningen (ENTEG), University of Groningen, Groningen 9747 AG, The Netherlands. 2. Groningen Cognitive Systems and Materials Center (CogniGron), University of Groningen, Groningen 9747 AG, The Netherlands. 3. Bio-Inspired Circuits and Systems (BICS) Laboratory, Zernike Institute for Advanced Materials (Zernike Inst Adv Mat), University of Groningen, Nijenborgh 4, Groningen NL-9747 AG, Netherlands. 4. MIT Sea Grant College Program, Massachusetts Institute of Technology (MIT), 77 Massachusetts Avenue, NW98-151, Cambridge, Massachusetts 02139, United States.
Abstract
During the past few decades, a significant amount of research effort has been dedicated toward developing skin-inspired sensors for real-time human motion monitoring and next-generation robotic devices. Although several flexible and wearable sensors have been developed in the past, the need of the hour is developing accurate, reliable, sophisticated, facile yet inexpensive flexible sensors coupled with neuromorphic systems or spiking neural networks to encode tactile information without the need for complex digital architectures, thus achieving true skin-like sensing with limited resources. In this work, we propose an approach entailing carbon nanofiber-polydimethylsiloxane composite-based piezoresistive sensors, coupled with spiking neural networks, to mimic skin-like sensing. The strain and pressure sensors have been combined with appropriately designed neural networks to encode analog voltages to spikes to recreate bioinspired tactile sensing and proprioception. To further validate the proprioceptive capability of the system, a gesture tracking smart glove, combined with a spiking neural network, was demonstrated. Wearable and flexible sensors with accompanying neural networks such as the ones proposed in this work will pave the way for a future generation of skin-mimetic sensors for advanced prosthetic devices, apparel integrable smart sensors for human motion monitoring, and human-machine interfaces.
During the past few decades, a significant amount of research effort has been dedicated toward developing skin-inspired sensors for real-time human motion monitoring and next-generation robotic devices. Although several flexible and wearable sensors have been developed in the past, the need of the hour is developing accurate, reliable, sophisticated, facile yet inexpensive flexible sensors coupled with neuromorphic systems or spiking neural networks to encode tactile information without the need for complex digital architectures, thus achieving true skin-like sensing with limited resources. In this work, we propose an approach entailing carbon nanofiber-polydimethylsiloxane composite-based piezoresistive sensors, coupled with spiking neural networks, to mimic skin-like sensing. The strain and pressure sensors have been combined with appropriately designed neural networks to encode analog voltages to spikes to recreate bioinspired tactile sensing and proprioception. To further validate the proprioceptive capability of the system, a gesture tracking smart glove, combined with a spiking neural network, was demonstrated. Wearable and flexible sensors with accompanying neural networks such as the ones proposed in this work will pave the way for a future generation of skin-mimetic sensors for advanced prosthetic devices, apparel integrable smart sensors for human motion monitoring, and human-machine interfaces.
With the recent progress in flexible electronics
and the wearable
smart technology, next-generation smart prosthetic devices are becoming
a reality. These devices can recreate the sense of touch in prosthetic
limbs and create sensing possibilities which assist toward realizing
smart apparels for real-time human vital monitoring and soft sensors
for human-machine interactions. In particular, skin-like artificial
sensors based on polymer–nanomaterial composites are widely
researched for wearable and next-generation prosthetic applications.
Various conductive nanomaterials such as carbon black, carbon nanotubes
(CNTs), graphene, Mxene, and silver nanowires in conjunction with
polymer elastomeric materials like ecoflex, polyimide, polydimethyl
siloxane (PDMS), and polyurethane have been used for developing skin-like
flexible and stretchable sensors.[1−6] In addition to the vast variety of two-dimensional (2D) nanomaterials
already reported in the literature, inexpensive nanomaterials like
electrospun carbon nanofibers (CNFs) have also been employed for developing
flexible and wearable sensors.[7,8] The electrospinning
method provides a facile and cheaper alternative to cumbersome processes
like chemical vapor deposition, laser ablation, arc deposition, chemical
and mechanical exfoliation, and graphite oxidation–reduction
employed for synthesizing carbonaceous nanomaterials like CNTs and
graphene. The crucial need of the hour is to develop facile, reliable,
and inexpensive skin-like artificial sensors integrated with intelligent
systems to achieve somatosensory perception in robots and next-generation
prosthetic devices. The present challenges can be solved by adapting
a bioinspiration approach entailing a two-pronged strategy, wherein
skin-like wearable/skin-mountable sensors could be employed to emulate
the skin, and an interface neuromorphic circuit could be used to generate
spike patterns, emulating neural firing and mimicking human sense
of touch.The sense of touch is a crucial ability in humans.
The exploration
of an unknown environment heavily relies on the usage of touch since
the first instance of every child’s life.[9] This sense enables the recognition of the properties of
objects like texture, shape, and softness, which are fundamental for
even a simple task like grasping a glass without breaking it. Most
of the complex somatosensory abilities observed in human beings lie
in the skin, which can be considered as a large-area pressure, tactile,
vibration, and temperature sensor. The skin is innervated by mechanosensory
afferents which enrich its somatosensory ability as shown by the schematic
in Figure a. For instance,
the glabrous skin of the human hand is populated by low-threshold
mechanoreceptors (LTMRs) consisting of a combination of rapid and
slow-adapting LTMRs (RA/SA-LTMRs).[10] While
the SA-LTMRs such as the Ruffini endings and the Merkel cells respond
to skin stretch and constant indentation, the RA-LTMRs such as the
Pacinian and Meissner corpuscles are more sensitive to dynamic stimuli
and (RA/SA-LTMRs).[10] The schematic in Figure a summarizes the
mechanoreceptors innervating the human skin. Robots should be endowed
with sensors similar to mechanoreceptors in order to perform complicated
tasks like interacting with human beings or unknown environments.
While performing simple tasks like object grasping, important information
is provided by touch in relation to both the shape of the held object,
the position in the hand, and the force exerted by the fingers.[10] Specifically, this requires both cutaneous tactile
information, like the intensity at which the hand is pressing the
object, and haptic proprioception, like the angle of the different
fingers or the position of the hand with respect to the arm.[11] Therefore, to achieve true skin-like sensing,
the system should have sensors capable of detecting pressure on the
fingers and strain at the joints and accompanying circuitry to convert
the sensor signal to neural impulses. While several biomimetic sensors
for cutaneous touch have been proposed in the literature,[12−14] few of them considered haptic proprioception and, to our knowledge,
the combination between these two is only conceptualized.[15]
Figure 1
(a) Schematic representation of glabrous and hairy skins
with their
underlying mechanosensory receptors enabling their somatosensory ability.
Reproduced with permission,[10] 2013, Cell
press. (b) Schematic representation of the process steps involved
in fabrication of the CNF-PDMS piezoresistive sensors. (c) Photograph
of the fabricated CNF-PDMS tactile sensor. (d) Conceptual scheme of
the final architecture. The glove will be equipped with tactile sensors
on the palm and on the fingertips, with a wrist bending sensor and
with stretch sensors able to detect gestures. The data will be then
collected using Wheatstone bridges, converted to currents, and fed
to a neural network which will use neural coding.
(a) Schematic representation of glabrous and hairy skins
with their
underlying mechanosensory receptors enabling their somatosensory ability.
Reproduced with permission,[10] 2013, Cell
press. (b) Schematic representation of the process steps involved
in fabrication of the CNF-PDMS piezoresistive sensors. (c) Photograph
of the fabricated CNF-PDMS tactile sensor. (d) Conceptual scheme of
the final architecture. The glove will be equipped with tactile sensors
on the palm and on the fingertips, with a wrist bending sensor and
with stretch sensors able to detect gestures. The data will be then
collected using Wheatstone bridges, converted to currents, and fed
to a neural network which will use neural coding.To pursue the combination between these two concepts, in this work,
CNF-PDMS-based piezoresistive sensors have been coupled with neuromorphic
spiking neural networks to achieve skin-like sensing and proprioception.
In this approach, models inspired by biological neurons were used
to encode oncoming signals from the sensors and to convert them into
digital pulses (called spikes), which preserve the information in
the time of the event.[16] Several studies
have already used this method for converting analog signals from tactile
stimuli into spiking activity.[12−14,17] Furthermore, the kinds of networks used in this study are increasingly
becoming popular due to their versatility and their ability to learn.[18] For example, in object tactile orientation detection,
several networks have been proposed for decoding the angle at which
a bar is pressed using only neurons, synapses, and learning.[19,20]In the first part of the paper, the sensing element is investigated.
The recorded behavior of a bundled CNF-based piezoresistive sensing
element is fed to a simulated neuron. The outcome of the latter is
shown to encode information through spike count, similar to an approach
proposed for prosthetics application demonstrated earlier.[21] In the second (wrist conformation) and third
(gesture recognition) parts instead, the strain sensor signals are
recorded and coupled with simulated neurons for emulating proprioception.
In the case of wrist conformation, the experiment is conducted by
sampling the bending angle of a wrist through the fabricated strain
sensor. The outcome is fed to a simulated neuron that can encode the
bending angle in spike count using the sensor’s voltage as
input. For the third case (gesture recognition), a more complex approach
has been adapted. By considering several CNF-PDMS strain sensors placed
at the finger joints, the simulated network was able to detect specific
gestures. This was done by a first layer of neurons, connected in
a one-to-one fashion coupled with a second layer, through synapses.
The latter was able to indicate which gesture was performed. The interplay
of these two layers of neurons, along with their synapses, formed
a spiking neural network. Such a combination of networks with flexible,
wearable yet facile and inexpensive sensors could be a key factor
for future generation of smart apparel with cutaneous and haptic abilities
capable of recreating the somatosensory perception in prosthetics
and robotic interfaces.
Results
System Description
The ultralightweight, stretchable,
and facile CNF-PDMS skin-like sensors utilized in this work have been
introduced in our earlier work, which describe the fabrication and
characterization of this sensing element.[7] The sensor design features a piezoresistive CNF bundle which acts
as a sensing element embedded in two layers of PDMS for encapsulation.
To demonstrate the capability of mimicking proprioceptive perception
and tactile sensing, the CNF sensing elements were configured into
various designs suitable for strain and pressure sensing. The details
of sensor fabrication are schematically represented in Figure b. Further details of the sensor
design and fabrication steps are presented in the Experimental Section. The sensing mechanism of the CNF-PDMS
strain sensor has been reported in our earlier works.[7,8] Within a CNF sensing element, electrons travel through the CNF percolation
network embedded in the PDMS elastomer matrix. Any external strain
or applied pressure results in the change in overlap area between
adjacent conductive domains, leading to an overall change in the resistance
of the sensor. In addition, electrons can also tunnel across a thin
(less than a nanometer) insulating barrier separating two conductive
domains, which leads to associated tunneling resistance which can
be predicted using Simmon’s tunneling resistance formula.[22,23] The tunneling resistance is extremely sensitive to the interdomain
separation, and any changes arising due to external pressure/strain
are also manifested as an overall resistance change of the sensor.
When a pressure/strain is applied on the CNF-PDMS sensor, its resistance
changes proportional to the magnitude of the applied force. Variation
of resistance can be sampled by ad hoc integrated circuits coupled
with a suitably designed Wheatstone bridge,[24] even with the possibility to gather multiple sensors using only
one circuit[25] which exhibits linear behavior.
In this work, an appropriately designed Wheatstone bridge circuit
is used to convert the resistance changes in the sensors into readable
voltage signal outputs. The realization of the stage converting the
pressure applied on a piezoresistive element to the current fed to
a neuron is not explored. Instead, an ideal linear dependence between
the pressure and the current output is used to run the simulation
in this work.In the following subsections, we demonstrate how
current coming from this ideal stage can be used in neuromorphic signal
processing instead of traditional digital signal processing. The information
in the former is conveyed using a new medium: spikes. In this work,
different types of encoding are shown with different properties of
touch to highlight how versatile and useful a conversion to spikes
can be. The signals, converted into currents, are fed into a leaky
integrate-and-fire (LIF) simulated artificial neuron. The LIF model
is a very common implementation of a simplified biological neuron,[26] composed of a resistor–capacitor network
and a threshold, as further explained in the methods. The choice of
this type of model is given by the great amount of existing hardware:
the literature shows implementations on FPGA,[27,28] microcontrollers,[29] and VLSI chips.[30−32] In the latter especially, the system does not require any overhead
for circuitry since the readout and the neurons can be integrated
in a single monolithic chip. This paper focuses on showing how, using
neurons instead of traditional digital communication, computation
can be achieved without using ADC or DSP. In the first experiment,
which involves applying different pressures on the sensor, analog
voltage response is observed, and the subsequent behavior of neurons
is investigated. In the second case, which involves measurement of
wrist bending, a similar approach is adopted to convert the sensor’s
voltage caused by the movement of a wrist into meaningful neural activity.
In the last part, related to identification of various hand gestures,
the combined activation of several neurons is exploited to reconstruct
the occurrence of different finger configurations.This technique
of converting analog signals into events has lower
limitations than traditional digital systems. They can encode analog
information in their time interval between spikes, keeping most of
the meaning and relaxing the Shannon theorem’s constraint.
This means that regardless of the stimulus’s transient speed,
a neuron is still able to sample most of the signal’s envelope.
Second, neurons are excited and spike only when a current is fed at
their input. This ability, called event-driven sampling, is useful
when the desired signal is sparse. Sparsity is a property typically
observed in nature where useful information is condensed in a small
fraction of the active time of a sampling agent. The event-driven
nature samples the environment only when necessary and with an almost
analog time step (only limited by the physical design of the neuron).
Mimicry of Tactile Sensing
Under the effect of an external
pressure stimulus, the resistance
of the CNF-PDMS sensor changes, which is manifested as voltage signal
output from the accompanying Wheatstone bridge circuit. To demonstrate
skin-like tactile sensing capability, the sensor was pressed four
times in quick succession, followed by a gap of 10 s and subsequent
repetition of the sequence six times. As shown by the sensor voltage
response plot in Figure a, the sensor can generate different voltages depending on the applied
tapping pressure which can be used to stimulate a simulated artificial
neuron. To be compatible with the neuron physical mechanism, the Wheatstone
bridge output is converted through an analog voltage-to-current converter
into current and then injected into the neuron.
Figure 2
(a) Sensor voltage generated
by different periodic pressures on
the sensor. Different pressure values have been used to explore the
response of the sensor with voltages up to 1.21 V. (b) Photo of the
sensor along with the sensor readout schematic. (c) Number of spikes
emitted by the connected neuron when the piezoresistor is stimulated.
(d) The membrane voltage reaches 0.4 V, its value is rebased to 0
V, and a spike is generated. (e) Correspondence between the values
of the sensor voltage (encoding the pressure) and the number of spikes
emitted by the sensor. The number of spikes is calculated per 100
ms.
(a) Sensor voltage generated
by different periodic pressures on
the sensor. Different pressure values have been used to explore the
response of the sensor with voltages up to 1.21 V. (b) Photo of the
sensor along with the sensor readout schematic. (c) Number of spikes
emitted by the connected neuron when the piezoresistor is stimulated.
(d) The membrane voltage reaches 0.4 V, its value is rebased to 0
V, and a spike is generated. (e) Correspondence between the values
of the sensor voltage (encoding the pressure) and the number of spikes
emitted by the sensor. The number of spikes is calculated per 100
ms.Once the current is injected in
the neuron, charge begins to accumulate
on its membrane capacitor that also has a leaky component. The voltage
difference present at the capacitor pins increases with the charge
as dVmem/dt = (Isensor/C – Vmem/(RC)), where C is the equivalent
capacitive value of the membrane and R is the equivalent
resistance of the leaky component. When the voltage reaches a specific
value (defined as threshold voltage or Vthr), the neuron emits a digital event (or spike). In Figure , the voltage on the membrane
(Vmem) is plotted to highlight how the
sensor’s voltage (Vsensor) influenced
the neuron behavior. In the formula, Isensor is related with the Vsensor by an ideal
component with linear conductance G so that Isensor = G × Vsensor (exemplified in Figure b by component G). The formula
as it is highlights that the membrane voltage acts as a low-pass filter.Figure also shows
the encoding ability of a neuron, which embeds the information about
the amplitude of the sensor voltage into its spiking activity. This
is visible in 2c and 2d, where spike count and the membrane potential
of the neuron are presented. Figure e highlights the close relationship between the sensor
voltage and the neuron activity by plotting the voltage of the sensor
and the equivalent spike response. For low voltages (less than 0.2
V), the neuron does not generate any spikes, while for higher voltages,
the neuron generates spikes proportional to the input. This is related
to the property of the LIF neuron, which does not reach the threshold
if the injected current is not high enough. This property can be efficiently
used to set an attention threshold under which the communication stream
is not active, avoiding the processing of noise or spurious signals.
Wrist
Conformation
SAII-LTMRs of the human skin are
extremely sensitive to skin stretch. For instance, the SAII-LTMRs
found in humans share their physiological traits with proprioceptors
which facilitate kinesthetic perception (finger shape or conformation).
The strain sensing mechanism of the CNF-PDMS sensors can also be exploited
for mimicking proprioceptors capable of detecting the bending of the
wrist. When a uniaxial strain is applied, the resistance of the sensor
changes owing to the conductive domain disconnection mechanism. The
sensor bending generated tensile and compressive strain, which resulted
in a voltage proportional to the applied angle then converted by the
accompanying Wheatstone bridge circuit. The sensor’s voltage
can be used to stimulate an LIF neuron, once converted into current,
which can also be applied for strain measurements. The information
about the bending can be encoded in the number of spikes that the
neuron emits for a given time step. The number of spikes correlates
well to the analog voltage generated by the neuron, as visible in Figure a. In the latter,
the higher the analog voltage, the denser the spike activity.
Figure 3
(a) Comparison
between the response to wrist bending of both the
sensor and the neuron. The sensor generates a voltage proportional
to the bending of the wrist. This voltage is converted into a current
and fed into a neuron, simulated on a computer. The number of spikes
per 100 ms is here considered as spike count (or spikes #). Different
trials for different angles are shown. The neuron follows the analog
voltage with the number of spikes but does not generate any spike
when the stimulus is only noise (this is visible in the lower part
of the figure, where the spikes are 0 when no bending is performed).
(b) Statistical analysis of the analog voltage and spike counts with
different bending angles. The statistical deviation of the sensor
is given by the human error in bending the wrist at a specific angle.
This uncertainty is reflected quite well in the neuron with the spike
count, which highlights the direct connection between a neuron spiking
activity and a current coming from a piezo-resistive readout. This
is to demonstrate the high degree of reproduction that the neuron
has with respect to the analog values that it receives.
(a) Comparison
between the response to wrist bending of both the
sensor and the neuron. The sensor generates a voltage proportional
to the bending of the wrist. This voltage is converted into a current
and fed into a neuron, simulated on a computer. The number of spikes
per 100 ms is here considered as spike count (or spikes #). Different
trials for different angles are shown. The neuron follows the analog
voltage with the number of spikes but does not generate any spike
when the stimulus is only noise (this is visible in the lower part
of the figure, where the spikes are 0 when no bending is performed).
(b) Statistical analysis of the analog voltage and spike counts with
different bending angles. The statistical deviation of the sensor
is given by the human error in bending the wrist at a specific angle.
This uncertainty is reflected quite well in the neuron with the spike
count, which highlights the direct connection between a neuron spiking
activity and a current coming from a piezo-resistive readout. This
is to demonstrate the high degree of reproduction that the neuron
has with respect to the analog values that it receives.From Figure a,
it can also be observed that the neuron employment shows an advantage
of filtering of noise in encoding. This phenomenon is possible thanks
to two properties of neural coding. First, the neuron’s membrane
capacitor acts as a low-pass filter, removing the high-frequency noise
components. Second, the neuron spikes only when a threshold voltage
is reached. This means that an isolated spurious signal that is injected
into the neuron through current will not be enough to generate a spike,
while a consistent stimulus is going to excite the neuron, resulting
in multiple spikes.In Figure b, the
statistical mean and standard variation among trials of both voltage
and spikes are plotted. It can be seen from the plot that not only
does the neuron have a response proportional to consistent stimulus
but also it is able to maintain the same statistical properties observed
in the original sensor’s voltage. The variation is due to the
psychophysical uncertainty of human movements which leads to subsequent
angular discrepancies while bending the wrist at certain prespecified
angles and the fact that the neurons transfer; also, this psychophysical
uncertainty can be used to process rich information, like fine grain
movement.
Gesture Recognition
To further demonstrate applications
of the CNF-PDMS sensors involving proprioceptive perception, a smart
glove system consisting of five identical sensors secured on a nitrile
glove coupled with appropriately designed Wheatstone bridge circuits
was developed. Different sensors were placed at the hand joints, while
a human volunteer moved the fingers in defined ways. The hand performed
several digits with the glove. Nominally, number 5 (composed of the
palm completely open), number 4 (composed of the thumb bending inward),
number 3 (made of the thumb and the pinky fingers closed), and numbers
2 and 1 shown by bending the middle and the index fingers and only
the index finger, respectively.Each of these finger’s
movements led to resistance changes in the sensors, subsequently leading
to voltage signals generated by the readout circuits. The simultaneous
activation of multiple sensors can be used to spot the number mimicked.
As seen in Figure a, the succession of the numbers (5, 4, 3, 2, and 1) generates in
the five sensors time-varying responses. Specifically, the bending
of the finger creates a high value, while the relaxed position gives
no response. The signals from the sensors are then fed into a simulated
layer of five LIF neurons. The response of different neurons, visible
in Figure c, highlights
that the layer (i.e., a group of neurons with the same task) is able
to transfer the analog value of the bending quite faithfully in the
number of spikes per time step. This approach, like the one used in
the case of tactile sensing mimicry previously, exploits the spike
activity of neurons to encode the amplitude of the sensor’s
voltage without noise and spurious activity.
Figure 4
(a) Response in voltage
output of the five different sensors placed
at the joint of the glove. The five sensors are plotted one over the
other, from the little finger to the thumb. The five different colors
superimposed to the graph highlight the different gestures performed
in that moment. (b) Order of execution of five different gesture tasks
performed with the glove. (c) Response in spike count of the five
different neurons connected to the five sensors, plus the Poisson
neuron, responsible for acting on the gesture “Five”.
The five different colors superimposed to the graph highlight the
different gestures performed in that moment. (d) Schematic describing
the network used in this example. (e) Response of the decoding layer
to the different gestures.
(a) Response in voltage
output of the five different sensors placed
at the joint of the glove. The five sensors are plotted one over the
other, from the little finger to the thumb. The five different colors
superimposed to the graph highlight the different gestures performed
in that moment. (b) Order of execution of five different gesture tasks
performed with the glove. (c) Response in spike count of the five
different neurons connected to the five sensors, plus the Poisson
neuron, responsible for acting on the gesture “Five”.
The five different colors superimposed to the graph highlight the
different gestures performed in that moment. (d) Schematic describing
the network used in this example. (e) Response of the decoding layer
to the different gestures.The coincidence activation of several neurons, connected in a one-to-one
fashion to the sensors, encodes the different digits performed with
the glove. Each number is obtained with the simultaneous bending of
several fingers. Therefore, by analyzing which sensor was active at
each moment, we can understand which number was performed. The decoding
part (i.e., understanding which number the sensors’ activity
represents), while being traditionally obtained using microcontrollers
or computers, can be done using neuromorphic components. This approach
can reduce the need for complex and power-consuming architectures.
The full network architecture is briefly shown in Figure d and described in detail in
the Experimental Section. The structure can
recognize which gesture was performed, without any digital support.
In Figure e, the outcome
of the network is visible for each gesture performed. In these, the
network responds with a spiking activity on the neuron carrying the
gestures semantic.The gesture task can be extended even further
considering that
the neuron’s ability to encode into spikes the analog voltage
of a sensor can be used to recognize more complex gestures. Figure shows the correspondence
between sensors’ output and spiking activity when the smart
glove is used to perform more refined gestures. A more complex neural
network could also decode these gestures, avoiding the need for further
steps in an acquisition chain.
Figure 5
Response of the five sensors and the five
neurons to various hand
gestures. The sensor output voltage sensors increase with the increasing
curvature of the finger, and the neuron response follows this behavior.
Response of the five sensors and the five
neurons to various hand
gestures. The sensor output voltage sensors increase with the increasing
curvature of the finger, and the neuron response follows this behavior.
Conclusions
In this work, we proposed
a combined approach between novel CNF-PDMS-based
piezoresistive sensors and spiking neural networks to encode proprioception
and tactile information without the need for digital architectures
like analog-to-digital converters or digital signal processing. The
fabricated piezoresistive sensor is exploited by accompanying neurons
connected afterward, and the resulting system has been used for converting
different types of stimuli, from tactile sensors or strain sensors,
into spikes, showing how the number of the pulses emitted by a neuron
can communicate analog values such as the tactile pressure or the
deformation. Furthermore, the potential behind this has been unveiled
in this work, showing how neural networks can also decode the information
contained in the sensors’ response without the need for any
digital processing as in the case of gesture recognition. A final
architecture, consisting of a direct interface between sensors and
neural network CMOS circuits can greatly improve the abilities of
robots, autonomous agents, or prosthetic devices to detect complex
stimuli, resulting in low power consumption and low latency conversion,[30] typical of embedded approaches.
Experimental Section
Electrospinning and CNF Synthesis
The detailed recipe
for electrospinning polyacrylonitrile nanofibers and the subsequent
pyrolysis has been reported by us previously.[7] Polyacrylonitrile (PAN) powder (150,000 g/mol) and N,N-dimethyl formamide (DMF) obtained from Sigma-Aldrich
was used to form 9% (w/v) PAN polymer solution in DMF for electrospinning.
An Inovenso NanoSpinner NE300 was used for electrospinning. The polymer
solution was fed through a 18G needle using a standard syringe pump
(model NE300) at a constant flow rate of 1 mL/h. The electrospinning
process was conducted for 30 min at 12 kV (applied between the needle
tip and a rotating mandrel collector) while maintaining a constant
tip to collector distance of 10 cm. The as-spun nanofibers were transferred
from the aluminum foil collector to a ceramic crucible and placed
inside a furnace for pyrolization (at 950 °C) to synthesize CNF
bundles. Further details regarding the pyrolization step are provided
in our earlier work.[7]
Sensor Fabrication
and Data Acquisition
The basic fabrication
steps include sandwiching a CNF bundle (bonded to copper tapes at
two ends) between two layers of PDMS, thus achieving a complete encapsulation.
In specific, the tactile sensor was composed of an electrically bonded
2 × 2 cm CNF bundle encapsulated with the PDMS elastomer. However,
the five identical strain sensors used for demonstrating proprioceptive
perception and gesture monitoring were composed of 4.5 × 0.2
cm bundled CNF films embedded in PDMS elastomeric layers.For
all the experiments involving tactile sensing and gesture monitoring,
appropriately designed resistance-matched Wheatstone bridge circuits
were developed, to which the CNF-PDMS sensors were connected. The
voltage outputs from the Wheatstone bridge circuit were obtained and
logged using a National Instruments data acquisition system (DAQ,
NI USB-6009) with National Instruments LabView software. A constant
power supply of 9 V was used for powering the Wheatstone bridge circuit
during all the experiments. Furthermore, the output signals from the
bridge circuit were obtained continuously at a sample acquisition
rate of 2 kHz in a differential configuration.
Neural Coding
Neuron Model
The conversion from the sensors’
voltage recorded during the different experiments was conducted on
a spiking neural network simulator called Brian2.[33] The latter is an open-source python library used for recreating
spiking neurons and synapses with custom properties. In all the experiments,
voltage has been converted to a linear current Isensor = GVsensor, where G is a transconductance constant, with value = 1 pS. This
is made to emulate an ideal voltage-to-current converter, able to
stimulate the neuron’s membrane capacitor.The model
used for simulating a neuron response is the LIF one. This model has
been chosen due to its simple software implementation and the possibility
to easily obtain a closed formula for an analytical computation. It
can be expressed with the formulawhere Vmem is
the voltage difference between the neuron inside and the neuron outside, C is the capacitive value of the membrane capacitor present
in biological neurons, τmem = RC is the time constant of the membrane originating from the capacitive
term and the leaky component, while Vthr is the threshold voltage that generates a spike. The capacitor integrates
the current coming either from the synapse (Isyn) or directly from the sensors (Isensor). Both these currents increase the voltage Vmem. The voltage leaks through the membrane with a time constant
of τmem. When the threshold is reached, Vmem is rebased to its resting voltage (Vrest), ready for integrating new current.Neurons
communicate with each other through spikes. Spikes are
converted into currents through the synapses. Every time a neuron
spikes, an excitatory post-spike current is generated and injected
in a neuron. This is typically observed in biological neurons, and
this also has a circuit equivalent in the literature. The spike is
here expressed as a Dirac δ that is 1 only when the spike comes.
Each synapse has a dimensionless weight, here expressed as W
Encoding
Neurons communicate with
each other using
different paradigms. In this work, we exploited specifically two different
cases: the rate encoding and the spatial encoding. In the rate encoding,
the information is condensed in the spike rate of a single neuron.
The injected current charges the capacitor voltage and makes the neuron
spike. The higher the current, more are the times the neuron spikes
with that current. The spike number directly communicates the amount
of injected current into the neuron. Another way neurons communicate
with each other is spatial coding. Different neurons in a specific
layer have different roles, with each neuron representing a specific
concept. In the case, for example, of four possible stimuli at the
input and four different neurons, each neuron spikes when the stimulus
that it is representing appears.
Network for Gesture Recognition
In neural networks,
the connection between two layers of neurons through synapses can
extract meaningful information from raw data, dividing them in semantic
classes. In this work, the neurons that encoded the five different
sensors were connected with synapses to five different output neurons
in a second layer. Synapses are electrical components that convert
the spike of a neuron (which is a voltage difference) into a current.
The amount of current into which a spike is converted depends on the
synapse strength. In this experiment, the strength of the different
synapses was tuned to excite the upcoming neurons only with the right
combination of input neurons. The combination was chosen to activate
the semantic neurons only when the right gesture was performed. The
neuron that must spike for the gesture “FOUR”, for example,
has a very strong connection with the thumb sensor. Neurons compete
between each other, and only the strongest one wins. In the case “FOUR”,
this neuron is the one winning when only the thumb is tilted. Gesture
“FIVE” instead needs a detailed description, given that
the open palm encodes for it. This means that no sensors are active
while performing the gesture. This problem can be solved by introducing
a sixth neuron in the input layer that is always spiking with a Poisson
distribution. When only the sixth neuron is active, then the network
understands the presence of the gesture “FIVE”.
Authors: Conor S Boland; Umar Khan; Claudia Backes; Arlene O'Neill; Joe McCauley; Shane Duane; Ravi Shanker; Yang Liu; Izabela Jurewicz; Alan B Dalton; Jonathan N Coleman Journal: ACS Nano Date: 2014-08-19 Impact factor: 15.881
Authors: Giacomo Indiveri; Bernabé Linares-Barranco; Tara Julia Hamilton; André van Schaik; Ralph Etienne-Cummings; Tobi Delbruck; Shih-Chii Liu; Piotr Dudek; Philipp Häfliger; Sylvie Renaud; Johannes Schemmel; Gert Cauwenberghs; John Arthur; Kai Hynna; Fopefolu Folowosele; Sylvain Saighi; Teresa Serrano-Gotarredona; Jayawan Wijekoon; Yingxue Wang; Kwabena Boahen Journal: Front Neurosci Date: 2011-05-31 Impact factor: 4.677
Authors: Luke E Osborn; Andrei Dragomir; Joseph L Betthauser; Christopher L Hunt; Harrison H Nguyen; Rahul R Kaliki; Nitish V Thakor Journal: Sci Robot Date: 2018-06-20