Tactile feedback display with spatial and temporal resolutions.

We report the electronic recording of the touch contact and pressure using an active matrix pressure sensor array made of transparent zinc oxide thin-film transistors and tactile feedback display using an array of diaphragm actuators made of an interpenetrating polymer elastomer network. Digital replay, editing and manipulation of the recorded touch events were demonstrated with both spatial and temporal resolutions. Analog reproduction of the force is also shown possible using the polymer actuators, despite of the high driving voltage. The ability to record, store, edit, and replay touch information adds an additional dimension to digital technologies and extends the capabilities of modern information exchange with the potential to revolutionize physical learning, social networking, e-commerce, robotics, gaming, medical and military applications.

Today's digital technologies, which record, store, and remotely reproduce audio and video information, allow users to immerse themselves in experiences from distant locations, or even from the past. The invention of microphones, cameras, and camcorders dramatically changed communication, learning, business, and human interaction in general; however, these technologies only take advantage of sight and hearing - two of the five senses (sight, hearing, touch, smell, and taste) that we use to perceive the world and communicate with each other. Integration of other sense information will enable additional dimensions in current digital technologies, potentially revolutionizing information exchange, and improving our lives. There has been significant interest in recording and reproduction of other senses. For example, smell has been recorded using electronic nose12, and produced or reproduced remotely using odor release devices34. The integration of smell recording and reproduction with video capture and replay56 has also been demonstrated.

Touch sense plays a significant role in our daily lives, particularly in personal interaction, learning and child development. For example, research has shown that a mother's touch, especially skin-to-skin care, plays an important role in the healthy development of an infant's head7. In general, human touch sensation includes the feeling of physical contact/force/pressure, texture/deformation, hot/cold, moisture, and pain/itch, which make it very difficult to fully record and reproduce. There has been significant progress on flexible and sensitive pressure sensors89101112131415161718 and tactile displays1920.Tactile feedback displays have also been demonstrated for different applications2122232425. In this article, we report the transparent ZnO thin-film transistor (TFT) based pressure sensors, the electronic recoding and analog reproduction of the force using the elastic polymer actuators. Digital replay and editing/manipulation of the recorded touch events were demonstrated with spatiotemporal resolution, using an 8 × 8 active matrix ZnO pressure sensor array, a data acquisition and processing system (sensor array reader circuit, computer, and actuator array driver circuit), and a semi-rigid 8 × 8 polymer diaphragm actuator array26272829.

Results

Figure 1 shows the structure and electrical characteristics of a ZnO TFT based pressure sensor. Figure 1A shows the cross-section and Figure 1B shows the top-view scanning electron micrograph (SEM) of a single device (see also Figure S1 in Supplementary Information (SI) for an optical image of an 8 × 8 array, also note that the ZnO sensor arrays are fully transparent). Figures 1C and 1D show the Id – Vds and Id - Vgs FET characteristics, respectively. The highest obtained ON/OFF ratio is 4.8 × 103 at Vd = −3 V, Vg = +/−20 V, while the typical value is 102–103. The ZnO TFT devices showed sensitive response to applied pressure at the ON state and little response at the OFF state (Figure S2). Figure 1E shows the drain current as different weights are applied on top of the TFT every 20 s using a pulsed measurement (see Figures S3 and S4). The percentage amount of current change relative to current at zero pressure as a function of pressure is plotted in Figure 1F and a linear relationship is observed (also see Figures S5–7). We designed and fabricated an 8 × 8 array of ZnO TFT sensors (Figure S8) to record pressure with a spatial resolution of 3 mm using a (x, y) coordinate system (Figures S9). The 3 mm pitch was chosen to match that of the actuator array for pressure reproduction.

Figure 1

Structure and electrical properties of a ZnO TFT pressure sensor.

(A) Cross section schematic and (B) SEM micrograph of a ZnO FET device with channel dimension of 10 μm × 10 μm. ITO bottom gate, ZnO channel, and ITO source and drain are highlighted for clarity. Scale bar is 20 μm. (C) Id-Vds and (D) Ids-Vgs characteristics of a single ZnO TFT. In (C), the curves at Vg = −20 V and Vg = −10 V overlap. (E) Drain current vs time as additional weights are added on top of TFT every 20 seconds. The contact area is fixed. (F) Percentage change of drain current from (D) as a function of applied pressure with linear fitting.

To reproduce force in an analog manner, we fabricated an 8 × 8 array of actuators using an acrylic based dielectric elastomer with an interpenetrating polymer networks (IPN) structure2627. Figure 2A shows the working mechanism of a thin film actuator. Applied voltage creates electrostatic force between the electrodes, which compresses the IPN film and causes a large area expansion. When the actuator's boundary is fixed (Figure 2B), an out-of-plane diaphragm type of actuation can be generated. Figure 2C shows the optical images of a diaphragm actuator in OFF (top) and ON (bottom) states. Figure 2D shows the vertical displacement of an actuator with an applied voltage of 3.4 kV and a charging time of 200 ms (see measurement setup in Figure S10). The actuation can be modulated by adjusting both voltage and charging time, as shown in Figures 2E and 2F.

Figure 2

Structure and performance of an IPN actuator.

(A) An elastomer thin film actuator with two patterned compliant electrodes with free boundary. The applied voltage increases area and reduces thickness. (B) A diaphragm polymer actuator with two patterned stretchable compliant electrodes. The applied voltage leads to off-the-plane actuation due to fixed boundary. (C) Polymer actuator in ON (bottom) and OFF (top) position. Scale bar is 1,500 μm. (D) Actuation vs time (top) vertical displacement measured by laser deflection, (middle) voltage developed across the actuator, (bottom) digital logic control voltage. (E) Maximum actuator displacement vs charging time under fixed 3.5 kV bias. (F) Maximum displacement vs applied voltage with fixed charging time of 200 ms.

The demonstration of pressure sensing and reproduction using a prototype tactile feedback display system (Figures S14, S15) is shown in Figure 3. Figure 3A shows a transparent 8 × 8 array of ZnO TFT sensors, with the active sensing area highlighted. The difference between acquired signal and baseline signal is proportional to applied pressure and shown as a surface map (three-dimensional, 3D) and two-dimensional (2D) intensity plot in Figures 3B and 3C, respectively. The position of local maxima indicates the point of pressure application, and the height of the maxima can be calibrated to the pressure intensity. Note that the columns 4 and 8 are floating due to a fabrication fault (Figure 3C). The peak response position is at pixel (7, 5). Each device in the array has different pressure response due to fabrication non-uniformity. We used an arbitrary threshold value to define the local maximum as the pressure point and use the location of local maximum as the input for the actuator activation. Figure 3D shows the resulting pattern, and Figure 3E displays the corresponding activation on the actuator array (highlighted by red circle). Figure 3F shows that it is possible to obtain a linear relationship between the output displacement on a polymer actuator and the input pressure applied to a ZnO sensor by adjusting the charging time based on Figures 1F and 2E (see Figure S16). Figures 3G–O and Video Clip A in the SI show force reproduction with temporal modification. Figures 3G–I show the 2D plot of the input signal from pressing on the ZnO sensor array sequentially at (2, 7), (5, 7), and (7, 7) positions, and Figures 3J–L show the real-time actuation from the corresponding pixels as indicated by the timestamps. Next, we play our recorded “touch movie” backwards in time. Figures 3M–O show the actuation of the same pixels but in a reverse order of (7, 7), (5, 7), and (2, 7).

Figure 3

Tactile feedback display system and demonstration of real-time reproduction and modification of touch contact with temporal and spatial resolutions.

(A) An 8 × 8 array of transparent ZnO TFT pressure sensors connected to the readout unit, with a soft insolating tip pressing in bottom right corner (soft coating eliminates mechanical damage and static charging). (B, C) 3D surface plot and 2D color map of the subtracted input signal. (D) Local maximum pattern, which is used as input to drive the actuator array. (E) Actuation of the corresponding pixel in the actuator array, with active pixel circled in red. (F) Proposed linear relationship of output actuation displacement vs the sensor input pressure. Sensing range is based on signal-to-noise ratio SNR = 3. (G–I) Time-lapsed input pressure 2D color map at different times. (J–L) Real-time actuation of the corresponding pixel to reproduce the input signals in G–I. The timestamp format is time:frames, where the last 2 digits display the frame count (0–30 at 30 fps). (M–O) The recorded pressure “movie” replayed backwards on the actuator array (modified reproduction with temporal resolution). The replay can be in any sequence. (P) 3D surface plot, (Q) 2D color map, and (R) local maxima of input signals with two pressure points. Pressure remains roughly constant for entire measurement. (S) Unmodified reproduction of contact pattern on the actuator array. (T) Transpose reproduction of initial pattern. (U) Spatially translated reproduction of initial pattern. The (x, y) coordinate for pixels on both the sensor and actuator array was shown at the bottom of the figure.

We also demonstrate real-time spatial editing of the touch sense in Video Clip B and Figure 3P–U. Two pixels at (4, 2) and (4, 7) were pressed simultaneously and pressure remained mostly constant during the experiment. Figures 3P and 3Q display a snapshot of the sensor responses to applied pressure in 3D and 2D maps, respectively, and a local maxima pattern (Figure 3R) was used to drive the actuator array. For the first 15 seconds, the input signal is directly reproduced on the actuator array. Figure 3S shows the direct replay of input signal with actuators at positions (4, 2) and (4, 7) being turned on, demonstrating real-time pressure reproduction with spatial resolution. For the next 15 seconds, we reproduce the transposed input signal with mapping (x, y) -> (y, x) and pixels (2, 4) and (7, 4) on the actuator array turned on (see Figure 3T). Next, the input signal is spatially translated, with the mapping (x, y)-> (x + 2, y-2) and reproduced on the actuator array with (6, 8) and (6, 5) turned on (0 = 8 in the 8 × 8 array addressing), as shown in Figure 3U.

Discussion

In this article, we report the transparent ZnO thin film pressure sensors with high sensitivity and active matrix sensor arrays. The sputtered ZnO thin film devices simultaneously function as a transistor and pressure sensor. ZnO thin films prepared by sputtering, CVD, or solution methods have been broadly researched for TFT applications30313233. There is a very large variation in reported device performance and ON/OFF ratios as large as 105–107 have been reported3033. The sputtered ZnO TFT's reported in this paper show excellent device performance, with the typical ON/OFF ratio of 102–103 and up to 4.8 × 103 at Vd = −3 V, Vg = +/−20 V. The ZnO TFT devices showed sensitive response to applied pressure in the ON state and little response in the OFF state (Figure S2).

The ZnO nanocrystals, with the c-axis normal to the substrate surface, respond to the pressure and create an overall piezopotential due to the piezoelectric effect3435. The piezopotential serves as an additional gate voltage and increases the current in the ZnO TFT channel (also see discussion in SI). Using pulsed measurement, the measurement noise is smaller than 1.6 nA (Figure S7). Considering a SNR = 3, the sensitivity of our pressure measurement system can be extrapolated to be around 15 KPa, which is near the level of a gentle human touch of approximately 10 KPa15. Note that, to match the pitch of the actuator array for pressure reproduction, we selected a 3 mm pitch for sensor array, which is much larger than some of the recent reported pressure sensor resolutions141518. However, the actual FET channel is only 5 μm × 190 μm. Moreover, much smaller devices (e.g. 5 μm × 5 μm) and thus very high density sensing resolution, even exceeding human capacity (about 45 μm)36, is achievable. Furthermore, the ZnO sensors and sensor arrays reported here are fully transparent (Figure S1), which makes it particularly interesting for touch-screen applications in mobile devices. On-going efforts involve the fabrication of flexible ZnO FET sensor arrays based on a Ni facilitated transfer technique37.

The polymer actuators were chosen for the tactile feedback display demonstration due to their large displacement. The polymer actuator is capable of adjusting the frequency and duty cycle (hence, displacement) for force reproduction. The working frequency of the actuators is limited mainly by the viscoelasticity of the IPN material, as well as the conductivity of the compliant electrode. For a typical actuator, the peak performance (~500 mm vertical deformation) can be fully realized at 1 Hz (Figure S12). Higher frequency is possible at the cost of reduced vertical deformation. Frequencies up to 1 KHz can be achieved using different polymers38. Improving the conductivity of the compliant electrode by using a thicker SWNT coating will lead to a better frequency response while the stiffened electrode will limit the maximum strain28. The actuation (displacement and force) can be modulated by adjusting both voltage and charging time. Figure S13 shows the measured force produced by an actuator with one and two IPN stacking layers. A typical blocking force of a 4-layer stack actuator (used in this study) is in the range of 5–10 mN, which corresponds to a pressure of 2.8 to 5.6 KPa. Multiple layers of IPN films were stacked to increase the actuation force output but displacement was reduced (Figure S14). New materials are being developed to reduce the viscoelasticity and improve the actuation performance29. Another critical issue regarding the tactile display is the operational stability. Diaphragm actuators for tactile feedback display are comprised of one or more electroactive polymer layers (IPN), compliant electrode coatings, and a top plastic layer with circular openings to allow the IPN polymer films to output large, out of plane deformation. With proper compliant electrode coatings2838, the IPN polymers could be actuated for as many as 1 million cycles at strains larger than 100% or raised height greater than 0.5 mm in a diaphragm configuration. In the diaphragm array (display) configuration, we have tested devices with cycle lifetime greater than 10,000 cycles. Potential failures can be caused by structure and materials failure. Stress concentration due to sharp points on the edge of the circular opening of the top polymer layer was found to be a major cause for the reduced stability. Electromechanical instability could also play a role, which can be resolved by new polymer synthesis39. Also note that for practical applications, different kind of or a combination of more than one kind of actuator technologies4041424344 can be used.

We have also shown that it is possible to obtain a linear relationship of actuator output (actuation) vs. pressure input recorded by the sensor (Figure 3f) with varied actuator driving conditions, although not directly demonstrated due to the sensor measurement and actuator driving circuitry limitations. The linear response of the current recording vs. pressure (digitizing pressure, in Figure 1f) and the linear actuator displacement vs. input pressure on sensors demonstrate the electronic recoding and analog reproduction of the force using the elastic polymer actuators. Our results strongly indicate the ability of the real-time recording and reproduction of touch (contact and pressure). On-going efforts involve the calibration of actuator output force with different displacements and improving uniformity of the pressure sensor array. Human perception of strength of pressure depends on actuator displacement (amplitude), driving frequency, and duty cycle (percentage of time actuator spends in the ON state vs the OFF state)45. For a realistic pressure reproduction, a human subject study is necessary to appropriately calibrate the actuator response, in which case, different kinds of actuators can be used.

The feeling of physical contact is the most important and required element of the touch sense. In our study, by selecting a local maximum and setting up a threshold, we have demonstrated digital recording of touch contact (1 or 0) using an 8 × 8 active matrix pressure ZnO sensor array, and subsequently the digital replay of the recorded touch events with spatiotemporal resolution by using a semi-rigid 8 × 8 polymer diaphragm actuator array. Editing and manipulation of input signal (acquisition and processing) and subsequent display on the actuator array with spatiotemporal resolution were also demonstrated. The ability to digitize the touch contact enables direct remote transfer of touch information, long-term memory storage, and “replay” at a later time. Moreover, the ability to reproduce and modify touch with both temporal and spatial resolutions makes “synthesized” touch possible. It could create experiences that do not exist in nature, similar to synthesized music, computer-generated imagery, and montage filmmaking. These results open up opportunities of enabling touch information as an additional dimension to digital technologies and thus extending the capabilities of modern information exchange that have revolutionary and positive impacts to education and physical training, digital media, social networking, medical rehabilitation, e-commerce, robotics, gaming, military applications, etc.

Methods

TFT and sensor fabrication

The ZnO TFT array was fabricated in a cleanroom using standard photolithography. Gate (150 nm ITO), channel (150 nm ZnO), source and drain (150 nm ITO) were formed by RF sputtering (Denton Discovery 18). PECVD (Oxford Plasmalab 80 Plus) was used to deposit 350 nm of SiNx as gate dielectric. The device was passivated by depositing 40 nm of Al2O3 using atomic layer deposition (ALD) (Beneq TFS200). The ZnO TFT single device has channel dimensions of 10 μm × 10 μm, while the devices in the 8 × 8 array have 5 μm channel length and 190 μm channel width. The array pitch is 3 mm. Figure S8 shows the array layout.

Single TFT device testing

We used a pulsed measurement technique to obtain the drain current (Id) as a function of time (for single device or array). First, the TFT is set to ON bias (Vd = −2 - −3 V, Vg = +20 V) and the current is integrated over a capacitor for 5 ms. The capacitor voltage is readout with an ADC. Next, the TFT is set to OFF bias (Vd = 2-3 V, Vg = −20 V) to allow the device to return to its initial state and the integration capacitor is discharged. This cycle is repeated continuously.

Pressure response measurement

For each 20 s weight application interval, the average value of current in the middle 10 s is calculated, from which the value at no pressure applied was then subtracted, and result was normalized by the original current level and plotted as a function of pressure in Figure 1F.

Pressure sensor array testing

We used time-domain multiplexing to readout the whole array of sensors sequentially. Device (i, j) is activated by measuring current of the row i (selected using a multiplexer), while the gate column j is set to +20 V and every other gate column to −20 V.

Actuator fabrication

Actuator array was fabricated using an acrylic dielectric elastomer with an interpenetration polymer network structure (IPN) as active layer26 and a spray-coated single wall carbon nanotubes (SWNTs) as compliant electrodes28. A plastic stencil was laser cut and used to generate the electrode array pattern. The films with patterned electrodes were then stacked and attached onto a plastic substrate.

Single actuator testing

The vertical displacement and blocking force of the actuators were characterized with a laser displacement sensor (Acuity AR200-6) and a load cell (Transducer Techniques GSO-10) separately. The laser spot was aligned to the center of the actuator dot where silver paste was smeared to improve the reflection. The shaft of the load cell was placed right above the actuator top surface to measure the blocking force.

Actuator array addressing

An overview of the actuator control circuit is shown in Figure S14. The microcontroller activates the relays to select which actuators are connected to the high voltage and ground. When an actuator receives connections to both, it is activated. A high voltage is applied to selected actuators within a row. Next, actuators are discharged by disconnecting the high voltage and next row is selected. A fast sweep of all rows gives the appearance of all actuators being active simultaneously.

Tactile feedback display demonstration

The sensing unit (sensor array and readout system) and actuation unit (actuator array, high voltage source, and driver system) were integrated through a PC running Matlab software over serial communication (Figure S15). For real-time touch reproduction, touch data is collected and digitized on a PC and then sent to the actuator array driver circuit. When digitized by the PC, arbitrary signal processing can be performed to manipulate the output, if desired.

Author Contributions

D.W. contributed the concept and development of tactile feedback display system and supervised the project. T.N. contributed in initiating the idea of recording and reproduction of touch sense and numerous valuable discussions. S.V. and B.L. contributed equally in the device fabrication, sensor reader and actuator driver design/fabrication, and all the data collection and plotting; X.N., P.B., Q.P. contributed polymer actuator work; A.K., K.S., N.P., M.Y., Y.J., Z.S. and C.L. contributed with ZnO FET and sensor array fabrication; M.K. contributed with circuit design and fabrication. S.V., B.L. and D.W. wrote the paper, and X.N., P.B., Q.P. contributed to the writing of the polymer actuator sections.