A touch-based multimodal and cryptographic bio-human-machine interface.

The awareness of individuals’ biological status is critical for creating interactive and adaptive environments that can actively assist the users to achieve optimal outcomes. Accordingly, specialized human–machine interfaces—equipped with bioperception and interpretation capabilities—are required. To this end, we devised a multimodal cryptographic bio-human–machine interface (CB-HMI), which seamlessly translates the user’s touch-based entries into encrypted biochemical, biophysical, and biometric indices. As its central component, the CB-HMI features thin hydrogel-coated chemical sensors and inference algorithms to noninvasively and inconspicuously acquire biochemical indices such as circulating molecules that partition onto the skin (here, ethanol and acetaminophen). Additionally, the CB-HMI hosts physical sensors and associated algorithms to simultaneously acquire the user’s heart rate, blood oxygen level, and fingerprint minutiae pattern. Supported by human subject studies, we demonstrated the CB-HMI’s capability in terms of acquiring physiologically relevant readouts of target bioindices, as well as user-identifying and biometrically encrypting/decrypting these indices in situ (leveraging the fingerprint feature). By upgrading the common surrounding objects with the CB-HMI, we created interactive solutions for driving safety and medication use. Specifically, we demonstrated a vehicle-activation system and a medication-dispensing system, where the integrated CB-HMI uniquely enabled user bioauthentication (on the basis of the user’s biological state and identity) prior to rendering the intended services. Harnessing the levels of bioperception achieved by the CB-HMI and other intelligent HMIs, we can equip our surroundings with a comprehensive and deep awareness of individuals’ psychophysiological state and needs.

Human–machine interfaces (HMIs)—with built-in data acquisition, processing capabilities, and data security measures—are necessary for acquiring awareness of individuals’ status and assisting them with achieving optimal outcomes. Accordingly, various classes of HMIs have been developed and integrated within the ubiquitous electronics ecosystem to capture user-specific inputs through various channels, including touch (1), voice (2), and gesture (3, 4). Such HMI modalities have laid the foundation for emerging technologies such as robotics (5), smart home (6), autonomous driving (7), and augmented reality (8), which are already transforming our current ways of living.

However, these HMIs provide limited awareness of individuals’ biological status, which in turn prohibits their utility to unlock broader applications such as those in the healthcare domain (9). This limitation is because of their intrinsic shortcomings in collectively accessing informative biochemical and biophysical indices—in other words, they lack “biological perception.” Toward devising a suitable HMI modality that bypasses such shortcomings, a key point of consideration is that biochemical indices (e.g., circulating biomarker molecules) are not trivially accessible due to the natural skin barrier function (10)—in contrast to putative biophysical indices (e.g., heart rate), which can be accessed inconspicuously using existing HMI-compatible sensors (e.g., optical devices). Also relevant to the context at hand, given the personal and sensitive nature of the target indices, any devised biocentered HMI modality should account for user identification and data protection (e.g., encryption) (11, 12).

Fingertips could serve as ideal human body sites for engineering a biocentered HMI for numerous reasons. First, many of the circulating biomarker molecules partition onto the skin surface of the fingertip (primarily via natural perspiration) with a relatively high flux (13–17). Leveraging this phenomenon, we previously introduced a thin hydrogel-based (TH) electrochemical sensing technique, wherein the hydrogel interface is employed as a medium for epidermal analyte sampling/sensing at the point of the fingertip (18). The inconspicuous nature of this sampling method allows for bypassing the challenges associated with conventional biomarker sampling modalities, which are invasive (e.g., fingerstick blood sampling), require external stimulation [e.g., iontophoretic sweat or interstitial fluid sampling (19–21)], and/or deviate from individuals’ routine behavior (e.g., drooling for saliva collection). Second, clinically significant biophysical indices such as heart rate (HR) and oxygen saturation level (SpO2) can be simultaneously acquired from fingertips using standard noninvasive methods (e.g., photoplethysmography). Third, the fingertip’s unique biometric feature (i.e., fingerprint) can be leveraged for user identification and as a personalized cryptographic key for in situ data encryption (22). Fourth, fingertip-based interactions are already the primary mode of communication in the readily proliferated HMIs (e.g., touch-sensitive screens, console controllers, trackpads, and keyboards).

Central to realizing the envisioned touch-based HMI, new TH-sensor-based bioanalytical capabilities must be developed to advance touch-based TH sensing for acquiring physiologically meaningful measures of biochemical indices. Specifically, a dedicated sensor design and a signal interpretation framework must be developed to account for the unique—yet previously unexamined—analyte mass transport behavior at hand. To elaborate, unlike the conventional sample-to-answer biochemical sensing scenarios, where the analyte concentration in a given biofluid sample is directly measured as the primary quantity of interest (in a quasi-static setting), in touch-based TH sensing the continuous epidermal influx of analytes into the hydrogel is primarily measured. The development of this framework is crucial for guiding the sensor development efforts, decoupling the influence of confounding factors, and ultimately translating the TH-sensor readouts to physiologically meaningful measurements. Furthermore, additional electrochemical-based TH-sensing methodologies must be developed to enable the detection of a broader spectrum of analytes and expansion of the potential application spaces of the envisioned HMI.

Addressing such TH-sensing limitations, here we present a dedicated framework for TH-sensor design and signal interpretation and illustrate its physiological utility by augmenting it with newly developed specialized TH sensors. We particularly formulated and validated the proposed framework via convergent mass transport modeling, simulation, and in vitro characterization (using an artificial fingertip setup). Critical for rendering physiologically relevant readouts, this framework was utilized to decouple the signal-distorting effect of the pressing force variability—a previously overlooked but major confounder—which is intrinsic to the touch-based HMI modality. Furthermore, guided by this framework, we thinned down the hydrogel interface to minimize the required fingertip/TH-sensor contact time (∼30 s) well below the wait/contact time reported in previous TH-sensor studies (on the order of minutes) (23, 24).

Our TH-sensor development efforts focused on targeting ethanol (for alcohol monitoring) and acetaminophen (APAP, a widely used analgesic medication), as example analytes with high significance in personal biomonitoring. For example, monitoring of alcohol could prevent personal/societal harmful outcomes such as alcohol abuse and driving under the influence (DUI) (25). Also, monitoring of APAP may be particularly beneficial in promoting patient compliance and safety, given that APAP supratherapeutic administration is the leading cause of liver failure in the United States (26). We specifically developed an amperometric TH sensor to enzymatically measure ethanol and an interference-free voltammetric TH sensor to target APAP. Particularly noteworthy is the importance of the devised voltammetric TH-sensing methodology in broadly enabling the detection of electroactive biomarkers such as many pharmaceuticals in the presence of nontargeted interfering electroactive species. This methodology is distinct from previously reported TH-sensing approaches that are mostly enzymatic. By employing the developed bioanalytical capabilities in human subject studies, we demonstrated the correlation of the temporal profiles of the epidermally accessed biochemical indices to their corresponding circulating profiles with a relatively high temporal granularity.

The developed bioanalytical capabilities and the findings from our physiological studies provided a foundation for engineering a touch-based cryptographic bio-HMI (CB-HMI; Fig. 1) capable of translating the user’s touch-based entries into encrypted biochemical, biophysical, and biometric indices. To engineer the CB-HMI, we integrated each of the developed TH sensors with a photoplethysmography (PPG) sensor and a fingerprint scanner within a unified platform, allowing for obtaining additional bioinputs such as the user’s HR and SpO2 levels and the minutiae pattern. For seamless and multimodal sensor signal acquisition and data processing, we augmented the sensor readings with dedicated signal conditioning circuitries and inference/encryption algorithms, forming a complete HMI solution.

Fig. 1.

Enabling bioperception and interpretation via CB-HMI. (A) Illustration of the translation of the user’s touch-based entry into bio-inputs. (Right) Exploded view of the CB-HMI, including the TH-sensor component and corresponding signal interpretation framework. (B) CB-HMI’s operational workflow, including its augmentation with feedback mechanisms. (C) Conceptual illustration of an ecosystem of objects, equipped with CB-HMI and conventional HMIs (e.g., touchpad and camera), forming a smart surrounding.

Harnessing its bio-perception/interpretation functionalities, the CB-HMI can be integrated within surrounding objects to intelligently and interactively determine and deliver the appropriate course of action for each user (feedback, Fig. 1). To this end, we applied the devised CB-HMI within two representative scenarios: driving safety and medication use, where the developed ethanol and APAP TH sensors are employed to acquire the relevant biochemical index, respectively. Accordingly, we demonstrated a vehicle-activation system and a medication-dispensing system, where the integrated CB-HMI uniquely enabled user bioauthentication (on the basis of the user’s biological state and identity) prior to rendering the intended services.

These demonstrated applications illustrate the CB-HMI’s capability in upgrading the surrounding objects to attain biological perception at an unprecedented level. Ultimately, the ubiquitous integration/deployment of the CB-HMI and other HMIs will create smart surroundings—equipped with deep and comprehensive awareness of the individuals’ psychophysiological state and needs—which can actively assist the users in reaching their optimal outcomes efficiently (Fig. 1).

Results

Formulation of an Epidermal TH-Sensing Signal Interpretation Framework.

Central to the realization of bioperception, to acquire physiologically interpretable TH-sensor readouts we formulated a generalizable TH-sensing signal interpretation framework. This framework specifically accounts for the unique mass transport context at hand, involving the continuous epidermal influx of analytes into the hydrogel medium, and subsequently onto the sensing surface. For simplicity and toward building intuition, we first assume that epidermally delivered analytes are diffused immediately and uniformly within the hydrogel. In this setting, the analyte concentration in the hydrogel CH during the sampling/measurement time t can be described aswhere N denotes the amount of analytes in the hydrogel, VH denotes the original volume of the hydrogel, and VT accounts for the added volume contributed by transepidermal processes such as the natural perspiration and water loss. Assuming that natural perspiration is the dominant process for epidermal analyte flux, CH can be represented as

In this equation, AH and h denote the hydrogel’s fingertip facing area and height, respectively. Further, Cp and Rp denote the perspired analyte concentration and natural perspiration rate (secreted volume per unit area and time) correspondingly, where their product effectively captures the analyte influx (J) in our case. Given the presumed relatively short sampling time, Cp, Rp, and subsequently J can be considered to be constant during the course of sampling, or equivalently, representable by their average values. Following the same assumption, the contributed volume by the perspiration can be neglected as compared to the volume of the hydrogel. In this way, the derived expression for CH can be simplified as

Examining this equation can yield insight in a number of ways. First, considering the inherent proportionality of the underlying sensor readout to CH, and J to CP, this equation allows for meaningfully correlating the sensor readout to the perspired analyte concentration, which serves as a noninvasive proxy measure of the analyte’s circulating level (27). Second, noting the time linear dependency of the sensor readout motivates defining the signal based on the first derivative of the sensor readout with respect to time. This definition allows for both extracting the analyte flux information and eliminating the influence of relatively time-independent confounding factors. Third, this equation illustrates that by thinning down the hydrogel (decreasing h) we can effectively amplify the signal, which could be particularly useful in quantifying analytes with low circulating levels (e.g., many pharmaceuticals and hormones).

Revisiting our formulated model, we can more specifically account for analyte diffusion within the hydrogel by considering the time that it takes for the introduced analytes to diffuse to the sensing surface (tD). In our context, tD can be estimated following Fick’s law:where D is the analyte diffusion coefficient within the hydrogel. This time scale manifests as a lag time between the fingertip contact and the observed sensor response, indicating another advantage of using thin hydrogels in TH sensing (i.e., facilitating rapid response).

To validate our simplified model, we performed a finite element analysis, simulating the analyte transport behavior (using COMSOL software). We primarily simulated the analyte concentration profile (within the hydrogel layer) across time for different levels of analyte flux inputs. As shown in Fig. 2, for a given input flux, after an initial lag-time (related to tD), the analyte concentration at the vicinity of the sensing surface increases linearly with time. The observed slope for each case is proportional to the input flux (Fig. 2). We further extended the simulation to capture the influence of hydrogel thickness on the TH-sensing performance. As shown in Fig. 2, a thinner hydrogel results in a reduced time lag and an enhanced on-sensor flux—aligned with the conclusions derived from our simplified model. Guided by these results, in our TH-sensor fabrication effort we particularly thinned down the hydrogel layer (by introducing a rehydration step in the fabrication process; ) to enhance the sensor performance.

Fig. 2.

Development and validation of a TH-sensing signal interpretation framework. (A–C) Simulation of analyte concentration profile within the TH. (A) Simulated temporal profiles of analyte concentration at the vicinity of the sensor surface for a representative input flux (TH thickness: 220 µm, input flux: 2.4 nmol⋅min−1⋅cm−2). (Inset) Side view schematic of the analyte delivery to the sensor surface. (B) Simulated on-sensor analyte flux (for t ≫ tD) versus epidermal analyte flux, illustrating a linear relationship. (Inset) The corresponding simulated temporal profiles of analyte concentration at the vicinity of the sensor surface. (C) TH-transport lag time and on-sensor flux versus hydrogel thickness, demonstrating the superior TH-sensor performance (reduced lag time and enhanced on-sensor flux) when using a thinner hydrogel. (D) Exploded view of the artificial fingertip. (Inset) The optical images of the laser-patterned micropore layer (with two different zoom-in views). (E) Ex situ characterization of the TH sensor via a microfluidic artificial fingertip. (F and G) Top-view photos and reaction schematics of the ethanol TH sensor (F) and the APAP TH sensor (G). WE, CE, and RE correspondingly denote working, counter, and reference electrodes. (Scale bar: 2 mm.) (H) Amperometric recording of an ethanol TH sensor upon contact with an ethanol-contained artificial fingertip (concentration: 2 mM). (I) Amperometric recording of an ethanol TH sensor in the presence of a varying pressing force profile. (Top) The exerted force profile. (Middle and Bottom) The measured TH sensor’s responses to the input fluid containing 0 and 2 mM ethanol.

Development of a Microfluidic Artificial Fingertip for Characterization and Empirical Validation of the TH Sensors and Signal Interpretation Framework.

To empirically validate the presented model and characterize the subsequently developed TH sensors in an ex situ setting, we created a microfluidic artificial fingertip, which emulates the analyte flux on a fingertip via natural perspiration (Fig. 2 and ). This setup is particularly useful for characterizing the TH sensors’ responses with respect to the analyte-related parameters (e.g., concentration and mass transport delivery rate) and CB-HMI–specific influential factors (e.g., fingertip mechanical contact force). The developed artificial fingertip consists of 1) a microfluidic chamber to facilitate the delivery of the input fluids (with dynamically varying composition and at adjustable flow rates) via a programmable pump (mimicking the thermoregulatory sweat secretion by the secretory coil); 2) a pressure regulation layer (28), mimicking the highly fluidically resistive sweat duct, to render a stable low flow rate [<1 μL⋅min−1⋅cm−2, corresponding to the natural perspiration rate (29)]; and 3) a laser-patterned pore layer [with the pore size comparable with the sweat duct diameter, ∼80 μm (30)], mimicking the fingertip skin surface (to be interfaced with the TH sensor). To characterize the constructed artificial fingertip from the standpoint of fluid secretion, we monitored the hydraulic pressure across the microfluidic chamber, under various input flow rates. As can be seen in , for each of the flow rates a relatively flat pressure profile was measured, indicating the artificial fingertip’s microfluidic capability to reliably deliver the intended analyte flux within the expected range of secretion rate.

Development and Ex Situ Characterization of TH Sensors for Epidermal Biosensing.

Recognizing the identical biofluidic origin of our target analytes in our epidermal TH-sensing and conventional sweat-sensing contexts, to develop our TH sensors we adapted electrochemical sensing methodologies that enable reliable analyte quantification in the complex sweat matrix. Accordingly, to construct the underlying sensing surface of the ethanol TH sensor (Fig. 2 and ) we utilized a mediator-free electro-enzymatic sensing methodology, which is advantageous over conventional mediator-based enzymatic sensing approaches that are prone to ionic interference (27). Furthermore, to construct the underlying sensing surface of the APAP TH sensor (Fig. 2 and ) we utilized a surface engineering technique that allows for creating an “undistorted potential window” within which the target analyte’s voltammetric response becomes dominant and interference is eliminated.

The constructed ethanol sensor is composed of 1) an enzyme layer (alcohol oxidase, AOx) to catalyze the oxidation of ethanol and generate hydrogen peroxide (H2O2) as a detectable by-product, 2) a permselective membrane (poly-m-phenylenediamine) to reject interfering electroactive species, and 3) an electroanalysis layer (platinum nanoparticle, PtNP) to detect the generated H2O2. The constructed APAP sensor is based on 1) a polymeric coating (Nafion) to mitigate the interference of the electroactive species and enhance the biofouling resistance and 2) a voltammetric sensing electrode (hydrogen-terminated boron-doped diamond electrode, H-BDDE) to selectively detect the oxidation peak of APAP. show the linear response (limit of detection: 0.13 mM for ethanol and 0.12 μM for APAP) and high selectivity of the developed sensing surfaces, demonstrating their suitability for the envisioned CB-HMI application. The developed sensing surfaces were coupled with a thin hydrogel layer to form the envisioned TH sensors and further augmented with our custom-developed signal interpretation framework to contextualize their readings for on-body applications.

To characterize the TH sensor’s response in relation to the envisioned touch-introduced analyte flux, we particularly exploited the ethanol sensor, which allows for tracking its response in real-time via amperometry. Accordingly, we interfaced the artificial fingertip with the ethanol TH sensor and configured it to continuously deliver ethanol molecules at a set flux (here, 0.64 nmol⋅min−1⋅cm−2). Also, we utilized a commercial benchtop potentiostat to record the generated sensor response. As illustrated in Fig. 2, after an initial current jump (corresponding to the formation of the microfluidic fingertip/hydrogel contact) the sensor presented increasingly current response levels with a constant slope (denoted as k). Since the sensor’s response is proportional to CH, aligned with our aforedescribed theoretical analysis, the observed slope is leveraged to represent the introduced analyte flux J.

Relevant to the envisioned touch-based sensing context, the same artificial fingertip characterization setup was utilized to investigate the potential confounding influence of variability in the strength of the pressing force. Accordingly, as the sensor response was being recorded continuously, different weights were mounted onto the artificial fingertip to mimic different strengths of pressing [corresponding to the normal pressure range exerted by finger pressing (31)]. As shown in Fig. 2, while initially a current jump was observed when making a contact between the weight-mounted artificial fingertip and the TH sensor, mounting additional weights did not significantly alter the sensor response. In particular, zero slope was consistently measured in the absence of analyte flux (i.e., using a blank buffer solution as fluid input) and a relatively unchanged finite slope was observed for the case of nonzero analyte flux.

Furthermore, the artificial fingertip was utilized to systematically validate the TH sensor’s ability to track the analyte flux for varying introduced analyte concentration and flow rate conditions. In our context, given the exogenous nature of the target analytes, the relative variations in analyte concentration (during the course of analyte circulation) is more significant than the relative changes in the natural perspiration rate [which in stationary settings is relatively stable (19, 32)]. Therefore, for this study we primarily focused on modulating the analyte concentration, while fixing the input flow rate within the physiologically relevant range.

Fig. 3 shows the real-time amperometric readouts of the ethanol TH sensor, where the observed slopes linearly scale with the input concentrations (concentration range: 0 to 10 mM, flow rate: 320 nL⋅min−1⋅cm−2). Fig. 3 illustrates the corresponding calibration curve constructed based on the measured slopes (R2 = 0.99). To characterize the APAP TH sensor, we captured the sensor’s voltammograms (3 min after the sample introduction) for the APAP concentrations ranging from 0 to 80 μM (introduced at the same aforementioned flow rate; Fig. 3). For each case, the voltammetric peak height (serving as the voltammetric signal) was extracted by applying our previously reported analytical framework and correcting for the effect of baseline variation (33). Fig. 3 shows the highly linear relationship between the input APAP concentration and the extracted signal (R2 = 0.99). The characterizations of the two TH sensors were further extended (using the same setup) by modulating the input flow rate (mimicking the natural perspiration rate variations) while keeping the respective analyte concentration constant. As shown in , for both cases linear relationships between the measured signals and the corresponding input flow rates were observed.

Fig. 3.

Ex vivo and in vivo characterization of TH sensors. (A–D) Ex vivo TH-sensors characterization using an artificial fingertip setup. (A) Amperometric responses of an ethanol TH sensor to input fluid with various ethanol concentrations (0, 2, 4, 6, 8, and 10 mM, all after contact with the artificial fingertip). (B) Ethanol TH-sensor calibration curve. Error bars indicate SE (three trials). (C) Differential pulse voltammograms of an APAP TH sensor 3 min after the introduction of the input fluid with various APAP concentrations (0, 20, 40, 60, and 80 μM). (D) APAP TH-sensor calibration curve. Error bars indicate SE (three trials). For all the experiments the input fluid was based on a phosphate-buffered saline solution, injected at 320 nL⋅min−1⋅cm−2. (E–J) In vivo TH-sensors characterization via human subject studies. (E and F) The captured and linear-fitted ethanol TH-sensor readouts from the index fingertip of a subject (entry at t = 0 s) before (E) and 60 min after (F) the intake of an alcoholic beverage (∼100 mL, 12.5% alcohol). (G) The temporal profile of the ethanol TH-sensor signal (obtained from the index fingertip of a subject) and concurrently measured BAC levels (20 min before and after an alcoholic beverage intake: ∼100 mL, 12.5% alcohol around t = 0 min). (H and I) The captured and baseline-fitted APAP TH-sensor readouts from the index fingertip of a subject before (H) and 60 min after (I) the intake of an APAP-based medication (650 mg APAP). (J) The temporal profile of the APAP TH-sensor signal (obtained from the index fingertip of a subject) and concurrently sampled salivary APAP levels (analyzed using liquid chromatography with tandem mass spectrometry) 5 min before and after an APAP-based medication (650 mg APAP at t = 0 min).

Building upon these ex situ sensor characterization results, we validated the TH sensors’ in situ epidermal detection capabilities by conducting human subject studies (Fig. 3 ). We specifically employed the ethanol TH sensor to assess whether the sensor is capable of differentiating between two states: no alcohol intake vs. recent intake of an alcoholic beverage. In this context, we defined the signal S as the slope of the sensor’s amperometric output between 10 and 30 s after fingertip contact, which accommodates for analyte transport lag time and settling of transient contact-induced disruptions and sensor response time. Also, the detection threshold is defined as Sfree, avg + 3 × Sfree, SD, where Sfree, avg and Sfree, SD are the average and SD of the signals obtained from the alcohol-free subject (three trials). In our studies, the threshold for determining the alcohol-free state was measured as 0.36 nA/min. We then performed touch-based TH-sensor measurements before (t = −20 min) and after (t = +60 min) the intake of an alcoholic beverage (∼100 mL, 12.5% alcohol). As shown in Fig. 3 , the ethanol TH sensor was capable of differentiating the pre/postintake events, as evident from the near-zero amperometric current slope for the preintake measurement and the relatively steep positive slope in the postintake measurement (S = 21.5 nA/min).

We adopted a similar study framework to evaluate the in situ detection capability of APAP TH sensor. For this context, the measured signal is defined as the voltammetric current peak height (IP), and the threshold for determining the APAP-free state was measured as 3.7 nA (three trials). Fig. 3 show the corresponding voltammograms measured by the APAP TH sensor, 10 min before and 60 min after the intake of an APAP-based medication (containing 650 mg APAP). The comparison of the two voltammograms and the corresponding extracted current peaks (I ∼0 nA vs. I ∼15 nA) validates that the developed voltammetric TH sensor is capable of detecting the presence of circulating APAP molecules via the devised touch-based modality.

The results from the TH sensor studies support the applicability of both mediator-free electro-enzymatic sensing and interference-free electroactive sensing methodologies that were originally formulated for direct sweat sensing (33, 34).

To further illustrate the physiological significance of the biochemical readings obtained from this modality, we extended our study framework to take fingertip-based measurements with a higher temporal resolution and compare the extracted profile with the target molecules’ circulating levels. Accordingly, the fingertip-based measurements were performed before and at intermittent time points after the alcoholic beverage/medication intake. To track the molecules’ circulating profile, standardized proxy measurements were performed: blood alcohol content (BAC) was estimated using an alcohol breathalyzer and salivary APAP was analyzed with mass spectrometry following our previously reported method (33). As shown in Fig. 3 , the temporal profiles of target molecules captured by the TH sensors closely mirror the molecules’ circulating profiles. A similar trend was observed in a second subject following the same study framework ().

Moreover, in a separate trial, we verified that the subject’s fingertip’s perspiration characteristic stays relatively stable in the context of our studies (), indicating that the observed trends of the analyte flux are due to the underlying changes in the analyte circulating levels but not the variations in perspiration rate. We further validated the consistency of the TH-sensor readouts by comparing the epidermally acquired temporal profiles of biochemical indices (e.g., APAP) across different fingertips, where minimal variations among respective measurements were observed ().

Integration of the Multimodal Data Acquisition and Processing Modules.

We integrated each of the developed TH sensors with a PPG sensor and a fingerprint scanner within a unified platform, allowing for additionally acquiring biophysical (HR and SpO2) and biometric (minutiae pattern) indices (). The platform is further integrated within an embedded system (), featuring dedicated signal conditioning circuitries and inference/encryption algorithms, to form a complete HMI solution.

Here, PPG was specifically selected for the acquisition of biophysical indices, because of its established clinical utility in assessing the user’s overall physiological status (e.g., cardiovascular health and respiration) and its noninvasive and touch-based nature (inline with our envisioned interaction modality) (35). The capability of the implemented PPG sensor and associated algorithms in deriving the underlying physiological signals such as HR and SpO2 were validated via human subject studies ().

To seamlessly associate the acquired biochemical and biophysical indices to the user, while preserving the user’s privacy, we developed a built-in user identification/data encryption scheme. This scheme capitalizes on the unique biometric feature of the fingerprint, which is inherently accessible in our envisioned HMI (via an implemented fingerprint scanner; ). The user identification is achieved by evaluating the similarity (s) between the minutiae patterns from the query and template fingerprints (). To demonstrate in situ data encryption/decryption, we realize a crypto-biometric system (CBS) based on a fuzzy vault scheme, which encrypts the secret (here, bioinputs) using a biometric key generated from the minutiae pattern () (36). In short, the bioinputs are encrypted in a vault using the minutiae-generated key A, which can be decrypted by query minutiae-generated key B only if B overlaps with A substantially (i.e., the two fingerprints are from the same individual). As shown in , this encryption process involves 1) transformation of the bioinputs into the coefficients of a polynomial equation, 2) projection of the fingerprint minutiae (here, distances among minutiae) onto the polynomial equation to create genuine points, 3) generation of polynomial-offset chaff points, and 4) creation of the vault list using the scrambled genuine points and chaff points. illustrates the corresponding decryption process. This process involves identifying the genuine points using the query minutiae feature and polynomial fitting to extract the coefficients, followed by reconstructing the bioinputs. Furthermore, shows the application of the implemented CBS given a hypothetical input, illustrating that the input is successfully encrypted/decrypted.

Development and Application of a CB-HMI–Enabled In-Vehicle Safety System.

To demonstrate its utility in a vehicle environment, which is an integral part of our daily experience, we first integrated the CB-HMI on a steering wheel to form an in-vehicle interactive system toward improving driving safety (Fig. 4 ). Enabled by the CB-HMI, the system can perceive the driver’s ethanol level (using the ethanol TH sensor), fingerprint, and PPG signal. These degrees of bioperception can be leveraged to bioauthenticate the drivers (interpretation) for vehicle activation (feedback). For bioauthentication, the system utilizes the bioinput entries to verify the driver’s alcohol-free state, biometric match, and liveness of the biometric input [to avoid tampering, by exploiting the PPG signal as a liveness indicator (37)]. Also applicable to this example setting, the derived HR and SpO2 information from the PPG measurements can serve as useful feedback to the driver, as they are informative measures of potential drowsiness (38, 39). As a proof of concept, here the authentication status, together with the acquired indices, are transmitted and displayed on a steering wheel–mounted liquid-crystal display (LCD) screen to provide visual feedback.

Fig. 4.

A CB-HMI–enabled in-vehicle safety system. (A) Operational workflow of the in-vehicle system. (B) Optical image of the in-vehicle system mounted on a steering wheel, illustrating the envisioned setting. (C–E) Validation of the system functionality in three hypothetical scenarios: nonowner/alcohol-free (C), owner with a recent alcohol intake (D, 1 h before fingertip entry, ∼100-mL beverage containing 12.5% alcohol), and owner/alcohol-free (E). For each scenario (i) the raw and processed ethanol TH-sensor readouts (upon a fingertip-based entry at t = 0 s); (ii) PPG sensor readouts (infrared and red channels); (iii) plot of relative coordinates and local ridge direction of the corresponding template and query fingerprints; and (iv) visual feedback to the user via an LCD, displaying the determined alcohol state of the subject, the derived HR and SpO2 information, and the concluded bioauthentication status. N.D. indicates no detection of alcohol.

To demonstrate the bioauthentication function of the developed in-vehicle interactive system, the system was tested in three hypothetical scenarios. For each scenario, upon finger pressing the ethanol readout, the PPG readout, and the fingerprint readout were acquired by the CB-HMI and the feedback information was communicated to the user via an LCD. Fig. 4 shows the input status and the raw and interpreted bioinputs (recorded and processed by the integrated modules), together with the generated feedback for each of the scenarios. In the first scenario (Fig. 4), while the presence of alcohol in the subject’s system was not detected (k < 0.36 nA/min), the activation request was rejected since the query fingerprint did not match the template of the vehicle owner. In the second scenario (Fig. 4), the activation request was rejected because of the detection of alcohol presence (k > 0.36 nA/min) in the driver’s circulating system (despite verification of the owner’s identity). In the last scenario (Fig. 4), biometric match and alcohol-free readouts were obtained (k < 0.36 nA/min), and the liveness of the biometric entry was verified, leading to granting the vehicle-activation request. For all these scenarios, the system acquired and interpreted the relevant bioinputs successfully and identified and communicated the intended course of action correctly.

Development and Application of a CB-HMI–Enabled Medication-Dispensing Robotic System.

To demonstrate the versatility of the CB-HMI and its enabling application, we illustrated its utility for assisting user(s) with their pharmacotherapy regimen. Accordingly, we integrated the CB-HMI into a custom-developed pill case to realize an unprecedented crypto/smart medication-dispensing robotic system (Fig. 5 ). This system perceives the users’ bioindices, including medication level (here APAP, using the APAP TH sensor), fingerprint, HR, and SpO2 to bioauthenticate the user (for medication dispensing), verify the medication intake, and update the personal electronic health records.

Fig. 5.

A CB-HMI–enabled medication dispensing robotic system. (A) Operational workflow of the medication-dispensing robotic system. t denotes the time of entry and t-t0 represents a 2-h period prior to the time of the entry. “✓” and “✗” correspondingly represent the detection of relatively high and low levels of APAP (as determined by the defined signal threshold). (B) Optical image of the fully integrated medication-dispensing system consisting of the CB-HMI, a medication delivery robotic arm, medication storage compartment, and an LCD. (C and D) The raw and processed APAP TH-sensor readouts (upon an index fingertip-based entry) and corresponding system response for two cases: a subject with a recent APAP intake (C, 1 h, 650 mg APAP) and an APAP-free subject (D). N.D. indicates no detection of APAP. (E) Acquired bioinputs and the corresponding status for three index fingertip entries at t = 0, +5, and +6 h. Bottom right subfigure tabulates the decrypted retrieved bioinputs and medication intake status at the end of day (EOD). (F) Encryption procedure for the second entry, illustrating the generation of 1) a secret-embedded polynomial using the acquired bioinputs, 2) a biometric key based on the extracted minutiae features, and 3) genuine and chaff points to construct the vault.

Specifically, for bioauthentication, the system exploits the bioinputs to verify the user’s identity and no/low medication circulating level as prerequisites for supplying the requested medication. Upon verification, the system dispenses the requested pill with the aid of a robotic arm (mechanical feedback). Similarly, by prompting the user to provide a follow-up touch-based entry, the same bioinputs can be used to confirm the medication intake (by verifying the elevated circulating level of the medication in tandem with the user’s identity). For both situations, the status of the bioauthentication/intake verification as well as the acquired bioindices are displayed on a pill case–mounted LCD screen to provide real-time visual feedback to the user. Optionally, this information can be biometrically encrypted and logged on a cloud-connected server. In this way, the personal electronic health record of the user can be seamlessly updated, while preserving the security of the collected information—all at the point of touch.

Fig. 5 demonstrate the bioauthentication function of the developed system for two representative cases: one with a subject with no recent APAP intake (>24 h), the other with the same subject but with a recent APAP intake (∼1 h). In the first case, the medication request was rejected because of the detection of APAP presence at a relatively high level (I > 3.7 nA; Fig. 5). In the second case, because no/minimal circulating drug was detected (I < 3.7 nA; Fig. 5), the medication request was granted, triggering the automatic delivery of a single pill by a robotic arm (as visualized in Movie S1).

To demonstrate all the enabling functionalities of the developed system, it was applied to assist a user under a regular APAP dosing schedule (650 mg every 6 h). In this illustrative study, three entries were provided by the user at predetermined time points (t = 0, +5, and +6 h), where the user had a recent intake (at t = −1 h), which was unbeknownst to the system. Fig. 5 shows the derived status for each entry in relation to the acquired bioinputs (detailed in ): a rejected medication request due to the detection of the presence of relatively high level of APAP (in agreement with the user’s APAP intake history), a granted medication request followed by automatic pill delivery (t = +5 h), and a successful intake verification (t = +6 h). For each entry, the acquired bioinputs were biometrically encrypted in situ (using our embedded CBS) and transmitted to a Google cloud server for storage. The description of the encryption process for a representative entry (t = +5 h) is shown in Fig. 5. At the end of the day, upon the user’s request—simply through a fingertip entry—all the stored encrypted electronic medical records were successfully retrieved and decrypted to render the longitudinal profile of the acquired bioindices.

Discussion and Outlook

We devised an HMI modality—with built-in cryptographic multimodal bioperception and interpretation capabilities—that translates the user’s touch-based entries into encrypted biochemical, biophysical, and biometric indices.

As its central component, our HMI (termed CB-HMI) features a TH-coated sensing interface and a signal interpretation framework to noninvasively and inconspicuously measure and contextualize the biochemical indices. The employed TH-sensing technique bypasses the fundamental challenge of circulating analyte accessibility (posed by the skin’s barrier function), enabling the perception of body’s chemistry at molecular levels. The devised signal interpretation framework, which was formulated and supported with the aid of an introduced analyte transport model and an artificial fingertip characterization setup, allowed for extracting the epidermal analyte partitioning flux information, decoupling the confounding effect of the pressing force variability, and correlating the epidermal readings to the corresponding analyte circulating levels. Expanding the detection capabilities of TH sensing, we demonstrated a voltammetric TH-sensing method, which is particularly useful for measuring electroactive species such as pharmaceuticals and central to the translation of the CB-HMI into broader biomedical applications (as illustrated here in the context of medication adherence monitoring).

Expanding the scope of the biological perception, the CB-HMI additionally features physical sensors (PPG and fingerprint scanner) to acquire the user’s biophysical (HR and SpO2) and biometric (fingerprint minutiae pattern) indices. Upon integrating the associated data acquisition and processing modules, we validated the CB-HMI’s complete bioperception/interpretation functionality via human subject studies. Collectively, the results demonstrated the CB-HMI’s capabilities in terms of acquiring physiologically relevant readouts of target bioindices, as well as user-identifying and biometrically encrypting these indices in situ.

By upgrading the common surrounding objects with the CB-HMI, and subsequently equipping them with biological perception/interpretation, we created interactive solutions to promote the user’s quality of life. In these solutions, the CB-HMI–enabled bioauthentication allowed for bypassing the limitations of existing sensor technology analogs (e.g., breathalyzer and pill counters), which are highly vulnerable to tampering or incapable of verifying the actual intake event (inherently nonspecific). The demonstrated interactive system for vehicle activation, which is capable of detecting the circulating alcohol level at well below the legal limit for driving (40), could be particularly helpful in preventing DUI—one of our modern societal challenges claiming >10,000 lives each year in the United States alone (41). The presented crypto-smart medication dispensing robotic system could serve as an ideal drug adherence monitoring solution, owing to its built-in user bioauthentication, medication intake verification, and seamless electronic medical record-keeping capabilities (42). In that regard, the CB-HMI’s biochemical sensing capabilities can be extended to target other medications and substances and augmented with advanced algorithms to determine appropriate drug dosing and cutoff values. The acquired insight can be further enriched and contextualized by incorporating a dedicated interface for logging user-reported symptoms. Moreover, the feedback functionality could also be extended to alert the subject of dose due/overdue. In this way, our technology can be positioned to address one of our great societal healthcare challenges: nonoptimized medication therapy, which is fueled by inappropriate dosing and patients’ poor medication adherence and results in 275,000 deaths and $530 billion in healthcare costs annually (42).

Toward deploying the presented HMI modality in practical settings, the influence of various confounding factors (e.g., skin and gland metabolism and perspiration rate variation) needs to be carefully characterized (via large-scale clinical studies), and if necessary mitigated (via engineering solutions and machine learning algorithms). To this end, the incorporation of auxiliary sensors (e.g., temperature and humidity) could be helpful to standardize the measurements and to account for underlying sources of inter/intraindividual variability. Moreover, to synergistically capitalize upon the multimodality of the acquired data rendered by the CB-HMI, the development and application of data-fusion algorithms are necessary, which in turn will aid with better contextualization of the data and personalization of the feedback. It is also noteworthy that the unique built-in data encryption feature of the CB-HMI inherently provides the layer of security necessary to protect such a wealth of personal information, while enabling decentralized biodata collection and processing within Internet of Things and blockchain infrastructures. Additionally, toward providing a comprehensive bioperception of the user, the devised HMI can be adapted to acquire a wider panel of psychological and physiological indices. For example, the presented TH-based sensing methodology can be applied to target endogenous molecular indicators of health, including hormones, nutrients, metabolites, and cytokines. In that regard, dedicated clinical investigations are required to establish the physiological significance of the TH-sensor measurements for each analyte and in relation to the analyte-specific blood–sweat partitioning mechanisms (43). Furthermore, the presented PPG-based sensing modality can be extended to derive other informative biophysical indices such as respiratory rate and blood pressure.

Ultimately, the maturity of the CB-HMI and other classes of HMIs, and their integration in proliferating intelligent systems such as robotics, will equip our surroundings with a comprehensive and deep awareness of individuals’ psychophysiological state and needs. This advancement will seed the foundation for creating interactive and adaptive environments to actively assist users in reaching their optimal outcomes efficiently.

Materials and Methods

A detailed description of the materials and methods can be found in , which includes relevant information pertaining to fabrication and characterization of TH sensors, finite element simulations of the mass transport process in TH sensing, CB-HMI construction, and human subject studies. The conducted human subject experiments were performed in compliance with the protocols that have been approved by the Institutional Review Board at the University of California, Los Angeles (IRB No. 17-000170). All subjects gave written informed consent before participation in the study.