Detection of Hand-to-Mouth Gestures Using a RF Operated Proximity Sensor for Monitoring Cigarette Smoking.

Common methods for monitoring of cigarette smoking, such as portable puff-topography instruments or self-report questionnaires, tend to be biased due to conscious or unconscious underreporting. Additionally, these methods may change the natural smoking behavior of individuals. Our long term objective is the development of a wearable non-invasive monitoring system (Personal Automatic Cigarette Tracker - PACT) to reliably monitor cigarette smoking behavior under free living conditions. PACT monitors smoking by observing characteristic breathing patterns of smoke inhalations that follow a cigarette-to-mouth hand gesture. As envisioned, PACT does not rely on self-report or require any conscious effort from the user. A major element of the PACT is a proximity sensor that detects typical cigarette-to-mouth gesture during cigarette smoking. This study describes the design and validation of a prototype RF proximity sensor that captures hand-to-mouth gestures with a high sensitivity (0.90), and a methodology that can reject up to 68% of artifacts gestures originating from activities other than cigarette smoking.

INTRODUCTION

At the present day, there are more than a billion regular smokers in the world and the tobacco use is on the rise, especially in developing countries [1]. Smoking is known to increase risk for various cancers, such as mouth, larynx and lung, as well as for heart attacks, stroke and several pulmonary diseases. This epidemic is the cause of 6 million preventable deaths yearly, with 10% of fatalities being non-smokers exposed to second-hand smoke [2]. The most prevalent mechanism for tobacco use is by means of cigarettes, for example, 23.3% of the U.S. population were regular cigarettes smokers in 2009, more than any other form of tobacco use [3]. Characterization of smoking habits such as number of cigarettes smoked per day, number of puffs per cigarette, duration and volume of a puff, etc., is important for evaluation of total smoke exposure, which has been shown to have the most significant impact of health consequences of smoking. Accurate estimates of smoking behavior and smoke exposure are also critical for improvement of clinical and pharmacological interventions and smoking cessation programs.

Self-reporting and portable puff topography devices are among the most popular current methods for monitoring cigarette smoking. However, the reliability of self-reporting methods is constrained to memory limitations and intentional miss-reporting of events [4]. On the other side, portable pufftopography instruments result in a limited assessment of frequency of smoking, since cigarettes have to be consumed through the instrument which might change the regular smoking pattern. Based on these two major limitations, our overall objective is the development of a wearable and non-invasive, practical, low cost sensor system (Personal Automatic Cigarette Tracker - PACT) that frees the user from the burden of conscious effort during monitoring of smoking habits. The PACT relies on monitoring of breathing and cigarette-to-mouth gestures to recognize smoke inhalations. A major component of the PACT is a proximity sensor for detection of cigarette-to-mouth gestures (or more generally, Hand-to-Mouth Gestures, HMGs). If an average smoker consumes 15 cigarettes per day with 8-16 puffs for each cigarette [5], the resulting number of smoking-related HMGs would be roughly 42,000 to 87,600 per year. This gesture is hard to disassociate from the smoking habit, and thus the reason why some methods for cessation of smoking use inhalers to mimic it [6].

A reliable HMG sensor should detect the user’s wrist proximity to the mouth and objectively capture the timing, duration and frequency of these events. Then, analysis of the respiratory patterns following a HMG can be used to identify the nature of the detected gesture. Inhalation of cigarette smoke has a unique breathing pattern that can potentially be automatically recognized by methods of machine learning.

A number of different methods and approaches have been reported for the detection of hand gestures. Accelerometers have been used to measure the velocity of hand movements [7], and to recognize different arm gestures [8]. Infrared detectors have been used to identify directional movements of the arms [9], and capacitive sensors [10] and video [11, 12] have been used to locate the arm’s exact position. Additionally, the use of gyroscopes has also been studied to identify the characteristic angular velocities associated with a user taking a bite of a meal [13]. These sensors can be very versatile and accurate, but they are limited to specific applications and are not practical for all situations. Accelerometers and gyroscopes are useful for detection of transitions of the arm position; however, they cannot provide information about absolute position of the arm and its proximity to the mouth. Machine video methods are very restrictive as they rely on observation of subjects with a confined space and do not allow for observation in the community. Thus, these sensors are not well suited for monitoring of cigarette smoking.

This paper describes a radio frequency (RF) based sensor for monitoring of HMGs in cigarette smokers. The proposed sensor design uses a transmitter/receiver pair to measure the distance of the transmitter positioned on a hand to the receiving antenna positioned on the user’s chest. This sensor design does not impair body movements in any way, is inconspicuous, miniature and operates in the RFID (Radio Frequency Identification) band. The design and human study presented in this paper are a substantial expansion of our early work in this direction [14]. Specifically, we describe the principle of operation and characteristics of an RF-based HMG sensor and describe the results of a human study testing the sensor for sensitivity to cigarette-to-mouth gestures and sensitivity to hand-to-mouth gestures originating from activities other than smoking. Based on the results of sensor testing, we suggest a methodology for a substantial reduction (68%) of artifact HMG not describing cigarette-to-mouth transitions.

MATERIALS AND METHODS

Fig. () shows the concept and a practical implementation of the HMG proximity sensor. The sensor system comprises two RF components: a transmitter and a receiver. The transmitter resides in a small bracelet on the dominant hand. The RF signal of the transmitter is emitted by a loop antenna which is positioned parallel to the inner side of user’s arm. The receiving antenna is positioned on the user’s chest. Due to the ∞–shaped directional propagation pattern of rectangular loop antennas in the receiver and the transmitter, the signal strength at the receiver will depend both on distance and relative orientation of the antennas. The maximum signal strength at a given distance is achieved when both antennas are co-axial and parallel with respect to each other. Such a property is highly desirable for detection of smoking HMGs, as antenna alignment naturally occurs during the most typical way of holding a cigarette at the mouth (Fig. ). HMGs originating from activities other than smoking (e.g. food intake, Fig. ) typically result in arm positions that orient receiving and transmitting antennas at an angle to each other and in turn result in a lower amplitude of the received signal at the same distance. While the proximity sensor amplitude is most likely not sufficient to differentiate HMGs of various origins, it provides a significant feature that can be used for detection of smoking.

RF Transmitter and Receiver

The RF transmitter, shown in Fig. (), is a sinusoidal oscillator that operates at a frequency of 125 kHz, typically used in RFID devices. The low transmission power from the sensor allows compliance with FCC and global standards for interference and range [15]. The transmitter uses a rectangular loop antenna (40x15x5 mm, 860±%10μH, 13ohm, Sonmicro Electronics LTD). The transmitter circuit and the antenna are incorporated into a small plastic enclosure with an adjustable wrist band. The power consumption of the transmitter circuit was measured to be 4mA @ 3.3V, which allows for 37+ hours of uninterrupted transmission with a 150 mAh Li-ion battery. The miniature dimensions (56x31x15 mm) and light weight (30 grams) of the transmitter, similar to a common wrist watch, allows the user to carry on normal daily activities with no interference from the device. The receiver is the second part of the proximity sensor (Fig. ). A custom-made, rectangular loop antenna is attached to the user’s chest by Velcro strips. The design of the receiver antenna was obtained by using standard calculation techniques [16]. The receiver antenna was intentionally designed with the aperture much larger than the transmitter antenna (110x110x5mm, 1080μH and 8.3ohm) to improve the sensitivity to HMGs that may position the transmitter off to the side of the alignment axis. This larger size allows a reduction of the differences in the sensor reading due to the variability of hand positions during HMGs. The receiving antenna is connected to a resonant circuit and rectifier which generates a signal proportional to the strength of the RF signal. This rectified signal, named Proximity Signal (PS(t)), is low-pass filtered, amplified and recorded by a data acquisition system.

For a typical cigarette-to-mouth gesture (Fig. ), the transmitter is located approximately at a distance of 10-15cm from the receiver. The sensitivity of the receiver is set to produce the maximum response of 3.0V at this distance with parallel and co-axial positioning of the antennas. Signal strength as a function of the distance in the range of 1-50cm is shown in Fig. ().

Testing for False Postive Detection

Since the operational frequency of 125 kHz is used by RFID devices, the described proximity sensor may be sensitive to interference from RFID equipment and other electromagnetic sources. An experiment was performed to analyze potential false positives arising from RF interference, that is, detecting interference from sources other than proximity sensor transmitter as HMGs. A volunteer was asked to wear the RF receiver on three different days without the matching RF transmitter; data for 20+ hours was recorded during unconstrained activities of daily living, including walking in an urban area, driving a car, washing dishes using a cell phone, and using household appliances that are strong emitters of electromagnetic noise (e.g. a microwave oven and a vacuum cleaner). A false positive HMG was counted if PS(t) exceeded an experimentally determined threshold (Th=100mV) above the electronic noise of the system.

Testing on Common Activities

Data collection was performed on twenty volunteer participants. These volunteers, ten males and ten females, ages 23.1±3.4 were considered regular smokers as determined by a carbon monoxide breath sample measure greater than 10ppm at the time of the experiment. The IRB at The University of Alabama approved the study and written informed consent was received from all participants after the experimental protocol was explained to them in detail. The participants were asked to perform twelve common activities, while wearing the proximity sensor. These activities where, in order: 1) sitting comfortably, 2) reading aloud, 3) standing still, 4) walking at a self-selected slow pace 5) walking at a self-selected fast pace, 6) using a laptop, 7) eating a meal using only their hands, 8) eating a meal using silverware for solid food and a straw for liquids, 9) walking outdoors, 10) smoking a cigarette in a sitting position, 11) resting in a self-selected posture, and 12) smoking a cigarette in standing position. These 12 activities were chosen to involve different behaviors featuring conscious (e.g. smoking, eating, computer work) and spontaneous (e.g. supporting the head while sitting) hand gestures, to analyze their impact on the proximity sensor developed. The real-world set of activities is obviously much greater and will be evaluated in future free-living studies. To record the proximity sensor signal, a miniature data logger (Logomatic V2.0, Sparkfun Electronics) digitized the PS(t) at 12bits and 100 Hz and saved the data to a microSD card for off-line processing. Fig. () shows a recording of the PS(t) obtained in one participant’s experiment during the 12 activities described above. The participants were videotaped by a DV camcorder during the each of the experiments. The video recordings served as a reference for annotation of the sensor signals. Annotation was performed by human raters that marked events of interest (e.g. cigarette-to-mouth gestures and smoke inhalations) by using custom software that enables time-synchronous review and mark-up of video and sensor signals. All annotated HMGs were analyzed and statistics were computed on the gestures characteristics associated with smoking and other activities.

Detection of Hand-to-Mouth Gestures

A HMG was detected when the amplitude of the PS(t) exceeded the 100mV threshold. Amplitude and time duration were measured for each of the detected gestures. Since this proximity sensor is dedicated to monitoring of smoking and smoking-related HMGs it is necessary to discriminate other ‘artifact’ gestures that are not related to this activity (for example, those originating from food intake). Two main characteristics, amplitude and duration of a HMG, were analyzed in order to reject obvious artifacts. Additionally, gestures with a very short gap between them were merged together; it has been observed that quick and prominent movements of the wrist during a HMG may result in detecting two separate gestures due to misalignment of the transmitter-receiver antennas within the same HMG gesture.

Artifact rejection and gesture merging were performed using the following thresholds: an amplitude threshold (Th) applied to the PS(t) to reject low amplitude gestures; short duration (Sd) or long duration (Ld) thresholds to reject artifacts with duration atypical to smoking-related gestures; and time separation threshold (Mt) to merge gestures if their separation was significantly short. The values of the parameters described, Th, Sd, Ld, Mt, were obtained based on the statistics computed across all participants and all activities. The results obtained represent the parameters that rejected as many HMG artifacts as possible, while ensuring that no gestures detected associated with smoking were discriminated. A detailed definition of these parameters is described next.

Amplitude Rejection

Rejection of non-smoking HMGs based on the amplitude of the PS(t) is based on a simple threshold Th:

For a given threshold Th, the PS(t) signal yields in a HMG set {H, where i = 1,…,n is the number of gestures and j = 1,…,m is the number of participants.

Duration Rejection

The duration for each HMG is defined as:

where i=1,…,n is the number of gestures, t is the starting time of a HMG and t is the ending time of a HMG, defined by the intersection of the PS(t) with Th.

Very short and very long duration HMGs are not uncommon, for example, when a participant moves the hand to scratch an area around the upper body (short gesture) or resting the head on the hand while reading (long duration).A rule was defined to discriminate these Sd and Ld artifacts:

Merging Gestures

As a last step after amplitude and duration rejection, adjacent HMGs that presented a significant short time gap between them were merged together into a single larger duration gesture. A merging threshold Mt was used in the following rule:

where t is the start time of the subsequent i-th HMG and t is the end time of the subsequent i-th HMG.

Smoking Gestures Detection Sensitivity

The ability of the proximity sensor to detect HMGs during smoking was evaluated by computing the number of True Positives (TP) in which each HMG was detected both by a human rater and the proximity sensor, and False Negatives (FN) in which a gesture was detected by a human rater but not by the sensor. The precision of the gesture sensor was then calculated as [17]:

The sensitivity metric is used here as a mean to evaluate how efficient is the sensor to capture HMG associated with smoking. In a similar fashion, artifact gestures were counted as False Positives (FP), in which a gesture is associated with a non-smoking activity. Recall was be calculated as:

The recall metric is used to analyze the significance of the artifact rejection described above.

RESULTS

An initial analysis of the HMG amplitude and duration detected by the described sensor was performed without considering artifact rejection. Duration of gestures, based on the PS(t), was found to be in average 3.78s (SD 5.42) for smoking, and 6.82s (SD 21.08) for all other activities. The average amplitude was found to be 81.0% (SD 21.5) of the maximum value for smoking gestures and 49.3% (SD 42.0) for gestures across all other activities.

Values obtained for the parameters described in Section III for artifact rejection are shown in Table . A comparison of the total number of HMGs before and after artifact rejection is presented in Table . Additionally, Fig. () shows the boxplots of the number of gestures detected across all participants for each activity before and after artifact rejection.

In Table , the p-values calculated based on the t-test statistical analysis with a 95% confidence interval are presented. These results show how significant the rejection of HMGs was for each activity across all participants. Fig. () displays a detailed description of the count of HMGs in the smoking activities before and after artifact rejection. Finally, based on the results obtained in Table , the sensitivity of the HMG sensor to smoking gestures was computed using equation (5), with a result of 0.90. Recall before and after artifact rejection was calculated with equation (6) based on the results in Table , resulting in values of 0.09 and 0.30 respectively.

DISCUSSION

The proposed RF proximity sensor proved to be an effective tool to identify HMGs originated from cigarette smoking and other common daily activities. Enabling the capture of cigarette-to-mouth gestures, this sensor constitutes a major feature of the PACT system. As Fig. () shows, the sensor is maximally sensitive within 30 cm of range, sufficient for reliable detection of gestures from cigarette smoking. For distances above 35cm, the sensor is virtually not sensitive to non-relevant arm movements. During false positive testing in noisy urban and household environments, the sensor did not register any HMGs as a result of interference from external RF sources. The methodology described in Section II suggests that this sensor is not affected by inter-subject behavioral variability.

The analysis of the statistics of duration and amplitude of all HMGs shows that, even though the average time duration of a HMG is similar between smoking and other activities, 3.78s (SD 5.42) and 6.82s (SD 21.08) respectively, the amplitude of cigarette-to-mouth gestures is on average more than 30% higher. This difference in signal amplitude could be explained by the propagation pattern of the loop antennas that have to be parallel and co-axial to produce a signal of highest amplitude for a given distance. It is important to note that higher amplitude by itself is not sufficiently descriptive to objectively separate smoking from artifact gestures, but it can be used in combination with the additional sensors of the PACT system, e.g. respiratory sensors, to correctly identify smoking by applying pattern recognition techniques to the respiration signals.

The testing of the sensor in different conditions (Fig. ) demonstrated its sensitivity to smoking gestures (total of 480, Table ) and to gestures originating from activities other than smoking (total of 4,592, Table ). Data from the human study also illustrated unequal distribution of HMGs among activities, as seen in the boxplots in Fig (). Across all activities not related to cigarette smoking, the lowest number of artifact HMGs was observed during standing and walking activities; this could be explained due to the normal arm lateral posture while standing still in one place, and the natural swing motion of arms during bipedal locomotion, respectively. The largest number of non-smoking related HMGs was observed during food intake and the second largest during unconstrained resting, where artifacts like nose, ear or head scratching, drinking, etc., are present in a sporadic fashion. Many of the gestures present in the food intake are very similar, both in amplitude and duration, to the gestures of cigarette smoking, suggesting the potential use of this proximity sensor for Monitoring of Ingestive Behavior applications [18, 19].

The artifact rejection rules based on amplitude and duration, and merging of gestures separated by a short gap, described in Sections II.5, II.6 and II.7 respectively, proved to be efficient in the rejection of non-smoking related HMGs. Using the parameter values shown in Table , this process was capable of reducing a significant number of artifacts from 5,133 to 1,637 (Table ), or 68% across all participants. Based on the results obtained from the statistical analysis (Fig. ), it can be concluded that the activities with a more significant rejection of artifacts (p<0.050) were: sitting, walking slow, using a laptop, both eating activities, resting and both smoking activities. There were no significant rejections (p>0.050) observed in reading, standing, walking fast and outside. These results are understandable since passive activities would be expected to have less upper body motion than more active ones, i.e. standing still vs. eating or smoking. By analyzing smoking independently from all other activities (activities 10 and 12), it can be observed that the number of artifacts was significantly reduced from 369 to 63, across all participants, as described in detail in Table and Fig. (). This result represents about 83% in the reduction of artifacts in the cigarette smoking activities. The number of false negatives observed in the detection of cigarette smoking gestures was 51 (out of 531 total cigarette smoking gestures) across all participants, which resulted in a sensitivity of 0.90. Such false negatives are explained by the behavior of some participants who unconsciously held the cigarette in their non-dominant hand, where no proximity sensor was worn. To improve the sensitivity, the use of two proximity sensors, one on each wrist, could be proposed. Overall, the proposed proximity sensor satisfies the needed monitoring requirements of the PACT system by detecting the absolute proximity of the user’s hand to the mouth, providing an estimated of the distance and allowing measurement of the hand gesture duration. On the other hand, the significant increase on the recall, from 0.09 to 0.30 demonstrates that artifact rejection eliminates a large number of non-smoking hand gestures, while preserving all of the smoking hand gestures (no change in sensitivity). However, the recall value of 0.3 indicates that some of the non-smoking hand gestures still remain in the dataset after the artifact rejection. The remaining hand gestures will be analyzed for coordination with breathing patterns as a way to recognize smoke inhalations which unlike any other activity are highly correlated with the breathing.

The hand RF transmitter has been designed to be minimally obtrusive by being comparable to wearing a common wrist watch. The miniaturization of the RF receiver electronic circuit and the antenna can enable the incorporation of the wearable sensor into an undergarment with the additional sensors of the PACT.

Further development of the sensor should also improve in some of the existing limitations. The sensitivity to smoking gestures can be improved by using an additional sensor on the non-dominant hand of the user, and using active RFID technology to differentiate gestures originating from different hands. Better artifact rejection may be developed using pattern recognition techniques that take into account the shape of the gesture and coordination of breathing with HMGs. Additionally, the 12 activities involved in this study may not be sufficient; it is important to analyze the performance of the sensor in more natural behavioral conditions, if the sensor would work properly for example, when the user is walking, smoking and using a cell phone at the same time. These improvements will be the objective of further research.

CONCLUSION

The RF proximity sensor described in this paper is a key component for developing the PACT, a wearable system aimed to accurately and objectively monitor cigarette smoking. In a human study with twenty participants performing a variety of different activities, this sensor demonstrated a high sensitivity to smoking hand-to-mouth gestures (0.90). As expected, the sensor was also capable of detecting artifact hand gestures originating from other activities; however, it was possible to reject 68% of the artifact gestures based on the amplitude, duration and time separation between them, while ensuring that all gestures associated to cigarette smoking were still detected. The future development of this sensor will include the incorporation of a digital RFID signature, and the implementation of pattern recognition methods to identify inhalation of cigarette smoke potentially following a hand-to-mouth gesture.

Table 1.

Rejection and Merging Parameters Obtained through an Exhaustive Search Across All Partici-pants and Activities. These Parameter Values En-sure that all HMG Associated to the Smoking Activi-ties are Considered for Analysis

Parameter	Value
Th	4.6%
Sd	0.68 Sec
Ld	22.87 Sec
Mt	0.96 Sec

Table 2.

Total Number of HMGs Found Across All Activities and Smoking of All Participants

Description	Initial	After Artifact Rejection
Total HMGs	5,113	1,637
Non-smoking activities HMGs	4,592	1,116
Smoking activities HMGs	890	584
Rater-detected smoking HMGs	531	531
Sensor-detected smoking HMGs (TP)	480	480
Cigarette lights detected	41	41
Smoking HMGs not detected by sensor (FN)	51	51
Artifacts during smoking	369	63

In this table, ‘Smoking activities HMGs’ denotes the total number of HMG inside both smoking activities across all participants; ‘Rater-detected smoking HMGs’ are the gestures reported by the human rater; TP are the smoking gestures captured by the sensor and FN are the smoking gestures not detected; ‘Artifacts during smoking’ are other gestures not associated with smoking during the smoking activities.

Table 3.

p-value Results Testing the Significance of Artifact Rejection in All Activities

Activity	p-value
Sitting	0.046
Reading	0.061
Standing	0.167
Walking Slow	0.012
Waking Fast	0.162
Laptop	0.014
Eating w/hands	0.001
Eating w/silverware	0.000
Walking Outside	0.098
Smoking/sitting	0.000
Resting	0.000
Smoke/standing	0.004
t-test statistical test with 95% confidence intervals