A novel hand gesture recognition method based on 2-channel sEMG.

Hand gesture recognition is getting more and more important in the area of rehabilitation and human machine interface (HMI). However, most current approaches are difficult to achieve practical application because of an excess of sensors. In this work, we proposed a method to recognize six common hand gestures and establish the optimal relationship between hand gesture and muscle by utilizing only two channels of surface electromyography (sEMG). We proposed an integrated approach to process the sEMG data including filtering, endpoint detection, feature extraction, and classifier. In this study, we used one-order digital lowpass infinite impulse response (IIR) filter with the cutoff frequency of 500 Hz to extract the envelope of the sEMG signals. The energy was utilized as a feature to detect the endpoint of motion. The short-time energy, zero-crossing rate and linear predictive coefficient (LPC) with 12 levels were chosen as the features and back propagation (BP) neural network was utilized to classify. In order to test the method, five subjects were involved in the experiment to test the hypothesis. With the proposed method, 96.41% to 99.70% recognition rate was obtained. The experimental results revealed that the proposed method is highly efficient both in sEMG data acquisition and hand motions recognition, and played a role in promoting hand rehabilitation and HMI.

Introduction

As the most common human body movements, hand gesture recognition has important significance in many aspects: (1) Rehabilitation: Many survivors of stroke or people with brain injury bear the upper limb motor function damage. In many cases, the lost motor function is not of the entire arm, but just the hand part [1]. However, the muscle of the forearm which has the responsibility of controlling the hand can still work normally. Therefore, people can use auxiliary equipment to carry out rehabilitation. The auxiliary equipment can execute the manipulator’s intention which the hand could not finish but can be judged from the state of forearm muscles [2, 3]. (2) Hand gesture recognition: People can present information through the location and shape of fingers, palm, wrist, and arm [4]. For example, ordinary people can use gestures to express greeting and thanks, and the deaf-mute can use standardized gestures to communicate what is even more important to them. At present, very few ordinary people can understand the sign language. If the sign language is accurately judged by the sEMG, it will significantly promote social communication [5]. (3) Human-machine interface: Tele-manipulation robot system is more and more used cause of its advantage which can be applied to a lot of environment which is dangerous for humans [6]. The robot system is now controlled by handle mostly which is not directly enough. On the contrary, if people control the system by recognizing the arm movements, the efficiency will be significantly improved [7]. At present, the efficiency of hand gesture recognition is one of the most challengeable problems in the human-machine interface. It includes the number of the sensor, the accuracy rate, real-time property, etc.

Recently, some studies have been developed to find the way for hand gesture recognition. Accelerometer and sEMG are widely utilized to recognize hand gestures [8, 9, 10]. A framework for hand gesture recognition based on the information fusion of a three-axis accelerometer and multichannel EMG sensors was developed by Zhang et al. [11]. In the framework, a decision tree and multistream hidden Markov models were utilized as a decision-level fusion to get the final results. And the effectiveness of the framework was demonstrated by the experimental results on the classification of 72 Chinese Sign Language words. Aim at reducing the noise influence, a method using a signal which is less sensitive noise was developed. Each base classifier was trained using the subset of channels selected randomly. Experimental results showed that the method had a satisfactory performance [12]. Both sEMG and accelerometry data were used to classify 40 hand movements in 5 amputated and 40 intact subjects. The results suggested that the inclusion of accelerometers may improve the real life prosthetics performance [13].

Some researchers utilized Kinect sensor to recognize hand gestures [14, 15, 16]. A robust part based hand gesture recognition system using Kinect sensor was built by Ren et al. They proposed a novel distance metric to measure the dissimilarity between hand shapes. The experiments demonstrate that the system is accurate and efficient [17]. A hand gesture recognition scheme was proposed targeted to leap motion data. The features were extracted from the depth computed from the Kinect and combined with the leap motion ones to improve the performance. Experimental results demonstrate that it is possible to achieve a very high accuracy in real-time [18]. A hand motion capture procedure which using the Kinect sensor to achieve more reliable tracking under unconstrained conditions for establishing the real gesture data set was proposed. The effectiveness of the framework was demonstrated by the experimental results [19].

However the approaches mentioned above have their own practical drawbacks. They used too many sensors to collect data, which can bring in complex data processing, thus these conditions limited their practical application. Differing from these approaches, only two channels of sEMG signals were utilized in this work to recognize the hand motions which tried to balance the recognition accuracy and information fusion.

In this study, we hypothesized that a hand motions recognition system with only two sensors could provide: (1) accurate sEMG signals; (2) acceptable identification of six hand motions which are hand close (HC), hand open (HO), wrist extension (WE), wrist flexion (WF), ‘OK’ sign (OS) and ‘V’ sign (VS).

To test the hypothesis, we designed a system for sEMG signal acquisition and preprocessing. Moreover, the neural network was utilized for pattern recognition. The main advantages of this system are: (1) Few channels are used. In this study, we only collect two channels sEMG signals to recognize the six hand motions which are superficial digital muscle and flex muscle. Obviously, the optimal situation of hand motions recognition is that as much information as possible can be expressed by the least muscles. This is because that the overmuch sensor will cause installation interference and is not good for system integration. (2) In view of the hardware system, we proposed a comprehensive method for sEMG signal processing including filtering, endpoint detection, feature extraction, and classifier.

The purpose of this study was to develop a novel method that could recognize the six hand motions by the sEMG of superficial digital muscle and flex muscle. Moreover, an experiment was done for verifying the effectiveness of the method. Five subjects participated in the experiment, and the results revealed that this method is highly efficient both in sEMG data acquisition and hand motions recognition.

Method

Declaration

All recruited subjects had signed the informed consents before experiment. The procedures conformed to the Declaration of Xuchang University.

Experiment protocol

A system based on the sEMG for the hand motions recognition was developed by the sensor and computer technology to test the hypothesis. This system can collect and record the sEMG data precisely. Figure 1 shows the structure of re cognition framework.

Figure 1.

Structure of recognition framework.

As shown in Fig. 1, there are mainly three parts in the recognition framework. Firstly, the subjects were asked to achieve the six hand motions one by one. Secondly, the sEMG signals would be obtained and recorded by the system, and then would be processed. Finally, the recognition results would be displayed. Four key technologies were included in this study: data acquisition, endpoint detection, feature extraction and pattern classification.

The method proposed by Su et al. utilized four sEMG sensors and one ACC sensor to collect the motion information and five features to classify [5]. In this study, only two sEMG sensors and three features were utilized to recognize. This will simplify the system and reduce the calculation. Six frequently-used hand motions were chosen to be recognized as shown in Fig. 2.

Figure 2.

Six hand motions. From left to right and up to down these are HC, HO, WE, WF, OS and VS.

Data collection

We designed a data acquisition subsystem which could provide the objective data about the muscle activity by measuring the sEMG signals of superficial digital muscle and flex muscle. To effectively reduce the interference, a high precision regulated power supply which can provide 12 V voltage was used. And a band-pass filter with a bandwidth of 11 Hz1940 Hz was applied to remove the noise in the circuit. Also, one-time button electrodes were placed on the skin after cleaning the skin with medicinal alcohol to guarantee the quality of the signal. During the experiment, the subjects were asked to make the six hand motions which are HC, HO, WE, WF, OS, and VS each five times in a turn. Four rounds were finished each subject. In case the muscle is overused, we set a two minutes interval between each two motion to rest the muscle.

Figure 3 shows the sEMG data of muscles while doing HC and HO where sEMG Channel 1 and sEMG Channel 2 mean the superficial digital muscle and flex muscle respectively. And the Fig. 3a and b mean making HC motion, the Fig. 3c and d mean making HO motion. It can be found that the sEMG data of the same muscle are different while making different hand motions. However the data need to be further processing to find the rule of muscles working.

Figure 3.

The sEMG data of superficial digital muscle and flex muscle while making HC and HO motion.

Endpoint detection

Because the sEMG is a very weak physical and non-stationary signal, various noises could not be fully removed no matter invasive or non-invasive technology is used for data acquisition. The non-invasive way will introduce more noise. The main types of the noises in sEMG signals are inherent noise in electronics equipment, ambient noise, motion artifact, inherent instability of signal, electrocardiographic artifacts and cross talk. Considering the various noises mentioned, the sEMG data must be filtered to achieve better effect of gesture recognition. In this study, we adopted IIR filter with the cutoff frequency of 500 Hz to extract the envelope and used the envelope signals with low frequency for endpoint detection. The formula of the IIR filter is as follow:

Where means the input data, means the output data.

Before feature extraction, the initial point and ending point need to be found in the long data. In this study, we calculated the whole energy of two channels sEMG and compared it with the threshold value (TV) defined beforehand. The process is as follows:

The sum of the energy of the two channels was calculated. The energy is the squared value of the amplitude of the current sample after filtering.

The average value of the two channels was compared with the TV. The result is as follows:

Also, we added 50 sample points at the beginning of the start point detection to avoid losing information. The result of endpoint detection was shown in Fig. 4. It can be seen that the performance of endpoint detection is good. The start point and end point of all six motions can be found clearly.

Figure 4.

The result of start point detection (solid line) and terminal point detection (dashed line): (a) HC; (b) HO; (c) WE; (d) WF; (e) OS; (f) VS.

Feature extraction

The purpose of feature extraction is to ensure the optimal performance of the classifier. To this end, the correlation of extracted features should be the lower, the better. In this study, the short-time energy, zero-crossing rate and linear predictive coefficient (LPC) with 12 levels were chosen as the features. Moreover, the data was processed by framing before the features were obtained.

The short-time energy was determined as:

Where means the data after framing, means the frame size.

The zero-crossing rate was determined as:

Where means the data after framing, means the frame size.

The LPC was determined by the transfer function of Auto-Regressive mode:

Here, means the gain factor.

The result of feature extraction was shown in Fig. 5. Where the Fig. 5a–f mean HC, HO, WE, WF, OS and VS, respectively. We chose 5 actions of each gesture to extract features.

Figure 5.

The result of feature extraction: (a) HC; (b) HO; (c) WE; (d) WF; (e) OS; (f) VS.

Classifier

In this study, we used BP neural network which is a multilayer feed forward neural network to classify. The main characteristic of BP neural network is that the signal transmits forward and the error transmits backward. The topology structure of the BP neural network is shown in Fig. 6. Here, the means the input value, means the predicted value, and mean the neural network weight. Here, is 26 which depends on the number of the features, and is 6.

Figure 6.

Topology structure of the BP neural network.

The number of total sets is 200 in this work which 60% for training and 40% for testing. The BP neural network training process consists of seven steps are described as follows:

Network initialization. Based on the input sequence and output sequence, we defined the number of nodes belonging to the input layer () is 26, the number of neurons belonging to the hidden layer () is 17, the number of nodes belonging to the output layer () is 6. The thresholds of the hidden layer () and output layer () were initialized.

The output of hidden layer calculation. The output of hidden layer () was acquired according to the following function:

Here, the activate function is .

The output of output layer calculation. The predicted output () was acquired according to the following function:

Error calculation. The error () was acquired according to the following function:

Here, means the desired output.

Weight value update. The weight value was acquired according to the following function:

Here, means the learning rate.

Threshold update. The threshold was acquired according to the following function:

Judging whether the iteration ends, if not, return to step 2.

Results

Five right-handed subjects were recruited in this study, four male (S2S5) and one female (S1), aged between 27 and 34 years old. All the subjects were all healthy with no history of joint and neuromuscular diseases and performed the six hand motions correctly. The recognition results were shown in Table 1.

Table 1

Recognition results of six hand motions

	S1	S2	S3	S4	S5	Mean ± standard deviation
HC	99.20%	99.31%	99.26%	99.34%	99.24%	99.27% ± 0.42%
HO	99.27%	99.36%	99.38%	99.37%	99.37%	99.35% ± 0.34%
WE	99.12%	99.70%	99.65%	99.44%	99.60%	99.50% ± 0.25%
WF	96.41%	97.80%	98.12%	98.11%	97.81%	97.69% ± 1.42%
OS	97.84%	98.41%	98.75%	98.63%	99.17%	98.56% ± 0.97%
VS	98.78%	98.81%	98.87%	98.92%	98.92%	98.86% ± 0.41%

As shown in Table 1, the recognition accuracy of S1 is the lowest for the individual. In all the tested six cases, the recognition accuracy of WE is the highest (99.50%), while the recognition accuracy of WF is the lowest (97.69%).

Discussion

In this study, we hypothesized that a hand motions recognition system with only two sensors could provide acceptable identification of six hand motions which are HC, HO, WE, WF, OS and VS. To meet the needs of the hand gesture recognition based on sEMG signal, we have developed a system which can gather and record precise sEMG data. Moreover, an innovative method for feature extracting including filtering, endpoint detection, feature extraction and BP neural network was proposed. Through the tests, we have shown that the system proposed in this study could collect the sEMG signal precisely.

In this study, we proposed a method for endpoint detection based on energy. As shown in Fig. 4, the start point and end point of the six hand motions could be found accurately in the long size data. It can remove the useless data and only leave the useful data which is very important for further data analysis.

Short-time energy, zero-crossing rate and linear predictive coefficient (LPC) with 12 levels were chosen as the features in this study. As shown in Fig. 5, the performances of short-time energy and zero-crossing rate are good. On the contrary, the result of LPC is a little confusion, especially the WF and OS.

The highest recognition accuracy is 99.70% while subject 2 making WE hand motion. The lowest recognition accuracy is 96.41% while subject 1 making WF hand motion. In addition, the recognition accuracy of the female is a little lower than the males. We suppose that it has to do with personal characteristics such as subcutaneous fat and muscle.

Compared with the vision-based technology such as Kinect sensor [14, 15, 16], sEMG technology has its own advantages which is less sensitive to environmental factors. In this study, we focused on exploring more concise relationship between hand gesture and muscle. Only two sEMG sensors and three features were utilized to recognize hand gesture compare to the method proposed by Su et al. utilized four sEMG sensors and one ACC sensor [5]. The experimental results revealed that the proposed method balance the recognition accuracy and systematic complexity well. This study will play a role in promoting the application of sEMG in rehabilitation because of the simplicity of the system and calculation.

However, there are some potential limitations in our present study. First, there is no patient participated in this study. One of our objectives is rehabilitation. But in this study, only five healthy people were employed. The effectiveness of this method is still unknown before experiment of people with motor function damaged is done. Second, the method proposed in this study is not appropriate for actual application at present. The work of data processing was finished off-line. And the experiment for estimating real-time performance is not done. Also, the changes of sensor location may influence the performance of the system after a long period using.

We intend to do patient experiments to determine the effectiveness of this study for people with hand motor function damaged. Moreover, the actual application of this system still requires more work in the future. The real-time performance and stability of this system need to be improved. However, the contents of this paper should be useful for further research of hand motions recognition based on sEMG.

Conclusions

In this study, we have demonstrated that this system is useful for sEMG signals collection. In addition, a comprehensive method was developed for sEMG signal processing including filtering, endpoint detection, feature extraction, and classifier. The recognition results of six hand motions reveal that this study would be useful for practical applications. In the future, we will focus on the real-time performance and stability of the system.