Continuous Recognition of Multifunctional Finger and Wrist Movements in Amputee Subjects Based on sEMG and Accelerometry.

BACKGROUND: While the classification of multifunctional finger and wrist movement based on surface electromyography (sEMG) signals in intact subjects can reach remarkable recognition rates, the performance obtained from amputated subjects remained low.
METHODS: In this paper, we proposed and evaluated the myoelectric control scheme of upper-limb prostheses by the continuous recognition of 17 multifunctional finger and wrist movements on 5 amputated subjects. Experimental validation was applied to select optimal features and classifiers for identifying sEMG and accelerometry (ACC) modalities under the windows-based analysis scheme. The majority vote is adopted to eliminate transient jumps and produces smooth output for window-based analysis scheme. Furthermore, principle component analysis was employed to reduce the dimension of features and to eliminate redundancy for ACC signal. Then a novel metric, namely movement error rate, was also employed to evaluate the performance of the continuous recognition framework proposed herein.
RESULTS: The average accuracy rates of classification were up to 88.7 ± 2.6% over 5 amputated subjects, which was an outstanding result in comparison with the previous literature.
CONCLUSION: The proposed technique was proven to be a potential candidate for intelligent prosthetic systems, which would increase quality of life for amputated subjects.

INTRODUCTION

Limb amputation is a major cause of disability in the world [1]. The key issues of accomplishing practical upper-limb prosthesis are functionality, controllability and cosmetics [2]. In recent decades, these prostheses controlled by the surface electromyography (sEMG) containing rich information of neuromuscular activity have been applied to replace those controlled by the original motor commands in a noninvasive way [2, 3]. Gradually, these approaches have developed from control of simple functional prosthesis, such as wrist flexion and extension, to multifunctional prostheses [3]. In spite of these technological advancements, the existing prostheses cannot fully meet the actual needs of the amputees in terms of dexterity. Hence, how to control a multifunctional and dexterous prosthesis becomes a tremendous challenge.

Peerdeman et al. [4] indicated that the various approaches proposed by the majority of studies were obtained based on intact subjects. And 4 to 16 electrodes, as the case maybe, would be placed on the mid-portion of the forearm. Among the existing state-of-the-art classifiers, linear discriminant analysis (LDA), artificial neural networks (ANN) and support vector machines (SVM) algorithms play significant roles in predicting the user’s intended movements with high accuracy [5]. In our previous paper [6], we obtained an excellent classification rate at 98.6% on five hand movements using intact subjects. However, it is still an unproved issue that the results acquired on intact subjects can apply equally to trans-radial amputated subjects (TRAS) [7]. Due to the practical and ethical issues lay in this field, the reports on the real-time prosthesis control especially for amputated subjects are rather limited. Moreover, some items should be noted as follows. Firstly, it is a suffering task to think and mimic the finger and wrist movement with amputated upper-limb throughout a few hours. Secondly, different amputation levels impact the performance of classification. Thirdly, amputees differ in learning ability. Thus, different scientific training treatments should be employed on different TRAS. In a general view of literatures using TRAS, Momen [8] showed an approximately 87.5% accuracy rate for 4 arm classes on only one amputee. A recent study included six TRAS and used 64-channel amplifier (12 movements, 87.8% accuracy [9]). Schultz et al. [10] demonstrated an 87.8% accuracy rate on 5 TRAS over 10 movements. It should be emphasized that the results obtained from amputees still remain to be improved.

In recent years, a few researchers paid close attention to accelerometry (ACC), which is relatively low-cost and easily integrated in a prosthetic socket. Zhou et al. [11] indicates that the EMG signals and tri-axial accelerometer mechanomyography signals can reduce the effect of limb position variation on classification performance. Fougner et al. [12] demonstrated that ACC provided useful supplementary for prosthesis controllers.

In this paper, an efficient continuous recognition scheme of multifunctional finger and wrist movements is proposed. Primarily, a root-mean-square (RMS) filter had been employed on the sEMG signals to smooth the data. Considering the practical applications in real-time control, a sliding window was employed. It should be emphasized that majority vote is adopted to eliminate transient jumps and produces smooth output for window-based analysis scheme. Subsequently, we followed the recommendation to use sEMG in conjunction with ACC as control modality [6]. However, the placement and regular orientation of the electrodes would cause redundant information. Principal component analysis (PCA) was hence utilized to reduce the dimension of features and to eliminate the correlation of ACC modality, which improved the classification rate. Six time domains features and one time-frequency domain feature are calculated respectively as the feature vectors of each sEMG segment. The features of sEMG and ACC are then fed into four types of classifiers. Finally, according to the experimental results, the significant features of sEMG and ACC were selected and fused into an integrated feature and then passed into the best-performing classifier, that is, SVM for ultimate classifications.

The rest of the paper is organized as follows: section 2 presents this details of the scheme for data analysis and feature selection is contained in this part as well; the results are demonstrated and assessed with a novel evaluation criterion in section 3 and discussed in section 4.

MATERIALS AND METHODS

Data Acquisition

The database utilized in this paper is the second version of publicly available Non Invasive Adaptive Prosthetics (NinaPro) database [13], which is determined to promote the state of sEMG controlled hand prosthetics for TRAS. The recruited subjects were explicitly instructed to think or mimic (if available) movies shown on the screen which was used as visual stimulus. After the stage of training, subjects were asked to perform 6 consecutive repetitions of 50 different movements. Each movement repetition lasted 5 s. The interval of each repetitive movement was a rest posture lasting 3 s. During the process of experiments, the EMG data was continuously acquired using 12 DelsysTM Trigno wireless electrodes at a rate of 2 kHz. Meanwhile, arm dynamics were recorded by a 3-axis accelerometer which was integrated into each of these electrodes. Eight Electrodes (from 1 to 8) are equally spaced around the forearm; electrodes 8 and 9 are positioned on the main activity spots of the flexor and the extensor digitorum superficialis, electrodes 11 and 12 were also placed on the main activity spots of the biceps brachii and of the triceps brachii. The details on the acquisition protocol and specific movements were described in [14]. Before making data publicly available, the sEMG had been cleaned of 50 Hz power frequency interference. Subsequently, the sEMG and ACC data streams were super-sampled to the relatively higher sampling frequency (2 kHz).

In this paper, we only employed the Exercise B of database 3 including data acquired from the 17 hand/wrist movements on 5 male TRAS. The details of the 17 movements are described in Table . The clinical characteristics including of the amputated subjects are described in Table [14], where the DASH (Disability of the Arm, Shoulder and Hand) score is an evaluative outcome measure for the upper extremity musculoskeletal conditions of patients.

Preprocessing

Filtering

Initially, in this work, the data was decimated at a sub-sample rate of 5. Then a high-pass Butterworth filter with corner frequency of 20 Hz was utilized to filter the sEMG data. In order to smooth the data, a root-mean-square (RMS) filter had been employed on the sEMG signals, which was equivalent to rectifying the sEMG amplitude by averaging over a moving window of 80 long samples, namely 200ms, considering the sub-sample frequency and relinearization by taking the square root [15, 16]. Simultaneously, the ACC data was low-pass filtered at a cutoff frequency of 5 Hz by a zero-phase second order Butterworth filter in order to remove high frequency noise components.

Relabeling

It is unrealistic to make the subjects perfectly mimic the kinematics of the video stimulus due to human reaction times. The author of the database had applied an offline generalized likelihood ratio algorithm to correct the resulting erroneous movement labels, which realigned the movement boundaries by maximizing the likelihood of a rest-movement-rest sequence [3].

Analyzing

Window-based Analysis Scheme

The signals of all channels were split into sliding windows in accordance with the real-time control scheme for multifunction myoelectricity proposed by Englehart et al. [17] as displayed in Fig. (). For each sliding window, a selected feature vector was calculated and fed to a pattern classifier which could achieve outstanding classification performance. L. Smith et al. [18] indicated that longer window resulted in higher accuracy at the cost of longer processing delays (the value of processing delays in this paper was computed on a 2.5 GHz AMD SempronTM X2 processor); meanwhile, the optimal window length was recommended to be in the range of 150-250 ms. Considering the configuration of processor and real-time prosthesis control in future, the overlap increment should be longer than 20ms. Consequently, a 250ms (i.e.100 samples) sliding window with a 25 ms (10 samples) incremental window was considered. Window-based analysis scheme are described in Fig. (). (For clarity reasons, the figure displays only three out of twelve electrodes. The data is decimated at a sub-sample rate of 10).

Then, the following experiment applied the majority vote (MV) to post-process the successive dense stream of test sequence generated by overlapped segmentation. The majority vote can be described as follow: (1) make a decision for (100 samples) sliding window every 25ms; (2) make N continuous decisions; (3) vote for the N decisions; (4) The decision of the maximum votes represents the final decision. The MV is an effective post-processing mechanism that eliminates transient jumps and produces smooth output for window-based analysis scheme [19]. However, the acceptable control delay in output should be less than 300ms. The parameter k (number of the MV) is constrained within the following inequality:

Where, increment is the overlap increment between two adjacent windows (25ms). Here, MV with k=3 was adopted to smooth the results (i.e. making a decision every 75ms).

Feature Extraction

Feature selection is an essential stage in sEMG classification. And the capability of presenting the features of different movements and the computational load in real-time control shall be focused. A significant amount of features widely used in myoelectric control design can roughly be divided into three categories, namely time domain, frequency domain, and time-frequency domain [19]. However, it had been proved that frequency domain features usually perform poorly when compared with the time domain features [20].

The most successfully used features in time domain is the Hudgins feature set [20, 21]. Following the recommendations in [20], we had selected six sEMG features including Mean Absolute Value (MAV), Modified Mean Absolute Value 1 (MAV1), Variance (VAR), Root Mean Square (RMS), Waveform Length (WL), and the first four autoregressive (AR) coefficients. Waveform Length is simply the cumulative length of the waveform over the time segment defined as:

where Δx = x - x (difference in consecutive sample voltage values). MAV1 is a robust modification of Mean Absolute Value with weighting window function (w). It is calculated by:

In addition, according to the classification performance of each single feature, a combination of different single features was also applied to explore more robust features.

Wavelet packet transform (WPT), an extension of discrete wavelet transform, is a more advanced representation in time-frequency domain. The transform is able to decompose a signal using the wavelet function at different levels, resulting in high-dimensional coefficients. In this work, we employed 3-level wavelet decomposition and Daubechies 3 (db3) as mother wavelet, then calculated the variances of each wavelet packet decomposition coefficient of the last level as part of sEMG features [6].

For ACC modality, we applied the mean value (MEAN) that performed excellent in previous work [11]. PCA was applied to reduce the dimension of feature for ACC data and over preliminary trails to ensure the optimal threshold to capture the tradeoff between reduction size and discrimination rate.

Classification

In this section, the classifiers considered here have all been successfully used in conjunction with sEMG signals, including linear discriminant analysis (LDA), k-nearest neighbors (k-NN) with k ϵ {1, 3, 5, 7, 9 and 11}, random forests (RF) and support vector machine (SVM) with a radial basis function (RBF) kernel as following:

For parameter optimization, the 4-fold cross validation scheme was used to evaluate the best choice of SVM parameters, namely penalty factor C and parameter γ of kernel function. In every experiment, 10-folds cross-validation was conducted. Following the same protocol presented by the authors of the database [13], we also adopted repetition 2 and 5 for testing set, so the remaining four repetitions created training set. Considering computational feasibility, the window sets for training and parameter optimization were subsampled at regular intervals of 3 and 5 respectively (i.e. a window increment of 75 ms and 125 ms). Then, 4 groups were alternately selected as training set to train classifier model and the other 2 groups were used to test the established model. Taking the distributional differences among repetitions into consideration, each of the folds corresponded to one of the four training repetitions instead of random splitting of the folds. In order to find an optimal configuration, parameters were selected using a two-stage grid search via coarse grids, and then fine grids [22]. In the coarse grids, C ϵ {2-8, 2, 22, 23, 24, and 25} and γ ϵ {2-8, 2-4, 2-2, 2, 22, and 23}. In the fine grids, these selected values in coarse grids were examined in a range of ±5%, ±10%, ±15%, and ±20%.

Evaluation

In addition to classification accuracy, a novel metric was also employed to evaluate the window-based control performance. Movement error rate (MER) proposed in [23] for offline evaluation of myoelectric control systems was utilized in this section. MER is a normalized version of edit distance used to count the minimum number of insertions, deletions and substitutions required for two sequences to match necessarily. Before computing the MER, a key process should be taken to regard movements rather than windows as the atoms by erasing adjacent duplicates in both sequences of true labels and the predict labels. Subsequently, the edit distance (MER) of two processed sequences was measured as defined as followed:

According to the definition of MER, lower MER implies that the predicted movements (not windows) achieve a higher similarity with true movements, namely the approach obtains better quality of hand movement control.

RESULTS

In this paper, all the analyses were conducted under the environment of MATLAB 2014a. A preliminary analysis was implemented to select effective features from the feature list we selected. Fig. () shows that the MAV and WPT perform best for single features (75.85±2.58% and 75.12±3.22% respectively), regardless of the training time consumed for WPT. Liner combining the MAV and RMS yields a more robust result than MAV and WPT. Interestingly, the accuracy of the MEAN features extracted from ACC is 86.6±2.59% and thus significantly higher than any of the considered sEMG features. Additionally, Fig. () indicates the MEAN feature after PCA obtains a remarkable improvement of approximately 1% (87.65±3.3%) in result than that without PCA. Thus, the first 18 principle components (PCs) have been used in the following experimental validation. The results in Fig. (). illustrate the average accuracy using either sEMG, or a combination of sEMG and accelerometry, compared with SVM, LDA, Random forests and k-NN classifiers. The SVM with sEMG+ACC modalities performs the best accuracy of 88.72±2.58%, which achieves a significant improvement of almost 13% over sEMG-only and roughly 1.1% over ACC-only. Subsequently, the predicted labels produced by the test sequences of the subject 5 presented in Fig. () were compared with the true labels. Finally, MER was calculated from both sEMG and sEMG+ACC as complementary evaluation criterion for window-based analysis (Table ).

DISCUSSION

Feature and Classifier

The results in Fig. (). indicate that the more advanced WPT feature is not much superior to the simple RMS and MAV, but suffering considerable computational consumption. Literature [3] also shows that MAV is a robust feature in window-based analysis scheme, which has been successfully used in online myoelectric control [24]. Due to the data segment technique, the non-stationary signal in a window gradually becomes stationary signal. Meanwhile, the LDA classifier that was successfully employed to recognize 4 to 10 hand movements [20] achieves miserable results. In any case, Random Forests and SVM performed very similarly for the case only employing sEMG-based scheme. After exploiting accelerometers, the non-liner SVM with RBF kernel is a more robust and discriminative classifier for high-dimensional and multi-class features.

Benefit of Accelerometers

The good performance of the sEMG+ACC confirms that ACC is indeed beneficial to multi-functional movement window-based recognition. Table presents a comparison between the results obtained through the approach proposed in this work and the results obtained through other methods previously published in [8, 9, 24, 26, 27]. It is obvious that our benchmark can recognize more movements when the accuracy is comparable with that in the literature. In addition, we draw a conclusion that it is more effective to integrate ACC rather than increase the sEMG electrode roughly after compared with [9]. Furthermore, the Movement error rates of 5 amputated subjects also obtain a remarkable decrease; i.e. exploiting accelerometers also raises the quality of every intact movement. Fig. () gives a further insight into the ACC modality that redundancy between electrodes influences movement classification. It was also mentioned in [23] that the dense placement and regular orientation of the 12 electrodes caused information redundancy. Hence it is a wise way to reduce the dimension of feature and eliminate the correlation of ACC signal using PCA. Although ACC performs better than sEMG on movement recognition, it cannot replace the function of sEMG on detecting the onset/offset of movement and predicting the muscle force [25].

Evaluation of Error Labels

Given a general view on the error labels presented in Fig. (), almost all the error labels occur near to the onset or offset of individual movements. On the other hand, the experimental results point out that a major of these error labels are misclassified as rest posture (label 0).To our knowledge, it can be explained as follows:

a) Due to the window-based analysis scheme, transitional segments between rest and movement would be produced from test sequences. In reality, these segments obtained test signals and finally were classified as rest labels.

b) These error labels were usually ignored [9] or regarded as prediction delay [23], which was also called motion-selection time in [10].

CONCLUSION

A majority of studies have compared different approaches to develop myoelectric control of upper-limb prosthesis. In this paper, we addressed this problem by classification of 17 functional movements on 5 amputated subjects. Our approach achieves an average accuracy of 88.7±2.6% by combining sEMG and ACC modalities, which confirms that our approach is appropriate for amputated subjects and ACC is an excellent complementary modality for myoelectric control. Furthermore, principle component analysis was employed to reduce the dimension of feature and to eliminate the correlation of ACC signal, which significantly improved the classification performance and reduced MER. In conclusion, the approach we present in this paper boosts the capabilities of dexterous myoelectric hand by hand amputated subjects.

ABBREVIATIONS

Abbreviation	Details	Abbreviation	Surface details
sEMG	Surface electromyography	ACC	Accelerometry
LDA	Linear discriminant analysis	ANN	Artificial neural networks
SVM	Support vector machine	TRAS	Trans-radial amputated subjects
PCA	Principal component analysis	NinaPro	Non Invasive Adaptive Prosthetics
DASH	Disability of the Arm, Shoulder and Hand	RMS	A root-mean-square
MV	The majority vote	MAV	Mean Absolute Value
MAV1	Modified Mean Absolute Value 1	VAR	Variance
WL	Waveform Length	AR	First four autoregressive coefficients
WPT	Wavelet packet transform	MEAN	Mean value
k-NN	k-nearest neighbors	RF	Random forests
RBF	Radial basis function	MER	Movement error rate
PCs	Principle components

Table 1

Description of the 17 movements.

Hand and wrist movements
0 Rest1 Thumb up2 Extension of index and middle finger while flexing others (cf. “V-sign”)3 Flexion of ring and little finger whileextending others4 Thumb opposing base of little finger5 Abduction of the fingers6 Fingers flexed together in fist	7 Pointing index8 Adduction of extended fingers9-10 Wrist supination and pronation(rotation axis through the middle finger)11-12 Wrist supination and pronation(rotation axis through the little finger)13-14 Wrist flexion and extension15-16 Wrist radial and ulnar deviation17 Wrist extension with closed hand

Table 2

Clinical data of hand amputated subjects.

Subject	AmputatedHand	Age	Height(cm)	Weight(Kg)	RemainingForearm (%)	DASH Score	Years sinceAmputation
12345	LeftRightRight & LeftRightRight	3550344445	183178166180183	8182689575	7030409090	15.1822.5086.673.3312.50	651145

Table 3

Average MER obtained from SVM with 3 major vote based sEMG modality and sEMG+ACC modalities over 5 amputated subjects.

Subject		1	2	3	4	5
MER	sEMG	2.913	5.753	3.280	3.260	3.130
	sEMG+ACC	0.97	1.275	0.826	1.122	0.594

Table 4

Researches on hand movement classification in amputated subjects.

References	Electrodes	Classes	Segmentation (ms)	Features	Classifier	Accuracy (%)	Number of amputee
[8]	2	4	2001/NM2	RMS	FCMs3	87.5	1
[26]	6	5	NM	NM	SVM	95	2
[27]	6	8	100/NM	CSSP4	LDA	80.3	1
[24]	8	7	250/505	MAV	KNN	79	5
[9]	64	12	200/25	4TD6	MLP7	87.8	1
[10]	12	10	150/50	4TD	LDA	84.4±7.2	5
This paper	12	18	250/25	MAV/sEMG+MEAN/ACC	SVM	88.7±2.6	5

1 Window length; 2 Not mentioned; 3 Fuzzy C-means; 4 Common Spatio-Spectral Pattern; 5Increment of window; 6Time domain feature; 7 Feedforward multi-layer perceptron.