Human Verification Using a Combination of Static and Dynamic Characteristics in Foot Pressure Images.

Since gait is the mixture of many complex movements, each individual can define with a unique foot pressure image that can be used as a reliable biometric scale for human verification. Foot pressure color images of Center for Biometrics and Security Research (CBSR) dataset from 45 men and 5 women were used in this study. Owing to the properties of this dataset, an index of foot pressure in addition to external feature and contourlet coefficient of images was extracted. A multilayer perceptron (MLP) was utilized for verification of subjects (it is a common practice to explain more about the training and test dataset). To validate the algorithm performance, results were obtained using a 5-fold cross validation approach. The results indicated accuracy of 99.14±0.65 and equal error rate (EER) of 0.02. These results demonstrated the reliability of proposed neural network in human verification application. Hence, it can be utilized in other verification systems.

Introduction

Foot pressure is an internal pressure which is created during daily motional activities or standing between foot and ground or any other supporting surface. Studying this pressure could be useful in some fields including: patients motional and balance difficulties, diabetic foot wound treatment, insole design, etc.[1]

Since walking consists of many complex movements, it could be said that foot pressure images and pressure distribution pattern are unique for each person and can be used as a biometric scale for identity verification. In initial studies, rubber and ink were used to record foot pressure images, and the distance between the big toe and hill were used as verification scale.[2] After that, sole geometrical characteristics such as distance and angle between each foot print pair,[3] steps length and width[4] have an important role in human verification. With the development in technology, and pressure recording sensors, other elements such as pressure center,[456789] maximum pressure area,[4689] and pressure–time curve[1011] were examined. Some restriction such as nonconformities between skin and sensor materials, sensors displacement[1] and low resolution[12] were noticed due to the sensor location inside the shoe.[89111314] In addition to geometrical elements of sole, other elements such as wavelet transform,[1315] modified Haar wavelet,[16] and also Principle Component Analysis (PCA) and linear discriminant analysis (LDA)[17] transform were highly noticed. In more recent years, pressure–time patterns have played a key role in diagnosis and control diseases such as diabetes, Parkinson syndrome, deformed foot, unbalanced walking[18] and also in examining pressure distribution in half paralyzed patients.

Human gait is the result of movement of muscles and joints. Body shape, length and weight, step length, and other elements have an important role in human gait.[19] Consider that humans are unique; this complex structure has made that impression that hundreds of different mechanical motions can be unique for each person.[3]

Foot pressure is an internal pressure, which is created during your daily movements or standing between foot and ground or any other supporting surface. Examining this pressure could be useful in walking analyses, biometrics, diagnosing diabetic foot, assessing treatment trend and motion improvement.[1] When a person steps or stands on sensors, foot pressure images are recorded by scanners [Figure 1]. In each image, red and purple colors are points with maximum pressure and green, blue, black and white colors indicate minimum pressure points.[20]

Figure 1

Foot scan system

Materials and Methods

Collecting pressure data

In this study, foot pressure images from Chinese Academy of Sciences, Institute of Automation (CASIA) have been used.[21] These data have been gathered from walking samples of 5 women and 48 men, who were all Asian, in an age range from 20 to 60 years. For each person, two images from right foot and one image from left foot in each sequence were taken, and this was repeated for five times. In total, 15 images were recorded for each candidate [Figure 2].

Figure 2

Recording of foot pressure images

Images were formed with red, yellow, green and blue colors, which respectively show the pressure from most to least.

Preprocessing

Data preparation includes removal of false, incomplete, duplicate, inconsistent or inappropriate structure. In reviewing data, left foot pressure image for one of the samples had not been recorded and was thus considered as missing datum. In this case, one of the images of the left foot, belonging to the same person, was randomly selected. Furthermore, the three samples of foot pressure images were not well recorded because of improper placement of the feet on screen on scanner system. So they did not follow the normal pattern for the rest of the pressure foot images, which we have to remove altogether [Figure 3]. Finally, 50 samples remained for us. Furthermore, the size of samples was different; with the help of Matlab software, all of those were set as 50*100.

Figure 3

An abnormal foot pressure image

Feature extraction

After data preparation and choosing suitable images for 50 candidates, it is now time for extracting the features. Important information in foot pressure images needs to be extracted. In this step, heel area, the angle of the internal arch of the foot, statistical characteristics of contourlet coefficients, and high pressure in each images were selected as static features. In addition, pressure changes over a step in the right foot as well as dynamic characteristics were investigated.

Heel area

One of the most frequent methods for classifying and separating image from its background is thresholding with the use of color histogram. To determine the boundary of the heel, the color factors that create most conflicts between pixels heel and background were selected to help them fit threshold to obtain a binary image. By examining some examples in the Red Green Blue model of heel, blue and green colors were considered as a main scale for separation.

Threshold values for green and blue colors were 0.95 and 0.5, respectively. These values were chosen as given below:

Pixel = if threshold green > 0.95 and blue < 0.5 otherwise      (1)

Since, there were a lot of pressure points with this threshold value in the below toes of foot, so in addition to threshold values, we introduced another condition on the basis of the total number of pixels in the heel area. Heel was bigger than other parts of the foot and could allocate more pixels to it. After checking of some examples, 20 values were set as a new threshold for number of pixels in the heel. So in every image, which the number of pixels was more than 20 and threshold values for green and blue colors were in the range of threshold, all pixels for this part was counted as heel area [Figure 4].

Figure 4

Extraction of heel image

The angle of the internal arch of the foot

There were many standards in assessing the internal arch of the foot, but none of them directly pointed to the calculation of the angle of the internal arch of the foot. Regarding this matter, this assessment was done with the help of simple math equations and use of highlighted points coordinates. First, the main foot pressure image had to be separated from the black background. As explained before, color histogram and examining each element in Red Green Blue model had a big role in this way. So again by using this method, threshold values for blue, red and green colors were determined as below:

Pixel = if threshold green and blue and red > 0.5 otherwise      (2)

Then, with the help of Eq. (2), color foot pressure image was separated from the background and transformed into a binary image. Next step was to achieve key point to calculate the angle of the internal arch. For this reason, first, we cut the borders of the sole, in a way that for each right and left foot, the lowest value and the highest value in each matrix line of foot image were chosen respectively. After indicating the border of the foot, we consider the innermost point of the arc as one of three key points. This point was beginning and end values of the data matrix respectively, for right and left foot. Then for left foot, from this point we moved toward the outermost point in the upper third of the data matrix which after that, pixel value is 0. This point was chosen as a second point for the left foot. Again, from the middle point, we moved downward to border and found the third point as the previous point. For the right foot, values which are close to the upper and lower corner of left side of the image, are considered as key points.

As can be seen in Figure 5, we can form a triangle using the three points. Our purpose is to calculate internal angle of this triangle.

Figure 5

Forming an imaginary triangle

If we assume coordinates of these three points as P(x1·y1), P(x2·y2), P(x3·y3), then length of the segment between any two points can be calculated from below equations:

In the above equations D is a length of the segment between any two points. Also if we assume that D(P1P2) = a, D(P2P3) = b, D(P1P3) = c; then the angle of the internal arch of the foot pressure image can be calculated as below:

where θ is the angle of the internal arch, which is considered as the second feature in this article.

Contourlet transform

Contourlet transform can be applied to image in different level and various directions; so different results could be achieved. Therefore, considering the current studies in the field of contourlet transform, new and distinct features can be extracted from its coefficients. For this purpose, in the multiscaled matter, Pyramid Laplacian Filter Bank was used, and in directive matter, the Directional Filter Bank was used. Catalysis was done in two levels, in the first level, a sub image in 128*128 size and in second level four sub images with same dimension were achieved; all of them had 16,384 coefficients. It was not easy to directly use the achieved coefficients because of their high volume, which resulted in slowing down the calculation. So we decided to extract other key points with the help of statistical components. In this way, mean, standard deviation, variance, skewness, and kurtosis were extracted of contourlet transform coefficients and then they were considered as the features.

Max pressure points in static mode

In color foot pressure images, red color indicates the highest pressure; on the other hand by examining of some samples, we found that red components are in three regions of the foot including: big toe, under toes and under heel which seemed normal, because these three parts play an important role during the step cycle.

After recalling the image in Matlab software, we separated red channel of each color image and demonstrated each pixel in three dimensional level, so that x and y indicate location coordinates and z indicates the value of the red component of that pixel. In this stage, we faced a large volume of pixels in a three dimensional environment so we decided to extract new components of these pixels in a three dimensional environment.

Our idea was that a surface should cross from all points and each surface has a unique equation and coefficients. In order to find the coefficients for the curve fitting equation the curve fitting toolbox is used. This toolbox can cross a line or complex surfaces through points and find their coefficients. First, x, y, and z values, which respectively indicate location coordinates and pressure amount or red component in foot pressure images, were inserted in curve-fitting toolbox. Since we had three variable inputs, instead of the curve, a surface was shown. In the next step, we used a polynomial surface for curve fitting, because it had less Root Mean Square Error than other models in the toolbox [Figure 6]. In addition, fitted surface was close to red points. This polynomial equation is shown below:

Figure 6

Curve fitting (EQUATION (5))

In above equation, p is the root polynomial, in which there are 21 of them for each foot pressure image. These coefficients are placed in a feature matrix.

Pressure changes in one-step cycle

There were two images of a right foot and one of left during a step cycle that saved in each person in the data center. Since the purpose was to study the pressure changes, therefore, in this section, foot pressure images of right foot were chosen. As can be seen in Figure 7, red color intensity is changed in paw, under the toes and heel during a step cycle.

Figure 7

Change of colors in gait cycle

Then, high-pressure pixels or red components from two-foot scan images were extracted and stored in two data matrices separately. In the next stage, we calculated difference of the two matrices. The resultant matrix was the changes in red color component during the step cycle of right foot. As before, a polynomial surface was fitted to these points. At the end, equation coefficients for this curve were saved as our next feature.

In total, 49 features were obtained for each person in data center which include: heel area, the angle of the internal arch of the foot, mean, standard deviation, variance, skewness and kurtosis of contourlet transform, 21 coefficients of high pressure in static mode, and the last/final 21 coefficients of pressure changes in one step.

Multilayer perceptron classifier

In this study, from 250 samples out of the 50 given data, 200 samples were used for network training and the rest were used for testing stage. For train-test ratio of 80:20 was used. But there are some unknowns which must be determined before training the network. Some of these parameters are as follows: number of hidden layers, number of neurons in each layer, the number of repetition, etc., which were further discussed in detail.

Data normalization

Inputting raw data reduce the speed and precision in the network; hence inputs must be normalized. In the first step, feature matrix was separately normalized with the help of the max–min method.

Reducing dimension

Because of high volume input matrix, PCA method was used for reducing dimension; in such a way that in a new environment, only feature, which has more than 95% energy, will be saved and in next step the rest are applied to classify as new inputs. In this new environment, without a dramatic drop in amount of data information, features dimension were reduced to 25, which in reality these features will count as network Input layer neurons.

Hidden layers

Since an artificial neural network with one hidden layer and providing suitable neuron numbers can be approximated in a general way, so in network design, with the choice of middle neurons, one hidden layer was applied. In this regard, a number of hidden layers were studied with 3–5 neurons in hidden layer and at the end, five neurons were chosen for hidden layer.

Activation function

In case of choosing nonlinear excitation function for hidden layer neurons, any kind of classification with any precision can be done while choosing only one hidden layer and there is no need for increasing them. We need to calculate the differentiation of excitation function in network training algorithm hence, we utilize same excitation function for hidden and output layer neurons. Furthur more, we apply sigmoid tangent instead of hyperbolic tangent, since it is faster in terms of executation time.

Number of output layer's neurons

Our purpose in this study is to verify the identity; therefore, outputs were determined through 0 and 1 class; which in fact accept or reject the user identity. Considering one output layer, the number of neurons regarding this layer is equal to number of classes, which is 2. Class 1 is correct answer (or confirming the examined candidate) and class 0 is the wrong answer (not confirming the candidate identity).

Training perceptron network

In this stage, first random values for synapses are chosen and then training process was done for the entire data. Modification of synapses weight was done by error Back Propagation method. Similarly and before beginning each stage, synapses weight was modified and above process was repeated. Learning rate in the training algorithm was 0.01.

Epoch numbers

If epoch numbers are low, network will be weak against new data, and in case of increase, they will only memorize training data. In this study, we have set epoch numbers as 100 for network.

Stop

There are some methods to stop a neural network; from which, classifier error rate is considered as stopping scale. If error gradient, while classifying sample, is less than wanted threshold limit, network will stop. Error gradient was 0.00166.

Results

Finally, for studying and assessing recommended network, binary classifier criteria such as accuracy, precision, and sensitivity as well as biometric systems-assessing coefficients namely equal error rate (EER) and 5-fold cross validation were studied. Since network performance was examined in 10 thresholds, all the results, for 50 candidates were separately calculated and their mean was calculated in those threshold. According to Table 1, accuracy equal to 98.06% has been achieved.

Table 1

Results of evaluation of classification

Also for more accurate assessment of work and considering the data quantity, 5-fold cross validation method has been used, in which all of the 250 samples were divided into five groups; so in each group, 50 samples were placed. Then in each learning stage of classifier, one of subgroups was assigned for testing and the rest were assigned for training. This is repeated for five times and finally average of accuracy, precision, sensitivity and EER were calculated.

According to Table 2, highest accuracy is equal to 99.14%. As shown, precision and accuracy have high values. In addition, False Acceptance Rate and False Recognition Rate graphs were calculated and plotted. Two graphs are shown in one [Figure 8].

Table 2

Results of evaluation of classification in 5-fold cross-validation

Figure 8

EER of MLP

Curves collision point in threshold is 0.94, which has become around 0.02; which in fact is Error Equal Rate (EER). This scale has been used to assess biometric system performance; when it goes lower, system performance will get better.

Discussion

Automatic systems on the basis of recorded images have become an appropriate field for research development due to advancing technology in recent years. Studying and analyzing human movement have attracted a lot of attention among world researchers in different fields. Identifying, tracking, and verifying a human's identity could be the purpose, while such systems and their results have many other applications including smart surveillance systems. Each person has a unique pattern and ground reaction force. This element is giving us the ability to study and examine walking and other components related to foot area.

In this study, a system was designed and analyzed, which is capable of verifying or rejecting anybody's identity from foot pressure images with help of visual machinery techniques. Data center includes data for 50 candidates. These data were consisted of two images from left foot and one image from right foot.

First, heel area and internal arch angle were considered physical characteristics of a sole. In previous studies, sole area was examined in eight different divisions, but extracting heel and calculating its area were not solely considered. On the other hand, internal arch angle of the foot was defined with help of standard scales in different articles such as Staheli Index, Arch Index, Chippaux-Smirak Index, Clarke Arch and Arch Length Index but the proposed method in this paper is not calculated in none of them. In addition to this, using internal arch angle as verification factor was not noticed in any of the studies done in the field of verification and identification.

Another property attracting our attention is variable colors in foot pressure scan. Considering that the colors display the amount of pressure of the foot, the stronger color, meaning red, showed the most pressure in that section and was considered as the main criteria in this section. Also, studying the recorded images in each walking cycle, which includes two images of the right foot and one image of the left foot, we have reached the conclusion that the intensity and area of the color red are different in each right foot. Therefore, the color red in the right foot was considered as the base of pressure changes in each step.

Our last benchmark is to apply contourlet transform instead of transforming wavelet and Haar wavelet on the samples. Since each sub image achieved by this transform, had covered many coefficients, so we decided to extract new features in addition to reducing the matrix size with the help of statistical components such as mean, variance, standard deviation, skewness, and kurtosis.

In final stage, for determining best inputs in multilayer perception network and after many experiments and testing different kind of samples, specific values for which their energy was more than 95% after applying PCA, were chosen as network input. We achieved our purpose to reduce error under 5% by using these features and choosing one middle layer and five neurons for it.

To assess more precisely, two similar articles with same subject and features were clustered in two groups. In the first study, combination of physical characteristics and pressure of foot images were compared with Takeda's method.[4] Since results in the Takeda's method were expressed according to EER, so foot pressure images data were divided into five categories and each time, four groups for training and the rest as test data were applied to the network. False Acceptance Rate and False Recognition Rate were calculated every time and the end after averaging five times, mean of intersection of these two curves were considered as EER. It is noteworthy that in Takeda's method pressure characteristics are obtained directly by pressure sensors. According to Table 3, EER is much lower than that of Takeda.

Table 3

Comparison between the results of the proposed method with Takeda's method [4]

In the second study, combination of physical characteristics, contourlet and pressure of foot images were compared with Zheng's method.[17] Results are shown in Table 4.

Table 4

Comparison between the results of the proposed method with Zheng's method [17]

Foot pressure images could be used in detecting damaged areas in ankles and foot, as well as to identify high pressure in the foot of people with diabetes and biometrics. However, if you wear shoes, cannot achieved information with high spatial resolution and the appropriate result is achieved without shoes.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.