Motion cue analysis for parkinsonian gait recognition.

This paper presents a computer-vision based marker-free method for gait-impairment detection in Patients with Parkinson's disease (PWP). The system is based upon the idea that a normal human body attains equilibrium during the gait by aligning the body posture with Axis-of-Gravity (AOG) using feet as the base of support. In contrast, PWP appear to be falling forward as they are less-able to align their body with AOG due to rigid muscular tone. A normal gait exhibits periodic stride-cycles with stride-angle around 45o between the legs, whereas PWP walk with shortened stride-angle with high variability between the stride-cycles. In order to analyze Parkinsonian-gait (PG), subjects were videotaped with several gait-cycles. The subject's body was segmented using a color-segmentation method to form a silhouette. The silhouette was skeletonized for motion cues extraction. The motion cues analyzed were stride-cycles (based on the cyclic leg motion of skeleton) and posture lean (based on the angle between leaned torso of skeleton and AOG). Cosine similarity between an imaginary perfect gait pattern and the subject gait patterns produced 100% recognition rate of PG for 4 normal-controls and 3 PWP. Results suggested that the method is a promising tool to be used for PG assessment in home-environment.

INTRODUCTION

Parkinsonian gait (PG) develops with the passage of time as a result of Parkinson’s disease (PD) features i.e. bradykinesia, postural stability and rigidity in muscular tone [1]. Normally a person walks upright with steady steps and even strides. Distinctive PG has features as stooped posture, short shuffling steps (festination) and slowness to start walking [2]. PWP appear to be falling forward [3]. Levodopa helps to reduce bradykinesia and rigidity in PG [4]. However with longer treatment, levodopa effects wear off. There may be fluctuations in drug response resulting in ON-OFF phenomenon [5]. Drug response fluctuations are corrected with smaller and more frequent dose adjustments which demands accurate timely assessment of gait impairment.

Wearable accelerometers and sensors have been used in gait assessment methods by researchers. Gait assessment systems were composed of Electrogoniometers and Moving Light Displays. Subjects were asked to wear joint markers which may cause patient’s discomfort [6] and consume significant setup time. These problems are tackled using marker-free systems because such systems do not require physical contact. Especially in biomedical applications (like rehabilitation monitoring after orthopedic surgeries and neurological disorder assessment) the hardware setup can be effectively reduced using a marker free gait assessment system giving the patient maximum possible comfort during the assessment.

Howard Lee et al. [7] investigated digital image processing methods for gait assessment to detect neurological disorders. Swing distances and joint angles of subjects were calculated from selected frames in a gait video. Based on these features, the system identified PG in the subjects. The constraint in this work was the setup procedure in which a novel colored dress had been specifically designed for patients for correct image segmentation. Working with still images was another drawback because information about gait cycle frequency and human posture could be obtained from the video clips. R.D. Green et al. [8] extended the research for PG recognition. The environment settings as used by Howard Lee et al. [7] were used to identify gait abnormalities in video images. On the basis of wave-form gait cycles, this system correctly classified 94% of subjects diagnosed with PD with one false negative. Cho CW et al. [6] applied Linear Discriminant Analysis on still images to identify PG. The system achieved 95.49% classification accuracy for 1529 test cases.

This paper aims to propose a simple vision-based PG recognition algorithm based on the assessment of the human posture lean and stride cycles during the gait. The main objective is to introduce a marker-free gait assessment in a realistic setup for monitoring. The paper is divided as follows. Section 2 gives an overview of the system. Results are elaborated in section 3. Section 4 depicts the conclusion and the future work.

SYSTEM DESCRIPTION

A block diagram of the proposed PG recognition algorithm is shown in Fig. (). The subject’s gait is first recorded using a camera. The foreground pixels of the human body are segmented from the background pixels based on the pixel brightness in each frame. In this way, the need for novel colored dress for laboratory experiment as used by RD Green and Howard Lee [7, 8] is eliminated. The image enhancements are done to extract the 2D silhouette from the gait video frames. The silhouette is isolated with a bounding rectangle to estimate the height and width of the subject. Once the height and width of silhouette is achieved, it is compared with a human model [9] to find head, torso and leg segments. Skeleton is formed by computing the medial points of each body segment. The motion cues are extracted from the skeleton. Two motion cues analyzed are the cyclic motion of legs and the posture lean of the subject during the gait. These two cues are compared with the cues of an imaginary perfect gait pattern to assess the gait impairment. The working units in the algorithm are detailed below. Functions available in the OpenCV [10] (Open Computer Vision) library are utilized to develop the algorithm in C++ programming environment.

Background Subtraction

The subjects were asked to walk parallel in front of the camera during a video recording of 20 seconds with frame rate kept as 5 frames per second (fps). Each video frame img[m,n] of pixel resolution ‘m x n’ is transformed from RGB color space to the HSI color space [11]. The foreground pixels of image img[m,n] are segmented from the background pixels to produce a binary image based on the brightness threshold ‘θ’as shown in equation 1.1.

Where, ‘S’ and ‘I’ are the thresholding values of Saturation and Intensity respectively for image img[m, n] in the HSI space. The dark pixel values of background are eliminated and the foreground pixels which represent the human body are preserved in the image. The binarized image is filtered in order to remove small blobs which possibly do not represent the body segments due to their small size. Anomalies in this filtered image are removed by the application of morphological dilation followed by erosion. These operators eliminate small holes in the obtained silhouette and robustify the pixels of thin body segments.

Silhouette Isolation

The obtained silhouette is isolated using a bounding rectangle as shown in Fig. (). The x and y coordinates of top-left vertex of the bounding rectangle is denoted as (xThe x and y coordinates of bottom-right vertex is represented as (xThe coordinates of the bounding box are used to find the silhouette’s height S and width S as given in equations 1.2 and 1.3 respectively.

Model Fitting and Skeletonization

The silhouette is mapped to a gait model (previously described by Jeffrey, Faezah and Safabaksh [9, 12]) using values of the silhouette’s height S and the width S. The head, torso and leg segments in the silhouette are estimated by dividing the bounding rectangle into the anatomical proportions shown in Fig. (). According to the anatomical proportions, the head segment comprises of the upper 13% portion of the silhouette’s height S. The width of the head segment is 10% of silhouette’s width S. The torso segment is comprised of 28.8% of S. The width of the torso segment is 17% of S. The legs are comprised of the lower 53% proportion of the silhouette’s height S.

Once the proportions of the body segments are separated from the silhouette, the segments are skeletonized by computing the medial points of each body segment as shown in Fig. (). The centre of gravity (COG) of the binarized silhouette is calculated using equation 1.4 and 1.5.

Where N represents the number of white pixels belonging to the silhouette and (x being the x and y coordinates of white pixels in the silhouette. The body is skeletonized by joining the tangents of the medial points to the COG as shown in Fig. (). The tangent which joins the head segment to the COG is called the Torso Vector (TV). The angle between the Axis of Gravity (AOG) and the TV denotes the lean angle of the silhouette’s posture. The two tangents that join the left and the right feet to the COG respectively are called the Feet Vectors (FVs). The angle between the two FVs represents the stride angle. Notice that the larger the stride angle of the subject during the gait, the larger will be the stride length in the gait cycle.

Motion Cue Extraction

The easiest parameter to be measured quantitatively among temporal and spatial gait parameters is the walking speed (meters/sec) because the motion and alignment of body joints are strongly influenced by the velocity throughout the gait cycle [13]. Studies revealed that cadence and speed are gait features which are robust to dopa-medication [4]. The speed during the gait is related to two variables; the stride cycles and the stride length. Further, the PWP have the tendency to lean forward during the gait [2]. The focus of this work is therefore emphasized on the two motion cues i.e. the posture lean and the stride cycles during the gait.

The motion cues are computed from the silhouette’s skeleton. The posture lean is estimated by computing the lean angle of skeleton in each video frame. To execute this approach, AOG is formed in the image img[m, n] by drawing a vertical line passing through the silhouette’s COG as given in line equation 1.6.

The x (i.e. x-coordinate of the point COG) acts as the x-intercept for the line AOG. The torso vector (TV) is computed by joining the x and y coordinates of the centre of head segment (denoted by xand yrespectively) to the COG with the tangent given in equation 1.7.

The cosine between the unit vectors TV and AOGis computed to find the posture-lean angle θ as given in equation 1.8.

The stride cycles are estimated from the cyclic leg movement of skeleton in each video frame. The stride angle is the maximum opening between the front and back feet in a stride. To estimate the stride angle, the feet vectors are computed by joining the bottom-most medial coordinates of left and the right feet segments with two tangents to the COG respectively as given in equations 1.9 and 1.10.

Where (x1, y1) and (x2, y2) are the bottom-most medial coordinates of the left and the right feet of skeleton respectively. The cosine between the unit vectors and is computed to find the stride angle θ as given in equation 1.11.

The posture lean angle θ and stride angle θ of a normal subject and a patient diagnosed with PD respectively are shown in Fig. ().

Motion Cues Matching

The main idea behind the matching of the motion cue patterns is to compute cosine similarity between subject’s gait pattern and an imaginary perfect gait pattern. Jeffrey et al. [12] revealed that the PWP walk with shortened stride angle with high and variable stride frequency. Murray P. et al. [14] depicted that, a normal gait exhibits constant stride frequency though that the stride length and stride angle varies depending upon the age of subject. They studied the walking patterns of 60 normal subjects based on the age and height distribution using a metronome. They found the mean stride length of a normal gait to be 78.4 ± 5.9 cm. We estimated an average leg height based on the human model proportions [9, 12] using the body heights of 60 normal subjects mentioned by Murray P. et al. [14].

Using the value of average leg height (93cms), an isosceles triangle is formed with two equal triangular sides as feet vectors and triangular base equal to the mean stride length (78.4cms) as shown in Fig. (). A bisector is drawn from the vertex COG against the triangular base which produces a right-triangle ΔABC. With the average leg height taken as perpendicular and the half of mean stride length taken as base , the angle is computed to yield θstride value in equation 2.

We called the θ value in equation 2 as a perfect stride angle value denoted by θ. The cosine of θ is 0.7. Note that the cosine for 0o is equal to 1 and that the cosine values decrease as the stride angle increases. A comparison between the normal subject’s strides and a PD patient’s strides is given in Fig. () when they were asked to walk from right to left in front of the camera. Notice that the normal gait (Fig. ) yielded periodic stride pattern during the gait-cycles whereas the PG (Fig. ) exhibited irregular pattern of strides during the gait cycles.

A peak-finder algorithm [15] has been used to detect the valleys in gait pattern. Note that a valley is a cos θ value in a gait cycle. We assumed a minimum θ between the FVs during a normal gait to be 25o. A threshold of 0.1 (i.e. cosθ = cos0) has been used to detect the valley points in the gait pattern. It is possible that the peak-finder algorithm may not detect a valley in the gait pattern with the given threshold. Especially in case of PG, the muscular constrictions in the feet of PWP results in θ shorter than 25o. In this condition, the algorithm is re-iterated with the adjusted threshold value to 0.01 so that the minimum θ up to 10o can be detected (i.e. cosθ = cos 0). The valleys i in patterns of normal gait and PG are shown in Fig. ( and ) respectively. Notice that the numbers of valleys are comparatively larger in number in the pattern of PG than the normal gait which illustrates the reduced step length and shorter stride angle in gait patterns of PWP.

Based on the assumption that an imaginary perfect gait exhibits constant stride frequency with stride angle θ, the subject’s strides may be matched with the perfect strides by computing the cosine distance d between the θ and θ for a gait cycle i (equation 3.1).

Where n is the total number of gait cycles. The distance equation depicts that the larger the value of d, the larger will be the cosine difference between the patterns of normal and the perfect gait. The stride variability in subject’s gait pattern can be represented by computing the mean of the overall distance d as given in equation 3.2.

Note that the PG pattern (shown in Fig. ) yields higher stride variability as compared to the normal gait pattern (shown in Fig. ).

The periodic gait pattern should yield constant time interval for each gait cycle. Time variations in the gait cycles may be found by computing the residuals between the intervals. The residuals r are computed by subtracting the time interval t in a gait cycle i by the overall mean (represented as t) of total intervals t for gait cycles n (equation 4.1).

The mean r of residual values r for gait cycles n (equation 4.2) is yet another feature for the assessment of gait anomalies in PWP as it detects shuffling and festination in gait.

The stooped posture of PWP during the gait is determined by the posture lean angle θ. We assume that a perfect gait exhibits an erect posture throughout the gait with the lean angle 0o denoted as θ . The cosine of θ is equal to 1. A comparison of the posture lean between the normal gait and the PG is shown in Figs. ( and ) respectively. The normal gait yielded a smoother graph for the posture lean angle per frame. The cosine values of posture lean angle θ in a normal gait were found closed to the cosine values of θ. In case of PG, abrupt changes in the graph of posture lean angle per frame were observed as well as the cosine values of posture lean angle per frame were found farther from the cosine value of θ.

Matching between the patterns of subject’s leaned posture and a perfect erect posture is performed by computing the cosine distance l between them (equation 5.1).

Where, m is the total number of gait video-frames. The mean l of cosine distance l has been computed to yield a single value to depict posture lean (equation 5.2).

The gait features i.e. stride variability d residual mean r and posture lean l are normalized within a range 0-to-1. A Support-Vector Machine (SVM) attribute evaluator based on Recursive Feature Elimination [16] has been used to weight each attribute. A set of 49 samples each with the three gait features have been trained in the SVM using 10-fold cross validation to rank the attributes. The square of the weight assigned by the SVM-based attribute evaluator is used to rank each class separately based on the one-vs.-all method. The classifier ranked the features l, dand r based on the given dataset as 3, 2 and 1 with the weights as 2.5, 2.5 and 1 respectively. The gait error E can be computed by the weighted average of gait features (equation 6.1). The difference of E from perfect gait G yields a gait match percentage (equation 6.2).

RESULTS

Several gait cycles of a group of four normal controls n and three PWP p were assessed using the described motion cue assessment algorithm. The gait features d r and l are computed for gait cycles n for each subject. The overall gait matching G% (equation 6.2) is computed for each subject. The results showing the gait quality for each subject respectively are shown in Table .

It is observed from the experiments that the gait patterns from the normal controls showed high percentage of gait matching G% with the perfect gait pattern. The normal controls n exhibited low stride variability d which resulted in higher G% values i.e. 92.6%, 87% and 88.3% respectively. The normal control n showed high stride variability d which resulted in low G% value of 71%. All of the normal controls n produced low values of posture lean l and residual mean r.

In case of PWP, the gait matching between the pathological gait patterns and the perfect gait pattern were fairly low i.e. G% was equal to 16.8%, 7.3% and 58.6% for p and prespectively. The subject p though showed low value of lean in the posture l (i.e. 0.033) which resulted in gait matching of 58.6%. An overall gait matching threshold of 70% was selected to discriminate between the normal and the Parkinsonian gait which resulted in 100% correct recognition for 7 subjects.

DISCUSSION

In conclusion, a simple computer-vision based methodology was developed for assessment of Parkinsonian gait. The result is a marker-free approach, which may be feasible in terms of portability in the home environment for the assessment of gait of PWP. It comprises a realistic approach based on the human body posture relative to the body’s centre of gravity for monitoring purposes. Features such as the posture lean angle, the stride variability and the mean of residuals between the stride intervals proved efficient features for the PG assessment. An SVM-based attribute evaluator was used to weight the different gait features. Based on these features and using Cosine Similarity matching with the imaginary perfect gait pattern, the system identified the PG in 7 subjects (3 PWP and 4 normal controls) with complete accuracy. The idea of using Cosine Similarity matching to a “perfect pattern” makes the algorithm computationally inexpensive.

The previous Linear Discriminant Analysis [6] and correlation [8] methods though produced 95% and 94% recognition rates for PG on other data sets but they are computationally complicated and expensive. These assessment methods required markers to be placed on the subject’s body. The approaches [6-8] using the Principal Component Analysis and General Regression Neural Network required extensive laboratory setups and prerequisites for the experiments thus making the methods unfeasible to be used for assessment in the home environment. The marker-free gait assessment method based on cosine similarity described in this paper is the least computationally expensive due to distance computations. The method does not require complex setup which makes it feasible to be used for gait assessment in home-environment.

This novel image processing technique has however a few limitations associated: The perfect stride angle θ was estimated from gait patterns recorded using a metronome [14] which was used as a benchmark to differentiate between the stride patterns of normal gait and PG. This is in contrast to the real-life where the normal gait speed and cadence vary between age and height groups. The other limitation in this method is the need of color contrast between subject’s apparel and background for color-segmentation.

In future, the proposed system will be tested on a larger clinically rated database to assess level of gait impairment based on the symptom severity. Different statistical methods and machine learning methods will be evaluated for ability to score overall gait impairment based on the gait features described in this paper. The algorithm will hopefully have the potential to aid clinicians in the following of Parkinson’s disease treatment, when used in conjunction with monitoring the other symptoms of the disease.

Table 1.

A comparison of Gait Assessment between 4 Normal Controls and 3 PWP

Subject	Posture Lean lavg	Stride Variability davg	Residual Mean ravg	Error E	Gait Match G%
n1	0.001	0.044	0.500	0.073	92.6
n2	0.001	0.063	2.222	0.129	87.0
n3	0.004	0.059	1.000	0.117	88.3
n4	0.016	0.158	0.680	0.29	71.0
p1	0.078	0.264	10.060	0.832	16.8
p2	0.109	0.268	8.250	0.926	7.3
p3	0.033	0.192	1.209	0.414	58.6