Comparison of gait speeds from wearable camera and accelerometer in structured and semi-structured environments.

A feasibility study was conducted to investigate the use of a wearable gait analysis system for classifying gait speed using a low-cost wearable camera in a semi-structured indoor setting. Data were collected from 19 participants who wore the system during indoor walk sequences at varying self-determined speeds (slow, medium, and fast). Gait parameters using this system were compared with parameters obtained from a vest comprising of a single triaxial accelerometer and from a marker-based optical motion-capture system. Computer-vision techniques and signal processing methods were used to generate frequency-domain gait parameters from each gait-recording device, and those parameters were analysed to determine the effectiveness of the different measurement systems in discriminating gait speed. Results indicate that the authors' low-cost, portable, vision-based system can be effectively used for in-home gait analysis.

Introduction

Gait analysis is an area of research that has seen an increasing focus due to its applicability to a wide range of age-related health issues, which may affect our growing elderly population [1]. For example, Beauchet et al. [2] found evidence that dementia can be predicted by poor gait performance. Similarly, Valkanova and Ebmeier [3] show that evidence strongly supports a relationship between gait and impairment of cognitive functions in patients with mild cognitive impairment and Alzheimer's disease. These findings illustrate the potential of using gait analysis in detecting symptoms of age-related illnesses.

While many objectively quantifiable gait parameters could be used for effective decision support and automated monitoring, the simple measure of gait speed has shown to be an accurate predictor of mobility, health, and even mortality [4-6]. Gait speed is thus a critical parameter for evaluating the utility of candidate gait analysis systems. Therefore, the goal of this Letter was to determine the feasibility of using our system, comprising of an affordable and non-invasive wearable camera and computer-vision-based processing methods, to classify gait speed of healthy individuals. Samples of gait were collected at three self-determined overground walking speeds (slow, medium, and fast). Since accelerometer-based methods have been successfully used to quantify gait, we deployed an accelerometer near the subject's right hip to provide a direct point of comparison for our system to a more widely used device. We also compared the capabilities of both devices to a research-grade optical motion-capture system, which represents the ‘gold standard’ for gait analysis. The comparison to both another wearable system and a high-precision standard provides validation for using our single-camera system for gait analysis tasks.

Related work

Expensive laboratory-based gait analysis systems can provide extremely robust quantification of human gait and locomotion. For example, highly precise three-dimensional (3D) motion-capture systems have recently been used to study detailed gait features across age groups in healthy individuals [7] as well as individuals with Parkinson's disease [8] and Alzheimer's disease [9]. While these systems can often provide an in-depth description of gait, they are not feasible for the use case of continuous monitoring due to their size, complexity, and cost. Continuous monitoring of patients in their natural environments during everyday activity provides a more constant and natural sampling of gait activity, enabling the detection of changes in performance over time. Additionally, user-friendly and lower-cost pervasive health monitoring systems reduce the burden on patients to make trips to a physician's office, which may be especially cumbersome for the elderly persons.

Recent work has focused on providing convenient, in-home solutions for activity recognition, eliminating the need for a laboratory setup. Video-based methods may offer inexpensive solutions with performance similar to more sophisticated motion-capture systems or floor sensors [10], but suffer from obstructed line-of-sight within the home environment. Audio-based systems have also been used to analyse gait in indoor environments [11, 12]. While both audio- and video-based systems have shown promise, their use is limited to a specifically preconfigured location. To overcome these issues, gait analysis is also being performed with wearable devices such as smartwatches [13], shoe-based wearable sensors [14], and wearable accelerometers [15-18]. By instrumenting the subject instead of the environment, the systems become portable and problems such as line-of-sight obstruction can be avoided. While these solutions provide promising results for gait analysis, they tend to be either still too complicated for in-home use (e.g. due to the number of components) or incapable of matching the level of precision and capturing gait performance as comprehensively as laboratory-grade motion analysis systems. Thus, there are trade-offs between accuracy and cost for laboratory-grade motion analysis systems and in-home wearable systems for gait analysis.

Our system uses a single head-worn camera to collect first-person video. Using computer-vision techniques, we can extract optical flow output from the video that mimics the ability of a low-resolution accelerometer to register movement parameters [19]. A benefit of a vision-based sensor such as our system is that the video data can provide additional context to the in-home monitoring scenario. For instance, if an unexpected event occurs for gait speed, the monitoring system could notify additional automated or manual review processes to analyse the specifically related video segment and determine whether the event was a clinically significant event such as a fall or simply an abrupt stop. It may also be possible to analyse the coarse direction of the subject's visual attention while walking and to identify objects that are being interacted with or other factors that may affect gait performance. Methods based entirely on accelerometer or pressure sensors are unable to explain variations or interruptions that are seen in daily gait activities and would not be able to inform further analysis processes. While vision-based in-home monitoring systems may raise privacy concerns by recording subjects and others in their home, automated methods of processing the video on recording would eliminate the need to store the raw video, mitigating the privacy risk. The storage or transmission of computed motion-based features removes nearly all-identifying information as these features are essentially equivalent to those recorded by inertial sensors, which do not raise the same concerns.

Our previous work based on the use of a wearable camera (and accelerometer for comparison as a well-established device) has involved collecting data from subjects on a treadmill [20]. However, limiting the data collection to occur on a treadmill was an artificial limitation, especially for the in-home use-case being described. In this Letter, we remove this limitation and also investigate the impact of changing this aspect of the experimental design to incorporate semi-structured, natural gait sequences from 19 participants.

Methods

Data collection

Accelerometer, motion capture, and first-person video data were collected from 19 participants as they walked overground six times over a distance of 4 m in a large motion-capture laboratory, free from any physical obstructions such as furniture or walls (Fig. 1). The participants were healthy college students ranging from 18 to 21 years old. About ten of the participants were male, and the remaining nine were female. Participants were instructed to walk at three self-determined speeds: slow, medium, and fast. Categorical speeds were used for multiple reasons. First, as participants were walking over the ground, and not on a treadmill, it would have been difficult to control gait speed adequately. Second, the time-series data from inertial sensors and the processed video features indicated the frequency of each subject's gait. Estimating the continuous gait speed requires knowledge of the exact stride length of a subject. Categorical gait speed was thus more appropriate as slow, medium, and fast gaits naturally correspond to an increasing step frequency regardless of stride length or distance covered.

Fig. 1

Motion-capture laboratory in which data was collected from participants

Each subject wore two commercial devices during data collection (see Fig. 2). The Pivothead SMART Architect Edition glasses [21] were used to record video of the activity in high definition resolution (1920 × 1080) at 30 fps. The device is worn as a pair of eyeglasses, and the camera is located at the centre of the glasses, above the nasal bridge, aimed directly forward. The glasses are nearly indistinguishable in shape and weight from a normal pair of glasses, providing a comfortable and natural sensor that is easily integrated into daily routine with no encumbrance or health risks to the wearer. The Hexoskin smart shirt [22, 23] was also worn, providing triaxial accelerometer readings at 64 Hz (data from the remaining sensors in the Hexoskin shirt were not analysed for this Letter). The accelerometer was located near the right hip on the torso of the subject. Gait data were also recorded with a 20-camera Motion Analysis Corporation Kestrel motion-capture system at a sampling rate of 120 Hz, and motion data were processed using Cortex v. 6.2 software (Motion Analysis Corp., Santa Cruz, CA). Each participant was instrumented with motion-capture markers, according to the Cleveland Clinic marker set. This model includes markers tracking the position of the feet, legs, trunk, arms, and head. While the position of each marker was recorded, our analysis focused on the head marker (for comparison with the head-worn glasses results) and estimates of whole-body centre of mass derived from the global marker set using a whole-body mass model calculated from Zatsiorsky–Seluyanov's body segment inertia parameters [24]. Each of the three systems independently and simultaneously recorded the gait sequences that were performed.

Fig. 2

Placement of sensors on the subject and alignment of axes between sensors

The Pivothead glasses were purchased for $300 United States dollar (USD) and the Hexoskin vest and device may be purchased together for $499 USD, making them easily available to consumers. While the exact price of the motion-capture system is not immediately available and will vary based on configuration, the cost of the 20-camera system is roughly $100,000 USD. A minimal set of four lower-precision cameras could be obtained for <$10,000 USD. Even considering the lower-cost motion-capture option, the consumer-grade devices have the advantage of being easy to use, while the motion-capture system requires an expert user and extensive instrumentation of the subject. While the motion-capture system presents an excellent means for collecting our high-precision truth data, the cost and complexity imply that the motion-capture system will only be feasible in a controlled clinical laboratory, and not for continuous, in-home monitoring. While we did not investigate real-time processing, the computational requirements for data from the single camera and inertial sensor would also be much lower than the 20-camera system.

Signal processing

The intent of incorporating the glasses worn camera into the experiment was to use the collected video to describe the subject's head motion in two dimensions (the frontal and vertical planes) throughout the gait sequence. Since the camera faces directly forward from the glasses, the frontal and vertical axes in physical space correspond to the x and y axes of the recorded video. While the camera device collects less data at a lower sampling rate and spatial resolution than a highly accurate 3D motion-capture system, the device is extremely portable, affordable, and simple to operate. However, the visual data requires processing to extrapolate information about the movement of the subject who is not in the view of the camera. The Lucas–Kanade optical flow technique was applied to the collected video samples to quantify participant motion from the videos [25]. This method outputs a displacement vector for a series of significant keypoints within a given video frame. An average vector was computed from all keypoint vectors per frame to find a single 2D vector, which represented the overall displacement in each frame of video. This approach was previously validated against other possible computer-vision techniques and was found to provide the most accurate representation of the actual displacement between frames [19].

The 2D (frontal and vertical) components of the optical flow displacement vectors were considered over time to generate two separate sets of time-series data. Three-axis time-series data were also collected from the body-worn accelerometer, and head and centre of mass data (each in three dimensions) from the motion-capture system were also analysed. The time-series data were manually separated into segments collected during each of the six trials. The collected data were manually segmented by examining the video and audio, and then recording the start and stop times of each gait segment within the video on a per-frame basis. None of the systems directly provide a determination of when a gait sequence occurs, though it would be feasible to automate this detection based on the collected features and the video data. For this initial work, such a system was not developed since the focus was the output of the system during gait activities. Motion-capture data were filtered in Cortex using a fourth-order low-pass Butterworth filter with a cut-off frequency of 6 Hz, which is the default setting for the low-pass filter in Cortex. Cut-off frequencies in motor control research generally range from 6 to 10 Hz, depending on the behaviour being observed. Given that the observed behaviour was walking, 6 Hz was much higher than the frequencies of interest and only filtered out sensor noise.

Since the wearable devices are commercial devices, which operate independently of each other, it was not possible to guarantee synchronisation of the data collection. This precluded any direct comparison of the raw time-series data from each of the sensing devices, since errors in synchronisation of the collected data would negatively affect any calculations. However, it is not necessary to directly compare the time-series data – gait speed alone has been shown in clinical applications to be a predictor of cognitive disorders such as dementia [26]. To overcome the limitation on direct comparison between the time-series data from each device, we moved data out of the time domain and instead derived frequency-based features in the following manner.

A periodogram calculation was applied over the entirety of each walk segment for each channel of data being considered. As shown in [19], the periodogram transformation can be used to identify main frequencies that occur in each time series. We identified for each time series the frequency with the highest amplitude from the computed periodogram to serve as our gait metric. While the periodogram provides an analysis of constant gait speed in [19] and a successful indication of gait speed in temporally short gait sequences collected for this Letter, longer gait sequences containing changes in gait speed may be better analysed with methods that consider time locality such as a spectrogram. For this Letter, we have assumed that gait speed is constant in each gait segment.

Statistical analysis

The goal of this work was to determine the feasibility of classifying gait samples categorically by their speed with a specific interest in the performance of the video-based wearable system. We determined whether the collected video-based features were impacted by gait speed in a manner similar to the features from more traditional gait analysis devices. We used analysis of variance (ANOVA) to determine whether our gait metric was significantly impacted by gait speed. Separate ANOVAs were conducted for each plane of motion for each gait measurement system. Data were screened for outliers (±2.5 standard deviations from the median) prior to analysis; six trials were identified as outliers, all for the motion-capture system in the sagittal plane and likely resulting from obstruction of one or more markers from the cameras’ view. Violations of the sphericity assumption were resolved by correcting the degrees of freedom of the statistical test using the Greenhouse–Geisser method.

Results

Fig. 3 illustrates the mean gait frequency detected in the vertical and frontal planes for each speed category in the study. ANOVA on the frequency-based gait metric derived from the eyeglass camera data revealed no significant differences across gait speed conditions in the vertical plane (, ). However, in the frontal plane, there was a significant effect of gait speed , , and . Pairwise post-hoc comparisons revealed significant differences among all three speed conditions (fast versus medium: Cohen's ; fast versus slow: ; medium versus slow: ; and all ). This indicates that using the frontal plane data (u for the glasses), the sensor was able to distinguish between gait speeds across the three categories.

Fig. 3

Mean plots with standard error bars for recorded features by gait speed

As was the case with the camera data, the accelerometer data did not discriminate gait speed conditions in the vertical plane ( and ). There was a significant effect of gait speed in the frontal plane , , and . Post-hoc tests again identified significant differences among all three speed conditions (fast versus medium: ; fast versus slow: ; medium versus slow: ; and all ).

For the motion-capture system, we first consider data from the head marker. A significant effect of gait speed condition was observed in the vertical plane , , and . All three speed conditions were found to differ significantly according to post-hoc tests (fast versus medium: ; fast versus slow: ; medium versus slow: ; and all ). A significant effect was also observed in the frontal plane , , and . All three speed conditions were found to differ significantly according to post-hoc tests (fast versus medium: ; fast versus slow: ; medium versus slow: ; and all ).

For the centre of mass displacements calculated from the motion-capture data, ANOVA revealed significant effects of gait speed in each plane of motion. In the vertical plane [, , and ], post-hoc tests revealed significant differences among all conditions (fast versus medium: ; fast versus slow: ; medium versus slow: ; and all ). Similarly, for the frontal plane [, , and ], all pairwise post-hoc comparisons were significant (fast versus medium: ; fast versus slow: ; medium versus slow: ; and all ).

Conclusion

In this Letter, we evaluated the performance of a gait analysis system, which uses only a wearable camera to collect 2D, first-person video. Optical flow and frequency-domain analyses were used to generate a dataset from video, and this dataset was then compared with data collected with a wearable triaxial accelerometer and a 20-camera motion-capture system. This is a crucial step to validate our single-camera wearable system against the vest device and the gold standard motion-capture system. While both of the latter devices (the accelerometer and motion-capture systems) have seen more use in the domain of wearable gait analysis than the wearable camera, our camera-based system has several advantages including cost, simplicity, and the ability to analyse the visual scene to provide context for the gait analysis data. Although the motion-capture system provided superior discrimination of gait speed in all planes of motion, as expected, the eyeglass-based camera system nonetheless discriminated gait speed significantly and outperformed the vest-based accelerometer system. This suggests considerable promise for its use for a variety of health-related gait applications. Plans include using the camera for in-home activity analysis to perform automated detection of gait activities (eliminating the need for manual segmentation) and measuring changes in gait speed over time as a possible measure of fall risk and physical decline in older adults.