A study of the effect of visual depth information on upper limb movement by use of measurement of smoothness.

[Purpose] This study verified that the smoothness of reaching movements is able to quantitatively evaluate the effects of two- and three-dimensional images on movement in healthy people. In addition, clinical data of cerebrovascular accident patients were also analyzed by the same method. [Subjects] Ten healthy adult volunteers and two male patients with previous cerebrovascular accidents participated. [Methods] The subjects were tasked with reaching for objects shown on a display. The target and virtual limb, rendered with computer graphics, were shown on the display. Movements of the virtual limb were synchronized with those of the subject. Healthy subjects reached for targets with their dominant arm, and cerebrovascular accident patients used their paretic arm. A polarized display and polarized glasses were used when the subjects were shown three-dimensional images. In the present study, jerk cost was used to quantify the smoothness of movement.
[Results] Six of the 10 healthy subjects had significantly smoother reaching movements when viewing the three-dimensional images. The two cerebrovascular accident patients tended to have smoother movements in response to the three-dimensional images.
[Conclusion] Analysis of the smoothness of movement was able to detect the influence of the depth cue in vision on movement quantitatively for the healthy subjects and cerebrovascular accident patients.

INTRODUCTION

Despite decreases in stroke mortality rates worldwide, the number of people with new-onset strokes, the number of stroke victims, and disability-adjusted life years (DALYs) lost continue to grow1). The many stroke survivors with upper limb motor impairment as sequelae often never fully regain function2,3,4). Such motor impairment limits activities of daily living and reduces the quality of life of affected stroke survivors5, 6). Alleviating upper limb motor impairment is thus a major focus of rehabilitation.

Intensive use and repetitive task training of the affected upper limb are considered effective rehabilitation techniques for upper limb motor impairment7). Constraint-induced movement therapy (CIT) is a prominent example of such techniques. In CIT, the affected limb is intensively trained while the intact limb is restrained. Although studies have shown CIT to be effective, many therapists and patients remain skeptical8, 9).

Recent advances in technology have brought greater use of computers to the field of rehabilitation. Virtual reality (VR) is a common application. Many investigations have found VR to be an effective tool in upper limb rehabilitation10,11,12,13,14,15,16,17,18). VR-based rehabilitation of stroke patients produces improvements based on neuroplasticity7, 19,20,21). Investigators have used VR in telerehabilitation and home rehabilitation conducted via the Internet22,23,24). The use of computer-based rehabilitation is expected to grow.

Use of VR helps keep patients motivated, allows greater training frequency, duration, and repetitions, and enhances feedback25, 26). VR rehabilitation provides visual, auditory, deep sensory, and kinesthetic feedback. Special devices such as vibrating units and robotic arms are needed for deep sensory and kinesthetic feedback27,28,29). Such devices are costly and often difficult to handle, fit to the patient, and maintain. Less costly and easier to use are displays and head-mounted displays used to provide visual feedback30,31,32).

Several investigators have evaluated the effects of visual feedback in VR rehabilitation for stroke patients. Bonan et al. found that stroke patients rely heavily on visual movement-related information when moving in a VR environment33). They also found that the effects of motor learning depend on the accuracy of the visual information provided33, 34). Visual feedback thus enhances the effects of VR rehabilitation.

Information with binocular parallax informs the perception of space and, specifically, depth perception. Either two- or three-dimensional (2D or 3D, respectively) images can be used to construct virtual spaces. Perspective and shading are used in 2D images to replicate depth information. However, with 3D images, depth from binocular parallax is expressed by showing each eye a slightly different image. Three-dimensional images thus provide a more spatially realistic environment than their 2D counterparts.

The efficacy of rehabilitation conducted with 3D images was demonstrated in studies that used batteries of clinical tests to evaluate motor recovery30, 31, 35). Lee et al. evaluated the finger-pointing performance of healthy subjects in a 3D environment created with a hemispherical display36). They noted that the subjects needed time to point at targets arranged before them. However, no study has quantitatively investigated the effects of 2D and 3D images on these movements using parameters other than time or human movement trajectories. In the present study, the efficacy of quantitative analysis of the effects of 2D and 3D images on movement was evaluated based on the smoothness of reaching movements of an upper limb in healthy people. In addition, clinical data of cerebrovascular accident (CVA) patients were also analyzed by the same method. Jerk cost was used to quantify the smoothness of movement. To properly reach, people must know the relationship between the positions of the hand and the target object37). Might differences in the depth information provided by 2D and 3D images be reflected in the patient’s perception of distance? Might smoothness of movement be affected? It was hypothesized that movement would be smoother for 3D images because of the rich depth information they provide.

SUBJECTS AND METHODS

Subjects

Ten healthy adult volunteers, 21.8±4.0 years of age (5 males, 5 females), participated in the study. No subject had a disease affecting visual-spatial perception or upper limb motor function (Table 1). Two male patients (64 and 71 years of age) with a previous CVA also participated. Neither CVA patient had a visual-spatial perception disorder, severe motor impairment, or ataxia that would have interfered with reaching ability (Table 2). The present study was part of a larger study approved by the ethics committee of Hokkaido University. After being informed about the study and its methods orally and in writing, all participants signed an informed consent form.

Table 1.

Data of the healthy subjects

Subject	Age	Gender	Dominant hand	Experimental order
A	23	Female	Right	2D → 3D
B	21	Female	Right	2D → 3D
C	26	Male	Right	2D → 3D
D	22	Male	Right	2D → 3D
E	31	Male	Right	2D → 3D
F	18	Female	Right	3D → 2D
G	18	Female	Right	3D → 2D
H	19	Male	Right	3D → 2D
I	20	Female	Right	3D → 2D
J	20	Male	Right	3D → 2D

Table 2.

Data of the CVA patients

Subject	Gender	Age	Diagnosis	Affected side	Period between onset and experiment (days)	Clinical assessment	Experimental order
K	Male	64	Cerebral infarction	Right	35	Br-Stage V/VI, FIM 75/35	2D → 3D
L	Male	71	Cerebral hemorrhage	Right	16	Br-Stage: V/V, FIM 57/33	3D → 2D

Br-Stage: Brunnstrom stage, upper extremity/lower extremity, FIM: Functional Independence Measure, motor score/cognition score

Methods

The objective of the study was to verify the possibility of quantitatively analyzing the effects of 2D and 3D images on upper limb movement using the smoothness of the movements. The subjects were tasked with reaching for objects shown on a display. The experimental setup is shown in Fig. 1. The target and virtual limb, rendered with computer graphics, were shown on the display. Movements of the virtual limb were synchronized with those of the subject. Microsoft’s Kinect (Microsoft, Redmond, WA, USA), a touchless sensor, was used to track movements of the subject’s limb. The Kinect is able to calculate spatial coordinates of all segments without the need to fit the subject with reflective markers. The subjects were asked to perform the tasks seated 1.8 meters from the display. A polarized display (Mitsubishi RDT234WX-3D, Tokyo, Japan) and polarized glasses were used when the subjects were shown 3D images. TriDef 3D middleware (Dynamic Digital Depth USA Inc., Los Angeles, CA, USA) was used to render 3D images. During the study, movement in the sagittal plane was recorded with a video camera.

Fig. 1.

Experimental system

A Kinect is fixed on a tripod. Subjects sit 1.8 m from the sensor.

The targets the subjects reached for were 4 boxes situated on lines 0°, 22.5°, and 45° from horizontal abduction across the 90° plane of shoulder flexion. The targets were white, blue, yellow, and red. The white box was placed at a fixed distance on the line 0° from horizontal abduction (30% of the upper limb length). The blue, yellow, and red boxes were then placed at random along one of the three lines at distances of 50%, 70%, or 90% of the upper limb length. This was done to ensure that no two colored boxes were placed on the same line or at the same distance from the white box (Fig. 2).

Fig. 2.

An example of the target arrangement

The subjects reached for the targets in the order of white, blue, yellow, and then red. The word “hit” was displayed to inform the subject when the virtual limb made contact with the target. The subjects were asked to keep touching the targets until they disappeared (5 s for the white box and 2 s for the other boxes). After a target disappeared, the subject was to reach for the next target. The subjects were asked to move their upper limb at a speed they were comfortable with. They were instructed to move the shortest distance between targets. Healthy subjects reached for targets with their dominant arm, and CVA patients used their paretic arm. Each subject underwent 15 trials. The first 5 trials were for practice. The data from the last 10 trials were included in the analyses. The 2D and 3D targets in the 15 trials were identically arranged for each subject.

Measurements were collected over 2 days. Five healthy subjects and 1 CVA patient undertook the trials in 2D on the first day. The other 5 healthy subjects and 1 CVA patient performed the trials in 3D on the first day. The subjects were switched from 2D to 3D or vice versa on the second day.

Smoothness of movements was compared to evaluate the effects of visual information on movement. Smoothness can be quantified under a concept called “jerk cost.” Jerk is defined as the percent change in acceleration over time. Decreases in jerk cost indicate an increase in movement smoothness. Normalized jerk cost (NJC) as presented by Kitazawa et al. was used in the present study38). NJC can be determined with the following formula: t1: Start of movementt2: End of movementlength: Length of reach trajectory

NJC was calculated from hand coordinates recorded with the Kinect while the subjects reached from target to target. Data were analyzed from the start of a reach movement to when the target was touched. The start of movement was defined as the point when velocity exceeded the mean velocity ± 2 standard deviations of 5 s of rest time. The mean NJC and standard deviation were determined for 10 trials. Reach time and trajectory length were also calculated for the same data analysis intervals, and then the means and standard deviations were determined for 10 trials. Data were excluded from the analysis when upper limb tracking failed.

Wilcoxon’s signed rank test was used to evaluate intra-subject differences and average NJCs, reach times, and trajectory lengths of all healthy subjects for 2D and 3D images (p<0.05). Statistical analysis was performed with PASW Statistics 18.

RESULTS

NJCs, reach times, and trajectory lengths are shown for the healthy subjects and CVA patients in Table 3. Six of the 10 healthy subjects had significantly smoother reaching movements when viewing the 3D images. Four of these 6 subjects were shown 2D images on the first day. The two CVA patients tended to have smoother movements in response to the 3D images. Changes in reaching velocity and acceleration in response to 2D and 3D images are shown in Fig. 3. The subjects with differences in NJCs frequently repeated acceleration and deceleration movements when viewing the 2D images.

Table 3.

Comparison of outcomes of the subjects

Subject		n	Time (sec)			Path length (m)			Normalized jerk cost
			(Median [IQR])			(Median [IQR])			(Median [IQR])
			2D	3D		2D	3D		2D	3D
Healthy subjet	A	27	0.67	0.50	**	0.23	0.25		154.9	117.0	**
			[0.42–1.08]	[0.33–0.83]		[0.20–0.35]	[0.16–0.29]		[66.5–605.0]	[31.3–330.1]
	B	25	1.00	0.75	*	0.27	0.29		459.1	268.8	*
			[0.67–1.33]	[0.58–0.92]		[0.19–0.34]	[0.18–0.31]		[184.6–790.1]	[150.4–470.6]
	C	26	1.17	0.92	*	0.31	0.29		777.5	441.6	*
			[0.76–1.6]	[0.72–1.16]		[0.22–0.37]	[0.21–0.36]		[214.2–1354.0]	[216.5–634.1]
	D	22	1.08	1.08		0.31	0.36		799.1	625.6
			[0.81–1.46]	[0.66–1.35]		[0.21–0.40]	[0.21–0.41]		[299.9–1053.9]	[170.7–1260.4]
	E	25	1.08	1.01	*	0.30	0.27		731.6	524.1	*
			[0.87–2.00]	[0.74–1.54]		[0.19–0.41]	[0.22–0.40]		[275.7–1934.2]	[275.6–1212.6]
	F	28	0.90	0.79		0.28	0.24		399.7	336.0
			[0.60–1.02]	[0.60–1.02]		[0.18–0.34]	[0.19–0.33]		[2–1.4–778.7]	[152.4–753.5]
	G	24	1.01	0.84	*	0.24	0.25		498.7	295.0	*
			[0.71–1.23]	[0.52–1.08]		[0.22–0.33]	[0.18–0.34]		[211.4–681.4]	[96.3–527.6]
	H	22	1.00	1.00		0.30	0.29		541.3	668.9
			[0.74–1.44]	[0.75–1.31]		[0.19–0.37]	[0.22–0.37]		[242.4–1003.7]	[332.8–1066.2]
	I	20	1.33	0.99	*	0.32	0.26	*	1210.6	680.6	*
			[0.87–1.72]	[0.70–1.42]		[0.21–0.42]	[0.19–0.35]		[471.1–2160.8]	[212.9–1194.5]
	J	21	1.00	0.76		0.26	0.25		522.9	306.3
			[0.55–1.42]	[0.58–1.09]		[0.19–0.36]	[0.20–0.32]		[101.9–1172.1]	[160.7–550.3]
CVA patients	K	27	1.83	1.83		0.45	0.41		2395.6	1484.8	*
			[1.50–2.50]	[1.41–2.50]		[0.28–0.80]	[0.31–0.58]		[878.0–6971.1]	[1,077.8–3011.9]
	L	25	2.17	1.83	*	0.39	0.37	*	2764.3	2517.4	**
			[1.79–3.63]	[1.46–2.54]		[0.33–0.63]	[0.24–0.44]		[2,231.9–10379.6]	[1,001.3–4719.5]

Comparisons were carried out using a Wilcoxon signed rank test.

* p<0.05, ** p<0.01

Fig. 3.

A representative example of a change of velocity and acceleration in healthy and CVA subjects during reaching motion. These graphs show the results for reaching movements to the same target in each subject.

Reach time was significantly longer for 2D images in 4 healthy subjects and 1 CVA patient (Table 3). Four of these 6 subjects were shown 2D images on the first day.

Trajectory length was significantly longer for 2D images in 1 healthy subject and 1 CVA patient (Table 3). The healthy subject was shown 2D images on the first day, and the CVA patient was shown 3D images on the first day.

The average NJC and reach time of all healthy subjects were significantly improved when the subjects viewed the 3D images (Table 4).

Table 4.

Comparison of the average values of all the healthy subjects

	n	Time (sec)			Path length (m)			Normalized jerk cost
		(Median [IQR])			(Median [IQR])			(Median [IQR])
		2D	3D		2D	3D		2D	3D
Healthy subjets	238	1.00	0.85	**	0.28	0.27		526.5	356.7	**
		[0.67–1.35]	[0.58–1.16]		[0.20–0.36]	[0.20–0.34]		[202.0–1,093.2]	[158.4–711.7]

Comparisons were carried out using a Wilcoxon signed rank test. **p<0.01

DISCUSSION

The purpose of this study was to verify that the smoothness of the movement is able to quantitatively detect differences in the effects of using 2D and 3D images on movement in a VR environment for healthy adults. Reach was the movement evaluated. Reach smoothness (normalized jerk cost, or NJC), reach time, and reach trajectory length were calculated to allow this quantitative evaluation. Reach trajectory length when 2D images were shown did not differ significantly from that when 3D images were shown. Reach time and NJC, however, were significantly shorter in more than half of the healthy subjects when 3D images were shown. Moreover, this study preliminary investigated whether this analysis could be applied to CVA patients. The results of the CVA patients were the same as those of the healthy subjects.

Trajectories did not differ when 2D and 3D images were shown. Viau et al. found no differences in upper limb reach strategies used in real and virtual space39). Visual guidance, however, is purported to be necessary to facilitate precise movement in complex motor tasks26, 29, 40). The linear arrangement and proximity of the targets in the present study required only simple motor skills. Synchronized display of the virtual limb, moreover, allowed the subjects to easily determine limb position and the direction to the target in the virtual environment. This likely explains the lack of a difference in reach trajectory length between the 2D and 3D conditions.

Reach times and NJCs were shorter when the subjects were shown 3D images. Multiple subjects reported that movement was easier under the 3D conditions because they were better able to perceive depth. These findings indicate that the subjects were able to more easily and smoothly reach for the targets when shown 3D images. Other researchers have investigated reaching in virtual environments. Lee et al. evaluated reaching by healthy subjects in a 3D environment created with a hemispherical display36). In their study, reaching in the depth dimension took the most time. They concluded that the extent of perception of depth information may influence movement in the depth dimension. Van Erp et al. evaluated pointing in a VR environment enhanced with shading and perspective41). Performance was poorer in the depth dimension than in the horizontal and vertical dimensions. Like Lee et al., they concluded that visual information is important for motor performance in the depth dimension. Images in these studies were displayed differently from those in the present study. These earlier studies provided positional information in space through monocular depth perception techniques such as movement parallax, occlusion, relative size, and shading. These techniques are commonly used in 2D graphics applications. Binocular depth perception, in contrast, provides additional depth information through binocular parallax and convergence. The 3D display used in the present study incorporated these techniques. The addition of depth information through binocular parallax to the information provided in monocular depth perception appears to have improved movement performance.

Figure 3 illustrates how the amount of depth information affects movement smoothness. Many of the subjects frequently repeated acceleration and deceleration movements when reaching under the 2D conditions, but they did so less frequently under the 3D conditions. Depth perception appears to have provided assistance. Under the 2D conditions, the subjects perceived depth through monocular depth perception and were therefore probably unable to judge the distance between the virtual limb and target. This indicates that the subjects had to frequently accelerate and decelerate, carefully probing for the target, before achieving a “hit.” The truer perception of depth through binocular depth perception under the 3D conditions appears to have allowed the subjects to accurately judge the distance between the virtual limb and target. It seems they were able to reach more smoothly for the targets without overly relying on hit information. Agreeing with previous findings, the findings of the present study indicate that visual information, and particularly depth information, may affect performance in the forward-backward direction.

In the CVA patients, the results showed that the effect of differences in depth information was apparent in the smoothness of the movements. CVA patients receive less afferent information from nonvisual sensory organs and therefore over-rely on visual information to compensate33, 34). Providing visual depth information was therefore likely beneficial to the CVA patients. However, this preliminary conclusion must be validated with a larger sample size. People reach to touch and grab nearby objects. Proper reaching requires smoothness of movement. Movement is less smooth in many stroke patients42) but can be improved through rehabilitation. Use of 3D images rendered with binocular parallax in VR rehabilitation could improve movement.

There were four subjects with a significant difference in smoothness of movements when tested with 3D images and then 2D images during the experiment. Also, there were two subjects with a significant difference in smoothness of movements when tested with 2D images and then 3D images during the experiment. For this reason, it will be necessary to analyze the stimulus order effects in detail with an increased number of subjects and a variety of conditions.

This study has two limitations. First, few CVA patients were enrolled because they were required to have mild motor impairment so that they could reach targets arranged identically to those of the healthy subjects. In future research, the effects of differences in how visual information is provided on the smoothness of movement will be investigated in patients with moderate motor impairment, with targets arranged to accommodate this greater severity. The scope of patients will be expanded beyond CVA patients to ataxic patients. Second, tracking errors by the Kinect reduced data quality. The Kinect recognizes the upper and lower limbs and the trunk by identifying points on the body. The points identified on the upper limbs are the palms, wrists, elbows, and shoulders. The linear arrangement of these points during reaching appears to have prevented recognition. The use of magnetic sensors could improve upper limb tracking precision.

This study verified that the effects of visual depth information on upper limb movement can be quantitatively analyzed using the smoothness of movement in healthy adults. Additionally, clinical data of CVA patients were also analyzed by the same method. The results showed that the smoothness of movement is able to quantitatively detect the effect of differences in depth cue. Furthermore, the results suggest that how depth information is provided may substantially affect rehabilitation. Computer-based rehabilitation better motivates patients, increasing training frequency and duration. VR is not limited to stroke patients, as it is applied in walking and balance rehabilitation and training for elderly people43,44,45,46). Computer-based rehabilitation can be offered not only in medical institutions but also in patient homes and via telerehabilitation. Its use is therefore expected to expand. Since movement quality depends on how images are provided, appropriate devices must be developed and used in systems when effective rehabilitation is the goal.