Development and evaluation of a new telerehabilitation system based on VR technology using multisensory feedback for patients with stroke.

[Purpose] The purpose of this study was to develop a new telerehabilitation system based on VR technology for training of paralyzed upper and lower extremities and poor balance in patients with stroke. Moreover, the effectiveness of the system was verified by analysis of the recovery of these patients. [Subjects] Five healthy persons and five people with motor paralysis, caused by cerebrovascular disease, participated. [Methods] The features of our system are as follows: (1) Our system can train upper and lower limbs and balancing with 3D images. (2) A Kinect(®) is used for user posture detection. (3) A vibrator is used for feedback to a sensory receptor in order to promote the learning effect of motion. Upper limb and balance training were conducted in this study.
[Results] The time necessary for the upper limb and balance training tasks was shortened for the participants with disabilities. The joint angle for the participants with disabilities tended to equate to that of the healthy participants over time. Moreover, our system had no side effects.
[Conclusion] These points suggest that our system is effective and safe. The user interface and assessment of the conditions of patients from a distance should be studied in the future.

INTRODUCTION

Japan, the population of which is rapidly aging, is experiencing incessant financial stress from high medical expenses1, 2). The governmental policy stance for improving this situation has been shifting swiftly to home care. Home rehabilitation has a serious problem with respect to maintenance of the quality of rehabilitation components because of limitation of the rehabilitation period in Japan3). Although the effectiveness of rehabilitation at home is evident4,5,6), the numbers of facilities and personnel are insufficient to fully meet the needs of distant or depopulated places at present3). Even in the United States, the inability of handicapped elderly people living in rural areas to receive medical services easily after leaving a hospital that provides rehabilitation is an important issue7).

Telerehabilitation using a computer has been proposed as a measure to solve this issue. In telerehabilitation, a therapist supports, evaluates, and assists handicapped persons from a distant location using telecommunication technology7). Reviews of telerehabilitation efforts have revealed its effectiveness not only in terms of clinical results but also in terms of user satisfaction and costs8). Telerehabilitation systems commonly use video or audio systems, but systems using virtual reality (VR) also exist9). The success of rehabilitation using VR has been demonstrated10,11,12,13,14,15). The accomplishments of a VR-based telerehabilitation system have also been reported16,17,18). Nevertheless previous studies of telerehabilitation systems have focused on individual movements, such as finger motion, upper limb motion, walking, and balancing, and no device yet produced can train upper and lower limbs and balancing comprehensively.

Visual and auditory approaches are used mainly as feedback to sensory receptors that are effective in the promotion of learning of motion. Studies of VR-based visual feedback mostly adopt a 2D display. A 2D image provides a feeling of depth with perspective and shadow methods, but it lacks information that can be detected or conveyed by binocular parallax, whereas a 3D image can present such information, so it allows spatial perception closer to nature19).

Aside from visual or auditory feedback, some systems adopt force feedback, which employs haptic or exoskeleton devices. However, issues of cost, handling, and safety remain with force feedback because of its specifications20). The effectiveness of feedback for deep sensation using vibratory stimulation has been shown as a sensory feedback system21, 22). Nevertheless, no existing report in the literature describes a study of a telerehabilitation system that uses vibratory stimulation for feedback. Because a telerehabilitation system can use a vibrator such as that commonly built into mobile phones or smart phones, it can be regarded as an inexpensive and safe feedback device for a distant operation.

The purpose of this study was to develop a new telerehabilitation system based on VR technology using multisensory feedback for training of paralyzed upper and lower extremities and poor balance in patients with stroke. Moreover, the effectiveness of the system was verified by analysis of recovery of the patients.

SUBJECTS AND METHODS

Subjects

Healthy people and people with disabilities using the present system as a stand-alone system were compared, and the changes resulting from using the system continuously were analyzed in people with disabilities to verify system feasibility. The performance of the system was also examined when it was operated from a distant location.

The participants in this study comprised five healthy persons with no upper limb disorder and five people with disabilities (Tables 1 and 2). One healthy participant and two participants with stroke received upper limb training in a sitting position with the stand-alone system. Four healthy participants and one participant with stroke received balance training in a standing position with the stand-alone system. One participant with stroke received upper limb training in a sitting position with the system controlled from a distant location. Among participants with disabilities, those who received upper limb training were administered a Simple Test for Evaluating Hand Function (STEF) and the Functional Independence Measure (FIM). Those who received balance training were administered the Functional Balance Scale (FBS) and FIM. Those who received rehabilitation with the system controlled from a distant location were evaluated for Brunnstrom Stage (Table 2). This research was part of a study that obtained approval from the ethics committees of the University of Hokkaido, the University of Tokyo, and Sapporo Shuyukai Hospital. The subjects received a thorough explanation about the study contents and methods both orally and in writing, and their written consent to participation was received.

Table 1.

Healthy subject data

Subject	Gender	Age (yrs)	Task
A	Male	26	Reaching while sitting
B	Female	26	Balance training while standing
C	Female	28	Balance training while standing
D	Female	29	Balance training while standing
E	Female	24	Balance training while standing

Table 2.

Patient data

Subject	Gender	Age(yrs)	Diagnosis	AffectedSide	Period between onsetand VR training (days)	Task	Clinical assessment
F	Female	50	Cerebral infarction	Right	17	Reaching while sitting	STEF 32 (100) / FIM 106 (126)
G	Male	83	Cerebral infarction	Right	32	Reaching while sitting	STEF 83 (100) / FIM 112 (112)
H	Female	61	Brain stem infarction	Left	712	Balance training while standing	FBS 41 (56) / FIM 100 (126)
I	Male	56	Cerebral hemorrhage	Right	506	Balance training while standing	FBS 46 (56) / FIM 112 (126)
J	Male	67	Cerebral infarction	Right	2230	Reaching while sitting on telerehabilitation	Br Stage U/E III

Methods

A block diagram of the telerehabilitation system prototype is presented in Fig. 1. Components of the rehabilitation program prepared using this system were presented using 3D image. Graphical data generated by the system were converted into stereoscopic 3D data through the Tridef® 3D middleware (Dynamic Digital Depth USA Inc.). A polarized display (Mitsubishi RDT234WX-3D) was used for the system, and the users wore polarized glasses so that they could see 3D images. Although polarized displays have the inherent disadvantage of half the resolution compared with normal displays, they have the advantages of being flicker-free and being able to present image information simultaneously to both eyes. In addition, polarized glasses are light enough that they can be worn comfortably. Even clip-on type devices can be furnished for fitting over glasses.

Fig. 1.

Block diagram of the telerehabilitation system

A Kinect® (Microsoft Corp.), which facilitates unconstrained measurement of body coordinates, was used for user posture detection. The Kinect® measures the distance from its camera to the body surface using infrared light. It then estimates the positions and attitudes of body segments. Because the Kinect® requires no adhesive markers, which are necessary for ordinary motion analysis devices, it reduces the load borne by the user.

A vibrator (Kyowa Electronic Instruments Co., Ltd.) was used to provide feedback to a sensory receptor. It was fixed to a Velcro® band. The users performed the component of the rehabilitation program with the band fixed to the hand.

Skype® (Microsoft Corp.), which permits users to make video calls at no charge, was used for conversations with video and audio between patients in distant locations and a therapist. In addition, the PubNub® Real-Time Network, a communication service provided by PubNub, Inc., was used for transmission and reception of commands or real-time data communication of states so that a therapist could monitor the components of the rehabilitation program and control the system from a distant location. Furthermore, the DropboxTM file-sharing service was used for collection of the raw position data necessary for detailed off-line data analysis. This system can be put in practical use as a stand-alone system or operated from a distant location.

For upper limb and balance training, the present system displayed virtual objects as targets. The participant was required to extend their upper limbs to touch them. The position of the targets in space and the movements of the upper limbs of the participant were shown on the 3D display using computer graphics (Fig. 2). When a reaching motion was made that touched a target, a tactile feeling was fed back by the vibration device attached to the hand. The targets were located on coordinate axes centered on the acromion, and their positions were determined according to upper limb length.

Fig. 2.

VR rehabilitation game scene

The 3D display presents the virtual targets and movement of the upper limbs of the participant using CG.

The upper limb training in a sitting position comprised an upward reaching motion, a lateral reaching motion, and a forward reaching motion. For the upward reaching motion, targets were positioned at a depth equivalent to 85% of the upper limb length and at flexion angles of the shoulder joint of 45°, 90°, and 135°, respectively. For the lateral reaching motion, one target was positioned on 85% of upper limb length at the 90 degrees shoulder flexion and two targets were also placed at the range of between the 45 degrees abduction and the 45 degrees adduction of the shoulder joint. For the forward reaching motion, three targets were aligned with an acromial line. The farthest target was set at a distance greater than the upper limb length. Upper limb training was initiated with all targets present. The participant performed 10 trials for each task per day in addition to ordinary rehabilitation. Participants with disabilities executed the training continually for 20 days.

The targets were positioned in two hemispheres centered on the unaffected acromion in the balance training in a standing position. The radii of the hemispheres were set to 60% and 85% of the upper limb length. Nine targets were placed in each hemisphere (a total of 18 targets). Targets were displayed randomly; a new target appeared when target was touched. The healthy participants and participants with disabilities performed three trials of balance training each day.

Regarding telerehabilitation, the upper limb training task was conducted in a sitting position by connecting the system in the participant’s home with a hospital via the Internet using mobile devices (E-mobile; LTE) (Fig. 3).

Fig. 3.

Test of operation of the telerehabilitation system from a distant location

The time necessary for task execution and the 3D coordinate data of each joint acquired from the measured data were used for analyses. Analyzed items included the average time necessary and joint angle for the upper limb training task, the average time required for the balance training task, and the average time required and coordinate changes in the telerehabilitation task. The measured joint angles were flexion of the shoulder joint, horizontal flexion of the shoulder joint, flexion of the elbow joint, and dorsal flexion of the wrist joint. Data of six trials, except for the first two and last two trials out of ten trials, were extracted for each upper limb training task for comparison between healthy participants and participants with disabilities. Moreover, the average times required in the first and last trials and the average of each joint angle were compared for the participants with disabilities. The average time necessary for three trials of the balance training task was calculated. Then the healthy participants and participants with disabilities were compared. The performance of the system when operated from a distant location was examined according to the circumstances of communication and real-time monitoring and the conditions of data transmission and reception.

RESULTS

Table 3 presents the average time necessary for each upper limb training task for the healthy participant and participants with disabilities. Patients required more time than the healthy participants to execute all tasks in the first trial, but the necessary time shorter in the last trial compared with the first trial for the participants with disabilities.

Table 3.

Average time necessary for each upper limb training task in a sitting position

		Upward reaching	Forward reaching	Lateral reaching
Control		2.9	2.4	4.0
Patient	First session	7.2±0.4	3.7±0.6	6.8±2.0
	Last session	3.5±0.65	2.4±0.9	3.7±0.96

(sec)

Table 4 shows the joint angle of each participant during reaching tasks in a sitting position. In the upward reaching task, the flexion of the shoulder joint was set to be 135°. The flexion of the shoulder joint of the healthy participant was close to the expected value of 135°. The flexion of the shoulder and elbow joint of disabled participant F approached that of the healthy participant in the last trial compared with the first trial. The flexion of the shoulder joint of disabled participant G did not change, but the flexion of his elbow joint eventually approached that of the healthy participant.

Table 4.

Changes in shoulder and elbow joint angles during each reaching task in a sitting position

In the forward reaching task, the subjects were asked to reach toward the farthest target set in front of them. The angle for the healthy participant was close to the anticipated value, but the flexion of the shoulder joint was smaller and that of the elbow joint was larger for the participants with disabilities compared with those for the healthy participant. Regarding secular changes in the participants with disabilities, flexion of the shoulder joint increased and the flexion of the elbow joint decreased in the last trial compared with the first trial.

When the horizontal flexion of the shoulder joint and the flexion of the elbow joint are setting to be 45° and 0°, respectively, in the lateral reach motion task. The flexion of the elbow joint was greater for the participants with disabilities than for the healthy participant. The flexion of the shoulder joint for disabled participant G increased finally. The flexion of the elbow joint for participants with disabilities F and G decline at the last trial to approach that of the healthy participant.

Table 5 presents the results of comparison of the average time required for the healthy participants and participant with disabilities in the balance task. The healthy participants were able to complete the task in an almost constant amount of time beginning at the start. Although the participant with disabilities required more time in the first trial, the time was shorter after the second trial. Neither healthy participants nor participants with disabilities exhibited a severe degree of loss of balance during performance of the task.

Table 5.

Average performance time in the balance task

	1st Trial	2nd Trial	3rd Trial
Control	58.0±5.2	43.1±3.3	42.8±3.4
Patient	377.0±190.3	114.8±23.3	104.1±27.7

(sec)

Network connectivity was established promptly for video calls and file transfers in operation of the telerehabilitation system. No delay in data transfer occurred that interfered with assessment or control of the situation. The data measured on the participant’s side were transmitted to the hospital in real time. The hospital was then able to assess the motion of the participant. Although some attitude data were lost in communication, a complete data file was transmitted promptly via the DropboxTM file-sharing service to the hospital within less than a minute completion of measurements.

DISCUSSION

In this study, a VR-based telerehabilitation system was developed. This system can be utilized to enable elderly and disabled people to receive the same quality of rehabilitation at home as they would in home-visit and day-care rehabilitation. The effectiveness and the feasibility of the system were examined for stroke patients.

Issues that must be examined when constructing a rehabilitation system that uses VR include usability, safety, and cost, as well as effectiveness. Because the users assumed in this study were handicapped elderly people, we determined that the system must be low-cost, user-friendly, and safe. Moreover, because operation from a distant location presumably involves a hospital, which would have robust network security, priority was assigned to ensuring that a stable connection could be made with the system in the home of a patient without altering any settings inside the hospital.

This system used a Kinect®, which is made by Microsoft Corp., for motion measurement. The Kinect® can perform motion capture to ascertain the posture of each joint without special calibration or markers. Therefore, the system can be operated without difficulty by just installing a Kinect® during system setup. Furthermore, the Kinect® is advantageous in that it is available at a low price compared with other commercial operation-analysis devices. Studies of VR rehabilitation system using the Kinect® have demonstrated its effectiveness23, 24). However, the Kinect® is occasionally unable to recognize a joint marker because of complex or fast joint movement, so a detailed analysis might require the correction of error data.

A therapist operated the computer because the Kinect® recognize user’s movement by maintaining the distance (approximately 1.8 m) between Kinect® and the user. Because the system currently requires an operator to control the software with a mouse or keyboard, conducting training in a location distant from the computer is difficult. The user interface must be improved in the future by making it possible to control the software without a mouse or keyboard by using the motion of a subject’s unaffected side taking advantage of the features of the Kinect®.

Information related to depth was given to the participants as visual information using a commercially available 3D display, with the aim of providing a training environment in an environment closer to reality. The actual users’ opinion demonstrated that this assisted them in comprehending the spatial relation between their hand and computer-generated targets. The effect of virtual reality sickness should be examined with respect to 3D graphics25). No symptom of virtual reality sickness was observed either in the present rehabilitation session or in a pilot survey performed in advance. Moreover, because a polarized display was used for the system, there were no complaints regarding flicker, trouble seeing images, or asthenopia. These points suggest that this system is a physiologically safe.

When a user reached to a target on the display, a vibrator stimulated a touch sensation of the user as a sensory feedback. The vibrator was fixed to a Velcro band so that it could be worn without difficulty. Because the vibrator used in this study is small and lightweight, it is suitable for mobile phones and smart phones. It is also regarded as safe. Moreover, many of the participants in this study reported that this method assisted them in comprehending when a target was touched.

Skype®, PubNub®, and DropboxTM, which were used in this study to provide a robust network environment, present benefits such as the ability to easily pass through a firewall, no need for a special server, and simple installation in a patient’s home or a hospital. Although some data were lost in real-time data transfer, it presented no great difficulty for monitoring. In addition, because raw data are useful for off-line analysis, satisfactorily detailed evaluation was assured.

Upper limb and balance training were conducted in this study to examine the components of a rehabilitation program. The time necessary for the upper limb training task decreased with time in the participants with disabilities. The joint angle for the participants with disabilities tended to become equivalent to that of the healthy participants over time. Consequently, kinematic data also supported the system effectiveness. The results demonstrating that the VR system improved motor functions show good agreement with the findings of previous research of VR use in stroke patients10,11,12,13,14,15,16,17,18). Upper limb motor function presumably improved because the affected side was exercised intensively using a VR game that was similar to constraint-induced therapy26). This system fed back a tactile sensation to a target contact using a vibrator. Feedback to a sensory receptor promotes effective relearning of motion27). It is therefore also regarded as having encouraged positive upper limb motion engendering the recovery of motor functions in the present study. The balance training task required a long time for the participants with disabilities in the first trial. However, the task could be completed in a shorter time after the second trial. The reasons for this include that the contents of the tasks performed in this study were easy to understand for participants with disabilities. Training will be continued, and the effects of balance training in a standing position will be observed in future studies.

In the present study, it was found that the participants with disabilities touched targets with smaller angular movements in each joint than anticipated in the upper limb training task. This showed that the subjects might be compensating for paralysis by moving the trunk. The compensatory action is presumed to be an inhibitor in the process of recovery from paralysis. Therefore, appropriate instruction for accurate movement of the trunk and upper extremities must be given during telerehabilitation.

Accordingly, we expect to improve this system so that a therapist can give appropriate instruction in real time through a display when operating the system from a distant location and so that the state of the trunk and lower limb of a user will be displayed on a screen using computer graphics. The present system adopted a simple task of touching a target so that elderly people or people with disabilities could understand it easily. However, the system has a few problems: operation of the PC to use the system is difficult, and the system cannot be applied to versatile patients. In the near future, the components of the rehabilitation system and the usability of the system will be enhanced so that the user’s motivation can be sustained.