Multimedia virtualized environment for shoulder pain rehabilitation.

[Purpose] Researchers imported games and virtual reality training to help participants train their shoulders in a relaxed environment.
[Subjects and Methods] This study included the use of Kinect somatosensory device with Unity software to develop 3-dimensional situational games. The data collected from this training process can be uploaded via the Internet to a cloud or server for participants to perform self-inspection. The data can be a reference for the medical staff to assess training effectiveness for those with impairments and plan patient rehabilitation courses.
[Results] In the training activities, 8 subjects with normal shoulder function demonstrated that the system has good stability and reproducibility. Six subjects with impaired shoulder underwent 6 weeks of training. During the third week of training, average performance stabilized. The t-test comparing 1-2 weeks to 3-4 weeks and 5-6 weeks showed significant differences.
[Conclusion] Using games as training methods improved patient concentration, interest in participation and allowed patients to forget about their body discomfort. The equipment utilized in this study is inexpensive, easy to obtain, and the system is easy to install. People can perform simple self-training both at home or in the office.

INTRODUCTION

Having healthy upper extremity function is imperative. Therefore, moderate exercise or training and appropriate maintenance are critical. For those with impaired limbs, it is more crucial to perform proper reasonable rehabilitation training to restore normal daily function.

Most human physical actions involve using the hands. According to reports from the U.S. National Security Agency, one-third of occupationally impaired body functions affect the upper limbs1). In its functional disability standards, the American Medical Association identifies that losing one arm is equal to losing 60% of the body mechanisms. The loss of a hand equates to the loss of 90% of the arm function or 54% of the entire body mechanism2); having healthy upper limbs is a matter not to be ignored.

Usually, patients with an upper limb dysfunction need to be trained repeatedly using appropriate rehabilitation equipment to recover. Clinically some of the most frequently used equipment include exercise skate of the arm, exercise skate of the hand, vertical tower, incline board, stacking cones and cura motion exercises. There are many therapeutic methods such as mechanical arms (passive or positive patient training through mechanical structures)3,4,5,6,7,8), video games (follow the instructions on the screen to move mechanical arms to help neural rehabilitation)9), and virtual reality (integrate and improve sound, video, graphics, and text) to make users feel they are experiencing it for real10,11,12,13,14).

The shoulder joint is frequently used in daily activities, making it prone to sprain and bruising, causing shoulder rotation or abduction disorders. Patients decrease the shoulder joint mobility because of the fear of pain, thereby affecting damaged function reconstruction. Shown in Fig. 1(a) is a traditional shoulder training activity called the shoulder finger ladder, which strengthens and increases the shoulder angle movement using finger movements. Patients face a wall (front and side) and move the fingers of the affected side upward along the ladder to their maximum reach. This training is very effective for patients who have frozen shoulders. Shown in Fig. 1(b) is a traditional training course named the single curved shoulder. The arm of the affected side is used to move a plastic piece from the left to right or from the right to left to train the initiation movement. The main body parts trained are the shoulder, elbow, and forearms.

Fig. 1.

Traditional training activities

Researchers are motivated to introduce virtual reality concepts to traditional rehabilitation training as it increases patient enthusiasm and repeatability. The primary goal of this research was to train the upper limbs. To facilitate participants using this training in daily life, researchers applied the Microsoft Kinect somatosensory devices (for Windows) for the 3D human motion capture system. This system detects human skeleton coordinates such as the palms, wrists, and both shoulders to develop Unity games. Participants can be trained through these games and scenes on the screen without actually having to touch a real entity. This study used mission-oriented training in applying Kinect somatosensory device software development with Unity 3D games to enhance the training effects. This approach is very convenient and safe because participants only need to touch the assigned virtual objects using their upper extremity.

Kurillo et al. argued that the Kinect-based 3D reachable workspace analysis provides sufficiently accurate and reliable results compared to motion capture systems, and that proposed methods could be promising for the clinical evaluation of upper extremity in neurological or musculoskeletal conditions15). Training using video games played on the Xbox Kinect may be an effective intervention for the rehabilitation of stroke patients16). Unity 3D is a low price, powerful, and intuitive game engine applied widely in industry. Even though it can be used to develop games it does not support a somatosensory application. Therefore, scenarios are played through Kinect. Kinect is designed to detect human skeleton information which is the key to developing somatosensory games. The signals are captured and transmitted to Unity 3D through Microsoft SDK (Kinect for Windows SDK) or Open NI (open nature interaction) to drive the game character actions.

SUBJECTS AND METHODS

Figure 2 shows the overall schematic diagram of this study. Human skeletons were displayed in action on a PC using Windows SDK, the Unity 3D software tool, and the Kinect’s sensing device. The game activities were customized by adopting the Unity3D software. Since the Kinect senses human skeleton data, this 3D coordinate data must be projected to the corresponding Unity3D’s virtual scenes on the PC screen. The virtual scenes can be used to construct and plan game scenes or express different design collision effects. The hardware interface used Microsoft’s product and the Kinect hardware for Windows to connect to the computer. The advantage of the Kinect is that its skeleton recognition technology can be used to determine actions while other relevant action information capture technology is captured through the physical installation of many sensing elements and cables. The development interface and application program part is mainly composed of the installation of drivers for the Kinect for Windows SDK and Unity3D software. Common programming language C# between these two was used to write a program, the game application was produced using the compilation and function calls of the Kinect SDK through Unity3D. This program could be used to measure data while the game was running, as well as recorded the 3D coordinates of the skeleton of the participants during the training process.

Fig. 2.

Overall scheme of the rehabilitation system, (a) Computing angle of shoulder joint

To capture human skeleton coordinates via the Kinect device, the angles could be measured by the bones that connect to the joints, and the angle differences in movements can help medical workers understand the accuracy of poses and action changes for patients in the process of training activities. Figure 2 (a) shows the collected data for the shoulder, elbow, and wrist joint coordinates, which were S(sx,sy,sz)·E(ex,ey,ez)·W(wx,wy,wz)

wherein

The elbow angle formula was

Figure 3(a) shows the training activities in the frontal plane − the shoulder finger ladder design used as training for upper extremity lifting or measurement purposes. The bottom is the Reference (Ref), the rectangular areas are reminders for the participants to touch a virtual position (represented by the letters from a–j) in sequence (height) by lifting their upper limbs. The red rectangular area (figure marked as a) is used for participants to touch according to the Ref (the very bottom of the figure) upon completion of the designated touch order (shown as a step). The patient virtually touched the point with their hands. At the same time, the red rectangular area automatically moves on the screen to show the completion of one round as soon as the participants have completed a–j in sequential order (or the individual’s maximum operating limit). The actual orders could be adjusted based on the design needs.

Fig. 3.

Design of the motion trajectory (take right hand as an example)

The following contains the explanation for each “ladder” planned for the shoulder finger ladder. The subject stands in front of the Kinect device at the start with their body captured completely. The captured coordinate data for the head, spine, hip center, elbow, wrist joint were H(hx,hy,hz)·S(sx,sy,sz)·HC(hcx,hcy,hcz)·E(ex,ey,ez)·W(wx,wy,wz).

The Ref position is defined as:

H(hx, hy−*ε, hz−*δ), 0.5 < ε < 0.95, 0.75 < δ < 1

The related positions are defined by the Ref on the starting position (Pos_Min) of the “ladder”:

S(sx,sy,sz−*δ), 0.75 < δ < 1

The ending position (Pos_Min) of the “ladder” is:

HC(hcx,hcy,hcz)−*δ), 0.75 < δ < 1

Adjacent to the “ladder” is the gap (if the area is divided into K):

,, iy=(0,1,0)

Figure 3(b) shows the training activities in the frontal plane − the planned design for the single curved shoulder. The subject sequentially touched the objects virtually from the left side of the hand toward the right, clockwise and then in the counterclockwise direction to the original starting point. This activity trained the shoulder, elbow, and forearm and was very helpful for hand-eye coordination and reaction cultivation.

Patients with frozen shoulders were trained with the shoulder finger ladder and the single curved shoulder training activities combined. The “ladder” point of the shoulder finger ladder activity could be the radius of curvature the single curved shoulder activity which provided better protection for the subject in the safety training process.

This experiment was performed in two parts: one for those with normal shoulder function and the other for those with shoulder disability. The participants used their hands according to the indicator points on the screen to virtually complete the exercise; the computer system recorded the coordinates and performed the statistical analysis, which can be referenced by clinic personnel later on. To confirm the reproducibility and stability of the system development, it was very helpful to understand whether this system was stable enough for clinical training activities for assessing and analyzing the test results from participants of different ages and body types or the same participant with various testing times17, 18).

In the normal shoulder function group, 8 participants conducted 12 rounds of training and testing with the device developed in this study. The collected training data were analyzed to examine the variations before and after the training to determine whether the system could provide consistent evaluations of the usually used hand, and to make sure that the system was stable with high repeatability. In the shoulder disability group, there were 4 participants conducting training for 6 weeks (twice a week), with 3 rounds of practice and 3 minutes of rest between each round. Subjects would complete 2 tests after 1 practice. The ethical committee at the Taipei Medical University Hospital approved the study, and written informed consent was obtained from each participant.

RESULTS

Eight subjects (age range 21–30, 31–40, 41–50, 51–60; 2 subjects in each group) with normal shoulder function were chosen to perform the shoulder finger ladder and the single curved shoulder exercise 12 times (3 rounds in each practice) of the training and testing to check reproducibility and stability.

Figure 4(a) shows the single curved shoulder test screen of the subjects. It shows that the subjects raised their right hand to touch the object (red) on the oval. Figure 4(b) shows the performance of subject A-1 (left handed) in 12 exercises. The left and right-hand average usage time reached stability at approximately up to 33.5sec and 41.2 seconds in the seventh and ninth test, respectively. The total average usage time for the left and right hand was 35.4 (SD = 2.35) and 42.8 (SD = 1.60) seconds, respectively. The results indicate that dominant side movement is more flexible than the non-dominant side.

Fig. 4.

The performance of subject A-1 (left handed) in the Shoulder finger ladder test

Table 1 shows the performance record of 8 (4 age groups) subjects with normal shoulder function. The results indicated that the average usage time for the dominant side of the subjects in the same age group were close to the shoulder finger ladder and single curved shoulder activity; indicating good system stability. The average usage time for both hands indicated that the dominant side movement is more flexible than the weak side. Moreover, the flexibility and ability to respond in elders were slightly inferior to that of the younger groups.

Table 1.

The performance record of 8 subjects with normal shoulder function

Year group	Subject/GenderDominant side	Average using time (sec)
		Shoulder finger ladder		Single curved shoulder
		Left hand	Right hand	Left hand	Right hand
21–30	A-1/Male/Left hand	21.6±0.8	23.9±0.8	35.4±2.4	42.8±2.6
	A-2/Female/Right hand	23.9±0.8	22.4±0.7	43.0±2.3	35.1±2.2
31–40	B-1/Female/Left hand	23.4±0.8	25.1±0.9	38.1±2.2	43.9±2.7
	B-2/Male/Right hand	25.4±0.9	23.1±0.8	44.6±2.8	37.4±2.3
41–50	C-1/Male/Left hand	25.1±0.8	28.6±0.9	40.2±2.6	48.1±2.6
	C-2/Female/Right hand	29.1±0.9	25.3±0.9	50.1±2.8	41.4±2.7
51–60	D-1/Male/Left hand	27.0±1.0	32.9±1.6	45.2±3.3	49.4±2.9
	D-2/Female/Right hand	32.2±1.7	26.9±1.0	51.2±2.9	43.6±3.3

The study included 4 subjects with impaired shoulders who participated in the test; the basic information of these subjects are shown in Table 2. Table 3 displays the result (average using time) of the 6 subjects (P1–P6) during the entire test period of the shoulder finger ladder and the single curved shoulder presented in a bi-week interval. Every subject’s performance showed positive results in the mid-stage, and stability in the post-stage.

Table 2.

Basic information of the 4 subjects with impaired shoulder

Subject	Age	Gender	Dominant side/Affected side	Symptoms
P-1	53	Male	Right / Right	Sports injury
P-2	48	Male	Right / Right	Frozen shoulder
P-3	55	Female	Left / Left	Frozen shoulder
P-4	67	Male	Right / Left	Traffic accident
P-5	46	Female	Right / Right	Vocational injury
P-6	63	Male	Left / Right	Traffic accident

Table 3.

Result of the “shoulder finger ladder” and “single curved shoulder” test on the affect side

Type	Subject	Average using time (sec)
		wk 1–2	wk 3–4	wk 5–6
Shoulder finger ladder	P-1	32.9±1.9	28.4±1.1	27.5±0.1
	P-2	35.7±2.0	32.1±1.4	31.3±0.4
	P-3	36.0±1.9	32.6±1.3	31.6±0.3
	P-4	39.8±2.4	36.5±2.0	35.0±1.0
	P-5	33.1±2.3	31.9±1.9	31.0±0.2
	P-6	39.2±2.8	36.2±2.9	35.1±0.9
Single curved shoulder	P-1	66.5±19.5	48.9±7.6	47.5±0.1
	P-2	68.9±18.5	53.8±12.3	52.9±1.3
	P-3	69.2±17.2	54.0±10.3	53.8±1.0
	P-4	78.7±18.7	68.6±9.9	67.4±1.3
	P-5	67.6±19.8	51.9±11.1	51.2±1.1
	P-6	77.1±17.6	67.1±9.1	66.3±1.2

Value are expressed as mean ± SD

To assess the average performance time of subjects in the 6 week test trial, researchers set the benchmark as the average score of tests given 4 times biweekly. Researchers performed pairwise statistical tests; see Table 4 for the results. The first and second weeks were adjusted for the stage for all subjects, however, the performance was not ideal. There was a significant improvement in the third-fourth and the fifth-sixth weeks. The t-test results comparing 1–2 weeks to 3–4 week and 1–2 weeks compared to 5–6 weeks showed a significant difference. The average performance reached stability after the third week, and there was no significant difference in the t-test results at 3–4 weeks and 5–6 weeks. The results indicated that this system was effective in the training of each subject.

Table 4.

Result of the statistical test (pairwise)

Type	Subject	Between the weeks
		wk 1–2 / wk 3–4	wk 3–4 / wk 5–6	wk 1–2 / wk 5–6
Shoulder finger ladder	P-1	32.9/28.4*	28.4/27.5	32.9/27.5*
	P-2	35.7/32.1*	32.1/31.3	35.7/31.3*
	P-3	36.0/32.6*	32.6/31.6	36.0/31.6*
	P-4	39.8/36.5*	36.5/35.0	39.8/35.0*
	P-5	33.1/31.9*	31.9/31.0	33.1/31.0*
	P-6	39.2/36.2*	36.2/35.1	39.2/35.1*
Single curved shoulder	P-1	66.5/48.9*	48.9/47.5	66.5/47.5*
	P-2	68.9/53.8*	53.8/52.9	68.9/52.9*
	P-3	69.2/54.0*	54.0/53.8	69.2/53.7*
	P-4	78.7/68.6*	68.6/67.4	78.7/67.4*
	P-5	67.6/51.9*	51.9/51.2	67.6/51.2*
	P-6	77.1/67.1*	67.1/66.3	77.1/66.3*

*paired-t test, p<0.05

DISCUSSION

The shoulder joint has the largest range of motion, the most complicated action form and is the most frequently used joint in physical activities, resulting in a higher injury frequency. People should maintain shoulder range of motion in their daily lives. Chronic degradation can occur if people do not take care of their shoulders. This study used the Kinect somatosensory device with Unity software to develop 3D situational games for upper extremity training activities. Using games as training methods helps improve concentration, patient interest in participation, and helps patients temporarily forget about their body discomfort. The equipment used in this study is inexpensive, easy to obtain and the system is easy to install. People can perform simple self-training at home or in the office. Our group will continue to recruit more cases with impaired upper limb function to conduct related research and develop suitable rehabilitation training games. To have more effective methods, researchers have introduced games and virtual reality training to help participants train their upper limbs in a relaxed environment.