Creation of virtual reality stimulus for a psychophysiological body balance measurement.

Virtual reality methods and equipment can be used to create stimulations for several psychophysiological measurements. Such stimulations can be flexibly modified and their versatility is wide; in principle even such could be prepared that are not possible in the real physical world like virtual flying in the space. For the generation of virtual reality stimulations we constructed a three-dimensional (3D) graphics engine to measure body balance variations in response to emotional stimulation. The stimulation was implemented as a virtual reality scene, where the subject watched a street view with a human figure approaching the subject at a constant speed. The functioning of the system was tested by measuring body balance variations while the approaching figure's facial expression was angry, neutral, or happy. The results showed that our young, healthy subjects were slightly, but still statistically significantly affected by the stimulation of the approaching figure. Thus, the whole system offers a tool for controlled experimental studies for analyzing body balance, for example, during stimulating approach-withdrawal behavior. There are several opportunities to further develop new visual psychophysiological stimulations and test forms.

Introduction

The investigation of human body balance is important for several reasons. For example, elderly people tend to lose some of their ability to automatically and rapidly correct unexpected changes in their body balance. From the psychological perspective changes in body balance are related to changes in attention, orientation, and emotional response. Carefully controlled experimental research on body balance with respect to for example, the above-mentioned phenomena requires that we have a good tool for both stimulation and measurement. Our goal was to construct a novel technique to form various suitable stimulations on the basis of virtual reality methods for research areas like these psychophysiological measurements.

From the research of human emotions it is widely known that emotional stimulation evokes significant changes in several levels of human behavior, that is, from the neurophysiological level to the behavioral level.1–8 For the behavioral level one significant argument has been that during emotional responding, depending on the quality of the response, there are significant changes in approach-withdrawal behavior. Although there may not always be visibly observable changes in this level (eg, flight or fight) it is likely that there are changes in the tendency to behave.9 Lang and associates have proposed a model in which emotions are regarded as action dispositions organized along two dimensions; affective valence (ie, from negative to positive dimension) and arousal (ie, from calm to highly aroused dimension).10–12 The first dimension reflects the presence of appetitive and aversive motive systems controlling the general direction of behavior (ie, approach-withdrawal). The second dimension reflects the intensity of either appetitive or aversive response systems. Thus, the valence dimension is seen as an index of motivational priming for appetitive or aversive behaviors and the arousal dimension as an index of the mobilization of attentive and metabolic resources.5

For the above reasons we wanted to utilize the idea of using emotional stimulations for virtual stimulations. From earlier research it is known that stimuli of facial emotional expressions (even subliminally presented) have frequently been found to evoke, for example, specific brain activity and changes in the activity of facial muscles that are related to facial emotional expressions.5,7 Because it is likely that facial expression stimuli are able to trigger emotional stimuli, we used three different types of facial expressions, angry, neutral, and happy. The stimuli were chosen from the widely known set of pictures of facial affect.13 The three stimuli (ie, angry, neutral, and happy faces) were all expressions from the same male actor. The expressions had been earlier judged to reflect the above mentioned facial messages well.10 See also a study on the sensitivity to the proportions of faces.14

In the second section we describe our research goals, present the design of the virtual reality stimulation and its implementation, and mention the software methods and tools used. The third section of this article deals with the structure of the three-dimensional (3D) engine implemented and the virtual reality equipment employed. The fourth section describes the virtual reality stimulation created and its use for balance measurements. In the fifth section we present balance measurement results and their computation. The discussion completes the article.

Objectives, design, and software methods and tools used

Virtual reality methods are used in numerous applications, eg, therapy and training.15,16 With virtual reality methods we were able to generate a wide range of stimulations and to vary their test parameters to explore subjects’ responses and reactions in various circumstances. The virtual reality world generated was watched using a head-mounted display (HMD; virtual reality helmet). Responses were measured by means of a force platform, which sensitively recorded a subject’s swaying. From the quality of swaying, or more precisely, from its measured sensor signals, we determined the influence of stimulations in the course of time.

We designed and implemented a system for psychological approach-withdrawal experiments, in which the “manuscript” is as follows: in a street of an open city setting an unknown virtual youngish male runs towards the spectator or subject. The speed of the running and countenance of the virtual runner can be changed. The virtual runner approaches the spectator, but just before a collision or treading on the spectator’s foot he dodges either to the right or left and vanishes around a street corner. With the rapid approach of the virtual runner as the main effect in the virtual environment, we were interested in investigating how a subject reacted, in the sense of his or her balance, while standing on the force platform, and what the possible impacts of the countenance of the runner were on a subject’s responses.

We prepared a 3D graphics engine and briefly describe its structure at high level. In the construction of the 3D engine, we discarded such advanced details as dynamic object occlusion culling that is applied to optimize the computation speed in graphics. We also left out audio effects and physical simulations such as collision response, because a collision was disallowed in our tests. An essential component of the system was also the interface using which a user of the system could input virtual reality parameters, ie, the properties of the virtual runner given. Balance recordings and analyses were performed using a standalone computer. We briefly describe how different techniques of computer graphics were applied to the present virtual reality system, a little like in, in order to offer a reader an opportunity to understand the whole of the system and the goals of its use.17

OpenGL was used to interface with the hardware. We applied a color buffer as double buffering, where an image is not directly drawn on the screen, but first on the back buffer in memory. After the completion of the image it is drawn in the display memory. A depth buffer was also used to maintain information about every data point to express how far away it is from the spectator. Texture mappings were executed using suitable surfaces present in a city milieu and also sky. Sky was simulated using a dome-shaped object with a texture of clouds. Walls, windows, doors, stairs and roofs of buildings, a pavement and a few street-lamps were drawn in the street environment. The city landscape was modeled using Autodesk 3D Studio MAX and exported through customized tools.18

Perhaps the most common view direction also applied by us is the “first person”: a view of the virtual reality scene seen through the spectator’s eyes. Provided that the spectator can freely move around the virtual world, this view direction means that almost all possible viewpoints over the virtual reality scene are taken into account. It is known that shadows improve the realism of a computer generated image and provide the user with important visual cues about object placement. We implemented a shadow algorithm based on shadow volumes described in its basic form for the first time in the references.19,20

Translation, scaling, and rotation of mere rigid bodies are insufficient for the animation of virtual reality models. We animated objects by applying the matrix bone skinning technique, in which the key idea is to define several affine transformations which affect a single model.21 The matrix bone skinning technique is advantageous compared to another option, the keyframe animation, since it requires less storage space and provides more flexibility.22 Our system needed a moving virtual runner, ie, a moving object, whose track was interpolated point by point along different curves.23 To speed up the rendering of virtual reality scenes, it is possible to apply space subdivision techniques to static geometry data for object representation. We employed the binary space partitioning (BSP) technique.24–26

We wanted to test whether the approach-withdrawal phenomenon can be achieved by stimulating subjects with the virtual reality system. At the beginning there were the following issues: what kind of stimulations and how to construct them. Another interest was focused on emotions aroused by various face images. Responses were measured analyzing body balance.

The structure of the 3D engine and virtual reality equipment

For 3D models an ultimate scene consists of geometry data and textures on the basis of arithmetic operations. A great part of the geometry data of the virtual reality world is probably unseen with regard to the ultimate scene. Therefore, it is feasible to prune invisible objects as much as possible from the geometry data at the earliest possible phase, which was implemented as a culling procedure in the graphics pipeline. The virtual reality world incorporates two different types of geometry: static and dynamic. For example, the soil and buildings are static, but any object which it is possible to move related to the static geometry can be seen as dynamic. In our system, there was the virtual running person. To optimize running times, we split the world into the two halves as in Figure 1(a).

Figure 1

Since most objects of an image are invisible, they can be culled away. (a) The culling of objects of static geometry is faster than of those in dynamic geometry. (b) Finally the data flow with the specific techniques applied generates the image seen by the test subjects.

The virtual reality world may include more objects than visible in a scene. To crop unnecessary objects from a scene, we used the BSP technique for the static geometry, which corresponds to the leftmost part of Figure 1(a). All necessary methods described in the introduction section were applied to build and use the 3D engine. Finally the data flow with specific techniques applied presented in Figure 1(b) generates the image seen by the test subjects.

The virtual reality system constructed was run in an AMD Athlon based system with a processor of 1.73 GHz, memory of 512 Mbytes and a 3D graphics accelerator of nVidia GeForce 4600. Our virtual reality helmet with stereo displays was the V8 HMD by Virtual Research that displays with a resolution of 640 × 480 pixels at a visual (diagonal) angle of 60° and refreshes the display at a frequency of at least 40 Hz, which was sufficiently high in order to yield animations for a rapidly moving target as the virtual runner without any delay or discontinuity. The view angle depends on rotations of the helmet and these were measured with the sensors of Intersense IS-300. Interpupillary distance (IPD), 5.8 cm on average, was taken into account by setting the HMD suitably and by adjusting IPD for each subject.

Virtual reality stimulation and balance measurement

Next we introduce the virtual reality stimulation implemented for the system. It is self-evident that the vision affects a human being’s balance. In fact, the older a person is, the greater the influence of the vision. Effects caused in a virtual reality environment also affect the balance.27–29 Our objective was to research effects of the approach-withdrawal reaction and emotions in the virtual reality environment.

All measurements were performed in our virtual reality laboratory designed for physiological recordings to acquire responses from subjects to virtual reality stimulations presented to them. The laboratory contains two consecutive rooms: one for measuring a subject and the other for micro-computers and a laboratory engineer. The former room was insulated against exterior noise, electrical noise and trembling of the building or ground. The laboratory was on the ground floor and the test room is like “a floating box” entirely disconnected from the walls and ceiling of the larger surrounding room. By using efficient mechanical insulation underneath its floor, outside vibrations were eliminated. The test room was throughout all its components painted with matte black to prevent even the slightest reflections of light.

We investigated the effects of the virtual reality stimulation on 20 healthy subjects, who were mainly students or from our research staff, but unacquainted with the system before their tests. Four of them were females and 16 were males, ages range from 23 to 34 years, but mainly between 23 and 28 years. Everyone first became acquainted with the balance test generally, but neither went through it nor especially looked at the stimulation before the test.

A virtual city environment was prepared by applying realistic textures extracted from actual photographed objects. Stereographic images and dynamic shadows were also used. As mentioned, we had a script, where a virtual running man approaches the viewer, ie, the subject in the street. The virtual runner passes the subject just before a seemingly inevitable collision either on the left or right side. The emotions of the modeled runner were simulated by varying his expression in one of three ways: happy, angry or neutral. The three expressions were made by changing the facial texture of the 3D model. They originated from photographs classified into different emotions. The face of the runner was also modeled to correspond to the faces in the photographs; for each expression only one photograph was used and it was taken directly in front of the person. Since these classified photographs were black-and-white pictures, we first colored them artificially with graphics software tools. Thus, we had two adjustable parameters: expression and passing side of the virtual runner. Actually, his speed could also be adjusted or the route varied or in principle several other properties, but all those were fixed to avoid very complicated test situations, for which balance effects would have been difficult to analyze and decode.

We tested the impact of the virtual stimulation in three sections, each consisting of a series of 12 s stimulations repeated 10 times in random order, ie, 360 s in all. The stimulations were varied in accordance with the expressions and passing sides. The whole series was performed for a subject in random order so that, for instance, the “angry expression” appeared in different positions for different subjects; the aim was to ensure that the possible effect of an expression would not depend on its position in the test series. Figure 2 presents the environment, the approach of the runner and three expressions.

Figure 2

The expressions of the approaching virtual runner: happy, neutral and angry. (The parts of the faces in the image are copyright © 2008. Reprinted with the permission of Paul Ekman).

The stimulation was presented to a spectator via the stereo displays of the virtual reality helmet worn by him or her. The reactions of a spectator’s balance were recorded using a force platform (Figure 3).27–29 The spectator stood on the platform, relaxed but alert, arms akimbo, and was told not to take any steps. Of course, the laboratory personnel controlled every experiment in order to stop a possible fall. No one fell because all subjects were young and healthy. The force platform measures by means of three sensors below the platform, balance as movements of average pressure (center of pressure, COP) and the vertical acceleration (approximately mg) caused by a subject’s mass m and the gravity g. When a subject is swaying, the location of the center of pressure is correspondingly moving in the plane as shown in Figure 4 and further as one-dimensional signals in Figure 5.

Figure 3

A subject stands on the force platform wearing the stereo display in the virtual reality helmet.

Figure 4

A stabilogram (generated by balance signals), in which swaying of the center of pressure is presented for 12 s in cm in the left-right direction (horizontal axis) and forward-backward direction (vertical axis).

Figure 5

A balance signal exemplar as a function of time: medio-lateral or left-right (M-L) and antero-posterior or forward-backward (A-P) during stimulation. Center of pressure (COP) in both directions and mass are measured. Naturally, nobody’s mass itself changes, but swaying affects its recording and the spikes in the mass signal are formed by heartbeats, breathing and muscle tension that change the pressure against the sensitive force platform on which a subject stands without shoes (see Figure 4).

After recording at a sampling frequency of 50 Hz, every signal was filtered with a recursive second-order Butterworth filter that was run back and forth to produce zero-phase lowpass filtering with a cut-off frequency of 10 Hz. The 50 Hz sampling frequency is typical and high enough to produce accurate information about a subject’s swaying, although sampling frequencies as low as 20 Hz have also been applied.30 The force platform was attached via an analog–digital converter (Data Translation 9800) to the stand-alone microcomputer; there were two computers, one for the creation of the virtual reality world and the other for the balance recording and analysis. It would have been an interesting and useful idea to link head positioning data with positional data from the force platform. Unfortunately, this was not possible, since precise synchronization control between the two devices was not available. Besides, we wanted to minimize any complicated synchronization problems.

Computation of results

It is essential to measure swaying left-right and forward-backward. Let X (t) be a random process in the course of time t and X (0) its starting location so that we have Y (t) = X (t)−X (0). Assuming X (t) to have a normal distribution for every t and tests to be independent, we can test whether X (t) differs from its start location according to the t test at time t by using samples extracted from Y.

Let {x1, …, x} be a random set of samples measured at a sampling frequency of 50 Hz. As usual, on the basis of the sample mean and variance and degree of freedom N−1 we can compute that, with probability α, average μ will be included in interval

Further, the t distribution is symmetric around its center t0.5 giving a confidence interval between points

Provided that Y (t) is normally distributed we can estimate its mean in the above way, since at any moment there are several hundred samples available. We may criticize the other assumption that the samples are independent, but, after all, there are at least 12 s between the samples of a certain observation moment and a subject’s balance is quite a random property. Moreover, there were 20 subjects, which averaged small intersubject variations.

Figure 6 shows the mean of forward-backward swaying of all the subjects calculated from the start location. At time 4.5–8.0 s they statistically significantly leant forward, ie, during that period in Figure 6 the subjects produced a clear response by leaning forward when the virtual runner had approached and almost reached them. This reflected a precaution for the seemingly probable, immediate collision against the virtual runner, since the movement of the center of pressure aids the start of a rapid evasive movement, but its maintenance requires more muscular work. The change in swaying was fairly small, circa 1 mm, and the average was maximally 2 mm forward from the position of no swaying (as if staying directly).

Figure 6

Movement (the left vertical axis in cm) in the forward-backward direction: average of all tests as continuous curve, 95% confidence limits as dashed curves and distance (from the spectator) of the virtual runner as dotted broken line (the right vertical axis in m).

Figure 7 presents the average results of the first test series in the forward-backward direction. Precaution at time 4.5–8.0 s is smaller than in the case of all series, which shows that the precaution response was damped after repetitions of experiments. Swaying in the left-right direction was of quite similar type, but its quantity was less, at most circa 0.5 mm.

Figure 7

Movement (the left vertical axis in cm) in the forward-backward direction: average of all first test series as continuous curve, 95% confidence limits as dashed curves and distance of the virtual runner as dotted broken line (the right vertical axis in m).

Ultimately, Figures 8 and 9 include results when the virtual runner passed the subject either on the left or right side. A small, but statistically significant evasive action appears in both figures. The variation of the expressions of the runner only slightly affected the balance.

Figure 8

Average movement and confidence limits in the left-right direction and distance of the virtual runner as dotted broken line (the right vertical axis in m), when the runner passed the subject on the left side.

Figure 9

Average movement and confidence limits in the left-right direction and distance of the virtual runner as dotted broken line (the right vertical axis in m), when the runner passed the subject on the right side.

In addition, we performed ANOVA tests for the signals recorded. The following models, in which t is time, e is emotion, and p is pass (of the virtual runner), were utilized:

Mean signals M (with error term) were compared according to different conditions: if the direction of pass (of the virtual runner) is taken into consideration, two signals are estimated, if emotion is taken, three signals are estimated, and if both are taken, two times three signals are used. The recordings were selected along with each variable combination and a mean signal was averaged from these.

We tested how a variable affects in the predictability of a model. A significant P value means that a current variable was useful; the opposite situation means that it does not significantly affect the predictability property of a model. Table 1 presents the results. The expression p|t, e shows whether p improves a model which also includes t and e. Values F and P in the first row of each model in Table 1 test the usefulness of the whole model, in other words whether the model predicts anything. They cannot be compared to the results of the other rows that test the usefulness of a variable inside a model. Degrees of freedom take into consideration how many parameters the deletion of a variable removes. For instance, emotion has 3 times 600 parameters, the direction of pass 2 times 600, and their combination 2 times 3 times 600.

Table 1

ANOVA results showing the test outcome (F) and significance (P) of averaged balance signals: modeled according to variables time (t), emotion (e) and pass of the virtual runner (p)

Model	Condition	Degree of freedom	F	P
Left-right, full	t, p, e	3600	1.93	0.00000
	e|t, p	2400	1.40	0.00000
	p|t, e	1800	1.60	0.00000
	e, p|t	3000	1.50	0.00000
Left-right, pass	t, p	1200	2.98	0.00000
	p|t	600	1.89	0.00000
Left-right, emotion	t, e	1800	2.26	0.00000
	e|t	1200	1.35	0.00000
Left-right, time	t	600	4.07	0.00000
Forward-backward, full	t, p, e	3600	1.28	0.00000
	e|t, p	2400	0.90	0.99969
	p|t, e	1800	0.51	1.00000
	e, p|t	3000	0.81	1.00000
Forward-backward, pass	t, p	1200	2.02	0.00000
	p|t	600	0.45	1.00000
Forward-backward, emotion	t, e	1800	2.05	0.00000
	e|t	1200	1.27	0.00000
Forward-backward, time	t	600	3.59	0.00000

To conclude, the prediction of the left-right direction takes advantage of all variables, ie both emotion and pass affect. The prediction of forward-backward swaying benefits significantly only from emotion. Nevertheless, if the direction of pass is also noted, it cannot be asserted that emotion notably improves model predictability.

Discussion

It was possible to build serviceable virtual stimulations to affect a subject’s balance using purely visual stimulations. We observed that suitable virtual reality stimulation affected a subject’s balance. Instead, possible emotional affections remained somewhat open. There are several possible reasons for this. To mention some first, maybe body balance does not change as a function of facial expressions. It may also be the case that while modeling the facial expressions in 3D format the clarity of the original facial expressions was compromised resulting in a case that they could not affect robustly enough. Finally, it is also possible that our preliminary way of analyzing the changes in the body balance was not suitable for tracking the perhaps very rapid contradictory changes in body movements. It may be that the dynamics of first forward and backward leans are corrected dynamically in the way the changes were summarized out.

Perhaps by introducing additional measurements in the future, we may be able to find evidence on the emotional influence. We may insert, for instance, electromyographic measurements of facial muscles or allow a subject to make gestures with his or her hands and arms, which was not allowed in the present experiments, but which would be recorded with sensors functioning in magnetic field. Allowing hand and arm gestures standing on the force platform would be more normal than our present tests. However, such a measurement would naturally be more difficult to analyze.

We found that the stereo display of the virtual reality helmet was crucial, since by subjectively assessing it created a fairly natural view into the virtual reality world and facilitated the stereoscopic effect which is the prerequisite for the effective immersion.15 Also, by taking head movements into consideration to show the changing virtual reality world around a subject was essential, which would not be possible without the appropriate virtual reality helmet (including motion sensors). Obviously, the good test laboratory with effective noise, trembling and electrical insulation was a very useful aid in producing precise and reliable recordings. The good test conditions also improved the immersion of a subject into the virtual reality world.

Because it was possible to achieve positive responses from the subjects in the present balance tests stimulated with virtual reality, it may be possible to expand measurements to include other physiological signals than those mentioned above. These could then be more widely applied in psychological experiments, perhaps even as therapeutic and evaluation tools, for example in rehabilitation for claustrophobia or other similar phobias.15,31