Augmented rotations in virtual reality for users with a reduced range of head movement.

INTRODUCTION: A large body of research in the field of virtual reality is focused on making user interfaces more natural and intuitive by leveraging natural body movements to explore a virtual environment. For example, head-tracked user interfaces allow users to naturally look around a virtual space by moving their head. However, such approaches may not be appropriate for users with temporary or permanent limitations of their head movement.
METHODS: In this paper, we present techniques that allow these users to get virtual benefits from a reduced range of physical movements. Specifically, we describe two techniques that augment virtual rotations relative to physical movement thresholds.
RESULTS: We describe how each of the two techniques can be implemented with either a head tracker or an eye tracker, e.g. in cases when no physical head rotations are possible.
CONCLUSIONS: We discuss their differences and limitations and we provide guidelines for the practical use of such augmented user interfaces.

Introduction

Research in the domain of virtual reality (VR) has recently reached the state where inexpensive devices, such as head-mounted displays (HMDs) with associated rendering and tracking technologies, have become available to a wide range of users through consumer outlets. Products in this field are trying to leverage the promise of VR to immerse and engage users in a virtual experience with more natural forms of interaction than possible with other types of displays and forms of input. Typically, the user’s head movements are tracked and mapped to that of a virtual camera such that they can look around the virtual world with a first-person perspective. Other forms of input include hand-held devices such as gamepads or joysticks to induce push-button rotations or translations which are widely used in many fields. In this paper, we present techniques that allow these users to get full-movement benefits from a reduced range of physical movements. Although our techniques could be applied to rotations specified via hand-based input devices, previous research has shown that physical head movements can result in an increased sense of presence[1-3] and improved perception and cognition.[4] As such, in this paper we focus primarily on head and eye movements.

The field of VR has made steady advances over the last several decades toward becoming a useful and even ubiquitous technology for society. Developments of VR consumer products are accelerating these advances, driven to a large degree by the field of home entertainment. The vast majority of products are mainly developed by and for non-disabled persons without any limitations in motor behavior. In particular, in the field of natural user interfaces (NUIs), the possibility that some users may not be able to perform such “natural” interactions at all or not to the same degree as a non-disabled person is mostly ignored. For instance, a non-disabled consumer playing a VR game while seated on a swivel chair at home may have a full 360-degree comfortable range of interaction by rotating their head and/or body or chair, but an individual seated in a wheelchair may not be able to rotate their head to the same degree, e.g. due to motor limitations, the wheelchair’s protective head cushion, or other peripheral equipment, which makes such head movements impossible or strenuous.[5] While people with limited or no head motion are unable to use such VR systems “out of the box,” one can substitute an alternate mapping of input to virtual view control. And while there are different kinds of assistive technologies such as foot pedals, mouth sticks, oversized trackball mouse, and many more,[6,7] in this paper we only focus on using head and eye motion as input.

We present and discuss two techniques that are aimed at providing users with the ability to rotate a full 360 degrees in a virtual world, even though the range of their physical head rotations might be limited in the real world. Scenarios where no physical head rotations can be performed are also in the scope of this paper, as long as the individual has control over a range of eye movements. We approach movement limitations systematically, presenting a framework that can be used to address low or high mobility of head or eye tracking in VR, and two techniques that can be realized with NUI tracking data. The proposed augmentation techniques are based on adaptive mappings that use (a) continuous rotations of the virtual world based on physical head/eye orientations or (b) discrete rotations that are triggered by physical head/eye orientations. Different types of eye movements, such as saccades and smooth pursuit, have varying characteristics that could induce disorienting rotations. To avoid this, developers can introduce a fixation time for gaze points that are intended to trigger the techniques. Blink patterns can also be used as triggers for the proposed techniques.

We make the following contributions:

We present two techniques to augment the range of virtual orientations for 360-degree interaction in VR that can be tuned to each individual’s range of comfortable head/eye rotations.

We analytically discuss the techniques and provide guidelines for the practical use of these techniques and the use of head/eye tracking data.

This paper is structured as follows. First, we discuss the related work. Afterward, two techniques are described for augmented natural rotations. Then, the techniques and their interplay are discussed in the discussion section followed by the last section that concludes the paper.

Related work

Many researchers have explored using VR for persons with disabilities for diagnosis, training, and rehabilitation. Kuhlen and Dohle described several of these computer systems that were developed alongside assistive technologies suited for different groups of persons with disabilities.[8] As an example, people who are paralyzed can perform tasks in a virtual environment (VE) that were not possible in the real world using assistive technologies that are capable of responding to minimum human output, which in this case was using a device that connects human bio-signals directly to a computer.[9,10] When studying the use of VR for persons with disabilities, Ford focused on benefits that might arise using VR with four strategies.[11] In one of the strategies, he introduced what he called shifts in valued functions, explaining how a paralyzed person’s interactions in a VE is not affected by them being paralyzed and not able to move, reducing the sense of being alienated. Promising feedback of using VR for disabled persons led us down the path of looking into the techniques that inspired us to create a more realistic virtual experience for a user with limited head and/or eye movements, specifically being able to have a full view of the VE.

One of these methods, mentioned earlier as Continuous Rotations was first introduced by Razzaque et al.[12] When proposing their method, one of their goals was to eliminate the use of controllers whilst interacting in a VE, and since they used a three-walled CAVE-like (Cave Automatic Virtual Environment) immersive projection system for their experiment, they needed to come up with a reorientation technique so the user would not turn toward the non-existing back wall. In their method, a user’s rotation in either direction would be followed up by an environment rotation in the opposite direction, with an amount determined based on factors such as the user’s angular velocity.

Another technique to accomplish the goal of making the VE more accessible to the user is Amplifying Rotations proposed by LaViola et al.,[13] which makes viewing a full 360-degree environment possible without using controllers. In their proposed method, since the users were able to move, they experimented with user’s head, torso, and waist orientation as input for the amplified rotation. They tested the method with both linear and nonlinear functions to map the input orientation to an amplified orientation. Later, Steinicke et al.[14] formalized amplified rotations with Rotation Gains depending only on head movements. Sargunam et al.[15] ran experiments using different reorientation techniques and tested factors such as spatial orientation and preference of use. They also came up with their own method of reorientation called Guided Head Rotations which has similarities with Razzaque et al.’s method and tries to realign the user’s head orientation to a predefined forward orientation taking into account the proximity of the user to virtual objects, and the rate at which the realignment happens.

In a study mostly focused on the effects of amplified rotations, Ragan et al.[16] ran experiments considering impacts of amplified rotation while doing a search task with the goal of exploring a 360-degree VE. While using an HMD or six-sided CAVE, one has access to the 360-degree VE through constant rotations and movements inside that environment, which might not be ideal for many circumstances. In their experiments, they tested different amplification factors and considered effects on a user’s spatial orientation and travel, search performance and simulator sickness. Their results validated the use of amplified rotations to view 360-degree VEs, noting that participants easily picked up on the new way of interacting with the environment.

As mentioned above, mapping head rotation to the rotation of the virtual camera inside the VE is not the only way of achieving a 360-degree view of the VE. There are certain types of medical conditions that greatly limit any sort of neck movement either permanently or temporarily, such as cervical cord dislocation, spinal cord injury, neck fracture, rheumatoid arthritis and many more. This gives rise to the need for another form of input as opposed to head rotations to control the view in the VE. For example, one might use bio-signals[10] or eye movements as the input controlling the full view of the virtual world. Several research groups are focused on developing new systems that are controlled by a user’s gaze for purposes such as controlling the movements of a wheelchair or general interaction with a computer.[17-20]

Augmentation techniques

In this section, we describe two techniques that can enable users with a restricted range of physical head and/or eye rotation to view a 360-degree VE while wearing an HMD. We first describe the setup and underlying coordinate systems before we describe the techniques in detail.

Setup and coordinate systems

Our setup consists of a commercial off-the-shelf HMD (an HTC VIVE) with integrated eye tracking capabilities provided by a Pupil Labs add-on system. The HMD has a resolution of 2160 × 1200 pixels (1080 × 1200 per eye) and a refresh rate of 90 Hz, with a nominal field of view of around 110 degrees, and a weight of around 470 g. Positional and rotational tracking is done by either 1 or 2 Lighthouse units delivered with the HTC VIVE. The head tracking data are available in real-time to an application implemented in the Unity3D graphics engine using the SteamVR 2018 plugin. The HTC VIVE also comes with hand-held controllers that can serve as an input device if needed. The eye tracker is integrated into the HTC VIVE using Pupil Labs’ official add-on for this HMD. It has 120 Hz eye cameras and a gaze accuracy of about one degree and a gaze precision of about 0.08 degrees at a latency of 5.7 ms (per the manufacturer). The HMD and the eye tracker are tethered to a graphics workstation with an Intel Xeon 2.4 GHz processor comprising 16 cores, 32 GB of main memory, and two Nvidia GeForce GTX 980 Ti graphics cards. Additionally, the eye tracking data are sent from the manufacturer’s Pupil Capture software to the Unity3D application via UDP using the Pupil Remote plugin and its Unity3D plugin.[21] Figure 1 shows an illustration of the considered setup and coordinate systems. It should be noted that while there are applications that make use of rotations about the pitch and roll axes, most use cases rely mainly on the yaw rotation, so we focus our descriptions on this axis. Pitch and roll rotations can be implemented using analogous approaches if desired. The techniques being proposed in this section can be applied to any VE, and a user’s range of head and eye motions can be collected both manually and automatically depending on the preference of the practitioner. The baseline range for head yaw rotations is 137.9 ± 13.6 degrees on either direction[5] and the baseline eye movement ranges for adduction and abduction are assumed to be 44.4 ± 6.9 and 44.8 ± 5.5 degrees, respectively.[22] These values should be adjusted for each individual.

Figure 1.

(a) Illustration of the head-centered coordinate system with yaw, pitch, and roll rotations in the considered VR setup. (b) Inside view of the HTC VIVE with the integrated Pupil Labs eye tracker consisting of infrared (IR) LEDs and binocular IR cameras.

Continuous Rotation technique

With this technique, the user can naturally look around the virtual world by rotating their head and/or eyes. However, since the range of physical head and/or eye rotations is limited, the technique has to provide a means to rotate the virtual view beyond the physical limitations. The Continuous Rotation technique does this by rotating the virtual view each rendering frame by an amount proportional to the angular difference between the user’s current head/eye orientation and the center orientation, i.e. when the user’s head/eyes are facing straight ahead. Hence, when the user rotates their head/eyes to look at an object toward the left, the view will slowly rotate the virtual world around the user’s head until that object is located in front of them. To enable the user to explore their restricted field of view without triggering the continuous rotation, we can set a threshold angle that prevents the view to rotate if the center orientation is not exceeded by a specific amount (onset-threshold). An example with head movements using this technique is illustrated in Figure 2.

Figure 2.

Illustration of the head-tracked Continuous Rotation technique in an example sequence of head movements. (a) User is facing straight ahead. (b) User rotates their heads towards the object on the left. (c) The virtual view is rotated a few degrees to the right each second. (d) After a few seconds, the view will stop rotating with object in front. The virtual environment is illustrated by the green checkerboard. Once the user’s head rotates to the left or right, the virtual view is rotated slowly in the opposite direction. If the user is facing toward a virtual object, the user will compensate for the subtle virtual rotation with physical head rotations until the object ends up directly in front of them and stops moving. The user can repeat this as often as desired. The eye-tracking based implementation uses the same approach except that the angles are taken from the eye tracker and not from the head tracker.

Head-tracked implementation

The first background rotation technique was introduced by Razzaque et al.,[12] where they effectively rotated a user’s viewpoint in a virtual world by a few degrees each frame of the rendering system. They argued that users may detect and respond to the continuously updated virtual camera orientation without recognizing the changes as external in origin, compensating for it by rotating their head if the magnitude is below the human susceptibility to such movements.[23] They applied the baseline rotation rate r = 0.145 degrees per second, but our impression, based on pilot testing and related work,[24] is that even rotation rates of up to r = 1 degree per second are not considered as noticeably disorienting or distracting by users.

In order to account for usability variations, we can further multiply a dynamic scaling factor to this continuous baseline rotation, such that the maximum rotation rate is only applied when needed most, i.e. when the user’s head is rotated far to the left or to the right. We therefore assume that the user’s tracked head yaw orientation is in a range defined by two thresholds τ, degrees around a predefined origin at degrees, which corresponds to the user’s head facing straight ahead. We assume that the two thresholds are predetermined for each individual to match their maximum comfortable head rotation range. Rotations beyond those thresholds should be avoided.

Using this approach, the velocity of the baseline rotation is scaled proportionally to the difference in yaw angle between the current head orientation and the orientation that corresponds to the user’s initial forward orientation. The gain factors for the left and right side: and , respectively, are calculated by computing the sine of the difference in yaw orientations up to the threshold, and applied in cases where and for the left and right, respectively. The resulting rotation rate is then computed each frame based on the updated scaling factor and maximum rotation rate in degrees per second, where . As stated before, we found that a maximum rotation rate of r = 1 provides useful results in most cases, but this value can be tuned for each individual based on their susceptibility to such baseline rotations.

Thus, when the user rotates their head to the right, the virtual camera rotates to the left; and when the user rotates their head to the left, the virtual camera rotates to the right. The scaling factor causes the continuous rotation to be stronger when the user is facing farther toward either side and zero when the user is facing straightforward. This is considered a rate control system.[25]

Eye-tracked implementation

We can implement a similar Continuous Rotation technique using eye movements instead of head rotations. Therefore, we denote the tracked gaze yaw angle as and define thresholds τ, degrees accordingly, just as in the head-tracked implementation in the previous section. The two thresholds are then predetermined for each individual to match their maximum comfortable eye rotation range, which should also be based on the range over which the eye tracker can accurately track the user’s gaze, e.g. when the user is wearing thick corrective glasses.

The effect is that when the user is looking at an object that is located to their right, the object in their focus is then slowly rotated counterclockwise around the user (together with the rest of the virtual world) until it ends up remaining stationary straight in front of the user after a few seconds. If the user then looks at a different object to the left or right, the process repeats. Hence, the gaze data enable users to intuitively indicate the direction they want the virtual camera to rotate. Since the virtual world rotates only slowly during a continuous fixation of the eyes, it does not impair their normal gaze behavior when looking around in the virtual view in front of them. An onset-threshold or a voice-triggered disabling of the mechanism may be applied.

As with the head-tracked implementation in the previous section, for the eye-tracked implementation it is possible to set a maximum rotation rate. We consider a rotation rate of 1 degree per second a reasonable amount of rotation. However, we should point out that our goal is more to provide a smooth and comfortable interaction for users than to optimize the values for navigation in a virtual world or for performance in a specific task. Practitioners may adjust these values as desired.

Discrete Rotation technique

The basic idea behind this technique is that users can naturally look around the virtual world in their individual comfortable range, e.g. by performing head rotations and/or looking around with their eyes alone. However, since this range is limited, a technique has to be introduced to rotate the view beyond that limit in the virtual world. With the Discrete Rotation technique, in order to rotate the view clockwise or counterclockwise from their current orientation, users can rotate their head or eyes in the desired direction until they approach their predefined individual threshold of comfortable head/eye rotations. Once movement approaching the threshold is detected by the head/eye tracker, the virtual view changes and is replaced by a new view that is rotated in the corresponding direction by a fixed amount. This amount is defined by the angular threshold, i.e. the view is rotated such that virtual content that was visible at the edge of the movement range before the discrete rotation is visible thereafter in the center of the user’s view in front of them. To prevent an unwillingly triggered discrete rotation, e.g. induced by pain-triggered and/or unintentional movement, the rotation is only applied after an onset threshold. This onset threshold is implemented as a minimum range before the actual discrete rotation is applied. Figure 3 gives an example of how this technique can be used to rotate the view in VR with head movements.

Figure 3.

Illustration of the head-tracked Discrete Rotation technique in an example sequence of head movements. (a) User is facing straight ahead. (b) User rotates to the right beyond the threshold. (c) The virtual view is rotated by that angle. (d) User can freely look around the rotated view. The virtual environment is illustrated by the green checkerboard. Once the user’s physical head orientation crosses the threshold τ or τ, the virtual view is rotated instantaneously in the opposite direction by the angle corresponding to that threshold. The user can repeat this as often as desired to rotate the view. The eye-tracking based implementation uses the same approach except that the angles are taken from the eye tracker and not from the head tracker.

In contrast to the Continuous Rotation technique, this technique does not involve any continuous background rotations in the virtual world, instead introducing brief discrete rotations of larger magnitudes which can be implemented having immediate effects within the next frame or a lerp over a predefined amount of time. Such techniques are also known as clutching techniques[26] since they involve repeated shifts in the VE by a fixed amount. By repeatedly moving back and forth with the head or eyes, the user can move that view window in any direction and over any angular distance.

In this case, we assume that the user’s head orientation is tracked by the HMD and that the user is capable of rotating their head, but only in a reduced range. Here, we define the variables and τ, τ the same way as in the head-tracked implementation for the Continuous Rotation technique.

When the user rotates their head, we look for orientations close to left and right thresholds denoted by or for some tolerance . Once such a head orientation is detected, the user’s virtual view is rotated instantaneously by the angle . After that triggered rotation the user can rotate their head back toward the comfortable range between the thresholds and resume natural head rotations (looking around the virtual world) within that range, or they can further/again rotate the virtual view by rotating to the left/right threshold.

Eye-tracked implementation

As an alternative to using head rotations, e.g. in case no physical head rotations are possible by the individual, it is possible to implement a similar clutching technique using eye movements. Here, we define the variables and τ, τ in the same way as in the head-tracked implementation for the Discrete Rotation technique.

The user can naturally look around the view on the HMD while we look for left/right gaze rotations that approach the left and right thresholds denoted by or for a tolerance of . Once such a gaze angle is detected, the user’s virtual view is rotated instantaneously by the angle . After that triggered rotation the user can look back toward the comfortable range between the thresholds, and resume natural looking around the virtual world, or they can further/again rotate the virtual view by again rotating their eyes to one side or the other, exceeding the thresholds.

An optional variation of this technique is to leverage eye blinks as an additional form of input. Instead of instantaneously rotating the view once the user crosses a threshold and looks toward the far angles of the display, the user can look in one direction and then perform a blink with their eyes to voluntarily trigger an instantaneous step in rotation. The advantage of this blink-induced technique is that users are less aware of the virtual world rotation, which can reduce discomfort associated with such brisk rotations. In the field of vision sciences, it is a well-researched concept that brief inter-stimulus intervals, such as when the eyes close and re-open during a blink, and associated perceptual masking processes can induce a phenomenon sometimes called change blindness,[27] which denotes the perceptual illusion that persons are unable to notice even large changes in their visual field if it happens exactly at the same time.[28] While using blinks is not mandatory in order to use this technique, it is our impression that such blink-induced rotations can greatly improve the visual comfort while using techniques such as this.[29] To prevent unintentional blinking-induced reorientation, we suggest blinking-patterns to be used for this approach.

Additionally, we propose an auto-adjustment for both techniques. Multiple attempts, e.g. numerous rapidly occurring eye movements toward one side of the visual field in order to reach a specific target indicates that the amount of discrete rotations is not sufficient for that particular user. Exceeding a threshold with this behavior would then trigger a dynamic adaptation of the rotation angle for each individual.

Discussion

In the previous section, we described two techniques with two realizations each that can be used to implement augmented virtual rotations, thus allowing users to look around a 360-degree VE presented on an HMD. The head-tracked implementations can be applied for users who can rotate their head over a reasonable range depending on each individual, whereas the eye-tracked implementations can be applied for the same users or even individuals who are incapable of rotating their head in the real world.

One advantage of the described techniques is that they do not require the user to manipulate any devices with their hands. As such, the techniques can be applied hands-free without the necessity for further instrumentation of the user. However, it should be noted that the described techniques only focused on rotations in a virtual world not translations. Not all applications in VR require the user to be able to change their position in the virtual world, such as when watching 360-degree VR movies. However, if desired, translations in the virtual world could be implemented through rotation-translation mode changes, e.g. via blinks or voice commands. We did not consider such translation techniques in the scope of this paper.

Another consideration when using these techniques is simulator sickness, which denotes possible sickness symptoms after longer term use of a VR system.[30,31] The most common cause of simulator sickness in VR systems is a visual-vestibular conflict, which can arise when the visual feedback received from the virtual world does not match exactly the vestibular feedback received from the user’s physical body senses. For instance, such conflicts arise when the virtual camera motion differs from the user’s physical head movements.[12,13,15,16] Although susceptibility to simulator sickness differs among individuals and sickness symptoms can be caused by different aspects of a VR system, many researchers have been working on identifying methods aimed at reducing simulator sickness such as vignetting,[32-34] blurring,[35] rotation snapping, and adding a virtual nose for the user.[36,37] The techniques presented in this paper can potentially induce more or less simulator sickness depending on the parameters and the person. We suggest that the thresholds and maximum rotations or rotation rates should be adjusted for each individual user to ensure that simulator sickness symptoms are kept within tolerance levels.

To systematically line out the field of applications for our techniques, we propose a rough classification of mobility impairments and best-practice reorientation techniques. These recommendations are based on past research on the techniques aimed at reorienting users while considering the induced simulator sickness reported for each technique; nonetheless other combinations can also be practiced.

As a practical guideline, if the user is able to perform natural eye movements (high) but has no or almost no mobility in head orientation (low) over a reasonable range depending on each individual, we recommend the use of the eye-tracked Discrete Rotation technique in conjunction with blinking as a triggering mechanism for the rotations. This approach has the benefit that it can be performed without requiring any physical head rotations. Moreover, this approach is likely to induce the lowest amount of simulator sickness, as there is no incongruent vestibular stimulation additional to the visual stimulation. Its main drawback is that the discrete rotations of the virtual world may induce moments of disorientation in a user, which should be taken into account. If the primary goal is to avoid disorientation, and the user is capable of a reasonable range of head rotations (head-mobility high, eye-mobility low), we recommend using the head-tracked Continuous Rotation technique. This technique has the benefit that it decouples the virtual camera rotations through the head orientation from the movement of the eyes. As such, the users will not be exposed to slight rotations of the virtual view whenever they explore the virtual world. In case there is a high mobility in both modalities, we propose a discrete reorientation triggered by eye blinking on top of the Continuous Rotation controlled by the head orientation.

The combination of both tracking systems can also be used for error prevention and correction of unintentional movement. A discrete orientation may only be applied if head/eye tracking thresholds are simultaneously exceeded, enabling the users to explore their limited field of view without unintentionally triggering the outlined mechanisms.

Conclusion

In this paper, we described two techniques based on continuous and discrete rotations of the virtual view in an HMD setup that can be applied to enable users to experience 360-degree rotations in a virtual world even when their physical head movements are severely restricted, e.g. when they are leaning against the headrest of a wheelchair. We presented two implementations for each of the two techniques, which are either based on head tracking data or eye tracking data. If the mobility allows both tracking modalities to be used, we proposed a combinational approach of both techniques that can greatly increase usability and reliability of our approaches. We discussed the advantages and disadvantages of the techniques and provided guidelines for practitioners in this field that may help them select one or multiple techniques with respect to the user’s mobility. In future work, we plan to perform a user study in the subject field. We also hope to extend the work by further considering input from brain–computer interfaces as an additional condition.