Materials and devices for immersive virtual reality.

The metaverse may change the way we live and interact with one another, and its potential applications range from entertainment to health care. Extended reality is the main technology to realize the highly realistic, interactive and immersive metaverse experience, and wearable electronic devices and materials are at its core. © Springer Nature Limited 2022.

The metaverse is a shared virtual-reality space bridging the material and immaterial worlds, where users can interact with other users and with computer-generated environments. The core of the metaverse is XR (extended reality) technologies, which include virtual reality, augmented reality and mixed reality. The current COVID-19 pandemic situation and the consequent demand for remote social interactions are accelerating the development of metaverse technologies. Moreover, Facebook’s acquisition of the virtual-reality technology company Oculus VR Inc. and its rebranding as Meta have dramatically increased the public interest in XR technologies and unlocked the true potential of the metaverse.

XR technologies are already used in entertainment (films, media and games), education (interactive virtual classes and laboratory experiments), teleconferencing (VR events, conferences and meetings) and product design, and are now expanding into health care (tele-surgery, surgery planning, medical training and virtual therapy), retail business (online VR shopping and virtual fitting rooms for e-commerce), tourism (virtual expeditions and field trips), job learning and training, advanced real-estate sales, and the fitness and art industries[1].

The further development of XR technologies requires advances in both software and hardware. Recent progress was driven by software developments, whereas the comparatively slower hardware development (devices and materials) is becoming an increasingly important limitation. XR devices produce realistic and highly immersive experiences by artificially reconstructing various human senses to create artificial perceptions of the virtual world. Users engage with XR devices, such as gloves or body suits, and, as such, all XR devices can be viewed as wearable electronic systems.

Wearable electronic devices for XR

The information delivered by XR devices is mainly classified according to the sensation type: visual and auditory information is provided by head mount displays (HMDs) and speakers, and tactile information by haptic actuators that exploit mechanical or electrical mechanisms.

Commercial VR products based on visual and auditory technologies are remarkably successful, but progress in haptic systems has been relatively slow. HMDs mainly cover the two major senses (sight and hearing), combining displays and headphones. Transparent and deformable displays are key components because they not only provide highly realistic virtual visual information but also serve as efficient user interfaces. By contrast, haptic interaction involves tactile sensation information exchange between a physical and a virtual space and requires sensors for haptic input (physical space to the virtual space) and actuators for haptic feedback (virtual space to the physical space).

Haptic technologies sense and replicate various tactile sensations generated by interactions with the environment, such as those due to dynamic variations in pressure, shear forces and temperature, with sensors and actuators. Sensors obtain haptic perception[1] to detect the user’s conditions (body motion, hand gestures and physiological condition) as well as the environmental conditions (temperature, pressure, texture and shape of objects in the virtual space), while actuators reconstruct this information artificially in the virtual space. Various haptic input devices have been integrated in wearable sensors, but most haptic feedback devices are still limited to the very simple vibrational modes of operation (vibro-haptic)[2] to replicate limited mechanical sensations. Various other haptic feedback devices, for example, based on cold and hot sensations (thermo-haptic)[3], are being developed to provide a more immersive experience in the virtual space.

Soft materials for XR

Currently, most commercially available XR devices are rigid. However, because haptic devices generate artificial sensations and transmit them from the virtual world directly to the users’ skin, the intimate and conformal contact between the skin and the XR devices has a very important role in the immersive experience. Current studies on next-generation XR materials therefore focus on soft, skin-like wearable electronic materials to overcome the intrinsic limitations of conventional rigid electronics.

Stretchable conductors

Stretchable, skin-mountable soft conductors are an integral part of soft haptic sensors in XR devices. Stretchable conductors are usually made from intrinsically stretchable functional conductors (carbon-based nanomaterial such as carbon nanotubes and graphene, metal nanowire percolation networks, liquid metals or conducting polymers) or serpentine-shaped electrodes made of non-stretchable conductors.

Stretchable thermoelectric materials

Thermo-haptic interfaces generate artificial cooling and heating sensations in XR[3]. The heating function is relatively simple to achieve by resistive heating with electrically conductive materials, but cooling is challenging in a wearable form. Conventional cooling by refrigeration cycles requires bulky and complex systems. Skin-like soft thermoelectric devices that use highly efficient organic thermoelectric materials, or rigid inorganic thermoelectric materials connected with stretchable serpentine metal conductors, represent promising options.

Variable-stiffness materials

Variable-stiffness materials[4] change their mechanical properties to generate an artificial sensation of softness or rigidity in a virtual space. Particles or layered thin sheets that can realize jamming, low-melting-point materials with phase changes, and magnetorheological or electrorheological materials can be used to produce these sensations.

Soft actuators

Actuators are integral parts of a vibro-haptic interface. Conventional rigid actuators are not ideal for XR applications due to discomfort that can occur at the skin interface and to reduced wearability. Various soft materials have been explored to realize, for example, dielectric elastomer actuators that involve compliant capacitors with passive elastomer films sandwiched between compliant electrodes, fluidic elastomer actuators that involve elastomeric composites of embedded pneumatic or hydraulic channels and materials with different elastic moduli to induce anisotropic deformations, and stimuli-responsive actuators made of materials such as liquid crystal elastomers or magnetic responsive materials that respond to various external stimuli, such as heat, electric or magnetic fields, or light with reversible shape changes or specific programmed patterns[2].

Challenges

Several challenges have to be overcome to develop more realistic immersive XR devices for future metaverse applications.

First, the ultimate objective of XR devices is to deceive users with realistic artificial sensations. XR devices capable of delivering sensations across all five basic human senses (vision, hearing, touch, smell and taste) will provide a richer and more realistic virtual experience, but olfactory and gustatory senses are still not captured in XR devices and need development. Realizing all five modes of sensation together will be essential for immersive and compelling XR experience.

Second, XR devices are worn by users. Besides functionality, maximum comfort and wearability are essential requirements for the successful commercialization of XR technologies. Currently, wearability and functionality cannot be achieved simultaneously and thus developers must compromise between them.

Third, if any sensation is missing, delayed or inaccurately reproduced, the users’ brain will receive conflicting signals between the virtual space and the physical space, which will result in nausea, dizziness and migraines. Information mismatch must be minimized by improving the realism of the generated sensations and minimizing the latency in XR devices.

Fourth, the usability of current XR devices is limited by the bulky electrical wires connected to the HMDs to support heavy computational loads and large power consumption. The wires constrain natural movements during use. Compact power supply and untethered operation are desirable for the development of future XR devices.

Finally, besides the technical issues, XR technologies involve critical ethics issues that must be considered, including data security, privacy protection and mental health risks[5]. The operation of advanced XR systems will involve the collection of vast amounts of personal data through large numbers of sensors. Technical and legal considerations must be navigated to protect privacy and personal information.

Outlook

We are now witnessing the beginning of a new paradigm for virtual interactions, in the form of the metaverse. Research on next-generation XR applications relies heavily on the development of advanced functional materials and devices such as soft haptic interfaces and sensors, and supporting flexible electronics. XR devices, when combined with wearable electronic technologies and soft robotic systems, offer exciting routes to new solutions in health care, rehabilitation and medical treatments to provide enhanced telesurgery, teletherapy or telemedicine experiences that overcome geographical limitations through the implementation of digital twins of 3D organs or of the entire human body. Further research will be needed to fully understand the fundamental relationship between measurable parameters and haptic perception for immersive XR devices. The ultimate XR devices may involve direct integration with the human body through interfaces with the brain. Such implantable XR devices have the potential to completely free users from external hardware. All these potential future directions rely critically on the development of advanced materials.