Immersive media experience: a survey of existing methods and tools for human influential factors assessment.

Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user's sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.

Introduction

It is predicted that the global augmented/virtual reality (AR/VR) market will reach US$814.7 billion by 2025 [1] through steady and continuous growth of new mobile applications and the appearance of the fifth-generation (5G) wireless networks worldwide. 5G networks promise faster speeds, lower latency, wider coverage, and more stable connections. Applications across multiple verticals are projected, including entertainment and media, gaming, healthcare, automobile, aerospace and defense, manufacturing, retail, and education, to name a few. In fact, as recently emphasized by Qualcomm, 5G coupled with VR will be essential for the development of next-generation immersive experiences and will enable applications, such as six degrees of freedom immersive content, automotive video streaming, crowded event sharing, remote control, and the tactile internet. Recent projections show that the number of users, in the United States alone, will double by 2022 [2]. To fully unlock this potential, however, the effectiveness of these emerging applications will not only depend on their technical capabilities, but also on the quality of experience (QoE) they provide to the user [3].

More generally, QoE refers to the “degree of delight or annoyance of applications or services resulting from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the users personality and current state” [4]. When it comes to immersive media and content, immersive media experiences (IMEx) [5] build on the QoE concept by also including factors such as sense of presence and immersion, as well as motion sickness (cybersickness), to name a few. In fact, QoE and IMEx are driven by three influential factors (IFs): system, context, and the (human) user.

Within immersive experiences, devices play a large role as system influential factors [5], as the capability of accurately tracking the user’s behaviors (e.g., body/head position, movements, eye tracking) can affect the interaction, as well as the sense of presence and immersion [6]. Device design in terms of portability, usability, field-of-view, visual quality, and ergonomic aspects have been shown to also impact the user experience [7, 8]. Moreover, multisensory experiences involving audiovisual media with e.g., haptic, olfactory, and gustatory stimuli in VR can also drastically affect the overall IMEx [9]. As highlighted in [5], in turn, contextual IFs describe the user’s environment, such as physical, temporal, social, economic, task, and technical characteristics. Sometimes it is difficult to separate system and context completely and they are evaluated together. On the other hand, human influential factors include “any variant or invariant property or characteristic of a human user,” i.e., factors related to emotional state, expectations, attention, among others [4]. As highlighted in [5], “the fact that not every human becomes equally immersed in the same book, movie, or game, illustrates that human IFs are of very high relevance for an IMEx.” Moreover, the sensitivity of each user towards incongruency and timing differences between perceptual modalities can lead to discomfort, visual fatigue, and motion sickness (also known as cybersickness), which has been reported to affect between 30 and 80% of users, with symptoms potentially lasting for several hours post VR exposure.

In order to continuously improve VR technology, constant evaluation is needed to measure IMEx, taking all three IFs into account. While system and context have been widely explored, the impact of human influential factors on IMEx is a less studied topic, thus will be the focus of this present paper. In particular, two assessment methods have been commonly used for this purpose, namely: subjective and instrumental. The latter can be further classified as behavioural and psycho-physiological. Conventionally, subjective methods consist of questionnaires, administered after the VR experience, that measure certain aspects of the human experience itself [10-14]. As can be expected, subjective tests can be biased, lack temporal resolution, and allow only for offline analysis, thus cannot be used to improve the IMEx in real-time. On the other hand, (instrumental) behavioural HIFs assessment is based on tracking user behaviours, such as facial expression, body gestures, and social interaction. These behaviours can be generated and controlled in a non-intentional, automatic manner and do not necessitate conscious introspection as in subjective methods.

Over the last few years, developments in biosensors [15] have allowed for numerous HIFs to be monitored in real-time in immersive environments [16]. As such, (instrumental) psycho-physiological methods have emerged and aim to find correlates between perceptual QoE/IMEx features and physiological metrics [17]. Physiological signals have been used, for instance, to measure stress [18], engagement [19], emotions [20], sense of presence [21], immersion [22], and overall experience [23, 24], thus can play a key role in advancing VR applications. Some physiological signals that have shown useful for experience assessment include the electroencephalogram (EEG), electrocardiogram (ECG) and measured heart rate (HR) or heart rate variability (HRV), electrooculogram (EOG) and eye blinks, electrodermal activity (EDA) [25, 26], and cerebral blood-flow measured via near-infrared spectroscopy (NIRS) [27]. Moreover, recent advances in dry/wireless electrodes [28, 29] and motion artifact suppression [30] have allowed for biosensors to be integrated directly into VR headsets (e.g., [31]) and to monitor human IFs in real-time [32], as shown in Fig. 1.

Fig. 1

Sensor-equipped VR headset with embedded sensors (left) and software suite (right) to save or live-stream biosignals, as well as measure signal quality and extract relevant human influential factors

Development of psycho-physiological methods is still in its infancy and much work is still needed in pre-processing the obtained signals. Unlike conventional video QoE assessment where users are sitting and static [4], VR experiments have users moving around and exploring the virtual environment, hence hampering signal quality and overall human IF measurement. Ultimately, monitoring of human biosignals while immersed in virtual reality will allow us to characterize factors such as cybersickness, perception of immersion, and overall experience, thus allowing for experiences to be adapted in real-time to maximize the IMEx. As VR becomes more popular, assessment of IMEx has become critical in order to develop applications that can be used and enjoyed by the masses.

This paper presents a survey of the literature on IMEx assessment, in particular on HIFs assessment, to find out what the latest trends and innovations are. We start by looking at existing subjective methods focusing on the user’s sense of presence, perception of immersion, cybersickness, emotional state, and experience. Next, we focus on behavioural and psycho-physiological measures, describing the latest biosensors and tools used. While the focus of this survey will be on psycho-physiological assessment, we provide a brief summary of the other methods and guide the reader to available review papers on the topics. Lastly, we venture into next-generation applications encompassing multiple senses, beyond audio-visual, to improve realism and immersion. We conclude with a brief discussion on limitations and provide the reader with recommendations for future studies.

Subjective IMEx assessment

Subjective evaluations are the most common method for IMEx/HIFs assessment. They are usually applied shortly after the end of an experiment through questionnaires or rating scales. Their composition can vary according to the purposes of a specific experiment or application, or could be more generic and applicable across several contexts. Moreover, due to the lack of a common understanding and definition of the terms presence and immersion, a plenitude of different questionnaires have been developed. Recently, questionnaires that have proven their effectiveness via pen-and-paper assessments have also been integrated into virtual reality [33-35], hence reducing study duration and discomfort to the users [36]. Here, we focus on four aspects of IMEx measured from subjective assessments, including: sense of presence and perception of immersion, user quality of experience, cybersickness, and mental/affective state.

Questionnaires for presence assessment

Presence within the context of virtual reality is defined as one’s sense of being in the virtual world. One of the fundamental aspects of VR is the ability to create and maximize the sense of presence of the user [12, 37, 38], hence making them feel like they are present in the virtual world. While sense of presence and immersion are closely related, numerous studies have categorized presence [39] based on what is being measured [11, 12, 40–43]. For example, [44] lists three types of presence in VR, namely personal presence (also called self-presence), i.e., the feeling of “being there,” social presence (also called co-presence), i.e., the sense of “being there with others,” and environmental presence (also called physical, telepresence or spatial presence), where participants feel immersed physically in the virtual environment and interact with virtual objects.

In other words, the sense of presence is influenced by a range of elements including equipment factors (as physical barriers and device awareness), user’s subjective factors (personality traits or immersion propensity), social factors (interactions with VR characters), and affective [45], such as the emotions about self (anxiety, paranoid ideation, detachment), emotions about others (loneliness, retrospective emotions, recognition of self), thoughts about self (memories, social judgement), thoughts about others (paranoid ideation, narrative), physiological reactions (anxiety, cybersickness), behaviour of avatars (narrative, duration of interaction, characteristics), interactivity with environment (movement, familiarity), and environmental characteristics (restrictions) [46].

Here, we present the most common questionnaires that have been created to measure sense of presence in VR. Table 1 lists the questionnaires, how many subjects they were validated on, subscales used, rating scale, number of items rated, as well as which media they are applicable to, namely: virtual environment (VE), cross-media (CM), shared virtual environment (SVE), and 2D screens. Moreover, citation numbers were taken from Google scholar and latest numbers were confirmed at the date of paper submission. As can be seen, most of the questionnaires were created more than 20 years ago. The most popular and most widely used to date is the so-called Presence Questionnaire (PQ) [12] that has been cited over 5250 times and measures involvement, sensory fidelity, adaptation/immersion, and the interface quality. The GlobalED Questionnaire [47] is the most popular for social presence, with over 2500 citations, and the Igroup Presence Questionnaire [40] for spatial presence, involvement, and the experienced realism. Although PQ continues to be the standard in virtual reality research, it has been argued that the instrument does not provide a measure of presence, but of the individual’s responses to various aspects of the virtual reality system, through a series of questions that involve the expression of the respondent’s opinion about the evaluated factors (i.e., control, sensory, distraction, and realism factors) [41]. The interested reader is referred to [48-50] for a complete in-depth review on subjective presence questionnaires.

Table 1

List of commonly used presence questionnaires for subjective IMEx assessment

References	Questionnaires	Subject	Subscale	Rating scale	Citations	Items	Media
[54]	Barfield et al. Questionnaire 1	86	Personal presence	0–100	329	2	VE
[55]	Barfield et al. Questionnaire 2	12	Monoscopic vs stereoscopic display, head-tracking, field-of-view	5-point	409	3	VE
[56]	Memory characteristic Questionnaire (MCQ)	90	Presence, Judgment, Attention, Coherence, and Field-of-view	7-point	1257	21	SVE
[43]	Slater-Usoh-Steed Questionnaire (SUS)	24	Presence from internal/external factors	7-point	908	6	VE
[57]	Lombard & Ditton Questionnaire	600	Social presence, Realism, Transportation, and Immersion	Not defined	332	103	CM
[47]	GlobalED Questionnaire	50	Social presence	5-point	2560	14	SVE
[58]	Kim & Biocca Questionnaire	96	Physical, Virtual or imaginary presence	8-point	860	8	2D screen
[59]	Reality Judgment and Presence Questionnaire (RJPQ)	124	Reality judgment, and Attention	10-point	225	18	VE
[12]	Presence Questionnaire (PQ)	152	Presence, Involvement, and Immersion	7-point	5254	32	VE
[60]	Thie & Van Wijk Questionnaire	48	Social presence	Not defined	38	Not defined	VE
[61]	Presence & Realism	Not defined	Virtual art exhibits	4-point	8	10
[62]	Dinh et al. Questionnaire	322	Visual, Olfactory, Auditory, and Tactile	0–100	537	14	VE
[63]	Murray et al. Questionnaire	10	Impact of the sound on the sense of presence	5-point	56	5
[64]	Nichols et al. Questionnaire	24	Influence of the headset, and Auditory stimuli	7-point	231	9	VE
[65]	Basdogan et al. Questionnaire	10	Social presence	7-point	497	8	SVE
[66]	ITC - Sense of Presence Inventory (ITC-SOPI)	604	Sense of Physical space, Engagement, Ecological validity, and Negative effects	Not defined	1121	44	CM
[67]	IPO Social Presence Questionnaire (IPO-SPQ)	34	Social presence	7-point	151	17	2S screen
[68]	Gerhard et al. Questionnaire	27	Immersion, Communication, Involvement, Awareness, Nature of the environment, and User interface	7-point	82	19	SVE
[69]	Krauss et al. Questionnaire	165	Presence	Rating scale	14	42	VE
[40]	Igroup Presence Questionnaire (IPQ)	246	Spatial Presence, Involvement, and Realism	5-point	1283	14	SVE
[70]	Swedish Viewer-User Presence Questionnaire (SVUP)	32	Enjoyment, Sound quality, Presence, and Cybersickness	5-point	72	19	VE
[71]	Schroeder et Al. Questionnaire	132	Physical and Social presence	5-point	177	11	SVE
[72]	Bailenson et al. Questionnaire	50	Social presence	7-point	493	5	VE
[73]	CMC Questionnaire/Social presence and Privacy Questionnaire (SPPQ)	310	Social presence	5-point	592	17	CM
[74]	Networked Minds	76	Co-presence, Psychological Involvement, and Behavioral Engagement	7-point	405	40/38	SVE
[75]	E^2I Questionnaire	10	Presence, and Enjoyment	7-point	451	14	VE
[76]	Nowak & Biocca Questionnaire	134	Telepresence, Copresence, and Social presence	7-point	833	29	SVE
[77]	Cho et al. Questionnaire	32	Visual realism, and Presence	0–100	40	4	VE
[78]	MEC-SPQ	Not defined	Spatial presence, and Attention	5-Point	206	8	VE
[79]	Temple Presence Inventory	46	Satial presence, Social presence-actor, Passive social presence, Active social presence, Presence as engagement, Presence as social richness, Presence as social realism, and Presence as perceptual realism	7-point	223	42	CM
[80]	Tendency Toward Presence Inventory	499	Cognitive Involvement (active and passive), Spatial Orientation, Introversion, Ability to Construct Mental Models, and Empathy	5-point	57	41	VE
[81]	The German VR Simulation Realism Scale	151	Visual Realism, Audience Behavior and Appearance, and Sound Realism	5-point	18	14	VE
[82]	Multimodal Presence Scale (MPS)	161/118	Physical, Social, and Self presence	5-point	62	38/15	VE
[83]	Short QoE questionnaire	36	Perceptual quality, Presence, Acceptability, and Cybersicknes	5-point	84	5	VE

Questionnaires for user experience assessment

User experience is defined as perceptions and responses resulting from the use of a system. It is assessed by various measures of involvement such as engagement, flow, immersion, and encapsulates the user’s preferences and behaviour during use. Immersion represents the instrumental level of sensory fidelity provided by a VR system, and in applications requiring a certain level of suspension of disbelief, it plays a crucial role in overall experience. Immersion also modulates user engagement and can result in achieving a flow state. In VR, sensory immersion is defined as “the degree in which the range of sensory channels are engaged by the virtual simulation” [51]. Moreover, flow experience is often considered as an important standard of ideal user experience and keeping users in the flow state is considered as one important goal in VR system design [52]. Here, we list the most commonly used user experience questionnaires in Table 2, alongside how many subjects they were validated on, the subscales used, rating scale, number of items rated, as well as which media they are applicable to, i.e., VE, SVE, CM, or 2D screens. The interested reader is referred to [53] for a complete in-depth review on subjective user experience questionnaires.

Table 2

List of commonly used user experience questionnaires for subjective IMEx assessment

References	Questionnaires	Subject	Subscale	Rating scale	Citations	Items	Media
[84]	Immersive Experience Questionnaire (IEQ)	244	Cognitive, Involvement, Emotional involvement, World dissociation, and Challenge	5-point	1791	31	CM
[85]	GameFlow Questionnaire	Not defined	Concentration, Player Skills, Control, Challenge, Feedback, Clear goals, Immersion, and Social Interaction.	GameFlow criteria	2715	35	CM
[86]	EGameFlow Questionnaire	Not defined	Concentration, Control, Knowledge Management, Challenge, Goal clarity, Immersion, Feedback, and Social Interaction	0–100	786	42	CM
[87]	Game Engagement Questionnaire (GengQ)	153	Immersion, Flow, Presence and Absorption	5-point	982	19	CM
[88]	Game Experience Questionnaire (GexpQ)	Not defined	Immersion, Competence, Flow, Negative effect, Positive effect, and Challenge	5-point	396	33	CM
[89]	EVE-GP questionnaire	2182	Multidimensional UX in video games	7-point	33	180	CM
[90]	Narrative game questionnaire	340	Curiosity, Concentration, Control, Challenge, Comprehension, and Empathy	7-point	266	27	CM
[91]	SCI Model Questionnaire 10	234	Sensory immersion, Challenge-based immersion, Imaginative immersion	5-point	1395	18	CM
[92]	Core Elements of the Gaming Experience (CEGE) questionnaire	15	Enjoyment, Frustration, Control, Puppetry, Facilitators, Ownership, Game-play, and Environment	7-point	252	38	CM
[93]	Unified questionnaire on User eXperience in Immersive Virtual Environment	116	Presence, Engagement, Immersion, Flow, Usability, Skill, Emotion, Experience consequence, Judgement, and Technology adoption	10-point	76	87	VE
[94]	Presence-Flow-Framework (PFF)	68	Perceptual experience, Situational involvement, and Competence	7-point	165	124	VE
[95]	Presence-Involvement-Flow Framework2(PIFF2)	91	Presence, Involvement, and Flow	7-point	79	139	VE
[96]	Virtual Reality Neuroscience Questionnaire (VRNQ)	40	QoE, Game mechanics, and In-game assistance	7-point	29	20	VE
[97]	User Experience Questionnaire (UEQ)	144	Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty	7-point	1297	26	CM

Questionnaires for cybersickness assessment

Cybersickness is one of the main limitations of VR as it induces physiological changes that affect the users’ sympathetic and parasympathetic activities. Reports suggest that cybersickness can affect between 30 and 80% of users and that symptoms can last for several hours [98]. The most common hypothesis to explain cybersickness is the sensory conflict theory. Indeed, cybersickness is the result of conflicts between three sensory systems: visual, vestibular and proprioceptive. Cybersickness is a complex phenomenon and, although motion cues play a primary role, multiple factors are known to contribute to the occurrence of sickness. These include factors related to the characteristics of the stimuli (e.g., spatial frequency, reactivity of the system, wideness of the field-of-view) and factors related to predispositions of the user (e.g., gender, age, predisposition to migraines) [99]. The most evident symptom of cybersickness is nausea, but there are also others, including general discomfort, headache, disorientation, and eye strain. Symptom intensity and duration are quite variable and depend on the characteristic of the stimulus, as well as user predisposition to cybersickness. In the majority of cases, the symptoms disappear a few minutes after the end of the stimulation. In particular cases, the symptoms could still be present 6 hours after the VR experience [100].

There have been a number of questionnaires developed to evaluate cybersickness, as shown in Table 3. Although the Simulator Sickness Questionnaire (SSQ) is widely used in VR research, it was originally developed to measure motion sickness in simulators [101]. It has been criticized for its psychometric qualities and applicability in VR as a measure cybersickness [102]. Recent questionnaires have since been developed specifically for HMDs, such as the Virtual Reality Symptom Questionnaire (VRSQ) [103], and have shown better indicators of validity [104]. Moreover, there are questionnaires that also focus on the severity [105] of cybersickness. The questionnaires have been regarded as being too long, so shorter versions have also been explored and validated, such as the MSSQ-Short [106], and the Simplified SSQ [107]. The interested reader is referred to [102, 108, 109] for an in-depth review on cybersickness assessment in VR.

Table 3

List of commonly used cybersickness questionnaires for subjective IMEx assessment

Reference	Questionnaires	Subject	Subscale	Rating scale	Citations	Items	Media
[101]	Simulator Sickness Questionnaire (SSQ)	Not defined	Nausea, Oculomotor, and Disorientation	4-point	4206	16	CM
[110]	Motion Sickness Assessment Questionnaire (MASQ)	310	Motion-sickness	9-point	297	16	CM
[106]	Motion Sickness Susceptibility Questionnaire-Short (MSSQ-Short)	257	Motion-sickness	4-point	273	18	VE
[105]	Fast Motion Sickness Scale (FMS)	126	Motion-sickness	0–20	277	1	VE
[103]	Virtual Reality Sickness Questionnaire (VRSQ)	24	Oculomotor, and Disorientation	4-point	212	9	VE
[111]	Misery Scale (MISC)	24	Motion-sickness	0–10	161	1	VE
[112]	Symptom Questionnaire	16	Motion-sickness	0–6	155	13	VE
[113]	Refactored SSQ	371	Nausea, and Oculomotor	4-point	140	16	VE
[107]	Simplified Simulator Sickness Questionnaire	158	Uneasiness, Visual Discomfort and Loss of Balance	5-point	2	9	CM

Questionnaires for affective/mental state monitoring

The word “experience” in the user experience expression implies that there is emotional involvement when users explore immersive media content. For example, users may feel happy, satisfied, frustrated, overjoyed, or disappointed by the experience. In general, emotional experiences can be described by three emotion primitives: valence (the pleasantness of a stimulus), arousal (the intensity of emotion provoked by a stimulus), and dominance (the degree of control exerted by a stimulus). The Self-Assessment Manikins is a picture-based questionnaire widely used to assess valence, arousal, and dominance [114]. Graphical tools allow users to report their feelings efficiently and intuitively by indicating or rating the part of the figure that best represents their current affective state. Graphical self-report instruments are appealing for the measurement of affective experiences since they do not require the users to verbalize their emotions. Instead, they rely on the human ability to intuitively and reliably attribute emotional meaning to (simple) graphical elements. However, the user may not be able to easily interpret the pictorials and therefore may have difficulty identifying with them [115]. More recently, a variant was proposed based on emojis (e.g., smiling or frowning faces). Emojis are pictographs or ideograms representing emotions, concepts, and ideas. Emoji-based rating tools are increasingly becoming popular tools as self-report instruments to measure user and consumer experience. The EmojiGrid, for example, has been proposed as a self-report tool for the assessment of VR-evoked emotions [116]. The interested reader is referred to [117-119] for more details on user emotional/affective state assessment in VR.

Instrumental IMEx/HIFs assessment

While subjective assessment methods aim to collect qualitative feedback from the users concerning their experiences, instrumental methods aim to measure them in a quantitative manner, thus allowing for easier replication and real-time (or quasi real-time) IMEx/HIFs monitoring. For example, behavioural measures can track the user’s facial expressions, movements, and eye gaze to monitor user reactions without the need for conscious introspection. Psycho-physiological methods, in turn, can be used to measure correlates of experience factors, such as engagement, attention, and cybersickness, to name a few. Commonly, instrumental methods rely on ground-truth labels obtained via subjective methods in order to build accurate and reliable models. Once the models are built, however, subjective methods are no longer required and real-time assessment can be achieved. With advances in sensor technology and wearable devices, instrumental methods are burgeoning. In the sections to follow, we present the findings from a survey of the literature spanning the years of 2015–2021, with particular focus placed on psycho-physiological methods, as the existing literature lacks a survey in this aspect.

Behavioural methods

With behavioural methods, the main goal is to assess whether the participants behave in the virtual environment as they would under similar conditions in the physical environment. For example, are the user’s physical movement and social interactions inline with those expected in the real world. Recent studies suggest that this can indeed be the case and participants can express verbal and bodily reactions in virtual reality in manners very similar to real situations [120]. Behaviors can provide unconscious cues about user experiences. A smile on one’s face typically indicates good user experience, while touching the HMD could imply poor fit or discomfort, and sweating and nausea typically indicates signs of cybersickness. In [121], for example, statistics showed that almost 50% of subjects touched their HMD at least once, suggesting discomfort and reduced sense of immersion. Over 76% of the participants, in turn, smiled at least once during the experiment, suggesting a positive reaction to the virtual content. Finally, roughly 10% of the participants reported sweating and nausea symptoms, suggesting visual fatigue and signs of motion sickness. The next sections report the various methods in which researchers have proposed to measure behaviours and what they represent in terms of IMEx.

Facial expressions

Facial expressions can tell a lot about the current emotional state of the user. Moreover, real-time facial expression recognition can improve the realism of virtual avatars, which can play a key role in user experience [122]. Recent innovations in sensor-equipped VR headsets have allowed for facial expressions to be monitored via facial electromyogram (EMG) electrodes placed directly on the VR faceplate of the head-mounted display (HMD) [31]. Alternatively, stand-alone facial EMGs have been placed in a transparent adhesive plastic film identical in shape and size to the actual HMD and placed around the eyes prior to donning the HMD [123]. Such a system was used to recognize 11 different facial expressions and on a sample size of 42 participants, an average expression recognition rate of 85% was achieved with eight sensors.

Eye tracking

Eye tracking provides information not only on where the users are focusing their attention at any particular point in time, but also on pupil dilation, blink rates, and eye movements indicative of e.g., cybersickness. Indeed, increases in pupil size reflect arousal associated with increased sympathetic activity [124]. In [125], a stressful experiment was conducted and showed that pupil dilatation and heart rate changes showed differences with stress. Furthermore, eye blink rate has a close direct relationship with dopamine activity in the brain, hence could be related to valence [126]. Moreover, increased attention levels have been shown to reduce blink frequency [127], as have more positive affective states [124]. Eye gaze information has also been linked to concentration levels and sense of presence in immersive environments [128]. More recently, pupillometry (more specifically, pupil dilation) was used to assess cognitive load in VR with uncontrolled scene lighting [129].

Eye tracking can typically be achieved with infrared cameras embedded into the HMD. Alternatively, electrooculography (EOG) sensors could also be embedded directly into the VR headset, as in [31], and used to track eye movements [32] and blink rates. Real-time knowledge of where the user is looking can also provide virtual environment developers cues about what captures users’ attention, what elements were more calming or stressful, and can allow for adaptive systems to be developed to make experiences more realistic (e.g., have an avatar look at you when you look at them), hence maximizing IMEx. More recently, eye movement information has also been shown to correlate with cybersickness, where atypical eye movements were indicative of motion sickness [130].

Movements and gestures

Body and head movements, along with arm gestures can also be indicative of IMEx related factors. For example, the work in [131, 132] showed correlations between head movements and user reports of valence, arousal, and emotional states. The work in [133], in turn, showed that different types of head movements (i.e., rotations and left/right tilts) could affect cybersickness. Postural stability was also shown to predict the likelihood of cybersickness in [134] and constrained movement was shown to reduce the sense of presence [46]. Commonly, hand tracking and gesture recognition has relied on VR controllers equipped with sensors. More recently, camera based systems have emerged that have allowed for controller-free hand/arm gesture tracking. While intuitively one would expect the controller-free setting to be more realistic, hence improve the overall IMEx, a recent study showed that controller-based interactions in VR were less demanding for the participants and resulted in fewer errors, thus in an overall improved IMEx [135]. The authors attributed this finding to the camera-based technology still being in its infancy (hence, not very reliable) and the learning curve of the user’s to a new technology. The interested reader is referred to [136] for an overview of gesture interactions in VR.

Psycho-physiological methods

Our bodies are an excellent canvas to convey our internal states. For example, our faces turn red when we are embarrassed, our heart rates go up when we are excited and/or stressed, our palms become sweaty when we are nervous or suffering from motion sickness, our heart rates and breathing rates synchronize when we are engaged. As biosensor technologies evolve and wearable devices become mainstream, psycho-physiological measurement has become a reality and has been incorporated into instrumental IMEx/HIFs assessment. In the sections to follow, we highlight methods that have been proposed in the literature over the last six years separated by biosensor modality. As the existing literature lacks a comprehensive survey of such instrumental methods, we aim to fill this gap with this paper.

Electrocardiogram and photoplethysmogram

Electrocardiogram (ECG) and photoplethysmogram (PPG) have become increasingly popular for studies in immersive virtual environments where a user’s heart rate (HR) and heart rate variability (HRV) need to be measured. While an ECG records the electrical activity of the heart, a PPG measures blood volume changes using optical sensors that measure changes in light absorption. Both methods provide information about heart rate, with PPG being the most widely used modality in wearables, as sensors can be easily embedded into bracelets and watch form factors. In both methods, it is common for a so-called RR time series to be derived from the interbeat/interpulse intervals and HR/HRV analysis is typically done on this heart rate series signal.

HRV analysis can be done in the time and frequency domains, as well as with nonlinear methods. Time-domain parameters rely on statistics computed directly from the RR series, such as standard deviation over certain window sizes. Frequency domain methods, in turn, rely on the power spectral density (PSD) of the RR time series, computed either via nonparametric (e.g., fast Fourier transform) or parametric (e.g., autoregressive models) methods. The PSD is then divided into different frequency bands, such as very low frequency (VLF: 0–0.04 Hz), low frequency (LF: 0.04–0.15 Hz), and high frequency (HF: 0.15–0.4 Hz), as these have shown to represent different aspects of the sympathetic and parasympathetic autonomic nervous systems. Commonly, absolute, relative, or normalized powers in the VLF, LF and HF bands, as well as their ratios, have been used to characterize heart rate variability. Lastly, as the RR time series exhibits complex non-linear behavior, non-linear measures have also been explored [137].

Table 4 presents a list of studies that have relied on HR and HRV measures to quantify different aspects of IMEx. As can be seen, measurement of stress is one of the leading aims. Dependent on the difficulty level of the game or the stressful sequences, an increase of HR is commonly observed [138, 139]. Moreover, a significant correlation between the valence emotional primitive and HRV has been demonstrated [140, 141]. An increase in HR was also seen during the last minutes when the user reported motion sickness [142]. The majority of the devices used were wearables-based, thus allowing the user to move during the immersive VR experience. It should also be noted that in the majority of the cases, multimodal systems were utilized, with HR/HRV coupled with other modalities; electrodermal activity (EDA) being the most prevalent [139, 140, 143–149]. The next sub-section is dedicated to the measurement IMEx correlates from the EDA.

Table 4

List of works relying on HR/HRV measurement for user IMEx assessment

Reference	# Subjects	Device	Measurement	Processing	Results	Questionnaire
[138]	21	AliveCor Kardia	Engagement, Concentration, Stress, Relaxation, and Emotion	HR	A low HR for Relaxation, an elevated HR for concentration, and an increase of HR during stress	PQ, and SUS
[162]	60	MP30 from Biopac System	Stress	Average of LF/HF ratio	Significant differences in the average ratios of LF/HF, as a function of plan configuration	Not applicable
[143]	18	e-Health Sensor Platform V2.0	QoE in terms of Quality, Frame-rate and Texture	Statistical features from HR: mean, min, max, median, std;	For Quality, no impact on the physiological responses	ACR, and SSQ
[144]	20	E4 from Empatica	Presence	Statistical features from HR: mean, LF, HF;	ECG features did not significantly vary between the presence and lack of factors of presence	PQ
[145]	33	g.USBamp and g.TRIGbox from g.tec	Mental immersion	Mean HR, and HRV;	HRV turned out to be significantly affected by network condition. Significant relationships between HRV and IEQ and gaming QoE	IEQ, Gaming QoE, and Video quality
[158]	10	Polar H10	User Experience	HRV, Time elapsed between two successive R-waves of the QRS signal (R-to-R interval), HF, LF, and VLF	HR and HRV are significantly different during resting once compared with the easy, medium, and hard difficulties	SSQ, Simulation performance, and Post-session interview
[146]	24	E4 from Empatica	Emotional responses	Mean HR, and std HR, root square of R-to-R, LF, and HF;	Higher classification accuracy of cognitive load against the HR data of 82.78%	Not applicable
[139]	24	ProComp Infinity from T &T	Gaming experience	Mean HR, and LF/HF ratio	all considered measures reported statistically significant increases due to playing in VR,	Demographics, System Usability Score, Visual Analogue Scale; SUS PQ
[147]	49	Brainproducts V-AMP 16	Fear effect on presence	Mean HR	Physiological responses in virtual heights leads to higher presence	Acrophobia Questionnaire, State-trait Anxiety Inventory, SSQ, MEC spatial, and PQ
[163]	33	E4 from Empatica	Influence of jerk on cybersickness	Inter-beat interval; HR	Lower HR with a high jerk effect. Correlation betweem HRs during collision periods and SS scores	IPQ, System Usability Scale, NASA task load assessment, SAM, and SSQ
[148]	600	Fitbit Charge	QoE	HR	A minor increase is noted in the tablet group as the mean HR increases by 1.8% over the test duration. The VR group experienced a slightly larger increase of 3.33%. Lastly, the AR group experienced the highest increase of 5.7%	Post-Test Questionnaire; video quality, audio quality, and audiovisual quality Questionnaire
[140]	24	Fitbit Charge	Emotions	Variation of HR and median for HR	Significant correlation between valence and variation of HR	Pre-test questionnaire, User Engagement Scale (UES), and SAM
[141]	13	BiosignalsPlux Explorer	Emotions	Mean HR, Mean RR	High emotional levels of valence and exaltation (SAM)	Self-assessment Manikin
[164]		BIOPAC’s MP150	Emotions	HRV, RR standard interval of the normal sinus of the human body, standard deviation of the difference between adjacent RR intervals	Mean and interval standard deviations of HR were equally significant in 2D and VR environments	SAM, Positive and Negative Affect Schedule, and SSQ
[142]	56	Biopac wireless sensors	Cybersickness	HF of HRV	augmentation of HF during the last minute	MSSQ
[149]	31	Zephyr OmniSense HR	SSQ

Electrodermal activity

Electrodermal activity (EDA), also known as galvanic skin response (GSR), measures the variation of the electrical conductance of the skin in response to sweat secretion. In the past, EDA has been associated with various aspects of psychological functioning, such as mechanisms underlying attention, information processing, emotion and stress. Several methods can be used to measure EDA, but a typical procedure consists of applying a constant voltage between two electrodes (commonly placed on the fingers, but also possible in other parts of the body, such as wrists and feet) to record conductivity variations, expressed in microsiemens (S). Three types of electrodermal measures are commonly used [150]: skin conductance level (SCL), representing a baseline measure of electrodermal conductance; nonspecific skin conductance responses (NS-SCRs), representing the frequency of spontaneous and momentarily changes in conductance, which are independent of external stimuli; and skin conductance responses (SCRs), which are momentary changes, similar to the NS-SCRs, but specifically elicited by external stimuli.

Table 5 lists the works that have relied on EDA signals for IMEx-related assessment. As can be seen, SCL peaks and amplitudes (and statistical measures over time) have been used to assess user experience, presence, and emotions. In particular, high sense of presence has shown to result in significantly more EDA peaks per minute than environment eliciting lower sense of presence [151]. Moreover, slow and steady increases in SCR have been shown to be correlated with cognitive activity [148]. EDA can also be attributed to an increase in mental workload or stress, as well as as significantly positively correlated with arousal states [124]. Lastly, during cybersickness events, researchers were able to observe a positive relationship between EDA responses and high jerk effects [145].

Table 5

List of works relying on EDA measurement for user IMEx assessment

Reference	# Subjects	Device	Measurement	Processing	Results	Questionnaire
[143]	18	e-Health Sensor Platform V2.0 for Arduino and Raspberry Pi	QoE in terms of Quality, Frame-rate and Texture	Peaks detection	For quality: no impact on the physiological responses	ACR, and SSQ
[144]	20	E4 from Empatica	Presence	Tonic and phasic decomposition	EDA features did not significantly vary between the presence and lack of factors of presence	PQ
[145]	30	g.USBamp, g.TRIGbox from g.tec hardware	Mental immersion	Peaks and amplitude in Skin Conductivity	No significantly effect of network condition and screen size on skin conductivity	IEQ, Gaming QoE, and Video quality
[146]	24	E4 from Empatica	Emotional responses	Mean, std, peak, strong peak, 20th percentile, 80th percentile, quartile deviation	EDA classification has returned low accuracy	Not defined
[139]	24	ProComp Infinity from T &T	Gaming experience	Skin conductance response (SCR)	Effect size revealed a large SCR	Demographics, System Usability Score, Visual Analogue Scale; SUS, and PQ
[147]	49	Brainproducts	Fear effect on presence	SCL	Physiological responses in virtual heights leads to higher presence	Acrophobia Questionnaire, State-trait Anxiety Inventory, SSQ, MEC spatial, and PQ
[163]	33	E4 from Empatica	Influence of jerk on cybersickness	Amplitude of SCR	Positive EDA responses with a high jerk effect	IPQ, System Usability Scale, NASA task load assessment, SAM, and SSQ
[148]	600	Pip Biosensor	QoE	SCL	Slow and steady increase in SCR can be correlated with an increase in cognitive activity, EDA can be attributed to an increase in mental workload or stress	Post-Test Questionnaire; video quality, audio quality and audiovisual quality Questionnaire
[124]	18	Shimmer3 Consensys	Determining affective responses	Skin conductance	Conductivity is significantly positively correlated with Arousal	Physical Activity Readiness Questionnaire (PAR-Q)
[151]	34	Shimmer3	Presence in video games	EDA amplitude and peak	High presence group had significant more EDA peaks/min than the low presence group	Demographics, MPS, SAM, and Emotional experiences questionnaire
[140]	24	Not defined	Emotions	Median and its variation	Fear situation: arousal is correlated with median of EDA	Pre-test questionnaire, UES, SAM
[149]	31	NeuLog EDA	Cybersickness	Average, percentage of change, min, and max of EDA	CNN-LSTM model can detect and predict cybersickness only the last two minutes of data with an accuracy of 97.44% and 87.38%	SSQ

Electroencephalograms

Electroencephalograms (EEG) measure electrical activity of the cortex using electrodes placed on the scalp. EEGs can be used to measure (cortical) neural activity in different parts of the brain, as well as connectivity patterns between different brain regions, which could be indicative of different affective states [152]. In fact, so-called affective EEG brain-computer interfaces have been used to model human influential factors for speech QoE modeling [24].

In recent years, several studies have explored the use of EEG for IMEx-related research. Tables 6 and 7, for example, list studies that have used different EEG features as correlates of IMEx parameters and cybersickness, in particular, respectively. As can be seen, event-related potential (ERP), ratio between the event-related desynchronization (ERD) and event-related synchronization (ERS), and statistical features such as the mean, the power of all frequency bands, and even the standard deviation of the EEG time series have been explored. EEG electrodes are typically positioned in the frontal, parietal, central, occipital, and temporal areas. Researchers have observed strong significant correlations between the subjective experience of presence and (i) the N1 ERP component (a large negative peak occurring roughly 80–120 ms post visual stimulus presentation) and (ii) the mismatch negativity (MMN), an ERP component resulting from the presentation of an odd stimulus in a sequence of stimuli, regardless of whether the subject is paying attention to the sequence or not [151]. Moreover, the total band power (1–45 Hz) in the frontal and frontal-left regions decreased with sense of presence and relative beta (16–30 Hz) and delta (below 4 Hz) powers increased in temporal and temporal right regions, respectively [144].

Table 6

List of works relying on EEG measurement for user IMEx assessment (excluding cybersickness)

Reference	# Subjects	Device	Electrode location	Measurement	Processing	Results	Questionnaire
[153]	36	8 channels LXE5208	Fp1, Fp2, F3, F4	Presence on affective responses and attitude	Arousal = Absolute β wave during stimulus - Absolute β wave during rest; level of arousal: Band-to-band β power	Increase in presence positively affected physiological arousal. Significantly higher arousal and attitude towards luge	Not defined
[151]	34	Advanced Brain Monitoring	Fz, F3, F4, Cz, C3, C4, POz, P3 and P4	Presence in video games	ERPv	Strong significant correlation between the subjective experience of presence and the early ERP components of N1 and MMN	Demographics, MPS, SAM, and Emotional experiences questionnaire
[157]	25	Mindwave	Fp1	Patient engagement	Banpowers; Engagement index: absolute power of β/(α+θ)	θ power in simulation conditions correlated with PQ scores to measure engagement an Increase of θ power and a decrease in α power	PQ
[158]	10	Biocybernetic Loop Engine	TP9, Fp1, Fp2, and TP10	Stress level	Absolute bandpowers; Engagement index; Frontal asymmetry index	Frontal θ values were significantly different between easy and hard difficulty levels	SSQ, Simulation performance, and Post-session interview
[138]	21	MyndPlay BrainBand	Fp1	User engagement, Concentration, Stress, Relaxation, and Level of emotion	Mean α, β, and θ	An elevated θ level and reduced α level for stress, and an elevated α level for relaxation, an elevated β level for concentration/focus,	PQ, and SUS
[199]	12	BrainVision 32 channel amplifier system	Fp1/2, F7/8, F3/4, Fz, FT7/8, FC3/4, T7/8, C3/4, Cz, TP7/8, CP3/4, CPz, P7/8, P3/4,Pz, O1/2, Oz and referenced to FCz	Spatial Presence	Eye blinks and muscles artefacts removal. Power of the α band. (ERD/ERS) = [(bandpower reference x band power test)/(band power reference)] x 100.	Strong spatial presence experiences are associated with increased ERD (cortical activity) in parietal/occipital areas of the brain together with decreased activity in frontal structures	Annett handedness questionnaire, and MEC-SPQ
[140]	24	Not defined	Not defined	Emotions	Median for all EEG bands	With Fear case: significant correlation between arousal and high γ and low β band and sensation of fear with with low γ band. Dominance is correlated with θ band. No fear case: user’s valence, arousal, and dominance, all of them with HighGamma band	Pre-test questionnaire, UES, and SAM
[144]	20	g.HIamp amplifier with the g.GAMMAcap2 EEG cap and g.SCARABEO active electrodes	F3, F4, T7, C3, C4, T8, P3, P4, PO7, PO8	Presence	Mean of EEG signal, Std of EEG signal, Signal power all frequency bands, Asymmetry index	band power in the frontal and frontal-left regions were decreased. The relative β and δ powers shows a significant increase in temporal and temporal right regions respectively	PQ
[200]	15	Neuroelectrics Enobio 32 using 8 gel-based AgCl electrodes	Fpz, F3, F4, Fz, P3, P4, Pz, Oz	Presence, Engagement, and Immersion	α and θ band power in frontal and parietal EEG	Increased α and θ band power following the VR exposure, θ band power significantly higher compare to baseline, α power increase reached statistical significance in the initial phase	NASA-TLX, and VR UX

Table 7

List of works relying on EEG measurement for cybersickness assessment

Reference	# Subjects	Device	Electrode location	Measurement	Processing	Results	Questionnaire
[159]	130 (65M, 65F)	Neurosky Mindwave Mobile	Fp1	Cybersikness	Low α, high α, low β, high β, θ, δ, low γ, and high γ waves, sickness detection including attention and meditation, and the sickness index: ∑α+∑θ∑β as input of binary LSTM network	Around 84% of accuracy for a window of 1, 5, and 10 mins	Not applicable
[201]	25	Emotiv Epoc+	AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4	Cybersickness	Image generation process of EEG data for an input to the CNN and DNN algorithms	Both algorithms gave 98% accuracy	Not applicable
[202]	202	Not defined	Fp1, Fp2, F3, F4, T3, T4, P3, P4	Cybersickness	Inter-correlation of the EEG channels and intra-correlation over the spectral and temporal information in each spectrogram as an input to a CNN	87% accuracy	SSQ
[203]	44	Emotiv Epoc EEG	AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, P7, P8, O1, O2	Cybersickness	EEG spectral power	Increase in spectral power, with respect to a baseline recording, is indicative of the onset of cybersickness	SSQ
[161]	18	OpenBCI system	FP1, FP2, C3, C4, P3, P4, O1 and O2	Cybersickness	Wavelet packet transform for EEG rhythm energy ratios of δ, θ, α and β	The average cybersickness recognition accuracy for single subject reaches 92.85%, and the cybersickness recognition accuracy to 18 subjects is also up to 79.25%	Not applicable
[160]	28	64-channel cap from AntNeuro	64-channel	Cybersickness	Relative power of each frequency band	Beta and LG showed significance only for the individuals suffering from headache, fullness of head, and blurred vision, while no other significances were found for an EEG parameter	MSSQ

EEGs have also been used to measure arousal states while in VR, especially via the use of the so-called frontal alpha (8–15 Hz) asymmetry index [153]. Alpha power changes were also seen with changes in attention levels in a target-response paradigm [154, 155]. Moreover, the ratios of theta (4–8 Hz), alpha and beta were used to assess alertness levels [156]. In [140], in turn, a fearful experiment showed significant correlations between arousal and the higher end of the gamma band powers and between arousal and (lower end) beta band power; the sensation of fear was shown to be correlated with the power in the lower end of the gamma band. In turn, dominance was shown to be correlated with theta band power. When fear was not induced valence, arousal, and dominance levels showed some correlations but only with the higher end of the gamma frequency band. Correlates of engagement in VR have also been proposed and they typically correspond to the ratio of the beta frequency band power (16–30 Hz) to the combined power in the alpha and theta ranges (4–15 Hz) [157, 158]. Lastly, the measurement of cybersickness using EEG has been proposed recently and deep neural network classifiers have been explored [159]. A sickness index relying on alpha, theta, and beta frequency subbands showed to achieved 84% in detecting cybersickness [160, 161].

In order to better understand the role of each brain region shown to correlate with sense of presence, some researchers have relied on functional magnetic resonance imaging or functional near-infrared spectroscopy to get a snapshot of which brain regions become active while in VR. In [165], for example, frontal, parietal and occipital regions showed involvement during free virtual navigation and activation in the dorsolateral prefrontal cortex was shown to be negatively correlated to sense of presence, hence corroborating some of the EEG findings. In turn, brain regions responsible for balance and vestibular (located in the cerebellum) inputs were shown to be active during cybersickness events [166].

Multiple-sensorial media applications

The majority of current applications stimulate only one (visual) or two senses (audio-visual). As the tactile Internet and Internet of Senses revolutions emerge, additional senses will be stimulated, including olfactory and somatosensory. Such media has been termed multiple-sensorial media, or mulsemedia, and within a VR framework, could lead to next-generation immersive systems with increased sense of realism and immersion [9, 167]. For example, inclusion of smells [168, 169] and haptics [170] have shown to improve sense of immersion. Haptics can be used to provide the user with cues about physical characteristics of an object (e.g., weight or texture) hence increase sense of realism [171, 172]. Vibrotactile feedback, in turn, provides feedback when interacting with virtual devices [173], thus also improving the sense of realism [174]. Table 8 lists some of the emerging work on mulsemedia QoE assessment via psycho-physiological methods. As an example, when using haptic gloves, a strong amplitude modulation or ERP signals occurred when the participants selected virtual objects and significant changes in the early negativity component of the ERP was seen during situations with haptic conflicts [175]. Moreover, while addition of olfactory stimuli showed a significant increase in sense of presence, it did not generate significant changes in EDA [176], hence suggesting that alternate modalities may be needed. It is important to emphasize that while our search period encompassed papers from 2015 to 2021, a great number of works in the mulsemedia domain appeared prior to 2015. The interested reader could refer to [177] for a detailed review of mulsemedia systems proposed prior to 2015.

Table 8

List of works using psycho-physiological measurements to assess QoE of immersive mulsemedia applications

Articles	# Subjects	Modality	Sense	Device	Measurement	Processing	Results	Questionnaire
[204]	27	ECG, EDA	Olfactory, Haptics, Thermal, Wind	Ambiotherm	Presence	HR, and SCL	A rise in HR was observed at the onset of the different wind/thermal stimuli and towards the end of the olfactory stimuli. Higher EDA values, which represent high arousal, have been noted to correlate with Negative Affect	Game experience Q, and PQ
[176]	60	EDA	Olfactory	Mindware MW3000A, Dreamreapers Inc.	Augment the exposure therapy process	Event related SCR	Olfactory stimuli increase presence but not EDA	IPQ, Quick Smell Identification Test, State-Trait Anxiety Inventory, Immersive Tendencies Questionnaire, Presence Visual-Analogue Scale
[175]	11	EEG	Haptics gloves	Model308-100, BrainProducts 64 chan	Detect conflicts in visuo-haptic sensory integration	ERPs	Strong amplitude modulation occurring when selection of objects; The early negativity component of the ERP is more pronounced during situations with haptic conflicts	Not applicable

Concluding remarks and suggestions for future work

Monitoring of human behaviour and psycho-physiological signals while immersed in virtual reality will allow models of human influential factors to be built, including to e.g., detect and even predict cybersickness, monitor the user’s perception of immersion and sense of presence, as well as overall immersive media experience. Ultimately, this information will allow for virtual environments and applications to be adjusted per user, thus maximizing the user experience. As emphasized by [178], the success or failure of any system for immersive communication will rely on the user experience that it provides and not necessarily on the technology it uses. Building IMEx systems that take into account system, context and human influential factors will be crucial for the development of the field.

Today, the most widely used measure of user IMEx remains subjective assessment. While subjective assessment can directly target specific IMEx dimensions (e.g., presence, immersion, cybersickness) with high validity, it requires offline evaluation, can be biased by subject responses [179], can be disruptive to the user experience with constant prompts, which, in turn, can increase the user’s cognitive load and indirectly affect the experience. Disruptions to answer subjective questionnaires can break the immersive experience and studies have reported that it can take some time before the sense of immersion is recovered post interruption [180]. Moreover, while VR-based questionnaires have been developed to replace paper-and-pencil ones (e.g., [33, 34]), their validity over time has yet to be confirmed. Future studies should explore this.

Furthermore, it is known that VR sickness drastically hampers IMEx. Studies have reported that women and children are more susceptible to cybersickness than men [181, 182], mostly due to a poor fit of their interpupillary distance to the VR headset [183]. More recently, an effect of smoking has also been reported [184]. While exposure and habituation can drastically reduce the prevalence and severity of cybersickness symptoms [185], especially over multiple sessions [186], getting through the first 20 min is crucial [187]. As such, being able to predict cybersickness at the beginning of a VR session could allow for mitigation strategies to be put in place “on the fly”, such as bringing in additional sensory modalities [188], hence improving the overall IMEx. Psycho-physiological measures are crucial for such real-time cybersickness evaluation. As shown here, however, only few works exist that have focused on cybersickness prediction, hence there is ample room for research. In particular, the methods relying on EEG signals have all used stand-alone EEG systems worn under the VR headset. This could lead to discomfort, hence indirectly affecting the IMEx. Future works should explore tools with sensors directly embedded into the VR headset, such as [31], where improved usability has been reported [189]. Moreover, the developed tools have mostly relied on deep neural networks, which could be power and storage hungry, thus not suitable for untethered applications in which the user is truly mobile. As such, future work should explore the use of feature engineering to find more robust features that can be coupled with simpler machine learning algorithms, as shown in [190].

As 5G and 6G wireless communications become more widespread, truly portable VR applications are emerging where the user is completely mobile and untethered to a computer [191]. Movement is known to generate artifacts that affect psycho-physiological signal quality and hamper human influential factors assessment. Existing enhancement algorithms, however, especially those developed for EEG signals, have not been tailored to such artifacts [192], hence new algorithms and movement artifact robust features will need to be developed. Adaptive systems are already starting to emerge (e.g., [193]) but further work is needed. Moreover, multimodal systems have shown to be useful in such mobile conditions where multiple signal modalities can account for certain confounding factors (e.g., fatigue on HRV) [194], but such systems within a portable immersive application are still needed to learn what confounding factors exist within an IMEx (e.g., how does fatigue effects on HRV affect the presence-related HRV features?). Future work should focus on better understanding these confounds.

And as wireless communication bandwidths and coverage increase, and latency decreases, future generation technologies, such as the tactile internet [195] or the Internet of Senses [196] will become the mainstream. In such scenarios, multiple senses will be stimulated, including olfactory, taste, and somatosensory systems, hence drastically improving the IMEx. As highlighted in [188], smells have already been explored and shown to reduce cybersickness symptoms, as have pleasant songs. Haptic feedback, in the form of vibrations and airflow time-aligned with visual cues, have also helped increase overall experience. The effect that such multisensory stimuli has on behavioural and psycho-physiological signals is still not well understood and only a few works have explored this direction (see Table 8). Future work should focus on multi-sensorial media and the overall impact it has on IMEx, including possible timing mismatch between different modalities.

Lastly, as highlighted by [5], IMEx is multi-faceted. Most of the works surveyed have touched only one or a few of the influential factors, hence only show a snapshot of what can be achieved with IMEx assessment. Recently, a unified user experience questionnaire was proposed containing 10 subscales to measure presence, engagement, immersion, flow, usability, skill, emotion, experience consequence, judgement, and technology adoption [93]. Future studies should explore the use of behavioural and psycho-physiological metrics to measure each of these components and measure their individual contributions to overall IMEx. Initial steps in this direction have been taken for speech (e.g., [24]), image [197], and video applications [198] where more than one influential factor has been explored and combined. Limited work exists, however, with immersive and mulsemedia applications. Future work should aim to fill this gap.