Examining the Academic Trends in Neuropsychological Tests for Executive Functions Using Virtual Reality: Systematic Literature Review.

BACKGROUND: In neuropsychology, fully immersive virtual reality (VR) has been spotlighted as a promising tool. It is considered that VR not only overcomes the existing limitation of neuropsychological tests but is also appropriate for treating executive functions (EFs) within activities of daily living (ADL) due to its high ecological validity. While fully immersive VR offers new possibilities of neuropsychological tests, there are few studies that overview the intellectual landscape and academic trends in the research related to mainly targeted EFs with fully immersive VR.
OBJECTIVE: The objective of this study is to get an overview of the research trends that use VR in neuropsychological tests and to analyze the research trends using fully immersive VR neuropsychological tests with experimental articles.
METHODS: This review was carried out according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Articles were searched in three web databases using keywords related to VR, EFs, and cognitive abilities. The study was conducted in two steps, keyword analysis and in-depth systematic review. In the web database search from 2000 to 2019, 1167 articles were initially collected, of which 234 articles in the eligibility phase were used to conduct keyword analysis and a total of 47 articles were included for systematic review.
RESULTS: In keyword analysis, the number of articles focused on dementia including the keywords "MCI," "SCD," and "dementia" were highlighted over the period, rather than other symptoms. In addition, we identified that the use of behavioral and physiological data in virtual environments (VEs) has dramatically increased in recent studies. In the systematic review, we focused on the purpose of study, assessment, treatment, and validation of usability and structure. We found that treatment studies and uncategorized studies including presence and cybersickness issues have emerged in the recent period. In addition, the target symptoms and range of participants were diversified.
CONCLUSIONS: There has been a continuously increasing interest in dealing with neuropsychology by using fully immersive VR. Target cognitive abilities have been diversified, as well as target symptoms. Moreover, the concept of embodied cognition was transplanted in this research area. ©Euisung Kim, Jieun Han, Hojin Choi, Yannick Prié, Toinon Vigier, Samuel Bulteau, Gyu Hyun Kwon. Originally published in JMIR Serious Games (https://games.jmir.org), 24.11.2021.

Introduction

Virtual reality (VR) is a state-of-the-art technology at present. With this technology, we can create realistic worlds from the real world and generate artificial experiences in real time [1]. For the past decade, technological progress has enabled VR to become more popular through decreasing costs and increasing convenience.

With regard to defining mixed reality, Milgram and Kishino [2] suggested the concept of the reality-virtuality continuum. It is powerful to overview the dimension of reality. Considering the scope of our research, we partly adopted their perspective and focused on “immersion” to classify VR. Immersion in VR can be explained as the perception that is created by various stimuli that provide an absorbing environment so that users think they are physically in a virtual world [3]. The VR system can be divided into fully immersive and semi- or non-immersive, depending on the degree of immersion [4]. Fully immersive VR provides 3D displays (eg, head-mounted display [HMD]) that effectively make the user feel they are existing inside the virtual environment (VE) for the highest level of immersion. Non-immersive VR is based on a desktop with a flat-screen monitor with low interaction (eg, personal computer, tablet). With a large, curved monitor or projector, semi-immersive VR offers moderate immersion and interaction (eg, Kinect). Thomas et al [5], who studied the impact of immersion with a virtual avatar, said that the degree of immersion significantly influences the virtual body ownership and the feeling of presence. In addition, an enhanced effect of fully immersive VR on presence was reported by Waltemate et al [6]. In this study, we define fully immersive VR as a cover with up to a 360° screen-like Cave Automatic Virtual Environment (CAVE), dome-shaped screens, and HMDs [7].

Neuropsychological assessment is one way of examining the brain by its behavioral product [8]. Neuropsychological tests are typically used to assess/treat one’s cognitive abilities and further diagnose mental illness. By using neuropsychological tests, findable neuropsychological domains are extensive, for example, attention/concentration, language, visuospatial abilities, motor coordination, and executive functions (EFs) [9].

There are some popular traditional neuropsychological tests such as the Stroop test and the trail-making task [10,11]. These tests are widely accepted in professional society. Nevertheless, there are a few questionable limitations. One of the most critical limitations that is commonly pointed out is the lack of ecological validity. It is difficult to exactly measure the cognitive demands of real-world activities of daily living (ADL) with existing traditional neuropsychological tests [12,13].

As mentioned, the progress in VR technology gives rise to the possibility of applying existing neuropsychological tests in VR as an advanced model of the tests [9]. Pen-and-paper tests and computerized tests are widely used in neuropsychology, and they are composed of a set of predefined stimuli that exist in a controlled environment [14]. It is still necessary to increase ecological validity in measuring real-world performance or cognitive impairment [15-18]. However, VR environments can provide a realistic experience by multisensory stimulation and interactive factors as in daily life [19,20], with a strong sense of presence (“being there”) [21]; the means to test multitasking ability [22]; the influence of distractors that may not be used in a real-world laboratory [22]; and a good level of motivation of participants [23]. By using VR neuropsychological tests, it is possible to measure and evaluate cognitive abilities and EFs in a daily-living situation, even the instrumental activities of daily living (IADL) [24], and also, collecting behavioral data is enabled. IADL indicate activities using instruments that allow an individual to improve their quality of life, such as cooking, cleaning, and managing finance. There are many previous studies that have reported the specific connection between IADL and EFs [25-28]. These support that VR neuropsychological tests are significant, in that the IADL situation can be experimentally implemented through VR to measure or manage related cognitive functions. Therefore, VR can be considered an alternative or complementary innovative neuropsychological tool [29].

The term “executive function” was first defined by Muriel Deutsch Lezak in 1982 [30]. She said that “executive functions comprise those mental capacities necessary for formulating goals, planning how to achieve them, and carrying out the plans effectively” and proposed four classes as capacities of (1) formulating goals, (2) planning, (3) carrying out the plan, (4) and performing effectively [30]. However, the definition and functional categories of EFs are prescribed slightly differently by other researchers. Anderson [31] described EFs as “a collection of inter-related processes responsible for purposeful, goal-directed behavior” such as “anticipation, goal selection, planning, initiation of activity, self-regulation, mental flexibility, deployment of attention, and utilization of feedback.” Similarly, Hughes [32] described an EF as “a complex cognitive construct that encompasses the set of processes that underlie flexible goal-directed behavior (e.g., planning, inhibitory control, attentional flexibility and working memory).”

Therefore, there is a common agreement for categorizing three core EFs in general [33,34]: inhibition, working memory, and shifting (also called cognitive flexibility). Higher-order EFs, such as decision making (also called reasoning), problem solving, and planning, are usually established by extending from the core EFs [35,36]. Moreover, one of the core EFs, working memory, can be divided into more detail, such as general working memory, which holds information in the mind, and updating, which conducts translating instructions into action plans and involving new information into thinking or action plans [37].

In this paper, with advice from experts in neuropsychology based on the theoretical background, we categorized the EFs into seven sub-abilities:

Inhibition: The ability to control impulsive and automatic responses and generate responses using attention and reasoning

Working memory: The ability to temporarily store and handle information in order to do complex cognitive tasks

Shifting: The ability to adapt your thoughts and behaviors to new, changing, and unexpected situations

Decision making: The ability to efficiently and thoughtfully choose an option among different alternatives

Problem solving: The ability to come to a logical conclusion when considering an unknown

Planning: The ability to think about future events and mentally anticipate the correct way to carry out a task or reach a specific goal

Updating: The ability to supervise behavior and ensure that you are properly carrying out the established plan of action

These days, fully immersive VR has been in the spotlight because it is considered an innovative tool to exceed the existing limitations of neuropsychological tests. Consequently, researchers who are studying in related areas have worked to improve the quality of VR and optimally apply it to neuropsychological tests. Many earlier studies have been conducted in the neuropsychological area with VR technology. However, there are only few studies that use fully immersive VR to mainly target EFs.

This systematic review article aims (1) to get an overview of the research trends using VR (including non-immersive) to conduct neuropsychological tests targeting EFs from 2000 to 2019 by keyword analysis and (2) to analyze the specific research trends using fully immersive VR with experimental articles.

Methods

Registration

This systematic review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [38]. In this study, we present a systematic review to summarize the past studies in the neuropsychological area with VR. Because there were few studies before 2000, the range of publishing years was limited from 2000 to 2019. The earlier studies were also significant and important. However, we assumed most of the concepts and topics in those studies could reflect on future studies, since there were a negligible number of studies that were related to VR and neuropsychological tests.

Search Strategy

Initial articles were searched in three web databases: Scopus, Web of Science, and PubMed. These engines are adequate to cover the wide spectrum of topics in the target area. The articles were searched in March 2020, and the search query string was as follows: (“virtual reality” OR “VR” OR “virtual environment”) AND (“executive function(s)” OR “EF(s)” OR “cognitive function(s)” OR “cognition”). The publishing years were limited from 2000 to 2019.

Inclusion Criteria

Journal articles and conference (proceeding) papers written in English were chosen, and other forms of publication, such as short reports, chapters of books, and dissertations, were excluded. To aim more deeply at EFs among the cognitive abilities, we included articles that were only related to “executive function” and “virtual reality” were used to see the effectiveness of neuropsychological tests in VR.

Exclusion Criteria

After duplicates were removed, all studies were checked by using keywords and abstracts in the screening process. Review and protocol articles and chapters of books were all excluded. Moreover, articles not mainly focusing on EFs were excluded. At the end of the screening process, 234 studies were left.

Evaluation of full-text articles was conducted during the eligibility process. In this phase, we excluded theoretical articles, reporting articles, and non-experimental articles. The term “virtual reality” commonly corresponds not only to articles about immersive environments but also to articles about non-immersive environments, such as the use of a flat-screen monitor, so this study excluded them as not fully immersive VR articles. A few records where we could not find the full text were also excluded.

Study Selection

The overall procedure of the study and the number of selected articles are shown in Figure 1. After searching the web database, 1167 articles were collected (Scopus 868, PubMed 102, and Web of Science 197). After removing duplicates and outliers, the remaining articles were used in the next phase. As a result of the screening phase, 702 articles were excluded by titles, keywords, and abstract review. With abstracts of 234 papers (data set 1), keyword analysis was conducted to examine the research trends. Then, we conducted a full-text review in the eligibility phase, and 187 papers were excluded based on the exclusion criteria. Finally, 47 papers (data set 2) were selected for the descriptive analysis.

Figure 1

Flowchart of the process of selecting studies. EF, executive function; VR, virtual reality.

Data Analysis

Data analysis was conducted by two sub-analysis methods.

First, a keyword analysis was conducted with data set 1. It was important to specify keywords for the analysis, so we conducted keyword extraction with Python NLTK. The Natural Language Toolkit (NLTK) is a Python package that has been developed for natural language processing and document analysis. The NLTK extracted top 200 keywords based on their frequency from each article’s title, keywords, and abstract. Among the results of the extraction, we selected significant keywords in line with the study’s purpose and categorized them according to measuring targets, names of mental disorders, and behavioral and physiological measures. Moreover, some keywords were added from the 47 full-text descriptive analysis.

Second, a descriptive analysis was conducted in detail with data set 2. The analysis was conducted with almost the same categories as the keyword analysis, but we also added some other elements: purpose of article, measuring targets, age of participants, names of mental disorders, real-time walking, and environment of VR.

All the analyses were conducted within a certain timescale. Considering the evolution of yearly publication frequencies, the time periods were divided into 4 stages of 5-year blocks: period 1, 2000-2004; period 2, 2005-2009; period 3, 2010-2014; and period 4, 2015-2019.

Results

Evolution of the Publication Frequency

In this section, we give an overview of the study of virtual neuropsychological tests in connection with EFs. There was a change of pace of the publication frequency from 2000 to 2019, with an exponential curve, as shown in Figure 2, with 936 articles after removing duplicates and outliers. Studies maintained a certain level all through the 2000s. The publication frequency began to increase conspicuously in 2009 and has rapidly increased in the past 10 years. It expresses the high interest in VR for dealing with mental disorders.

Figure 2

The evolution of publication frequency.

Keyword Analysis

A keyword analysis was applied to data set 1 that mainly focused on EFs. For the keyword analysis, we segmented the total period (2000-2019) into four periods: 2000-2004, 2005-2009, 2010-2014, and 2015-2019. A significant increase in the publications in Figure 2 was the main rationale to classify the periods. In the keyword analysis, we examined the trends in VR applying to neuropsychology.

Target Cognitive Abilities

Cognitive ability includes various information-handling processes that occur in the human brain [39]. Target EFs brought from the theoretical background were complemented with “memory,” “attention,” and “spatial” since these frequently appeared with keyword extraction. Memory and attention are wide and basic concepts when assessing cognitive abilities [39,40]. They had a high proportion through all periods (Figure 3) because they occurred when measuring specific cognitive abilities together.

Figure 3

The evolution of cognitive abilities targeted in the 234 screened studies.

In the early 2000s, period 1, all studies tried measuring basic abilities, memory, and attention, but researchers began to try to measure other targets, including higher-order EFs. It seems that the diversification accelerated in line with the popularization of VR in the late 2000s.

Target Symptoms

Our mental disorder list was formed by the result of keyword extraction, and we manually added several mental disorders from the 47 articles we reviewed for full-text descriptive analysis. There was a transition stage from the late 1990s to the early 2000s, when traditional pen-and-paper tests turned into computerized tests [41]. As shown in Figure 4, only a few studies conducted experimental attempts to apply VR to neuropsychology. Period 1 was the beginning stage of using VR technology for neuropsychological assessment, and this was ongoing through period 2. After that, from the early 2010s, studies expanded to treat various types of mental disorders. Until period 3, brain injury (stroke is included in brain injury in a broad view) was the most common mental disorder in this research area, maybe due to its high incidence and fatal long-term consequences [42]. However, in period 4, studies gradually spread to other brain disorders.

Figure 4

The evolution of symptoms targeted in the 234 screened studies. AD: Alzheimer’s disease; ADHD, attention deficit hyperactivity disorder; MCI: mild cognitive impairment; SCD: subjective cognitive decline.

Through period 4, research on the topics mild cognitive impairment (MCI), subjective cognitive decline (SCD), dementia, Alzheimer’s disease (AD), attention deficit hyperactivity disorder (ADHD), and schizophrenia increased. Among them, MCI, SCD, AD, and dementia are age-related symptoms that increased from period 3 (36.4%) to period 4 (43.8%). In this article, we view SCD, MCI, AD, and dementia as one continuous process and call it “age-related cognitive decline.” In addition, studies on ADHD increased from period 3 (6.1%) to period 4 (11.2%), and studies on schizophrenia increased from 3.0% to 9.0%. Despite a high prevalence compared with schizophrenia, autism, ADHD, and depression remained less investigated.

Use of Behavioral and Physiological Data

Attempts to obtain a subject’s behavior data continuously increased from the early 2000s, while attempts to obtain physiological data rapidly increased in period 4 (Figure 5). In terms of content, there were attempts to use head tracking, gaze (eye) tracking, body tracking, hand tracking, and gait (leg) tracking, as shown in Figure 5. In the graph for physiological data, there were studies completed in EDA, HRV, NIRS, and respiration measurement, but EDA was the only attempt before period 4, and the others were conducted in period 4.

Figure 5

(Left) The use of behavioral data over each period. (Right) The use of physiological data over each period. EDA: electrodermal activity; HRV: heart rate variability; NIRS: near-infrared spectroscopy.

Descriptive Analysis

The final 47 articles included through PRISMA were experimental studies that conducted neuropsychological tests using fully immersive VR. The article set was divided into the same four periods as the keyword analysis. Studies on fully immersive VR were encouraged in the early 2010s, and most of the research was done within the past 5 years (period 4) with every year. Table 1 represents a summary of the 47 studies.

Table 1

Summary of the characteristics of the 47 studies.

Study	Journal/conference	Purpose of study	Target cognitive functions	Subject status	VEa	BEb
Areces et al (2018) [43]	PLoS ONE	AscVal.Sd	AttentionInhibition	ADHDeAge 6-16 years	Classroom	—
Bailey et al (2019) [44]	Journal of Applied Developmental Psychology	Unf	Inhibition	Normal onlyAge 4-6 years	Other(s)	—
Blume et al (2017) [45]	Trials	TrgVal.Uh	AttentionInhibitionWorking memory	ADHDAge 6-10 years	Classroom	Oi
Chicchi et al (2019) [46]	Applied Neuropsychology: Adult	Val.U	AttentionDecision makingInhibitionPlanningShifting	Normal onlyMean age 30.14 years	Other(s)	—
Chicchi et al (2019) [47]	Frontiers in Psychology	Un	AttentionPlanningShifting	Normal onlyAge 18-48 years	Kitchen	O
Climent et al (2019) [48]	Applied Neuropsychology: Adult	Un	AttentionWorking memory	Normative studyAge 16-90 years	Aquarium	—
Coleman et al (2019) [49]	Frontiers in Psychology	Tr	AttentionWorking memory	ADHDAge 6-13 years	Classroom	—
Dahdah et al (2017) [50]	Neurorehabilitation	TrVal.U	Inhibition	Stroke, TBIjMean age 40.3 years	ApartmentClassroom	—
Davison et al (2018) [51]	Acta Neuropsychiatrica	Val.SVal.U	PlanningWorking memoryMotor	Normal onlyAge 19-24, 66-77 years	Parking lotLaboratory	—
De Lillo et al (2014) [52]	American Journal of Primatology	Un	Working memory	Normal onlyAge 18-36 years	Other(s)	—
De Luca et al (2020) [42]	Applied Neuropsychology: Child	Tr	InhibitionPlanningProblem solvingUpdatingWorking memoryMotorSpatial	TBIAge 15 years	A number of VEs	O
Díaz-Orueta et al (2014) [53]	Child Neuropsychology	AsVal.U	AttentionInhibition	ADHDAge 6-16 years	Classroom	—
Eom et al (2019) [54]	Cyberpsychology, Behavior and Social Networking	AsUn	AttentionInhibition	ADHDAge 6-17 years	Classroom	—
Grewe et al (2013) [55]	Journal of NeuroEngineering and Rehabilitation	Val.S	Working memorySpatial	EpilepsyAge 25-47 years	Market	—
Grewe et al (2014) [56]	Epilepsy and Behavior	AsVal.S	Working memorySpatial	EpilepsyAge>18 years	Market	—
Henry et al (2012) [57]	Journal of Neuroscience Methods	Val.U	AttentionInhibitionMotor	Normal onlyAge 19-58 years	Apartment	—
Huang (2020) [58]	Cyberpsychology Behavior and Social Networking	TrUn	InhibitionShiftingWorking memory	Normal onlyAge>50 years	Other(s)	—
Ijaz et al (2019) [59]	Journal of Medical Internet Research	AsVal.S	Working memorySpatial	Normal onlyMean age 73.22 years	Google Street View	O
Iriarte et al (2012) [60]	Journal of Attention Disorder	Un	AttentionInhibition	Normative studyAge 6-16 years	Classroom	—
Kang et al (2008) [61]	Cyberpsychology and Behavior	As	AttentionPlanningWorking memory	StrokeMean age 55.4 years	Market	—
Lalonde et al (2013) [29]	Journal of Neuroscience Methods	Val.U	Inhibition	Normal onlyAge 13-17 years	Classroom	—
Liao et al (2019) [62]	Frontiers in Aging Neuroscience	TrVal.U	AttentionInhibitionPlanningShiftingWorking memoryMotor	MCIkAge>65 years	A number of VEs	O
Lo Priore et al (2003) [63]	Cyberpsychology and Behavior	Un	AttentionPlanningShifting	Normal onlyYoung age	Market	O
Mondellini et al (2018) [64]	2018 IEEE 6th International Conference on Serious Games and Appli cations for Health, SeGAH 2018	Tr	Working memory	Normal onlyAge<40 years	Market	—
Negut et al (2017) [65]	Child Neuropsychology	AsVal.SVal.U	AttentionInhibition	ADHDAge 7-13 years	Classroom	—
Nolin et al (2009) [66]	Annual Review of CyberTherapy and Telemedicine	Val.U	AttentionInhibition	TBIAge 8-12 years	Classroom	—
Nolin et al (2016) [67]	Computers in Human Behavior	Val.SUn	AttentionInhibition	Normal onlyAge 7-16 years	Classroom	—
Ouellet et al (2018) [68]	Journal of Neuroscience Methods	Val.SUn	Working memory	SCDlMean age 67.2 years	Market	—
Parsons et al (2013) [69]	Journal of Clinical and Experimental Neuropsy chology	Val.U	AttentionInhibition	Normal onlyAge 18-28 years	In a car	O
Parsons et al (2016) [70]	Journal of Autism and Developmental Disorder	AsVal.U	Inhibition	AutismAge 18-34 years	Classroom	—
Parsons et al (2017) [71]	Journal of Neuroscience Methods	AsVal.S	Working memory	Normal onlyAge>17 years	Market	—
Parsons et al (2018) [72]	IEEE Transactions on Affective Computing	As	AttentionInhibition	Normal onlyAge 18-28 years	In a car	O
Parsons et al (2018) [73]	Journal of Neuroscience Methods	Val.U	Inhibition	Normal onlyAge 18-30 years	Apartment	—
Parsons et al (2019) [74]	Applied Neuropsychology: Adult	As	AttentionInhibition	Normal onlyOlder adults and undergraduate young adults	Apartment	—
Plechatá et al (2019) [75]	Frontiers in Psychology	AsUn	Decision makingShiftingWorking memory	Normal onlyAge 19-39, 60-91 years	Market	—
Plotnik et al (2017) [76]	International Conference on Virtual Rehabilitation (ICVR)	Val.S	AttentionPlanning	Normal onlyMean age 37.1 years	Other(s)	O
Proulx et al (2018) [77]	Annual Review of CyberTherapy and Teleme dicine	Un	AttentionInhibitionWorking memory	Normal onlyAge 20-60 years	LaboratoryTent	—
Rizzo et al (2000) [78]	Cyberpsychology and Behavior	As	AttentionInhibition	ADHDAge 8-12 years	Classroom	—
Robitaille et al (2017) [79]	Disability and Rehabili tation: Assistive Technology	As	AttentionMotor	TBIMean age 30.3 years	Middle-east Village	O
Serino et al (2018) [80]	Sensors (Switzerland)	As	Working memorySpatial	ADmAge>65 years	Other(s)	—
Tarnanas et al (2013) [81]	International Conference on Virtual Re habilitation, ICVR 2013	AsUn	InhibitionPlanningWorking memoryMotor	MCIAge >60 years	Apartment	O
Tarnanas et al (2013) [82]	Journal of Medical Internet Research Serious Game	AsVal.S	InhibitionShiftingUpdating	MCI, ADAge>65 years	Apartment	O
Tarnanas et al (2014) [83]	Alzheimer’s and Dementia	As	InhibitionShiftingUpdating	MCIMean age 71.6 years	Apartment	O
Voinescu et al (2019) [84]	26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 – Proceeding	Val.SVal.U	AttentionInhibition	Normal onlyAge 23-51 years	Aquarium	—
Yasuda et al (2017) [85]	Topics in Stroke Rehabilitation	TrVal.S	Spatial	StrokeAge 45-85 years	Other(s)	—
Yeh et al (2012) [86]	2012 IEEE-EMBS Conference on Biomedical Engineering and Scien ces, IECBES 2012	As	AttentionDecision makingShiftingWorking memory	ADHDAge 7-13 years	Classroom	O
Yeh et al (2012) [87]	2012 IEEE-EMBS Conference on Biomedical Engineering and Scien ces, IECBES 2012	As	PlanningWorking memory	DementiaAge 60-90 years	Market	O

aVE: virtual environment.

bBE: behavior or ecological measures.

cAs: assessment.

dVal.S: structure validation.

eADHD: attention deficit hyperactivity disorder.

fUn: uncategorized.

gTr: treatment.

hVal.U: usability validation.

iO: includes BE measures.

jTBI: traumatic brain injury.

kMCI: mild cognitive impairment.

lSCD: subjective cognitive decline.

mAD: Alzheimer’s disease.

Analysis of Purpose of the Studies

This study identified that the purpose of the 47 articles is mainly clustered into four groups, and some additional purposes also existed. Thus, in this study, we divided the articles into five groups: assessment, treatment, usability validation, structure validation, and, for additional purposes, uncategorized: (1) assessment: to assess, diagnose, or evaluate a neuropsychological condition of the subject; (2) treatment: to treat or rehabilitate the patient; (3) usability validation: to validate the efficiency of a VR application by comparing it with an existing traditional neuropsychological test; (4) structure validation: to attempt to test the construct integrity and ecological and temporal stability of the VR itself; and (5) uncategorized: to evaluate presence, cybersickness, aging, and normative studies. Since there could be multiple purposes in one article, we checked all articles for all purposes.

As shown in Figure 6, the majority of the studies were conducted for assessment and validation. The number of validation studies was nearly steady through all periods. The number of studies dramatically increased as we moved to period 3. In period 3, 7 studies (41.2%) reported the results of assessing EF(s), 7 studies (41.2%) tried to validate their VR neuropsychological tests, but there were no studies for applying a VR test to treatment until period 4. Treatment studies first appeared in 2017, and the tendency began to expand. In period 4, 10 (21.7%), 8 (17.4%), and 18 (39.2%) studies were conducted on assessment, treatment, and validation, respectively.

Figure 6

The evolution of the objective of the final 47 studies.

Meanwhile, the proportion of uncategorized studies increased rapidly in period 4. Before period 4, there were just 4 studies, but 10 studies corresponded to the uncategorized type. Of the four studies, one was a study for the sense of presence that was conducted early in VR history, and the others were conducted in period 3 to obtain information about certain cognitive functions associated with functional impairment in ADL. However, the uncategorized studies in period 4 were slightly different: 7 (70%) of the 10 studies were conducted to evaluate the sense of presence and cybersickness of users in the VE, and the remainder were studies on age-specific differences.

Cognitive Abilities and VEs

The VEs used to target EF(s) are shown in Table 2. In addition to the cognitive abilities selected in the Target Cognitive Abilities section, we added spatial (navigation, spatial perception, etc) and motor (psychomotor, motion, movement, etc) abilities, because these are significant features within fully immersive VR tests. In general, most articles not only target a certain cognitive ability but also treat several abilities, so we allowed for multiple checking.

Table 2

Frequency of VEsa by EFsb.

Cognitive ability	VEs	Reference	Number of papers that studied the specific cognitive ability (n)
Inhibition (n=27)	Apartment	[50,57,73,74,81-83]	7
	Classroom	[29,43,45,50,53,54,60,65-67,70,78]	12
	Aquarium	[84]	1
	Laboratory	[77]	1
	Tent outdoors	[77]	1
	In a car	[69,72]	2
	Othersc	[42,44,46,58,62]	5
Working memory (n=21)	Laboratory	[51,77]	2
	Parking lot	[51]	1
	Market	[55,56,61,64,68,71,75,87]	8
	Classroom	[45,49,86]	3
	Tent outdoors	[77]	1
	Google Street View	[59]	1
	Aquarium	[48]	1
	Others	[42,52,58,62,80]	5
Shifting (n=9)	Apartment	[82,83]	2
	Market	[63,75]	2
	Classroom	[86]	1
	Kitchen	[47]	1
	Others	[46,58,62]	3
Decision making (n=3)	Classroom	[86]	1
	Market	[75]	1
	Others	[46]	1
Problem solving (n=1)	Others	[42]	1
Planning (n=10)	Parking lot	[51]	1
	Laboratory	[51]	1
	Market	[61,63,87]	3
	Apartment	[81]	1
	Kitchen	[47]	1
	Others	[42,46,62,76]	4
Updating (n=3)	Apartment	[82,83]	2
	Others	[42]	1
Attention (n=24)	Classroom	[43,45,49,53,54,60,65-67,78,86]	11
	Market	[61,63]	2
	Aquarium	[48,84]	2
	In a car	[69,72]	2
	Kitchen	[47]	1
	Laboratory	[77]	1
	Tent outdoors	[77]	1
	Apartment	[57]	1
	Middle-east village	[79]	1
	Others	[46,62,76]	3
Spatial (n=6)	Market	[55,56]	2
	Google Street View	[59]	1
	Others	[42,80,85]	3
Motor (n=6)	Parking lot	[51]	1
	Laboratory	[51]	1
	Apartment	[57,81]	2
	Middle-east village	[79]	1
	Others	[42,62]	2

aVE: virtual environment.

bEF: executive function.

cOthers: does not use specific environments, uses multiple environments as a minigame with multiple scenarios, or uses a certain unrealistic environment that could not be experienced in real life.

Since the study of fully immersive VR began, numerous VEs have been designed. In the initial stage, until 2012, only classrooms and supermarkets were studied, classrooms four times [60,66,78,86] and supermarkets three times [61,63,87], but VEs began to diversify as research progressed.

Inhibition and attention were targeted the most, as they are deeply related functionally and conceptually. They were studied 27 and 24 times, respectively. For inhibition and attention, virtual classroom environments were mostly used, and virtual apartments were the next most common type of environment. The next most targeted EF was working memory, which was studied 21 times. For working memory, various VEs were used, but markets were the most common, with eight studies. Next, planning was studied 10 times, shifting was studied 9 times, spatial and motor were examined 6 times each, and decision making, updating, and problem solving were studied 3, 2, and 1 times, respectively.

Description of Participants

An overall distribution of the participants’ age is presented in Figure 7. For age sorting, the United Nations’ age classification guideline for health, health services, and nutrition was applied [88]. Some articles that did not perfectly fit the guideline were checked multiple times according to their range of participants.

Figure 7

Research based on the age of the participant and symptoms of two age groups, under 14 and over 65 years. AD: Alzheimer’s disease; ADHD: attention deficit hyperactivity disorder; MCI: mild cognitive impairment; SCD: subjective cognitive decline; TBI: traumatic brain injury.

The age distribution of participants in all the articles was similar in all periods. The proportion of 15-24-year-olds was the highest, and the proportion of those 65 years and was the third. In the under-14 age group, there is just one article that studied traumatic brain injury (TBI), and the others examined ADHD. Among those above 65 years, the age-related cognitive decline corresponded to nearly 80% of all articles. In addition, there were many studies with only a healthy group.

The Emphasis on Real-Time Walking

The use of a multidirectional treadmill to allow a subject the possibility of walking by themselves first appeared in period 3. Research using treadmills usually targeted patients with TBI and MCI to rehabilitate their gate and motor ability. Entering period 4, some VR applications began to provide real-time walking without any equipment. In fully immersive VR, supermarkets, kitchens, laboratories, and small squares were used as VEs, but supermarkets corresponded to half of all environments.

Fully immersive VR provides a sense of reality and of presence in virtual spaces with a special apparatus, such as an HMD [21]. Furthermore, another merit of fully immersive VR is that it has a high level of ecological validity by allowing free movement in a certain space. The real-time walking studies can be separated into two categories, real-time walking and walking on a treadmill. In the case of a treadmill, fully immersive environments are built with a 180°/360° or a dome-shaped screen [42,76,83]. For real-time walking, VR can function as a tool allowing ADL.

Discussion

Expansion of Research: From Assessment to Treatment and From Simple Tasks to ADL

The most noticeable phenomenon is the appearance of treatment studies in period 4. In the early periods, when VR was first used in neuropsychological tests, it was only used for assessing cognitive states. However, high ecological validity, reproducibility of daily life, and the possibility of measuring behavior responses that were not available with traditional neuropsychological tests opened a new venue for cognitive rehabilitation [42,64,89,90]. Therefore, the research trend changed from being assessment oriented toward an additional interest for treatments. The possibility of treatment of cognitive problems using VR seems to expand continuously. In these studies, researchers have tried to train or rehabilitate the participants’ cognitive abilities (mainly inhibition and working memory) through regular and repetitive VR tasks that consisted of daily-living scenarios or game-like applications [42,58,62,64]. Some research argues that the biomarker and the degree of brain activation obtained as kinematic outcomes are indicators of treatment response [91,92].

From the perspective of changes in validation, it seems that there are two main streams. The first perspective is about transplanting existing traditional tests into VR, such as the Stroop test [29,50,57,67,69,70,73] and the trail-making task [76]. This includes most of the usability validation studies in data set 2. Because the Stroop test is a simple and ocular task, it has been considered suitable to convert into a VE. In addition, there is one study that brings the trail-making test into the VE. Second, an interesting thing is that most of the structure validation studies were conducted in supermarket environments based on the multiple-errand test [55,56,68,71]. Supermarket environments are popular due to their deep relation with daily living. Researchers have also tried to design various scenarios that are related to daily living, such as parking simulation [51] and pathfinding with Google Street View [59]. However, continuing to adapt parameters or structural properties to suit VR neuropsychological tests remains a challenge.

The increase in uncategorized studies in period 4 could also be seen in the same context. Most of the uncategorized studies examined the sense of presence and cybersickness [44,47,54,58,63,67,75,77]. These are indeed the two major problems for wider acceptance of VR for more age groups and for obtaining more ecological validity. The increase in studies to solve these problems aims for participants to feel more comfortable, which could further lead to the development of daily-living contents and expand the area of treatment. Therefore, solving these problems seems a key point to improve fully immersive neuropsychological testing in VR, but there is still a long way to go.

Diversification of Targeted EFs and VEs

As shown in Figure 3, diversification started in line with the popularization of VR in the late 2000s, and the research has grown quantitatively and qualitatively ever since. Initially, studies mostly targeted low-order EFs, such as inhibition, attention, and working memory [61,66,78]. However, in recent years, VR technology has evolved to overcome the problem of presence and cybersickness. This enabled researchers to implement higher ecological ADL with various practical scenarios and VEs. Consequently, it has led to studies that deal with higher-order EFs, such as decision making, planning, and problem solving.

Inhibition and attention are functions deeply related to ADHD. Because most subjects with ADHD are children and youth, it seems that researchers have actively used classroom and apartment (more accurately, the living room) environments that those ages are familiar with [43,45,53,54,65,78]. As the Stroop test is famous for assessing one’s inhibition, many studies on inhibition and attention have translated it into VR. Especially, most studies conducted under virtual apartment environments have developed their application based on the Stroop test [50,57,73,74]. Since familiarity with the participants’ daily-living situation induces psychological comfortability, application can sophisticatedly measure the natural state of cognitive abilities.

The supermarket environment is most frequently used for measuring working memory. Planning [61,63,87] and decision making [75] are also associated with the supermarket environment. For higher-order EFs, various other environments and daily-living scenarios have been tried, such as laboratories [51,77], kitchens [47], and parking simulation [51]. It is unrealistic to precisely measure one isolated EF due to its interdependent property. Higher-order EFs should be comprehensively measured. Only one case measured planning alone, a study that was the first attempt to apply the trail-making test to immersive VR [76]. Lastly, studies on spatial and motor abilities were tried after 2012 using a treadmill or motion-tracking systems.

Since one of the EFs closely connected to other EFs we defined, EFs in ADL tend to be treated comprehensively rather than alone. These deal with specific situations, the emergence of distractors, and well-organized scenarios that assist the original task.

Diversification of Target Symptoms and Expansion of the Age Group

As shown in Figure 4, target symptoms began to diversify after period 2. Stroke and brain injury accounted for the largest percentage until period 3, but in period 4, this changed. This indicates that VR studies on neuropsychology are being conducted in a wide variety of conditions. The thing to keep an eye on is the conspicuous increase of age-related cognitive decline and a relatively high proportion of ADHD.

The many VR studies have focused on ADHD, which is a popular symptom of young age in the early stage of VR research. In Figure 7, the under-14 age group is composed of TBI and ADHD, and ADHD represents the majority of these studies. It is not surprising as almost 5% of children are diagnosed with ADHD [86]. Many previous studies have reported the positive effect of VR on children who have a mental disorder [93-95] and surely on ADHD [43,78,86]. In addition, the ease of converting the Stroop test into immersive VR would have played a role, since the Stroop test is a neuropsychological assessment to measure attention and inhibition, which are major deficit features of ADHD [43]. If we regard under 25 years as one unique group, the activeness of related research can be seen as familiarity and accessibility to VR and resistance to cybersickness of adolescents. Young people are more adaptable to VR, even immersive VR [96].

As VR technology advances and related research accumulates, obstacles that blocked the application of VR neuropsychological tests to the elderly are being resolved [97]. Figure 7 also shows the expansion of the research to the elderly. In the 65+ age group, about 80% of papers are related to age-related cognitive decline. As mentioned before, we consider SCD, MCI, AD, and dementia together as age-related cognitive decline. MCI is a symptom that decreases one’s memory, attention, and cognitive function, and it could lead to dementia or AD [98]. SCD is an early state when someone’s cognitive ability is beginning to decline [99]. In other words, SCD may indicate a pre-MCI condition, even an early marker for the symptomatic manifestation of dementia. Age-related cognitive decline symptoms are common in the elderly. In addition, depression is a quite under-investigated area despite a huge impact on daily-living executive dysfunction in 60% of depressive states in the elderly, with an exponential growth of prevalence with age, a poorer response to antidepressants, and more evolution toward dementia compared to depression without dysexecutive syndrome [100-102].

Investigating VR as a neuropsychological tool for the elderly seems increasingly realistic since VR has become more familiar even for seniors. As we have seen before, there has been a lot of research on and technological advancement in presence and cybersickness in VR, and it is possible that the public and even the elderly could comfortably access it [103-105], which enables expansion to the elderly and causes rapid growth of the research that targets age-related cognitive decline symptoms after period 4.

Overall, immersive VR offers a high number of possibilities to deal with neuropsychological symptoms in various age groups. It shows the versatility of VR, which is applicable from youth with ADHD to the elderly with age-related cognitive decline.

Combining Behavioral and Physiological Data Measurement

It seems that measuring behavioral data and physiological data is becoming increasingly important. These kinds of data are used to extend the concept of embodied cognition to measure the neuropsychological status and to overcome the weaknesses of VR.

Embodied cognition is a broad notion from embodiment thesis in philosophy, and it states that an agent’s cognition is powerfully affected by aspects of their body beyond the brain itself. This means that sensorimotor experiences and actions are crucial to cognitive processing [106]. These aspects are thought to be crucial for cognitive performances in the elderly [107]. In this context, measuring behavioral and physiological variables in immersive VR would be a key method of assessing the neuropsychological status. Thus, the growth of associated research seems to be increasing because of the ecological characteristics of VR environments. In addition, when we looked deeper into gait or walking, there was a large difference in the appearance of walking on a treadmill and real-time walking in periods 3 and 4. With real walking in a VR neuropsychological test, researchers were able to apply the concept of embodied cognition and measure it in ADL.

Furthermore, an increase in physiological data measurements should be considered as these can provide auxiliary ways to assess the neuropsychological status of a patient. In addition, it is also considered to be a helpful way to overcome cybersickness [108-110]. Cybersickness is a major obstacle in applying VR to older people and people who have difficulties adapting to VR because they feel dizzy. In addition, research on emotion recognition and presence in VR uses physiological data [111-114]. With these efforts added, VR neuropsychological tests’ versatility for all ages continues to improve.

Overall, by using behavioral and physiological data measurement, VR neuropsychological tests are not only getting over existing drawbacks but also expanding to the concept of embodied cognition to improve its measurement capability.

Limitations

This review had a few limitations. First, keyword analysis was conducted by only using titles and abstracts of the articles. This means that there is a possibility of a difference with the articles’ main texts. Second, we only included articles that described research conducted with real participants, so conceptual and theoretical trends in the area were not covered. Finally, not every article markedly described the target EFs. Likewise, there were some ambiguities in clearly assigning the purpose of each article to assessment, treatment, structure validation, or usability validation. Even though we attempted to judge based on the description of what EFs the articles targeted, and what their intended purpose was, there was still room for error. Future research could include broader studies in the area and use more rigorous methods to analyze the trends.

Conclusion

This review will assist researchers in understanding the trends in VR neuropsychological tests over the past 20 years. Associated research has been on the rise and has sharply increased in recent years. In this process, the advancements in technology and various approaches have led to diversified target cognitive abilities, including EFs, as well as target symptoms. Moreover, collecting behavioral and physiological data enables a wide understanding about treating EFs by using the concept of embodied cognition. As a result, VR neuropsychological tests now cover a wide range of age groups and extend beyond assessment tools to treatment tools. This review shows that there is a continuously increased interest in dealing with neuropsychology by using fully immersive VR, and it is expected that this will help advance research in this area.