Virtual reality and speech analysis for the assessment of impulsivity and decision-making: protocol for a comparison with neuropsychological tasks and self-administered questionnaires.

INTRODUCTION: Impulsivity is present in a range of mental disorders and has been associated with suicide. Traditional measures of impulsivity have certain limitations, such as the lack of ecological validity. Virtual reality (VR) may overcome these issues. This study aims to validate the VR assessment tool 'Spheres & Shield Maze Task' and speech analysis by comparing them with traditional measures. We hypothesise that these innovative tools will be reliable and acceptable by patients, potentially improving the simultaneous assessment of impulsivity and decision-making. METHODS AND ANALYSIS: This study will be carried out at the University Hospital Fundación Jiménez Díaz (Madrid, Spain). Our sample will consist of adults divided into three groups: psychiatric outpatients with a history of suicidal thoughts and/or behaviours, psychiatric outpatients without such a history and healthy volunteers. The target sample size was established at 300 participants (100 per group). Participants will complete the Barratt Impulsiveness Scale 11; the Urgency, Premeditation, Perseverance, Sensation Seeking, Positive Urgency, Impulsive Behaviour Scale; Iowa Gambling Task; Continuous Performance Test; Stop signal Task, and Go/no-go task, three questions of emotional affect, the Spheres & Shield Maze Task and two satisfaction surveys. During these tasks, participant speech will be recorded. Construct validity of the VR environment will be calculated. We will also explore the association between VR-assessed impulsivity and history of suicidal thoughts and/or behaviour, and the association between speech and impulsivity and decision-making. ETHICS AND DISSEMINATION: This study was approved by the Ethics Committee of the University Hospital Fundación Jiménez Díaz (PIC128-21_FJD). Participants will be required to provide written informed consent. The findings will be presented in a series of manuscripts that will be submitted to peer-reviewed journals for publication. TRIAL REGISTRATION NUMBER: NCT05109845; Pre-results. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study stands out for its innovation, as there are few studies exploring the assessment of impulsivity through virtual reality (VR), and no previous studies validating the Spheres & Shield Maze Task in clinical settings.

This study also stands out for its scope including speech analysis, VR and traditional measures of impulsivity and decision-making.

This study further stands out for its methodological rigour and its implementation in a clinical setting.

A limitation of the study is the unfamiliarity of some participants with the digital environment, which will be overcome by the explanation and support of the trained psychologists who will supervise the development of the task.

Introduction

Impulsivity is the tendency to act without prior reflection. It implies poor self-control of behaviour, leading to quick reactions without regard for consequences.1 Impulsive behaviour is preceded by increasing tension and followed by gratification or relief.2 Impulsivity is strongly present in some mental disorders such as borderline personality disorder, substance abuse or bipolar disorder.1 3 Impulsivity is also strongly associated with suicidal thoughts and behaviours (STB) and may define a particular suicidal phenotype in which action predominates over ideation.4

The dimensional model of impulsivity considers three factors: decision-making, action and personality traits.5 The first two factors are the behavioural domains of impulsivity, and they are assessed with neuropsychological tests, while impulsivity as a personality trait is assessed with personality questionnaires.6 The third factor, decision-making, has been strongly associated with suicidal behaviour.7

Decision-making is a complex cognitive function in which learning, previous experience and sensitivity to feedback interact.8 It can be intuitive or reflexive, involving different brain structures: intuitive processes are mediated by the amygdala, ventral striatum and orbitofrontal cortex, while reflexive processes are mediated by the dorsolateral prefrontal cortex, anterior cingulate and posterior parietal lobe.9 For people who engage in suicidal behaviour, it may be difficult to make decisions in risky situations, as they tend to decide more emotionally than rationally.10 Studies have found that more than half of all suicide attempts are impulsive.11 Furthermore, increased impulsivity and aggression have been associated with death by suicide.12–14

Assessment of impulsivity

Self-administered questionnaires are the most common approach to assessing impulsivity. The Barrat Impulsivity Scale (BIS-11), which assesses impulsive personality traits, is a commonly used measure.15 Despite its usefulness, self-report has significant limitations: the cultural influence of the questionnaires hinders their generalisability.16 Furthermore, impulsive people may find it challenging to describe their own personality, and they may report unreliably.17 Computerised neuropsychological tests overcome some of these limitations, as the cultural background does not influence them, and they do not rely on the capacity for introspection and self-awareness. Moreover, they are more sensitive to change and can be applied at any age.6 Tasks of this type are the Iowa Gambling Task (IGT),18 which measures decision-making, and the Go/No-go Task,19 which measures response inhibition and impulsive action. Despite their advantages over questionnaires, neuropsychological measures are not without limitations: the situations and stimuli presented to participants are not very ecological, that is, they do not usually have a translation into real life, which may compromise their validity.20 Virtual reality (VR) may increase ecological validity, as there is empirical evidence showing that the neural mechanisms that people experience when immersed in a VR environment are similar to those of real life.21 Serious VR games have become an innovative, engaging and customisable way to provide healthcare.22–24 VR has been applied in mental health interventions25 and the field of neuropsychological assessment,26 with good results.

Speech and mental health

Speech has been increasingly explored for mental health assessment. Speech samples can be easily collected from patients and can provide helpful information, particularly if machine learning techniques are used for analysing speech data.27 While the selected acoustic features used for classification may differ between models trained with free or constrained speech, studies show that free speech responses are equal28 or even more accurate29 in detecting depression.

Many features can be extracted from speech. Particularly, jitter, shimmer and frequency have shown to be related to many psychiatric disorders. Predictive models for diagnosis using speech features have been developed for depression, Post-Traumatic Stress Disorder (PTSD), schizophrenia, anxiety and bipolar disorder with promising results.30 In contrast, there is a lack of research on speech analysis for assessing impulsivity and decision-making. However, there are studies concerning attention deficit/hyperactivity disorder (ADHD), a disorder in which impulsivity plays an important role. For instance, Hamdan et al31 compared children with and without ADHD and found that the voices of children with ADHD had more hoarseness, breathiness and straining. Further studies are needed to validate speech analysis for assessing mental health problems.

Aims and hypotheses

This study presents the VR assessment tool ‘Spheres & Shield Maze Task’ (SSMT), a virtual maze that participants must go through while overcoming several obstacles, which provides an immersive experience that can approximate real-life conditions, thus increasing ecological validity. The VR tool has been previously evaluated in non-clinical settings in two studies.32 33 In the first pilot study, the SSMT was explored in 41 students. The authors found that scores obtained through VR correlated with the self-reported assessment of risky behaviour.32 In another study, 98 community-dwelling participants were recruited. The VR task successfully discriminated between low-risk and high-risk individuals.33

However, the SSMT has not been validated in clinical settings. Similarly, speech analysis has not been sufficiently studied as an ecological measure for analysing impulsivity in mental healthcare settings.

This work aims to validate the SSMT VR tool and speech analysis by comparing them with traditional measures of impulsivity and decision-making in psychiatric patients with and without a history of STB and healthy controls. We hypothesise that both the SSMT and speech analysis will be well accepted by patients, which will translate into high scores in the satisfaction surveys. This work also aims to explore if there is an association between suicidal behaviour and impulsivity and how speech analysis correlates with impulsivity measures, electrodermal activity and eye-tracking.

We also hypothesise that we will find significant differences among the three groups of participants, with a gradient of impulsivity from lowest to highest between healthy controls

Methods and analysis

Setting and design

This is a cross-sectional study that will be carried out in the Department of Psychiatry of the University Hospital Fundación Jiménez Díaz (Madrid, Spain). Data will be collected over an estimated period of 1 year.

Sample

Our sample will be divided into three groups:

Psychiatric outpatients with a history of suicide thoughts and/or behaviours.

Psychiatric outpatients without a history of suicide thoughts and/or behaviours.

Community-dwelling healthy volunteers, serving as controls.

Inclusion criteria will be as follows:

Being 18 years of age or older.

Being able to understand and sign the informed consent form.

Being fluent in Spanish (to be able to understand the information sheet, the consent form, and the questionnaires and tasks).

Exclusion criteria will be as follows:

Inability to understand and sign the informed consent form for any reason.

History of intellectual disability.

History of dizziness or other balance pathology, hearing impairment or visual impairment that precludes carrying out the VR assessment.

Severe upper limb disability that precludes operating the VR game controls.

Sample size calculation

In the original validation study of the VR environment in healthy volunteers, a sample of 98 participants was used.34 A recent meta-analysis35 on studies comparing differences in decision-making between suicidal patients, non-suicidal patients and healthy controls found a pooled moderate effect size (0.4) in neuropsychological markers of vulnerability to suicidal behaviour. Therefore, we estimated a sample size based on a moderate effect size of 0.4, an alpha error of 0.5 and a power of 0.8. Sample size estimated was 63 for each group. On the other hand, a systematic review about speech analysis applied to mental health recommends a minimum of 74 participants per group for null-hypothesis testing.31

Finally, and considering our recruiting capacity and the risk of dropout, our target sample size was determined at 100 participants per group (300 in total).

Patient and public involvement

Before starting the study, a pilot will be conducted, in approximately five psychiatric patients and five healthy volunteers, who will test the VR environment and give their impressions on the quality of the software and the ease of use and inform of any possible technical or human errors.

Procedure

Psychiatric patients will be recruited by their attending psychiatrist during regular visits. Healthy volunteers will be recruited following the snowball methodology generated from mental health centre workers, trainees, and medicine and psychology students.

After signing the informed consent, participants will be scheduled for the assessment session, which will last approximately 1 hour. In this session, the participant will be recorded throughout using a lapel microphone, Rode smartLav+, connected to a digital recorder and will be asked two questions with emotional affect followed by questionnaires and computerised neuropsychological tests. They will then be taken to another room and administered the VR task that is recorded by the dual microphones in the Vive Pro Eye VR headset. The participant will first be asked a question with neutral emotional affect and then continue with the VR task. Trained psychologists will administer the questionnaires and assist the participant during the VR task. All personnel will undergo specific training to be able to supervise the VR task adequately. At the end of the VR task, participants will complete two satisfaction surveys used in previous studies.26 One of this satisfaction surveys will be quantitative and will explore general satisfaction with the different assessments, usability of the VR tool, perceived usefulness of the project and extent to which users would recommend participating in the project to a family member or friend. The other satisfaction survey will be qualitative and will explore comments of the users, possible complaints and suggestions for improvement.

A brief report on their impulsivity scores will be given to participants as feedback shortly after concluding the project.

The piloting is expected to take place in January 2022. Recruitment and assessment are expected to start in March 2022 and to be completed by March 2023. Data analysis will take place from June to October 2023. We expect to publish partial results by December 2023 and final results by summer 2024.

Measures

The following assessments will be carried out: (1) collection of sociodemographic and clinical data, (2) self-reported assessment of impulsive personality traits, (3) computerised neuropsychological tests, (4) VR-based impulsivity assessment, (5) recording of speech and (6) satisfaction surveys.

Data collection will be carried out in an electronic data collection notebook (www.memind.net). The recorded spontaneous speech will also be uploaded. In addition, the resolution of the virtual maze will be automatically recorded by the software, together with the corresponding scores, which will be saved on an external hard disk.

Sociodemographic and clinical data

An ad hoc questionnaire will be developed to record sociodemographic variables of interest, including gender, age, marital status, employment status and educational level. History of suicidal behaviour will also be assessed using the Columbia Suicide Severity Rating Scale.36 Additional questionnaires will be the Patients’ Health Questionnaire-9 (PHQ-9), the Positive And Negative Symptoms Scale (PANSS)—only in case of psychosis—and the General Anxiety Disorder-7 (GAD-7). Participants will also be inquired about substance use—including alcohol, tobacco, cannabis, cocaine, hallucinogens and other drugs.

Impulsive personality traits

Barratt Impulsivity Scale (BIS-11)

The BIS-11 was designed to assess impulsivity as a personality trait. It consists of 30 items that are grouped into three subscales: attentional impulsiveness, motor impulsiveness and non-planning impulsiveness. Each item is scored from 1 to 4, with higher scores indicating greater impulsivity.15

Urgency, Premeditation, Perseverance, Sensation Seeking, Positive Urgency, Impulsive Behaviour Scale (UPPS)

This is a questionnaire that measures impulsive personality traits through 45 items, scored from 1 to 4, with higher scores indicating greater impulsivity.37 38

Computerised neuropsychological tests

Iowa Gambling Task (IGT)

This is a neuropsychological test that measures decision-making through a game in which the participant must bet on one of four decks of cards. Feedback is given to users so that they can determine the best option (the one with the highest payoff) if the reflect on it. The score is calculated by subtracting wrong decisions from right decisions. Higher scores reflect a more thoughtful and less impulsive decision-making process.18

Continuous Performance Test (CPT)

This test assesses the ability to inhibit an ongoing motor response. The stimuli consist of individual letters, which are presented on the computer screen for 250 ms. Participants have to press the space bar as quickly as possible in response to each stimulus presentation, unless the stimulus is the letter X. Impulsive responses are defined as errors of commission (ie, pressing the space bar when the letter X is displayed).39

Stop signal Task

This task assesses inhibitory control over an already initiated response. The stimuli presented to the participant consist of four letters: two are assigned to one key and two to another. After a certain number of times, an acoustic stop signal is emitted, indicating that the participant should refrain from pressing the corresponding keys any further. Delays in obeying the stop signal (and thus continuing to press the keys instead of refraining) indicate increased impulsivity.40

Go/No-go Task

The purpose of these tasks is to measure attention and inhibitory control. In response to certain stimuli, the participant must respond by pressing the space bar (go), while in response to other stimuli, the participant must inhibit (no go). False alarm errors (pressing the space bar in response to the No-go stimulus), omission errors (not pressing the space bar in response to the Go stimulus) and reaction time to the Go stimulus are calculated.19

Emotional affective questions for speech analysis

Good news question

At the beginning of the assessment session, the participant will be asked, ‘Can you recall some recent good news you had and how did that make you feel?’. Additionally, the patient will be asked to respond for 2 min, and the interviewing psychologist will refrain from speaking during this period and timestamp the recording. This question is designed to elicit an emotional response related to a positive experience.29

Sadness characteristic question

After the Good News Question, the participant will be asked, ‘Do you get a characteristic feeling when you’re sad or down and what do you normally do to cheer yourself up?’. Additionally, the patient will be asked to respond for 2 min, and the interviewing psychologist will refrain from speaking during this period and timestamp the recording. This question is designed to elicit an emotional response related to positive and negative emotions.29

Average day question

At the beginning of the VR task, the participant will be asked to ‘Describe your average day.’ Additionally, the patient will be asked to respond for 2 min, and the interviewer will refrain from speaking during this time. The recording by the VR headset will be automatically separated. This question is designed to be neutral and to not elicit any emotion. This question will work as a baseline for comparison with spontaneous speech during the VR task.

VR environment

Our VR environment, called Spheres & Shield Maze Task, has been developed by the Laboratory of Immersive Neurotechnologies (LabLENI) of the Universitat Politècnica de València. To implement the virtual environment, we will use a VIVE pro eye VR headset, a VNET INTEL I7 computer and a Shimmer GSR (galvanic skin response) kit. The game consists of a maze that the user must solve by moving. The movement is performed by combining a touchpad with front-facing orientation.

The user will encounter different hazards in the maze, such as narrow walkways from which the player can fall (figure 1), lightning clouds that strike in intervals, flying insects (figure 2) or locked rooms (figure 3). These hazards have different consequences, such as time penalty or energy loss. Users can activate a shield to defend themselves from some of these hazards at the cost of being unable to gather green orbs (figure 2). When used, the shield requires a recharge time, so it is a limited resource that the player must optimise. Hazards may also be avoided completely by choosing another path through the maze.

Figure 1

In-game screenshot of green orbs around a narrow wooden bar from which the player can fall.

Figure 2

In-game screenshot exhibiting the use of the shield to protect from flying insects.

Figure 3

In-game screenshot of a locked room that requires a key to escape.

The screen displays an indicator of the energy, which increases by gathering green orbs and decreases with movement. If the energy decreases, the movement is slowed down, giving the user auditory feedback of heavy breathing. These consequences may influence the user’s decision-making. Energy can be recharged by consuming green orbs along the route.

The user can interrupt the gameplay and go to the break room at any time by pressing a button. In the break room, the participant will be asked a question: ‘What would you like to do right now?’, with the following options:

I need to try even harder to improve my score.

I feel guilty about my performance.

I need to express how I feel.

I wish I had never started this game.

I would like to share what is happening to me with someone.

I need to remember what just happened to analyse why I got stressed and see it from a different point of view.

I prefer not to think about what I have just experienced and move on.

I need to be alone.

I would like to take a minute to relax.

Figure 4 shows the maze diagram.

Figure 4

Diagram of the virtual reality-based assessment tool ‘Spheres & Shield Maze Task’.

The user will start the game with 100% of time, 75% of energy and 100% of shield. When the energy is at 50%, the movement speed will drop by 1 point. When the energy is at 25%, the movement speed will drop by 2 points. When there are 30 s left, the time indicator will flash red, and a ‘30 seconds left to find the exit’ message will appear. If time runs out and the user has not found the exit, the time will start to count below 0.

The maze will be scored as the time taken to find the exit. Additionally, data will be collected on behavioural and psychophysiological measures during the execution of the task, including distance travelled, number of stops, trajectory, decision-making at bifurcations, total energy obtained, number of orbs collected, use of the shield and use of the break room. Electrodermal activity, eye movement and spontaneous speech will also be recorded. The playing style used to traverse the maze, together with the time taken, yields a score on impulsivity using predesigned algorithms. In this way, a ‘stealth assessment’ of decision-making and impulsivity is obtained through implicit behavioural measures, as explained by De Juan Ripoll et al.32 33

Eye-tracking and electrodermal activity

The Vive Eye Pro headset will be configured for each participant to assure comfort, clear vision and the proper function of eye-tracking software built into the headset. While the patient performs the VR task, the headset will automatically track eye movement, and these data will be automatically stored. Eye-tracking has been previously used as a metric for processing of features and attention, and lower processing of features has been related to impulsivity.41

In addition, participants will wear a Shimmer3 GSR unit with finger electrodes and an optical pulse sensor to measure skin conductance, electrodermal activity and heart rate. Skin conductance and electrodermal activity have both been studied as markers of arousal, risk processing and impulsivity.42 43

Eye movement and electrodermal activity are a part of the SSMT VR task that was incorporated in the latest updates of the task to improve the general scores of the VR task.33 As such, they will not be analysed separately from the VR task.

Satisfaction surveys

At the end of the session, participants will complete a quantitative and a qualitative satisfaction survey, which have been used previously.26

Outcomes

Main outcomes will be impulsivity and decision-making style as measured through the VR task using predesigned algorithms. VR task-based measures will be compared with measures obtained through personality assessment (impulsivity traits of personality: BIS-11 and UPPS questionnaires) and neuropsychological tests (decision-making: IGT task; impulsivity: CPT test, Stop signal Task, Go/No-go Task).

Statistical analysis

Statistical analyses will be carried out using SPSS V.24.0, R, Matlab and Python statistical software. External consistency of the VR task will be assessed using traditional neuropsychological measures as gold standard, and internal consistency will be assessed using Cronbach’s alpha and intraclass correlation coefficient. To explore the differences between groups (suicidal patients, non-suicidal patients and healthy controls), the mean scores obtained in the variables collected will be compared using analysis of variance test or its non-parametric equivalent if the distribution is not normal. We will also compare the groups suicidal versus non-suicidal psychiatric patients (groups 1 vs group 2) and psychiatric patients versus healthy controls (group 1+group 2 vs group 3) using logistic regression analyses. We will also carry out exploratory analyses concerning other variables.

Audio recordings will be analysed in OpenSMILE, an open-source offline software for audio analysis.44 A diarisation step will be performed to label the participant, silences and the interviewing psychologist. Participant audio will be summarised by statistics of mean and variance on multiple speech features, such as signal energy, loudness, pitch, formants, mel frequency cepstral coefficients, spectral features and frequency. Machine learning techniques will then be used for feature selection and examining the relation between a participant’s speech, impulsivity, suicidality and mental health questionnaires. Additionally, analysis of recorded speech or utterance before and during the VR task will be used to predict satisfaction of the participant.

Descriptive statistics from the quantitative satisfaction survey and content analysis for the qualitative satisfaction survey will also be provided. Satisfaction scores of the traditional assessments will be compared with satisfaction scores of the VR task.

All tests will be two-tailed with statistical significance established at p<0.005 and 95% CIs.

Ethics and dissemination

Study approval and consent

This study was approved by the Ethics Committee of the University Hospital Fundación Jiménez Díaz (PIC128-21_FJD), and it follows the guidelines formulated by the World Medical Association’s Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects.45 The protocol is registered in ClinicalTrials.gov (NCT05109845). No data that could identify participants will appear in the publications derived from this study. Each participant will be assigned an alphanumeric code. The co-IPs (EB-G and MLB) will be the only people who will be able to access each participant’s data. The web database of the participants will be hosted in a secure server, and all communications will be encrypted using HTTPS protocol and protected by a security firewall.

All participants will sign an informed consent form. They will be given sufficient time to read the sheet and ask questions. It will be made clear that participation is entirely voluntary and that they may leave the study at any time if they wish to do so. There are no costs nor remuneration for participation in this study.

Dissemination plans

We expect the VR-based assessment to be comparable to the traditional self-report and neuropsychological assessment and to be better accepted by the participants. Thus, we will have a useful assessment tool with greater ecological validity. This study will also contribute to increasing knowledge about the association between impulsivity and suicidal behaviour and the use of speech analysis in a clinical setting.

This study will result in a series of manuscripts that will be submitted to peer-reviewed journals for publication. We will report both significant and non-significant results. Once the results of the project are available, the mechanisms underlying the potential associations between variables will be discussed. Our findings will also be presented in the form of oral communications and symposia at national and international psychiatric conferences.