Verbal memory measurement towards digital perspectives in first-episode psychosis: A review.

BACKGROUND: Even in the early phases of psychotic spectrum illnesses such as schizophrenia, patients can experience cognitive decline or deficits prior to the onset of psychotic symptoms such as delusions and hallucinations. In this systematic review, we assessed which verbal memory assessments are most widely used in first-episode psychosis and may be applied via digital technologies (smartphone applications, etc.) for use in early detection.
METHODS: In November 2019, we searched for studies measuring verbal memory in first episode psychosis or schizophrenia over the past 10 years on PubMed and PsycINFO. We screened abstracts of these studies and excluded review studies. Full-texts of included studies were used to identify the verbal memory measurement tests, follow-up frequencies, and sample sizes.
RESULTS: We screened 233 reports and found that 120 original research studies measured verbal memory in first episode psychosis over the past 10 years. Four of these studies specified using a computer, 24 (20%) used a paper-pen format, 1(1%) used both, and 91 (76%) studies did not specify their administration tools or suggest there were offered in digital formats. Thirty-five (30%) studies had follow-up measurements of verbal memory, while 85 (70%) had only a single verbal memory measurement. DISCUSSION: While many scales are commonly used to measure verbal memory in first episode psychosis, they are not often administered via digital technology. There is an emerging opportunity to administer these and other tests via digital technologies for expanding access to early detection of cognitive decline in clinical high risk and first-episode psychosis.

Introduction

Psychosis is a clinical syndrome associated with impairments in multiple domains including functioning, thought, behavior, and cognition. Most patients experience onset of psychosis in early adulthood, although most mental health systems fail to detect psychosis until a year or more has elapsed (Perkins et al., 2005). However, early detection and intervention may prevent or slow the progress of the illness (Lieberman et al., 2019; Santesteban-Echarri et al., 2017). There are a growing number of psychosis studies focused on predicting the onset of psychosis, examining features including speech (Bedi et al., 2015) and cognitive deficits (Zanelli et al., 2019). Strategies to leverage technology and digital tools, including smartphones, to increase scalability and offer assessments and therapy directly to patients, hold high potential for the future (Camacho et al., 2019; Liu et al., 2019; Torous et al., 2017; Torous and Keshavan, 2018; Vaidyam et al., 2019).

In the early phases of psychotic spectrum illnesses like schizophrenia, patients can experience cognitive decline prior to onset of other symptoms such as delusions and hallucinations (Bora and Murray, 2014). Cognitive deficits are already present before and during the prodromal phase (Bora and Murray, 2014), but progress to more severe impairments in first-episode psychosis and may decline further afterwards (Bora et al., 2014; Sanchez-Torres et al., 2018). Cognitive deficits in first-episode psychosis can be experienced across all cognitive domains, with impairments of varying severity in general cognition, social cognition, processing speed, and verbal memory (Corigliano et al., 2014; Healey et al., 2016; Mesholam-Gately et al., 2009; Schaefer et al., 2013; Sheffield et al., 2018; Woodberry et al., 2016). Among these cognitive domains, verbal memory is often impacted and may be predictive for prognosis and functional recovery in first-episode psychosis (Aas et al., 2014; Allott et al., 2011; Fatouros-Bergman et al., 2014; Mesholam-Gately et al., 2009; Seidman et al., 2016; Szoke et al., 2008; Woodberry et al., 2016; Zhang et al., 2015). However, all cognitive domains decline before or in the first-episode with emerging evidence of continued decline after (Fett et al., 2019), including verbal memory (Bozikas and Andreou, 2011; Zanelli et al., 2019). Research shows a possible underlying neurobiological mechanism for verbal memory deficits in first-episode psychosis, including gray matter decreases in left temporal lobe, hippocampal system, and frontotemporal circuits (Saykin et al., 1994). Early detection is critical as cognitive remediation significantly improves verbal memory compared to other cognitive variables (Revell et al., 2015) and eventually mediates functional gains (Lee et al., 2013).

Verbal memory refers to the memory of verbally represented information (Tatsumi and Watanabe, 2009) and has two main subdomains, which are declarative and non-declarative verbal memory. Among these two subdomains, declarative memory can be divided into semantic (fact memory) and episodic memory (event memory) (Aas et al., 2014; Spreen and Strauss, 1998). In order to measure verbal memory, a broad range of tasks are available including story recall, learning of word lists or sequences of paired words (Tatsumi and Watanabe, 2009). Instruments used to measure verbal memory in psychiatric studies include Weschler Memory Scale (WMS), Rey-Auditory Verbal Learning Test (RAVLT), California Verbal Learning Test (CVLT), Hopkins Verbal Learning Test (HVLT), Buschke Selective Reminding Test (BSRT), Brief Assessment of Cognition in Schizophrenia (BACS), Cambridge Cognition Paired Associates Learning Test, and many others. While the literature shows the importance of verbal memory for early detection and intervention in first-episode psychosis, it is still not clear which tasks or tests are most widely used to measure verbal memory, what differential features they possess, and which may be most amenable to being translated to a digital format. There is a lack of information about the methods (computer, digital, pen-paper) used to administer these tests and frequency of follow-up measurements. In this systematic literature review, we aimed to find which verbal memory assessment tests are most widely used in first-episode psychosis (FEP) and could be applied via digital technologies (smartphone applications, chatbots etc.) with the goal of capturing dense and longitudinal data during this period of clinical and neurobiological transitions surrounding the first episode.

Methods

In November 2019, we searched English language studies measuring verbal memory in first episode psychosis or first episode schizophrenia over past 10 years on PubMed (Medline) and PsycINFO databases. We chose 10 years as digital versions on mobile phones and portable consumer devices would not be available earlier. The following keywords were used in searching papers published from 2010 to date: “first episode psychosis and verbal memory” and “first episode schizophrenia and verbal memory”. Inclusion criteria were first-episode psychosis and verbal memory measurement for patients. In this study, we identified a test as ‘digital’ if it was administered largely via a mobile device or tablet that was capable of also gathering dense and longitudinal data. While there may not be validated digital versions of some assessments, it is possible research on early adaptations has been conducted and we sought to identify these papers.

Two investigators independently screened the titles and abstracts. We excluded review studies, meta-analyses, commentaries, and editorials. We downloaded full-texts of included studies to identify the verbal memory measurement tests and tools used, follow-up frequencies, and sample sizes. The PRISMA diagram below shows the process (Fig. 1). A standardized, pre-piloted form was used to extract data from the included studies for assessment of study quality and evidence synthesis.

Fig. 1

PRISMA 2009 flow diagram.

Results

We screened 233 papers and found that 120 original research studies measured verbal memory in first episode psychosis over past 10 years. Of these 120 studies, 31(%26) used Rey Auditory Verbal Learning Test (RAVLT), 29(24%) used California Verbal Learning Test (CVLT), 27(22%) used Weschler Memory Scale-Logical Memory Scale (WMS), 14(12%) used Hopkins Verbal Learning Test-Revised (HVLT), 4(3%) used both WMS and CVLT, 3(2%) used both WMS and RAVLT, and 11(10%) used other tests to measure verbal memory. Four (3%) of these studies specified that they used a computer, 24(20%) used paper-pen, 1(1%) study used both computer and pen-paper, and 91(76%) studies did not specify their verbal measurement application tools. 35(30%) studies had follow-up measurements of verbal memory, while 85(70%) studies had only a single time point verbal memory measurement during study period. Study sample sizes range from 6 to 498 (Table 2).

Table 2

Studies and Verbal Memory Tests.

Test	N and % of studies used including the studies used two tests (total N = 120)	N studies used paper-pen (PP) tools	N studies used computerized (CMP) tools	N studies did not specify tools (NOS)	Studies have follow-up assessments (excluding baseline assessment)	List of studies
CVLT	33(27%)	9	0	24	9 studies: (Albus et al., 2019)(3 fup), (Gjerde et al., 2019)(1 fup), (Amoretti et al., 2018)(1 fup), (Vila-Rodriguez et al., 2017)(1 fup), (Amoretti et al., 2016) (1 fup), (Helene E. Barder et al., 2013)(4 fup), (Helene Eidsmo Barder et al., 2013)(3 fup), (Baitz et al., 2012)(2 fup), (Becker et al., 2010)(1 fup)	PP: (Albus et al., 2019; Becker et al., 2010; Benoit et al., 2015; Bioque et al., 2016; Egloff et al., 2019; Gjerde et al., 2019; Ittig et al., 2015; Moura et al., 2019; Zabala et al., 2010)CMP: –Both PP and CMP: –NOS: (Amoretti et al., 2018, 2016; Bachman et al., 2012; Baitz et al., 2012; Ballesteros et al., 2018; Bang et al., 2015; Helene E. Barder et al., 2013; Helene Eidsmo Barder et al., 2013; Cabrera et al., 2016; Cuesta et al., 2015; de la Serna et al., 2013; de La Serna et al., 2010; Egloff et al., 2018; Faerden et al., 2013; Gisselgård et al., 2014; González-Pinto et al., 2016; Heitz et al., 2019; Kim et al., 2011; Mezquida et al., 2016; Nuechterlein et al., 2011; Núñez et al., 2016; Pflueger et al., 2018; Stain et al., 2012; Vila-Rodriguez et al., 2017)
HVLT	14(12%)	7	0	7	6 studies: (Fu et al., 2017)(5 fup), (Heeramun-Aubeeluck et al., 2015)(2 fup), (Fisher et al., 2015) (1 fup), (Olivier et al., 2015)(2 fup), (Trampush et al., 2015)(1 fup), (Vesterager et al., 2012)(2 fup)	PP: (Compton et al., 2013; Fu et al., 2017; Heeramun-Aubeeluck et al., 2015; Scoriels et al., 2012; Trampush et al., 2015; Vesterager et al., 2012; Zhou et al., 2014)CMP: –Both PP and CMP:–NOS: (Carrión et al., 2018; Fisher et al., 2015; Guo et al., 2014; Hou et al., 2016; Hu et al., 2011; McEwen et al., 2015; Olivier et al., 2015)
RAVLT	34(28%)	4	1	29	9 studies: (Rojnic Kuzman et al., 2019)(1 fup), (Kravariti et al., 2019)(1 fup), (Caruana et al., 2015)(1 fup), (Molina et al., 2014) (1 fup), (Matsuda et al., 2014)(2 fup), (Ayesa-Arriola et al., 2013) (2 fup), (Urben et al., 2012)(1 fup), (Rodríguez-Sánchez et al., 2013)(2 fup), (Simon et al., 2012) (2 fup)	PP: (Huber et al., 2012; Ngoma et al., 2010; Rodríguez-Sánchez et al., 2013; Urben et al., 2012)CMP: –Both PP and CMP: (Knytl et al., 2019)NOS: (Allott et al., 2013; Ayesa-Arriola et al., 2019, Ayesa-Arriola et al., 2014, Ayesa-Arriola et al., 2013; Caruana et al., 2015; Galderisi et al., 2009; González-Blanch et al., 2010; Gutiérrez-Galve et al., 2010; Hasan et al., 2014; Havelka et al., 2016; Hermens et al., 2010; Hickling et al., 2018; Karambelas et al., 2019; Koelkebeck et al., 2010; Kravariti et al., 2019, Kravariti et al., 2009; Leeson et al., 2010, Leeson et al., 2009; Mallawaarachchi et al., 2019; Matsuda et al., 2014; Mellacqua et al., 2012; Molina et al., 2014; Rodriguez et al., 2019; Rojnic Kuzman et al., 2019; Setién-Suero et al., 2018; Simon et al., 2012; Üçok et al., 2013; Varela-Gomez et al., 2015; Zanelli et al., 2010)
WMS	34(28%)	4	1	29	10 studies: (Rojnic Kuzman et al., 2019)(1 fup), (Albus et al., 2019) (3 fup), (Wannan et al., 2018)(1 fup), (Liu et al., 2015)(1 fup), (Benoit et al., 2014)(1 fup), (Wing Chung Chang et al., 2013)(3 fup), (Hovington et al., 2013) (1 fup), (W. C. Chang et al., 2013) (3 fup), (Baitz et al., 2012)(2 fup), (Pollice et al., 2012)(1 fup),	PP: (Albus et al., 2019; Benoit et al., 2015; Campbell et al., 2013; Makowski et al., 2019)CMP: –Both PP and CMP: (Knytl et al., 2019)NOS: (Aas et al., 2011; Andreou et al., 2015; Bachman et al., 2012; Bartholomeusz et al., 2011; Béchard-Evans et al., 2010; Benoit et al., 2014; Bodnar et al., 2011; Buchy et al., 2010; Cassidy et al., 2014; Chan et al., 2010; W. C. Chang et al., 2013; Wing Chung Chang et al., 2013; Hovington et al., 2013; Hui et al., 2019; Ilonen and Salokangas, 2016; Jordan et al., 2018, Jordan et al., 2014; Liu et al., 2015; Lui et al., 2015; Lutgens et al., 2014; Martínez-Cengotitabengoa et al., 2012; O'Connor et al., 2012; Pollice et al., 2012; Rodriguez et al., 2019; Rojnic Kuzman et al., 2019; Ruiz-Veguilla et al., 2017; Wannan et al., 2018; Wiffen et al., 2012)
Other	12(10%)	2	4	6	3 studies: (Firth et al., 2018a)(1fup), (Allott et al., 2018)(1 fup), (Firth et al., 2018b)(2 fup)	PP: (Andersen et al., 2013; Higuchi et al., 2013)CMP: (Caldiroli et al., 2016; Dieleman et al., 2015; Firth et al., 2018a, Firth et al., 2018b)Both PP and CMP:NOS: (Allott et al., 2018; Jepsen et al., 2010; Molina et al., 2018; Palomares et al., 2019; Sawada et al., 2017; Suazo et al., 2012)

Thirty-five (30%) studies had follow-up measurements of verbal memory, while 85(70%) studies had only a single time point verbal memory measurement during study period. Of these 35 studies, follow-up intervals range from 10 weeks to 10 years. Thirteen (39%) of these 35 studies have 1-year follow-up intervals, followed by 5(14%) studies with 6-month follow-up intervals.

Verbal memory tests (Table 1 and Table 2)

California Verbal Learning Test (CVLT)

Developed in 1987 (Delis et al., n.d.) and revised in 2000 (Delis et al., n.d.), CVLT is one of the most widely used verbal learning and memory tests in neuropsychology (Rabin et al., 2005). It can be used to assess both recall and recognition of the words from 4 different semantic categories (Spreen and Strauss, 1998). Administration time is approximately 20 min excluding 30 min delay intervals. Steps of administration are described in Table 1.

Table 1

Verbal Memory Assessments.

Test	Developed	Updated version	Assessed age	Administration	Key process	Measures obtained	Alternate versions available
CVLT	1987	CVLT-II (2000)	16–89	20 min and 30 min delay intervals: (1)5 trials recalling 16-word list A, (2) 1 trial Interference list B, (3) Short delay free and cued recall trials of list A, (4) 20 min delay interval of nonverbal tasks, (5) long-delay free-recall, long delay cued-recall of list A, (6) yes/no recognition trials of list A	Explicit recall and recognition of semantically related words presented over multiple trials	Immediate and delayed recall, retrieval efficacy, acquisition rate, learning strategy, proactive and retroactive interference, motivation	Yes
HVLT	1991	HVLT-R(2001)	13–80+	15 min and 20 min delay interval: (1) 3 learning trials of 12-word list, (2) a delay interval, (3) a delayed recall trial, (4) yes/no delayed recognition trial	Explicit recall and recognition of semantically related words presented over multiple trials	Long Term memory, recognition, forgetting	Yes
RAVLT	1957	RAVLT (1983 is the most commonly used)	6–97	15 min and 20 min delay interval:(1) 5 trials of free recall of 15-word list A, (2) Interference list of 15-word list B, (3) delayed recall of list A, (4) 20 min delay interval, (5) Recall of list A, (6) A story that uses all the words from list A and identifying of these words of list A	Explicit recall and recognition of unrelated words presented over	Working and long-term memory, learning, recognition, forgetting, proactive and retroactive interference, temporal order, motivation	Yes
WMS	1945	WMS-IV (2009)	16–89	30 min and 30 min delay interval: (1) Immediate recall of two paragraphs read aloud, (2) 30 min delay interval, (3) Delayed recall of two paragraphs, (4) Yes/no recognition test	Explicit recall and recognition of auditory information	Immediate and delayed recall and recognition of auditory material	No

This table is adapted from “A compendium of neuropsychological tests” (Spreen and Strauss, 1998). Copyright by the Oxford Publishing Limited. Reproduced with permission of the Licensor through PLSclear.

In our study, we found that 33 (27%) of the studies that measured verbal memory in patients with first-episode psychosis in the past ten years used CVLT. This result shows that CVLT is the second most widely used verbal memory measurement test in first-episode psychosis studies of past ten years. Of these 33 studies, 9 used paper-pen, none used computerized methods, and 24 study did not specify the tool. Nine of these studies had follow-up assessments of verbal memory while 24 of them had only a single assessment (Table 2).

Hopkins Verbal Learning Test (HVLT)

Devised in 1991 (Brandt, 1991) and updated in 1998 (Benedict et al., 1998), HVLT is one of the most widely used verbal memory assessments, especially in dementia (Spreen and Strauss, 1998). HVLT is widely used by schizophrenia researchers as a part of Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Consensus Cognitive Battery (Nuechterlein et al., 2008). HVLT assesses verbal recall and recognition of the words from 3 different semantic categories. Administration time is approximately 15 min excluding a 20 min delay interval. Steps of administration are described in Table 1.

Our results show that 14(12%) of the studies that measured verbal memory in patients with first-episode psychosis in the past ten years used HVLT. Seven of these studies used paper-pen, none used computerized methods, and 7 did not specify the tool. Six of these studies had follow-up assessments of verbal memory while 8 of them had only one (Table 2).

Rey Auditory Verbal Learning Test (RAVLT)

Developed in early 1900s by Eduard Claparede (Claparède, 1919) and published in 1958(“Lezak (1983) Neuropsychological Assessment. 2nd Edition, Oxford University Press, New York. - References - Scientific Research Publishing,” n.d.), RAVLT is one of the first verbal memory measurement tests. Although it has been updated many times, the most widely used version is published in 1983(“Lezak (1983) Neuropsychological Assessment. 2nd Edition, Oxford University Press, New York. - References - Scientific Research Publishing,” n.d.). RAVLT assesses working and long-term memory, learning, recognition, forgetting, proactive and retroactive interference, temporal order and motivation. Administration time is approximately 15 min excluding 20 min delay interval. Steps of administration are described in Table 1.

We found 34 (28%) of the studies in the past ten years used RAVLT to assess verbal memory in first episode psychosis patient population. This shows that, excluding the studies using two different verbal memory measurement tests, RAVLT is the most widely used verbal memory tests to assess first-episode psychosis patients in psychiatric studies of the past decade. Four of these studies used paper-pen, one used computerized methods, and 29 did not specify the tool. Nine of these studies had follow-up assessments of verbal memory while 25 study had only one initial assessment (Table 2).

Weschler Memory Scale-Logical Memory Scale (WMS)

WMS -LM was developed in 1945 (Wechsler, 1945) and updated to current form (WMS-IV) in 2009(“WMS-IV Wechsler Memory Scale 4th Edition,” n.d.). It is ranked the most widely used verbal memory measurement in neuropsychology field (Rabin et al., 2005; Spreen and Strauss, 1998). Among the subtests of WMS, Logical Memory Scale assesses verbal memory. In particular, this subtest assesses immediate and delayed recall and recognition of auditory material. Administration time is approximately 30 min excluding 30 min delay interval. Steps of administration are as stated in the Table 1.

We found that 34 (28%) of the studies in the past ten years used WMS to assess verbal memory in first episode psychosis patient population. This result shows that, besides of RAVLT, WMS is one of the two most widely used tests in first-episode psychosis research of past ten years including the studies used two different verbal memory measurement tests. Four of these studies used paper-pen, 1 study used computerized methods, and 29 study did not specify the tool. Ten of these studies had follow-up assessment of verbal memory while 24 study had only initial assessment (Table 2).

Other verbal memory measurement tests

Out of 120 studies identified from our search, 12(10%) study used wide range of verbal measurement tools other than CVLT, WMS, RAVLT, and HVLT. Four studies (Higuchi et al., 2013; Molina et al., 2018; Sawada et al., 2017; Suazo et al., 2012) used Brief Assessment of Cognition in Schizophrenia (BACS), 3 studies (Andersen et al., 2013; Jepsen et al., 2010; Palomares et al., 2019) used Buschke Selective Reminding Test (BSRT), 2 studies (Firth et al., 2018a, Firth et al., 2018b) used 12 Word Verbal Recall (CANTAB Software), 1 study (Dieleman et al., 2015) used 60 Word Test, 1 study (Caldiroli et al., 2016) used Word Learning Test (WLT), and 1 study (Allott et al., 2018) used Paired Associate Learning Test (PAL) to assess verbal memory. Two of these studies used paper-pen, 4 studies used computerized methods, and 6 studies did not specify the tool. Three of these studies had follow-up assessment of verbal memory while 9 studies had only initial assessment (Table 2).

Discussion

We found no standardized verbal memory measurement across studies of first-episode psychosis. Although there has been an increase in use of technology in mental health services over the past decade, only a small number of the studies reviewed (5/120; 4%) used computerized methods to measure verbal memory. To date, continuous and periodic verbal memory measurement are not widely used to monitor cognitive function in first-episode psychosis. Additionally, we found that only 30% of the 120 studies assessed had a follow-up assessment of verbal memory, with the majority being cross-sectional studies. Additionally, we found that only 36 (30%) of the 120 studies had a follow-up assessment of verbal memory, with the majority being single time point studies. Most of the follow-up measurements in these studies show remarkable and continuous decline of verbal memory performance in psychosis patients. To that extent, follow-up measurements also prove that verbal memory decline in FEP can be improved or slowed with timely therapeutic interventions. However, most of these studies have 1 year or longer follow-up intervals, which decreases possibility of early detection and intervention in verbal memory decline. To date, continuous and periodic verbal memory measurement are not widely used to monitor cognitive function in first-episode psychosis.

Our results show that the most widely used verbal memory measurement tests are the RAVLT (28%) and WMS (28%), followed by CVLT (27%), and HVLT (12%). Each of the four presented here have notable differences. Three (CVLT, HVLT, RAVLT) consist of word lists while WMS consists of reading paragraphs. Additionally, WMS is the only test without alternative versions for follow-up assessments. Studies using HVLT had the greatest follow-up ratio (43%) followed by WMS (29%), CVLT (27%), and RAVLT (26%). While use of any tool does not determine whether the study will feature follow up, it is interesting to note that HVLT was featured in more studies with follow up compared to the three others. CVLT is the most comprehensive, reliable, and predictive verbal memory measurement test among the 4 main tests (Beck et al., 2012; Silva et al., 2012; Spreen and Strauss, 1998), and can be used for more detailed and multiple assessments of verbal memory measurement.

Few studies used technology to administer assessments. Only 5/120 studies used computerized methods and none used digital formats to assess verbal memory. Of these 5 studies, 2 (Firth et al., 2018a, Firth et al., 2018b) used 12 Word Verbal Recall (CANTAB Software), 1 study (Dieleman et al., 2015) used 60 Word Test, 1 study (Caldiroli et al., 2016) used Word Learning Test (WLT), and 1 study (Knytl et al., 2019) used both RAVLT and WMS. In addition, of these 5 studies, only 1 study (Firth et al., 2018b) a follow-up assessment of verbal memory. There is no detailed explanation of the computerized administration methods of WMS and RAVLT in the study of (Knytl et al., 2019), and no visible examples of computerized or digitalized administration of HVLT or CVLT in first-episode psychosis research.

The successful digitization and validation of BACS (Atkins et al., 2017) called the BAC App offers a practical model for digital tool conversation and adaption, as the tablet-based version offers results consistent with the paper BACS (Atkins et al., 2017). Such results support the translation of other paper scales into other digital modalities such as smartphones. Given the increased scalability, accessibility, and portability of digital scales compared to their paper predecessors, the comparability of results supports further paper to digital translations. Other considerations to guide paper-to-digital scale translation, especially for scales to be used for longitudinal administration, include user experience (UX) testing to minimize use attrition in patients, and data security to ensure patients are comfortable engaging with digitized versions of these scales. When considering adopting digitized versions of these scales, comparisons against paper versions will need to be made in the context of measures such as convergent and discriminant validity (i.e. what are the degrees of relatedness and independence of different scale constructs). One recent study examined these validities using PCA (principal component analysis) results when comparing online (MyCQ) against paper-based (CANTAB) assessments of cognition in OCD, schizophrenia/schizoaffective disorder, and MDD patients (Domen et al., 2019). Another study has shown that results from digitally-administered tests can be used for separating first-episode psychosis patients from controls (Liu et al., 2019), which may be more useful in contributing to clinically actionable insights.

We propose that a brief verbal memory measurement via smartphone applications may facilitate longitudinal research and clinical care. Further benefits beyond longitudinal data include that timing of word reading would be uniform and spoken without an accent that might be consistent with a participant's expectation. Also, there would be no need for the rater to write words down while testing although often today it remains necessary for the rater to touch a box indicting the word was spoken. In the near future, natural language processing may enable fully remote testing and recognition of spoken words where a rater is not necessary. Software programs today can also automatically score results, removing any chance for human error. However, although one source of error will be removed, these tests are still scoring humans. As such, test performance validity will need to be assessed: is digital test performance impacted by patient engagement to the point that the test is no longer reflective of neurocognitive function (Greher and Wodushek, 2017)? It will be important for these digital tests to consider their scored results in the context of phenomena such as patient effort, in order to concurrently assess performance and embedded validity indicators. The CVLT already contains such an embedded test that has been utilized for assessing psychosis inpatients (Bayan et al., 2018), and creating and validating a digital version of this instrument will be key for fitting into a broader literature studying the interplay between effort and validity (e.g. Shura et al., 2018). Although efforts have been made to create a method that would lend itself to automated assessment of performance validity (Wolfe et al., 2010), attempts to generalize these methods have encountered varying measures of success (Donders and Strong, 2011). In addition, these assessments will need to be considered in the context of other digital systems that may affect effort and engagement, such as adherence-promoting reward systems (Pani and Keefe, 2019) and other gamification-based elements of apps that integrate testing with multiple dimensions of care.

While validating any new assessment is not a simple process, the potential of longitudinal cognitive data combined with digital phenotyping data offering concomitant data on environments, social functioning, symptoms, and even physiology offers a new window into cognitive assessment the field should not ignore. Such a digitalized brief verbal memory assessment may be able to detect and monitor verbal memory changes in first-episode psychosis patients remotely and continuously, without burnout of the users and administrators. Considering smartphone use is common among patients with psychosis (Torous et al., 2017; Torous and Keshavan, 2018), we believe that a digital verbal memory assessment using the most-widely used assessments and digital mental health applications bears investigation and validation. Indeed, verbal memory tests are already being administered in a digital format and enabling automated algorithm-based scoring of story recall (Holmlund et al., 2020). Validation and developing norms around new digital scales via international deployment will be an especially salient opportunity given the previously discussed scalability of these technologies. While pen-and-paper neurocognitive assessments have been extensively assessed, global studies are now possible that take advantage of ubiquitous technologies such as smartphones that will allow for cross-cultural testing, rapid prototyping and retesting, and portable updates that can be applied to multiple platforms concurrently and in a monitorable, quantifiable manner. It will be informative study how these digital formats can best be taken advantage of different languages and cultures in standardizing assessments.

Our paper has several limitations. The search term used was broad and will have missed several studies that did not feature the keywords searched for in the title and abstract. For example, several studies utilize the verbal memory tests in the BACS and CANTAB battery but these did not show up in our title and abstract search. We also did not search clinicalstudies.gov and thus our results reflect published research and not ongoing efforts which may be more likely to utilize newer technologies.

Conclusion

Verbal memory measurement can predict clinical and functional disability in first-episode psychosis patients. Although the literature supports the importance of verbal memory measurement in first-episode psychosis, we found no universally used and recommended verbal memory measurement test in first-episode psychosis literature. Although many verbal memory tests are widely used, almost none of them have a computerized or digitalized administration example – although some like the BACS do. Creating and validating further assessment tools that can gather longitudinal data offers a new window in cognition that when combined with other data streams from mobile devices could present a useful new tool for the field.

Declaration of competing interest

CMK, JB, DO, JT have no relevant COI. DO is supported by funding from the NIMH P50MH115846. JT is supported by funding from the NIMH K23 MH116130–02.

RSEK is an employee of Duke University and in the past 3 years has received research funding from the National Institute of Mental Health and Boehringer Ingelheim; in the past 3 years he has also received honoraria, served as a consultant, speaker, or advisory board member for AbbVie, Acadia, Aeglea, Akebia, Akili, Alkermes, Allergan, ArmaGen, Astellas, Avanir, AviNeuro/ChemRar, Axovant, Biogen, Boehringer-Ingelheim, Cerecor, CoMentis, Critical Path Institute, FORUM, Gammon Howard & Zeszotarski, Global Medical Education (GME), GW Pharmaceuticals, Intracellular Therapeutics, Janssen, Kempharm, Lundbeck, Lysogene, MedScape, Mentis Cura, Merck, Merrakris Therapetics, Minerva Neurosciences Inc., Mitsubishi, Montana State University, Monteris, Moscow Research Institute of Psychiatry, Neuralstem, Neuronix, Novartis, NY State Office of Mental Health, Orygen, Otsuka, Paradigm Testing, Percept Solutions, Pfizer, Pharm-Olam, Regenix Bio, Reviva, Roche, Sangamo, Sanofi, SOBI, Six Degrees Medical, Sunovion, Takeda, Targacept, Teague Rotenstreich Stanaland Fox & Holt, Thrombosis Research Institute, University of Moscow, University of Southern California, University of Texas Southwest Medical Center, WebMD, and Wilson Therapeutics. He receives royalties from versions of the BAC testing battery, the MATRICS Battery (BACS Symbol Coding), and the Virtual Reality Functional Capacity Assessment Tool (VRFCAT) and is a shareholder in VeraSci and Sengenix.