A systematic review of research on neuropsychological measures in psychotic disorders from low and middle-income countries: The question of clinical utility.

INTRODUCTION: Several studies of neuropsychological measures have been undertaken in patients with psychotic disorders from low- and middle-income countries (LMICs). It is, however, unclear if the measures used in these studies are appropriate for cognitive screening in clinical settings. We undertook a systematic review to determine if measures investigated in research on psychotic disorders in LMICs meet the clinical utility criteria proposed by The Working Group on Screening and Assessment.
METHODS: Preferred Reporting Items for Systematic Reviews and Meta-Analyses were employed. We determined if tests had been validated against a comprehensive test battery, the duration and scope of the tests, the personnel administering the tests, and the means of administration.
RESULTS: A total of 31 articles were included in the review, of which 11 were from Africa. The studies included 3254 participants with psychosis and 1331 controls. 3 studies reported on the validation of the test against a comprehensive cognitive battery. Assessments took 1 h or less to administer in 6/31 studies. The average number of cognitive domains assessed was four. Nonspecialized staff were used in only 3/31 studies, and most studies used pen and paper tests (17/31).
CONCLUSION: Neuropsychological measures used in research on psychotic disorders in LMICs typically do not meet the Working Group on Screening and Assessment clinical utility criteria for cognitive screening. Measures that have been validated in high-income countries but not in LMICs that do meet these criteria, such as the Brief Assessment of Cognition in Schizophrenia, therefore deserve further study in LMIC settings.

Introduction

Various neuropsychological measures have been developed for cognitive screening in patients with psychotic disorders (Reichenberg, 2010; Keefe and Fenton, 2007). This is because cognitive impairment is a key predictor of outcomes like quality of life in patients with psychotic disorders. Cognitive impairment contributes a larger portion of disease burden than behavioral, positive or negative symptoms of psychosis (Green et al., 2019; Emsley et al., 2008; Whiteford et al., 2013). Cognitive screening is therefore an essential component of routine care for patients with psychotic disorders (American Psychiatric Association, 2013; World Health Organization, 1992). Cognitive screening in psychotic disorders involves administering neuropsychological measures to assess for cognitive impairment. Although various cognitive domains can be impaired, research by the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) initiative recommended seven key domains for neuropsychological assessment in patients with psychotic disorders. These are i) working memory, ii) attention/vigilance, iii) verbal learning and memory, iv) visual learning and memory, v) reasoning and problem solving, vi) information processing speed, and vii) social cognition (Green et al., 2004; Nuechterlein et al., 2008).

The American Psychological Association's Working Group on Screening and Assessment (WGSA) have provided guidelines for assessing whether neuropsychological measures are appropriate for cognitive screening in clinical settings (American Psychological Association, 2014). WGSA was a collaboration of the American Psychological Association's Board of Professional Affairs and the Committee for the Advancement of Professional Practice of the American Psychological Association to help distinguish cognitive screening from comprehensive psychological evaluations (Roebuck-Spencer et al., 2017). Briefly, the guidelines state that for a measure to have clinical utility for cognitive screening it must be: a) able to identify early on individuals at high risk for impairment, b) be sensitive enough to determine those who need further review; c) be brief and narrow in scope; d) be administered as part of a routine clinic visit; e) be administered by clinicians or support staff or with electronic devices and; f) be used to monitor progress and outcomes (Roebuck-Spencer et al., 2017). Several brief neuropsychological measures such as the Brief Assessment of Cognition in Schizophrenia (BACS) have been shown to meet these criteria in high-income countries (HIC) (Fervaha et al., n.d.; Hurford et al., 2011).

In low- and middle-income countries (countries with a gross national income of less than $5101 https://data.worldbank.org/income-level/low-and-middle-income) there is a growing literature on neuropsychological measures for psychotic disorders (Araújo et al., 2015; Ayres et al., 2007; Nakasujja et al., 2012b; Ngoma et al., 2010). Such work has been useful in demonstrating the large burden of impairment and its association with poor outcomes (Ayres et al., 2007). However, it is unclear if measures that have been researched are appropriate for cognitive screening. In particular, it is unclear whether these neuropsychological measures meet the criteria for cognitive screening as outlined by the WGSA. It would be useful to know if the tests are used early in the course of the illness, whether these assessments have been validated against comprehensive neuropsychological batteries, duration and scope of the tests, setting where the tests are performed, the personnel administering the tests and whether the tests are used for follow up of patients.

Here we aimed to determine if the neuropsychological measures used in research on patients with psychotic disorders in low- and middle-income country contexts meet the six WSGA clinical utility criteria for cognitive screening. A systematic review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009). The study protocol was registered prior to data collection in the open access online registry, PROSPERO, University of York, York, United Kingdom, registration number CRD42018047872. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42018047872.

Methods

Study selection

Exclusion and inclusion criteria were determined using the PICOSS (Population, Intervention, Comparator, Outcomes, Study design, Setting) framework (Robinson et al., 2011). We considered articles written in English with no time limit on when the studies were conducted. The population of interest was participants with psychosis. Psychosis was defined as participants with schizophrenia spectrum and related psychotic disorders, bipolar affective disorder and depression with psychotic features. We selected these disorders given the current literature that highlights their shared genetic and neurobiological underpinnings (Rosen et al., 2012; Mark and Toulopoulou, 2016). The intervention included any study in which neuropsychological measures were performed in at least one cognitive domain. This was done to ensure that neuropsychological measures that are used for assessment and not screening, such as the MATRICS consensus cognitive battery, were excluded. The comparator was healthy controls. Our outcomes included the clinical diagnosis, whether these assessments had been validated against comprehensive batteries, duration and scope of the tests, setting where the tests were performed, the personnel who administered the tests and whether the tests are used for follow up of patients. All study designs irrespective of sample size were included in the review. The review was limited to the low and middle-income country setting. This was done due to the disparity in care between high income (GNI > $5101) and low-income countries (GNI < $5101).

Data sources, search strategy, screening and abstraction

In consultation with a librarian (RS), data sources included (a) electronic search of databases, (b) search for gray literature (conference proceedings, clinical trial registers) and (c) using the reference bibliography of full text articles to identify potentially relevant studies. The electronic search strategy followed the PICOS approach (Population, Intervention, Comparator, Outcome and Study design/setting), and was conducted in three databases including PubMed, Embase and PsychINFO. Only English language articles were included into the review. The complete search strategy is in the supplementary files. The search strategy used Boolean logic to combine terms in the PICOS framework. Articles were saved into Endnote (Brahmi and Gall, 2006), duplicates removed and two authors (EKM & JLO) independently screened the titles and abstracts to determine which articles were eligible for the review in parallel, before retrieving full texts. A consensus meeting was held when there was disagreement. For each study, abstracted data included name of the test, the domains they assessed, duration of assessments, personnel administering the tests, the types of assessments (paper vs computerized), and whether the tests had been validated against a gold standard.

Quality assessment

Studies with a poor risk of bias assessment were not excluded from the final analysis. However, duplicate publications were removed to limit publication bias. Bias assessment was undertaken by the primary reviewer (EKM) in consultation with DA, JLG and EAO.

Data synthesis

We performed a structured narrative synthesis were words and text are used to summarize and explain the findings of the review (Popay et al., 2006). Quantitative data was analyzed using Stata version 14. (StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP).

The criterion (a) of a test being able to identify early on individuals at high risk for impairment was assessed by abstracting the clinical diagnosis to determine which tests were performed early at the first episode of psychosis. The criterion (b) of a test needing to be sensitive enough to determine those who need further review was assessed by determining the tests that had been validated against a comprehensive neuropsychological battery. The criterion (c) of a test being brief and narrow in scope was assessed by the number of domains assessed and the duration of the assessment. The criterion (d) of a test administered as part of a routine clinic visit was assessed by determining the setting (inpatient versus outpatient) in which the tests were performed. These settings were chosen since neuropsychological assessments are performed on resolution of psychotic symptoms which is often in outpatient not inpatient settings (Harvey, 2013; Reichenberg, 2010). The criterion (e) of a test administered by clinicians or support staff with electronic devices was assessed by determining the mode of delivery of the test (pen and paper versus computerized) and the personnel administering the tests. Finally, the criterion (f) of a test used to monitor progress and outcomes was assessed by determining the studies that employed a longitudinal study design as well as those that assessed quality of life in participants.

Results

Study setting and population

The last search using PubMed was undertaken on April 10th, 2020, while the last search using PsycINFO & Embase, was undertaken on 18th October 2018. After removal of duplicates, eligible titles and abstracts were screened according to the inclusion criteria until a final list was agreed upon. The process is highlighted in Fig. 1. Thirty-one studies were included in the final analysis. The articles were published between 1994 and 2018 with many (15/31) published between the years 2000 and 2010. Seven (7) studies were from Central and South American countries, thirteen (13) were from Asian countries and eleven (11) were from African countries. South Africa had the highest number of individual studies making up 7 of the 31 studies. In total, the final table included 3254 participants with psychosis and 1331 controls.

Fig. 1

Article selection using PRISMA guidelines.

Early neuropsychological assessment

Only 3 studies were performed early among patients with a first episode of psychosis. The different diagnostic characteristics of the participants are shown in Table 1.

Table 1

Summary of studies included in the review.

Year	Author	Country	Study design	Population	Population group	Total number of participants	Diagnostic type	Number
1994	Gureje	Nigeria	CS	Inpatient	Black	128	Schizophrenia	43
							Mania	32
							HC	53
1997	Mattson	South Africa	CS	Outpatient	Caucasian	40	Schizophrenia positive symptoms	20
							Schizophrenia negative symptoms	20
2002	Ertugrul	Turkey	Case control	Outpatient	Caucasian	90	Schizophrenia	60
							HC	30
2002	Harvey	South Africa	CS	Outpatient	Caucasian	29	Schizophrenia English speaking	5
					Black		Schizophrenia Afrikaans speaking	24
2005	Aleptekin	Turkey	CS	Outpatient	Caucasian	69	Schizophrenia	38
							HC	31
2006	Leppanen	South Africa	CS	Outpatient	Black	84	Schizophrenia	44
							HC	40
2007	Salgado	Brazil	CS	Outpatient	Caucasian	40	Schizophrenia	20
				Inpatient			HC	20
2007	Trivedi	India	CS	Outpatient	Oriental	45	BPD	15
							Schizophrenia	15
							HC	15
2007	Ayres	Brazil	CS	Outpatient	Caucasian	553	Schizophrenia	98
							BPD	41
							Depression with psychosis	31
							HC	383
2008	Pradhan	Brazil	CS	Outpatient	Caucasian	103	BPD	48
				Inpatient			Schizophrenia	32
							HC	23
2008	Leppanen	South Africa	CS	Outpatient	African	81	Psychosis	36
							Siblings	23
							HC	22
2008	Savitz	South Africa	CS	Outpatient	Caucasian	230	HC	65
							BPD I	49
							BPD II	19
2008	Schneider	Brazil	CS	Outpatient	Caucasian	94	BPD	66
							HC	28
2009	Savitz	South Africa	CS	Outpatient	Caucasian	110	BPD with psychosis	25
							BPD without psychosis	24
							HC	61
2010	Ayres	Brazil	CS	Outpatient	Caucasian	160	Schizophrenia	56
							Affective psychosis	34
							HC	70
2010	Ngoma	Democratic Republic of Congo	CS	Inpatient	Black	341	HC	153
							Brief psychotic disorder	68
							Schizophreniform	50
							Schizophrenia	70
2010	Cabral-Calderin	Cuba	Longitudinal	Outpatient	Caucasian	68	Schizophrenia	34
							HC	34
2011	Mehta	India	CS	Outpatient	Oriental	18	Schizophrenia	9
							HC	9
2011	Guo	China	Longitudinal	Outpatient	Oriental	698	Schizophrenia	578
							Schizophreniform	120
2012	Nakasujja	Uganda	Longitudinal	Inpatient	Black	483	Mania	312
							Psychosis NOS	16
							Schizophrenia	100
							Depression	55
2013	Santosh	India	CS	Outpatient	Oriental	100	Schizophrenia	100
2014	Heeramun- Aubeeluck	China	Longitudinal	Outpatient	Oriental	101	FEP	101
2014	Okasha	Egypt	CS	Outpatient	Black	90	BPD	60
							HC	30
2014	Mazhari	Iran	CS	Outpatient	Persian	100	Schizophrenia	50
							HC	50
2015	Arau´ jo	Brazil	CS	Outpatient	Caucasian	174	Schizophrenia	116
							HC	58
2016	Hou	China	CS	Outpatient	Oriental	80	FEP	40
							HC	40
2016	Tang	China	CS	Outpatient	Oriental	148	Schizophrenia	94
							HC	54
2017	Charernboon	Thailand	CS	Outpatient	Oriental	72	Schizophrenia	36
							HC	36
2017	Zhou	China	Longitudinal	Inpatient	Oriental	49	FES	32
							HC	17
2017	Hendricks	South Africa	CS	Inpatient	Caucasian	29	Alcohol induced psychosis	13
							Alcohol use	16
2018	Sagar	India	Longitudinal	Outpatient	Oriental	178	BPD depressed	36
							BPD manic	41
							BPD euthymic	52
							HC	49

FES=first episode schizophrenia, HC = healthy controls, BPD = bipolar affective disorder, FEP = first episode psychosis, NOS = not otherwise specified.

Validation of tests

Only 3 out of 31 studies specifically evaluated a measure against a comprehensive neuropsychological battery. A summary of the publications and associated validation statistics are shown in Table 2.

Table 2

Studies in which a brief test was compared to a complete battery.

Author	Validated	Comparison group-selection criteria	Comparison group-size	Comparison tool	Sensitivity	Specificity	Reliability	Concurrent validity
Mehta, 2011	NR	NR	NR	NR	84.2	81	0.71	NR
Mazhari,2014	YES	HC	50	Standard battery	98.0	66.0	0.74	NR
Arau´ jo, 2015	YES	HC	58	BACS- French	NR	NR	0.874	0.625

BACS - Brief Assessment of Cognition in Schizophrenia; MOCA - Montreal Cognitive Assessment.

Scope and brevity of tests

The choice of the number of domains assessed differed across the publications with 8 out of 31 publications (25.8%) assessing for impairment in only one domain while 5 out of 31(17%) studies assessed for impairment in six domains. The proportions of domains assessed are shown in the bar graph (Fig. 2) below. Most tests assessed for impairment in the reasoning and problem-solving domain accounting for 24.64% of all tests in the studies. Fig. 3 highlights the proportions of tests that assessed the other domains. In 6 studies, the time taken to perform the assessments was less than hour. Domains assessed and the time taken to perform the tests is highlighted in Table 3.

Fig. 2

Bar graph showing number of domains assessed in the publications.

Fig. 3

Bar chart showing the proportion of domains assessed.

Table 3

Summary of tests used, domains they assess, duration and who performed test.

Author	Subtest/scale/battery	Domains assessed	Administration time (hours)	Mode of delivery	Person administering test	Training received
Gureje, 1994	. Verbal memory. Verbal Fluency. Design fluency. Wechsler Adult Intelligence Scale (WAIS) (Performance subtests). Wechsler Adult Intelligence Scale (WAIS) (Verbal subtests)	VLM, RP, WM, AV	NR	Pen and paper	Neuropsychologist	NR
Mattson, 1997	. Rey Auditory Verbal Learning Test (RAVLT). Wisconsin Card Sorting Test (Modified). Austin Maze. Rey Complex Figure (RCF). Controlled Oral Word Association Test (COWAT). Trail making test. Stroop Color and Word Test	VLM, RP, IP, AV	NR	Pen and paper	Clinical psychologist	NR
Ertugrul. 2002	. Wechsler Memory Scale Revised. Wisconsin card sorting test	AV, WM, RP, VSM	2	Pen and paper	NR	NR
Harvey, 2002	. Wechsler Memory scale (revised). Rey Auditory Verbal Learning Test (RAVLT). Continuous performance test (IP version). Verbal fluency. Wechsler Adult Intelligence Scale (WAIS). Wisconsin Card Sorting Test	WM, VLM, AV, IP, RP	NR	Pen and paper	Research assistants	YES
Aleptekin, 2005	. Wechsler Adult Intelligence Scale (WAIS). Controlled Oral Word Association Test (COWAT)	AV, WM, RP	NR	Pen and paper	NR	NR
Leppanen, 2006	. Facial affect recognition	Social cognition	NR	Computer	NR	NR
Salgado, 2007	. 15 item word list. Digit sequencing task. Token motor task. Category fluency. Symbol coding. Tower of London	VLM, WM, AV. RP, IP	0.72	Pen and paper	Psychiatrist	Single
Trivedi, 2007	. Wisconsin Card Sorting Test. Continuous performance test	RP, AV	NR	Computer	NR	NR
Ayres, 2007	. Controlled Oral Word Association Test (COWAT). Wechsler Adult Intelligence Scale (WAIS)	VSM, AV, WM	NR	Pen and paper		NR
Pradhan, 2008	. Wisconsin Card Sorting Test. Trail B. Controlled Words Association Test. PGI Memory scale. Bender Visual Motor Gestalt Test. Trail A	RP, VLM, WM, IP, AV	3.5	Pen and paper	NR	NR
Leppanen, 2008	. Facial affect recognition	Social cognition	NR	Computer	NR	NR
Savitz, 2008	. Wechsler Adult Intelligence Scale (WAIS). Controlled Oral Word Association Test (COWAT). Rey Complex Figure (RCF). Stroop Color and Word test. Rey Auditory Verbal Learning Test (RAVLT). Wisconsin Card Sorting Test (64 item)	AV, WM, VLM, VLM, IP, RP	1	Pen and paper	NeuropsychologistPsychiatric nurseGraduate students	Yes
Schneider, 2008	. Wechsler Adult Intelligence Scale (WAIS) III	VSM, WM, IP	NR	NR	NR	NR
Savitz, 2009	. Digits span. Controlled Oral Word Association Test (COWAT). Rey Complex Figure (RCF). Stroop Color and Word test. Rey Auditory Verbal Learning Test (RAVLT). Wisconsin Card Sorting Test (64 item)	WM, AV, VLM, RP, IP	1	Pen and paper	NeuropsychologistPsychiatric nurseGraduate students	Yes
Ayres, 2010	. Controlled Oral Word Association Test (COWAT). Wechsler Adult Intelligence Scale (WAIS)	VSM, AV, WM	NR	Pen and paper		NR
Ngoma, 2010	. Rey 15 Item. Rey Complex Figure (RCF). Letter number sequence task. Test of attention. Trail making test. Motor speed. Controlled Oral Word Association Test (COWAT). Stroop Color and Word test. Wisconsin Card Sorting Test (256 version). Trail making test	VLM, VSM, WM, AV, MS, RP	NR	Pen and paper	Clinical psychologist	NR
Cabral-Calderin, 2010	. Emotional Expression Multimorph task	social Cognition	NR	NR	NR	NR
Mehta, 2011	. Social cognition rating scale in Indian Settings	Social cognition	NR	NR	NR	NR
Guo, 2011	. Wechsler Adult Intelligence Scale (WAIS) (Revised). Wisconsin card sorting test. Wechsler Adult Intelligence Scale (WAIS) (Revised). Wechsler Memory Scale (Revised)	IP, RP, WM, VSM	NR	Pen and paper	Neuropsychologist	NR
Nakasujja, 2012	. WHO UCLA Auditory verbal learning test. Symbol digit modalities test. Verbal fluency. Wechsler Adult Intelligence Scale version III (WAIS)	VLM, AV, WM, RP, IP	NR	Pen and paper	NR	NR
Santosh, 2013	. Trail making test part B. Trail making test part A. Stroop test. Digit span. Verbal fluency test	RP, IP, AV, WM, VLM	NR	NR	NR	NR
Heeramun-Aubeeluck, 2014	. Paced Auditory Serial. Wechsler Memory Scale. Wechsler Adult Intelligence Scale (WAIS). Trail making. Hopkins Verbal Learning Test (Revised). Brief Visuospatial Memory Test (Revised)	WM, IP, VLM, VSM	NR	NR	NR	Yes
Okasha, 2014	. Weschler memory scale. Continuous performance tests. Wisconsin Card Sorting test	WM, VSM, VLM, AV, RP,	3.5	pen and paper	Research assistants	NR
Mazhari, 2014	. 15 item word list. Digit sequencing task. Token motor task. COWAT. Symbol coding. Tower of London. Trail making B	VLM, WM, AV. RP, IP	0.67	Pen and paper	NR	NR
Arau´ jo, 2015	. Rey Auditory-Verbal Learning Test. Wechsler Adult Intelligence Scale (WAIS) (Version III). Trail Making test. Controlled Oral Word Association Test (COWAT). Wisconsin Card Sort Test (128 cards)	VLM, WM, IP, VSM, AV, RP	0.68	Pen and paper	NR	NR
Hou, 2016	. Trail making. Stroop color word test. Hopkins Verbal Learning Test-Revised (HVLT-R)	IP, AV, VLM	NR	NR	NR	NR
Tang, 2016	. Facial emotional recognition task	SC	NR	NR	NR	NR
Charernboon, 2017	. Emotion perception. Theory of mind. Social knowledge	SC	NR	NR	NR	NR
Zhou, 2017	. Hopkins Verbal Learning Test-revised. The Verbal Fluency Test, Chinese version. The Color Trails Test. Stroop Color Word Test Chinese version. Cambridge PM Test (C-CAMPROMPT)	WM, VLM, AV, RP	NR	Computer	NR	NR
Hendricks, 2017	. Controlled Oral Word Association Test (COWAT). Trail Making Test. Rey Auditory Verbal Learning Test (RAVLT). Visual Reproduction Trails. Rey Complex Figure (RCF). Rey 15 Item. Wechsler Adult Intelligence Scale (WAIS) (South African). clock drawing test	VLM, AV, IP, WM, VSM, RP	NR	Pen and paper	Neuropsychologist	NR
Sagar, 2018	. Post graduate institute memory scale. National Institute of Mental Health and Neuro-Sciences neuropsychology battery. Verbal working memory	AV, WM,RP, VLM	1	Pen and Paper	Neuropsychologist	NR

WM = working memory, AV = attention/vigilance, VLM = verbal learning and memory, VSM = visual learning and memory, RP = reasoning and problem solving, IP = information processing speed, and SC = social cognition. NR = Not reported.

Setting where tests were performed

Table 1 highlights the clinic setting in which the tests were performed. Twenty-four studies were conducted in an outpatient population, 5 were carried out among inpatients and 2 in both outpatient and inpatient populations.

Administration of tests with support of technology

Four studies (Harvey et al., 2003; Okasha et al., 2014; Savitz et al., 2009; Savitz et al., 2008) used nonspecialised health professionals to perform the tests. Most tests were performed by a neuropsychologist; or by trained research assistant/graduate trainee. Four out of 31 studies used computerized assessments. This is highlighted in Table 3.

Follow up of participants

Only one study (Alptekin et al., 2005) reported on the quality of life of the participants. Only 6/31 studies (19.4%) utilised a longitudinal study design. These longitudinal studies are highlighted in Table 1.

Risk of bias across studies

There was extensive publication bias (p ≤ 0.005) as shown in the funnel plot (Fig. 4). Published studies (circles) and unpublished studies (squares) in the funnel plot were estimated from the trim-and-fill method. The solid line corresponds to adjustments for the impact of publication bias summary effect and the dashed line to the unadjusted summary effect.

Fig. 4

Filled funnel plot with pseudo 95% confidence limits showing publication bias.

Discussion

The research done to date suggests several gaps in the field. Only three studies performed neuropsychological assessments in patients with a first episode of psychosis (Heeramun-Aubeeluck et al., 2015; Hou et al., 2016; Zhou et al., 2017). It is recommended that assessments are performed at the earliest opportunity (American Psychiatric Association, 2013; World Health Organization, 1992; Reichenberg, 2010; Keefe and Fenton, 2007). There is need to validate tools for use among patients with a first episode of psychosis in LMICs (González-Blanch et al., 2011; Moreno-Granados et al., 2014).

The number of studies in which tests were validated against a comprehensive neuropsychological battery was low (Mehta et al., 2011; Mazhari et al., 2014; Araujo et al., 2015). More studies in which the performance of brief tests is compared against comprehensive batteries like the MATRICS consensus cognitive battery (MCCB) are needed. To date most studies have compared the performance of comprehensive batteries like the CogState and the MOCA which have little utility in LMICs with MCCB (Gil-Berrozpe et al., 2020; Lees et al., 2015). However, the results of these studies provide some support for validity.

In 6 out of 31 studies (Salgado et al., 2007; Savitz et al., 2008; Savitz et al., 2009; Mazhari et al., 2014; Araujo et al., 2015; Sagar et al., 2018), the assessment took <1 h to assess for impairment in five domains, which is attractive for clinical application. Most tests assessed for impairment in the reasoning and problem-solving domain (Gureje et al., 1994; Mattson et al., 1997; Ertuğrul and Uluğ, 2002; Harvey et al., 2003; Alptekin et al., 2005; Ayres et al., 2007; Salgado et al., 2007; Trivedi et al., 2007; Pradhan et al., 2008; Savitz et al., 2008; Savitz et al., 2009; Ngoma et al., 2010; Guo et al., 2011; Nakasujja et al., 2012a; Mazhari et al., 2014; Araujo et al., 2015; Hendricks et al., 2017; Zhou et al., 2017; Sagar et al., 2018). This seems clinically useful given literature from high-income countries that the greatest burden of cognitive impairment is in the cognitive domains of attention/vigilance, memory and reasoning and problem-solving among chronic patients (Rund, 2002).

In this review, only three studies used nonspecialised staff to perform the neuropsychological assessments (Harvey et al., 2003; Savitz et al., 2008; Savitz et al., 2009). This raises concern about the clinical utility of these measures in LMICs, where there are few specialized staff (Evans-Lacko et al., 2019; Mugisha et al., 2017; Semrau et al., 2015). mHealth apps may be more cost-efficient and feasible for delivery by non-specialized staff (Istepanian et al., 2004; Nicholas et al., n.d.; Robbins et al., 2018). Evidence suggests that tests delivered via mHealth applications are more efficient, accurate, accessible and interactive than assessments delivered via pen and paper (Bakkour et al., 2014).

Most tests were performed using pen and paper tests (Gureje et al., 1994; Mattson et al., 1997; Ertuğrul and Uluğ, 2002; Harvey et al., 2003; Alptekin et al., 2005; Ayres et al., 2007; Salgado et al., 2007; Pradhan et al., 2008; Savitz et al., 2008; Savitz et al., 2009; de Mello Ayres et al., 2010; Ngoma et al., 2010; Guo et al., 2011; Nakasujja et al., 2012a; Mazhari et al., 2014; Okasha et al., 2014; Araujo et al., 2015; Hendricks et al., 2017; Sagar et al., 2018). Limitations of paper based assessments include human error in data collection, the additional time required to score the assessments after they have been administered, costs associated with obtaining copyrighted and proprietary forms, and the burden of transporting and storing hard-copy questionnaires (Robbins et al., 2014a). Further work is warranted on the use of electronic assessments using mobile technology (mHealth applications, or “apps”) as is already being done in other populations such as persons living with HIV/AIDS (Brian and Ben-Zeev, n.d.; Robbins et al., 2014b; Robbins et al., 2018).

Few studies were of a longitudinal study design, and so it is unclear whether these tests are useful for monitoring progress and outcomes (Cabral-Calderin et al., 2010; Guo et al., 2011; Nakasujja et al., 2012a; Heeramun-Aubeeluck et al., 2015; Zhou et al., 2017; Sagar et al., 2018). Also, only one study assessed for quality of life as an outcome (Alptekin et al., 2005) highlighting the strong associations between cognitive impairment and quality of life. There is need for further studies on the ability of tests to be used in longitudinal studies.

One limitation of our own work deserves emphasis: we searched for English publications only and so may have missed studies published in other languages. Also, the WGSA criteria are not entirely specific on what characteristics constitute the threshold for meeting a criterion. We welcome scrutiny of our study descriptions into what may constitute meeting the WGSA criteria. A further limitation is that our criteria were based on findings from research on these measures, whereas the criteria are intended to address clinical use of these measures.

In conclusion, measures that have been used in research on psychotic disorders in low- and middle-income countries meet only some WCGA clinical utility criteria. Several candidate assessments are, however, attractive in terms of their scope and duration, and at least one of these, the Brief Assessment of Cognition in Schizophrenia; has been validated in high-income settings (Chianetta et al., 2008; Keefe et al., 2008; Salgado et al., 2007; Mazhari et al., 2014; Araujo et al., 2015). Further work on the administration of measures performed by non-specialized staff using mHealth apps is recommended in low and middle-income contexts.

List of abbreviations

low- and middle-income country

application

mobile health

CRediT authorship contribution statement

Emmanuel K. Mwesiga:Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing - original draft, Writing - review & editing.Dickens Akena:Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Validation, Writing - original draft, Writing - review & editing.Nastassja Koen:Conceptualization, Writing - review & editing.Richard Senono:Data curation, Software, Writing - review & editing.Ekwaro A. Obuku:Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing - review & editing.Joy Louise Gumikiriza:Data curation, Investigation, Validation, Writing - review & editing.Reuben N. Robbins:Resources, Writing - original draft, Writing - review & editing.Noeline Nakasujja:Conceptualization, Supervision, Writing - review & editing.Dan J. Stein:Conceptualization, Funding acquisition, Resources, Writing - original draft, Writing - review & editing.

Declaration of competing interest

Dr. Robbins is supported by funding from the US (P30-MH43520; PI: Robert H. Remien; R01-HD095256; PI: Reuben N. Robbins; R21-HD098035; PI: Reuben N. Robbins). The other authors declare no conflict of interest.