Cognitive deficits characterization using the CogState Research Battery in first-episode psychosis patients.

The computer-based CogState Research Battery (CSRB) proposes a test structure which follows MATRICS recommended cognitive domains but lacks direct comparison to pen and paper batteries in first-episode psychosis (FEP). The aim of this study was to compare performances obtained with the CSRB and a pen and paper battery in a historical cohort of FEP patients. Among patients entering an early intervention program between 2003 and 2014, separate cohorts completed the traditional pen and paper cognitive battery (n = 182) and the CSRB (n = 97). Composite z-scores were derived using normative data of matched controls (n = 64 pen and paper, n = 69 CSRB) and were compared between the two batteries for the 7 cognitive domains. The cohort tested using the CSRB performed better on the domains of processing speed, attention, visual memory, and verbal memory than the cohort tested using the pen and paper battery (all p < 0.001). Performance did not differ between the two types of batteries for the working memory, executive functions, and social cognition domains. Cognitive profiles identified in the two patient cohorts were similar, with verbal memory being the most impaired domain. Better performances on the CSRB may be primarily due to the minimal demand of the computerized tests on graphomotor abilities and reading speed compared to the pen and paper tests. Our investigation offers a better understanding on how the results obtained with computerized batteries may compare to earlier work done with traditional tests.

Introduction

In recent years, computerized cognitive batteries have been increasingly used in schizophrenia research, particularly in randomized-control trials which require retesting participants. This testing modality is attractive to researchers for many reasons, namely precise stimulus presentation and response recording, automated administration and scoring, reduced administration and scoring time, and the possibility of multiple equivalent alternate (Collie et al., 2001, Collie et al., 2003). It is important to better understand how the results from such batteries compare to previous reports using traditional pen and paper neuropsychological tests. So far, research focusing on computerized neurocognitive batteries involved mostly patients with chronic schizophrenia and investigated their construct validity (Forbes et al., 2012, Irani et al., 2012, Pietrzak et al., 2009a, Pietrzak et al., 2009b, Ritsner et al., 2006, Silverstein et al., 2010, Snyder et al., 2008, Yoshida et al., 2011).

The use of computerized cognitive batteries in the early phases of schizophrenia or related psychotic disorders has not yet been sufficiently validated. The CogState Research Battery’s (CSRB) structure follows the recommended MATRICS cognitive domains of processing speed, attention, working memory, visual learning and memory, verbal learning and memory, executive functions, and social cognition for research in schizophrenia (Nuechterlein et al., 2004).This battery has been previously validated in a chronic schizophrenia sample against the pen and paper MATRICS Consensus Cognitive Battery (MCCB) in chronic schizophrenia (Pietrzak et al., 2009a). Strong correlations were found between the performance on each test of the CSRB and the MCCB in control and patient samples and sensitivity to deficits was comparable (Pietrzak et al., 2009a). To date, the CSRB has not been directly compared to a pen and paper battery in a sample of patients experiencing psychosis in first-episode psychosis (FEP). It is unclear whether resulting cognitive performance would be equivalent in this population since factors like age, familiarity with computers, and inclusion of all subtypes of psychosis in a sample of mostly out-patients as opposed to samples limited to schizophrenia or low-functioning patients may influence the outcome.

This study aims to compare the cognitive performance of FEP patients as measured by two different batteries covering the 7 MATRICS-suggested cognitive domains; one in conventional pen and paper form and the other in computerized form. Since 2003, patients enrolled in the Prevention and Early Intervention Program for Psychoses (PEPP-Montreal) are approached for cognitive testing. In 2010, the program switched from their pen and paper battery to the CSRB, giving us the unique opportunity to compare the results of these batteries in a historical cohort design. Since we are studying a FEP cohort, we expect their level of cognitive impairment to be milder than what was reported in the investigation of Pietrzak et al. (2009a) of patients who have been ill for an average of 20 years. We also expect that cognitive profiles will be similar between the two batteries but that performance may differ when separate cognitive domains are compared according to testing modalities.

Methods

Participants

All patients were recruited and treated through the PEPP-Montreal, a specialized early intervention service at the Douglas Mental Health University Institute in Montreal, Canada. People aged 14 to 35 years from the local catchment area suffering from affective and non-affective psychosis were admitted to the program, the majority as out-patients. Patients enrolled had not taken antipsychotic medication for more than one month, and had an IQ higher than 70. Diagnoses were established according to the Structured Clinical Interview for DSM-IV (First et al., 1998) and confirmed between two senior research psychiatrists co-authors of this study (A.M. & R.J.). For complete program details see Malla et al. (2003) or ). All patients entering the program were presented information regarding a longitudinal study of cognition outcome in FEP and data from patients who signed the consent form were used in this study.

Healthy control participants were also recruited from the community for the same longitudinal study and were matched for age, gender, and parental socioeconomic status (SES) to the patients. Inclusion criteria for healthy controls included no history of Axis I disorders, neurological diseases, or traumatic brain injury, and no first-degree family member suffering from schizophrenia or related psychosis.

Research protocols were approved by the Douglas Mental Health University Institute research ethics board and McGill University Faculty of Medicine review board.

At the time of this analysis, 521 patients met the inclusion criteria for this study. Of those, 87 refused to sign the consent form, 104 could not be scheduled for testing and 51 had incomplete neuropsychological data, and therefore had to be excluded. Final analyses include 182 patients and 64 healthy controls who completed the pen and paper battery and 97 patients and 69 controls who completed the CSRB.

Patients with complete neuropsychological data did not differ from those with missing data on sex (Χ = 1.472, p = 0.225) or age (t = 1.403, p = 0.152). However, patients who did not complete the evaluation had significantly lower symptom levels on the Scale for Assessment of Positive Symptoms (SAPS; (Andreasen, 1984b) (t = 2.430, p = 0.015) and the Scale for the Assessment of Negative Symptoms (SANS; (Andreasen, 1984a) (t = 2.153, p = 0.032). Patients with more severe symptoms may have been utilizing clinical services more frequently and consequently may have been easier to schedule for testing compared to patients with minimal symptoms which visited the clinic less frequently. The demographic information for the groups included in the main analyses can be found in Table 1.

Table 1

Demographic and clinical variables.

	Patients Paper group (n = 182)	Patients CSRB group (n = 97)	Controls Paper group (n = 64)	Controls CSRB group (n = 69)	Statistic	p
Age	23.71 (3.63)	24.09 (5.10)	24.09 (2.91)	24.77 (6.39)	F = 0.929	0.427
Gender (M:F)	129:53	76:21	43:21	49:20	χ2 = 2.826	0.419
Parental SESa, b	2.86 (1.10)	2.71 (1.01)	3.00 (0.95)	2.79 (0.99)	χ2 = 10.018	0.349
Years of educationc	11.74 (2.34)	11.85 (2.40)	14.61 (2.51)	13.57 (2.04)	F = 26.837	< 0.001
Antipsychotics total dose (mg/day in clz equivalents)a	484.40 (859.34)	312.56 (243.71)	–	–	t = 2.370	0.019
Symptom levelsa
SAPS total	15.18 (15.96)	11.40 (12.54)	–	–	t = 2.110	0.036
SANS total	24.41 (13.44)	20.19 (12.49)	–	–	t = 2.444	0.015

Due to incomplete records sample sizes are as follows for Parental SES: Patients Paper group n = 113, Patients CSRB group n = 36, Controls Paper group n = 63 and Controls CSRB group n = 67; for Years of education: Patient Paper n = 177, Patients CSRB n = 85, Controls Paper n = 61, Controls CSRB n = 68; for Antipsychotics total dose: Patient Paper group n = 159, Patients CSRB group n = 97;for SAPS total: Patients Paper group n = 171, Patients CSRB group n = 91;and for SANS total: Patients Paper group n = 170, Patients CSRB group n = 88.

Hollingshead parental socio-economic status, in which 1 = highest and 4 = lowest.

Post-hoc analyses show: Patient Paper, Patient CSRB < Controls Paper, Controls CSRB.

Measurements

All patients were approached within the first 6 months of treatment for the cognitive assessment, which had been in pen and paper format between 2003 and 2010. Since September 2010, a new testing protocol was put in place, which includes the CSRB and therefore, the type of cognitive assessment could not be randomized. Testing with healthy controls was done during the same time periods with each battery. Cognitive assessments of patients were done on average 13.74 weeks (median: 8.29) after intake into treatment for those who did the pen and paper battery and 16.18 weeks (median: 9.86) after intake for those who did the CSRB, however the difference was not statistically significant (t = − 0.920, p = 0.358). Assessments took place when patients were in a stable enough but not necessarily an asymptomatic condition, and were administered by trained research staff or graduate students under the supervision of an accredited neuropsychologist (M.L.). Cognitive assessments were done on site at PEPP-Montreal and evaluators were supervised by a neuropsychologist (M.L.) for both pen and paper and CSRB protocols. Regarding the CSRB administration, the evaluator remained in the room while the patients underwent testing to explain the tasks, verify that they were performed optimally and record any relevant observations. This also ensured that data were recorded properly; therefore only two patients had incomplete data among those eligible for the CSRB, due to a computer problem (excluded due to incomplete data: pen & paper: 15%, CSRB: 1%; χ2 = 22.31, p < 0.001).

Demographic and clinical data

The following information was collected at the time of neurocognitive testing to characterize the two patient groups: age, type and dosage of antipsychotics taken daily, and parental SES measured with the Hollingshead two-factor index (Hollingshead, 1965). Symptom assessments with the SAPS and SANS are done at multiple predetermined time-points during the course of treatment and ratings done within a month of cognitive testing were used if available. The SANS total score was calculated without the attention items because of their overlap with the cognitive variables. We also removed the items of poverty of content of speech (alogia scale) and inappropriate affect (affective flattening scale) since factor analytic studies have shown them to belong to the disorganized symptom cluster (Liddle, 1987, Malla et al., 1993). All other items of the affective flattening and alogia scale were used as well as all items of the avolition–apathy and anhedonia–asociality scales. Antipsychotic doses were converted to chlorpromazine equivalents according to the literature (Jensen and Regier, 2010, Leucht et al., 2014, Woods, 2003) to allow comparison between patient groups.

Neurocognitive batteries

The tests composing each battery and their outcome measures are described in Table 2. The pen and paper battery took approximately 2 h to administer, while the CSRB took 1 h.

Table 2

Batteries and tests descriptions.

	Pen & Paper Battery		CogState Research Battery
Cognitive domain	Test	Outcome measure	Test	Test description	Outcome measure
Processing speed	Digit symbola	Number of correct symbols in 120 s	Groton Maze Chase Test	Click the tiles of a grid to trace the path of a target which moves when it is reached.	Average number of correct moves per second
	Trail Making Test Ab	Completion time	Detection task	Click as soon as a playing card turns over.	Average reaction time for correct responses
	Stroop Test: Wordc	Number of correct words read in 1 min
	Stroop Test: Colorc	Number of correct colors named in 1 min
Attention	D2 testd	Concentration performance	Identification task	Determine whether a playing card is red or black as fast as possible after it has flipped over by clicking the left or right mouse button.	Response accuracy
	Stroop Test: Inhibitionc	Number of correct colors named in 1 min
Working memory	Digit Spana	Raw score	One-back task	Determine whether the current playing card shown is the same or different from the previous one or the second-previous one.	Response accuracy
	Corsi Spatial Spane	Raw score	Two-back task
Visual memory	Visual Reproduction: Immediate Recalle	Raw score	One-Card Learning task	Click the left mouse button if he or she had seen the playing card shown previously during the task.	Response accuracy
	Visual Reproduction: Delayed Recalle	Raw score	Continuous Paired Associate	Learn and remember abstract pictures hidden beneath different locations on the screen.	Response accuracy
			Groton Maze Learning task: Delayed Recall	Find the hidden pathway by clicking tiles while avoiding to click tiles located diagonally to the last correct tile, click the same tile twice or move backwards on the previously discovered pathway. This is done over 5 trials. At the end of the test battery the participant is asked to remember the hidden pathway previously learned.	Total number of errors
Verbal Memory	Logical Memory: Immediate Recalle	Raw score	International Shopping List: Immediate Recall	A list of 16 words is read to the participant for 3 consecutive trials; after each trial the participant is asked to recall as many of the words as possible. Participants are also asked to recall as many words as possible at the end of the test battery.	Total number of words recalled over 3 trials
	Logical Memory: Delayed Recalle	Raw score	International Shopping List: Delayed Recall		Total number of words recalled
	Logical Memory: Recognitione	Raw score
Executive functions	Block designa	Raw score	Groton Maze Learning task	As described above	Total number of errors after 5 trials
	Trail Making Test Bb	Completion time	Set-Shifting Task	A playing card appears on the screen; if the word “color” appears above the card, the participant has to guess whether red or black is correct; if the word is “number” they have to guess whether the number on the card is correct. Feedback is provided with each response to teach the participant the underlying rule of each trial which changes without warning; participants are told at the beginning of the task that the correct card will change throughout.	Response accuracy
Social Cognition	Missing Cartoonsf	Number of correct responses	Social–Emotional Cognition task	The participant is shown 4 pictures at a time and must decide which one is different from the others as quickly as possible. Stimuli include emotional faces or eyes.	Response accuracy

(Wechsler, 1997a).

(Reitan, 1992).

(Stroop, 1935).

(Brikenkamp and Zillmer, 1998).

(Wechsler, 1997b).

(O'Sullivan and Guilford, 1976).

Statistical analyses

A composite score was calculated for each cognitive domain in both neurocognitive batteries by averaging the z-scores for all tests within each domain and an overall composite score was calculated for each battery by averaging z-scores of all individual tests. The performance of the matched control groups was used as normative data to derive these z-scores. To verify matching, the two patient groups and the two control groups were compared on age and number of years of education using one-way ANOVAs, and on sex and parental SES with Chi-square tests. Cognitive performance between the two patient groups was compared using a mixed-design ANOVA with cognitive domain as the within group factor and type of neurocognitive battery as the between group factor. Independent t-tests were used to compare antipsychotic dosage and symptom totals.

All analyses were conducted using SPSS version 20 (SPSS, Chicago, IL, USA) and were two-tailed with a critical p-value of 0.05, except where noted. The Visual Reproduction immediate and delayed recalls had to be log transformed and the Logical Memory: Recognition subtest had to be square root transformed due to significant negative skewness. All other variables were normally distributed.

Results

Demographic variables

The 4 participant groups did not differ on age, gender or parental SES but patients had fewer years of education than controls (see Table 1). The two patient groups differed significantly on their average total dose of antipsychotics taken daily (p = 0.019) and symptom levels (p = 0.036 for SAPS and p = 0.015 for SANS) at time of testing. Correlations were explored between these variables and cognitive performance yielding only weak coefficients and therefore were not used as covariates in the main analysis. At the time of cognitive testing, total dose of antipsychotics taken daily showed its strongest correlation with processing speed (r = − 0.232), SANS total score correlated most with verbal memory (r = − 0.241) and SAPS total score showed no correlation with most of the cognitive domains.

Cognitive performance

The interaction between cognitive domains and type of cognitive battery used was statistically significant (F(6,1662) = 18.63, p < 0.001) and therefore post-hoc comparisons were performed on each domain corrected for multiple comparisons (p = 0.05/7). The cohort tested using the CSRB performed better on the domains of processing speed, attention, visual memory, and verbal memory than the cohort tested using the pen and paper battery (all p < 0.001). Performance did not differ between the cohorts tested using the two types of batteries in the working memory (p = 0.16), executive functions (p = 0.26), and social cognition domains (p = 0.14). Fig. 1 shows the cognitive profiles obtained. Table 3 reports patients’ z-scores for individual tests.

Fig. 1

Cognitive performance of patients on the pen and paper battery and the CSRB (Wolf, 1986).

Table 3

Z-scores for individual neurocognitive tests.

	Pen & Paper (n = 182)		CSRB (n = 97)
Cognitive domain	Test	z-score M (SD)	Test	z-score M (SD)
Processing speed	Digit symbol	− 1.35 (1.05)	Groton Maze Chase Test	0.15 (1.00)
	Trail Making Test A	− 0.39 (1.26)	Detection task	0.57 (1.58)
	Stroop Test: Word	− 0.96 (1.11)
	Stroop Test: Color	− 1.22 (1.22)
Attention	D2 test	− 1.38 (1.23)	Identification task	− 0.19 (1.14)
	Stroop Test: Inhibition	− 0.84 (1.05)
Working memory	Digit Span	− 0.76 (0.97)	One-back task	− 0.54 (1.23)
	Corsi Spatial Span	− 0.77 (0.98)	Two-back task	− 0.66 (1.27)
Visual memory	Visual Reproduction: Immediate Recall	− 1.06 (1.36)	One-Card Learning task	− 0.52 (1.24)
	Visual Reproduction: Delayed Recall	− 1.09 (1.07)	Continuous Paired Associate	− 0.33 (0.80)
			Groton Maze Learning task: Delayed Recall	− 0.76 (1.40)
Verbal Memory	Logical Memory: Immediate Recall	− 1.66 (1.19)	International Shopping List: Immediate Recall	− 1.00 (1.34)
	Logical Memory: Delayed Recall	− 1.30 (1.08)	International Shopping List: Delayed Recall	− 0.77 (1.25)
	Logical Memory: Recognition	− 1.27 (1.32)
Executive functions	Block design	− 0.67 (1.21)	Groton Maze Learning task	− 0.55 (1.31)
	Trail Making Test B	− 1.14 (1.38)	Set-Shifting Task	− 0.94 (1.39)
Social Cognition	Missing Cartoons	− 0.69 (1.09)	Social- Emotional Cognition task	− 0.48 (1.22)

A significant main effect of type of neurocognitive battery used was found (F(1,277) = 36.88, p < 0.001, partial η2 = 0.12) where patients who did the CSRB performed better overall than those who did the pen and paper battery. There was also a main effect of cognitive domain (F(6,1662) = 24.84, p < 0.001, partial η2 = 0.08) indicating that level of impairment was not similar on all cognitive domains when results on both batteries were combined. Performance on processing speed was significantly better than that of all other domains (all p < 0.01). Verbal memory was significantly more impaired than all other domains (all p ≤ 0.001) whereas performance on attention, working memory, visual memory, and executive functions did not significantly differ (all p > 0.1). Social cognition performance was not different from attention and working memory but patients had a significantly better performance on the social cognition domain compared to executive functions (p = 0.03).

Discussion

The goal of this study was to compare impairment levels of FEP patients on a traditional pen and paper cognitive battery and the computerized CSRB. Performance on working memory, executive functions and social cognition did not significantly differ but processing speed, attention, visual and verbal memory performance of patients was better when assessed with the CSRB (see Fig. 1). We also observed that verbal memory was the most impaired domain regardless of type of battery used. Keeping in mind the limitations of our research design, we have gained a better understanding of the advantages that each battery has to offer. Overall, the pen and paper battery is more sensitive to deficits in some domains, while sensitivity is similar to that of the CSRB in others. Other than the considerably reduced administration and scoring time required by the CSRB, it should be noted that fewer patients were excluded from analyses due to incomplete data using the CSRB compared to the pen and paper battery. It is fundamental to keep these differences in mind when deciding which testing modality is best suited to a particular research question. We believe the possible explanations for the differences in processing speed, attention and memory pertain mostly to the nature of the tasks used, and therefore how cognitive domains are operationalized in each battery.

The most striking difference between batteries is in the processing speed and attention. An important factor which could explain this variation is the difference in graphomotor ability demands. The pen and paper battery uses a variety of tasks in the processing speed and attention domains which require writing or tracing (Digit Symbol Coding, Trail A, D2) while this process is completely eliminated in the computerized tasks. This could mean that even in the very early stages of treatment, patients may show some fine motor skills difficulties, however this remains to be directly studied. Some reports suggest that even short neuroleptic exposure can have an impact on fine motor speed in patients when compared to controls (Grootens et al., 2009, Krieger et al., 2001). Interestingly, we found that patients tested with the pen and paper battery were on average more medicated than those who did the CSRB based on chlorpromazine equivalents. However, this may be an artifact of the conversion formulas: the prescription profile across the 11-year span of our data collection shows a significant difference in the proportion of patients taking oral versus injectable antipsychotics (pen and paper: 7% injectable, CSRB: 22% injectable; χ2 = 10.48, p = 0.001). When converting injectable medication to chlorpromazine equivalents the only available formula is for the oral form of the same molecule but the minimum efficient dose may not be the same. Further investigations are needed to evaluate the best conversion factor for commonly prescribed injectable antipsychotics to chlorpromazine equivalents.

The Stroop test was used in the processing speed and attention domains for the pen and paper battery. In addition to the increased complexity of the reading-based task compared to the reaction-time based CSRB tasks, Stroop performance has been shown to correlate with level of education (Van der Elst et al., 2006), which could have further disadvantaged the patients on the pen and paper protocol. Furthermore, the tasks comprising the processing speed and attention domains of the pen and paper battery require more complex processing than those used to compose the same domains in the CSRB. Specifically, the Digit Symbol Coding task relies on many other cognitive skills over and beyond processing speed, namely visual scanning, attention, working memory, and mental flexibility among others. Performance on this task is one of the most impaired in schizophrenia among traditional neuropsychological tests (Dickinson et al., 2007); however, in the CSRB, processing speed is operationalized differently and this leads to discrepant results. Similarly, the Stroop word-reading task is known to activate the lexical pathway while the CSRB avoids demands on language in most tasks by using playing cards as stimuli, which reduces cultural confounds.

Visual memory and verbal memory performances were also significantly different between batteries. Graphomotor ability could have been a factor in these domains as well, since the WMS-III Visual Reproduction requires drawing whereas the CSRB visual memory tasks do not. Furthermore, the CSRB tasks rely less on recall than recognition, unlike the pen and paper tasks, and although still impaired, deficits in recognition are not as prominent as those of recall (Pelletier et al., 2005) and this may have influenced the results. As for the verbal memory, it remains the most impaired cognitive domain regardless of battery used. Compared to the performance of healthy controls, FEP patients did better when asked to recall a list of words which can be grouped in 3 semantic categories than when asked to recall short stories. Both these tasks offer the possibility of using semantic information to improve performance but the list presentation provides increased structure and therefore supports better encoding. However a finer analysis of the encoding and recall strategies would be needed to better explain this difference.

We have obtained different results than the previous validation study of CSRB in chronic schizophrenia. Pietrzak et al. (2009a) which concluded that both batteries were equally sensitive to detect cognitive deficits since the effect sizes for the composite scores were of − 1.50 for both the CSRB and MATRICS battery (Pietrzak et al., 2009a). The effect sizes in our investigation were smaller in our pen and paper battery (d = − 1.03) and CSRB (d = − 0.48). This is most likely due to the differences in clinical characteristics of patients between the two studies. While Pietrzak et al. (2009a) have limited their investigation to schizophrenia, our cohort includes other types of FEP such as patients with bipolar disorder and major depression disorder which are known to be less cognitively impaired than first-episode schizophrenia patients (Barch, 2009, Hill et al., 2009). It should be noted also that Pietrzak et al. (2009a) did not use all the CSRB tasks available in the software (one task per cognitive domain except for working memory was used) and this may have influenced patient performance differently (Pietrzak et al., 2009a).

Limitations

This study has some limitations. First, the administration of the two types of batteries could not be randomized or counter-balanced since this is a historical cohort study. Second, there were significant differences between the two patient groups in symptoms at the time of testing. These differences could be partly explained by the 3 week difference in the administration of the battery between the two groups even though this difference was not significant, or, as stated earlier, an artifact of the available chlorpromazine equivalents conversion equations. As noted earlier, the symptom severity was not assessed on the day of cognitive testing but within 1 month of testing, which may not always reflect precisely the symptom severity on the day of testing. Finally, the number of patients in each group was uneven and this may have affected the magnitude of statistical effects calculated.

Role of funding source

The Canadian Institute of Health Research and the Sackler Foundation had no role in study desing, data collection, analysis, or interpretation; in writing of the report or the decision to submit the paper for publication.

Contributors

PEPP-Montreal research staff and clinicians.

Conflict of interest

Dr Ridha Joober sits on the advisory boards and speakers’ bureaus of Pfizer Canada, Janssen Ortho Canada, BMS and Sunovian, Myelin Canada, Otsuka Canada and Perdue Pharmaceuticals; he has received grant funding from AstraZeneca and Lundbeck Canada as well as honoraria from Janssen Ortho Canada, Shire and from Pfizer Canada for CME presentations and royalties from Henry Stewart talks. Dr Ashok K. Malla is supported by the Canada Research Chair and has received grant and research support from Pfizer, Janssen-Ortho, AstraZeneca, and Bristol-Myers-Squibb. Dr Martin Lepage has received honoraria from Janssen-Ortho & Lilly, is on the advisory board for Roche Canada and has a research grant from Otsuka. All other authors declare that they have no potential conflict of interest.