A preliminary validity study of the cambridge neuropsychological test automated battery for the assessment of executive function in schizophrenia and bipolar disorder.

OBJECTIVE: Although the executive function subtests of the Cambridge Neuropsychological Test Automated Battery (CANTAB) have been used to assess cognitive function in diverse psychiatric illnesses, few studies have verified the validity of this battery for Korean psychiatric patients. Therefore, this preliminary study evaluated the construct and concurrent validity of the executive function subtests of the CANTAB for Korean psychiatric patients by comparing it with subtests of the Computerized Neuropsychological Test (CNT).
METHODS: Three subtests of the CANTAB and three subtests of the CNT were administered to 36 patients diagnosed with either schizophrenia or bipolar disorder. Subtests of the CANTAB included the Intra/Extra-Dimensional Set Shift (IED), Stockings of Cambridge (SOC), and Spatial Working Memory (SWM). Differences between groups on each subtest as well as correlations between the subtests of the CANTAB and the CNT were assessed.
RESULTS: The schizophrenia group performed significantly more poorly on the IED and the Wisconsin Card Sorting Test (WCST) compared with the bipolar disorder group. Additionally, correlation analyses revealed a significant correlation between the IED and the WCST; a positive correlation between the SOC and the Trail Making Test, Part B and the Stroop test; and a significant correlation between the SWM and the Stroop test.
CONCLUSION: This study verified the construct and concurrent validity of the executive function subtests of the CANTAB for Korean psychiatric patients and suggests that the subtests of this battery would be useful and appropriate for assessing deficits in executive function in Korean clinical settings.

INTRODUCTION

Deficits in executive function are known to be a salient characteristic of patients with schizophrenia.1 Previous studies have consistently found that patients with schizophrenia perform very poorly on tasks that measure executive abilities mediated by frontal lobe function.2,3 Generally, executive functions are defined as the higher-level cognitive functions that are necessary to plan and execute goal-directed behaviors4,5 and may include cognitive flexibility, creativity, planning ability, abstract thinking, concept formation, and inhibition.6 It has recently been shown that those with schizophrenia, as well as those with bipolar disorder or anxiety disorders, exhibit deficits in executive function relative to controls.7,8 Executive function capability is an important predictor of the treatment, prognosis, and functional outcomes of these disorders.9

The Wisconsin Card Sorting Test (WCST), the Tower of Hanoi test, the Stroop test, the Trail Making Test, the Controlled Oral Word Association test, and the verbal fluency test are widely used as measures of executive function.3 Recently, efforts have been made to appraise neurocognitive deficits more efficiently and precisely by adapting traditional neuropsychological tests into computerized assessments that more accurately measure reaction times or use complex automated scoring systems.10,11 The Computerized Neuropsychological Test (CNT), a computer-based standardized assessment used in clinical settings in South Korea, includes the WCST, the Stroop test, and the Trail Making test, all of which are measures that evaluate executive function.12 The CNT may be usefully employed to measure sub-elements, such as those involved in planning, initiation, and sequencing a behavior, cognitive flexibility, and response inhibition, that support executive function. Computerized tests such as the CNT are advantageous as they efficiently and precisely enable evaluation of diverse neurocognitive functions under conditions that minimize environmental impact.13

In countries other than South Korea, the Cambridge Neuropsychological Test Automated Battery (CANTAB) is actively utilized in experimental and clinical settings to assess cognitive dysfunction in various psychiatric disorders.14,15 The CANTAB was developed in 1986 at the University of Cambridge by Barbara Sahakian, Trevor Robbins, and colleagues and has been used as a tool for cognitively assessing subjects with attention deficit-hyperactivity disorder, dementia, schizophrenia, and mood disorders.16,17,18 More specifically, the CANTAB includes a variety of subtests, such as the Intra/Extra-dimensional Set Shift test (IED), Stop Signal Test (SST), Stockings Of Cambridge (SOC) test, Spatial Working Memory (SWM) test, and Cambridge Gambling Task (CGT), that evaluate executive function and measure decision-making ability, response inhibition, and working memory.11 Patients with schizophrenia exhibit severe deficits in attentional set-shifting2,19 and perform more poorly on the SST test, which is similar to the go/no-go paradigm, relative to healthy controls,20,21 demonstrating that these patients have deficits in response inhibition. This type of executive dysfunction in patients with schizophrenia is persistent, difficult to treat, and linked to impairments in functional outcome.22

The CANTAB has been used to assess cognitive function in several types of diverse psychiatric illnesses during the past several decades. However, validity studies for this battery in Korean psychiatric patients are lacking, and no standardization of the battery has been developed for Korean patients. Therefore, the purpose of this study was to examine the use of the executive function subtests of the CANTAB with Korean psychiatric patients. This is a preliminary study examining the construct and concurrent validity of the CANTAB by comparing its subtests with those of the CNT.

METHODS

Participants

This study included 36 subjects aged between 18 and 50 years who were either hospitalized or receiving outpatient care; 20 patients were diagnosed with schizophrenia and 16 with bipolar disorder type I or II according to the criteria in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV).23 The schizophrenia group consisted of 13 male patients and seven female patients (mean age: 26.1 years, SD=8.2), 14 of whom had paranoid schizophrenia and six of whom exhibited an undifferentiated type of schizophrenia. During the study period, all patients with schizophrenia received daily treatment with the following atypical antipsychotic medications: clozapine (206 mg; n=11), quetiapine (700 mg; n=1), risperidone (1.75 mg; n=2), olanzapine (10 mg; n=4), aripiprazole (15 mg; n=2), and/or paliperidone (9 mg; n=1). The bipolar group consisted of 15 patients with bipolar type I and one with bipolar type II disorder (mean age: 21.6 years, SD=3.8). All patients with bipolar disorder received medication throughout the study; four patients received atypical antipsychotic medication, one received a mood stabilizer, and 11 received an atypical antipsychotic drug along with a mood stabilizer. At the time of assessment, 14 patients were in the manic state and two were in the depressive state. The following individuals were recruited and then excluded: patients with substance dependence (e.g., alcohol) and those with neurological or medical conditions that can cause cognitive damage, including head injuries. This study was approved by the Institutional Review Board of Seoul National University College of Medicine (No. H-1207-039-417).

Measures

Cambridge Neuropsychological Test Automated Battery

In this study, three CANTAB subtests were administered for a total of approximately 25 min to assess the executive function of the recruited patients. The IED test assesses rule acquisition and attentional set-shifting ability using a two-dimensional space in which a colored shape and a white line appear as two types of patterns. Successfully passing each step requires correct and consecutive prediction of the defined rules, and the test ends when the subject fails to predict the rules on 50 attempts during any step. The SOC test examines spatial planning and motor control and is similar to the Tower of London test. Subjects viewed an upper and lower screen with three balls of different colors that appeared as though they were inside a hanging bag. Subjects were required to touch the lower screen to move the balls in the bag so that they matched the pattern on the upper screen. The SWM test assesses visuospatial memory and strategy use. Subjects were required to find a blue square in a colored square box appearing on the screen and to move it to an empty pillar on the right. They memorized the identified box while the other boxes were touched until the empty pillar was full.

Computerized Neuropsychological Test

Of the many cognitive subtests of the CNT, three that assess executive function were administered to the recruited subjects. A standardization study of CNT subtests revealed the following significant test-retest reliability scores: 0.543-0.862 for the WCST, 0.609-0.905 for the Word-Color test, and 0.531 for the Trail Making Test (TMT-B) test.12,24 Total time required to complete the three subtests was approximately 25 min. The Word-Color Test, also known as the Stroop test, assesses selective attention and response inhibition. It measures response time when reading words and requires saying the color of the ink in which each word appears when the word and the color in which the word is written either match or do not match. The WCST assesses problem-solving skills, abstraction, and the ability to shift strategies efficiently in response to environmental feedback. Subjects are required to discern the appropriate criterion for sorting (i.e., color, form, or number). If the criterion is correctly identified, "right" is displayed. A new criterion is introduced after subjects consecutively provide a given number of correct answers and the number of responses required to identify each criterion is recorded. The TMT-B assesses abilities related to selective attention, sequencing, mental flexibility, visual search, and motor function. Subjects were required to touch the monitor to connect numbers (1-25) and letters in the Korean alphabet in alternating order.

Clinical assessments

We used the Positive and Negative Syndrome Scale (PA-NSS)25 to assess the severity of psychotic symptoms in the schizophrenia group. This scale includes 30 questions under categories such as positive symptoms, negative symptoms, and general psychopathology. The Young Mania Rating Scale (YMRS)26 was used to assess the severity of the manic symptoms in the bipolar disorder group. Additionally, two self-report questionnaires, the Beck Depression Index (BDI)27 and Beck Anxiety Index (BAI),28 were also administered to all subjects to assess the severity of depressive and anxious symptoms, respectively.

Statistical analysis

Data were analyzed using SPSS 12.0 for Windows (SPSS Inc., Chicago, IL, USA). Student's t-test was used to identify significant differences between the schizophrenia and bipolar disorder groups to examine the construct validity of the CANTAB. Additionally, an analysis of covariance (ANCOVA), controlling for duration of illness as a covariate, was performed to identify group differences. The concurrent validity of the CANTAB was assessed using a Pearson's correlation between the subtests of the CANTAB and CNT, and the relationship between symptoms and subtasks was examined using Pearson's correlation coefficients.

RESULTS

Demographic and clinical characteristics

We found no differences in age or educational level between groups, but the schizophrenia group contained significantly more men than did the bipolar group (Table 1). The duration of illness was significantly shorter in the bipolar compared with the schizophrenia group, but no significant differences in IQ, the BDI, or the BAI were observed. The mean PANSS total score was 59.6 (SD=12.7) in the schizophrenia group and the mean YMRS score was 14.4 (SD=7.9) in the bipolar group.

Table 1

Comparison of general characteristics of patients with schizophrenia and bipolar disorder

*p<0.05. IQ: intelligence quotient, PANSS: Positive and Negative Syndrome Scale, YMRS: Young Mania Rating Scale, BDI: Beck Depression Index, BAI: Beck Anxiety Index

Comparisons between groups on the CANTAB and CNT

The results of the CANTAB and CNT executive function subtests are presented in Table 2 for patients with schizophrenia and bipolar disorder. We found no significant differences between groups on the SOC and SWM of the CANTAB. However, patients with schizophrenia performed significantly more poorly over the total trials (t=2.62, p<0.01) and made more total errors (t=1.88, p=0.069) on the IED compared with the bipolar group. No significant differences between the groups were observed on the Stroop test or the TMT-B test of the CNT. However, patients with schizophrenia performed significantly more poorly with regard to completed categories (t=-2.88, p<0.01) and total errors (t=2.51, p<0.05) on the WCST relative to patients with bipolar disorder. Patients with schizophrenia also tended to do worse in terms of total trials (t=1.90, p=0.065). The results of the IED and WCST tests were similar to those of the CANTAB and CNT executive function subtests (Figure 1) such that the schizophrenia group perform-ed significantly more poorly on the IED and WCST subtests compared with the bipolar group. In terms of the other executive function subtests, the performance of the schizophrenia group was similar to that of the bipolar disorder group. However, even after controlling for duration of illness, the schizophrenia group still performed more poorly on the total trials of the IED (F=5.34, p=0.028) and the completed categories of the WCST (F=5.31, p=0.028) compared with the bipolar group.

Table 2

Comparison of the CANTAB and CNT scores between patients with schizophrenia and bipolar disorder

*p<0.05, **p<0.01, †this variable specifies the mean number of moves used to solve problems with solutions that can be achieved in a minimum of four moves, ‡this variable specifies the number of times the subject revisited a box in which a token had previously been found. This this value was calculated for all trials with four or more tokens, §this variable specifies the minimum and maximum numbers of boxes for which we calculated the SWM strategy using the Boxes (lower bound) and Boxes (upper bound) options. SOC: Stockings of Cambridge, SWM: Spatial Working Memory, IED: Intra/Extra Dimensional Shift, TMT-B: Trail Making Test-B, WCST: Wisconsin Card Sorting Test,. CANTAB: Cambridge Neuropsychological Test Automated Battery, CNT: Computerized Neuropsychological Test

Figure 1

Performance of schizophrenic and bipolar disorder patients on executive function subtests using Student t-tests. *p<0.05, **p<0.01. SOC: Stockings of Cambridge, SWM: Spatial Working Memory, IED: Intra/Extra Dimensional Shift, Stroop C-W: Stroop Color-Word, TMT-B: Trail Making Test-B, WCST: Wisconsin Card Sorting Test, CANTAB: Cambridge Neuropsychological Test Automated Battery, CNT: Computerized Neuropsychological Test.

Concurrent validity of the executive function subtests of CANTAB

Correlations between the CANTAB and CNT subtests were calculated for all 36 subjects to determine concurrent validity (Table 3). The analysis revealed significant positive correlations between the SOC subtest of the CANTAB and the Stroop test of the CNT such that problems performing the SOC were significantly correlated with color-word errors on the Stroop test (r=0.35, p<0.05). Moreover, total errors and between errors on the SWM of the CANTAB were significantly correlated with color-word errors on the Stroop test (p<0.05), and total trials and total errors on the IED was significantly positively correlated with total errors on the WCST (p<0.05) and strongly correlated with total trials and categories on the WCST (p<0.01).

Table 3

Correlation between subtests of the CANTAB and the CNT in patient groups (N=36)

*p<0.05, **p<0.01, †this variable specifies the number of occasions on which the subject successfully completed a test problem using the lowest possible number of moves. TMT-B: Trail Making Test-B, CANTAB: Cambridge Neuropsychological Test Automated Battery, CNT: Computerized Neuropsychological Test

Relationships between the CANTAB and clinical symptoms

We found significant relationships between the CANTAB subtests and various clinical variables. In the schizophrenia group, significant correlations between PANSS negative scores and completed stages (r=-0.474, p=0.04) and total errors (r=0.565, p=0.012) on the IED were observed. In the bipolar group, a significant correlation was observed between YMRS scores and total errors (r=0.679, p=0.004), between errors (r=0.665, p=0.005), and strategy (r=0.594, p=0.015) on the SWM. Additionally, we observed significant correlations between the BAI and SOC mean 4 moves (r=0.345, p=0.039) and mean 5 moves (r=0.456, p=0.005) for all patients. No other significant correlations between the CNT subtests and clinical symptoms were identified.

DISCUSSION

The IED, SOC, and SWM subtests of the CANTAB are sensitive to frontal lobe dysfunction and are reliable and valid instruments for measuring executive function.29 Nevertheless, there is a relative lack of studies evaluating the validity of the executive function subtests of the CANTAB in Korean psychiatric patients. Thus, the current study investigated the construct and concurrent validity of these subtests by analyzing the differences and relationships between performance on the CANTAB and CNT in Korean patients with schizophrenia or bipolar disorder.

In the current study, patients with schizophrenia performed significantly more poorly on the IED of the CANTAB and the WCST of the CNT compared with patients with bipolar disorder. Several previous studies have compared executive function in schizophrenia and bipolar disorder.30,32,33,34,35 A previous study using the WCST found that, although patients with bipolar disorder have mild deficits in executive function, patients with schizophrenia are qualitatively more impaired.30 A componential analysis of WCST performance revealed that, among the components of executive function, patients with schizophrenia showed specific deficits in set-shifting ability.19 Similarly, patients with schizophrenia made more errors in the extra-dimensional shift stage of the IED, reflecting impairments in set-shifting ability.31 These findings demonstrate that the deficits in executive function, including set-shifting ability, found in patients with schizophrenia are measured comparably by the IED and the WCST. Additionally, the IED subtest was significantly correlated with negative symptoms in patients with schizophrenia. Previous studies have found that the deficits in cognitive functioning seen in patients with schizophrenia are most commonly related to negative symptoms.32 Indeed, Tam and Liu33 have suggested that the negative symptoms of schizophrenia are more highly correlated with poor executive functioning. This is consistent with the present finding of a large effect size between the performance of schizophrenic patients with negative symptoms and that of patients with bipolar disorder.

In current study, we found no significant differences between groups on the Stroop test and TMT-B of the CNT. How-ever, a comparison with the percentiles for CNT normative data on the Stroop test revealed that both groups exhibited impairment below the 0.5 percentile (mean=10.1, SD=1.6). This is consistent with findings from previous studies showing that patients with schizophrenia or bipolar disorder demonstrate significant deficits in the interference task of the Stroop test relative to controls.34,35 Moreover, patients with schizophrenia or bipolar disorder with severe psychotic features have scores below two SDs of the normative data on the Stroop Test.36

We found no significant differences concerning SWM strategy between groups; however, both groups performed below the average level reflected in the normative data (mean=13.2, SD=5.2). It has been consistently reported that patients with schizophrenia have deficits in spatial working memory,37 and that these deficits also exist in their relatives.38 Badcock et al.39 found that patients with schizophrenia or bipolar disorder display deficits on spatial span tasks compared with he-althy controls. Here, significant correlations between SWM subscores and the severity of manic symptoms were observed in the bipolar group. Recent studies have suggested that the severity and progression of symptoms in bipolar disorder are connected with the persistence of cognitive impairment.40 This appears to be particularly true for cognitive deficits in working memory, spatial attention, and problem solving and seem to be more pronounced in patients experiencing manic or mixed states.41

In the present study, correlation analyses revealed a significant correlation between the IED of the CANTAB and the WCST of the CNT in all patients. Since its introduction, the WCST has been used to assess prefrontal lobe dysfunction in patients with brain lesions.42 Prior neuroimaging studies using the WCST have reported a significant increase in neural activity within the frontal lobe or prefrontal cortical regions when subjects are engaged in the task.43 The IED is the computerized analog of the WCST; thus, the extra-dimensional shift stage of the IED can be considered equivalent to the category shift of the WCST.44 Although similarities exist, test-retest reliability results indicate that the coefficient was slightly higher for the IED (r=0.75) than for the WCST (r=0.65).45 Thus, the IED possesses high validity and is an appropriate alternative test for evaluations of executive function.

In the present study, the SOC of the CANTAB exhibited positive correlations with the TMT-B and the Stroop test of the CNT. Although cognitive planning ability is important when performing the SOC, complex tasks such as this also require the operation and integration of multiple cognitive domains, including visuospatial working memory and attention.3 More specifically, subjects must mentally plan their sequence of movements prior initiating them to perform well on the SOC test. Although sufficient directions are given before the test, starting to solve the problem without appropriate forethought can hinder planning ability.46 Thus, the present findings indicate that motor function, general inhibitory mechanisms, and attention are reflected in the SOC.

This study found that the SWM test of the CANTAB, which assesses visuospatial working memory and executive function, was significantly correlated with the Stroop test. Specifically, visuospatial working memory was closely linked to executive processes.47 Previous studies have found that com-ponents of working memory are goal maintenance, activation of goals and maintenance of information that is needed to control attention and response selection.48,49 The goal maintenance ability allows one to choose task-relevant information and prevent interference. The Stroop task also requires that working memory be used to maintain focus on a crucial goal. Kane & Engle (2003)50 reported that individual differences in working memory capacity predicted performance on the Stroop task, which also requires goal maintenance. Therefore, this correlation suggests that the deficits in spatial working memory found in patient groups are related to impaired selective attention and attentional control.

The current study has several limitations. First, a normal comparison group was not recruited as the control group. However, the two patient groups in the present study may be compared with the normal mean values from previous standardization studies of the CANTAB and CNT. Another limitation was that the Structured Clinical Interview for DSM-IV (SCID) interview tool was not used. However, the patient groups in the present study were recruited as subjects only if they were diagnosed with DSM-IV Axis-I diagnoses by psychiatrists. Third, although the sample size of patients was modest, statistical analyses revealed significant findings. Finally, the Stroop test and the TMT-B of the CNT cannot be considered the gold standards for the SOC and SWM of the CANTAB. However, we selected the Stroop test and TMT-B, computer-based standardized tests whose reliability and validity have already been examined, because they both measure executive function. Additional research with large samples will be needed for standardization and validation.

In conclusion, the current study demonstrated the construct and concurrent validity of the executive function subtests of the CANTAB for Korean patients with schizophrenia or bipolar disorder. Because computerized assessments such as the CANTAB have a high degree of efficiency, these findings are expected to contribute to clinical studies and the assessment of various neurocognitive functions in clinical settings in South Korea. Further standardization studies with large Korean population-based samples of various ages and educational levels will be needed for clinical application.