Dispersion of cognitive performance test scores on the MATRICS Consensus Cognitive Battery: A different perspective.

Objective: Persons with schizophrenia exhibit greater neurocognitive test score dispersion. Here, we seek to characterize dispersion on the Neurocognitive Composite subtests of the Measurement of Treatment Research to Improve Cognition in Schizophrena Consensus Cognitive Battery (MCCB) and determine the relative effects of different antipsychotic formulations on dispersion and mean performance. Method: In this post hoc analysis of the DREaM study (NCT02431702), which compared treatment with paliperidone palmitate (PP) long-acting injectable with oral antipsychotic (OAP) treatment over 18 months, dispersion in MCCB neurocognitive subtest performance was calculated for each participant by visit (test occasion).
Results: Over 18 months, mean neurocognitive performance improved in a manner consistent with the expected effects of practice in both groups (p < 0.05); this improvement was observed during the first 9 months (PP: p < 0.05, OAP: p < 0.001), followed by stable performance over the second 9 months (PP: p = 0.821, OAP: p = 0.375). Rates of change did not differ between groups (treatment-by-visit interaction: p = 0.548). In contrast, analyses of dispersion focusing on contrasts between baselines and end points of the first and second 9 months revealed different patterns. Over the first 9 months, dispersion in both groups lessened to a similar extent. However, over the second 9 months, dispersion remained stable in the PP group, whereas neurocognitive performance became significantly more variable in the OAP group (p < 0.01).
Conclusion: Dispersion of neurocognitive test scores provides a different index of cognitive change than that provided by composite scores. Long-term maintenance of therapeutic levels provided by PP over time may limit (relative to oral AP) the extent to which cognitive performance becomes more variable.

Introduction

Cognitive impairment is a well-established feature of schizophrenia, and considerable research has focused on approaches to measuring it. The methodology used to assess cognition can have implications for understanding patterns of cognitive strengths and weaknesses, the way these patterns evolve over time, and the extent to which these patterns vary as a function of patient or treatment characteristics. Consequently, the National Institute of Mental Health established the Measurement of Treatment Research to Improve Cognition in Schizophrenia (MATRICS) initiative (Green and Nuechterlein, 2004). Through a collaboration between industry, academia, the National Institute of Mental Health, and the US Food and Drug Administration, the MATRICS Consensus Cognitive Battery (MCCB) was developed by selecting 10 tests that assess 7 cognitive domains and can be combined to produce a single composite end point (Nuechterlein et al., 2008). Over time, these measures have become widely accepted and the MCCB (or batteries based upon it) have evolved into the standard means for assessing cognition in schizophrenia clinical trials (Georgiades et al., 2017).

The 10 tests of the MCCB comprise 7 cognitive domains with 1 test that is used to assess social cognition. The 9 neurocognitive tests, which do not include social cognition, can be combined to produce a Neurocognitive Composite score (Georgiades et al., 2017; Harvey et al., 2020; Nuechterlein and Green, 2016). Although individual domains of the MCCB and similar batteries have been examined in several research studies (Georgiades et al., 2017), the MCCB Overall Composite score and Neurocognitive Composite score tend to be the measures of primary interest in clinical trials. Most commonly, results are evaluated in terms of changes in mean composite performance over some period of time, often before and after an intervention (Keefe et al., 2011).

One of the limitations of using a composite score as the unit of measurement that is compared across treatment sessions is that this approach does not account for changes in the variability of performances within a single testing session. The study of intra-individual variability (IIV) typically takes one of two forms: (1) inconsistency, or variability among responses within a reaction-time task, or (2) dispersion, or variability among test performances (relative to applicable norms) within a single neurocognitive battery (Cole et al., 2011; Hill et al., 2013; Merritt et al., 2021; Vance et al., 2021).

Variability of intra-individual performance is clinically relevant because it has been linked to changes in neural integrity associated with age, pathology, and developmental differences (Halliday et al., 2019; MacDonald et al., 2006; Wallace et al., 2019). In some cases, increased variability in performance precedes obvious declines in average levels of performance, making it a potentially useful marker of early stages of cognitive decline (Halliday et al., 2019; MacDonald et al., 2006; Reichenberg et al., 2006b).

The dispersion of performance within a neurocognitive battery has proven to be a valuable metric for a variety of therapeutic states. Greater dispersion within single-session test batteries relative to matched controls has been demonstrated in individuals with bipolar disorder (Depp et al., 2012), HIV infection (Hines et al., 2016; Vance et al., 2021), hepatitis C virus infection (Morgan et al., 2012), West Nile virus infection (Sheppard et al., 2018), veterans with mild traumatic brain injury (Merritt et al., 2018; Sorg et al., 2020), athletes with loss of consciousness secondary to concussion (Merritt et al., 2021), increased risk of progression to Alzheimer's disease (Halliday et al., 2018), and greater severity of traumatic brain injury (Hill et al., 2013). In many of these investigations, the sensitivity of dispersion to deficits appears to be independent of mean level of cognitive performance (Cole et al., 2011; Hines et al., 2016; MacDonald et al., 2006; Reichenberg et al., 2006b).

Dispersion has also been examined in patients with schizophrenia (Ahn et al., 2019; Cole et al., 2011; Reichenberg et al., 2006a). Reichenberg and colleagues (Reichenberg et al., 2006a) obtained the results of IQ testing from evaluations of 53,731 unselected 17-year-old Israeli participants without history of psychotic symptoms who were evaluated prior to entering military service. Using a 4-subtest IQ measure, this group found higher levels of dispersion were linked to increased risk for future hospitalization for schizophrenia. Cole et al. (2011) found that higher levels of score dispersion were observed in patients with schizophrenia and siblings compared to matched controls when evaluated using a comprehensive neurocognitive battery assessing the factors of verbal memory, working memory (n-back), visual memory, processing speed, card sorting, and verbal span. These differences remained significant after controlling for the overall levels of performance, as reflected in the respective composite scores. Finally, in a single-arm study of 38 patients with schizophrenia, Ahn et al. (2019) identified a relationship between dispersion and white matter integrity, as reflected in fractional anisotropy values in the genu of the corpus callosum. Because this was a single-arm study, it did not identify the degree to which this relationship is specific to schizophrenia; however, the link between dispersion and white matter integrity has also been suggested by other groups (MacDonald et al., 2006).

Despite the wealth of findings suggesting the potential utility of dispersion in characterizing neurocognitive performance, and the ubiquity of the MCCB as a measure of cognition in studies of patients with schizophrenia, there are no published investigations of dispersion of subtest performance within the MCCB. In this post hoc analysis of a clinical trial dataset, we sought to determine whether within-administration subtest score dispersion on the MCCB provides additional information beyond that provided by the MCCB Neurocognitive Composite score.

Methods

This is a post hoc analysis of the Disease Recovery Evaluation and Modification (DREaM) trial, a prospective, delayed-start, matched-control, double-randomized, open-label, flexible-dose multicenter study (NCT02431702) comparing treatment with paliperidone palmitate (PP) long-acting injectable with oral antipsychotic (OAP) treatment in participants with recent-onset schizophrenia or schizoaffective disorder (Alphs et al., 2022). DREaM had three parts (see Fig. 1): a 2-month oral run-in (Part I), a 9-month disease progression phase (Part II: PP or OAP), and a 9-month disease modification phase (Part III: PP/PP; OAP rerandomized: OAP/OAP or OAP/PP). The focus of DREAM was to compare those participants taking PP or OAP for the duration of the 18-month extended disease progression analysis. The focus of this post hoc analysis was to examine the extent to which intrasession dispersion of neurocognitive performance on the MCCB changed relative to changes observed in mean MCCB performance at the conclusion of predefined experimental periods.

Fig. 1

DREaM study design. Abbreviations: ER, extended release; EDP, extended disease progression phase; OAP, oral antipsychotic; PP, paliperidone palmitate; PP1M, paliperidone palmitate once-monthly; PP3M, paliperidone palmitate once-every-3-months.

Participants

DREaM participants were enrolled at 34 sites in the United States, Brazil, and Mexico. Participants were men and women aged 18–35 years who met DSM-5 criteria (American Psychiatric Association, 2013) for schizophrenia or schizophreniform disorder, confirmed by the Structural Clinical Interview for DSM-5 disorders, and were within 24 months of their first psychotic episode. Full details on the study population are reported in the primary study publication (Alphs et al., 2022). Participants were assessed using the MCCB during screening and at Day −29 (Part I), Day 0 (baseline of Part II, randomization day 1), Day 260 after randomization of Part II/beginning of Part III, and Day 520 after randomization of Part III (Fig. 1). The DREaM study protocol and amendments were reviewed by an independent ethics committee/institutional review board. The study was conducted in accordance with the ethical principles that have their origin in the Declaration of Helsinki and are consistent with Good Clinical Practices and applicable regulatory requirements (Alphs et al., 2022).

MATRICS Consensus Cognitive Battery

The MCCB was administered by a qualified rater on protocol-specified days: Day −29 (Part I), Day 1 (day of randomization, Part II BL), Days 92 and 176 after randomization (Part II), Day 260 after randomization (end of Part II/BL of Part III), Days 352 and 436 after randomization (Part III), and Day 520 after randomization (end of Part III). When possible, the MCCB was administered at the same time of day (±1 h) and by the same person at each assessment. All MCCB data were reviewed and scored by central raters who were blinded to treatment information. Central raters also assessed neurocognitive data quality and corrected any errors. This extensive blinded processing was performed to ensure that issues with data and reporting were minimized and analyses were reflective of true cognitive performance. Central raters entered the data for each participant into the MCCB Computer Scoring Program to generate T-scores for individual tests and for composite scores. Per the default settings used in the MCCB scoring software, T-scores were corrected for age and reported gender. Using the nine MCCB neurocognitive test scores (i.e., excluding the Managing Emotions branch of the Mayer-Salovey-Caruso Emotional Intelligence Test), a commonly used dispersion metric of intra-individual standard deviation relative to mean individual performance within a single testing occasion (Hill et al., 2013; Vance et al., 2021) was derived for each participant.

Data analyses

All data analyses were performed with SAS® 9.4 Companion for Windows, Fifth Edition: SAS Institute Inc., Cary, NC, USA. Timepoints for analysis were keyed to transition points in the experimental design (Fig. 1): (1) the beginning of randomization (Part II baseline, randomization Day 1), (2) the end of Part II (Part II end point/Part III baseline, Day 260 after randomization), and (3) the end of Part III (Day 520 after randomization). The MCCB Neurocognitive Composite score and dispersion of performance across the 9 MCCB neurocognitive subtests were analyzed with a repeated measure mixed-effects model, with treatment and visit as fixed effects and a treatment-by-visit interaction. This analysis was carried out once using unstructured covariance matrix to account for within-subject correlation. The visit effect is a measure of deviation from the null hypothesis of constancy of the score over time for all treatment groups combined after aggregating subjects. The treatment-by-visit interaction tests the null hypothesis of parallel response profiles over time by treatment groups. A significant treatment-by-time interaction indicates that changes in response over time differ between treatments; in other words, the magnitude of the difference between treatments varies over time. The correlation of the repeated measures within participants was modeled with an unstructured covariance matrix. Because primary analyses were not powered to detect differences in dispersion, this post hoc analysis examined contrasts to identify potential differential temporal patterns within groups. Using this model, least squares means and standard errors were computed at each visit.

Results

The numbers of participants with MCCB data on the day of randomization (baseline of Part II), Day 260 after randomization (end of Part II), and Day 520 after randomization Day 520 (end of Part III), respectively, were 49, 49, and 39 (PP), and 63, 59, and 44 (OAP). Demographic and psychiatric history characteristics were similar between the PP and OAP groups at Part II baseline (Table 1).

Table 1

Participant demographics and psychiatry history characteristics at Part II baseline.

	PPn = 49	OAPn = 63
Mean age (SD), years	23.7 (4.65)	23.0 (4.30)
Male, n (%)	41 (83.7)	47 (74.6)
Race, n (%)
White	24 (49.0)	23 (36.5)
Black/African American	17 (34.7)	25 (39.7)
Other	8 (16.3)	15 (23.8)
Education, n (%)
High school or less (≤12 years)	29 (59.2)	39 (61.9)
>12 years	20 (40.8)	24 (38.1)
Duration of illness, n (%), months
<6	27 (55.1)	40 (63.5)
6–12	8 (16.3)	7 (11.1)
>12	14 (28.6)	16 (25.4)
Mean CGI-S score (SD)	3.3 (1.09)	3.2 (1.15)
Mean no. of previous psychiatric hospitalizations (SD)	1.1 (1.30)	1.2 (1.12)
Mean MCCB Neurocognitive Composite score (SD)	30.6 (12.14)	30.1 (14.58)

All PP vs. OAP comparisons were P > 0.05.

Abbreviations: CGI-S, Clinical Global Impression-Severity scale; MCCB, MATRICS Consensus Cognitive Battery; OAP, oral antipsychotic; PP, paliperidone palmitate.

Spearman rank correlations revealed a small, inverse relationship between dispersion and the Neurocognitive Composite at baseline (r = −0.163, p < 0.05); however, the indices were essentially independent thereafter (Day 260 after randomization, Part II: r = −0.057, p = 0.461: Day 520 after randomization, Part III: r = −0.015, p = 0.869).

Analyses of cognitive performance using the Neurocognitive Composite T-score revealed that mean performance improved in both groups (p < 0.05) over the 18-month extended disease progression phase (Fig. 2). For both groups, this improvement took the form of significant improvement during Part II (PP: p < 0.05, OAP: p < 0.001), followed by stable performance over Part III (PP: p = 0.821, OAP: p = 0.375). Rates of change in the mean Neurocognitive Composite T-scores did not differ significantly between groups (treatment-by-visit interaction: p = 0.548).

Fig. 2

Within-phase changes in MCCB Neurocognitive Composite score. *P < 0.05 for Day 260 (Part II) vs. Part II baseline. **P < 0.001 for Day 260 (Part II) vs. Part II baseline. Abbreviations: LSM, least squares mean; MCCB, MATRICS Consensus Cognitive Battery; OAP, oral antipsychotic; PP, paliperidone palmitate; SE, standard error.

A similar overall pattern was seen in dispersion over the 18-month extended disease progression analysis; there was numeric improvement over time that fell just short of statistical significance (p = 0.073), but the overall rates of change across time did not differ by group (p = 0.993). However, analyses of dispersion focusing on contrasts between baselines and end points of the first and second 9 months revealed different patterns than seen with mean scores (Fig. 3). Over Part II, dispersion in both groups lessened (indicating more consistent performance) to a similar extent that failed to reach statistical significance for either group. Throughout Part III, in contrast, although dispersion remained stable in the PP group, neurocognitive performance became significantly more variable in the OAP group (p < 0.01).

Fig. 3

Within-phase changes in dispersion of MCCB Neurocognitive Composite subtests. *P < 0.01 for Part 2 Day 260 vs. Part 3 Day 520. Abbreviations: LSM, least squares mean; MCCB, MATRICS Consensus Cognitive Battery; OAP, oral antipsychotic; PP, paliperidone palmitate; SE, standard error.

Discussion

To our knowledge, this is the first report examining test score dispersion within the MCCB. Results suggest that this index may provide information distinct from that provided by mean level of performance on the same tests, a result similar to that seen in previously reported investigations in schizophrenia and other disease states (Hill et al., 2013; Hines et al., 2016; MacDonald et al., 2006; Vance et al., 2021). The Neurocognitive Composite T-scores of the treatment groups improved as would be expected from the effects of practice (Keefe et al., 2017) in Part II but did not change differentially by medication group over time. In contrast, the dispersion of scores suggested that both groups improved modestly during the first 9 months of treatment after oral run-in, but that these gains were maintained in the PP group and lost in the OAP group during the ensuing 9 months, suggesting that, over time, more consistent maintenance of therapeutic antipsychotic levels provided by PP may provide a stronger buffer against increasing variability in cognitive performance than that provided by OAP. This contrast suggests that the IIV index might be sensitive to some drug effect beyond the overall level of cognitive performance. This interpretation is limited, in that the extent to which dispersion changes purely as a function of time and repeated assessments, regardless of treatment, has yet to be investigated.

An intriguing parallel that merits further investigation is the relationship of differences in MCCB dispersion to real-world differences in the presentation of patients with schizophrenia. Cole and colleagues reported a modest relationship between dispersion and clinicians' Global Assessment of Function (Cole et al., 2011). In a sample of 173 individuals at ultra-high risk for psychosis between the ages of 14 and 29 assessed over 24 months, Lam and colleagues noted that “gradual increases in variability of test performance over time suggest the possibility that the underlying cognitive architecture may have devolved in converters (to psychosis) and nonremitters (of psychosis) during follow-up” (Lam et al., 2018). An intriguing observation is the parallel in our dataset to the clinical outcome pattern for these patients reported by Baker et al. (2020): within the OAP group, more inconsistent neurocognitive performance is evident in the second 9 months of treatment, over the same time frame in which hospitalizations and/or arrests became more prevalent in the OAP group relative to the PP group. The extent to which these parallels are related given their post hoc nature and the small samples involved is difficult to gauge. In addition, other potential factors might be expected to impact these outcomes, such as group differences in treatment adherence, and the individual or joint influence of such factors is presently unknown. Given previous findings from other research groups, investigation of such relationships in a larger, more heterogenous sample merits consideration.

In addition, although data suggest that dispersion may, in some cases, serve as an early indicator of subtle cognitive change, the impact of dispersion on function in daily life remains largely unexplored. The extent to which diagnostic group, age, cognitive reserve, task instruction, and other demographic variables may influence observed values, and the extent to which those influences overlap with the impact of the same variables on mean scores, is essentially speculative at this point, though some relevant findings have been published with regard to IIV in reaction time (Garrett et al., 2012).

Potential mechanisms of increased dispersion in neurocognitive performance in clinical groups have been proposed. In studies of elderly subjects and veterans, dispersion increased with reduced microstructural white matter integrity (Halliday et al., 2019; Sorg et al., 2020), and disturbances in white matter microstructure were a feature of schizophrenia early in the disease, including in patients who were not being treated (Kraguljac et al., 2019). Other groups that demonstrated the presence of higher levels of dispersion of neurocognitive performance in schizophrenia-related cohorts have noted that neuronal dysconnectivity and frontal lobe dysfunction may contribute to these findings (Reichenberg et al.).

Finally, the role of more variable levels of task engagement among patients with schizophrenia cannot be excluded. Neurocognitive performance dispersion has been reported to vary according to both severity of injury and consistency of engagement in the cognitive testing procedures in a sample of patients with traumatic brain injury (Hill et al., 2013). The rate at which patients with schizophrenia fail to remain consistently engaged in cognitive testing procedures, whatever the reason, may be substantial (Kreis et al., 2020; Morra et al., 2015; Ruiz et al., 2020). Interpretation of such measures are, at best, more complicated in patients with schizophrenia, due to the common, disease-related impairment in the ability to sustain the levels of motivation or attention required to perform optimally, reactions to the stress of recognizing that one is performing poorly to a series of cognitive tests, the potential social or economic motivations to perform less than optimally, and the finding that “effort test” failure may more often relate to demonstrable “real-world” functional limitations (Whearty et al., 2015). In this investigation, however, even if inconsistent engagement in testing procedures contributes to the results, there is no obvious reason to postulate that it would impact one treatment group more than the other. Likewise, the possibility that consistent use of antipsychotic medication might enable more consistent engagement in neurocognitive testing cannot be excluded, but this has not been studied.

Limitations of this study will merit attention in future investigations. The absolute magnitude of the changes observed in dispersion were relatively small (just over one-tenth of a standard deviation). Although group differences may sometimes be meaningful at smaller magnitudes than the same magnitude of difference at the individual level because of the averaging of multiple outcomes (Montgomery and Möller, 2009), magnitudes of difference in MCCB test dispersion have not yet been linked to functional or pathological correlates. The sample sizes that we examined are relatively small, and our sample is limited to patients diagnosed with schizophrenia or schizophreniform disorder within 2 years of their first psychotic episode. Replication in larger and more heterogenous samples would be informative.

Data sharing statement

The data sharing policy of Janssen Pharmaceutical Companies of Johnson & Johnson is available at https://www.janssen.com/clinical-trials/transparency. As noted on this site, requests for access to the study data can be submitted through Yale Open Data Access (YODA) Project site at http://yoda.yale.edu.

CRediT authorship contribution statement

David J. Williamson: Writing – original draft, Visualization, Supervision, Writing – review & editing. Keith H. Nuechterlein: Conceptualization, Methodology, Writing – review & editing. Todd Tishler: Conceptualization, Methodology, Writing – review & editing. Joseph Ventura: Conceptualization, Methodology, Writing – review & editing. Benjamin M. Ellingson: Conceptualization, Methodology, Writing – review & editing. Ibrahim Turkoz: Formal analysis, Data curation, Writing – review & editing. Richard S.E. Keefe: Conceptualization, Methodology, Writing – review & editing. Larry Alphs: Conceptualization, Methodology, Writing – review & editing.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

IT and AO are employees of Janssen Pharmaceuticals and hold stock in Johnson & Johnson. DW and LA are former employees of Janssen Pharmaceuticals and holds stock in Johnson & Johnson. KN has received research grant support from Janssen Scientific Affairs, LLC, and has served as a consultant to Astellas, Genentech, Janssen, Medincell, Otsuka, ReCognify, Takeda, and Teva. TT has no conflicts of interest to disclose. JV has received research grants from Posit Science and Genentech, Inc., and has been a consultant to Boehringer-Ingelheim and Posit Science, Inc. BE has received research grant support from Siemens, Janssen, VBL, the National Brain Tumor Society, and the American Cancer Society; has served as a consultant to the Image Analysis Group (IAG), Oncoceutics, Inc., BeiGene, Tocagen, and the Global Coalition for Adaptive Research (GCAR); and has also attended an advisory board or served as a paid consultant to Agios Pharmaceuticals, Imaging Endpoints, Kazia/Novogen, Medicenna, MedQIA, LLC, NeoSoma, the NIH/NCI Cancer Imaging Steering Committee, NW Pharmaceuticals, and Siemens. RK owns VeraSci, a for-profit company that provides comprehensive services to >100 business entities, most of which are pharmaceutical companies, including Janssen Pharmaceuticals.