Model selection and prediction of outcomes in recent onset schizophrenia patients who undergo cognitive training.

Predicting treatment outcomes in psychiatric populations remains a challenge, but is increasingly important in the pursuit of personalized medicine. Patients with schizophrenia have deficits in cognition, and targeted cognitive training (TCT) of auditory processing and working memory has been shown to improve some of these impairments; but little is known about the baseline patient characteristics predictive of cognitive improvement. Here we use a model selection and regression approach called least absolute shrinkage and selection operator (LASSO) to examine predictors of cognitive improvement in response to TCT for patients with recent onset schizophrenia. Forty-three individuals with recent onset schizophrenia randomized to undergo TCT were assessed at baseline on measures of cognition, symptoms, functioning, illness duration, and demographic variables. We carried out 10-fold cross-validation of LASSO for model selection and regression. We followed up on these results using linear models for statistical inference. No individual variable was found to correlate with improvement in global cognition using a Pearson correlation approach, and a linear model including all variables was also found not to be significant. However, the LASSO model identified baseline global cognition, education, and gender in a model predictive of improvement on global cognition following TCT. These findings offer guidelines for personalized approaches to cognitive training for patients with schizophrenia.

RCT Entities:   Population Interventions Outcomes

Introduction

Targeted cognitive training (TCT) of auditory processing and verbal working memory for schizophrenia has shown efficacy for improving cognition (Fisher et al., 2014, Fisher et al., 2009), but little is understood about which patient factors predict positive outcomes. Previous meta-analytic work suggests that symptoms may be predictive of response to cognitive remediation more broadly (Wykes et al., 2011), though other findings suggest that baseline cognition and other treatment factors may also predict response (Kurtz et al., 2009, Fiszdon et al., 2005; Joanna M Fiszdon et al., 2006a, Fiszdon et al., 2006b, Vita et al., 2013, Lindenmayer et al., 2017). In TCT, baseline reward anticipation was associated with improvements in verbal memory as well as global cognition (Fisher et al., 2014), and improvement in auditory processing speed is also a predictor of subsequent cognitive improvements (Biagianti et al., 2016). However, the pre-treatment cognitive and demographic profile of individuals who may benefit from this intervention remains an open question. Efforts to predict treatment outcomes in psychiatric populations including schizophrenia remains a critical goal, especially as psychiatry as a field continues to pursue personalized interventions (McGorry, 2013). However, developing predictive models for psychiatric treatment and outcome poses numerous challenges, and novel analytic tools will likely be required to solve these problems.

Previous work to identify variables predictive of cognitive training outcomes in schizophrenia has been limited by problems with multiple comparisons. Despite numerous clinical trials collecting a wealth of data on measures including cognition, functioning, symptomology, and demographics, there is often insufficient power to test each variable, and few a priori predictions to guide principled analyses. Moreover, building predictive models using linear regression with this type of data raises concerns of “over-fitting,” where inflated R2 values may adequately characterize variance in a single dataset, but hamper its predictive ability in the general population. These challenges have therefore given way to more advanced and iterative statistical procedures, that allow for the inclusion of large numbers of predictor variables, without being penalized for their inclusion.

Least absolute shrinkage and selection operator (LASSO) is one such regression procedure that allows for testing large numbers of predictor variables (including when the number of predictors is greater than N) while minimizing the model error as well as minimizing the risk of over-fitting (Tibshirani, 1996). LASSO has been previously used to examine questions regarding genetics (Wu et al., 2009), neuroimaging (Shimizu et al., 2015), and clinical outcomes (Bertocci et al., 2016). The current study seeks to use LASSO to identify a model of baseline cognitive, functional, symptom, and demographic factors that may be predictive of response to TCT in recent onset schizophrenia (SZ); such information could guide personalized treatments for this population. Here we re-examine data from 42 SZ patients who underwent 40 h of targeted cognitive training (TCT) of auditory processing and working memory. Participants were examined at baseline on the basis of global cognition, symptoms, functioning, estimated intelligence quotient (IQ), duration of illness, and demographic variables including age, gender, and education attainment. Previous findings in these participants suggested that the treatment elicited improvement on measures of verbal memory, problem solving, and global cognition (Fisher et al., 2014). We hypothesized that LASSO would more adequately identify a model predictive of global cognitive improvement (measured by the MATRICS Consensus Cognitive Battery) in response to TCT, compared to simple correlation or multiple regression models.

Methods

Participants

Participants in the current study included 43 individuals randomized to the active treatment group (TCT) of a study described previously (Fisher et al., 2014) (Clinicaltrials.gov NTC00694889). Briefly, patients with recent onset SZ were recruited from the Early Psychosis Clinics at the University of California, San Francisco and the University of California, Davis. Participants were required to meet the following criteria: (1) Structured Clinical Interview for DSM-IV (SCID) confirmed diagnosis of SZ or schizoaffective disorder; (2) recent onset of psychotic episode within the last 5 years (M = 18.81; SD = 15.45); (3) good general physical health; (4) age between 14 and 30 years old; (5) fluent and proficient in English; (6) IQ of 70 or greater; (7) no known neurological disorder; (8) no substance dependence in the past year. All eligible participants had achieved outpatient status for at least 3 months, and were stable on psychiatric medications. Participants 18 and older provided informed consent, while participants under 18 provided assent as well as legal parent/guardian consent. All consenting and baseline assessment procedures were conducted prior to random assignment to the training condition. One participant was subsequently removed due to missing clinical data. All study procedures were approved by the IRBs at the University of California, San Francisco and University of California, Davis.

Training procedure

Participants were loaned a laptop computer to complete the intervention independently at home. Individuals were asked to participate for 40 h over the course of 8 weeks (1 h/day, 5 days/week). Participants were contacted 1–2 times per week by telephone to check in on progress, in addition to offering coaching/support if there were difficulties completing the training. Check-in appointments were also conducted after every 10 sessions, where participants were paid $5 for each completed session, an additional $20 for 10 completed sessions, and $30 after completing 40 h of training. Participants also received $20 for each pre- or post-training assessment. TCT consisted of adaptive computerized exercises designed to improve the speed and accuracy of early auditory processing while engaging in auditory and verbal working memory tasks (Fisher et al., 2009), and was provided by Posit Science, Inc. Exercises are individually adaptive, and designed to dynamically shift the difficulty level to maintain an 80–85% accuracy rate. During a one-hour session, each participant completed four to six exercises, with their compliance monitored via remote upload of their data. Participants completed on average 32.93 h of training (SD = 10.45), over the course of the 8 weeks.

Assessment procedures

Assessments were conducted blind to group assignment immediately before and after TCT. All assessment staff were trained and monitored by the same senior researcher (M.F.) ensuring cross-site consistency. Assessments included an abbreviated version of the MATRICS Battery to measure global cognition (Nuechterlein et al., 2008), the Positive and Negative Syndrome Scale (PANSS) to measure symptoms (Kay et al., 1987), Strauss Carpenter Outcome Scale to measure social contact, hospitalizations, and engagement in school/work (Heinrichs et al., 1984), the Global Functioning Role and Social Scales, and an estimate of Full-Scale Intelligence Quotient based on verbal reasoning. Means and standard deviations of all predictors are described in Table 1A. Standard demographic variables including age, gender, education, and duration of illness were also collected. Together, 10 total variables were included in the statistical models (summarized in Table 1).

Table 1

Baseline statistics, correlations, and linear regression model predicting change in global cognition.

Measure	A) Baseline		B) Correlation		C) Linear regression
	Mean	SD	r-Value	p-Value	Estimate	Std. error	t-Value	p-Value
(Intercept)	–	–	–	–	− 0.84	1.49	− 0.56	0.58
Global cognition	− 0.85	0.74	− 0.31	0.04	− 0.21	0.13	− 1.67	0.11
Symptoms	58.21	12.57	− 0.02	0.89	− 0.001	0.01	− 0.28	0.78
GFR	4.83	2.44	0.03	0.83	− 0.02	0.04	− 0.6	0.55
GFS	5.79	1.39	− 0.1	0.53	− 0.09	0.07	− 1.25	0.22
Strauss	8	2.23	0.02	0.92	0.06	0.05	1.18	0.25
Duration	18.67	15.61	0.06	0.72	− 0.001	0.004	− 0.203	0.84
Age	21.69	3.29	0.04	0.78	− 0.02	0.03	− 0.56	0.58
FSIQ	102.76	12.24	− 0.15	0.33	− 0.003	0.01	− 0.41	0.68
Gender	30 male	–	–	–	0.18	0.16	1.1	0.28
Education	12.88	1.62	0.29	0.06	0.13	0.05	2.53	0.02

Note. Baseline statistics and correlation and linear regression models of all predictors and their relationship to change in global cognition score. (A) Means and standard deviations of all predictors at baseline. (B) No correlations between baseline measures and global cognition were found to be significant after correcting for multiple comparisons (critical p = 0.005). (C) The linear regression of all predictors only showed Education to be a significant predictor of change in global cognition score. However, the full model was not significant (F = 1.61; p = 0.15; Multiple R2 = 0.34).

Planned analysis

Preliminary analyses sought to characterize the relationship between baseline demographic variables and change in cognition. To do so, we used Pearson correlations to determine whether baseline measures related to change in global cognition (post-pre). Next we entered all measures into a general linear model to examine relationships holding all other variables constant. We then used the ‘GLMNET’ package in R (Friedman et al., 2015) to perform a LASSO regression, which is an extension of generalized regression that minimizes residual sum of squares by shrinking some coefficients to zero, thereby eliminating them from the model (Tibshirani, 1996). This allows for stable model selection and regression without over-fitting.

Here we carry out a 10-fold cross validation of LASSO to produce an optimal tuning parameter (minimum value of lambda) that would minimize the cross-validated error and also protect against tests already run by the model. The final model retained all predictors with coefficients not equal to zero. Though inferential statistics for LASSO have been proposed (Lockhart et al., 2014), these estimates are vulnerable to substantial bias, and are therefore subject to further study and not included current statistical packages. Instead we conduct a post-hoc linear regression with the retained predictors to estimate inferential statistics, to compare to the previous full model.

We also sought to determine whether there was a curvilinear relationship between change in global cognition and any of the predictors or their interactions. We constructed the second order terms from the 10 original predictors, resulting in a new set of predictors consisting of 75 variables (10 original variables, 10 variables that were the square of the original variables, and 55 interaction terms between the original variables). We again used the 10-fold cross validation LASSO procedure to identify the non-zero coefficients. We followed up this analysis with another linear regression that also included any polynomial terms. Last, we conducted post-hoc LASSO analyses separating out the individual cognitive domains as predictors (i.e. speed of processing, verbal working memory, visual working memory, and problem solving) as opposed to using the global cognition score. We also tested the LASSO model in subjects randomized to the study's computer game control condition (N = 43) to determine whether observed changes in cognition may be reflecting regression to the mean. Finally, we also tested the cross-validated LASSO model in a separate sample of chronic schizophrenia patients (N = 29) who underwent TCT in a related randomized controlled trial (Fisher et al., 2009).

Results

Pearson correlations between all baseline measures and change in global cognition were conducted. No measure showed a significant relationship with change in global cognition after correcting for multiple comparisons (critical p = 0.005; Table 1B). Next we entered all variables into a regression model and found that only years of education was predictive of increased general cognition following TCT (t = 2.53; p < 0.05) when holding all other variables constant (Table 1C). However, the full model fit was found not to be significant (F = 1.61; p = 0.15; Multiple R2 = 0.34).

Next we used a 10-fold cross-validation of LASSO to determine variable retention and their regression coefficients. Only baseline global cognition, gender, and education were found to be predictive of improvement and were found to explain 23% of the variance in improvement in global cognition (Table 2A). We then conducted a post-hoc linear regression model entering the retained predictors from LASSO (baseline global cognition, gender, and education) to draw statistical inferences on the model, and compare them to the previous models (Table 2B). Both baseline global cognition and baseline education were observed to be significantly predictive of increased general cognition improvement, while gender was not. The full model was found to be significant (F = 4.94; p = 0.005; Multiple R2 = 0.28).

Table 2

LASSO model predicting change in global cognition.

Measure	A) LASSO	B) Linear regression
	Coefficient	Estimate	Std. error	t-Value	p-Value
(Intercept)	− 0.74	− 1.58	0.58	− 2.72	0.01
Global cognition	− 0.16	− 0.25	0.08	− 2.94	0.01⁎
Gender	0.05	0.18	0.13	1.37	0.18
Education	0.07	0.12	0.04	3.05	0.004⁎

Note. LASSO and linear regression models of retained predictors after 10-fold cross validation of LASSO. (A) Predictors and their coefficients retained after 10-fold cross validation of LASSO. (B) The linear regression of the retained predictors showed both baseline global cognition and baseline education to be significantly predictive of change in global cognition. This model, including baseline global cognition, gender, and education was found to be significant (F = 4.94 (df = 38); p = 0.005; Multiple R2 = 0.28).

indicates significance at p < .05.

We followed this analysis by examining a polynomial regression model with LASSO, containing the 10 original predictors, the squares of those predictors, and 55 interaction terms. 10-fold cross validation of LASSO again yielded three non-zero coefficients, including baseline education, the squared baseline global cognition term, and the interaction between education and gender (Table 3A). Again, we conducted a post-hoc linear regression with the retained predictors, and found that both education and squared global cognition were significant, with the full model also achieving significance (F = 6.64 (df = 38); p = 0.001; Multiple R2 = 0.34). The LASSO procedure found no significant predictors when using the cognitive domain scores as individual predictors in the model. Additionally, the initial LASSO model was found not to be significant when applied to a separate group of more chronic schizophrenia patients who underwent a similar TCT intervention (Fisher et al., 2009). Finally, we also assessed whether improvements in global cognition may be reflecting a regression to the mean. We assessed the correlation between baseline global cognition and change in global cognition in the parent study's computer games control group (N = 43) (Fisher et al., 2014) and found no significant relationship (r = − 0.04; p = 0.78). Additionally, no significant predictors were obtained when submitting the subjects in the control condition to the LASSO procedure.

Table 3

Polynomial LASSO model predicting change in global cognition.

Measure	A) LASSO	B) Polynomial regression
	Coefficient	Estimate	Std. error	t-Value	p-Value
(Intercept)	− 0.44	− 1.21	0.48	− 2.51	0.02
Education	0.05	0.1	0.04	2.65	0.01⁎
Global cognition2	0.08	0.12	0.03	3.64	0.001⁎
Education × gender	0.003	0.01	0.01	1.2	0.24

Note. LASSO and linear regression models of retained polynomial predictors after 10-fold cross validation of LASSO. (A) Polynomial and linear predictors and their coefficients retained after 10-fold cross validation of LASSO. (B) The linear regression of the retained predictors showed both baseline education and baseline squared global cognition to be significantly predictive of change in global cognition. This model, including baseline education, squared global cognition, and an education by gender interaction was found to be significant (F = 6.64 (df = 38); p = 0.001; Multiple R2 = 0.34).

indicates significance at p < .05.

Discussion

The current study used LASSO model selection and regression to identify baseline measures predictive of response to TCT of auditory systems in individuals with recent onset schizophrenia. SZ patients with lower baseline global cognition scores and more years of education were found to have greater improvements in global cognition following TCT. This relationship was not observed in a post-hoc analysis of the individuals randomized to undergo a computer games control condition. We also found that these relationships were even more robust when accounting for the quadratic relationships between baseline global cognition and change in this measure. A model with baseline education, baseline squared global cognition score, and an interaction between baseline education and gender was found to further improve model fit. We also establish cross validation of LASSO regression as a viable method for predicting treatment outcome in a psychiatric population. Though this method has been established in genetic and neuroimaging research, it is gaining prominence in the examination of psychological data for the purposes of behavioral predictions (Yarkoni and Westfall, 2017). Here we show here that it may be appropriately used to model psychiatric outcomes data, and may be crucial tool in computational approaches to psychiatric research.

Previous work has demonstrated that baseline cognitive status plays a predictive role in response to domain-focused cognitive remediation and strategy coaching interventions for SZ. These studies find that higher baseline cognitive status is predictive of response to treatment (Kurtz et al., 2009, Medalia and Richardson, 2005; J M Fiszdon et al., 2006a, Fiszdon et al., 2006b, Lindenmayer et al., 2017), possibly indicating that “the rich get richer” in response to such an intervention. In contrast, interventions that use a compensatory skill learning or decision-making heuristic training approach have demonstrated that lower baseline cognition is predictive of stronger cognitive gains following the intervention (Rodewald et al., 2014, Twamley et al., 2011).

The current study also suggests that lower baseline cognition is predictive of improvement in recent onset SZ patients in response to intensive training that targets early auditory processing and working memory operations. This indicates that TCT may be an ideal intervention for lower functioning individuals. Differing from both the domain-focused training, as well as compensatory cognitive remediation approaches, TCT emphasizes training of early phases of perceptual processing while simultaneously requiring close engagement of attentional control and prefrontal predictive and response selection operations. This intervention has been demonstrated to evoke neuroplasticity in sensory (Dale et al., 2015, Popov et al., 2011), subcortical (Ramsay et al., 2017), and higher-order (Subramaniam et al., 2014) cortical regions.

Education also appears to be a critical factor predictive of response to treatment, suggesting that those who have experienced higher levels of education (i.e. some college) may be more responsive to TCT than those with lower educational attainment. This has been shown in studies examining cognitive remediation for schizophrenia (Lindenmayer et al., 2017), perhaps indicating that individuals with more practical academic skills (i.e. time management, task planning, goal setting) are likely to benefit more from cognitive training interventions. These findings may also be consistent with the ‘cognitive reserve hypothesis,’ which proposes that individuals with higher IQ, occupational, or educational attainment show more resilience to neuropsychiatric disorders including schizophrenia (Barnett et al., 2006, Koenen et al., 2009). Individuals with more educational attainment, and subsequent cognitive reserve, may also have more neuroplastic capacity, which has been demonstrated in healthy older adults (Barulli and Stern, 2013, Lövdén et al., 2010). This could be contributing to mechanisms that support neuroplasticity in response to TCT and similar interventions for schizophrenia (Ramsay and MacDonald, 2015). Numerous factors contribute to high school and college dropout in individuals with schizophrenia (Goulding et al., 2010), but less is known about factors that may promote resilience in an academic setting. Overall these and other findings suggest that higher education can contribute to training outcomes, and should be considered when making clinical recommendations. However, future studies will be required to examine this effect further.

Finally, though not significant in the linear regression model itself, gender was observed to contribute to overall fit, suggesting that females had slightly better outcomes in response to TCT. This is largely consistent with literature indicating that males with schizophrenia have poorer premorbid adjustment and more severe negative symptoms, possibly contributing to poorer outcomes overall (Abel et al., 2010). Future studies will be required to more carefully parse gender differences in response to TCT interventions for schizophrenia, and how this may influence cognition.

In the current study the LASSO model proved not to be predictive when breaking down cognition by individual domains. No variables survived the LASSO procedure when including speed of processing, verbal working memory, visual working memory, or problem solving individually. This may suggest that a “general” cognitive factor is better suited to predict global cognitive improvements, which is consistent with the observation that cognitive disruptions in schizophrenia are characterized by a “generalized deficit” (Mohamed et al., 1999, Schaefer et al., 2013). Future studies will be required to parse out the relative contributions of baseline abilities within different cognitive domains and their mediating effects on cognitive improvement following TCT. Relatedly, we note that while lower baseline cognitive scores might naturally move closer to the average upon retest, we did not observe this relationship in the post-hoc analysis of the control condition. Therefore, we can more confidently conclude that improvements in response to the intervention did not simply reflect a regression to the mean, and are likely a reflection of the TCT more specifically.

Last, we highlight that these results were limited to patients with recent onset schizophrenia, mostly within the first 5 years of the illness. While meta-analytic findings suggest that there are few differences between early and chronic schizophrenia with regard to cognitive training treatment response (Revell et al., 2015), some specific interventions have observed more generalized and robust influence on cognition in early schizophrenia patients (Bowie et al., 2014). In the current study, we applied the identified model to a separate cognitive training dataset that examined chronic schizophrenia patients, and found that the relationship between baseline cognition, education, and gender was not significant. This indicates that the current model's predictive ability is more relevant in people early in the illness course, and may support the hypothesis that younger individuals may be more receptive to intensive neuroplasticity-based interventions (Wykes et al., 2009). Future work will be required to extend the current findings and test the model's predictive capacity in other samples and to determine its clinical utility.

Conflict of interest statement

The training software used in this study was supplied free of charge by Posit Science. Sophia Vinogradov has also served as a paid consultant to Posit Science Inc. There are no other competing interests.