Do cognitive impairments limit treatment gains in a standalone digital intervention for psychosis? A test of the digital divide.

Digital mental health interventions, such as those provided by smartphone applications (apps), show promise as cost-effective approaches to increasing access to evidence-based psychosocial interventions for psychosis. Although it is well known that limited financial resources can reduce the benefits of digital approaches to mental healthcare, the extent to which cognitive functioning in this population could impact capacity to engage in and benefit from these interventions is less studied. In the current study we examined the extent to which cognitive functioning (premorbid cognitive abilities and social cognition) were related to treatment engagement and outcome in a standalone digital intervention for social functioning. Premorbid cognitive abilities generally showed no association with aggregated treatment engagement markers, including proportion of notifications responded to and degree of interest in working on app content, though there was a small positive association with improvements in social functioning. Social cognition, as measured using facial affect recognition ability, was unrelated to treatment engagement or outcome. These preliminary findings suggest that cognitive functioning is generally not associated with engagement or outcomes in a standalone digital intervention designed for and with people with schizophrenia spectrum disorders.

Introduction

Evidence-based psychosocial treatments for psychosis have strong support for improving symptoms and recovery outcomes among people with serious mental illness (SMI), including schizophrenia spectrum disorders (Bighelli et al., 2021; Frawley et al., 2021; Turner et al., 2018). However, as few as 10% of people with a diagnosed SMI receive these interventions (Dixon et al., 2010). Lack of access leads to poorer outcomes, including higher rates of relapse and hospitalization, and limited community engagement (Lehman et al., 1998). Digital mental health interventions, such as those provided by smartphone applications (apps), show promise as cost-effective approaches to increasing access to evidence-based psychosocial interventions for psychosis (Ben-Zeev et al., 2019).

Limited access to technology is often a barrier to reaping the benefits of digital approaches to mental healthcare in SMI (Fulford and Mote, 2019). Estimates show a large range of ownership of digital technology, including smartphones. For example, fewer than 50% of patients with schizophrenia receiving care in the community report owning a smartphone (Klee et al., 2016; Young et al., 2020), while the rate is as high as 90% in patients with SMI in a private clinic (Torous et al., 2018). The varying rates are tied to socio-demographic factors, including age and economic status. Not surprisingly, older people with SMI report lower rates of smartphone ownership than younger ones (Watson et al., 2021), and receipt of disability payments is a strong predictor of reduced likelihood of smartphone ownership (Young et al., 2020). While there are affordable options for smartphones, the financial resources needed for purchasing data plans can be prohibitive for those living in poverty (Vogels, n.d.).

Even if such technologies can be provided to people with limited resources, such as through a public-funded initiative, it is unclear if general lack of familiarity with the technology could limit the potential benefits. Furthermore, characteristic symptoms and impairments in people with SMI themselves could interfere with capacity to engage in, and thus benefit from, digital interventions. One such characteristic concerns intellectual and cognitive ability. Studies across all stages of psychosis suggest cognitive impairment is a key feature of SMI (Fett et al., 2020; Seidman et al., 2006). Although low educational attainment does not appear to limit gains in traditionally-delivered evidence-based treatments for psychosis, such as CBT (Turner et al., 2020), digital interventions such as those implemented through smartphone apps require active participation, often with limited human support, which could create additional challenges in engagement for those with cognitive impairment. Thus, it is important to evaluate whether intellectual and cognitive functioning is related to digital treatment engagement and outcomes.

A related challenge for people with SMI is in social cognitive impairment, which impacts social outcomes (Couture et al., 2006; Fett et al., 2011) and could potentially limit the gains of interventions designed to improve social relationships. Interventions designed to directly improve social cognition, such as Social Cognition Training or Cognitive Enhancement Therapy, demonstrate positive benefits in social cognitive abilities and functioning (Nijman et al., 2020); however, the extent to which social cognitive abilities impact the benefits of social goal interventions, particularly those delivered digitally, is unknown. That is, difficulties in identifying emotions and thoughts in others—through impaired affect recognition and theory of mind abilities—could limit successful implementation of strategies designed to form and maintaining interpersonal bonds, such as in supporting social goal attainment.

We previously developed a smartphone app to provide guided support in improving social functioning for people with psychosis. This app—Motivation and Skills Support (MASS)—was designed with direct input from people diagnosed with schizophrenia spectrum disorders, with a key aim to minimize the potential impact of cognitive or motivational impairments on treatment engagement. We used e-design tools validated to reduce burden of these impairments in people with serious mental illness (Rotondi et al., 2017), and focused on enhancing engagement through empirical work on the temporal experience of pleasure model (Fulford et al., 2018; Gard et al., 2014). Intervention development occurred in two phases—one in which participants engaged with the app over brief periods of time and provided qualitative feedback, allowing for iterative design (Fulford et al., 2020); in the second phase we examined usability and preliminary efficacy on target outcomes (Fulford et al., 2021). Our ultimate goal was to develop an empirically- and theoretically-informed digital intervention that could be used in daily life without regular clinician or caregiver support.

In the current study we examined the extent to which intellectual and cognitive abilities were associated with treatment engagement and outcome in a pilot evaluation of the MASS app digital intervention. We hypothesized that premorbid cognitive and social cognitive abilities would not be associated with treatment engagement, including responses to smartphone notifications or interest/motivation in app content (i.e., working toward a social goal), given our user-centered approach to design elements and usability. Relatedly, we hypothesized that premorbid cognitive ability would not be significantly associated with changes in the primary treatment outcome of social functioning. Conversely, we hypothesized that higher levels of social cognition would be significantly related to greater gains in social functioning over the intervention, given their role in providing a foundation for effective social skills and functioning.

Method

Data presented in this paper were collected as part of an open pilot intervention study that took place in the Boston and San Francisco Bay areas (registered clinical trial NCT03404219; see Fulford et al., 2021). Clinical and outcome data were collected before and immediately following the eight-week intervention, as well as three months following treatment termination. Interested participants who met screening criteria completed an assessment of eligibility and baseline clinical measures, followed by the intervention. Participants had to be fluent in English and between the ages of 18 and 65 years, and receiving current standard care (medication, psychotherapy, or both). Participants were not eligible if they endorsed: 1) current substance use disorder (i.e., over the past six months) or 2) suicidal ideation, or 3) reported a diagnosed neurological disorder. In the current study, we present findings on participants who completed the intervention and had all available data for planned analyses.

Thirty-one individuals diagnosed with schizophrenia (n = 17) or schizoaffective disorder (n = 14) (mean [SD] age = 46 [11] years) were recruited using fliers and in-person presentations at community treatment and rehabilitation centers that serve people with SMI. Nine (29%) graduated from college, 11 (36%) were employed at least part-time, and 21 (75%) received disability payments. Severity of positive and negative symptoms was mild to moderate at baseline (see Fulford et al., 2021).

Motivation and Skills Support (MASS) smartphone app

The primary focus of the MASS app was to support participants in working toward a collaboratively identified social goal. Participants met with a research assistant at baseline to identify a goal that was feasible to achieve during the intervention, and this goal was pre-programmed into the app. Examples of goals include making a new friend by going to events and improving an existing relationship. The research assistant introduced participants to the smartphone (Samsung Galaxy S8, provided by the study team) and the app, and they were asked to demonstrate functions on the phone and app to ensure comprehension. Phones included prepaid data and call and text plans for the duration of the eight-week intervention.

The MASS App was a standalone smartphone intervention through which social goal support content was delivered via push notifications. Participants had access to the app at any time, but were sent two notifications with a direct link to the app, twice per day over the eight-week intervention. The app included a set of questions designed to gather data on social behavior (e.g., quality and quantity of social interactions), and the option to access a list of steps to take in pursuit of their social goal. Examples of steps toward a goal of making a new friend might include: 1) identifying an event by searching online, 2) finding transportation to the event, and 3) introducing oneself to someone at the event. Each of these steps would have additional details to support successful implementation. Participants could also access social skills training video content, which included recorded displays of social skills in the context of two storylines—one in which two old friends reconnect, and another in which two individuals worked to build a new relationship. For more details on the MASS intervention development, see Fulford et al. (2020), and for information on primary outcomes of the intervention, see Fulford et al. (2021).

Measures

Socio-demographic variables

We collected basic demographic information through self-report and interview. Relevant variables for the current study include age in years; gender (male, female, non-binary); race and ethnicity; level of education—an ordinal variable ranging from 1 (through grade 6 or less) to 7 (professional or graduate degree); employment (employed at least part-time or unemployed); disability (receiving disability payments or not).

Cognitive ability

Our primary measure of cognitive ability was the Wide Range of Achievement Test, 4th edition – Word Reading (WRAT-WR) subtest (Wilkinson and Robertson, 2006). In this subtest, respondents are asked to accurately pronounce words of increasing difficulty. When ten consecutive words are pronounced incorrectly, the test stops. The total correct words are then converted to a scaled score based on the participant's age. WRAT-WR subtest scores have established evidence of criterion validity for a measure of cognitive abilities, above and beyond educational attainment (Sayegh et al., 2014). Although WRAT-WR scores are typically considered to be markers of premorbid cognitive function, they are strongly correlated with neurocognitive abilities in schizophrenia (Wilk et al., 2004). In the current study, three participants were missing WRAT-WR data due to errors in study procedures.

Our measure of social cognition was the Penn Emotion Recognition Test (ER-40; (Kohler et al., 2003, Kohler et al., 2004)). The ER-40 is a test of facial affect recognition—one of several components of social cognition—in which participants are shown photographs of 40 faces and select among a list of possible emotions expressed in the picture. We examined total accuracy (number of correctly identified emotions) as an indicator of social cognitive ability.

Digital treatment engagement and outcome

We operationalized the degree of engagement in the digital treatment using three metrics: 1) proportion of smartphone notifications responded to (out of a possible 120); 2) proportion of completed smartphone surveys in which participants elected to work on their social goal (i.e., affirmative response to “Would you like to take any steps towards your social goal today?”); and 3) average degree of motivation to work on their social goal, reported in the app (i.e., “How motivated are you to work on this step?”, with five response options ranging from ‘Not at all motivated’ to ‘As motivated as possible’). The primary treatment outcome was social functioning as assessed by the Social Functioning Scale (SFS; (Birchwood et al., 1990)). In the current paper, we examined SFS scores at baseline, and change scores from baseline to treatment termination.

Analyses

We first examined distributions of our variables of interest, including the presence of univariate and multivariate outliers. We then examined associations between our two primary measures of cognitive abilities, WRAT-WR and ER-40, and demographic variables, including age, gender, ethnicity/race, educational attainment, and employment status. We also explored associations between markers of digital treatment engagement (i.e., adherence) and socioeconomic status (i.e., education, employment, and disability status).

We then ran bivariate correlations between measures of cognitive ability (WRAT-WR) and social cognition (ER-40) and markers of treatment engagement and outcome. We used Spearman's rho given the ordinal nature of the treatment engagement data. To aid in interpretation of potential associations, we examined mean values of treatment engagement and outcome data by those high and low in premorbid cognitive ability, based on a WRAT-WR standard score of 100. Given our sample size for primary measures (n = 28–31), we were powered to detect moderate correlations (r values 0.48 to 0.51 and above) with a probability of Type II error (beta) of 0.20 and two-tailed Type I error (alpha) of 0.05.

We calculated Bayes factors to evaluate support for null hypotheses (i.e., no association between cognition and treatment engagement and outcome). Bayes factors provide information as to whether the observed data are more likely under one model than the other (Schmalz et al., 2021). By calculating the ratio between the likelihood of the alternative hypothesis relative to the null hypothesis, the Bayes factor can be interpreted as the degree of evidence for the probability of the alternative hypothesis; thus, in this case a large value provides evidence in favor of a positive association between cognition and treatment engagement/outcome, while a small value provides evidence in favor of the null (no association between these variables); a value of 1 indicates that observed data are equally likely under either model. In general, Bayes factors less than 0.33 indicate evidence for the null hypothesis, while those greater than 3 indicate evidence for the alternative hypothesis; those greater than 0.33 and less than one are considered anecdotal evidence for the null, while those greater than 1 but less than 3 are considered anecdotal evidence of the alternative (Lee and Wagenmakers, 2014). Given the lack of existing data relevant for these hypotheses, we used default priors for all correlations (i.e., a stretched beta distribution of 1). Bayesian analyses were performed in JASP ().

Results

Data were normally distributed and there were no individual univariate or multivariate outliers. Mean WRAT-WR and ER-40 scores were slightly higher but relatively comparable to those documented in previous studies in schizophrenia (i.e., WRAT-WR were 2–3 scaled points higher, and ER-40 scores were 1% higher, than in prior studies; see (Hill et al., 2013; Pinkham et al., 2018)). Bivariate correlations between premorbid cognition (WRAT-WR standard score) and social cognition (ER-40 total accuracy) and demographic variables indicated only one significant association—higher WRAT-WR was associated with higher level of education (rho = 0.40, p = 0.04). Higher ER-40 was also associated with higher level of education, though this was not statistically significant (rho = 0.31, p = 0.09). Indicators of socioeconomic status (education level, employment, receipt of disability payments) were unrelated to aggregated markers of treatment engagement (see Table 1).

Table 1

Socio-demographic variables and associations with cognition.

WRAT-WR standard score
	n	Mean	SD	t-test	Rho/r	p
Age				–	−0.10	0.60
Education level				–	0.40⁎	0.04
Male	15	91.47	11.93	−1.69	–	0.11
Female	12	100.92	16.20
Disability payments	19	95.11	15.84	−1.39	–	0.19
No disability payments	7	104.14	14.69
White	13	94.15	14.67	−0.78	–	0.44
Non-white	15	98.60	15.41
Employed	9	94.11	13.97	0.61	–	0.55
Unemployed	19	97.68	15.65
Total	28	96.50	15.00	–	–	–

Note. ER-40 = Penn Emotion Recognition Test; WRAT-WR = Wide Range Achievement Test – Word Reading. For gender, one participant identified as non-binary and was not included in analyses. One participant declined to answer whether they received disability payments, and two others were unsure. Total possible n for WRAT-WR data = 28.

p < 0.05.

WRAT-WR scores showed small to moderate negative associations with aggregated treatment engagement markers, including proportion of notifications responded to (rho = −0.33), proportion of notifications in which the participant elected to work on their goal (rho = −0.12), and degree of interest in working toward the goal (rho = −0.02, see Fig. 1 and Table 2). When examining proportion of notifications responded to as a function of WRAT-WR standard score, those higher in premorbid cognition (i.e., above 100 standard score) did not differ in the proportion of completed data than those lower in premorbid cognition—54% vs. 42%, respectively (t = 1.10, p = 0.28). WRAT-WR scores also showed a small positive association with change in social functioning (i.e., scores at termination, controlling for baseline scores: b [SE] = 0.12 [0.07], t = 1.84, p = 0.08, Δ R2 = 0.05). ER-40 total accuracy showed small associations with treatment engagement and outcome (rho values < ±0.20).

Fig. 1

Scatterplots of association between WRAT-WR and treatment engagement.

Table 2

Associations between cognition and treatment engagement and outcome.

WRAT-WR standardized score
Treatment engagement	Rho	B10	95% CI of B10
Notifications responded to	0.33	+0.962	−0.049–0.605
Elect to work on social goal	−0.12	*0.318	−0.479–0.219
Degree of interest in social goal	−0.02	*0.240	−0.379–0.343
Treatment outcome
Change in SFS	0.38	♦1.530	0.001–0.637

Note. B = Bayes factor (support for alternative hypothesis); CI = Credible Interval; ER-40 = Penn Emotion Recognition Test; SFS = Social Functioning Scale; WRAT-WR = Wide Range Achievement Test – Word Reading.

*Moderate evidence for null hypothesis.

+Anecdotal evidence for null hypothesis.

♦Anecdotal evidence for alternative hypothesis

Generally, Bayes factors suggested anecdotal evidence for the hypotheses of no association between WRAT-WR and EMA notifications responded to (B10 = 0.962) and election to work on the social goal (B10 = 0.318), and moderate evidence of no association between WRAT-WR and motivation to work on the social goal (B10 = 0.240). For the association between WRAT-WR and change in social functioning, there was anecdotal evidence of the alternative hypothesis (i.e., a positive correlation; B10 = 1.530). There was also moderate evidence of no associations between ER-40 and treatment engagement and outcome (B10 range: 0.23 to 0.34). Results are presented in Table 2.

Discussion

It is unknown to what extent impaired cognitive abilities may reduce potential benefits of standalone digital interventions for psychosis, due to problems such as difficulty navigating software and hardware in the latest mobile technologies. These challenges could limit the initiation and sustained use of technologies in daily life and contribute to a ‘digital divide’ between those who do and do not benefit from such interventions, diminishing the potential for increased access to evidence-based interventions in this hard to reach population (Fulford and Mote, 2019). In the current study we sought to provide a preliminary evaluation of this question.

Our findings suggest that in a middle-aged sample of outpatient participants with a schizophrenia spectrum disorder, premorbid cognitive ability was not associated with indicators of engagement in a digital intervention for social functioning. This preliminary finding is encouraging and quells some concerns that cognitive abilities could diminish the accessibility of standalone mobile interventions designed for this population. Importantly, the MASS intervention was developed for and with people with schizophrenia spectrum disorders. As such, our findings cannot address the extent to which cognitive impairments might impact engagement with mental health apps available to the general public.

There was a small association between higher cognitive ability and greater improvement in social functioning over the intervention—WRAT-WR scores explained 5% of the variance in SFS score improvement, with a Bayes factor that suggested anecdotal evidence for the alternative hypothesis. The clinical significance of this small association may not be especially meaningful, but does indicate that cognitive ability could have some impact on outcomes in digital psychosocial interventions, consistent with findings that people with greater cognitive impairment may be less likely to benefit from community-based psychosocial rehabilitation for schizophrenia (Kurtz et al., 2011). More work is needed to provide more robust tests of such potential associations, in larger samples and with more comprehensive measures of cognition and treatment engagement.

We also examined other potential contributors to the digital divide, including associations between treatment engagement and indicators of socioeconomic status. Education level, employment, and disability payment status were largely unrelated to proportion of notifications responded to, or participation or motivation for social goal support. These findings further suggest that availability of financial and other resources may not be robustly associated with the potential benefits of digital interventions among people with SMI. It is important to note, however, that participants in the study were provided with phones and data plans—thus, we can only speak to the extent to which such limited resources might impact the use (and not availability or access to) such digital interventions.

Finally, social cognitive ability at baseline, as assessed via facial affect recognition, was unrelated to treatment engagement or outcome. Social cognition is a multifaceted construct that includes several related information processing abilities that support basic capacity for interpersonal communication (Frith, 2008). Other facets of social cognition that might support treatment engagement and/or outcome, but that we did not measure in this study, include theory of mind or mentalizing, joint attention, and empathy. Furthermore, other interpersonal factors, such as social skills and motivation, may be more directly related to social functioning benefits (Fulford et al., 2018). Inclusion of measures of these constructs in future studies could provide evidence of the relevance of such factors for digital treatment engagement and outcome.

There are several limitations in the current study that should be mentioned. Our sample size was relatively small, limiting the potential to detect more subtle associations. We were only powered to detect moderate or higher correlations. Indeed, there were several relationships between cognition and intervention engagement and outcome that were positive but small in magnitude, which with a larger sample could have passed thresholds of statistical significance. Also, while the WRAT-WR has demonstrated evidence of convergent validity with more comprehensive measures of neurocognition, this scale is typically considered to be a measure of premorbid cognitive ability or intellectual function. There may be specific cognitive abilities that are more strongly tied to digital intervention engagement and outcomes. For example, executive function may be especially relevant for self-management skills that support sustained use of digital interventions.

In sum, in a standalone digital intervention for social function in psychosis, we did not find strong associations between premorbid cognitive ability or social cognition and treatment engagement and outcome. These findings suggest that while limited resources may certainly impact access to mobile devices, such as smartphones and associated mental healthcare apps, characteristic cognitive impairment among people with SMI may not contribute robustly to such a digital divide.

CRediT authorship contribution statement

Fulford: Conceptualization, Methodology, Writing – Original Draft, Writing – Review and Editing, Supervision, Funding acquisition; Schupbach: Data curation, Writing – Review and Editing; Gard: Conceptualization, Methodology, Writing – Review and Editing, Supervision, Funding acquisition; Mueser: Conceptualization, Methodology, Writing – Review and Editing, Funding acquisition; Mow: Investigation, Data curation; Leung: Investigation, Data curation.

Declaration of competing interest

The authors declare no conflicts of interest associated with the data presented in this manuscript.