Measuring functional outcomes in schizophrenia in an increasingly digital world.

With an unmet clinical need for effective interventions for cognitive and negative symptoms in patients with schizophrenia, measures of functional status (often a co-primary endpoint) remain key clinical trial outcomes. This review aims to give an overview of the different types of functional assessments commonly used in clinical trials and research involving patients with schizophrenia and highlight pertinent challenges surrounding the use of these as reliable, sensitive, and specific assessments in intervention trials. We provide examples of commonly used functional measures and highlight emerging real-time digital assessment tools. Informant- and clinician-rated functional outcome measures and functional capacity assessments are valid, commonly used measures of functional status that try to overcome the need for often overly ambitious and insensitive 'real world' milestones. The wide range of scientific and practical challenges associated with these different tools leave room for the development of improved functional outcome measures for use in clinical trials. In particular, many existing measures fail to capture small, but meaningful, functional changes that may occur over the course of typically short intervention trials. Adding passive digital data collection and short active real-time digital assessments whilst patients go about their day offers the opportunity to build a more fine-grained picture of functional improvements that, if thoughtfully developed and carefully applied, could provide the sensitivity needed to accurately evaluate functional status in intervention studies, aiding the development of desperately needed treatments.

Introduction

Schizophrenia is a major cause of disability. It tends to develop in adolescence, with negative symptoms (e.g. anhedonia and apathy) and cognitive symptoms (e.g. impaired attention and memory) typically emerging before positive symptoms (e.g. hallucinations and delusions). Current treatments target positive symptoms, but there is a significant unmet clinical need for effective treatments for negative symptoms and cognitive impairment in patients with schizophrenia (Talpos, 2017). Negative and cognitive symptoms persist after positive symptoms have remitted and are predictive of functional outcomes (e.g. living independently and working; (Fett et al., 2011; Shamsi et al., 2011; Hunter and Barry, 2012; Green et al., 2015)). This impairment is costly, with 50–85% of economic costs associated with schizophrenia being indirect (e.g. unemployment; (Chong et al., 2016)). Functional outcomes are important to patients' quality of life (Cotter et al., 2019; Shepherd et al., 2012) and were negatively affected by the changes to everyday life imposed by the recent COVID-19 pandemic (Kozloff et al., 2020). Hence, it is increasingly important to drive treatment development that can help to rectify functional impairment with substantial real-world benefits for patients, their families, and society.

To achieve this, it is crucial to ensure valid and reliable measurement of functional outcomes. In this narrative review, we will give an overview of the main types of measures of ‘function’ in schizophrenia by discussing selected, commonly used examples. We also showcase some more recent digital developments and explore how these could refine the measure of ‘functioning’ in patients with schizophrenia.

Types of tools used to measure functioning in patients with schizophrenia

Whilst real-world milestones (such as employment) provide clear treatment goals for patients, they are often beyond the scope or timeframe of a typical drug intervention study. This led to the development of a multitude of proxy-measures of functioning in people with schizophrenia. These can be broadly categorized into activity- and capacity-based measures. Activity-based measures are typically further subdivided into (1) interview-based clinician-rated scales that assess the frequency and quality of patient engagement in social, occupational and academic activity, and (2) subjective measures of functioning and the overlapping concept of quality of life using patient self-report questionnaires. Furthermore, the related concept of (3) functional capacity can be measured through (a) clinician-based ratings of these abilities or (b) performance-based measures that assess an individual's ability to perform day-to-day tasks in a controlled environment.

This section provides an overview of these categories, using examples of associated tools commonly used in schizophrenia research and clinical studies. The most common assessments of patient functioning in clinical trials are interviewer-rated measures, with both the Personal and Social Performance scale (PSP; (Morosini et al., 2000)) and the Global Assessment of Functioning (GAF; (Hall, 1995)) having been used in over 20 studies within the last 5 years. A variety of different self-reported quality of life measures (e.g. Social Adjustment Scale-Self Report (SAS; (Weissman, 1999)) are more commonly included in trials than self-reported function measures like the Social Functioning Scale (SFS; (Birchwood et al., 1990)). Functional capacity is assessed in approximately half of the clinical trials investigating functioning. The most commonly used measures of functional capacity are the interviewer-rated Schizophrenia Cognition Rating Scale (SCoRS; (Keefe et al., 2006, Keefe et al., 2015)) and the University of California, San Diego Performance-Based Skills Assessment (UPSA; (Patterson et al., 2001)). In the last year, 4 trials employing a newly developed digital tool, the Virtual Reality Functional Capacity Assessment Tool (VRFCAT; (Keefe et al., 2016)) have been launched.

Interviewer-rated measures of functioning

Interview-based, clinician-rated measures are commonly used to assess functioning in people with severe mental illnesses. The Global Assessment of Functioning (GAF; (Hall, 1995)) and the Social and Occupational Functioning Assessment Scale (SOFAS; (Goldman et al., 1992)) have been among the most widely used measures of global functioning in patients with schizophrenia. They provide useful measures of overall impairment, but lack sensitivity to dysfunction within specific subdomains (e.g. social, occupational). Milestone achievements (e.g. employment, residential status) are weakly associated with ‘global’ scores, but strongly map onto subscale scores of different functional domains (e.g. vocational, residential; (Harvey et al., 2012)). Hence, collapsing functioning into a single ‘global’ score may obscure the ability to detect deficits or changes in particular domains (Harvey, 2013). Rating the GAF and SOFAS can be difficult, requires extensive training and experience to ensure reliability and consistency within and between raters. Limited scoring information is provided, leading to questions regarding the reliability and validity of these tools (Aas, 2011). Scoring biases and lack of sensitivity for subdomains have been addressed by updated assessments, such as the Personal and Social Performance scale (PSP; (Morosini et al., 2000)), which uses more formally operationalised scores in four discrete domains of social and occupational functioning. Interviewer-rated measures of functioning are the most commonly used functional assessments in clinical studies within the last 5 years, with the PSP and GAF being predominantly used.

Interview-based assessments may suffer from patients being unable or unwilling to provide an accurate account of their everyday activities and behaviors (Harvey et al., 2011), further complicated by exacerbations in clinical symptoms, cognitive difficulties (e.g. memory problems), socio-cultural pressures or lack of insight (Bowie et al., 2007; Gould et al., 2013; Harvey and Bellack, 2009; Sabbag et al., 2012). However, experienced clinicians with frequent contact and knowledge of the patient can mitigate these effects (Sabbag et al., 2011). Alternatively, a familiar caregiver or caseworker can be interviewed, as in the Specific Level of Functioning (SLOF; (Schneider and Struening, 1983)) scale. Endorsed by the VALERO initiative as a suitable measure to assess ability-relevant real life functioning in the context of cognition-related studies (Harvey et al., 2011; Leifker et al., 2011), this measure assesses behaviors across up to six subscales spanning physical, personal, social and occupational functioning.

Self-reported measures of functioning and quality of life

Some clinician and informant-rated instruments, such as the Social Functioning Scale (SFS; (Birchwood et al., 1990)) and World Health Organization Disability Assessment Schedule 2.0 (WHODAS-II), have been administered as self-report measures in order to assess patients' own experiences of daily functioning. However, self-reported measures of subjective illness burden and disability have been reported to exhibit little correlation with performance-based functional capacity assessments (McKibbin et al., 2004) or clinician-rated scales (Chopra et al., 2004). Objectively rated functional assessments (by a clinician well known to the patient and ideally an additional informant) and patient-reported outcome measures are generally considered to provide distinct, but informative, perspectives on the recovery process.

Patient-reported quality of life is an overlapping, but conceptually distinct, construct from ‘real-world’ functioning. Whilst improvements in quality of life may be difficult to achieve within the short duration of most intervention trials, quality of life measures help clinicians and researchers to understand patients' subjective satisfaction with their social, vocational and residential circumstances. Relevant tools include items or subscales relating to everyday functioning. For example, the Social Adjustment Scale-Self Report (SAS-SR; (Weissman and Bothwell, 1976)) is an instrument administered to patients to assess perceived performance and satisfaction over the past 2 weeks in 6 domains spanning personal, social and occupational aspects. Other scales, such as the Modular System for Quality of Life (MSQoL; (Pukrop et al., 2000, Pukrop et al., 2003)), additionally include self-reported items relating to interpersonal relationships, hobbies and independence. Importantly, generic quality of life measures that are often used in clinical settings, such as the EuroQol-5D (EQ-5D) and Short Form Health Survey (SF-36), have been demonstrated to be heavily weighted on physical symptoms and may not capture the aspects of everyday functioning pertinent to people with schizophrenia (Connell et al., 2014; Papaioannou et al., 2011).

Functional capacity assessments

Instead of measuring function through ‘real-world’ behavior, functional capacity assessments evaluate an individual's ability to complete these activities under controlled conditions. Enhancing functional capacity is thought to be an important intermediary step for enabling behavioral change, and has been shown to mediate the association between cognitive ability and ‘real-world’ functional outcomes (Bowie et al., 2006). This concept has received support from various key opinion leaders, including the MATRICS working group, for use in the context of clinical trials (Green et al., 2008). It can elucidate more precisely where problems are arising and what should be targeted to improve functional outcomes (i.e. behavioral or cognitive change). Despite this, capacity doesn't always translate into outcome, i.e. being able to perform a skill in the lab does not mean patients do perform it in the real world.

Interviewer-rated measures of functional capacity

Clinician-rated measures of functional capacity have recently been developed that do not require the specialist equipment necessary to complete performance-based assessments. The Schizophrenia Cognition Rating Scale (SCoRS) is arguably the most well-known of these and is among the most frequently used co-primary functional endpoints in pro-cognitive pharmacotherapy trials (Keefe et al., 2006, Keefe et al., 2015). The cognitive domains assessed by the SCoRS correspond with the seven cognitive domains identified by the MATRICS initiative as being associated with functional outcomes in patients with schizophrenia (Nuechterlein et al., 2008). A global rating is assigned in addition to the 20 individual items that evaluate cognitive deficits and the degree to which these deficits impair the patient's ability to manage cognitively demanding, functionally relevant, everyday tasks such as conversations, watching television, and using electronic devices (Keefe et al., 2015, Keefe et al., 2006).

The SCoRS assessment collects information from the patient (interview), an ‘informant’ (interview with family member, social worker, or friend who has regular contact with the patient in everyday situations) and the clinician (a rating based on the clinical judgement of the clinician who administered the scale to the patient and informant). Ratings generated by clinicians and informants seem to have similar properties, though ratings provided by patients appear to be less reliable (Harvey et al., 2019). However, informants can be biased and it has been suggested that up to 25% of patients with schizophrenia do not have a suitable informant (Harvey et al., 2019). The SCoRS has been reported to have good psychometric properties including test re-test and inter-rater reliability, as well as convergence with cognitive performance and real-world functioning (Harvey et al., 2019).

Performance-based functional capacity assessment

Performance-based functional capacity assessments require participants to complete various everyday tasks in simulated situations. The most widely used measure is the University of California, San Diego Performance-Based Skills Assessment (UPSA; (Patterson et al., 2001)), a pen-and-paper-based role-play test in which patients are asked to use props to demonstrate how well they can perform a number of routine activities, e.g. creating a shopping list, using public transport, or paying bills. The UPSA evaluates up to six domains of daily functioning: (1) household management, (2) communication, (3) financial skills, (4) transportation, (5) comprehension/planning, and (6) medication management (Patterson et al., 2001). UPSA administration requires approximately 30 min, though abbreviated versions of this assessment, including a selection of subscales, such as the UPSA-Brief (Mausbach et al., 2007, Mausbach et al., 2010) and UPSA-2 (Bowie et al., 2006), are available.

The UPSA has received support from the MATRICS initiative (Green et al., 2011) and exhibits moderate to strong correlations with both neurocognitive ability and ‘real-world’ functional activities (Becattini-Oliveira et al., 2018). The measures exhibit good test-retest reliability and are well tolerated among patients with schizophrenia (Becattini-Oliveira et al., 2018). The task requires specialist resources and training to administer and score, whilst a lack of alternate forms can result in large practice effects (Gold et al., 2018).

Computerised performance-based measures of functional capacity

To overcome the practical limitations associated with most traditional performance-based measures, computerised performance-based functional capacity assessments have been developed, including an iPad-based version of the UPSA (Moore et al., 2015). These computer or smartphone-based gamified assessments replicate ‘real-world’ activities (e.g. recall of a shopping list, managing money, etc.). Compared to pen-and-paper tests, these reduce subjectivity (standardized audio instructions, automatic scoring), improve practicality (no need for props), improve access (potential to be delivered remotely), are easily adaptable, and might more accurately reflect how tasks are performed in today's technology driven world (Czaja et al., 2017). Different versions of these gamified assessments are currently under development and seem to be accepted by patients. These simulations show convergence with standard measures of functional capacity, providing valid assessments whilst overcoming many of the constraints (e.g. need for trained rater, travel to administration location) faced by more traditional assessments. Additionally, some digital measures of functional capacity have shown sensitivity to age (e.g. Atkins et al., 2015), making them good candidates to assess functioning on an empirically driven, transdiagnostic basis in line with the U.S. National Institute of Mental Health initiated Research Domain Criteria (rDOC) project.

A high-profile example of recent developments in this area includes the Virtual Reality Functional Capacity Assessment Tool (VRFCAT), a computer-based test that simulates instrumental activities of daily living in a realistic and interactive virtual environment (Lindenmayer et al., 2020). Specifically developed for clinical trial use (Keefe et al., 2016), the VRFCAT is now implemented to detect functionally meaningful improvements in clinical trials (Nahum et al., 2020) and has been deployed in four clinical trials that started within the last year. It has demonstrated sensitivity to relevant deficits in patients with schizophrenia (Keefe et al., 2016; Ventura et al., 2020) related to four different functional abilities: (1) checking for the availability of items to complete a recipe, (2) taking a bus, (3) shopping in a store, and (4) managing currency. Six alternate forms of each scenario limit practice effects, and can be adapted as necessary (Keefe et al., 2016).

Emerging technologies for real-time remote assessment of functioning

Introduction

Major advances in digital capabilities over the last decade offer new clinical opportunities to acquire active and passive information on various aspects of patients' daily functioning (Cook et al., 2019). A range of physiological sensors have now been integrated into smartphones, smartwatches and other wearable devices, making digital phenotyping and enhanced patient monitoring a possibility. The use of high-frequency functional monitoring using digital devices brings its own scientific, ethical and practical challenges. Yet, these technologies provide a unique opportunity to acquire ‘real-world’ functional data with the potential to demonstrate the effectiveness of treatments for unmet clinical needs in schizophrenia. Importantly, many of these technologies are widely available and used throughout the world, including in many low and middle-income countries, offering opportunities to widen access and enhance clinical research and treatment globally.

Here, we will introduce two emerging types of functional measures, active ecological momentary assessments (EMAs), and passive sensory-driven data collection. We discuss examples of how these assessments are currently being used, before highlighting the most important challenges pertaining to digital assessments of functioning. An overview of the different types of functional assessments discussed in this article is shown in Fig. 1.

Fig. 1

Categories of measures used to assess functioning in intervention studies with patients with schizophrenia. Examples of each category are given (Hall, 1995; Morosini et al., 2000; Niendam et al., 2006; Auther et al., 2006; Hodgekins et al., 2015; Birchwood et al., 1990; Chopra et al., 2004; Herdman et al., 2011; Keefe et al., 2006, Keefe et al., 2015; Patterson et al., 2001; Mausbach et al., 2007; Moore et al., 2015; Atkins et al., 2015; Ruse et al., 2014; Granholm et al., 2020; Jagesar et al., 2021b).

Active assessments

Delivery of short self-report measures or tasks onto patients' smart phones or wearable devices has become a more common feature of clinical trials. Popularity has further increased in response to the COVID-19 pandemic, due to the remote nature of these techniques. Ecological momentary assessments (EMAs) offer the chance to capture patients' functional status right in the moment, limiting confounds arising due to memory biases and problems that may impact ratings typically provided during single ‘snapshot’ assessments at the start and end of a trial. The ease with which these assessments can be carried out facilitates high frequency testing designs, which enable detailed pictures of change in function over time and in relation to treatment, all from data captured remotely. These approaches not only create more data, but they also reduce patient burden and trial costs, both of which are important goals for those designing and conducting clinical trials. The feasibility, reliability and validity of different EMAs is being shown in order to establish these more novel measures, often alongside traditional endpoints (e.g. Granholm et al. (2019), Harvey et al. (2021)). Choosing the right momentary assessment is key for high ecological validity and convergence with conventional measures, as highlighted by a recent systematic review (Wright et al., 2021). Brief daily patient and/or informant assessments using targeted stand-alone questions or selected items from e.g., the SCoRS, can quantify the functional impact of cognitive dysfunction in real-time.

Passive assessments

Passive assessment of real-world behaviors, such as leaving the house (via location tracking; (Depp et al., 2019)), physical activity (through monitoring heart rate, step count or accelerometry; (Deenik et al., 2017)), and sociability (based on ingoing/outgoing messages) may provide an indication of subtle functional changes associated with treatment of negative and cognitive symptoms. One application developed to collect such data whilst continuously running in the background of a smartphone is the passive monitoring platform BEHAPP (https://behapp.org). It is used in formal research studies and collects data, which can be related to communicative or exploration acts and has successfully been deployed in studies examining schizophrenia (Bilderbeck et al., 2019; Jagesar et al., 2021a).

High-frequency testing and simultaneous data acquisition

High-frequency measures can highlight daily fluctuations in behavior, measuring both ‘good’ and ‘bad’ days or even particularly challenging parts of a day. Feasibility, reliability and validity of frequent daily EMAs is being established, for example the completion of social and functioning self-report measures 7 times a day for 7 days by Granholm et al. (2019). Such frequent information provides a more granular picture of the ups and down patients experience and might show patterns that would otherwise remain hidden and might provide treatment targets. Adding active anchors to passive assessments can give meaningful, unique insights into a patient's true behavior. A recent study for example found associations between self-reported mood and passively collected data from geolocation relating to patients' routines (Henson et al., 2020).

Discussion of emerging technologies

Studies have reported a largely positive view among people with psychosis on the potential use of digital health technologies to investigate, manage and treat their mental health problems (Birnbaum et al., 2018; Bucci et al., 2018; Gay et al., 2016). Many hope that digital phenotyping and monitoring will eventually have the potential to make the process of drug development cheaper, more efficient, and safer through the use of more detailed and frequent patient assessments (Smith, 2018).

The digitisation of tools for the assessment of functioning in schizophrenia, including cognition, provides a powerful framework to enhance many aspects of such assessments. This includes the ability to capture data related to real world activities directly in a patient's environment in a timely manner, improved patient access and reduction in travel to sites, and electronic storage and management of data. But with great power comes great responsibility. On the one hand, the use of digital technologies to collect patient data can make a tremendous contribution towards furthering understanding of important mechanisms of functioning in patients with schizophrenia. On the other hand, patients with schizophrenia are a vulnerable group and technology used to measures their everyday functioning needs to be developed to high scientific, technical and ethical standards. Most of these emerging technologies are not classed as medical devices and are hence operating outside a strict regulatory framework. To foster consensus and provide a standard, the FDA has recently published a draft guidance entitled ‘Digital Health Technologies for Remote Data Acquisition in Clinical Investigations’ (FDA, 2021). The document outlines recommendations intended to facilitate the meaningful use of digital health technologies in clinical investigation and gives a good overview of the current challenges relating to design, testing, deployment, data safety and analysis of digital health technologies. Here, we would like to highlight some of the most pertinent issues regarding the use of digital technology in patients with schizophrenia in order to encourage the reader to further explore this topic.

It is of uttermost importance that digital measures of functioning are ‘fit-for-purpose’, i.e., developed to high scientific standards (verification, validation) and usable by patients with schizophrenia. Verification and analytical validity have been obtained for many sensory-driven measures (e.g. actigraphy, HRV), but establishing clinical validity in patients with schizophrenia is a more complex process. It is difficult to establish the ‘true’ measure of a novel biomarker if a gold standard for detailed comparison is lacking, i.e. when actively assessing fluctuations in cognitive function at high-frequency and only being able to compare to less frequently employed, more standardized test batteries. This needs to be carefully thought about though, as these are the areas where novel digital endpoints could bring the biggest gains. Digital endpoints, both active and sensory-driven, have been used in ‘validation-type’ studies to build a picture of their utility and comparability in relation to more established measures. A number of smartphone and wearable-based active and passive digital assessment studies have been conducted in patients with schizophrenia, including several that simultaneously evaluated various elements of functioning, symptomatology or involved cognitive assessments (e.g. Ben-Zeev et al., 2017; Liu et al., 2019; Moran et al., 2018; Mulligan et al., 2016). First results from such, predominantly small, trials are promising (e.g. Harvey et al., 2021). Encouraged by meaningful results and good levels of adherence and engagement, larger-scale studies using a variety of active and passive measures related to functioning have been launched (e.g. Begemann et al., 2020; Bilderbeck et al., 2019; Rasmussen et al., 2021) and will help to get a better sense of clinical validity of such measures.

A patient-centric approach will help to ensure new tasks are fit-for purpose. Ensuring low patient burden, simplicity, and ease of use should be a priority, given the propensity for distractibility and the occurrence of positive symptoms, which may conflict with independently-administered highly gamified or elaborate apps and tasks. Delivery of measures needs to be carefully planned and issues like inclusivity, provision of technology, technical literacy, appropriate user support and continued patient engagement need to be considered. Whilst technology can enable decentralized trials, increasing accessibility for many, new factors, e.g. smartphone ownership (ranging from 10 to 83% among those with schizophrenia (Wright et al., 2021)) might emerge as new potential barriers. These can be mitigated by providing well-designed and easy to use study devices, providing technical support and keeping patient burden low (short assessments, integration of technology for easy use, appropriate timing of assessments, customization to each participant and provision of targeted reminders to facilitate compliance). Assessments must be developed and employed in a target-population specific manner, taking into account additional external factors like language use and necessary cultural adaptation.

Digital biomarkers need to be reliable, meaningful and confidential and both clinical risks and privacy-related risks need to be identified and minimized. Lastly, data storage, analysis and interpretation need to be safe and meaningful. Machine learning algorithms, which are often needed to translate the vast amount of real-time data into functional outcomes, need to be validated with the respective data in patients with schizophrenia and once a valid link has been established, changes to data collection or algorithm updates will need to be closely monitored in order to ensure comparability of outcomes across time.

Conclusion

Despite the recognised importance of functional recovery by patients, carers, clinicians and policy makers, long-term functional impairment remains a common, debilitating and costly aspect of schizophrenia with an unmet need for effective therapeutic interventions. A lack of consensus on the optimal functional measure(s) to include in clinical trials, and the various practical and psychometric issues associated with existing tools complicates the use of meaningful functional outcome assessments.

Digital tools (smartphones, wearables, personal computers) enable cost-effective, high-frequency capture of rich ‘real-world’ data, and offer solutions to the practical and scientific limitations among traditional assessments including the potential inability to capture the subtle changes in everyday behaviors that may be reasonably expected to occur during a brief clinical trial. Such measures, when considerately designed, could increase trial precision and sensitivity, and therewith help to address the unmet patient need for successful intervention to improve cognition and daily functioning.

Declaration of competing interest

AS, LA and JHB are employees of Cambridge Cognition. ML and JC were formerly employees of Cambridge Cognition (where this work was conducted). JC is currently an employee of Reckitt.