Issues and developments related to assessing function in serious mental illness.

Serious mental illness (SMI) results in functional disability that imposes a significant burden on individuals, caregivers, and society. Development of novel treatments is under way in an effort to improve the illness domains of cognitive impairment and negative symptoms and subsequently to improve functional outcomes. The assessment of functional outcomes in SMI faces a number of challenges, including the proliferation of assessment instruments and the differential prioritization of functional goals among stakeholder groups. Functional assessments relying on self- and informant report present a number of limitations. Identifying alternative strategies to assess functioning that are reliable, valid, and sensitive to change is necessary for use in clinical trials. Measures of functional capacity have been proposed for clinical trials investigating compounds to treat cognitive impairment in schizophrenia. Alternative approaches employing effort-based decision making or daily activity recording using instruments such as the Daily Activity Report may be more appropriate for studies focused on improving negative symptoms.

Introduction

A primary goal of treatment development for serious mental illness (SMI) is to reduce the disability commonly associated with this type of psychiatric diagnosis. Disability associated with SMI imposes significant personal and economic burdens across multiple domains of life activity.[1-2] Schizophrenia and schizoaffective disorders are among the leading global causes of disability.[3,4] Including care expenditures and disability, the costs of SMIs are more than $317 billion annually in the United States or more than $1000 per year for every man, woman, and child.[5] More than three-quarters of this amount is associated with loss in productivity.[6,7] Many areas of functioning are affected by SMI, including activities of daily living (ADLs), vocation, finances, social and family relationships, leisure activity, and health care.[8] Even when successful for improving some aspects of illness, such as positive symptoms, treatment with medication can fail to improve or restore functioning.[9-12]

Cognitive impairment and negative symptoms have been identified as strong contributors to functional impairment. There have been multiple recent efforts to develop novel compounds designed to improve cognition and negative symptoms in schizophrenia and the functional disability associated with these illness attributes.[13] With respect to cognition, the US Food and Drug Administration has indicated that improvements in the results of neuropsychological testing would be insufficient for a medication to receive an indication for improving cognitive function in schizophrenia.[14] Improvement in functional outcome has been required as a coprimary measure in such investigations.[14] The requirement of coprimary outcome measures in clinical trials is not unique to schizophrenia. It has long served as the standard in Alzheimer disease research, with varying levels of success. Similarly to schizophrenia, assessment of functioning in Alzheimer disease is complicated by factors such as heterogeneity of the illness and stage of the disease. Assessments appropriate for measuring function in moderate or severe stages of the illness are not able to detect the subtle expressions of the illness in the very early stages. This has become an increasingly important issue in Alzheimer disease research, given a more recent focus on developing early intervention strategies.[15] Coprimary measures incorporating caregiver report to assess functioning in mid- and late-stage Alzheimer disease have been more successful, given the high degree of contact caregivers have with these patients. Even with the limitations of informant reporting, availability of informants for the Alzheimer disease population differs greatly from that of schizophrenia, where informants are often unavailable and tend to have less frequent contact with patients.

Although coprimary measures have not been required for studies of investigational compounds designed to address negative symptoms, these studies have incorporated measures of functional outcome as important secondary outcome measures.[16] Assessing functional outcome in a reliable and valid manner for studies of cognitive impairment and negative symptoms is complicated by a host of problems. We discuss issues that are particularly relevant for such studies and describe several recently developed novel approaches to assessment, which may address some of these issues. Table I provides a list and brief description of various assessment measures referenced in the following sections.

What is functional outcome and when should it change?

A particularly daunting issue with respect to the assessment of functioning in SMI is the breadth of potentially relevant outcome domains. Functional outcomes can refer to independent living skills, including paying bills, basic self-care, taking care of one's living space, social skills, social network size, work and academic role functioning, and the ability to complete goals. To further complicate the enterprise, functioning in each of these areas can be assessed using a variety of methods. Selection of measures can be difficult, particularly in illnesses such as schizophrenia in which clinical presentation and functional impairments can be heterogeneous. In addition, the relationships between functional outcomes and primary measures are variable. These issues are further complicated by a proliferation of available measures purported to assess functional outcomes. As one example of the proliferation of measures of functional outcome, the VALERO study (VALidation of Everyday Real-world Outcomes) asked 48 experts to nominate measures, resulting in 59 potential functional assessments for further examination.[17] In another study that examined social functioning measures, a review of 301 studies identified 80 potentially relevant measures of social functioning.[18] Given the proliferation of measures, it is not hard to understand why some may question the need for novel measures.

In addition to the difficulties in selecting from the vast array of measures, functioning means different things to different stakeholders. Practitioners, individuals with lived experience of SMI, families, and payers may prioritize various types of functional outcome very differently.[19] There is a recent push toward patient-reported outcomes or PROs. Careful thought must guide the development and use of these PROs to ensure that these outcomes are relevant to patients and other stakeholders.

It is also unclear what should be expected of measures of functional outcome in terms of time course for change. Many studies of novel pharmacologic compounds are of short duration, lasting only weeks. In contrast, some changes in real-world functioning may take many months or years. Green[20] has recommended a focus on more proximal or intermediate measures of functioning in short-term clinical trials rather than a focus on real-world functional changes, such as increased number of friends, active participation in dailylife, or work or vocational performance. Examples of intermediate measures include those that assess functional capacity, such as the University of California San Diego (UCSD) Performance-based Skills Assessment (UPSA)[21] or Brief International Functional Capacity Assessment (BIFCA), formerly known as the Functional Assessment Battery (FAB),[22] and those that are interview-based measures of functional impairment associated with impaired cognition, such as the Cognitive Assessment Interview (CAI).[23,24]

Intermediate measures such as those mentioned above are designed to be more sensitive to change in the context of a brief clinical trial, but a number of limitations associated with these measures remain unresolved. Interview measures typically rely on self-report or collateral information from a caregiver informant. Issues about sources of information in ratings are discussed below. Functional capacity assessments that require the performance of specific functional tasks in front of an examiner (eg, calling a doctor's office to schedule an appointment) are informative regarding how well a person can perform a functional behavior in a research setting, but they offer little data regarding what the person is doing in their day-to-day environment. Any information about an individual's day-to-day functioning can only be inferred from these types of intermediate measures.

Who reports about functional outcome?

A large number of functional outcome measures, including intermediate measures such as the CAI, rely primarily on the report of the person being assessed. The person is asked to report on how well they get along with others; how often they engage with others socially; how well they perform independent living skills; the quality of their work; and to what extent problems with attention, memory, or planning interfere with these activities, etc. A number of problems with self-report data have been identified. Difficulties with memory over long periods of time (usually the past week, month, or 3 months) can render retrospective self-report of activities inaccurate.[25-28] This is particularly problematic in an illness such as schizophrenia where individuals score nearly two standard deviations below control subjects on measures of memory.[29,30] In addition, insight is known to be impaired for individuals with SMI. This may impact how the person views their competence at specific tasks, whether the person is aware of cognitive impairment and its consequences in daily life, who they consider to be “friends,” and how they view the quality of their work.[31] A recent study by Harvey et al (VALERO) found that patient ratings on the Specific Levels of Functioning scale (SLOF)[32] were not well correlated with cognitive impairment and other ratings of functional outcome.[33] On the other hand, Harvey et al found that ratings by caregivers and ratings by a trained clinician using all sources of information were correlated with both cognitive test scores and with other functional outcome assessments. These data indicate that for some individuals, self-report is unlikely to produce reliable data.

Using caregivers to provide information on functioning greatly improves the concurrent validity of ratings on the SLOF.[33] It is not clear how much time a person may need to spend with a participant or how much access to the person's residence would be needed to be able to report reliably as a caregiver. In the VALERO study, informants were typically case managers who saw patients at least once per week. The inability to identify an appropriate informant to supply rating information has been reported by as many as one-half to one-third of patients diagnosed with schizophrenia.[34] In clinical trials, requiring the participation of caregivers can often increase the length of time to recruit participants and the cost of the study. In addition, requiring caregiver participation may have substantial consequences for the generalizability of findings, as it would result in the de facto exclusion of an entire participant subgroup unable to identify caregivers. Functional capacity measures resolve the issue of who is reporting by changing the assessment to one based on objective task performance.

In deciding who should report on functional outcome, the type of assessment needs to be taken into account. Patients can be very reliable reporters of specific behavior over brief periods. For example, with respect to reporting on adherence to medication, the correlation between patient-reported adherence over a 2-week period and objective measures such as in-home pill counts or electronic monitoring was close to zero.[35] However, in a separate study, when patients were asked daily—using a smart pill container—to report their adherence, self-reported adherence rates were highly correlated with objective measures (r=0.61; P<0.0001; study using Med-eMonitor devices for electronic monitoring).[36] It seems that individuals with SMI can be good reporters of what they do over short periods, but may do less well at reporting over long periods of time (weeks) and at identifying their level of competence in certain domains.

Several novel measures have been developed to address shortcomings in available instruments. We are particularly interested in discussing these developments as they pertain to studies in which cognitive impairment or negative symptoms are primary outcomes. In the next two sections we will describe assessment development in the area of functional capacity measures. Then, we will go on to discuss assessment development for intermediate and real-world measures designed specifically for trials in which negative symptoms are a primary target.

Does selection of functional outcome measure depend on the primary study target?

When assessing functional outcome, it is important to consider the construct that is being investigated and how this construct may affect functioning. For example, functional capacity measures are intermediate measures that are highly correlated with neuropsychological test scores (r≈0.60).[37] Consequently, it makes sense to use measures of functional capacity for studies in which change in cognition is a primary outcome. However, if negative symptoms are a primary study target, intermediate measures that assess motivation or effort in a laboratory setting or actual engagement in daily functional activities may be more appropriate to include.

The development and refinement of functional capacity measures

The UPSA[21] and its brief version the UPSA-B[38] have served as well accepted and validated measures to assess functional ability in individuals with SMI. The UPSA assesses functional performance across five subscales including household chores, communication, finance, transportation, and recreational activity planning. The full UPSA takes approximately 30 minutes to administer, and UPSA-B, which includes only the finance and communications subscales of the UPSA, takes 10 to 15 minutes to administer. An important potential advantage reported for the UPSA-B is its sensitivity to change in response to a treatment designed to improve functioning.[38] The UPSA is strongly correlated with cognitive functioning, and this relationship has been replicated across multiple countries.[39-43]

The Test of Adaptive Behavior in Schizophrenia (TABS) is a functional capacity measure designed to address limitations of other available measures, including inadequate assessment of the ability to initiate and of the ability to self-identify problems that occur in the course of performing functional activities.[25] For example, capacity tests tend to ask a participant how they would solve a particular problem. However, it is not clear that the individual would have been able to identify the fact that a problem existed. In addition, since capacity tests tend to ask the individual to respond to a contrived situation, there is little room to assess the individual's ability to initiate a response.

The TABS is designed to assess the abilities necessary to perform goal-directed activity, including initiation, planning, problem identification, problem solving, sequencing, appropriate inhibition, and persistence in the context of six functional areas (work and productivity, medication management, independent living, shopping, basic hygiene, and social skills). Items in the TABS demand considerable initiation (eg, spontaneously naming items that would be necessary to stock an empty bathroom), allow the subject the chance to identify specific problems on his or her own before the problems are pointed out by the examiner (eg, identify that he or she was short-changed, identifying that he or she will run out of medication), and provide additional points for spontaneously offering solutions (eg, spontaneously announcing a plan to remedy a problem with running out of medication). TABS scores are calculated as percentages, with higher scores indicating better adaptive behavior. TABS scores were found to be moderately to strongly correlated with other measures of functional outcome, negative symptoms, and neuropsychological test scores (convergent validity). A brief form of the TABS containing three subtests (assessing medication management and work/productivity) takes approximately 15 minutes to administer and score.

As part of the MATRICS initiative (Measurement and Treatment Research to Improve Cognition in Schizophrenia),[44,45] the VIM (Validation of Intermediate Measures) study assessed the reliability, validity, and utility of a number of intermediate measures assessing functional outcome in schizophrenia.[37] Findings indicated that the UPSA, UPSA-B, and TABS were the instruments with the most favorable psychometric properties in a US sample. However, the cultural adaptation of these instruments was not taken into account.

Many efficacy studies of novel compounds are now conducted as multisite international trials. This necessitates that these measures be applicable in culturally distinct locations. Different cultures may influence the degree to which the activities investigated are relevant to individuals and subgroups in different countries. We therefore examined ratings made by experienced researchers at 31 sites in eight countries in order to assess the cross-cultural adaptability of three functional capacity measures including the Independent Living Scales (ILS),[46] UPSA, and TABS. English-speaking research staff familiar with conducting medication trials rated the extent to which each subscale of each intermediate measure could be applied to their culture and to subgroups within their culture on the basis of gender, geographic region, ethnicity, and socioeconomic status. Problems were identified for specific subscales on all the performance-based assessments across multiple countries. India, China, and Mexico presented the greatest challenges in adaptation.[47]

In a follow-up study within the overall MATRICS initiative, the psychometric properties of the BIFCA, comprised of two subtests from the UPSA and one from the TABS,[22] were examined. These subtests were rated by clinical trials experts in the study described above[47] to be the most appropriate functional capacity assessments across different cultural contexts in both Western and non-Western cultures. The reliability and validity of the BIFCA in four sites in India were assessed. Researchers administered the BIFCA, the UPSA-B, the MATRICS Consensus Cognitive Battery (MCCB)[44,45] and used other scales to assess measures of symptomatology and a measure of global functional outcome in 141 individuals with schizophrenia at baseline and four weeks later. The psychometric properties of the BIFCA and UPSA-B were similar in India. Given the previous results that the subtests on the BIFCA are more relevant to non-Western cultures,[47] the BIFCA has been translated into multiple languages for potential use in international studies of novel medications seeking an indication for improving cognition in schizophrenia. It is unclear whether this battery will be adopted as a coprimary outcome in future studies examining cognitive impairment in schizophrenia.

Identifying limitations in the representativeness of role-play scenarios for complex, real-life situations, the Canadian Objective Assessment of Life Skills (COALS) measure was developed to assess functional competence.[48] The developers identified two particular challenges faced by measures of functional capacity. The first involves the sufficient representation and engagement of component processes that make up a functional behavior and the sensitivity to capture it. The second involves the need for new measures to demonstrate incremental validity above neurocognitive tasks previously- shown to predict functional outcome. In the construction of the COALS, McDermid Vaz et al incorporated multiple component processes (eg, working memory, comprehension, and social cognition) required to perform functional tasks into the development of the assessment tasks. An important feature of the COALS development was the collaboration through focus groups including clinicians and peer specialists to ensure that all domains were covered and that constructs were well assessed.[48] The COALS provides a competence measure in two functional domains, including procedural knowledge routines (eg, how to cook a meal) and executive operations (ie, knowledge and flexibility necessary to carry out tasks and successfully overcome challenges that may present during task performance).[48] The COALS consists of scenarios tapping into five domains, including health and hygiene, time management, transportation, crisis management, and domestic activities. The COALS total scores and scale scores demonstrated good concurrent validity with measures of the UPSA and MCCB and good discriminant validity. The COALS was also shown to add incremental validity by accounting for additional variance in the prediction of community independence over neurocognitive measures alone.[48] As a recently developed measure, it has yet to be determined whether the COALS provides a measure of functional capacity that is superior to existing measures. It is also unclear to what extent the COALS could be adapted for use across cultures.

What about studies of compounds designed to improve negative symptoms?

In studies investigating novel compounds to improve negative symptoms, choice of functional outcome is an important issue. Functional capacity, which may have little to do with increasing motivation, effort, and emotional responsiveness, may not be the ideal choice. This is supported by relatively weak correlations among functional capacity measures and negative symptoms.[49] At least two approaches to the development of intermediate or proximal measures for studies of negative symptoms have been applied. One examines laboratory measures of the expenditure of effort as a proxy for motivation, and another attempts to rate the amount or real-world activity engaged in by the research participant in multiple functional domains. It is possible that these types of intermediate measures would be ideally suited for studies in which negative symptoms are the primary clinical target.

Green and colleagues have done some elegant preliminary work on measures of effort-based decision making.[50,51] Basically, effort-based decision making examines how much effort one is willing to expend for a given level of reward. For example, someone is offered a choice between something that requires little effort and results in little reward and something that requires significantly more effort but offers greater reward. By manipulating the difficulty of the tasks and levels of reward, these tasks allow us to infer a person's level of motivation. Negative symptoms are believed to influence this effort-based decision making. Particularly, amotivation is thought to be associated with the willingness to expend less effort in such paradigms and has been associated with poorer functional outcomes. Patients with schizophrenia have in fact been found to choose easier tasks over more difficult tasks in many of these paradigms.[50,51] Reddy and colleagues[52] examined multiple measures of effort in the domains of perceptual effort, physical effort, and cognitive effort in 94 individuals with stable schizophrenia. These tasks were found to have low-to-moderate correlations with negative symptoms, cognitive functioning, and functional outcome measures, and were generally- not related to positive symptoms. Their study provides preliminary support for further investigation of these effort-based measures as intermediate measures in trials investigating functional outcomes associated with negative symptoms.[52]

Another way to assess more proximal functional outcome in studies of compounds designed to address negative symptoms is by assessing changes in activity in functional domains. The Daily Activity Report (DAR) examines real-world functioning in a unique way by obtaining a report from the patient of all activities over a 7-day period.[53] The DAR was developed following a comprehensive review of the literature, as well as from input in focus groups from patients, nonprofessional caregivers, and treatment team members. Patients rated different activities for importance and complexity, forming the basis for the DAR scoring system. The DAR differs from an existing measure developed by Jolley,[54] which examines functional behavior over 7 days in a number of important ways.[53] The DAR is based on frequent contact with the patient (three telephone calls daily for 7 days) rather than a long period (1 week) of retrospective recall. Moreover, the DAR uses a structured interview to rate activities and their complexity in three different domains (domestic, social, educational/ occupational), with scoring based directly on the input of stakeholders. The data present a complete picture of what the person is doing during a week. The data can be summarized in multiple ways and can provide information that would allow a researcher to make ratings on global instruments assessing functional outcome, such as the Personal and Social Performance Scale (PSP).[55] Global scales are typically rated during an interview with the patient, and scores are either based on self-report of activities recalled over months or on the report of significant others.[55,56]

We administered the DAR and additional assessments of functional outcome, functional capacity, cognition, and symptomatology to 50 individuals with schizophrenia at two time points, 1 month apart, and to 25 healthy controls. Early indications are that the DAR is well tolerated by patients, with 88% of those approached consenting to participate, and 94% of those who consented completing both the baseline and follow-up assessment phases. In addition, patients and controls did not differ in the number of calls completed during the 1-week assessment, with a mean number of 17.77 (standard deviation [SD]=3.83) and 18.32 (SD=2.53) calls respectively. Inter-item consistency was high. Test retest reliability across 1 month for the total DAR score was 0.67 (P<0.0001). The total DAR score, as well as scores for social activity and nondomestic work/school, differed significantly between control and patient participants. DAR domain scores were moderately-to-highly associated with negative symptoms and with functional outcomes.[53] Interestingly, the work/school dimension of the DAR explained most of the variance in the DAR's relationship to global functioning. This relationship suggested that global measures of functional outcome might primarily be tapping whether the person works. Work status is unlikely to change in a brief clinical trial and this may be why some of these global measures of functional outcome can be relatively insensitive to change. DAR scores were only weakly and nonsignificantly related to positive symptoms, providing important evidence of discriminant validity. This study provided preliminary support for the reliability and validity of the DAR using interviewer administration.[53] Current work is focused on testing a patient-reported outcome version of the DAR, using SMS texting technology. This version of the DAR automatically scores activity on the basis of patient responses and eliminates the need for interviewers. Such a system, which can capture all of a patient's waking activity without the use of a research assistant, will probably be both highly informative and cost effective.

These two approaches to functional assessment for studies investigating compounds to improve negative symptoms represent important moves forward. However, it is important to point out that neither of these measures examines the emotional responsiveness dimension of negative symptoms. One approach to creating a performance-based measure of emotional responsiveness not reliant on self-report would be to rate affect in response to emotional video clips, or affect during role-play scenarios. In addition, behavioral sampling using smartphone or text technology could be used to capture what a person is feeling at multiple times during the day.[57-59] Such assessments would be particularly valuable if a compound specifically targeted emotional changes.

How can new technologies improve functional assessment?

The use of new technologies provides great opportunity to bridge some of the shortcomings in the assessment of functioning, more generally, and the assessment of real-world functioning, specifically.[57-59] Remote data collection and remote reporting technologies include the use of portable electronic devices, such as tablet computers and smartphones, that provide opportunity for realtime assessment of day-to-day functioning. A major advantage of utilizing this technology is that it can enable the means to collect data on functioning over a period of days or weeks. This offers the potential to greatly improve measurement reliability while reducing resource expenditures in both cost and personnel hours, which have previously impeded the use of comprehensive functional assessments. Applied to clinical trials, this would allow for collection of real-world functional data over the course of the study that could include a well-established baseline before treatment and continuous collection of functional data over the course of treatment. The amount of data collected on functioning may help to significantly enhance sensitivity for detecting change in functioning, which is a particularly important consideration for clinical trials. Exploiting these technologies to good effect requires efforts to develop assessment applications and psychometric evaluation of these applications.

TABLE I.

Referenced functional measures including brief descriptions.

Measure	Description
UCSD Performance-based Skills Assessment (UPSA)	Performance-based measure of functional ability employing 5 tasks serving as real-life functioning analogs.
UCSD Performance-based Skills Assessment - Brief version (UPSA-B)	A brief version of the UPSA, designed to be administered in 10-15 minutes. It includes the communication and finance subscales of the UPSA.
Test of Adaptive Behavior in Schizophrenia (TABS)	Performance-based assessment of functional capacity, including assessment of task initiation and identification of problems with a task.
Brief International Functional Capacity Assessment (BIFCA)	Multicultural validated measure of functional capacity incorporating components from the UPSA and TABS.
Canadian Objective Assessment of Life Skills (COALS)	Performance-based assessment of functional competence based on incorporating component processes required for tasks in domains that use procedural knowledge routines and executive operations.
Specific Levels of Functioning scale (SLOF)	Informant-rated scale assessing function across 6 behavioral domains.
Independent Living Scales (ILS)	Performance-based assessment of functional ability across 5 domains relevant to self-care.
Cognitive Assessment Interview (CAI)	Clinician-rated brief interview of cognitive function.
The Daily Activity Report (DAR)	Daily tracking of activity using multiple daily activity recordings. Designed to enhance self-report accuracy for activity reporting. Functioning is assessed on the basis of activities tracked over a week.
Personal and Social Performance Scale (PSP)	Clinician-rated global assessment of functioning.
UCSD, University of California at San Diego