Feasibility and validity of computerized ecological momentary assessment in schizophrenia.

BACKGROUND: Computerized Ecological Momentary Assessment (EMAc) techniques permit the assessment of daily life behaviors and experiences. The present investigation examined the feasibility and validity of this assessment methodology in outpatients with schizophrenia.
METHODS: Outpatients with schizophrenia or schizoaffective disorder (n = 54) received a battery of standard laboratory clinical and functional outcome measures and then completed electronic questionnaires on a personal digital assistant (PDA) microcomputer 4 times per day for 1 week.
RESULTS: Generally good compliance (87%) with EMAc was found, and participants rated their experience with the study positively. The data collected in daily life demonstrated expected patterns across the assessment week and were significantly associated with scores from standard laboratory instruments measuring similar constructs.
CONCLUSIONS: EMAc is a feasible and valid approach to data collection in community-dwelling people with schizophrenia, and it may provide important information that is inaccessible via standard clinical and functional outcome measures administered in the laboratory.

Introduction

Self-report, proxy report, and clinician-administered measures are the most common assessments for people with severe mental illness. These measures, however, may be heavily influenced by retrospective recall errors and state-dependent biases,1,2 particularly for disorders like schizophrenia that are often characterized by cognitive deficits and poor insight. Performance-based skills assessments constitute a useful supplement to standard clinical measures because they provide direct information about basic functional skill capacities of the individual. However, a limitation of performance tests is that they are conducted within a single laboratory or hospital setting and therefore may not accurately describe the actual daily life functioning or experiences of the participants in the real world. For these reasons, the ecological validity of data gathered with existing laboratory measures remains uncertain, in particular, concerning functioning in natural contexts.

Ambulatory or in vivo monitoring techniques increase the ecological validity of assessment data and overcome many issues associated with retrospective reporting. Some of these approaches, such as Ecological Momentary Assessment (EMA) or the Experience Sampling Method, share many of the advantages of direct observation but are less labor intensive.3 EMA provides real-time, real-world monitoring of behavior, experiences, and symptoms through the use of electronic devices that prompt participants to provide information to the investigator several times each day. EMA has been applied with success in the study of a wide variety of psychiatric conditions, including personality disorders, anxiety and depression, psychosis, and substance abuse.4–14 However, one concern with EMA is that severe psychiatric populations may be unable or unwilling to comply with intensive assessments in daily life. The majority of prior studies using EMA in different populations have used paper-based assessments, which have questionable validity because participants may not respond to questionnaires at the appropriate times or may complete them in mass the night before returning assessment booklets to the investigator.3,15

Computerized EMA (or “EMAc”) avoids many of these concerns by offering greater ease of use through the automatic administration of questions and by permitting the exact time of each response to be electronically verified. The use of handheld microcomputers also increases the flexibility of assessment (eg, by allowing supplemental questions branching to specific responses), simplifies data management, and improves confidentiality of participant responses.3,16,17 Although most investigations of schizophrenia or psychotic symptoms have used paper-based measures,5,6,8,14 a recent study found encouraging support for the use of EMAc in this population.17 However, this study was based on a small sample (N = 10) who participated within a structured inpatient hospital setting, and participants were asked to complete computerized assessments for only 1 day. Given the nature of severe psychiatric disorders such as schizophrenia, the overall feasibility of EMAc techniques over longer durations remains unclear, and in particular concerning the quality and validity of data relative to daily life functioning outside of the hospital.

The present investigation examined the feasibility and validity of EMAc in the assessment of outpatients with schizophrenia. Feasibility was assessed by examining compliance with EMAc over a 1-week period, as well as through subjective participant ratings of the difficulty, time burden, and overall acceptability of the methodology. Validity was assessed by examining the presence of expected patterns among variables assessed using EMAc (eg, associations between daily stressors and mood states) as well as by the concordance between EMAc and standard laboratory measures of the same construct.

METHODS

Participants

This study was approved by the Institutional Review Board for the University of California, San Diego. Participants were recruited from a larger psychosocial treatment outcome study that required participants be physically and clinically stable enough to participate in outpatient group therapy and not have received cognitive-behavioral therapy in the past 5 years. Fifty-six community-dwelling people with schizophrenia (n = 44) or schizoaffective disorder (n = 12) were invited to participate. The Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, was used to verify diagnoses. Two participants refused, one due to paranoid ideas concerning the study and one due to fears of losing the study material. The 54 participants who enrolled had a mean age of 44.06 years (SD = 10.46), reported a mean of 12.11 years of education (SD = 1.05), and were 63% male, 57% white, 13% African American, and 15% Hispanic. The mean length of illness reported by participants was 23.83 years (SD = 11.45, range = 1.8–54 years), and 59.3% reported 5 or more psychiatric hospitalizations in the past. The sample experienced moderate symptom severity (Positive and Negative Syndrome Scale or PANSS18 total, M = 61.89, SD = 15.23; positive symptom score, M = 17.65, SD = 6.33; negative symptom score, M = 13.67, SD = 5.17), and 70% of participants resided in assisted living facilities (“board and care”). The vast majority of participants reported no prior experience with personal digital assistants (PDAs) (92.6%).

Procedures

A battery of laboratory-based self-report and interview measures was administered according to standardized procedures to assess mood, symptoms, and functioning in the weeks prior to the day of testing. EMAc methods were then used to assess these domains during the week after the laboratory questionnaires and instruments were administered. The measures completed in the laboratory, therefore, referred to the weeks prior to the day of testing, while the real-time EMAc assessment was done during the week after the laboratory assessments. EMAc sampling was not conducted during the week prior to laboratory assessments (ie, for the same reference period as the laboratory measures) to avoid any potential impact of EMAc sampling (eg, heightened awareness of symptoms and behaviors) on the laboratory measures, which could artificially increase the association between the 2 assessment methods or otherwise compromise the typical standardized administration of the laboratory measures.

The laboratory assessments included measures of psychotic symptoms (PANSS),18 anxiety (Beck Anxiety Inventory, BAI),19 depression (Beck Depression Inventory, BDI-II),20 and self-reported functioning (Independent Living Skills Survey, ILSS).21 Participants also completed a comprehensive battery of tests from the Measurement and Treatment Research to Improve Cognition in Schizophrenia Consensus Cognitive Battery22 including speed of processing (Category Fluency, Symbol Coding, and Trail Making A),22 working memory (Letter-Number Span and Wechsler Memory Scale-III spatial span),22–24 and verbal (Hopkins Verbal Learning Test-Revised)22,25 and nonverbal (Brief Visuospatial Memory Test-Revised)22,26 learning and memory. All raw scores were converted to demographically corrected T-scores, and a mean global impairment T-score was computed. Following these assessments, a 30- to 45-minute training session was provided on how to operate the PDA (Palm Zire 31), the meaning of all questions and response choices, and procedures for carrying the device and responding to alarms. Two practice EMAc questionnaires were completed in the laboratory, the second in response to an alarm signal. Direct examiner observations of the practice assessments in the laboratory confirmed that 93% of participants provided accurate descriptions of their physical location, 85% provided accurate descriptions of their social company, and 97% provided accurate descriptions of their principal activity. Individuals demonstrating greater difficulty in understanding questions or operating the device were provided additional training.

Participants were then given PDAs to carry with them for 7 days, with each device being programmed to administer 4 electronic interviews per day. The assessment times were fixed for each participant and randomized across participants. The signals occurred within each of the following time periods: 9:00 AM to 12:00 noon, 12:00 noon to 3:00 PM, 3:00 PM to 6:00 PM, and 6:00 PM to 9:00 PM. The sampling windows were adjusted to accommodate each participant's typical sleep and wake schedules, and they were also given the option of silencing alarms for 30-minute intervals (eg, during church, driving, naps). This strategy allowed for individuals to participate without modifying their usual daily schedule, while also permitting a maximum of waking hours to be assessed. Finally, participants were given the option of using a belt pack or clip to facilitate carrying the device. All information about sampling procedures, battery charging, pager number to call with questions, etc, were provided to participants in writing. All participants were contacted once by telephone on the third day of sampling to conduct any problem solving and remind participants to charge the PDA. Participants received $35 for completing the weeklong EMAc assessments. Data entries across the week were time-stamped, and the PDA program enabled responses to be provided only within a 15-minute period following the signal.

EMAc Questionnaire

The PDAs were programmed using a modified version of the Purdue Momentary Assessment Tool version 2.1.2.27 The EMAc questions used visual analog scales and box-checking formats to collect all responses. Initial pilot work demonstrated that this population had difficulty remembering options between screens and often did not scroll to the next screen without prompting. As a result, the number of possible responses was limited to the number that would fit on one screen (about 5) to avoid participants needing to scroll between screens. While some questions sampled immediate experience (eg, current location, context, activities, and mood state), others sampled experiences since the previous assessment (eg, social interactions, stressful events, hallucinations, and delusions). This strategy increased the probability of assessing events of brief duration by allowing retrospective reporting over periods limited to 3 hours. The assessment domains and specific items in the EMAc questionnaire were consistent with general outcome domains commonly assessed in psychiatry research, such as daily functioning, symptom severity, and mood (see table 1 for sample questions).

Table 1.

Sample EMAc Questions

Functional Behavior and Environment

Specific activities, social contexts, and physical environments were assessed through EMAc by 5 broad categories of current location, 5 broad categories of current behavior, and number of social contacts since the last electronic questionnaire (see table 1).

Psychotic Symptoms

Psychotic symptoms were assessed by 6 yes-or-no questions (see table 1) concerning specific thoughts or experiences during the time since the last electronic interview. Five of these questions assessed delusions relative to being spied on, mind reading, thought insertion, thought broadcasting, and having special powers. An additional question assessed the experience of visual or auditory hallucinations.

State Moods and Perceived Stress

Sad, anxious, and happy mood states were assessed by separate 7-point Likert scale that asked participants to evaluate their mood at that moment by tapping on an arrow bar (see table 1). The impact of daily stressors was assessed by asking participants to rate the event that had the greatest impact on them since the last electronic questionnaire on a 7-point Likert scale ranging from 1 (very positive) to 7 (very negative).

Overview of Analyses

Repeated-measures data were analyzed using the Hierarchical Linear and Nonlinear Modeling Program 5.04.28 Expected patterns among continuous EMAc variables (eg, stress ratings and positive or negative moods) were examined using means-as-outcomes models. To examine whether scores from standard clinical instruments (PANSS, BDI-II, BAI, and ILSS) were associated with corresponding variables in daily life, means-as-outcomes models for continuous EMAc measures (eg, depressed mood) and Bernoulli models for dichotomous EMAc outcomes (eg, psychotic symptoms) were used. These models are analogous to linear and logistic regressions, respectively, but are adapted to repeated-measures data because they take into account the dependency among the multiple observations generated by each participant. The coefficient γ10 represents the average within-person association of 2 EMAc repeated-measures variables (eg, depressed mood and daily stress), whereas γ01 represents the within-person association of one EMAc repeated-measures variable and a given clinical score per individual (eg, depressed mood and a BDI score).

Results

Feasibility and Acceptability

With one exception, all PDAs were returned by participants in working order (one PDA was lost, but the participant successfully completed the EMAc assessments at a later date). Seven participants (13%) were noncompliant with the EMAc procedures (demonstrating no more than 4 completed assessments or the equivalent of one full day of participation). These individuals did not differ from the overall sample with regard to age, sex, or symptom severity using the PANSS total or positive symptom scores. However, noncompliant individuals demonstrated greater overall cognitive impairment (M global impairment T-score = 25.54, SD = 7.19) than compliant participants (M global impairment T-score = 34.22, SD = 7.49), t(46) = −2.67, P < .05. Compliance among the remaining 47 participants (87% of the original sample) was variable but generally high: An average of over two thirds of the 1218 possible electronic interviews were successfully completed across daily life contexts (mean response rate = 69%, SD = 22%). The extent of EMAc missing data was unrelated to age, sex, PANSS total, PANSS positive subscale total, or PANSS negative subscale total. Missing data were also not associated with number of days in the study (γ01 = −0.031, P > .05), indicating that potential fatigue effects were likely to be minor or that they did not directly influence participation rate. The average length of EMAc electronic interviews was 3 minutes 35 seconds (SD = 2 minutes, 22 seconds).

In addition to these objective measures of compliance, feasibility was also examined by asking participants to rate their experience with EMAc. As shown in table 2, EMAc was experienced positively by the sample as a whole. Most notably, the average ratings by participants indicated few difficulties with the EMAc methodology and a high willingness to participate in similar studies in the future.

Table 2.

Participant Ratings of EMAc Difficulty and Acceptability (n = 47)

Itema	M	SD
I had difficulties understanding the questions	1.37	0.77
I had difficulties typing my responses	1.83	1.18
I had difficulties operating the PDA	1.37	0.73
The PDA was comfortable to carry	3.57	1.56
The beeps interfered with my activities	2.06	1.33
Overall, this experience was pleasant	3.74	1.40
Overall, this experience was challenging	3.14	1.54
Overall, this experience was stressful	2.23	1.40
I would be interested to participate in similar studies in the future	4.34	1.16
I would recommend to others to participate in a similar study	4.03	1.29

Note: EMAc, Computerized Ecological Momentary Assessment; PDA, personal digital assistant.

All responses were ratings from 1 “not at all” to 5 “very much.”

Validity

The presence of expected patterns among responses were examined for EMAc variables that were expected to covary in a predictable manner. Increases in the perceived severity of environmental stress were positively associated with feeling anxious (γ10 = 0.19, P < .001) and sad (γ10 = 0.24, P < .001) and negatively associated with feeling happy (γ10 = −0.18, P < .01).

EMAc data were then examined relative to similar constructs assessed by 4 standard laboratory interviews and self-report measures (ILSS, PANSS, BDI-II, BAI; see table 3). For the majority of EMAc questionnaires, participants reported being home, alone, and inactive, with infrequent reports of adaptive functioning activities (eg, self-care, shopping, chores, work, school, or active leisure). Higher ILSS total scores (indicating higher functioning) were associated with spending less time in one's own home (γ01 = −4.541, P < .001), as well as with less time spent alone (γ01= −1.914, P < .05). The co-occurrence of these 2 EMAc variables (being alone in one's own home) was also decreased in individuals having higher total ILSS scores (γ01 = −3.367, P < .001). Concerning adaptive behaviors, participants with higher ILSS total scores were more likely to be at work or school (γ01 = 4.739, P < .05), visiting the homes of friends or family (γ01 = 4.025, P < .01), shopping or doing chores outside the house (γ01 = 3.739, P < .05), and having interactions with strangers and people other than family or friends (γ01 = 4.280, P < .05). Subscales of the ILSS measuring specific functioning domains were also associated with the frequency of behaviors or environmental contexts in expected directions. Those with greater transportation skills were more likely to be outside their own home (γ01 = −1.489, P < .05) and more likely to be at the home of friends or family (γ01 = 2.469, P < .01). Similarly, those with higher ILSS scores for taking care of one's appearance and cooking predicted the frequency of specific self-care behaviors such as hygiene, dressing, and eating in daily life (γ01 = 1.614, P < .01). The BDI-II and BAI totals were significant predictors of depressed mood (γ01 = 0.03, P < .01) and anxious mood severity (γ01 = .04, P < .05), respectively. Finally, the PANSS positive symptom subscale predicted all forms of psychotic symptoms experienced in daily life, including ideas of being spied on (γ01 = .14, P < .001), mind reading (γ01 = 0.26, P < .001), thought insertion (γ01 = 0.19, P < .001), thought broadcasting (γ01 = 0.13, P < .01), having special powers (γ01 = 0–0.15, P < .001), and visual or auditory hallucinations (γ01 = 0.17, P < .001).

Table 3.

EMAc Variable Frequency and Corresponding Laboratory Assessment Scores (n = 47)

EMAc Assessment	M	SD	Laboratory Assessment	M	SD
Mood and stress
Sad mood	2.61	0.96	BDI-II total	15.38	10.64
Anxious mood	2.84	1.15	BAI total	13.60	10.35
Happy mood	4.59	0.98
Stress severity	2.87	0.96
Psychotic symptoms
Visual or auditory hallucinations	28.34%	33.83	PANSS positive total score	17.60	6.45
Delusions:
Being spied on	16.04%	23.95
Thought reading	19.60%	29.87
Thought broadcasting	14.19%	24.24
Thought insertion	16.98%	27.72
Special powers	11.98%	23.44
Environment and Functioning
Environment
At home	70.30%	20.00	ILSS global score	0.72	0.10
At home of friends/family	8.49%	12.04
At work or school	4.31%	5.91
Other location (inside)	8.36%	8.73
Other location (outside)	8.54%	10.92
Social context
Alone	48.36%	20.06
Family or friends	35.59%	21.17
Coworker, colleagues	5.38%	6.98
Stranger	4.00%	6.80
Other person	8.08%	11.80
Activity
Inactivity	52.48%	26.65
Self-care	10.55%	10.34
Shopping, chores	6.70%	8.67
Work/school, active leisure	10.42%	9.63
Other activity	21.48%	24.12

Note: EMAc frequencies are percentage of each participant's completed questionnaires. EMAc, Computerized Ecological Momentary Assessment; BDI-II, Beck Depression Inventory-II; BAI, Beck Anxiety Inventory; PANSS, Positive and Negative Syndrome Scale (positive symptom subscale total); ILSS, Independent Living Skills Survey.

Discussion

EMAc permits the monitoring of behaviors, moods, and cognitions in natural contexts and therefore provides important information that is largely inaccessible to standard laboratory self-report and interview measures.3,6,16 This powerful, ecologically valid measurement technology has rarely been used in investigations of severe psychiatric disorders. The purpose of the present study was to examine the feasibility and validity of EMAc in outpatients with schizophrenia. With regard to overall compliance, 2 people (less than 5%) declined participation, and only 13% of those who agreed to participate were noncompliant with the EMAc procedures. It is notable that 87% completed an average of over two thirds of the 28 programmed electronic questionnaires. These rates indicate that people with schizophrenia are willing to participate in EMAc protocols and complete enough assessments to examine both within- and across-day variation, while the compliance rate found in this study was somewhat lower than previous EMAc studies of nonpsychiatric (90%–96%)29–31 and higher functioning psychiatric samples (86%–92%).32,33 The number of completed questionnaires remained informative and adequate for data analysis. Moreover, no salient fatigue effects were observed over to course of the study, and feedback from the participants themselves indicated that EMAc was highly acceptable. Taken together, the compliance and acceptability findings suggest that EMAc methods are highly feasible for severe mental illnesses, such as schizophrenia. Several safeguards were nonetheless used in the present study to encourage compliance, such as telephone contacts during the data collection phase and monetary incentives for completing EMAc assessments. The number, type, or format of questions posed to participants may also need to be adapted depending on the clinical characteristics of the samples studied. The use of such procedures, however, should not substantially reduce the feasibility of this novel and powerful approach to assessment in real-world contexts.

The validity of EMAc data was examined in several ways. The relationships among EMAc variables were consistent with expectations (eg, increased stress perception was associated with decreased feeling happy). The presence of such expected patterns in EMAc responses was possible only if participants consistently rated the 2 variables in the opposite direction on Likert scale, a finding that confirms deliberate responding by the participants. Finally, significant moderate to strong associations were found when several laboratory self-report and interview instruments were used to predict variance in corresponding constructs measured by EMAc. In this way, EMAc appeared to measure the target construct (eg, functional behaviors, mood states, and psychotic symptoms) while still providing unique information concerning relationships between daily life contexts, behaviors, and experiences. Applications of data generated from EMAc are widespread, and relatively simple research protocols may provide rich information relative to daily life experiences. Concerning the present data, eg, variables assessed at any given assessment (t) may be used to prospectively predict the onset or changes in psychotic symptoms in subsequent assessments (t + 1). Conversely, the presence of psychotic symptoms can be used to predict behavioral, social, or emotional consequences that occur over subsequent hours. The acquisition of data demonstrating real-time symptom expression provides a novel outcome for controlled clinical trials, and information concerning discrete functional behaviors provides the ecological validity that is currently lacking in this domain. From a clinical point of view, EMAc may also permit treatment advances through verification of medication compliance and the completion of desired exercises or the detection of early warning signs of relapse. The findings were consistent with the positive feasibility findings in the only other study that used EMAc in people with schizophrenia,17 but the present study also demonstrated that EMAc can be used with outpatients over a week of monitoring. The capacity of people with schizophrenia to independently use EMAc in nonsupervised conditions is of key importance for promoting research on functional assessment and the daily life expression of this disorder. These findings should, nevertheless, be interpreted relative to the characteristics of the participants and methodology used in this study. The feasibility of these methods may be decreased in patients with greater cognitive deficits, eg, as was demonstrated in the present sample. In examining the predictive validity of laboratory measures, only major or frequent outcomes and psychological experiences in daily life were examined, such as self-reported living skills, psychotic symptoms, mood, and perceived stress. Less frequent but important outcome variables, such as suicidal ideation or specific clinical events, may require larger sample sizes and possibly different assessment methods. It should also be considered that each EMAc electronic interview was designed to be brief and easy to complete. The content of all clinical laboratory questionnaires (eg, all ILSS items) used in the concurrent validity analysis could not be exactly duplicated in daily assessments due to the necessary brevity of EMAc. More detailed lengthy protocols may, therefore, induce greater fatigue effects or lower compliance rates with different samples. The high compliance rates found were also for a convenience sample that was already participating in a psychotherapy treatment outcome project with the researchers. Consequently, refusal and noncompliance rates may be higher for a more general community sample. Finally, with regard to sample characteristics, the participants were not preselected for having experience with electronic devices of any kind, so it is reasonable to assume that the findings should be generalizable to cohorts of the same or younger age. It is possible, however, that older samples may require different methods. For example, very old individuals (mean age ≥ 80 years) may be unable to use PDAs but can readily use cell phones if such applications involve direct communication with an investigator, rather than use of electronically coded response options.34

It is also important to note that this investigation provided 4 assessments per day and utilized fixed assessments for each participant (randomized across individuals) as part of a research program with specific scientific goals. Increasing the number of assessments per day may be more adapted for other objectives, such as time budget assessments of behavior and experiences in daily life. Concerning random vs fixed assessments, either may be justified concerning the goals of a given study. Should fixed assessments be used, however, investigators may wish to examine if the frequency of variables differs as a function of day of study and adjust statistical models for time-dependent effects that may reflect a participant's increasing ability to anticipate signals over the course of the investigation. Such potential reactive effects, as well as response rates in general, are also more readily assessed through computerized techniques than paper-based measures.15,35 Future investigations may benefit from considering these issues, while testing other EMAc assessments and techniques for consumer self-reporting, including those using cellular technology.

Funding

Office of Rehabilitation Research and Development, Department of Veterans Affairs, National Institute of Mental Health (R01MH071410); UCSD Advanced Center for Interventions and Services Research (NIMH P30MH66248).