Routine outcomes monitoring to support improving care for schizophrenia: report from the VA Mental Health QUERI.

In schizophrenia, treatments that improve outcomes have not been reliably disseminated. A major barrier to improving care has been a lack of routinely collected outcomes data that identify patients who are failing to improve or not receiving effective treatments. To support high quality care, the VA Mental Health QUERI used literature review, expert interviews, and a national panel process to increase consensus regarding outcomes monitoring instruments and strategies that support quality improvement. There was very good consensus in the domains of psychotic symptoms, side-effects, drugs and alcohol, depression, caregivers, vocational functioning, and community tenure. There are validated instruments and assessment strategies that are feasible for quality improvement in routine practice.

Introduction

Psychosocial and medication treatments have consistently and repeatedly been found to improve outcomes in schizophrenia (American Psychiatric Association 2004; Kreyenbuhl et al. 2010). However, key evidence-based treatments have not been widely disseminated (Institute of Medicine 2006; Lehman 1999; President’s New Freedom Commission on Mental Health 2003). Increasingly, policy makers and healthcare organizations are interested in improving the quality of care. One leading approach to improving care starts by identifying patients who would benefit from a specific evidence-based treatment, and then ensuring that they have access to that treatment (Rubenstein et al. 2000). While this approach has improved outcomes in chronic disorders such as diabetes (Asch et al. 2004; Kupersmith et al. 2007), it has proven very difficult to implement in mental illness, and especially schizophrenia (Young et al. 2004). A major challenge has been a lack of valid, routinely collected outcomes data to identify patients who are failing to improve clinically and functionally, or who are not receiving appropriate care. In part, the lack of such data reflects uncertainty regarding which outcome domains to monitor and how to feasibly collect routine outcomes data in large populations of patients. Quality improvement efforts would benefit from better consensus regarding methods for routine monitoring of outcomes of schizophrenia.

Most successful efforts to improve healthcare quality have been conducted outside of mental health, and few have occurred in schizophrenia. In diabetes, for example, medical records and electronic data routinely include information needed to evaluate and improve care. These include data on demographics, hemoglobin A1c (HgA1c, a measure of diabetes severity), prescriptions, and procedures received. Since HgA1c and prescription data are routinely collected and recorded electronically, policy makers have been able to identify patients who have poorly controlled diabetes and apply national quality improvement efforts to improve diabetes care. In schizophrenia, there are also measures of illness severity and need for treatment. At present, however, these outcomes are not reliably documented in medical records (Young et al. 2004). As a result, policy makers have had difficulty determining the extent to which quality improvement and additional treatment resources would have value in schizophrenia.

There are multiple barriers to gathering outcome data in schizophrenia. Outcome assessment methods used in research protocols have proved too costly and time-intensive to widely adopt in routine practice. Clinicians can be trained to achieve high levels of reliability in standardized instruments such as the Brief Psychiatric Rating Scale (Ventura et al. 1993). However, substantial staff time is required for administration and for training required to achieve and maintain adequate reliability. Most provider organizations cannot afford to dedicate staff to this assessment role or to intensively train staff to reliably perform assessments.

There are alternative strategies for outcomes monitoring. These include monitoring based on billing or administrative data, assessment by clinicians without intensive training in standardized instruments, or patient self-assessment. Administrative and billing data are relatively easy to obtain for large patient populations, and can be used to track basic demographics and sometimes medication prescriptions. However, these data can be inaccurate or incomplete, and lack detailed clinical content. Alternatively, it is possible to ask clinicians to routinely complete rating instruments as part of treatment visits, without intensive training and reliability checks. Some organizations have, for example, mandated regular assessments using the Global Assessment of Functioning (GAF) Scale. This has met with limited success. Reliability with untrained raters is very poor (Niv et al. 2007b) and many clinicians resist incorporating standardized rating and documentation into routine practice. In contrast, monitoring based on patient self-report and/or self-assessment is relatively easy to implement in clinical settings (Brown et al. 2005; Chinman et al. 2007). While validity varies, it can be quite good with certain domains and instruments (Eisen et al. 2004). In addition, there is inherent value in patient-centered assessment (Niv et al. 2007a).

This manuscript describes a process within the U.S. Department of Veterans Affairs (VA) to (1) identify outcome indicators that can be used for routine monitoring in support of projects to improve the quality of care for schizophrenia, and (2) identify instruments and methods for monitoring these outcome indicators. The VA provides care for more than 100,000 people with schizophrenia and is a national leader in healthcare quality and the use of clinical information systems to improve care (Jha et al. 2003; Kupersmith et al. 2007). In 1998, the VA established a national Quality Enhancement Research Initiative (QUERI) to inform quality improvement and identify effective strategies for implementing research evidence into routine clinical practice across nine disorders (Francis and Perlin 2006; McQueen et al. 2004). The Mental Health QUERI focuses on improving care delivered to individuals with depression and schizophrenia, and is a partnership between leading national researchers, policy makers, and treatment experts. Since prior efforts to develop outcome measurement in schizophrenia (Cuffel et al. 1997; Veterans Health Administration 2003) did not meet the QUERI’s goals, a Schizophrenia Outcomes Workgroup was convened in August 2004. This manuscript presents the methods used and products developed by this Workgroup.

Methods

The first task of the Workgroup was to identify outcome domains that can support implementation of high-priority, evidence-based treatments. Based on a review of national treatment guidelines and the new emphasis on recovery, the Workgroup chose to focus on the following outcomes: psychotic symptoms, medication side-effects, drug and alcohol use, depression, family and caregivers, vocational functioning, community tenure and housing. The domains of negative and cognitive symptoms were acknowledged to also be important, but were not included because national guidelines do not contain evidence-based practices to improve these outcomes. Comorbid medical conditions were also acknowledged as critically important, since patients with schizophrenia are at high risk for a variety of medical problems, including diabetes mellitus, cardiovascular disease, hypertension, chronic obstructive pulmonary disease, and infectious diseases (American Psychiatric Association 2004). However, development of consensus in this area would require review of a very large body of guidelines from outside of specialty mental health, and was beyond the scope of this Workgroup.

For each domain, the Workgroup sought to identify indicators of: (a) need for treatment, and (b) change in clinical status associated with treatment. Potential assessment strategies, including patient self-assessment, rater or clinician-based assessment, and use of administrative or billing data, were reviewed. It was agreed that feasibility of routine assessment and documentation was critical, and therefore this manuscript focuses on self-report and administrative data approaches. The Workgroup recognized that to improve care, it is also necessary to know whether patients who need treatment have received it. This can be assessed using data on treatment utilization and treatment fidelity. While this data is quite important, discussion of alternatives for assessing treatment utilization and fidelity indicators was beyond the scope of this Workgroup.

A modified Delphi process was used to increase consensus regarding outcomes monitoring options (Fitch et al. 2001). This process included literature review, expert interviews, and a national panel of experts and stakeholders. The literature review covered published journal articles and books, unpublished documents on treatment and outcome assessment in schizophrenia, plus communication via phone and email with key informants. These informants included experts who had published on schizophrenia outcome assessment, Workgroup members, and individuals who had key unpublished work. Each informant was asked for literature they were aware of on the subject (published and unpublished) and for the names of other individuals who are knowledgeable in this area. Following several iterations, the identified documents were reviewed. A draft paper was prepared summarizing, for each domain, evidence-based treatments, common problems with treatment quality, and potential outcome indicators, instruments and assessment modalities.

A national panel was assembled that included experts in clinical treatment, quality improvement, and outcomes assessment, as well as national and regional policy-makers. In September 2005, this panel met via video conference. Prior to the conference, panel members reviewed the results of the literature review. During the conference there was, for each targeted domain, review of the evidence supporting treatments in the domain and common problems with treatment implementation. Within each domain, the panel suggested constructs that can identify (a) need for treatment, and (b) change in status associated with treatment. For each construct identified, alternative strategies and instruments were discussed that make use of administrative or billing data, medical records, patient self-assessment, clinician assessment, or interviewer-based assessments. Recommendations were made regarding alternative outcomes monitoring strategies. Panel members discussed and negotiated areas of disagreement to increase consensus regarding the recommendations. Following the conference, panel members provided additional feedback by email and phone. A document was prepared summarizing project consensus. This was circulated to the Workgroup. An iterative process of revisions led to the final consensus reported in this manuscript and endorsed by the Mental Health QUERI. The authors of this manuscript have no known conflicts of interest, and certify their responsibility for this article.

Results

Psychotic Symptoms

Symptom Severity

National guidelines concur that the treatment of psychosis includes the prescription and use of appropriate medication to control psychotic symptoms (American Psychiatric Association 2004; Kreyenbuhl et al. 2010; Miller et al. 2004). By “appropriateness,” we mean the extent to which clinicians are using effective strategies to address symptoms and side-effects (Young 1999). Almost all patients who are in treatment for schizophrenia are prescribed an antipsychotic medication (Lehman 1999). However, many patients do not receive treatment that appropriately addresses their symptoms or side-effects (Young et al. 1998). While choice of antipsychotic medication is controversial, there is agreement that symptoms and side-effects need to be managed, and there are strategies for doing so. It is useful to identify which patients are not receiving appropriate medication management for psychosis, so that quality improvement efforts can focus on their needs. The appropriateness of prescribing can be evaluated using data on the severity of psychotic symptoms in combination with data on medications prescribed (Young et al. 1998). Unfortunately, administrative data and chart review are not reliable or valid methods for assessing psychosis (Young et al. 1998). One alternative data source is patient self-assessment. Self-report measures of psychosis severity can be found in Table 1.

Table 1

Treatment targets and outcome measures

Treatment target and outcome measures	Description of outcome measures
Psychotic symptoms: symptom severity
Self-report
Revised Behavior And Symptom Identification Scale-Revised (BASIS-R)	The BASIS-R is a 24-item self-report instrument with six scales: psychosis, depression/functioning, interpersonal problems, alcohol/drug use, self-harm, and emotional lability. The BASIS-R has good reliability and validity (Eisen et al. 2004) and can accurately identify moderate and severe psychosis (Niv et al. 2007a).
Symptom Checklist-90-Revised (SCL-90-R)	The SCL-90-R is a 90-item self-assessment tool that measures psychoticism and paranoid ideation in addition to seven other symptom scales. Although Derogatis reported acceptable reliability and validity of the scale (Derogatis and Melisaratos 1983), subsequent studies suggest that the instrument measures a single global distress factor instead of the nine independent subscales reported (Clark and Friedman 1983; Rauter et al. 1996).
Brief Symptom Inventory (BSI)	The BSI is a 53-item self-administered scale developed from the SCL-90-R. The BSI has good psychometric properties and is an acceptable brief alternative to the SCL-90-R. However, its validity for measuring psychosis is unclear. In a study of persons with schizophrenia, the BSI had only one factor structure of general psychological distress (Hoe and Brekke 2009). As with the SCL-90-R, the psychoticism and paranoid ideation scales did not emerge. Another study compared the BSI to interviewer assessment and did not find a relationship between the psychoticism scale and the unusual thought content and hallucination items by interview (Morlan and Tan 1998). They also did not find a significant relationship between the BSI paranoid ideation scale and suspiciousness by interview. The BSI is, however, capable of differentiating between those who were classified as high versus low on positive symptoms by interview (Preston and Harrison 2003).
Psychotic symptoms: medication adherence
Administrative data
Medication possession ratio (MPR)	MPR assesses the extent to which dispensed medications provide coverage for a given interval (e.g., 6 months). MPR is inversely correlated with the probability of hospitalization (Gilmer et al. 2004; Valenstein et al. 2002; Weiden et al. 2004). Although there does not seem to be a distinct threshold at which partial antipsychotic adherence becomes problematic, taking 80% or more of prescribed medications (MPRs ≥ 0.8) has been used as a cut-point (Osterberg and Blaschke 2005). A limitation of this approach is that calculations require 6 months or more of data, so results may be obtained too late to be clinically useful.
Gap measure	Gap measure is based on gaps in medication prescribing or failure to refill prescriptions (Bryson et al. 2007). A gap measure assessed 10 days after a refill is due can provide information that is timely enough to allow clinicians to intervene, though with a greater number of false positives (Law et al. 2008).
Microelectronic monitoring systems	These pill bottles have a bottle cap that records the date and time of every bottle opening. Monitoring bottles are costly to use widely.
Blood and urine tests	Blood and urine tests can identify patients who have no recent medication ingestion (Cramer 1991). However, this is a relatively expensive option that is not well accepted by clinicians and patients.
Self-report
Count missed doses	This approach appears to be inaccurate (Byerly et al. 2007; Cramer 1991).
Drug Attitude Inventory	Assesses attitudes toward medications rather than adherence behavior (Hogan et al. 1983).
Psychotic symptoms: treatment resistance
Administrative data	Set a benchmark to detect under-use of clozapine.
Self-report	Use instruments described above to identify patients who may benefit from clozapine (i.e., are medication adherent and have persistent high levels of psychosis).
Medication side-effects
Administrative data
Body Mass Index (BMI)	BMI is calculated from height and weight, and is the most commonly recommended metric for monitoring weight gain. A BMI of 18.5–24.9 is normal, 25.0–29.9 is overweight, and 30 or higher is obese. Guidelines indicate that BMI should be recorded before medication initiations or changes, at every follow-up visit for 6 months following medication initiation or change, and at least every 3 months thereafter (Marder et al. 2004).
Fasting glucose or HgA1c	Hyperglycemia and diabetes can be tracked through monitoring of fasting plasma glucose values. However, it may be difficult for patients with schizophrenia to fast reliably. Use of random blood glucose values, or HgA1c, is likely more practical. The possibility of diabetes should be considered in the event of fasting plasma glucose >126 mg/dl, random plasma glucose value >200 mg/dl, or HgA1c >6.1% (Marder et al. 2004). Testing should be done before beginning a new medication. Patients with diabetes risk factors or weight gain should have monitoring 4 months after starting a new medication and annually thereafter.
Lipids	A fasting lipid panel should be completed at least every 5 years.
Self-report
Height and weight	Patients can also report their weight to the clinician, though this may be inaccurate because patients may not weigh themselves regularly, have unreliable scales, or inaccurate memory.
LUNSERS	The Liverpool University Neuroleptic Side Effect Rating Scale (LUNSERS)(Day et al. 2005) covers extrapyramidal and other side-effects but is limited in its assessment of sexual side-effects.
Arizona Sexual Experience Scale (ASEX)	The ASEX is a 5-item scale commonly used to assess sexual dysfunction and has been validated among patients with schizophrenia (Byerly et al. 2006; McGahuey et al. 2000). Sexual functioning should be assessed prior to antipsychotic medication initiation and yearly thereafter (Marder et al. 2004). If the patient is prescribed a first-generation antipsychotic, risperidone or paliperidone, sexual functioning should be assessed at every visit until dose is stabilized.
Comorbid drug and alcohol use
Self-report
Michigan Alcoholism Screening Test (MAST)	The MAST is a 25-item yes/no questionnaire used to identify alcohol abuse disorders (Selzer 1971). Although the MAST has been shown to be reliable and valid in non-psychiatric populations, it has generated mixed results in patients with schizophrenia (McHugo et al. 1993; Toland and Moss 1989).
CAGE	The CAGE, a 4-item screen for subjective negative consequences of alcohol abuse, has acceptable sensitivity and specificity in psychiatric populations (Mayfield et al. 1974).
Alcohol Use Disorders Identification Test (AUDIT)	The AUDIT screens for alcohol abuse by identifying harmful personal and social consequences of alcohol use (Bohn et al. 1995). This 10-item questionnaire is brief, easily administered, and is highly reliable and valid in schizophrenia (Dawe et al. 2000; Saunders et al. 1993). The AUDIT requires only a seventh grade reading level and is a better measure of recent alcohol consumption than the MAST or the CAGE, which focus on lifetime use (Hays et al. 1995).
AUDIT-C	The AUDIT-C is a brief version of the AUDIT consisting of the three consumption-related items of the AUDIT (Bush et al. 1998; Dawson et al. 2005).
T-ACE	The T-ACE (Russell 1994) is a 4-item, self-report alcohol screen.
TWEAK	The TWEAK (Russell and Bigler 1979) is a 4-item, self-report alcohol screen.that consists of items from the MAST, the CAGE, and the T-ACE (Russell 1994). The TWEAK was slightly more accurate than the T-ACE in detecting alcohol abuse in patients with schizophrenia, and both of these questionnaires have been shown to be more accurate than the MAST or the CAGE (Russell 1994; Wolford et al. 1999).
Drug Abuse Screening Test (DAST)	The DAST (Skinner 1982) is a leading self-administered drug abuse screen and has been validated in psychiatric populations (Staley and el-Guebaly 1990).
ASSIST	The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) is a 8-item scale used to detect substance abuse in a primary care setting (WHO ASSIST Working Group 2002).
Two-item conjoint screen (TICS)	The Two-Item Conjoint Screen (TICS) has been used in primary care to identify patients with current alcohol or drug problems (Brown et al. 1997).
Comorbid depression and suicidality
Self-report
Beck Depression Inventory—II (BDI-II)	The BDI—II is a 21-item scale with good reliability and validity (Beck et al. 1996).
BDI—primary care version (BDI—PC)	The BDI—PC is a 7-item screening device designed for use in primary care settings (Beck 1993).
Patient Health Questionnaire-9 (PHQ-9)	The PHQ-9 is a 9-item, self-report scale that assesses frequency of depressive symptoms over the past 2 weeks (Kroenke et al. 2001) and has been shown to be responsive to improvements with treatment (Lowe et al. 2004).
Beck Scale for Suicidal Ideation (BSI)	The BSI (Beck and Steer 1991) is a 21-item, self-report measure that screens for suicidality and has good psychometric properties in schizophrenia (Pinninti et al. 2002). Alternatives include the two-item Clinical Suicidality Assessment scale (CSA)(Pfeffer et al. 1988), and the two-item self-harm subscale from the BASIS-R.
Clinical Suicidality Assessment Scale (CSA)	The CSA is a 2-item suicidiality screen (Pfeffer et al. 1988).
BASIS-R	The BASIS-R includes in 2-item self-harm subscale (Eisen et al. 2004).
Family and caregivers
Administrative data	Hospitalization data.
Self-report
Family Environment Scale (FES)	The FES is a 90-item, true/false questionnaire measuring family cohesion, conflict, and expressiveness (Morisky et al. 1986).
Family Attitude Scale (FAS)	The FAS is a 30-item measure of family stress and expressed anger (Kavanagh et al. 1997).
Level of Expressed Emotion Scale (LEE)	The LEE is a 60-item measure of expressed emotion (Cole and Kazarian 1988).
Patient Rejection Scale (PRS)	The PRS an 11-item measure of relatives’ hostile and critical attitudes toward their ill family member (Kreisman et al. 1979).
FACES IV	FACES IV is a 42-item measure that yields scores in a number of domains, including family cohesion, adaptability, communication, and satisfaction (Gorall et al. 2004).
McMaster Family Assessment Device	The McMaster Family Assessment Device is a 53-item measure that assesses the health of family functioning (Miller et al. 1985).
Quality of Life Interview (QOLI)	Items from the QOLI can be used to assess frequency of family contact and satisfaction with family relationships (Lehman 1988).
Vocational functioning
Self-report
Employment status	Employment status (i.e., competitive employment, sheltered employment, unemployed); number of hours worked per week; number of paid work days in the past 30 days; job tenure; salary per hour. While dollars earned is a critical outcome, there has been limited research on the validity of obtaining this information from patients.
Indiana Job Satisfaction Scale (IJSS)	The IJSS is a brief job satisfaction questionnaire developed for individuals with severe mental illness (Resnick and Bond 2001).
Social Attainment Scale—II (SAS-II)	SAS-II has a patient self-report version that measures missed days of work, objective and subjective performance adequacy, interpersonal friction, distress, and satisfaction (Schooler et al. 1979). For patients who are unemployed, an item from the SAS-II can be used to measure desire to work.
Quality of Life Interview (QOLI)	The QOLI includes a number of items assessing employment status and satisfaction.
Community tenure and housing
Administrative data	Hospitalization data.
Self-report
Residential Follow-Back Calendar	The Residential Follow-Back calendar records days of stable community housing, institutional days, homeless days, and marginal homelessness (Bebout et al. 1997).
Schizophrenia Outcomes Module	The Schizophrenia Outcomes Module includes valid measures of housing status (Cuffel et al. 1997).
Quality of Life Interview (QOLI)	The QOLI includes items assessing satisfaction with housing.

Medication Adherence

Another common challenge is poor medication adherence (Lacro et al. 2002). It is useful to identify which patients are poorly adherent (see Table 1 for measures), so that quality improvement efforts can implement interventions that improve adherence. There are evidence-based strategies that improve adherence, including cognitive-behavioral approaches and long-acting injectable medications (Zygmunt et al. 2002).

Treatment Resistance

Clozapine is the only antipsychotic that has superior efficacy in patients with treatment-refractory psychosis (Chakos et al. 2001; McEvoy et al. 2006). According to national guidelines, it should be used in patients who have had at least two trials of other antipsychotic medications, but who continue to have clinically significant psychotic symptoms (Kreyenbuhl et al. 2010). However, clozapine is more complex to prescribe than other medications and is underused (Busch et al. 2004; Leslie and Rosenheck 2007). It is likely that prescription data can be used to detect under-use of clozapine. Treatment-refractory patients might be identified by detecting polypharmacy or high dosages of antipsychotic agents, but this approach has not yet been validated. Another approach is to identify patients who have persistent high levels of psychotic symptoms, adequate adherence, and trials of at least two antipsychotic medications. Alternatively, there is no reason to believe that treatment resistance varies by clinic. Therefore, it should be possible to establish a benchmark for clozapine use that would reflect a reasonable prevalence of clozapine use. It has been estimated that between 25 and 40% of patients are refractory to treatment, and up to a half of these individuals have substantial improvement with clozapine (Kane and Marder 2005). High performing states have had about 15% of patients on clozapine (Covell et al. 2002).

Recommendations

Further research is needed to determine the extent to which assessment can inform efforts to improve treatment, reduce psychosis, and improve psychosis-related outcomes, as well as the optimal intervals for assessments. Adherence is clearly important and can be monitored using pharmacy data. To improve treatment of psychosis, it will be necessary to systematically monitor the severity of psychosis in patients. Self-report is a valid method for screening for psychosis (Niv et al. 2007a), and the panel recommends the BSI or BASIS-R. When psychosis assessment is combined with data on medication prescribing, it is possible to identify patients who are not improving with their current regimen. Under-use of clozapine can be detected with this approach, or by comparing prescribing rates to benchmarks.

Medication Side-Effects

Most antipsychotic prescriptions are now for second generation agents. The most common side-effect with this medication class is weight gain and related metabolic complications, such as diabetes and hyperlipidemia. These place patients at elevated risk for cardiovascular disease. Less common side-effects include extrapyramidal side-effects (especially tardive dyskinesia and akathisia), and sexual dysfunction. National panels have developed recommendations for the screening and monitoring of antipsychotic medication side-effects (American Diabetes Association 2004; American Psychiatric Association 2004; Marder et al. 2004) (see Table 1 for measures).

The most important side-effects of second-generation antipsychotic agents are weight gain and related medical conditions. Patients can self-monitor their weight. Clinicians should routinely calculate BMI and perform laboratory monitoring for diabetes and hyperlipidemia according to national guidelines. Weight and metabolic data can be combined with data on prescribing and other treatments. These data identify patients who are overweight or gaining weight, and who are not receiving medication changes or psychosocial interventions that improve weight (Newcomer 2007; Newcomer et al. 2008). Similarly, it is possible to identify patients with diabetes or hyperlipidemia who are not receiving needed medical services.

Patients who are prescribed first-generation agents should be assessed for extrapyramidal side-effects. Patients who are prescribed a first-generation agent, risperidone, or paliperidone should be monitored for sexual dysfunction. Extrapyramidal and sexual side-effects can be reliably evaluated by self-assessment.

Drug and Alcohol Use

Evidence-based care for schizophrenia and comorbid substance abuse consists of coordinated, simultaneous provision of mental health and substance abuse treatment (McHugo et al. 2004). To ensure the provision of evidence-based services for the dually diagnosed, it is necessary to know when patients are abusing drugs or alcohol and, more specifically, what substances are being used, the frequency and amount of use, and impairments in functioning resulting from substance use. Objective assessment of substance use, e.g., urine toxicology screens and breathalyzer data, is the gold standard for guiding treatment. Since objective data are often not available, monitoring may also be based on administrative data, clinician assessment, or self-assessment. Administrative data tend to be of limited use because rates of detection and documentation of substance abuse in routine care are low and of questionable accuracy. Also, when diagnoses are made, there is often limited documentation regarding the severity and recency of the use. Self-administered assessments may be useful in a clinical setting as they are easier to obtain than clinician-administered assessments and have better predictive utility (Wolford et al. 1999). Patients appear to be more forthcoming with self-administered assessments than during interviews. Self-report measures of alcohol and drug use can be found in Table 1.

Regular screening should be used to identify substance abuse and associated problems and to guide treatment. Self-administered patient reports can provide data of similar or better quality than clinician assessment while reducing clinician burden. The AUDIT or AUDIT-C can be used to screen for alcohol use disorders and the DAST can be used to screen for drug use disorders.

Depression and Suicidality

Depressive symptoms have been shown to be important determinants of quality of life among patients with schizophrenia. Some treatment guidelines support the use of antidepressant medication in this population for those who continue to experience depression after a reduction in positive psychotic symptoms (American Psychiatric Association 2004). These guideline recommendations have been made even though no pharmacological or psychosocial treatments for comorbid depression and schizophrenia have consistently proven to improve outcomes. Suicide is a common cause of premature death in schizophrenia, and individuals with schizophrenia are nine times as likely to die by suicide as the general population. Self-report measures of depression and suicidality are valid, and can be found in Table 1.

Routine assessment of depression and suicidality is important, though there is limited evidence to guide the frequency of these assessments. Screening and assessment for depression can be done by self-assessment using the BDI or PHQ-9. The PHQ-9 is quicker to administer. The BASIS-R suicidality items are a reasonable choice to screen for suicidality. However, further research is needed to develop and evaluate psychosocial and medication interventions that improve depression and suicidality in schizophrenia.

Family and Caregivers

Family psychoeducation is included in national treatment guidelines based on its positive effect on outcomes for patients and their family members (Kreyenbuhl et al. 2010). Family interventions have been shown to reduce relapse and hospitalizations, and have broader benefits on psychosocial adjustment, such as vocational functioning (McFarlane et al. 2000; Pilling et al. 2002; Pitschel-Walz et al. 2001). Despite mandates to implement family services and family-driven care (President’s New Freedom Commission on Mental Health 2003), research indicates that most families do not have even minimal contact with mental health clinicians (Resnick et al. 2005). To ensure that families and other caregivers have access to services and clinicians, it is necessary to know which patients are in close contact with family or caregivers. This information is not reliably found in administrative data or medical records. However, it can be reliably obtained through patient self-report. Although there is no cut-off for intensity of contact that defines an “involved caregiver,” this is sometimes defined as contact of at least several times a month. Research on the potential benefits of family involvement suggests that high priority should be given to engaging the families of patients who have severe psychotic symptoms, low medication adherence, or who report a negative family environment. Strategies for identifying the first two groups are found in the section on psychotic symptoms. There are several scales that assess family environment and relationships (see Table 1). However, research does not support limiting interventions to families with particular scores on these instruments. Clinicians may find them helpful, but none is recommended for routine assessment. Structured, long-term family services have been associated with reduced psychosis and less hospitalization. Evaluation of psychosis is described above. Hospitalizations and related crisis services can be identified using a combination of administrative data and self-report. There are also established scales to measure family outcomes, including knowledge of illness and family burden, though the validity and reliability of these instruments is uncertain.

Family members and other caregivers who might benefit from increased involvement in care should be identified and engaged through outreach. The presence of an involved family or caregiver can be ascertained by patient self-report. At a minimum, caregivers should be provided with information, given access to the treatment team, and given referrals to family support organizations. Evidence-based family interventions should be offered. In regard to the outcomes of family services, relevant domains for assessment include psychotic symptoms, medication compliance, and hospitalization.

Vocational Functioning

Supported employment (SE) is an evidence-based practice that substantially increases rates of competitive employment in populations with schizophrenia (Crowther et al. 2001). According to guidelines, SE should be offered to individuals who are not employed and wish to find work, and to individuals who are currently working but find their work unsatisfying or too stressful (Kreyenbuhl et al. 2010). It is often impossible to determine from administrative data or medical records which patients should be offered SE, since work status and goals are not consistently documented. While research on assessment is limited, it is likely that these domains can be assessed by self-report (see Table 1).

Self-report measures should be used to identify eligible patients and to assess the outcomes of SE. Employment status can be assessed by self-report, including dollars earned and hours worked. Job satisfaction can be assessed using the Indiana Job Satisfaction Scale.

Community Tenure and Housing

A key component of recovery for people with schizophrenia is successfully living in their communities; for example, not being hospitalized, incarcerated, or homeless. About 10% of people with schizophrenia are homeless (American Psychiatric Association 2004), and there is a spectrum of autonomy in housing arrangements. The Workgroup saw value in measuring the strength of an individual’s integration into their community, but there was little research or consensus regarding how to measure this construct. Therefore, the focus was restricted to housing and duration of community tenure.

Some individuals with schizophrenia require specific psychosocial services to live in the community or require long-term, structured living environments. Assertive community treatment (ACT) is effective in improving community tenure and is included in national treatment guidelines. Patients may be eligible for ACT if they are homeless, hospitalized, incarcerated, or at high risk for these outcomes. Repeated hospitalizations can be identified using billing or administrative data. Since these datasets typically do not include all hospitalizations, they can be augmented by self-report. In addition to ACT, there are other care models that improve access to housing (Young and Magnabosco 2004).

Important outcomes of ACT and housing services include housing status and satisfaction with one’s living situation. Administrative and billing data typically lack accurate and current information regarding community tenure and housing. There has been limited research on the validity of self-administered measures of housing, although one study found patient reports of housing status to be highly correlated with family and clinician reports (Cuffel et al. 1997). There are widely used interviewer-administered instruments that are likely to provide an accurate assessment of housing status, including homelessness, and stable or unstable community housing (Table 1).

Consumer-driven interventions, such as peer support or consumer outreach workers, also show promise for improving community tenure (Chinman et al. 2006). However, there is insufficient evidence to know which individuals should receive consumer-driven interventions, how to provide them, or how to characterize their effectiveness. They are not yet covered in national treatment guidelines, and the Workgroup was not able to characterize relevant needs for outcome assessment.

Eligibility for ACT should be determined using a combination of administrative and patient self-administered data assessing housing, hospitalizations and incarceration. The major challenge to using these data to evaluate access to ACT is the nature of the patient population being served, which may be homeless and minimally engaged in treatment. Novel approaches for identifying eligible individuals for ACT should be developed and studied. In regard to housing services, homelessness and unstable housing should be assessed by self-assessment. Risk for homelessness, while important, has not been well characterized. Satisfaction with housing should be assessed using items from the Quality of Life Interview (Lehman 1988).

Discussion

The Schizophrenia Outcomes Workgroup used literature review, expert interviews, and a national expert panel process to increase consensus regarding outcomes monitoring strategies that can be used to support quality improvement and the delivery of evidence-based treatments for schizophrenia. It is clear that outcome measurement is better developed in some domains than others. Good consensus was reached for the domains of psychotic symptoms, medication side-effects, drug and alcohol use, depression, family and caregivers, vocational functioning, community tenure, and housing. In these domains, there are well developed, validated, self-report outcomes assessment instruments, and often multiple excellent choices for routine outcomes monitoring. However, there are areas that would benefit from further research on measurement, including, for example, identifying eligible patients for clozapine or supported employment. It was not possible to develop consensus in other domains, such as negative symptoms, cognitive symptoms, and use of peer support interventions, because efficacy research is not sufficiently developed. National treatment guidelines offer limited guidance regarding treatments that should be provided to patients and, therefore, outcomes to be monitored.

Although there are high quality outcomes assessment instruments, some assessment strategies are much easier than others to implement in routine treatment settings. The most feasible strategies for obtaining outcomes data are the use of administrative or billing data, and self-administered assessments. Administrative data are typically not designed for this purpose, and often contain inaccurate or incomplete information on outcomes and limited clinical content. This is somewhat better in systems that have implemented Electronic Medical Records (EMRs). However, even with EMRs, outcomes data are inconsistently documented or encoded in fields that are readily usable for quality improvement (Young 2007). To standardize medical records data, healthcare organizations sometimes rely on clinicians to document outcomes, an approach that has not worked well. Although many clinicians could be trained to high levels of reliability in standardized instruments, they typically do not have the time or interest necessary for training, routine performance, and entry of outcomes data into medical records.

Self-administered patient assessments are another strategy for outcome monitoring. They requires less time from busy clinicians, do not require repeated trainings, and alleviate concerns about clinician gaming of assessments. Methods of self-assessment vary and can be determined by the needs of specific projects and healthcare organizations. Methods of self-assessment have included traditional paper and pencil forms, scannable forms, electronic self-entry devices, and internet access by patients. Electronic devices at clinics can include dedicated kiosks or tablet computers with touch-screen and audio interfaces. Self-entry systems can generate reliable and valid data, are well accepted by clinic populations with schizophrenia, and can be feasible to implement in clinic settings (Chinman et al. 2004, 2007).

For quality improvement, it is necessary to know which patients would benefit from specific treatments, and which of these patients have not been receiving those treatments. The Workgroup did not review methods for monitoring the utilization or appropriateness of services that patients receive. However, there was agreement that, in addition to monitoring outcomes, it will also be important to monitor the provision of effective care. For psychosocial treatments in particular, it is important to monitor the extent to which treatments demonstrate fidelity to practices that are known to improve outcomes.

This project had a number of limitations. First, the outcome strategies and instruments recommended are not intended to replace clinical decision-making in individual patients. Rather, they were developed to support organizational efforts to improve care in populations of patients. Also, the instruments reviewed are not an exhaustive list. There are likely to be other instruments with good psychometric properties that are suitable for monitoring outcomes in these domains. In any specific instance of quality-oriented outcomes monitoring, selection of specific strategies and instruments must be a function of the goals of the project and the characteristics of the healthcare system.

Healthcare organizations have succeeded in improving care by making use of quality improvement that includes routine collection of outcome data. The routine and systematic collection of data allows for identification of unmet needs, and redesign of services to improve access to effective practices and improve outcomes. There have been fewer successes with this approach in schizophrenia than in other chronic medical disorders, and there are specific challenges to collection and encoding of outcomes data in this population. However, the Workgroup believes that there is substantial potential to improve access to effective treatments in populations with schizophrenia. This begins with systematic monitoring of outcomes. Providing outcomes data to clinicians would not by itself be expected to improve the quality of care (Fihn et al. 2004; Stone et al. 2002). Rather, this data needs to be used as part of systematic projects to reorganize service delivery. Systematic research will bee needed to evaluate these efforts to improve care.

There was strong consensus regarding viable strategies for routine outcomes monitoring. There are some domains, such as psychotic symptoms, medication side-effects, and medication adherence, where there are existing instruments and data sources that are ready now for widespread implementation and use. There are other domains, such as clozapine, employment, and community reintegration where useful instruments are available, but measurement could be substantially strengthened with additional research. It was not possible, or desirable, to definitively recommend one specific instrument and measurement strategy for all quality improvement efforts. Each measurement strategy and instrument has pros and cons that must be weighed. And, to be successful, quality improvement needs to attend both to evidence and also local context when selecting measurement strategies. Therefore, when there were multiple acceptable instruments, the Workgroup did not select one choice, but left this choice to the quality improvement project that will use the data. With regard to alternative measurement strategies, patient self-report appears to be the most broadly feasible method in many domains. Self-report should be supplemented, where helpful, by clinician documentation and administrative data. Whatever path an organization selects with regard to domains, instruments, and data collection strategies, there are important opportunities to use outcomes data as part of projects to improve care for schizophrenia.