Implementing Residential Integrated Treatment for Co-occurring Disorders.

Objective: Much is now known about effective treatment for co-occurring substance abuse and psychiatric difficulties and many evidence-based practice recommendations have been disseminated. Implementation of these recommended interventions in daily clinical practice has been more of a struggle. This article describes successful implementation of integrated treatment for co-occurring disorders in a small residential program.
Methods: A traditional 28-day addiction service was transformed into a 3-month integrated treatment program and 155 individuals with co-occurring disorders agreed to participate in its evaluation. The transformation entailed a completely new manualized service, training in a number of clinical interventions for all program clinicians, ongoing clinical supervision, and formal measurement of clients' backgrounds, substance abuse, quality of life, mental health symptoms, self-esteem, and satisfaction with the program. We also obtained collateral informants' reports on participants' symptoms, substance use, and quality of life. Fidelity to the treatment model was continuously assessed, as were participants' knowledge and skill acquisition. In addition, impact of the implementation on the program clinicians' morale and attitudes toward evidence-based practices was assessed, as was staff turnover and per diem costs.
Results: Despite very problematic clinical and sociodemographic histories, the 86 participants who completed the program showed clinically significant mental health symptom improvement, acquisition of knowledge and skill, and high self-esteem and satisfaction with the program. Program fidelity, clinician morale, commitment to the program, and attitudes toward evidence-based practice were uniformly high. These successes were achieved while maintaining the lowest per-inpatient day cost of all hospital inpatient units. Conclusions: The findings support the contention that evidence-based integrated treatment can be implemented with fidelity in regular clinical practice to the benefit of participants, staff, and the hospital. Our experience was that having a scientist-practitioner working as a staff member on the program to lead the implementation was a key element. Future reports will focus on longer-term follow-up of substance use and quality of life outcomes.

Estimates of lifetime diagnosis of substance use disorder for individuals with major mental illness are approximately 50% (Essock et al., 2006; Mueser et al., 2000). Nevertheless, substance abuse is underdiagnosed in people experiencing mental illness (Mueser, Noordsy, Drake, & Fox, 2003). Much evidence informs best practice treatment for people with mental illness and substance use disorders. Implementation of evidence-based recommendations, however, has lagged behind. This gap can have serious implications, such as increased probability of rehospitalization, depression, suicide, incarceration, relapse, inability to manage finances, unstable housing, medication noncompliance, compromised physical health, poor family relationships, and higher service use and cost (Dickey & Azeni, 1996; Drake, Mueser, Clark, & Wallach, 1996; Drake et al., 1998; Mueser et al., 2003; Polcin, 1992; Rosenberg et al., 2001; Wallen & Weiner, 1989). Because of the gap between evidence and practice, consumers often “bounce” from addiction treatment to psychiatric care and back again, benefitting little from either (Center for Substance Abuse Treatment, 2005).

Newer integrated treatment programs have better outcomes (Detrick & Stiepock, 1992; Dixon et al., 2010; Drake, McHugo, & Noordsy, 1993; Drake, O'Neal, & Wallach, 2008; Drake, Yovetich, Bebout, Harris, & McHugo, 1997; Durrell, Lechtenberg, Corse, & Frances, 1993; Godley, Hoewing-Robertson, & Godley, 1994; Meisler, Blankertz, Santos, & McKay, 1997; Ziedonis et al., 2005). In integrated treatment, both disorders are primary and treated by one team simultaneously. Services are comprehensive, assertive, harm-reducing, long-term, motivation-based, and multimodal (group, individual, and family formats; Mueser et al., 2003; Health Canada, 2001). A focus on recovery (Center for Substance Abuse Treatment, 2005) is important (Hipolito, Carpenter-Song, & Whitley, 2011), with formats including residential, outpatient, and informal consumer and peer support (Center for Substance Abuse Treatment, 2005; Health Canada, 2001). Treatments are specific to the challenges faced at each stage and include cognitive behavior therapy, motivational interviewing, relapse prevention, and skills building (Center for Substance Abuse Treatment, 2005).

Fully integrated treatment is still not available to all who need it (Hogan, 2011), probably because implementing evidence-based, integrated services for a population with complex needs is difficult (Drake & Bond, 2010). Too frequently, evidence about outcomes (or clients’ preference) does not drive practice; rather, considerations such as finances and practitioner preferences tend to be the driving factors. Approximately half of implementation efforts fail (Klein & Knight, 2005). Thus, the crucial issue is less the ideal service for those with co-occurring disorders but how that ideal can be realized. As distinct from optimal service itself, studying implementation requires measuring appropriateness, affordability, fidelity, effectiveness, and penetration (Torrey, Tepper, & Greenwold, 2011). Torrey et al. (2011) reported on many integrated treatment programs’ struggles with fidelity, but those providing supervision were more successful because clinical leaders mastered the skills and knowledge necessary to provide ongoing clinical training and monitoring. Similarly, therapeutic fidelity, best measured with work samples (Stirman et al., 2010), is most strongly associated with implementation success and improved outcomes (Drake et al., 2001). Torrey et al. (2011) also stressed the importance of a committed, results-oriented change leader with organizational and managerial support. In this paper, we demonstrate the implementation of evidence-based integrated treatment (Center for Substance Abuse Treatment, 2005; Mueser et al., 2003) and report measures of clinical fidelity and the impact on clients, clinicians, and the organization. We present data to show that the program targeted appropriate clients; faithfully implemented treatment as designed; yielded improvements (compared to waitlist controls) in skills, knowledge, and symptoms; and achieved high client and staff satisfaction. Data also address whether clinical improvements were merely a function of selection biases and whether the improvements were achieved via clinicians’ commitment to the therapeutic model. Last, we present data on costs.

The Program

A residential service in a rural psychiatric hospital in Canada's largest and most ethnically diverse province opened in 1970, delivering traditional 28-day addiction therapy comprised of readings and discussion of addiction-related topics and weekly 12-step meetings. In 2003, the program's mandate changed to serving clients who had a substance use disorder and mental illness and it was renamed a “concurrent disorders program.” The clinical service remained essentially unchanged, however, because no one had been designated to lead any changes. In 2005, the second author proposed to the hospital's administration that the first author (who was interested in co-occurring disorders) be hired to oversee an actual transition to integrated treatment in line with best practice recommendations (Center for Substance Abuse Treatment, 2005; Mueser et al., 2003) as well as to evaluate the resulting new service. With the support of executives and the program manager, the duration of treatment was extended to 90 days with pre-admission (waitlist) engagement comprising regular contact with program clinicians and aftercare comprising time-unlimited weekly groups to review and practice skills. Manuals and fidelity scales for all program elements were developed (Center for Substance Abuse Treatment, 2005; Mueser et al., 2003). Clinical services were suspended for 2 weeks (after clients in the 28-day program graduated) to permit training of all program clinicians by the first author in behavioral and cognitive behavior therapy (CBT). This was a highly effective way to get intensive training to all program clinicians, and it had the additional benefit of providing a distinct demarcation between the “old program” and the new.

The 12-week, full-time program was fully integrated throughout so that mental illness and substance use were addressed together and manualized in two streams: CBT and Skills. The CBT stream curriculum was geared toward clients with a diagnosis of a mood or anxiety disorder and the Skills stream toward clients with a diagnosis of schizophrenia or schizoaffective disorder. Single-session psychoeducation, based on topics suggested by Mueser et al. (2003), was common to all (see Appendix for a full list). Also, weekly groups addressed anger management, self-esteem, spirituality, art therapy, therapeutic recreation, fitness, and discharge planning. Clients also attended self-help (AA, NA, CA, and Double Recovery) and weekly employment or peer-led “Pathways to Recovery” groups (Ridgway, McDiarmid, Davidson, Bayes, & Ratzlaff, 2002). Smoking cessation involved motivational enhancement, psychoeducation, and information about the tobacco industry. CBT entailed weekly 90-minute group therapy and 60-minute homework sessions. The manual was adapted from Mind over Mood (Greenberger & Padesky, 1995). Although originally intended for clients in the CBT stream, it became clear that most clients with schizophrenia and schizoaffective disorder would also benefit from CBT. Those few clients with these latter diagnoses who did not participate in CBT attended a session which consisted of a customized board game that provided practice in such key skills as drug refusal, getting a sponsor, and reporting symptoms and side effects to a physician. There were also weekly Seeking Safety groups (Najavits, 2002) for those with posttraumatic stress disorder (PTSD). All clients in the Skills stream participated in sessions based on Overcoming Addictions (Roberts, Shaner, & Eckman, 1999), involving weekly drug refusal, assertiveness, and social skills training sessions using modeling and behavioral rehearsal. Finally, because this population is at risk of physical health disparities (Sullivan, Han, Moore, & Kotrla, 2006; Thornicroft, 2011), clients were offered thorough medical examinations, including relevant diagnostic (including HIV) tests. The medical service also facilitated access to a dentist, nurse practitioner, and any indicated medical specialists.

Although residential integrated treatment for co-occurring disorders has been associated with positive outcomes (Brunette, Mueser, & Drake, 2004), residential services might afford an “artificial sanctuary” such that program gains do not generalize to post-program environments. In consultation with K. Mueser (personal communication, 2005), a schedule of graduated weekend passes was implemented to facilitate identification of high-risk situations and requisite coping skills. Newly identified risks and skills were discussed every Monday and strategies added to relapse prevention/recovery plans. In addition, clients attended biweekly discharge planning groups in which supports for successful transition to the community were determined (e.g., housing, finances, interpersonal relationships).

METHODS

Participants

This study was approved by the facility's Research Ethics Board in accordance with the Declaration of Helsinki. After a complete discussion of the study, 155 people with co-occurring disorders who completed at least one assessment consented to participate; one person declined. Of the 155, 121 were admitted and 86 completed the residential program. People were eligible if (a) they were at least 16 years old; (b) they met Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) diagnostic criteria for a mood, anxiety, or psychotic disorder; (c) they met DSM-IV criteria for a substance use disorder; and (d) their symptoms were sufficiently stable that they could participate in structured group therapy. Each participant was asked to nominate a collateral informant (113 participants nominated a collateral, typically a family member) authorized to report on the participant's background and complete questionnaires about the participant's quality of life, substance use, and mental health symptoms. The length of time participants stayed on the waitlist varied from a few days to 5 months.

Measures

Background, Substance Abuse, and Quality of Life Measures

A client information form was used to record clinical history and sociodemographic variables gathered through review of participants’ hospital charts and interview. A quality of life questionnaire was developed to assess engagement in school or work, financial circumstances, housing, personal relationships, recreation, self-esteem, and support from professionals in the 30 days prior to first referral to the program or receipt of inpatient services. A substance abuse questionnaire was developed to assess the frequency, amount, and type of substance use (and gambling) in the 30 days prior to first referral to the program or receipt of inpatient services. Participants indicated whether they experienced any negative social, vocational, educational, or legal consequences due to substance use. This questionnaire also asked about motives for using substances (coping with negative affect or enjoyment) and about lying about or hiding substance use. Self-reported substance use has been shown to have good concordance with urine screens and is a valid indicator in most circumstances (Jackson, Covell, Frisman, & Essock, 2004). Parallel quality of life and substance abuse surveys were designed for collaterals, as well as a 32-item inventory for collaterals to report on participants’ mental health, including symptoms of anxiety, depression, bipolar disorder, and psychotic disorders.

Symptom Measures

The following battery of validated assessments was administered monthly on the waitlist, on admission, halfway through the program, and on discharge: the Brief Psychiatric Rating Scale (BPRS)–Expanded Version (Lukoff, Nuechterlein, & Ventura, 1986; Greenwood & Burt, 2000; Hedlund & Vieweg, 1980; Morlan & Tan, 1998); the Beck Depression Inventory (BDI-II; Beck, Steer, & Brown, 1996; Segal, Coolidge, Cahill, & O'Riley, 2008); the Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, & Steer, 1988; Steer, Ranieri, Beck, & Clark, 1993); and the Rosenberg Self-Esteem Scale (RSS; Rosenberg, 1965; 1989; Heatherton & Wyland, 2003; Sinclair et al., 2010).

A number of additional assessments were also completed. The Brief Trauma Questionnaire (Schnurr, Vielhauer, Weathers, & Findler, 1999) is a 10-item scale that assessed whether DSM-IV PTSD Criterion A was met. Respondents indicated whether they experienced any of 10 potentially traumatic events. Two follow-up questions ascertained perceived and actual harm. Inter-rater reliability kappa coefficients for the presence of trauma have been shown to be above .70 (range, 0.74–1.00) for all events except physical illness (0.60; Schnurr, Spiro, Vielhauer, Findler, & Hamblen, 2002). Participants who reported any traumatic event were administered the Posttraumatic Stress Disorder Checklist–Specific Version (PCL-S; Weathers, Litz, Herman, Huska, & Keane, 1993), a 17-item measure to assess the severity of posttraumatic symptoms. Scores of 50 or higher are likely to meet diagnostic criteria for PTSD (McDonald & Calhoun, 2010; Weathers et al., 1993). The PCL-S was administered once on admission and again on discharge if the participant completed the 3-month program.

The Personality Diagnostic Questionnaire 4 + (PDQ-4+; Hyler, 1994) is a 99-item, self-report screener for DSM-IV personality disorders completed while on the waitlist. A total PDQ-4 + score of 30 or greater is purported to correspond to a substantial likelihood of significant personality disturbance. Overall, the measure has adequate reliability and validity (Hyler, Skodol, Kellman, Oldham, & Rosnick, 1990; Reich, Yates, & Nduaguba, 1989; Trull & Larson, 1994). The Wide Range Achievement Test–4th Edition, Reading Subtest (WRAT-4; Wilkinson & Robertson, 2006) is an assessment of word-reading ability included in the larger WRAT-4 battery and was administered on admission to the program. Total reading scores can be converted into a school grade equivalent and the Reading Subtest has excellent alternate-form, test–retest reliability, and internal consistency (Wilkinson & Robertson, 2006).

Clinician Measures

Clinicians (n = 14) anonymously completed the Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004; Aarons, McDonald, Sheehan, & Walrath-Greene, 2007) almost 6 years after integrated treatment began. The EBPAS assessed attitudes toward adoption of evidence-based practices (EBPs) and innovations and consists of four subscales: Appeal (the extent to which one would adopt EBP if intuitively appealing); Requirements (the extent to which one would adopt EBP if required); Openness (the extent to which one is open to new interventions); and Divergence (the extent to which one perceives EBP as not as useful or important as clinical experience; Aarons et al., 2010). The EBPAS has good psychometric properties (Aarons, 2004; Aarons et al., 2007; Aarons et al., 2010). We also obtained data on staff morale 3 years before the transformation to integrated treatment, at the beginning of the new service, and at the end of the present data collection, 5 years later.

Program Measures

Fidelity to treatment manuals was measured by recording therapy sessions and randomly sampling for review by the first author, with a random subsample of 10 also rated by the third author (intraclass correlation coefficient of inter-rater reliability = .92, p < .001). Therapists received continual feedback about fidelity scores. For each major therapeutic component, participants completed a paper-and-pencil or role-play assessment of the associated concepts and skills. There was also a short 3- or 4-item quiz administered after each psychoeducation session. All those completing residential treatment were asked to complete a satisfaction questionnaire. Finally, we rated the program on the Integrated Treatment Fidelity Scale (ITFS; Substance Abuse and Mental Health Services Administration, 2009), a measure of the extent to which 14 best practice recommendations were implemented. The first and third author based independent ratings on interviews with the program director, five other clinicians, and three participants, as well as ten charts chosen at random, observation of a clinical meeting, a recording of a group therapy session, and knowledge of the program. Disagreement on three items was resolved by adopting the lower of the two ratings. The raters tried to be conservative and the intraclass correlation coefficient of inter-rater reliability was good (.77, p < .01). Last, we obtained data on staff turnover and costs in comparison to hospital averages.

Data Analysis Plan

Descriptive statistics were used to summarize data for all 155 participants at program initiation on measures of symptoms. We also computed summary data for participants who remained in treatment after 6 weeks (n = 100), those who completed the 12-week inpatient program (n = 86), and those who remained on the waitlist after 1 month (n = 90), and 3 months (n = 12). Means and 95% confidence intervals were used to assess differences in clinical improvement between active treatment and the waitlist. For those in active treatment, an overall index of clinical improvement was derived by computing the pre–post difference on each symptom measure then standardizing and summing to yield a single score to which each measure contributed equally. This index was used in analyses exploring predictors of clinical improvement. A dichotomous variable was used to examine predictors of program completion. Descriptive statistics were used to summarize the tests of knowledge and skill, and client satisfaction. The intercorrelations of the overall index of clinical improvement, the measures of knowledge and skill, and the satisfaction survey were examined. We summarized the scores of clinical fidelity, clinicians’ attitudes toward evidence-based practice, and morale at three time points for which there were comparison data for the rest of the hospital's inpatient services.

RESULTS

Some of the participants’ background characteristics are shown in Table 1 and generally indicate problematic clinical and sociodemographic histories. Almost all had received prior mental health treatment and many had a prior attempt at formal therapy for addiction. Many came from families where addiction had been a problem, many had not lived with both parents, and most suffered abuse or neglect as children. For most, substance abuse had been a problem since their teens. Many had positive screening results for PTSD. Few were employed and, on average, their adult socioeconomic status was lower than it had been when they were children. Most had financial problems and depended on government assistance. Many endorsed characteristics of personality disorder, and the overall index of personality disturbance (PDQ-4 + total) was very high (M = 40.8; SD = 13.3). Many had some history of property, drug-related, or assaultive crime.

TABLE 1

Background Characteristics of the Sample (N = 155)

Characteristic	M (SD) or n (%)
Demographics
Gender (male)	93 (60%)
Age (years)	39.0 (11)
Highest grade completed	11.6 (2.10)
Reading grade level (WRAT-4)	11.5 (2.3)
Juvenile history
Evidence of childhood abuse or neglect (n = 116)	60 (52%)
Parental, family SES rating[a] (n = 135)	8.81 (1.72)
Alcohol problem under age 18 (n = 133)	112 (84%)
Drug problem under age 18 (n = 136)	107 (79%)
Mental health contact under age 18 (n = 127)	40 (32%)
Adult sociodemographic history
Never married or common law (n = 152)	41 (27%)
No employment or school participation at admission (n = 151)	121 (80%)
Participant's adult SES rating[a] (n = 146)	6.62 (1.83)
Unstable housing (n = 149)	42 (28%)
Government disability or welfare (n = 148)	75 (51%)
Nonviolent offense history (n = 139)	82 (59%)
Violent offense history (n = 136)	44 (32%)
Prior mental health contact as an adult (n = 146)	134 (92%)
Number of professional contacts in month prior to admission (n = 149)	4.4 (3.8)
Prior addiction treatment (n = 142)	67 (47%)
Self-reported substance abuse in 30 days prior to admission
Alcoholic drinks (for those reporting an alcohol problem; n = 105)	69 (92)
Times using illicit drugs[b] (for those reporting a drug problem; n = 94)	18.6 (20.7)
Average number of cigarettes per day (n = 153)	13.2 (12.8)
Principal substance of abuse (n = 154)
Alcohol	94 (61%)
Cannabis	17 (11%)
Cocaine	12 (8%)
Opioids	7 (5%)
Other	4 (3%)
Polysubstance abuse	19 (12%)
Symptoms on admission
BAI score (n = 154)	22.9 (12.8)
BDI score (n = 153)	29.0 (13.2)
Rosenberg score (n = 153)	12.0 (5.6)
BPRS total score (n = 150)	39.5 (7.7)
PDQ-4 + total score (n = 146)	40.8 (13.3)
Screening positive for PTSD (n = 115)	59 (51%)

Note. SES = socioeconomic status; BAI = Beck Anxiety Inventory; BDI = Beck Depression Inventory; BPRS = Brief Psychiatric Rating Scale; PDQ-4 + = Personality Diagnostic Questionnaire 4 +; WRAT-4 = Wide Range Achievement Test–4th Edition, Reading Subtest; PTSD = Posttraumatic stress disorder.

SES is based on an ordinal occupational scale from 1 (none) to 12 (postgraduate or professional).

Includes the misuse of prescription drugs.

Participants’ responses to the quality of life and substance abuse questionnaires for the 30 days prior to their referral to the program indicated that most were neither working nor engaged in education and most reported financial problems. For those whose primary substance of abuse was alcohol, the average number of drinks in those 30 days was 69 (SD = 92) with high variability (range = 0–600). Participants reported using illicit drugs an average of 19 (SD = 20) times in those 30 days, again with wide variation (range = 0– 150). Most reported significant problems arising from substance use, and alcohol was the most common substance of abuse.

There were statistically significant overall correlations between participant and collateral reports of substance use and quality of life in the month prior to entering the program, r = .43, p < .01, and r = .57, p < .01, respectively. Collateral reports of depression and anxiety symptoms in the month prior to entering the program correlated significantly with participants’ initial BDI-II and BAI scores, r = .43, p < .01, and r = .36, p < .01, respectively. Collateral reports of psychotic symptoms, however, were not related to BPRS Psychosis subscale scores (r = .04). Of interest, collaterals reported more psychotic symptoms than did participants.

Progress in treatment was examined by comparing scores on the principal clinical measures at three time points for all participants in treatment to scores at the same time points for participants on the waitlist. Thus, the first measurement entailed the 155 participants who received an initial assessment. The second measurement for the treatment group entailed all 100 participants still present after 6 weeks, and the third measurement entailed all 86 participants who completed residential treatment. These scores were compared to those obtained for all 90 participants on the waitlist after 1 month and 12 on the waitlist after 3 months (Figure 1).[1] All measures showed the same pattern: Treatment participation was associated with large clinically and statistically significant improvements while the same period on the waitlist was associated with much smaller, usually nonsignificant, changes.

FIGURE 1

Changes in symptoms as a function of treatment participation. Mean Beck Anxiety Inventory (BAI), A; Beck Depression Inventory (BDI), B; Rosenberg Self-Esteem Scale, C; Brief Psychiatric Rating Scale (BPRS), D; BPRS Psychosis subscale, E; and BPRS Depression subscale, F; scores as a function of three time points: for therapy participants, at baseline, after 6 weeks of treatment, and at program completion (usually 12 weeks); and for waitlist controls, at baseline, at 4 weeks, and at 12 weeks. Error bars reflect the 95% confidence interval for each mean.

The means for the Beck inventories at program completion were in the range for nonclinical samples. Not shown in Figure 1, for those with positive screening results for PTSD, mean PCL-S score was 59.4 (SD = 11) on admission and 37.76 (SD = 12.7) on discharge (n = 42), indicating that most did not endorse clinically significant levels of PTSD on discharge.

Comparing the 86 participants who completed residential treatment to the 69 who did not on the six principal clinical measures at the initial assessment yielded no significant differences. The scores were indistinguishable, indicating that program completion was not a proxy for less serious initial clinical presentation. We then examined whether background characteristics predicted program completion. Those characteristics that were negatively associated with program completion included childhood abuse or neglect, being separated from their parents before the age of 16, lower grade level completed in school, alcohol and drug problems before the age of 18, and a violent offense history.

We derived an overall index of each participant's clinical improvement by subtracting the initial score from the final post-treatment score on each of the BAI, BDI, RSS, and BPRS. Then, to ensure that each measure contributed equally, each difference score was standardized to a mean of zero and standard deviation of 1.0, and the overall clinical improvement score was the sum of these four z-scores. Overall improvement scores were positively related to few background characteristics: female gender, higher grade completion in school, hours per week worked by those employed or in school, number of contacts with helping professionals in the month prior to admission, and fewer nonviolent offenses in their history. Overall clinical improvement was also unrelated to participants’ principal substance(s) of abuse.

Clinicians

Clinical staff attitudes toward evidence-based practices, as measured by the EBPAS, showed higher mean Openness, Requirements, and Appeal and lower Divergence ratings than published EBPAS norms. The average EBPAS total score in the normative sample was 2.73 (SD = .49; Aarons et al., 2010), compared to an average score for the present report of 3.16 (SD = .48). There were three relevant formal anonymous surveys of staff morale. Two (3 years before and then coincident with implementation of integrated treatment) employed an identical method and indicated that the new program was associated with a statistically significant increase. That is, based on the total proportion of responses rated more positively than “neutral,” the program's clinicians increased in morale and engagement, 61% versus 56%, p < .05. The second survey also yielded a statistically significantly higher overall morale rating compared to the hospital total, 61% versus 52%, p < .05, and the integrated treatment program yielded higher ratings than any other inpatient unit. The third survey, using a different method, conducted 5 years after implementation of integrated treatment, confirmed that high morale continued—the program received the highest ratings among inpatient clinical units and was statistically significantly higher than the rest of the hospital, 76% versus 57%, p < .01.

Program Outcomes

We examined each participant's acquisition of the skills and knowledge the program sought to impart. Out of 349 possibilities, only 10 (< 3%) observations were consistent with a knowledge or skill decrement, and, on average, every score increased statistically significantly, in almost every instance, by more than a standard deviation. Table 2 shows the pre and post scores for those participants who completed both assessments. Interestingly, the magnitude of skill and knowledge improvements was unrelated to the magnitude of the overall clinical improvement score. There were also 21 simple 3- to-5-item quizzes administered after each psychoeducation session. Average scores on these were uniformly high with little variability, mean = 93% (SD = 5.99), average n = 65. Participants who completed the 3-month residential program gave high satisfaction ratings, mean = 4.51 (SD = 0.59) on a 5-point scale. Again, interestingly, satisfaction ratings were related neither to clinical improvement score, r = .11 (n = 83), nor to the measures of skill and knowledge acquisition, all p > .10. Ten clients participated in a hospital-wide survey administered by peer support workers and, on average, rated the integrated treatment program more positively than did the patients of all hospital units; 27 of the 35 survey items received ratings at least 10% higher than the rest of the hospital.

TABLE 2

Changes in Participants’ Group-Specific Tests of Skill and Knowledge From Admission to Program Completion

Group	Admission M (SD)	Program Completion M (SD)	t (df)
Seeking Safety	9.21 (1.64)	10.28 (1.33)	4.04 (42)
Gambling prevention	12.86 (3.22)	16.25 (3.25)	11.58 (82)
Cognitive behavioral therapy	7.02 (2.44)	9.46 (2.04)	7.34 (55)
Anger management	2.60 (1.33)	4.54 (1.12)	9.43 (64)
Skills group	32.18 (7.91)	52.64 (5.80)	8.55 (21)

Note. df = Degrees of freedom. All p < .0001.

Ratings of adherence to the manualized material (clarity of delivery, completeness, etc.) as well as therapeutic skill (engagement, therapeutic alliance, etc.) were made on 7-point scales. Over the entire course of the data collection, average fidelity was high, mean = 4.6 out of 6, representing excellent fidelity (a score of 4 was considered very good fidelity and quality) with very low variability, SD = 0.56. The program scored 60 out of 70 on the ITFS. Six items were scored 5 out of 5; six scored 4 out of 5; and two (pharmacological treatment and secondary interventions for nonresponders), 3 out of 5.

Excluding retirements, in the 5 years between implementation of integrated treatment and the present data collection, there was only a single resignation (a strong supporter of the integrated program who asserted that clinical changes had nothing to do with the decision) from among the clinical staff and no requests for transfer to other hospital units. There are ongoing requests from clinicians in other hospital units wanting transfers to the integrated treatment program. The high morale and engagement ratings could not be attributed to integrated treatment costing more. Based on total program budgets (all clinical and management salaries plus all other patient-related costs) and occupancy numbers available at the end of the present data collection, the integrated program registered a mean cost of $450 per inpatient day, compared to $700 for the rest of the hospital's civil inpatient units. It is noteworthy that the entire transformation, implementation, and present evaluation were accomplished with essentially no additional expenditure other than the salary of the change leader (the first author).

DISCUSSION

The present findings support the conclusion that implementation of evidence-based, integrated treatment for concurrent disorders yielded high therapeutic fidelity, strong clinician commitment to the therapeutic model, minimal staff turnover, high practitioner morale, and significant in-program clinical improvement, all achieved at less cost than routine hospital care. Five years after implementation, clinicians reported very positive attitudes toward evidence-based interventions. The findings support the conclusion that participants, whose backgrounds indicated serious and long-standing addiction and psychiatric difficulties, made substantial gains in the knowledge and skills imparted by the program. These participants also exhibited clinically significant improvements in mental health symptoms and reported high satisfaction and engagement with the therapy. The data also indicate that improvements cannot be attributed to the mere passage of time or to selecting participants with less serious difficulties for treatment.

Interestingly, few participants’ characteristics were related to the measures of clinical improvement. Female participants, those with more education, those who were engaged with professional services and were in school or had jobs, and those with less criminal offending exhibited greater improvements in psychiatric symptoms and self-esteem. We had intended to examine the association between fidelity ratings and clinical improvement, knowledge and skill gains, and client satisfaction, but fidelity scores were too uniformly high with insufficient variability to conduct such analyses. Although assessing work samples is time-consuming, the investment very probably contributed to the high fidelity ratings because the therapists were aware that any session could be chosen for rating.

The ITFS was not developed for small residential programs so that some items were difficult to rate. However, ratings indicated areas for improvement, including better means to detect nonresponders and use of such adjunctive medications as naltrexone. We plan to have the program re-rated by external assessors in conjunction with the assessment of 1-year post-treatment outcomes. Even in the absence of follow-up data, the present findings offer clues for improving the program. For example, the high PDQ-4 + scores suggest interventions for personality disorders (e.g., dialectical behavior therapy; Linehan, 1993), procedures to detect and boost effects for nonresponders, better long-term aftercare, and better efforts to facilitate employment and education. These will be entertained as improvements after the collection and analysis of 1-year follow-up outcomes. Future revisions to the program will be based on the simultaneous analysis of follow-up data with background, clinical improvement findings, fidelity scores, and aftercare arrangements. Such analyses comprise the optimal means to effect continuous quality improvement.

Research has demonstrated the importance of change leaders (e.g., Torrey et al., 2011), and we suggest that the present change leader's simultaneous roles as clinician and evidence-based practice expert was key to success—the program had been mandated for 2 years to convert to integrated treatment, but no progress was made until the change leader surveyed the empirical literature, identified best practice, designed program manuals, developed outcome measures, provided training, and implemented fidelity assessment and clinical supervision protocols. We also suggest that this coherent approach to designing and monitoring clinical services was indeed one cause of the very high morale reported by the program's therapists. A further advantage was that the change leader had mostly program development and evaluation duties and did not have to schedule these around a full clinical caseload. We suggest the ideal change leader is a scientist-practitioner who understands the interventions, delivers some clinical services, evaluates the relevant literature, and undertakes program evaluation.

Conclusions

Despite somewhat pessimistic statistics on implementation of integrated treatment, the present findings show it is possible

There need be no practical impediments to the implementation of manualized, high-fidelity, evidence-based best practice for co-occurring disorders. A traditional residential service was transformed to an integrated treatment program for co-occurring disorders in such a way that participation in the program led to clients exhibiting clinically significant improvements in symptoms, gains in relevant knowledge and skills, and high satisfaction. In addition to these benefits to participants, the integrated program had lower cost than any of the institution's other inpatient units and the program's clinicians exhibited high morale and commitment and strong support for the specific program and evidence-based practices in general.