A simplified method for routine outcome monitoring after drug abuse treatment.

THE ROUTINE COLLECTION OF DRUG TREATMENT OUTCOMES TO MANAGE QUALITY OF CARE, IMPROVE PATIENT SATISFACTION, AND ALLOCATE TREATMENT RESOURCES IS CURRENTLY HAMPERED BY TWO KEY DIFFICULTIES: (1) problems locating clients once they leave treatment; and (2) the prohibitive cost of obtaining meaningful and reliable post-treatment data. This pilot describes precise methods for an economical staff-based routine outcome monitoring (ROM) system using an 18-item core measure telephone survey. As implemented at Narconon™ of Oklahoma, a behavioral and social skills based, residential drug rehabilitation program, the system was psychometrically adequate for aggregate reporting while providing clinically useful information. Standardized procedures for staff training, collecting client contact information, structuring exit interviews and maintaining post-treatment telephone contact produced follow-up rates that improved from 57.6% to 100% over the course of the project. Aggregate data was used to improve program delivery and thereby post-treatment substance use and social outcomes. These methods and use of data may contribute to the discussion on how to best monitor outcomes.

Background

Expectations for addiction treatment—outcomes and quality

Today’s medical management model aims at improving health care quality and costs, in part through mandating the use of “evidence-based” services. What constitutes “evidence,” and how it is best obtained, is still under discussion and is collaboratively evolving as treatment providers, researchers, and stakeholders examine and re-examine their needs and available resources.1

While treatment efficacy can be readily quantified using research-driven clinical trial methods under very strict and controlled research parameters, such stringent design requirements—and associated costs—are rarely practical for the typical community-based treatment program.2 Although it can be argued that clinical trial-type evidence best suggests “what works,” routine monitoring of how well a given program maintains consistent results over time is also vital to demonstrating responsible investment of public and private resources. The goals of routine monitoring are straightforward: to ensure reproducible treatment effectiveness, consistency, and cost-effectiveness; to improve the overall quality of treatment; and to ensure accountability of health service providers to funding sources by monitoring their outcomes and maintaining treatment quality.1

For this reason, decades of State and Federal legislation, funder initiatives and accrediting agency criteria include directives to monitor post-discharge program outcomes in addition to those that are obtained while the client is still at the facility. The extent to which this is accomplished, however, is still less than desired.

The 1993 Government Performance and Results Act (GPRA) and the GPRA Modernization Act of 2010 charged Federal agencies with accountability for reduction in substance abuse problems and gains in employability, safety, responsiveness, and program quality.3 A standard for important outcome measures, the 135-item Center for Substance Abuse Treatment (CSAT) GPRA tool is comprised of National Outcome Measures (NOMs) that “represent meaningful, real life outcomes for people who are striving to attain and sustain recovery, build resilience, and work, learn, live, and participate fully in their communities.” All programs receiving discretionary grants are required to submit staff-collected enrollment, discharge, and 3 or 6-month post-enrollment follow-up data using this instrument. Discretionary grants include 10–20 percent funding for dedicated follow- up activities that are expected to attempt contact with 100 percent of all discharged clients and, minimally, obtain data from 80 percent.4,5

Some states have incorporated NOMs sets in the reporting required for licensing procedures, and a few states—California, Minnesota, Connecticut, Delaware, Illinois, and New York—are establishing permanent statewide outcomes monitoring systems.6 Although aggregate data has provided valuable information regarding predictors of treatment success, as well as a greater understanding of special populations that may not be otherwise represented,7,8 post-treatment reporting systems have yet to be consistently implemented.

For example, in 1998 the California Treatment Outcome Project (CalTOP) implemented a 44-center, staff-based pilot project with a sophisticated client-tracking database to collect client information at enrollment, 3-months and 9-months post enrollment. Staff at participating sites randomly selected 20 percent of the 15,618 admissions for follow-up and produced valuable data regarding services received, outcomes and cost-effectiveness, as well as areas for quality improvement such as lower-than-expected retention rates and poor matching of severity with type and intensity of services received.9 The statewide system that evolved from this pilot, the California Outcomes Monitoring System (CalOMS), currently monitors only measures at enrollment and discharge, but plans to include a follow-up component in the future.

The Minnesota Department of Human Services is required by statute to collect sufficient information to evaluate the efficiency and effectiveness of treatment for chemical dependency. In Minnesota, the Drug and Alcohol Abuse Normative Evaluation System (DAANES) is used to meet both state and federal reporting requirements. Currently, treatment staff submit web-based data at three points in time: admission, six-month review (opioid replacement therapy clients only) and discharge. Although the original DAANES design included a 6-month outcomes measure of the percentage of patients who remained abstinent, attempts to obtain this follow-up data were hampered by non-compliance and it is not collected at this time.10 Early efforts returned a non-representative sample size (22 percent in 199111), but as recently as 2001 program staff from small facilities reported difficulties with the long reporting requirements; less than two-thirds submitted even baseline data.7

Insurers can use follow-up data to assess and recommend guidelines for appropriate care. Kaiser Permanente, in a study by Chi et al, determined that a continuing care model (yearly primary care visits with specialty referrals as needed) improved abstinence or non-problematic use rates and resulted in health plan cost savings.12 A shorter subset of outcomes data that could be routinely collected would likely contribute to additional treatment refinements.

Accrediting agencies have also placed greater attention on outcomes and other quality standards that are aimed at continual quality improvement. Both the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and the Commission on Accreditation of Rehabilitation Facilities (CARF) provide assistance to healthcare providers in meeting or exceeding quality standards, including requirements for outcome monitoring (Joint Commission on Accreditation of Healthcare Organizations 1998).13

Outcomes monitoring also has application in the corporate setting, as a tool that Fortune 500 companies with large Employee Assistance Program (EAP) vendors (eg, Caterpillar, Archer Daniels Midland, and ConocoPhillips) can use for selecting treatment and prevention programs. The Workplace Outcomes Suite (WOS) can detect statistically significant changes in absenteeism, presenteeism, work engagement, life satisfaction, and workplace distress over 90-day follow-up periods, and with sample sizes as small as 50 clients. (Lennox RD and Sharer D, Sensitivity analysis of pre-treatment post-treatment scores in the Workplace Outcome Suite (WOS). Unpublished Analysis. 2011). Where client groups are heterogeneous and treatments are non-protocol driven, such as those found in many out-sourced or external EAPs, a facility-driven routine outcome monitoring effort could return useful data to the EAP.

Obtaining real world outcomes data for the evidence base

The systematic tracking of clients after they have completed a full course of intervention, when they are operating under minimal supervision as a member of their family, workforce and community, is one of the most convincing methods for demonstrating real-world effectiveness of behavioral health programs. Despite stated aims to gather meaningful post-discharge data, most performance-monitoring efforts are still in development stages. Limitations to data collection include: (1) difficulty tracking clients once they leave the treatment setting; (2) using treatment staff to collect follow-up data when their main function is providing chemical dependency services; and (3) relying on personal interviews and other time-consuming protocols that are costly, require substantial staff training, and may result in data drift or loss due to follow-up complexity. According to the Alcohol and Drug Services Study (ADSS), the typical clinic has a therapeutic staff of less than a dozen people.14 Such an organization lacks adequate administrative resources for anything but a highly efficient follow-up and monitoring methodology.

A practical way to include quantitative follow-up methods may be attainable within the infrastructure of the growing continuing care model. Research at Chestnut Health Systems shows that while telephone follow-up solely for organizational feedback does not appear to improve post-treatment outcomes,15 15–20 minute telephone-based post-treatment counseling support can be an effective form of step-down treatment that increases engagement and sobriety while lowering relapse rates.16,17 For this reason, treatment programs are adding continuing care strategies that include a counseling component.

This paper describes a streamlined, telephone-based routine outcome monitoring process conducted within a continuing care system. This system is designed to provide meaningful data regarding treatment success that can be used for quality management purposes by a treatment facility. Utilizing a short outcomes survey with scales that focus on core treatment indicators, it is sufficiently simple to fit within the routines of staff at smaller facilities. Feasible for use by individuals who have not been trained in research methods, this system provides timely feedback to clinicians who may need to intervene with individual cases and, with certain limitations, can contribute to larger, aggregate pools of data used by those involved in basic research, treatment policy and funding decisions.

Aims

The purpose of this study was to evaluate the efficacy of a post-treatment Routine Outcome Monitoring (ROM) system as a tool for measuring, and improving, results from drug rehabilitation services. Study objectives included: (1) characterize and codify the necessary during-treatment actions that would result in meaningful post-treatment follow-up rates; (2) develop a short survey based on questions common to self-reported substance abuse measures in the U.S., including the Addiction Severity Index (ASI) and CSAT GPRA, that could be aggregated into clinically-useful reports; (3) characterize basic psychometric properties of this instrument, including internal construct validity, and (4) corroborate client self-report data with information collected from collateral sources such as relatives and continuing care records. Although CSAT and other performance efforts require contact with all clients regardless of discharge reason, this first phase of developing a quality assurance system was confined only to those who completed the residential treatment plan.

Methods

Narconon International and Psychometric Technologies Incorporated collaborated to develop a scientifically grounded methodology that would result in data that would be useful to treatment providers. The project included construction of a short, psychometrically-validated assessment of drug abuse treatment outcomes, and it built on enrollment, case management and client follow-up systems that were already in place at Narconon.

Evaluation setting: Narconon™ substance abuse treatment program

The ROM instrument and reporting system were developed in a pilot study at Narconon of Oklahoma, a 230-bed Level III Clinically Managed High Intensity Residential Service (American Society Placement Criteria for the Treatment of Substance Abuse Disorders standards). The international Narconon network includes nearly 130 facilities and makes use of secular materials developed by the philosopher and humanitarian, L. Ron Hubbard. The complete Narconon rehabilitation program involves ten distinct therapeutic modalities:18

: Nutritional and physical assists under 24-hour nursing supervision (preceded by medical withdrawal when necessary);

: Social Skills-based activities that address discomfort in the presence of others, aggressive or withdrawn behaviors and verbal communication skills;

: Light aerobic exercise and dry-heat sauna concurrent with nutrient support of vitamins, minerals and cold-pressed oils aimed at addressing the effects of residual drug and other protracted withdrawal symptoms.19,20 Participation requires a full physical exam and medical approval;

: This life skills module trains participants to recognize and overcome barriers to study and comprehension;

: A series of cognitive and behavioral exercises and drills aimed at changing the tendency of the addicted person to “live in the past” and improving their ability to objectively view people and experiences in the present;

: This module is designed to help prevent relapse by training the individual to recognize the characteristics of two distinct types of personalities—the social personality and the antisocial personality—and the ways that relationships with each type affect individual success, survival, drug use, or criminality;

: A therapeutic step to help the individual formulate personal ethical guidelines and address the consequences of past activities that were outside these guidelines;

: A relapse prevention and life skills process that incorporates ethical principles such as honesty, integrity, dependability and commitment to work, and helps participants work out the exact steps they need to take to improve areas of their lives such as healthy living, family relations, employment, etc.;

: As an additional tool to support their efforts to remain stable and productive following release, clients review essays on a group of precepts that comprise a non-religious, social skills, common sense moral code. Precepts include concepts such as “Take care of yourself”, “Don’t Do Anything Illegal”, “Be Industrious”, “Respect the Religious Beliefs of Others”, etc.;

: With the help of staff counselors, each client creates a discharge plan that addresses their goals as an individual, as a family member, in work relationships and any other situations unique to the individual, along with specific plans of action to achieve these goals.

Key aspects of the Narconon Drug Rehabilitation program were advantageous to developing the follow-up methodology and questionnaire:

A variety of client data is collected during enrollment in the Narconon program, including the Addiction Severity Index (ASI);21

The program is characterized by a lower ratio of staff to participants, more hours of staff contact time and close monitoring of treatment implementation protocols, characteristics common to programs with better retention and engagement in treatment.22–24 This suggested a strong likelihood of obtaining meaningful data with which to validate an outcome monitoring system;

An existing system of during-treatment quality control procedures established to assure that care is delivered according to codified standards more commonly seen only in “research therapies.”25 Because the Narconon “corporate culture” includes an error detection and correction system, it seemed likely that the project’s monitoring tasks would be supported by existing case management and administrative lines.

The follow-up assessment program was implemented as part of normal operation, and a system was developed to report findings back to the participating treatment center for quality management purposes.

Community re-integration

Congruent with a behavioral and social skills approach to rehabilitation, aftercare staff provide support for any sub-clinical problems or difficulties which, if left unchecked, might ultimately result in drug reversion. Should drug reversion occur, aftercare staff can help make arrangements for the client to re-enter treatment.

Approximately one week before program completion and final discharge, a trained staff member who is not involved in treatment helps the client formulate an individualized community re-entry plan following a standardized outline. The re-entry plan is developed by: (1) Listing client risk factors and problems noted in intake data and clinical records; (2) In coordination with clinical staff, rating the extent to which these conditions were addressed during the client’s participation in the program and noting any that may still need to be addressed, including health and emotional status, drug craving, communication, or antagonistic family members; (3) Listing client goals and objectives for community re-entry, including participation in social groups and drug free activities, obtaining work, developing additional skills, adequate housing, etc.; (4) Verifying client and collateral contact information and (5) Setting an agreed schedule of follow-up calls.

Staff and client keep signed copies of the plan, which includes follow-up telephone calls at these intervals: once a week from months 1 to 3; once every two weeks from months 4 to 6; once per month from months 6 to 12 and once every three months from months 13 to 34. During each follow-up call the specialist discusses progress or obstacles regarding each of the goals defined in the discharge plan. Changes are made in the plan if necessary. Key data and any recommendations are recorded in the client records.

If the individual does not move successfully into his new life, the Aftercare specialist will help the client work out how to apply the relevant skills by referring to program manuals that the client used while at Narconon and now has at home as reference materials. If the situation is severe enough to warrant it, the staff member will encourage the graduate to return to the Narconon center so that the difficulties can be reviewed, addressed in depth, and corrected. It should be noted that this also triggers a management review to identify possible flaws in the delivery and supervision of specific modules of the program.

Client tracking and outcomes assessment by treatment staff

At the start of this project, Community Reintegration follow-up was hampered by difficulty contacting clients. To more easily maintain contact with clients once they leave the facility, the project implemented a client tracking system to include:

Informing clients of the post-completion follow-up at the beginning of their treatment and obtaining permission to also contact supportive relatives or close associates as needed during and after the program;

Helping clients understand that the community reintegration and follow-up components are integral to their recovery goals, thus increasing the likelihood that they would participate in outcome monitoring.

Collecting complete locator information at intake including home, business and cell phone numbers, email address and street address;

Obtaining the same contact information for up to three relatives or close associates who do not live with the client but who the client states would know about their treatment and are supportive of their recovery;

For systematic and quantitative measurements, clients were contacted to complete the Routine Outcome-Monitoring (ROM) questionnaire within one month of the six-month anniversary of Narconon program completion. Key steps for obtaining meaningful outcomes included:

Immediately prior to the ROM survey, staff reviewed all notes from prior Community Re-integration follow-up telephone calls as a means to verify client answers and also to personalize the interview as appropriate;

Staff were trained to ensure that each outcomes assessment question is answered and that deviation to discuss problems is not part of the outcomes survey interview. On completion of all outcomes survey questions, staff can provide assistance or make an appointment for another follow-up call;

Initially, staff were allowed two weeks to obtain outcomes from both the client and a close relative or associate, for verification. Because the collateral data correlated highly (see results section) the final method instructed staff to make no more than three calls to the client, after which they could instead survey a relative or close associate. The time limit of two weeks was maintained.

Completed survey forms, coded only by an internal tracking number, were submitted to a statistical analysis group for scanned data entry and analysis. A quarterly report returned to the facility provided interpretation of the aggregate findings.

The outcome instrument

The outcome questionnaire, presented in Figure 1, was developed by isolating core questions from the 135-item GPRA CSAT treatment outcome module5 and including those that addressed key recovery goals additional to cessation of substance use: employment and education, family/social relations and housing.27,28 The original 10-item questionnaire assessed last 30-day substance use, drug-related legal status, employment and living status, and program satisfaction. Ten items were thought to obtain clinically relevant information during a short telephone interaction.

Figure 1

The routine outcome monitoring questionnaire.

During the project, Aftercare and Case Management staff provided valuable feedback regarding survey length and important clinical questions. Items were added to address last-30-day drug-related emotional status and last-30-day health status, as these important problem areas do not necessarily track with changed misuse of substances.29 Additionally, staff requested inclusion of questions regarding alcohol use to intoxication or illicit substance use since graduation as a means to assess the usefulness of the Community Re-integration component (see also Process Feedback and Quality Management section).

The final survey consists of six items directed at self-reported drug and alcohol use in the past 30 days, two items directed at general drug use since leaving treatment, five items directed at quality of life issues in the past 30 days, one item inquiring about use of other treatment services, three items inquiring about living conditions and one item inquiring about general health status. Finally, the instrument includes three qualitative interview questions that obtain clinically relevant information.

Subjects

For purposes of developing and evaluating the ROM methodology, this project was initially limited to only those individuals who had completed the full Narconon program (called “graduates”). Twice each project year between 2004 and 2007, non-clinical staff compiled a list of all individuals for whom the current month marked the six-month anniversary of their discharge. This compilation excluded program graduates who elected to remain on location to participate in post-treatment staff training. The list was given to the Aftercare specialist team, who completed the ROM surveys by telephone interview within two weeks. Any incomplete surveys were marked as missing data.

During the first three sampling periods, 2004–5, telephone contact was attempted for both the graduate and their closest relative, for purposes of verifying the graduate self-report and evaluating the viability of relying on family member reports. After the third sample period, collateral correlation was analyzed (see results section) and it was determined that the graduate data was reliable. Following this test, data was obtained from the graduate only, or by surveying a relative or close associate only if the graduate was not available after three contact attempts. Staff also noted those cases where collaterals did not have recent contact with program clients; data was not acquired if the collateral could not give a recent report and the case was categorized as missing data.

Data analysis

Observing Federal and local confidentiality rules, de-identified data coded only by a site-assigned, unique identification number was sent to Psychometrics Technologies Incorporated for entry into an Excel spreadsheet. All data was then imported into SPSS for further analysis.

Results

Three elements of feasibility are examined: (1) the ability to produce a follow-up rate of 80 percent; (2) the ability of the ROM instrument to produce acceptably accurate and clinically-meaningful aggregate information, as verified by a two-stage construct validity analysis and the distribution characteristics of self-reports; (3) resources required, including the time spent collecting extensive contact information and the time spent locating the clients and administering a short outcome questionnaire. Treatment center and staff feedback is also described.

Severity of the treatment population

A total of 419 subjects were identified who completed the program and returned to their uncontrolled, natural environment. Table 1 presents the means and standard deviations of the ASI scores at intake for all subjects as well as ASI means based on source of follow-up data.

Table 1

Mean ASI scores at intake.

ASI subscales	All subjectsn = 419	SD	Follow-up data obtained from
			Graduaten = 237	Relativen = 96	Missingn = 86
ASI employment	0.44	0.31	0.45	0.50*	0.45
ASI alcohol use	0.23	0.28	0.25	0.20	0.25
ASI drug use	0.26	0.17	0.26	0.29	0.26
ASI legal	0.24	0.30	0.22	0.25	0.22
ASI social	0.19	0.22	0.18	0.18	0.18
ASI psychological	0.05	0.20	0.05	0.08	0.05
ASI medical	0.09	0.20	0.08	0.07	0.08

Note:

P value = 0.03 all others not significant.

The Narconon treatment group appears to be generally consistent with clients seen in private residential treatment settings.21,30 Standardized on a scale of 0 to 1, the highest level of impairment in the ASI score appears in the Employment scale, approaching a 0.50 level. Alcohol Use, Drug Use and Social Impairment are all at the 0.25 range, indicating a moderate level of problems. Relatively low scores on Medical and Psychological subscales likely reflect the fact that clients (or their families) are attempting to address abuse-related problems before they have become debilitating, chronic conditions.

There were no statistical differences in baseline ASI characteristics when analyzed by source of outcomes data: program graduates, collaterals, or missing, with the exception of the employment scale. Initial addiction severity score differences does not associate with unavailability for follow-up interview.

Respondent analysis

Data was obtained from 323 of the 419 subjects who returned to their community; leaving 22.9% with missing data. The first sampling point had an inadequate follow-up rate. By reviewing the successful actions of Desmond et al,31 improvements were made as follows: (1) Staff were trained to use a simple enrollment form to collect multiple phone and email contact information from the client as well as multiple collateral phone and address contact data; (2) All contact information was verified and updated at discharge and, (3) A written checklist was implemented to organize each step of the follow-up process. As a result of these improvements, post-treatment contact rate consistently improved to upwards of 80 percent. Follow-up rates and respondent mix for each sampling point are presented in Table 2.

Table 2

Contact rates.

	1st 2004*N = 66	2nd 2004*N = 44	1st 2005*N = 71	2nd 2005N = 78	1st 2006N = 63	2nd 2006N = 63	1st 2007N = 34
Graduate	28 (42.4%)	29 (65.9%)	42 (59.1%)	42 (53.8%)	38 (60.3%)	34 (54.0%)	25 (73.5%)
Collateral	34 (51.5%)	38 (86.3%)	54 (76.1%)	15 (19.2%)	13 (15.9%)	17 (27.0%)	9 (23.1%)
Missing	28 (42.4%)	6 (13.6%)	17 (23.9%)	21 (26.9%)	12 (19.0%)	12 (19.0%)	0 (0.0%)

Note:

During the first three sample points, data was obtained from both graduate and collateral for verification of answers. Thereafter, staff were instructed to call collateral contacts after 4 unsuccessful attempts at reaching program graduate.

Staff turnover had some influence on follow-up rates. There were new Aftercare staff in second quarter 2005 and first quarter 2007; in first quarter 2006 and first quarter 2007 there was only one Aftercare staff person.

Construct validation of the drug problem items

Construct validity is tested by the extent to which similar, yet distinct, measures of treatment effectiveness give consistent answers. Although not all patients are expected to show positive outcomes on all measures, there should be a fair degree of association between measures—that is, there should be a general pattern or movement in a direction that reflects the success (or lack of success) of the drug abuse treatment. Such positive correlations are evidence of construct validity. Table 3 presents the bivariate correlations between thirteen measures.

Table 3

Intercorrelations among drug-related problems for graduate respondents.

	Means	1	2	3	4	5	6	7	8	9	10	11	12
1. 30 days: any alcohol	0.75
2. 30 days: alcohol to intoxication	0.27	0.58**
3. 30 days: cocaine	0.29	0.02	−0.04
4. 30 days: marijuana/hashish	0.15	0.15	0.28**	0.33**
5. 30 days: heroin	0.22	0.03	−0.00	0.32**	0.02
6. 30 days: other illegal drugs	0.27	0.20**	−0.02	0.34**	0.28**	0.20**
7. Since discharge: alcohol to intoxication	1.15	0.44**	0.36**	0.10	0.06	0.19**	0.14*
8. Since discharge: illegal drugs	1.17	0.04	0.03	0.43**	0.39**	0.35**	0.35**	0.29**
9. 30 days: arrested for drugs	0.11	0.01	0.03	0.16*	0.09	−0.03	0.14*	0.16*	0.32**
10. 30 days: spent night in jail	0.22	0.14*	0.06	−0.03	−0.03	−0.02	0.23**	0.25**	0.15*	0.37**
11. 30 days: stressed by drugs	0.48	0.22*	0.22*	0.23**	0.61**	0.61**	0.35**	0.13	0.49**	0.27**	0.25**
12. 30 days: reduced activities	0.33	0.12	0.19*	0.00	0.17	0.24**	−0.03	0.24**	0.30**	0.19*	0.16	0.42**
13. 30 days: emotional problems	0.57	0.21*	0.18*	0.31**	0.59**	0.51**	0.49**	0.20*	0.55**	0.22**	0.20*	0.82**	0.56**

Notes:

p < 0.05;

p < 0.01.

As can be seen in the table of correlations, roughly two thirds of the graduate reports are significantly intercorrelated with one another at the P < 0.05 level. The exceptions are in the use of “any alcohol” and the use of “alcohol to intoxication,” which include self-reports associated with subclinical levels (ie, non-problematic use) and may have meaning only if the post-treatment goals included abstinence from what is considered a legal activity.

Although the intercorrelations are not particularly large, there is sufficient commonality among the responses to support the inclusion of each variable in a broad measure of drug abuse-related quality of life. Taken together, the results presented in Table 3 support the validity of the data collected by staff over the telephone at the six-month follow-up.

As the questionnaire uses mostly single-item measures, attempting to define latent constructs from these items was deemed risky both because they are very disparate and because they were not designed to measure an underlying structure. The authors contend that factor analysis of latent variable analysis without a prior structures procedure would be inappropriate.

Comparison of self and collateral reports

To verify self-reported measures of drug abuse, the first three sampling periods included data collected by both self-report and collateral family reports. Approximately 91 pairs of self and collateral reports were available for the 10 sets of drug-related problems included in the original survey.

Due to the fact that the six-point Likert format is not considered an interval-scale of measurement, we used the Spearman rank-order correlations in our analysis of the validation data. Table 4 presents the Spearman rank correlation coefficients (r) and significance levels (p) for these ten variables. With the exception of one variable (alcohol to the point of intoxication), all of the correlations were positive, substantial, and statistically significant, indicating a high degree of correspondence between the two ratings. The significant correlations between the types of respondents range from a .40 for days of any alcohol, to a perfect correspondence for days of heroin and days of illegal drugs. Even the questions regarding the use of alcohol and drugs since leaving treatment correlate well and easily exceed traditional cutoffs for inter-rater reliability. These strong correlations suggest that graduates and their relatives agree when there is evidence of a problem, as well as when there is not.

Table 4

Correlations with family ratings.

	r	P
1. 30 days: any alcohol	0.40	0.0005
2. 30 days: alcohol to the point of intoxication	−0.03	ns
3. 30 days: cocaine	0.76	0.0005
4. 30 days: marijuana/hashish	0.78	0.0005
5. 30 days: heroin	1.00	0.0005
6. 30 days: other illegal drugs	1.00	0.0005
7. Since discharge: alcohol to intoxication	0.67	0.0005
8. Since discharge: used other illegal drugs	0.77	0.0005
9. 30 days: been arrested for drug related offenses	0.54	0.0005
10. 30 days: spent the night in jail	0.98	0.0005

The results in Table 4 support the validity and inter-rater reliability of the measures in terms of correspondence between the reports of graduates and relatives. Because of the small sample size and the binary nature of the outcomes, the three coefficients of 1.00 should be interpreted with caution. The fact that collaterals include a variety of contact types (mothers, fathers, spouses, and other relatives) who do not have direct access to drug use information and may be potentially affected by self-serving bias, one would expect only modest correlations. Although each of these factors has the potential to introduce substantial variability, this does not appear problematic; the observed level of convergence is consistent with that seen in other self-peer validation studies.

Analysis of recurrence of drug problems

The analysis of drug use in the past 30 days from the total sample of respondents is presented in Table 5 as separate results for the total of all respondents, the subset of respondents who were graduates and the subset of respondents who were relatives. Of the 323 subjects for whom data was obtained, 72.1% of the data came from the graduates themselves. The sample sizes reported at the top of the respective columns reflect the number of clients contacted. As the actual sample sizes for each cell vary slightly due to missing data, the percentages in the columns are based on the number of valid responses for a particular variable within that analytic dataset.

Table 5

Recurrence of drug-related problems.

Drug use and problems	Drug problems at follow-up
	Graduate(N = 238)		Relative(N = 94)
	None	1+ days	None	1+ days
1. 30 days: any alcohol	180 (76)	58 (24)	62 (66)	32 (34)
2. 30 days: alcohol to the point of intoxication	216 (91)	22 (09)	85 (90)	9 (10)
3. 30 days: cocaine	228 (96)	10 (05)	79 (85)	14 (15)
4. 30 days: marijuana/hashish	224 (94)	14 (06)	89 (96)	4 (04)
5. 30 days: heroin	233 (98)	5 (02)	83 (89)	10 (11)
6. 30 days: other illegal drugs	230 (97)	8 (03)	80 (86)	13 (14)
7. Since graduating: alcohol to intoxication	165 (69)	73 (31)	52 (56)	41 (44)
8. Since graduating: used other illegal drugs	173 (73)	65 (27)	53 (57)	40 (43)
9. 30 days: been arrested for drug related offenses	228 (96)	19 (04)	85 (91)	8 (09)
10. 30 days: spent the night in jail	231 (97)	7 (03)	82 (88)	11 (12)
11. 30 days: been stressed because of your drug use	119 (90)	14 (11)	43 (80)	11 (20)
12. 30 days: reduced or given up important activities	128 (96)	5 (04)	42 (78)	12 (22)
13. 30 days: experienced emotional problems	120 (90)	13 (10)	39 (74)	14 (26)

Note: Numbers in parentheses are percent of valid responses.

Analysis of graduate responses showed a very similar pattern: alcohol had the highest recurrence of last-30-days use at less than 24 percent and the last-30-days use of each illegal drug was less than ten percent. A separate analysis of the relative-only sub sample showed a similar but somewhat attenuated pattern.

Questions were also asked to assess the use of any alcohol or illegal drug since graduation. As expected, recurrence of these behaviors over this longer time period was slightly higher than for the last-30-day assessment.

Need for additional treatment

As can be seen in Table 6, the frequency of re-entry into treatment since graduation is estimated at a modest 12 percent, approximately one-third of those who reported substance use after program completion. This suggests that, for many, the reported post-program use levels may not be severe enough to require new treatment. Potentially, assistance provided as a component of the community re-integration follow-up helped these graduates implement their discharge plan—including early recognition and assistance with problem areas before clinical treatment became necessary.

Table 6

Need for additional treatment services since graduating.

	Need for additional rehabilitation services
	All respondents(N = 332)		Graduate(N = 238)		Relative(94)
	No	Yes	No	Yes	No	Yes
14. Since graduation required rehab services	243 (88)	32 (12)	175 (88)	26 (12)	68 (89)	8 (11)

Note: Numbers in parentheses are percent of valid responses.

Resources for follow-up

This 230-bed facility enrolls an average of 43 individuals each month. Enrollment staff did not see a change in workload when the enrollment form was changed to collect additional personal and collateral contact information.

For those clients who complete the full program, two full-time staff are responsible for all aspects of discharge planning, community re-integration follow-up, and the six-month routine outcome monitoring interview. These staff verified contact information at discharge, then averaged three attempts before reaching a client for phone interview followed by the 10–15 minute interview itself at each follow-up point as described in the methods section. At this facility, the staff requirement to complete just the ROM questionnaire on each client was 0.25 FTE. With the help of Narconon staff, a complete implementation manual has been written that should further reduce staff burden by providing comprehensive training on all aspects of this system. This manual describes in detail the process for verifying contact information, for review of treatment folders, for implementing a follow-up method aimed at transitioning a client back into their families and communities, and for obtaining answers when asked questions by telephone survey.

Other costs associated with implementing the system were in a range that would not be prohibitive to smaller facilities. A scanner with appropriate data software was a one-time purchase of approximately $1000. Review of aggregate information and reporting by a qualified professional cost about $1000 per analysis and can be scheduled quarterly, annually, or as appropriate to the facility size and needs.

Process feedback and quality management

Ongoing process feedback from treatment staff provided important information throughout this pilot. Initially, to restrict interview length, the ROM questionnaire focused on last-30-day drug use. Midway through the project, staff requested inclusion of questions to measure alcohol use to intoxication and any illicit drug use after program completion. Staff also requested questions to assess social aspects of client recovery, including activities of their own choosing, emotional problems, and employment or enrollment in school. These questions are in line with social-educational prevention and treatment models where vulnerability to relapse can be ameliorated by reducing key predictive risk factors32—one aim of a community reintegration component. Further, staff recommended a clinically useful reporting format that the center subsequently used during certification by the Commission on the Accreditation of Rehabilitation Facilities (CARF).

The next ROM version will address several points: One item, #14, reads “Since graduating the program have you required additional drug rehabilitation services (not including support groups)?” Use of the word “required” is problematic when the intent is to determine whether additional services were “utilized.” Section III needs a more appropriate scale and wording to adequately assess how many days worked, went to school, or lived in own home or apartment. Finally, question #8, asking about “any illegal drug,” should be further clarified to ... “including prescription drugs used other than as prescribed.”

Initial aggregate data revealed that alcohol use after leaving the program was a potential problem spot for more graduates than had previously been thought. This data led the center to spend more time addressing alcohol-related beliefs and behaviors and addressing alcohol as a drug with subsequent clients. These steps appear to have improved overall outcomes at subsequent time points (data not shown).

Discussion and Conclusions

This report shows the feasibility of a Routine Outcome Monitoring (ROM) system for use in drug abuse treatment facilities. Telephone-based recovery management can address several factors including: (1) Stabilizing and sustaining recovery congruent with an individualized care philosophy; (2) Any stigma associated with returning to a facility after completing treatment; (3) Reaching clients who live at a distance from the treatment facility; and, (4) Potential staff and financial burden.

In contrast to large treatment studies such as the Drug Abuse Treatment Outcome Study (DATOS), the National Treatment Improvement Evaluation Study (NTIES), the Services Research Outcome Study (SROS) and California Drug and Alcohol Treatment Assessment (CALDATA) that invest up to 20 interviewer hours per patient reaching response rates of between 48% and 70%,33,34 meaningful Routine Outcome Monitoring must be accomplished using shorter surveys that focus on core indicators.

Similar projects have also demonstrated feasibility. Oudjeans et al describe an independent telephone center ROM that achieved a follow up rate of 53% at 9-months following intake in a mixed inpatient/outpatient subject group. This project developed a 20-question survey from ASI questions and estimated a per-client cost of $57.35 Tiet et al used a mail-back form developed from the ASI to achieve a follow-up rate of 66.8% at 6-months following intake. This project estimated a $187 per-client cost to follow a mixed inpatient/outpatient/methadone Veterans Administration subject group.36 Stanford, Banerjee and Garner37 tested feasibility of counselor-mediated telephone follow-up aimed at early detection of drug reversion for recovery into treatment. In 6–10 minutes, counselors assessed four questions asking how things are going personally, with family members, at work and school and in life. This was followed by a brief risk-of-relapse assessment based on American Society of Addiction Medicine (ASAM) placement criteria.38 Counselors made 138 calls to contact 32 subjects approximately every four-six weeks.

Data obtained using the ROM system seems adequate for compliance with grant reporting or accrediting agency requirements. Excepting the first follow-up point, this project consistently produced a representative sampling exceeding 80 percent and required very little staff training. Collateral verification in the early part of the project indicated minimal bias from “grateful testimonials”—where the client might not want to hurt the counselor’s feelings by reporting lack of success39—possibly due to using non-treatment staff for follow-up.

This project was undertaken at the request of a program desiring to continuously improve treatment results including those that are not as expected. Engaging treatment program staff in the process of monitoring their own outcomes and incorporating the data obtained into management decisions will likely lead to greater responsibility for improving treatment results. Calculations of treatment effectiveness for purposes such as third-party funding decisions could be strengthened by including a 10% random sample verification by independent telephone-based ROM.

Independent verification of a treatment subsample would provide centers using this system the necessary accuracy and transparency for funding and policy decisions and can be structured to meet HIPAA and other confidentiality procedures. This staff-based component is likely to be the most economical and time-efficient means to provide assistance and recovery management for individual clients, feedback into individual treatment centers, and larger data sets to build the state and national evidence base such as described by Oudjeans.35

This project focused only on those clients who completed the program. Follow-up rates, and especially outcomes data, can be generalized only to those who complete the full course of care. Future projects to further refine this ROM system should explore feasibility with all discharge categories. For quality assurance and program improvement purposes, it is important to obtain data from people who leave the program prior to completion.

Quality treatment should be effective at changing drug use behavior after the patient leaves treatment. The primary objective of any ROM system is to provide an ongoing trail of long-term outcome data with which to assess any changes in treatment outcomes that might reflect changes in quality or therapy drift. While such systems may not be rigorous enough to supplant all other measures of treatment efficacy, they can provide valuable insight into the real-world results that are being achieved. Increasing the reliability of systems that can be used to measure outcomes on an ongoing basis could mark a step forward for rehabilitation clients and providers alike.

The project described in this paper placed a quantitative measure within a continuing care activity. The pilot data in this report shows that existing treatment and administrative staff utilizing this system are able to:

Track the location of a patient using extensive contact information procedures at intake and then verified at discharge to obtain post-treatment contact rates upwards of 80 percent;

Administer a short, telephone-based outcome monitoring survey to either the graduate and/or a member of his or her family;

Collect valid and reliable data that can be used to assess the quality of care that clients originally received by measuring the treatment’s impact on post-treatment drug use behavior and key social correlates;

Use a structured interview to assess individual need for additional treatment;

Utilize aggregate data to detect underperforming areas and strengthen the program.

Good administrative practices starting with enrollment simplify the process of locating past clients suggesting that ROM procedures could likely be used by even smaller facilities. This project demonstrates that a ROM system can be implemented with low staff burden and generate outcomes data that is useful, reliable and valid for most needs.