Research-practice partners assess their first joint project.

Investigators from RAND Corporation and community treatment providers at Behavioral Health Services joined forces to test an intervention to improve services for patients with co-occurring mental disorders. In the course of working together, the partners confronted many of the issues that typify research-practice collaborations in community settings. The researchers' applied theoretical understanding and the counselors' intimacy with patient responses combined to strengthen the intervention. However, counselors' discomfort with some protocols and changes reflecting the extremely dynamic nature of the community-based research setting complicated the study execution and interpretation. Despite these challenges, the intervention improved the counselors' ability to identify and respond appropriately to patients' co-occurring disorders, and one of its components was associated with improved patient outcomes. The experience also demonstrated the advisability of consulting collaboratively with clinic staff during the planning of studies and the pretesting of study protocols.

Katherine Watkins, a researcher at RAND Corporation in Santa Monica, California, set out in 1999 to find a community-based agency to collaborate on a study. Spurred by a conviction that “we have to conduct efficacy research in the actual environments where most substance abuse treatment is delivered,” Watkins had two objectives: to test an intervention for substance-abusing patients with co-occurring mental illness, and to forge a long-term research-practice partnership.

Jim Gilmore, director of patient services at Behavioral Health Services (BHS) in Los Angeles, had been concerned for decades that “the substance abuse and mental health treatment systems both push away patients with comorbid conditions. Each believes the other should intervene first, with the result that these patients never get the help they need.” Having made service improvements for dual-diagnosis patients a primary goal for his agency, Gilmore saw in Watkins’ proposal a chance to excite and educate his staff toward this end. As well, Gilmore saw research-practice collaboration with RAND as an opportunity to “put BHS on the map” as a place to go for innovative, science-based substance abuse treatment.

Gilmore and Watkins agreed to join forces. Their joint project over the next 3 years documented some positive effects from their intervention and suggested that it can enhance patient outcomes (Hunter et al., 2005; Watkins, 2003; Watkins et al., 2004). Along the way, the partners encountered the gamut of issues and dynamics that make research-practice collaborative trials rewarding, but also challenging to execute.

For this article, RAND researchers and BHS administrators and line staff shared their experiences with Science & Practice Perspectives in a series of interviews summarizing the advantages of partnership as well as the difficulties, the solutions, and the lessons learned.

THE INTERVENTION AND HYPOTHESIS

Watkins and Gilmore agreed to test an intervention adapted from a protocol RAND had used successfully with depressed patients. Grounded in the Partners in Care theoretical model (see www.rand.org/health/projects/pic for a full description of the model), the intervention would address the problems of patients with both substance abuse and mental illness on three fronts:

Staff skills – RAND would train BHS staff to identify, assess, and work with patients with co-occurring mental illnesses and to know when to refer them to the Department of Mental Health (DMH);

Patient awareness and activation – The collaborators would implement three specialty groups within BHS to teach patients to monitor their mental health, to control their own symptoms to the extent possible, and to seek help when needed. All study patients would be asked to participate in a “health and wellness” general information group, and patients who reported they had psychiatric problems or were referred by a counselor could join either a general “dual diagnosis” group or one focused specifically on helping patients cope with depression;

Access to the wider system of mental health treatment resources – RAND and BHS would work to facilitate referral links with DMH, with regularly scheduled case conferences to discuss shared patients.

Watkins and her RAND colleagues hypothesized that their intervention, in comparison with BHS’s standard practices, would improve patient outcomes at the end of treatment and 6 and 12 months later. RAND’s previous success with a similar intervention for patients with depression was grounds for hope that the hypothesis would prove out, but they also knew their intervention had two potential drawbacks.

First, it had many pieces, so the BHS staff would have a great deal to learn, implement, and coordinate. Second, any improvements the intervention produced would likely be modest in scale. After all, the researchers reasoned, the BHS staff had little experience working with co-occurring disorders. Moreover, even if the staff did an excellent job, the intervention’s main therapeutic ingredients— patient awareness and activation—could go only so far toward resolving longstanding, deeply embedded problems of substance abuse and mental illness. Because the intervention’s anticipated advantage over treatment as usual was small to begin with, there was a significant chance that weaknesses in implementation or chance events could wipe it out—producing an undeserved null result.

“STRONGEST POSSIBLE DESIGN, GIVEN THE CIRCUMSTANCES”

When possible, investigators usually will test a new intervention with a randomized controlled trial (RCT), which provides the best possible evidence that the intervention being tested, rather than some unidentified chance factor, is the cause of any improved outcomes. However, logistical considerations ruled out use of an RCT design for this project. The researchers settled instead on a quasi-experimental design. They would deliver the intervention at the largest BHS clinic, located in Gardena, California, and use BHS clinics in nearby Inglewood and Wilmington as comparison clinics.

A concern in this type of study is that selecting patients for treatment and control groups nonrandomly—in this case, assigning them on the basis of which clinic they attended—leaves open the possibility that inherent differences between the groups may affect the outcome independently of the intervention. Such differences would threaten the study’s internal validity and the researchers’ ability to understand the impact of the intervention. For example, the patients in the Gardena clinic, who would receive the intervention, were more racially mixed than patients in the comparison clinics in Inglewood (roughly half African-American and half Hispanic) and Wilmington (nearly all Hispanic). Researchers would therefore need to think carefully about the possibility that racial differences might be at least partly responsible for any differences they might find in patient outcomes.

“We chose the strongest possible design, given the circumstances,” says Suzanne Wenzel a principal investigator on the study. “Of all BHS’s clinics, we chose for our comparison sites the ones that were most similar to the Gardena clinic. We also were vigilant in monitoring what was happening in the clinics at all times, trying to notice everything that might happen to affect the study outcomes.”

“All in all,” says Sarah Hunter, RAND’s conductor of qualitative evaluation,“I don’t think the trial design had a big impact on our findings. We were able to control for many of the observed differences across the clinics in our statistical analysis.”

PRESENTING THE STUDY TO THE LINE STAFF

Community clinic staff members generally have an array of reasons to be enthusiastic, cautious, or opposed to invitations to become involved in research. When Watkins and Gilmore presented their plans to the BHS line staff, they encountered all three reactions.

Yolanda Farley, program director at the Gardena site, voiced the most common response—she welcomed the project: “I was really jazzed. Personally I have always liked the cognitive part of treatment. I also remembered that in 1992 or ’93, Sara [Huish, a counselor at Gardena] came back from a convention saying, ‘The face of treatment is getting ready to change. Mental health is going to be a big part of it.’ She was right—we have since seen mental health become a major part of substance abuse therapy.”

A second staff contingent took the news cautiously, keying on the fact that there would be unpaid extra work. At the Gardena clinic, along with recruiting patients and delivering the intervention, the staff would assume a hefty burden of schedule adjustments, meetings, and recordkeeping.

Counselors at the comparison clinics would incur fewer additions and disruptions to their normal practice, but also would receive less professional satisfaction. Roselva Romero, program director at the Inglewood site, says, “My counselors were disappointed because the counselors at Gardena would get the training and do the intervention, while we only did the screening part.” To get buy-in from these counselors, the RAND team promised to train all BHS staff in the mental health intervention after the study.

A few of the staff rejected the intervention and its goal. “The traditional concept of substance abuse treatment stigmatizes mental health,” Farley says. “Some of our veteran counselors were stuck in this mindset, and a couple of them resigned.”

THE PARTNERSHIP: CONTRIBUTIONS AND COMPROMISES

Research-practice collaborators invariably find they have to adapt their communication styles to comprehend each other. As an example, while researchers are well accustomed to obtaining information and ideas through lectures and reading, counselors may learn more easily through demonstration and practice. In the RAND-BHS study, Watkins reports, “The staff told us right off they would rather we change from lectures to more experiential training, with group discussions and role playing.”

Researchers and clinicians share the same ultimate goal—helping people overcome drug abuse and addiction so that they can lead healthy, productive lives— but the two groups’ sensibilities and short-term agendas can diverge or clash. As a result, BHS staff clinicians sometimes provided advice that strengthened the intervention and its implementation. They also sometimes asserted differences that required negotiation and compromise.

Where Perspectives Converged: Training and Design

Early in the staff training phase of the study, the counselors pointed out to RAND trainers that the classes they were providing did not fully reflect the researchers’ own philosophical foundation. Says Watkins, “The staff said the training was focused too narrowly on mental health, that it should truly integrate mental health and substance abuse issues. They were absolutely right; after that, our training got better.”

Drawing on their unequalled familiarity with patients’ views and attitudes, the BHS counselors contributed improvements to the specialty mental health groups that were a key part of the intervention. Watkins says, “We originally called our self-awareness group for patients ‘Mental Health 101.’ The counselors told us, ‘You’ll never get anybody to sign up if you call it that.’ The substitute name they gave us—‘Health and Wellness’—worked much better. The counselors also helped us solve an early problem with this group: Some clients were saying, ‘This material doesn’t apply to me,’ and tuning out. The counselors told us to add sleep and nutrition to the course content, because every client recognized that good sleep and good nutritional habits mattered to them.”

Where Perspectives Diverged: Recruitment Protocols

The protocols for patient recruitment raised the most numerous and consequential differences between the RAND researchers and BHS counselors. The underlying issues were fundamental enough to potentially affect almost any research-practice collaborative study (See “Key Challenges That Arose During the Interventions: Underlying Issues, Researchers’ Responses, Impacts, and Lessons”).

The RAND team enlisted BHS intake personnel to recruit patients—a deviation from usual research practice, but one the researchers felt was necessary. “In traditional research, investigators hire their own recruiting staff,” explains Hunter. “The recruiters’ entire job is to present the study to every eligible new patient who enters the clinic. But we wanted our intervention to be one that BHS could sustain with their own resources after the study when we—and our funding—left the scene. Therefore, their intake staff needed to be able to present our intervention to the patients along with everything else they do.”

The investigators trained the BHS intake staff to invite each new patient to participate in the study. This involved giving each new patient a “screener” packet with information on the study and helping willing patients fill out a baseline health questionnaire and a consent form.

Months later, recruitment lagged far behind schedule, raising the prospect that the researchers might not have enough data to reach any conclusions at all at the end of the time allocated for the study. Questioning the staff to learn why so few patients were signing up, the researchers discovered three primary concerns.

Inducting Versus Recruiting

The clinical staff sometimes omitted telling new patients about the study out of concern that doing so might interfere with treatment induction. “Clinicians are employed to get patients into treatment,” says Patricia Ebener, RAND’s chief data collector. “They’re willing to go along with research protocols to a degree, but not when doing so threatens their prime mandate. Now, imagine that you have a client who’s agitated and on the fence about making the commitment to treatment. You have managed to keep her from bolting throughout an hour-long intake process, and now you’re supposed to ask her to fill out a study questionnaire. You may well worry that if you do, it’ll be the last straw. You might very well decide to drop the screening, rather than add to the risk of never seeing her again.”

To resolve this issue, the RAND team agreed to let intake staff introduce the study and screen at any time during the intake interview instead of always at the end, as originally specified. Although this method might produce a study sample of patients that was less than ideally representative of BHS’s normal population, it added a valuable element to recruitment: the counselors’ clinical judgment concerning when clients might be receptive.

Informed Consent: Vagueness Versus “Honesty”

The BHS intake staff objected to the wording of RAND’s informed consent form. Some doubted patients could understand its legalese. As well, Ebener says, “One group of counselors didn’t like the fact that the forms presented our study as being simply about health. They thought we should tell patients it was about mental health, that it was dishonest not to do so.”

“Counselors tend to believe that being honest with oneself is the most essential step in recovery,” James Morrow, program director at the Wilmington site, comments, “so acting any way other than completely up-front with patients is the worst example they can set.”

RAND researchers had minimal room for compromise on this issue, says Ebener. “One goal of our study was to see how well the counselors, once they were trained, could tell which clients in their normal case mix had mental health needs. We wouldn’t be able to find that out if the recruiters inadvertently changed the mix by enrolling mostly patients with mental health problems. That’s what might happen if they disclosed that the study was about mental health. Patients who knew or thought they had mental health issues might enroll, while others might opt out. And patients with or without mental health issues might shy away for fear of the stigma associated with mental illness.”

The RAND team explained why the recruitment pitch had to be vague about the study’s goals. Hoping they had made their point, to get recruitment back on track, they conceded to the counselors the flexibility to explain the consent form in their own words. Recruitment improved after that, but—with counselors presenting the study to patients ad lib—the investigators henceforth felt insecure about potential bias in their participant pool.

“In my mind, the way the staff presented the study to new admissions will always be a question mark,” says Ebener. “In the future, when we evolve our consent protocols for such studies, we will pay more attention to the experiences the counselors have to offer.”

Compensation

At the outset of the study, the RAND team paid BHS for staff time spent in recruitment. This system of compensation proved inadequate as an incentive. It also failed to provide counselors with any guidance on an issue some found vexing: How hard should they press patients to join the study?

The investigators had worked out their compensation system in consultation with RAND’s institutional review board (IRB), a committee charged with overseeing the technical and ethical soundness of the project. When the system did not work, the team obtained the board’s permission to modify the pay formula and give $5 for each screen handed out. The intake staff would receive the money even if the patient checked “not interested” on the cover sheet.

“The modified incentives clearly signaled to the staff that the idea was not to try to persuade or coerce the client, but just to present the information and let the client make up his or her own mind,” says Ebener. “But the cost was much higher.”

After the major recruitment issues were resolved, the BHS staff obtained signed initial questionnaires from 90 percent of new clients. Ultimately, however, added costs and time expended during the period of slow recruitment forced the researchers to drop planned preintervention data-gathering and a planned 12-month followup.

COMMUNITY DYNAMISM

The dynamism of community clinic environments is the antithesis of the stability and control investigators enjoy in traditional research settings. The RAND-BHS collaborators encountered one very common problem—high staff turnover—and others that vividly illustrate the wisdom of anticipating the unexpected.

Staff Turnover

Staff turnover rates of 50 percent annually are common in community substance abuse clinics. BHS proved no exception to the rule of perpetual staff replacements: During the RAND-BHS collaboration, the individuals filling more than half of the dozen positions with roles in the study changed, in some cases several times.

“Some counselors left voluntarily, some involuntarily,” says Gilmore. “There came a point when we recognized turnover could have an impact on the study. We alerted RAND, but there wasn’t much to say except, ‘Brace yourselves, because it’s probably going to keep happening.’ We tried to maximize the use of the continuing staff and ease the new folks into their study tasks, but there were times, especially with screening, when we couldn’t do that.”

“We were constantly training new staff, which is an expensive proposition,” says Hunter. “Another consequence was that, whereas original staff members had received their training prior to starting to deliver the intervention, their replacements had to learn the intervention and perform it at the same time.” With partially trained counselors delivering the intervention, the researchers had further cause to worry that its anticipated modest margin of advantage over treatment as usual might dwindle to nothing.

Comparison Site Quality Improvement

In a typical new intervention study in a traditional research setting, investigators will establish a “state of the art” standard for the comparison treatment and stick to it. During BHS’s collaboration with RAND, the treatment program’s continuous quality improvement created a problem for the researchers.

“One day when I visited the Wilmington comparison site, staff members told me they had forged a great relationship with their local mental health provider,” recalls Hunter. “They were exchanging referrals and communicating with this provider on a regular basis. In effect, they had duplicated one component of our intervention.”

Wilmington’s choice of initiatives, Gilmore says, was partly in response to a Los Angeles County policy of encouraging substance abuse programs to enhance services for patients with co-occurring mental disorders. Hunter believes competitiveness also may have played a role. “In a double-blind study in a university research setting,” says Hunter, “the clinicians delivering treatment and the research participants often don’t have any idea whether they are in the treatment or control arm. In our study, though, the people in the comparison sites were well aware that our study was about mental health and knew they were being compared to Gardena. They wanted to look good.”

Removing one difference between the intervention and comparison treatment would make it harder to demonstrate an advantage for the intervention. Nevertheless, the researchers could only accept it. “I made it very clear that clinicians in all the clinics were to deliver the best care they could,” says Watkins. “Ethically, you can’t say anything else. You can’t encourage health care providers to not try to do better for their patients.”

Systemic Tension and Conflict

Investigators working in a university or teaching hospital usually can count on cooperation among the entire study team. In community settings, the interfaces between professional groups may generate friction.

Months into the study, Gardena counselors complained that their liaison with DMH was not going smoothly. Consuela Jackson recounts, “We were working to become better at diagnosing and assessing clients in need of psychiatric treatments. We identified a fair number of people and referred them to the DMH. That was part of the intervention, but it turned out the DMH providers weren’t keen on opening the door for our patients.”

“In their own way, I think the DMH providers have the same problem with folks with co-occurring disorders that our folks can have,” says Gilmore. “Each side thinks the other should deal with these people first.”

Farley felt that arrogance also contributed to the problem: “Substance abuse counselors are labeled as paraprofessionals, mental health people as professionals. Some DMH personnel were exhibiting the attitude, ‘How dare you presume to know this client needs to be referred to us?’”

The RAND team investigated and discovered that one DMH employee— unfortunately, the team’s main liaison at the referral clinic—was the main source of the difficulty. The partners quickly established new procedures to work around this individual, but some damage had already been done. While the problem lasted, poor coordination of care had reduced the intervention’s potential to enhance patient outcomes.

Huish says the discord ultimately cost patients the most: “I felt like we worked to fight against the stigma of mental illness, to open our clients’ minds to the point where they could accept treatment, and then DMH closed the door on them.”

Proposition 36

Community clinics often have to adapt their operations quickly and extensively in response to local, State, and national policy decisions. During the RAND-BHS collaboration, California voters approved a referendum—Proposition 36—that gave most nonviolent drug-abusing criminal offenders the option of entering substance abuse treatment instead of prison. The impact on the study was substantial.

BHS, like treatment programs throughout the State, began receiving a wave of patients with histories of crime and incarceration. During the course of their institutionalization, many of these new clients had been linked to mental health care.

“One of the goals of our intervention was to increase patients’ use of mental health services,” says Hunter. “Obviously, it couldn’t do that if patients were already in treatment. After Proposition 36, about one in four new BHS patients already had a mental health provider. That diluted our chances of demonstrating an advantage for the intervention.” Working with the mental health providers in the criminal justice system was also difficult, as these providers were geographically dispersed and did not know about the study.

“Investigators doing research in community clinics need to be braced for all kinds of shifts,” says Ebener. “There is a seasonality to these programs’ operations. Policies change, funding sources change. As we found out, even population bases can change without anyone being able to predict or control it.”

THE RESULTS: TWO VIEWS

When the RAND team analyzed their data, they concluded that the Gardena counselors had understood and implemented the intervention well. Based on interviews at the beginning, midpoint, and end of the study, supplemented by examination of patients’ charts, the counselors had clearly improved in some key respects: For example, they more often detected patient mental health issues, addressed these issues in the treatment plan, and appropriately referred patients to mental health providers.

The researchers found one statistically significant indication that their intervention improved patient outcomes: Among patients in the Gardena clinic with mental health issues, those who had attended more specialty mental health group sessions reported fewer symptoms and less drug use 6 months into treatment.

The data comparing outcomes between Gardena and the two other clinics were ambiguous: Gardena patients reported better mental health and drug abuse outcomes, and they stayed in treatment longer. However, these advantages were not large enough to demonstrate the intervention’s superiority with statistical certainty.

The RAND team also analyzed whether the patients who received the intervention engaged in more health-promoting activities as a result. They found that over the course of the study, Gardena patients increased their use of mental health services and psychiatric medications more than patients in the comparison clinics. However, despite the intervention’s focus on patient education and referrals, the difference amounted to only a statistical trend, not a proof of effect.

The RAND team shared these results with BHS staff. Not surprisingly, researchers and clinicians looked at them in somewhat different lights and took away overlapping but somewhat contrasting lessons.

The Investigators’ Conclusions

From their data, RAND researchers concluded that their intervention had successfully raised the BHS staff ’s skills in treating patients with substance abuse and mental health issues. They had not, however, achieved the benefits they had hypothesized.

“Our intervention achieved its first objective—improving the Gardena staff’s understanding and responses to patient mental health issues,” says Watkins. “Those improvements, if they are large enough, should lead to better patient outcomes. Unfortunately, several features of the community clinic setting undercut our ability to demonstrate better outcomes. Staff turnover and other issues made for inconsistent implementation. Other events pared down the differences between the intervention and comparison treatments over the course of the study.”

Ultimately, says Watkins, “Our intervention’s main benefit turned out to be something we didn’t originally conceptualize as one of its goals: Participating BHS staff began to think more like health care professionals. Instead of saying, ‘I recovered this way, so that’s how my patients have to recover, too,’ they learned to elicit objective information and use it to generate patient-specific treatment plans. For example, they performed mental status exams and considered which BHS interventions would best fit each patient’s mental health and substance abuse symptoms.

“This cultural change is incredibly important to move substance abuse treatment forward. It was already under way at BHS, but the intervention greatly accelerated it.”

The Clinicians’ Conclusions

The Gardena staff came away from the study convinced that the intervention markedly improved patients’ progress in recovery. Huish expresses the general feeling: “With this intervention, I saw people staying longer and becoming stable while taking medication. We saw clients improving in their cognition to where they could start to comprehend the goals and objectives of their treatment plans. Progress occurs, I believe, when a person improves his or her way of thinking, when hallucinations become less intrusive, when people are less tied up with their psychiatric symptoms.”

“The staff members were surprised when RAND showed them the results of the data analysis,” says Gilmore. “Their reaction was, ‘How can this be? We know our clients are getting better.’ We’ve been having discussions to try to explain it.”

One proposed explanation echoes the RAND team’s surmise that flaws in implementing the intervention diluted its efficacy. “The Gardena staff feel their clients’ progress didn’t show up because we started collecting data before they got good at performing the intervention,” says Hunter. “They feel the data don’t reflect their current practices, and that the intervention is now working. I think there’s probably some truth to that.”

BHS staff members also suggested that outcome measures different from those used in the study might have been better suited to capturing patients’ progress. For example, based on their clinical experience, the counselors questioned the researchers’ use of a global question about patients’ quality of life. “Patients generally said their quality of life hadn’t improved, but I think that is to be expected from addicted patients in early recovery,” says Shirley Summers, head of clinical services at BHS. “Patients are clean and sober, but not used to it. Even though objectively their lives may be better—they may have a job and a car, their relationship may be back together—they’re probably still going to feel uncomfortable just being in their own skin.” The staff suggested that the quality of patients’ participation in therapy might provide the best indicator of their progress in early treatment.

GAINS AND FUTURE OF THE COLLABORATION

RAND and BHS are well along in discussions about their next collaboration. “From the beginning, this study was part of a larger project to build a partnership with BHS for the long run,” says Watkins. “We anticipate that the collaboration will grow easier with each new joint endeavor.”

The next collaboration will probably also focus on mental health and drug abuse comorbidity. Gilmore says gains from the first project will enable the partners to start the next one further along: “Treating co-occurring disorders is now the standard at BHS. As a result of the study, our counselors are more aware of these patients, and they have more tools to help them. The research also verified how important it is to do this: Around half the folks who walk through the door at Gardena have more than one disorder. Now that we are more attuned, we are also finding that often, once someone’s substance abuse clears a little bit, it becomes very apparent that he or she has a co-occurring disorder.”

The staff in all three participating BHS clinics expressed enthusiasm about the study already done and the ones to come. “It was a lot of work, but time well spent,” Jackson says. “The experience boosted my confidence and effectiveness with clients who come into treatment with multiple histories. I had learned about dual diagnosis in school, but RAND’s training was more extensive. I had prior experience with cognitive-behavioral therapy, but it was nothing like working alongside the RAND psychologist.”

In their next try, Watkins will apply the major lessons taken home by the RAND team: “Community-based research designs must be extra hardy. In future studies, we will field an intervention and recruit a sample large enough to absorb more of the dynamics of community environments and still show changes in outcomes. We won’t start evaluating the intervention as soon as we put it in place, as our funding timeline forced us to do here. Instead, ideally, we’ll take 6 months to pilot-test the intervention with the staff, get their feedback, and work out the kinks.

“Community-based research is difficult—it takes a long time, and it can be costly, but we must do it. There is only so much we can learn about treatment effectiveness in laboratory and university settings.”

SUMMARY AND LESSONS

In this research-practice collaborative study, RAND researchers and BHS community treatment providers complemented and challenged each other. The researchers fielded a better intervention thanks to the counselors’ input, and the counselors gained skills in identifying and assisting patients with co-occurring mental health disorders.

The partners encountered many challenges. Some arose because of the somewhat conflicting goals of research and treatment: For example, the researchers wanted clinicians to always present the study to new patients at the end of the intake interview, while the counselors wanted to present it when their clinical sense told them doing so would not interfere with treatment induction. Both positions had important practical rationales: Researchers try to eliminate differences between patients’ experiences that can bias study results, while clinicians seek to establish a therapeutic relationship with each patient on an individual basis. Although negotiating differences may never be simple when such fundamental principles come into play, the partners learned that it’s vital to engage all levels of participating staff very early in the planning of a study.

A second set of issues reflected changes within the community treatment environment. These varied in their degree of predictability—for example, the RAND-BHS partners can anticipate that counseling staff turnover is likely to be high in future studies they undertake together, but systemic changes on the order of California’s implementation of Proposition 36 are likely to always have impacts that are very hard to foresee. The takeaway lessons from these experiences are the need to be alert to changing situations throughout the planning and conduct of studies and the advisability, if possible, of building in reserve capacity—in staff training, time, funding, and statistical power relative to the anticipated effectiveness of the intervention being tested.

RAND and BHS conceived of this study as the initial project in a long-term partnership. Thanks to the lessons learned from their first experience, the partners will start their next project—already in planning—further forward. RAND researchers will incorporate an expanded awareness of the community treatment culture and environment. BHS counseling staff, say all concerned, will start the next project with a more scientific outlook, marked by a new level of commitment to evidence-based clinical decisionmaking.

KEY CHALLENGES THAT AROSE DURING THE INTERVENTIONS: UNDERLYING ISSUES, RESEARCHERS’ RESPONSES, IMPACTS, AND LESSONS

EVENT	UNDERLYING ISSUE	RESEARCHERS’ RESPONSE	POTENTIAL IMPACT ON STUDY RESULT	LESSON
Recruitment fell behind schedule.	Intake staff felt the protocol for recruiting patients into the study might hinder treatment induction.	Relaxed the protocol: Allowed intake staff to present the study at any time during the intake interview, rather than only at the end.	Less uniformity in presentation of study may have resulted in recruitment of a nonrepresentative sample of patients.	Involve staff at all levels in early planning for study. Counseling staff are likely to resist enacting any protocol that may interfere with their primary work goals—in this case, motivating often-ambivalent drug abusers to enter treatment.
	Intake staff felt patients might not understand informed consent.	Allowed intake staff to present informed consent materials in their own words.	Same as above.	Same as above. Counselors tend to be protective of their patients’ interests, and may interpret them in ways different from researchers.
	Intake staff felt the informed consent did not tell patients enough about the purpose of the study.	Explained why the consent needs to be vague, but conceded enough leeway in the presentation to make the intake staff comfortable.	Same as above.	Same as above.
	With the original compensation plan, counselors were uncertain how hard they should press patients to participate in the study.	Revised the plan to link compensation to number of recruitment pitches to patients, but kept it delinked from patients’ agreement or refusal to participate.	Same as above.	Same as above.
Trained staff left the program.	Like many community clinics, BHS has turnover rates approaching 50% annually among line counselors.	Trained incoming staff as quickly as possible.	Added study costs and time requirement. New staff learned the intervention and delivered it at the same time, potentially reducing its effectiveness.	Anticipate high staff turnover; if possible, train reserve staff in treatment interventions.
A comparison clinic instituted new practices that duplicated part of the trial intervention.	The comparison clinic was practicing continuous quality improvement and responding to encouragement from the county to enhance services for the mentally ill. Counselors may have felt competitive with those in the intervention clinic.	Accepted the fait accompli, believing it would be unethical to discourage the comparison clinic staff from trying to improve their services.	The study was unable to show whether or not the duplicated part of the intervention was effective. The study’s ability to show that the entire intervention was superior to standard care was reduced.	Closely monitor organizational practices at all participating study sites to avoid unwanted influences on outcomes.
The percentage of patients who were already receiving mental health care upon presentation to BHS increased.	California’s Proposition 36 mandated the option of treatment as an alternative to prison for nonviolent offenders, many of whom were linked to mental health services in the course of prior institutionalizations.	Accepted the situation.	Dilution of positive intervention results. The inclusion of patients who already had a mental health provider made it difficult to demonstrate the advantages of the intervention.	Anticipate the unexpected.