Self- Evaluation Tool to Support Implementation of Treatment for Tobacco Use Disorder in Behavioral Health Programs.

Tobacco use in people with behavioral health conditions remain two to three times higher than the general population causing premature death and impacting recovery negatively across several domains. Intermediary organizations can provide practical tools, training, and technical assistance to help programs improve capacity to treat tobacco use. This report describes the construction and application of the Tobacco Integration Self-Evaluation Tool (TiSET) for behavioral health programs, a 20-item scale inspired by the DDCMHT and additional content from the Facility Tobacco Policy and Treatment Practices Self-Evaluation tool that one of the study authors (JW) used previously with addiction treatment programs. Completing the TiSET is an important step for behavioral health programs to evaluate their ability to effectively treat people that use tobacco. An important next step is to use those results to facilitate a quality improvement process. We include large agency example illustrating how the TiSET can be applied in real-world practice.

Introduction

Tobacco use rates in individuals with a behavioral health condition remain two to three times higher than the general population (Lasser et al., 2000; Lawrence et al., 2009; CDC, 2013) and this group consumes at least a third of the tobacco sold in the United States (Grant et al., 2004; Lawrence et al., 2009). Tobacco causes a staggering 50% of deaths in those with serious mental illness (SMI; Callaghan et al., 2014) and kills more substance users than their primary substance (Hurt et al., 1996; Veldhuizen & Callaghan, 2014). Tobacco also threatens recovery by impacting negatively on finances (Steinberg et al., 2004), employability (Houle & Siegel, 2009), housing (Vijayaraghavan et al., 2016), mental health symptoms (Taylor et al., 2014) and abstinence from other substances (Prochaska et al., 2004; Weinberger et al., 2017). Smokers with behavioral health conditions have not benefited as greatly from public health and tobacco control efforts making them a high-risk/high-cost population in urgent need of intervention (Lê Cook et al., 2014; Lawrence & Williams, 2016).

Adding to the disparity, behavioral health systems have been slow to adopt tobacco-related policies (e.g., tobacco-free clinic grounds) and requirements for clinical interventions. Exact reasons are unclear, but barriers cited by psychiatrists include that the people they serve have more immediate issues to address and are not motivated to quit and that behavioral health staff are unfamiliar with cessation interventions (Himelhoch Riddle & Goldman, 2014). Smoking was historically used as a behavioral reward in psychiatric inpatient units and continues to serve as a shared social activity for many people with behavioral health conditions. The result is that smoking became an accepted part of the culture in most mental health and addiction treatment centers and residential facilities.

Despite some progress in problem recognition, tobacco is not treated as a co-occurring disorder, and buy-in among staff is still low. Stigma contributes to the belief that this group can’t or won’t give up tobacco because it’s “all they have”. Even groups that champion recovery models for overcoming behavioral health disorders seem to undermine smokers’ sense that they can recover from tobacco dependence as well. Interestingly, similar claims are not made to justify use of other addicting and deadly substances in behavioral health care. Less than half of mental health sites in the US even ask people about smoking (Himelhoch et al., 2014; Marynak et al., 2018). An additional contributing factor may be the knowledge deficit of evidence-based practices for treating tobacco among psychiatrists and behavioral health staff (Ratschen et al., 2009; Williams et al., 2015, 2019). Despite barriers, behavioral health providers are well suited to deliver intensive tobacco cessation treatments, as these professionals have experience in treating other substance use disorders and are experts in behavioral therapies. The co-occurring model to integrate tobacco into usual treatment, matching interventions to motivational level, and taking a long-term perspective, would be beneficial, as it is with other substance use disorders.

While training is necessary, it is just one component of an overall approach of systems change. Implementation science shows that processes for addressing the local culture and engaging leadership, as well as the specific processes for change (including having champions and developing planning and evaluating) are essential for the success of the initiative (Miech et al., 2018). Through a comprehensive system change effort, clinical treatments for tobacco use disorder can be implemented, and policies can be revised to address the environment of care, support treatment and de-normalize the use of tobacco.

Program changes can occur in all kinds of settings and at all levels of care (outpatient, inpatient, residential, etc.), and can be implemented as part of an institutional change plan with specific goals, deadlines, and dedicated staff. The changes may be limited or expansive, involving brief or easier steps, or more comprehensive ones that institute a range of treatment services and place limits on staff and client tobacco use. A program might consider the ultimate step of creating an entirely tobacco-free treatment environment that reinforces the message of health and hope, to recipients of services.

Although interested in improving services for tobacco use disorder, behavioral health programs may lack the pragmatic guidance on how to change. Several models now exist describing the recommended steps to address tobacco in a behavioral health program, although many emphasize the development of tobacco-free environments and treatment grounds (Hoffman et al., 1997; Williams et al., 2005; Guydish et al., 2012). We adapted these steps into a program model that emphasizes incorporating tobacco treatment into a behavioral health program, in order to provide the staff with the tools needed to intervene clinically. Creating an implementation plan that identifies the tasks and actions needed to move the program towards fully integrating tobacco treatment into program services is perhaps the most important step in this process.

The New York City Tobacco Cessation Treatment and Technical Assistance Center (TCTTAC; https://nyctcttac.org/) is an intermediary organization providing training and technical assistance to New York City (NYC) behavioral healthcare providers to improve their capacity to treat tobacco use disorder in the people they serve. Intermediary organizations (Franks & Bory, 2015; Proctor et al., 2019) work with programs, as part of the implementation team (a group of people, including program staff and administrators), tasked with ensuring that implementation “happens” (Fixsen et al., 2019). Funded by the NYC Department of Health and Mental Hygiene, TCTTAC is designed to address certain barriers to tobacco treatment by ensuring that providers have the necessary training and organizational support to use skills and knowledge about treatment of tobacco use disorder in their daily work. Since 2018, TCTTAC has trained 472 providers from 127 programs representing 46 agencies in NYC.

A central feature of the TCTTAC model is support for the behavioral health program to develop an implementation plan that includes goals, tasks and action steps As we began working with programs to implement integrated treatment for behavioral health and tobacco use disorder, we quickly discovered that it would be helpful to have some measure of self-assessment and fidelity that would identify strength and deficit areas and allow for tracking progress. However, we were unable to locate a validated measure in the literature.

The Dual Diagnosis Capability in Mental Health Treatment (DDCMHT) index was first developed in 2004 and is a reliable and valid standardized measure developed to help programs improve their ability to provide integrated treatment for co-occurring mental health and substance use disorders (Gotham et al., 2009, 2011). The DDCMHT demonstrates excellent total score reliability, variable subscale reliability, high inter-rater reliability, and moderate construct validity (Gotham et al., 2013) and has been used to measure systems implementation and guide program change (e.g., Gotham et al., 2010; McGovern et al., 2010; Chaple & Sacks, 2016). The DDCMHT evaluates 35 program elements in dimensions such as program culture, assessment, treatment, and training and supports for staff. Conducting an initial DDCMHT assessment serves as a “baseline” measure of capability to treat co-occurring disorders, and allows programs to go on to develop co-occurring treatment implementation plans akin to typical treatment plans that include goals, objectives, interventions, responsible persons, and projected target dates. Since the measure can be re-administered, it can also be used to measure the success (or sustainability) of these changes (Covell et al., 2021a, b).

Although the DDCMHT is widely used for co-occurring disorders, we sought to develop a modification with items specific to tobacco use policy and treatment, which could be used to guide and evaluate the implementation of treatment for tobacco use disorder in behavioral health settings. An advantage of using the DDCMHT is that our approach to addressing tobacco defines tobacco use as fundamentally a co-occurring substance use disorder for individuals with mental illness. The TCTTAC goal is to engage all individuals who use tobacco in treatment and work with those who are at different levels of motivation to change. Language around tobacco “cessation” is avoided since it engages only those who are in the preparation stage and ready to take immediate steps.

Method

Scale Construction

We set out to create a self-evaluation tool that was inspired by the DDCMHT and included additional content from the Facility Tobacco Policy and Treatment Practices Self-Evaluation tool that one of the study authors (JW) used previously in other work on the implementation of tobacco treatment into substance abuse services (Williams et al., 2005).

We began by examining the 35 organizational elements that fall into one of seven dimensions covered by the DDCMHT (Gotham et al., 2011) to determine which were most critical for addressing tobacco use disorder. Similar to the first dimension of the DDCMHT, Program Structure, which includes whether program characteristics such as mission statement, licensure, and financial incentives that support integrated treatment, we created a “policy and administrative” section that included two items related to the presence of tobacco use policy and capacity for tracking tobacco related outcomes. Paralleling the second dimension of the DDCMHT, Program Milieu, which includes whether the program routinely welcomes people with co-occurring disorders and the availability of materials to address both disorders, we created an “Environment” section that included one item each examining the availability of tobacco educational materials and signage related to smoking on or near the premises. We retained the third and fourth dimensions of the DDCMHT, Assessment and Treatment, with five items each detailing whether specific clinical activities around screening, assessment and treatment support treatment of tobacco use. We did not include the fifth dimension of the DDCMHT, Continuity of Care, which focuses on continuity of care after program discharge since we were working with mainly outpatient programs. We did, however retain the sixth and seventh dimensions of the DDCMHT, Staff and Training, in which we included four items related to the availability of clinically appropriate team members (e.g., psychiatrists, peer providers), access to supervision and coaching, and two items related to staff training around treating tobacco use disorder.

This process produced the Tobacco Integration Self-Evaluation Tool (TiSET) for behavioral health programs, a 20-item scale covering 6 dimensions (available through our website at https://nyctcttac.org/). The 6 Dimensions of the TiSET are Policy/ Administrative, Environment, Screening/ Assessment, Treatment, Staff and Training. One of the strengths of the DDCMHT is clearly defined anchors for rating, and we wanted to retain the anchors and scoring system for the tobacco self-evaluation tool. Each item on the DDCMHT receives a score from one through five, and averages of items within subscales and across all items create subscale and overall scores ranging from one to five, respectively. A score of one is considered “mental health services only” (able to treat only people with mental health disorders), a score of three is dual diagnosis “capable” (able to treat most people with both mental health and substance use disorders), and a score of five is considered dual diagnosis “enhanced” (able to treat all people with both mental health and substance use disorders, including those with the highest level of symptoms and/or impact from their co-occurring disorder) (Center for Substance Abuse Treatment, 2005; Gotham et al., 2011), with scores between indicating that a program is making progress toward the next level (e.g., two indicates a program is between mental health services only and capable). We revised the anchors for the retained items to be specific for tobacco use and retained the scoring convention of one to five (i.e. a score of three is tobacco “capable”, able to provide some services to individuals with co-occurring tobacco use and mental health conditions).

Procedures

To help programs complete the self-assessment, we created several supporting documents including steps for completing the self-evaluation, interview guides for talking with program leadership and clinical staff, and a chart review form (all available through our website at https://nyctcttac.org/). These documents, which parallel DDCMHT tools (Gotham et al., 2011), provide stepwise guidance for completing the self-evaluation. An important first step is to convene a workgroup that will plan and carry out the self-evaluation process. The workgroup should be composed of representatives from different roles within the program or clinic including program directors, prescribing clinicians, clinicians, care managers, peer specialists, support staff, and clients. This representativeness is critical to ensure the inclusion of different perspectives on the program’s practices and procedures.

Once the workgroup has been identified, it is important that the workgroup communicate the purpose of the self-evaluation to all staff and clients. This should emphasize that the self-evaluation is meant to help the program/clinic learn about its current services for persons with tobacco use disorders, and to identify its strengths and possible areas for improvement. It is important to stress that the self-evaluation is a collaborative process and not an audit comprised of criticisms and judgments.

The workgroup then selects the sources of the information from which to rate the 20 items on the TiSET. The self-evaluation uses observational methods in addition to chart reviews to gather information. Sources of information should include: direct observation of the milieu; tour of the physical settings; input from clinical team meetings; focused but open-ended interviews with key informants including program directors, supervisors, prescribing clinicians, nurses, clinicians, care managers, support staff, and clients; review of documentation, medical records, policy and procedures, brochures and other handouts. Input from peer counselors, families and tobacco users may be particularly valuable in providing perspective and initiating meaningful dialogues with stakeholders.

Prior to the evaluation, the workgroup should review and understand the definition and response coding for each TiSET item. Scheduling the evaluation should include time for 2–3 workgroup members to conduct separate group interviews with the program/clinic leaders and supervisors, select clinicians, and clients. Separate interviews allow for different perspectives on the program’s practices and procedures. The schedule should also include time to tour the program’s physical site and time to review documents, such as brochures, policy and procedure manuals, and other pertinent materials. Time should also be scheduled to review eight to ten medical records, all for individuals who currently use tobacco. Specific forms to conduct chart reviews or question administrators assist in data gathering. The program self-evaluation will take between a half to a full day to complete. It is important to allocate sufficient time to do the TiSET self-evaluation. Information collected from the various sources is used to rate the 20 TiSET items, with workgroup members resolving any discrepancies between data sources to reach consensus. A larger agency can decide to conduct a TiSET self-evaluation at the individual program level to assess differences that may occur across the continuum of care (for example, in the outpatient versus housing setting).

Reviewing the TiSET Report

Programs in NYC that complete the TiSET self-evaluation submit their ratings online to our team via a Qualtrics form. The TCTTAC team scores the self-evaluation creating an average item score for the scale, overall, and within each of the six dimensions. The resulting overall and subscale scores range from one to five, with a score of one being considered “mental health services only,” a score of three, “capable,” indicating the program is able to provide some services to individuals with co-occurring tobacco use and mental health conditions, and a score of five, “enhanced”, where the program is able to provide services to all persons with co-occurring tobacco use and mental health conditions. When working with agencies, the TCTTAC team also creates average scores across participating programs or participating program types, depending upon the needs of the agency. The TCTTAC team creates a five-page program-specific summary report with detailed recommendations for each item, based upon the program’s rating. The report provides both written and graphic feedback (including a comparison to all participating programs) on each of the six Dimensions of the TiSET (Policy/ Administrative, Environment, Screening/ Assessment, Treatment, Staff and Training) indicating the degree of current implementation in each area.

We recommend that this report be submitted to the Executive Leadership and shared with stakeholders for their input and feedback. We also stress that the report should be viewed as a positive and affirming exercise, with an emphasis on current activities and program strengths, before discussing any possible areas for improvement.

Programs in New York State (NYS) outside of NYC are eligible to receive this report through the Center for Practice Innovations Focus on Integrated Treatment initiative. Programs outside of NYS can conduct manual scoring, following the guidelines above, to produce their own summary report.

Using the TiSET Report to Guide Further Implementation Planning

The next step after reviewing the TiSET Report is to have the workgroup to develop a Tobacco Use Disorder (TUD) implementation plan. An implementation plan is important as it provides a blueprint for how the program will reach its goals. It is the first step in the Plan-Do-Study-Act (PDSA) cycle, an iterative implementation process a workgroup can use to identify steps (Plan), act on those steps (Do), evaluate the results (Study), and then incorporate successful improvements and make any adjustments or changes needed (Act) (Deming, 2000). An implementation plan also increases accountability as tasks are assigned to specific staff and have clear target dates. Further, a good implementation plan includes strategies for sustaining successful changes.

We typically encourage programs to identify a range of goals that vary in how challenging they will be to implement. For each area identified, we encourage programs to have a defined goal (what is it that the program would like to achieve in that area) and specific measurable steps to achieve each goal. Using improving the environment as an example, the goal might be to “provide a welcoming environment that encourages healthy choices.” A few of the steps might be to acquire tobacco related pamphlets and posters and signs prohibiting tobacco use and display them, respectively, in waiting areas and throughout program/agency grounds.

Regular meetings help the implementation workgroup to keep track of progress and adjust as needed. We encourage programs to adopt quality improvement cycles of three to six months, addressing revised and new goals with each iteration. We strongly encourage programs to repeat the TiSET after six months or one year to measure progress and track updates in the identified areas. The follow-up report is submitted to and scored by the TCTTAC team as described above, with one notable difference. The follow-up report includes a graph comparing the baseline to follow-up scores and details improvements made, in addition to providing specific feedback to inform the next quality improvement project. In this way, the implementation process is ongoing and reflective of the changes made over time.

Participating Agency

We invited a large agency that has collaborated with TCTTAC to describe how they have used the TiSET to drive implementation. The Institute for Community Living (ICL), Inc., serves over 10,000 individuals throughout NYC each year, including adults, children and families experiencing mental illness, substance misuse, and homelessness. The 1200 person staff serves in more than 100 behavioral health and housing programs in sites across the five boroughs. Among people served, 51% (N = 3271/6375) identified as female, 57% (N = 2779/4848) identified as Black, 57% (N = 3458/6104) were age 45 or older, 75% (N = 3628/4840) had a serious mental illness, (39% [N = 1902/4840] had a primary diagnosis of schizophrenia or schizoaffective disorder), 57% (N = 1063/1878) were obese, 50% (N = 886/1757) were hypertensive, and 49% (N = 1028/2089) were smokers. Through these programs, ICL offers behavioral health treatment, care management, supportive housing, transitional shelters, and special community teams to reach the most hard-to-reach individuals. ICL’s work is grounded in a whole health approach which means that they treat the person—not the symptoms or diagnoses—and understand the inextricable link between physical and mental health. Services address all aspects of a person’s life experience including social and economic needs. ICL’s Clinical Excellence Committee, a multi-disciplinary group of staff from all levels of the agency (leadership, peer staff, direct care) and representative of diverse programs, are charged with achievement of ICL’s tagline, “People get better with us.” ICL has been nationally recognized as an innovative leader in the field, and their semiannual health surveys show continuously improving health and mental health outcomes for the people they serve.

In 2018, ICL opened their East New York Health Hub (Hub), a comprehensive center for integrated care. The Hub offers a spectrum of behavioral health services including clinic treatment, Personalize Recovery Oriented Services (PROS), Assertive Community Treatment (ACT), Home and Community-based Services (HCBS), care coordination, outreach programs, Family Resource Center, and supportive housing. The Hub is co-located with a Federally Qualified Healthcare Center (FQHC) operated by Community Healthcare Network. In partnership with this FQHC, ICL developed a model of fully integrated care including systematic protocols for referral, information sharing, case conferencing, and quality metrics. All policies, including the tobacco-related policies discussed here, were shared and reviewed with the FQHC to ensure alignment, resulting in a revised agency-wide tobacco free environment policy. The New York State Psychiatric Institute Institutional Review Board determined that this evaluation did not meet the definition of human subjects research.

Results

ICL began collecting health indicator data semi-annually in 2017 and added tobacco use as an indicator in 2018. Noticing that the rates of tobacco use remained at about 50% at every collection point, the Clinical Excellence Committee recognized the need to address tobacco using a systemic, multi-pronged approach and engaged with TCTTAC for training and support. We obtained agency-wide buy-in with a kickoff meeting in 2019. In the first Quarter of 2020, we invited 56 programs representing treatment, housing, outreach, and shelters to complete a baseline TiSET and submit those ratings to TCTTAC. TCTTAC provided ICL with average scores overall and by program type and created separate detailed reports for each participating program.

While ICL initially considered asking each program to develop its own implementation plan, we determined that it would be more feasible to develop an agency-wide workgroup and implementation plan, informed by the TiSET aggregated ratings. The first goal was to build workforce competency and self-efficacy by providing staff with training and educational materials on tobacco treatment and recovery. Specific action steps for this goal included having staff participate in TCTTAC in-person and self-paced trainings, providing additional in-house basic training at staff meetings, creating a repository of educational materials and resources on ICL’s intranet, and sending monthly emails with relevant information and tips for program directors to share with their teams. Across the 56 participating programs, 113 staff participated in training in the first year of implementation.

A second goal was that staff would routinely provide tobacco-related interventions that are matched to a person's stage of change. Specific action steps for this goal included having staff document how the person felt about addressing their tobacco use and describing treatment provided or offered by the staff, which might include medications or smoking reduction groups. During the first year of implementation, ICL transitioned from one electronic health record (EHR) to another, and attention was paid to include smoking-related questions in screening and assessment tools.

Other intervention-related action steps included items from other TiSET domains, specifically: making posters and educational material readily available and visible for people receiving services (Environment Domain), and routinely including discussion of tobacco treatment into clinical supervision (Staff Domain). Agency prescribers participated in a competition to offer nicotine replacement to the most people for whom it was indicated, to stimulate prescriber awareness and action around medication-assisted treatment.

A third goal was to provide support for employees interested in addressing their own tobacco use. Specific action steps included engaging with the employee assistance program (EAP) to offer workshops and resources, reviewing ICL policies around employee tobacco use, and making nicotine replacement therapy available for staff in staff lounges. This goal may actually have generated the greatest culture change at ICL, with discussions prompting staff to consider their own attitudes and assumptions around tobacco.

The workgroup, including TCTTAC staff, met monthly to review the plan and track progress toward these goals. Some goals, such as participation in training, were easy to track in an ongoing way (as people completed the training). Others, such as whether the treatment provided matched a person’s stage of change, were assessed after six months via a survey and chart audit. In early 2021, programs were invited to complete a follow-up TiSET to measure progress. Of the 56 programs completing a baseline TiSET, 40 (71%) completed a follow-up, showing significant gains in the policy and administrative and the training domains and a marginally significant improvement in the Environment domain (Table 1). ICL plans to use these findings and follow-up program reports to update and refine our implementation plan.

Table 1

Means, standard deviations, and paired t-tests for TiSET baseline and follow-up

Domain baseline	Follow-up		Paired t test*		t test	p value
	M	SD	M	SD
Overall score	3.02	0.87	3.27	0.71	1.61	0.116
Policy & Administrative	3.59	1.17	4.11	0.92	2.28	0.028
Environment	3.20	1.07	3.56	1.11	1.80	0.080
Screening & Assessment	3.51	0.97	3.47	0.98	− 0.19	0.850
Treatment	2.56	0.97	2.50	0.81	− 0.46	0.648
Staff	2.58	1.23	2.77	1.04	0.83	0.410
Training	2.68	1.42	3.21	1.15	2.09	0.043

*df = 39

Interpretation of score: capability to treat co-occurring behavioral health condition and tobacco use:

1–1.99. MHOS (Program focused on mental health only services)

2—2.99. Intermediary between MHOS and DDC

3—3.49. DDC (Dual Diagnosis capable; Able to treat some people with both mental health and tobacco use disorders)

3.5—4.49. Intermediary between DDC and DDE

4.5–5.0. DDE (Dual diagnosis enhanced; Able to treat all people with both mental health and tobacco use disorders, including those with the highest level of symptoms and/or impact from their co-occurring disorder)

Discussion

Completing the TiSET is an important step for behavioral health programs to evaluate their ability to effectively treat and work with people that use tobacco. The results shed light on a program’s strengths while also highlighting areas for improvement. An important next step is to use those results to facilitate a quality improvement process (Bertram et al., 2015), and intermediary organizations like TCTTAC, who are focused on tobacco treatment, can help facilitate that process. The large agency example illustrates how this process can be applied in real-world practice. Specifically, the baseline TiSET informed an implementation plan with three major goals: increasing staff competency through training, improving treatment for co-occurring tobacco use, and offering assistance to employees who smoke. This emphasis helped the organization set priorities and focus subsequent efforts, while developing a timeline for specific tasks to meet these goals.

Programs in this agency that completed both a baseline and follow-up TiSET demonstrated significant gains in two domains in the first year of implementation, consistent with the agency’s plan. Improvement in the Policy & Administrative domain likely reflected increased leadership attention to tobacco treatment and the resultant agency-wide culture shift, while the Training domain change was easily supported by training attendance.

During this same period, there were not significant increases in the other TiSET domains. One important confounding factor was that these major changes in service delivery were occurring due to the COVID-19 pandemic, which greatly changed the ways in which assessment and treatment were provided during the implementation year. This particularly impacted achievement on the Treatment domain, where one of the main action steps had been to offer tobacco treatment groups. Whereas, previously group treatment was offered routinely throughout the agency, the need to change groups to a remote format made offering them more challenging, with initiation of a tobacco-specific treatment group only occurring towards the end of the first year. As more persons served access technology and agency staff gain expertise in running virtual groups, and when more in-person services resume, group tobacco treatment is expected to increase. The fact that many staff were rarely or never in the office during this year also likely impacted scores on the Environment domain—for example, many staff may never have had a chance to see the nicotine gum in staff lounges, resulting in only a trend to improvement on this domain.

The agency also transitioned to a new EHR during this year, which placed additional stress on systems, impacting capacity for change. Documentation forms and processes for screening, assessment, and service plans were modified. Staff were just learning to use the new systems by the end of this first year, which may have impacted perception of Screening and Assessment domain changes or gains. Adaptation to these workflow changes also likely impacted scores in the Treatment and Staff domains, with staff and supervisor energies being split between the EHR transition and quality improvement goals including tobacco treatment.

These confounding factors aside, we will likely also find that there is an ontogeny to change on the TiSET domains, where Policy and Administrative and Training changes are necessarily the forerunners of a larger culture shift. Leadership buy-in, standard-setting and staff competency must precede changes in Environment (easiest to achieve), followed by Screening and Assessment and Staff (easier to mandate through forms and supervisory processes). The ultimate goal of improving Treatment quality is the hardest to achieve and most dependent on change in the behavior of individual staff members, which takes time and critical mass.

As is evident from this example, systems change efforts are not trivial and can take months and even years to successfully implement. These efforts work best with leadership and administrative support, and ideally the buy-in of the entire organization. Systems change efforts may be necessary to affect the culture of behavioral health, which has historically supported or been ambivalent about tobacco. Taking steps that help service recipients and staff de-normalize tobacco use and rethink the risks and rewards of tobacco can tip the balance towards change. Systems change efforts can increase the availability of treatment services and the demand for these services from tobacco users, which can lead to more people quitting tobacco. This report illustrates how practical tools, like the TiSET, can help facilitate implementation. The TCTTAC website includes publicly facing materials (i.e., the TiSET along with guidance documents) that can help interested agencies apply this approach to implementing treatment for tobacco use. Added support from an intermediary organization, like TCTTAC, can increase the chances of success.

There are several limitations worth noting. First, the instrument has not been subjected to rigorous reliability and validity testing, which was beyond the scope of what the intermediary was funded to provide. We sought to mitigate some of these concerns by modifying an established valid and reliable scale, the DDCMHT, for assessing co-occurring disorders more broadly and tailoring it for one substance, tobacco. Additionally, given that the DDCMHT and its variants have been used widely across behavioral health settings, the TiSET tool may offer program staff a familiar approach to assessing a program’s compliance with specific elements of the program model. Second, while the gold standard for programs ratings is to have an objective rater visit a program and complete and assessment, (Bond & Drake, 2019) the TiSET is designed as a self-assessment instrument. While self-assessment might lead to bias under certain conditions (e.g., when licensing depends upon proving model fidelity), there is some emerging evidence that, when supported as part of a quality-improvement process without penalty, self-assessment and independent fidelity ratings do not differ significantly from one another (Covell et al., 2021a, b; Margolies et al., 2017). Further, one study demonstrated a significant positive relationship between self-reported fidelity and outcomes (employment) that was sustained for a year (Margolies et al., 2018). Third, the agency described herein is a large well-resourced agency. It is possible that smaller agencies with fewer resources may not have the resources to participate in a similar process, though it is also possible that greater change could be achieved with a less heterogenous group. Future studies should include smaller agencies to determine whether this approach is feasible or whether modifications are needed to support agencies with fewer resources. Finally, to date, the use of the TiSET has been supported by an intermediary organization that is funded by The New York City Department of Health and Mental Hygiene to provide free essential support to help organizations use the ratings to develop an implementation plan to enact and sustain changes. Across evidence-based practices, this support can greatly enhance successful implementation (e.g., Margolies et al., 2015; Cheron et al., 2019; Thorning & Dixon, 2020; Covell et al., 2021a, b). While we designed the TiSET and supporting documents to allow organizations to self-direct this process, some programs may not have the resources to proceed without external support. Additionally, in areas where intermediary assistance is not free, programs may not have the resources to pay for this support. Future studies should examine the extent to which organizations are able to use these resources to effect change in a self-directed way. Additionally, state and city governments, along with other payers, should strongly consider funding intermediaries to support their behavioral health providers implement evidence-based practices broadly, including co-occurring behavioral health and tobacco addiction.