Fidelity Assessment of the Treatnet family (TF): A family-based intervention for adolescents with substance use disorders and their families.

INTRODUCTION: In transporting family-based interventions to community settings, establishing and maintaining fidelity to intervention is important. This exploratory study was implemented in the framework of a United Nations Office on Drugs and Crime (UNODC) global programme on Drug Dependence Treatment and Care. It is the first to examine an evidence-informed family-based intervention ("Treatnet Family"; TF) adherence for the treatment of adolescents with substance use disorders (SUD) among practitioners in community settings in Jakarta, Indonesia.
METHOD: Twenty-three practitioners at five community-based counselling centres were trained in Treatnet Family and delivered it to 19 adolescents with SUD and their family members over a 6-week period. One of the five local Treatnet Family-trained supervisors randomly selected one session of the family-based intervention (TF) and observed the extent to which the practitioner's adhered to the TF manual.
RESULTS: According to the supervisors' observation, all the practitioners used the Treatnet Family core skills such as positive reframing, positive relational reframing, perspective taking, relational questions, and going with resistance. There was a high level of agreement between practitioners' and supervisors' rating on the practitioners' use of specific therapeutic skills as measured using the Inventory of Therapy Techniques (ITT).
CONCLUSION: Results suggest that Treatnet Family can be delivered with adherence by practitioners in community-based settings.

Introduction

Substance use disorders (SUD) continue to be among one of the most common disorders in adolescence, with a lifetime prevalence of SUD is estimated to range from 3% to 32% (Essau and Delfabbro, 2020, Fergusson et al., 1993, Lewinsohn et al., 1993, Feehan et al., 1994, Merikangas et al., 2011, Swendon et al., 2012). Family-based interventions have empirically demonstrated significant effects in the reduction of adolescent substance use (Lindstrøm et al., 2015, Rigter et al., 2013). Tambling, Russell and D’Aniello (2021, p. 9) recently concluded that family-based intervention has “become the gold standard in care, demonstrating positive outcomes and superiority over individually-oriented or family-involved models”. Priorities in recent years have shifted toward transporting evidence-based treatments from research/university to community practice settings (Hailemariam et al., 2019, National Academies of Sciences, Engineering, and Medicine, 2017). In transporting family-based interventions to community settings, establishing and maintaining fidelity to intervention is important. When fidelity is not fully assessed, it is not known if the changes in the study outcomes are due to the intervention being investigated, or due to differences in its implementation (Toomey, Matthews, & Hurley, 2017). As argued by some authors, differences in outcomes of evidence-based interventions that have been transported from research to routine clinical settings could be attributed to differences in the way practitioners comply to the intervention procotol (Collyer, Eisler, & Woolgar, 2020).

Intervention fidelity refers to the degree to which the intervention is delivered by the practitioners as intended by intervention developers (Dusenbury et al., 2003, Hill et al., 2007). Two important aspects of intervention fidelity which have attracted much research attention are adherence and competence. Adherence refers to the extent to which the key components of the interventions are delivered as designed (Hill et al., 2007), or the degree to which an intervention is implemented according to the theoretical and procedural aspects of the intervention model (Hogue, Dauber, Samuolis, & Liddle, 2006). Competence refers to how well practitioners apply their skills in delivering the intervention (Hogue, Henderson, Dauber, Barajas, Fried, Liddle, 2008); competence may also include practitioner’s characteristics (e.g., enthusiasm, preparedness, attitude, responsiveness) or skills (e.g., in using the techniques prescribed by the intervention) with which interventions are delivered (Dane & Schneider, 1998). The ability to demonstrate both competence and adherence is believed to increase confidence in the results of the intervention (Borrelli, 2011, Southam-Gerow and McLeod, 2013). In this sense, a strong fidelity is critical for transporting research-based interventions to community settings protocols (Henggeler, Pickrel, & Brondino, 1997).

Evaluating intervention fidelity enables early detection of protocol deviations from becoming widespread, which could have an impact on the study findings (Collyer, Eisler, & Woolgar, 2020). Intervention fidelity is particularly important for studies that are conducted in various sites; this is to ensure that interventions are delivered in the same manner across sites, which will help to reduce the possibility of site by treatment interactions (Baer et al., 2007, Waltz et al., 1993). Intervention adherence and competence has been assessed using various methods. In some studies, adherence was assessed using therapist and family member self-reports of adherencing to the key components of the intervention (Henggeler, Melton, Brondino et al., 1997); in some other studies (Hogue et al., 2008), observational methods were used. Both methods have advantages and disadvantages. Although the use of self-reports is less expensive for monitoring fidelity than observational method, the practitioner’s answers may be influenced by social desirability which could give inaccurate reports of fidelity (Schoenwald & Garland, 2013). Observational method could be costly and labor-intensive as it involves training the intervention observers. However, observational method is considered as the optimal standard for fidelity assessment (Schoenwald & Garland, 2013) as it provides a more objective assessment of practitioners' and adolescents' behavior than self-report; this method also helps to understand both the nuances of the intervention and its facilitation (Bishop, Pankratz, Hansen, Albritton, Albritton, & Strack, 2013).

Therapist adherence to the intervention protocol has consistently been reported to predict clinical outcomes (Hogue et al., 2008). Specifically, the higher the adherence level, the better the clinical outcomes (Hogue et al., 2008). For example, in a study by Schoenwald et al. (2008), greater adherence to the multi-systemic therapy (MST) for youth with drug use disorders and behavioral problems fidelity was related to better short- and long-term outcomes. In another study, high adherence to MST was associated with good outcomes for violent and chronic juvenile offenders and their families (Henggeler et al., 1997). However, treatment adherence is not always associated with better outcomes. According to some authors, adherence to intervention protocols may reflect training and supervision, both of which could have an impact on therapist behavior (Proctor et al., 2011, Schoenwald et al., 2011). Several authors (Barber et al., 2006, Hogue et al., 2008) argued that adherence that is either too lax or too strict to the intervention protocol might limit a therapist’s effectiveness. For example, over-adherence to the intervention protocol may indicate lack of flexibility to respond to client needs (Hogue et al., 2008).

Establishing and maintaining family-based intervention with fidelity in routine community settings has been reported to pose several challenges (McHugh & Barlow, 2010) as investigators have less control over practitioner selection and lack of procedures to monitor clinical activities. At the same time, practitioners in community settings have heavier caseloads than therapists in research settings. Furthermore, it has been argued that most studies published on the effectiveness of evidence-based intervention such as family-based interventions have been conducted at highly specialised university clinics with above average competence and experience in the clinical treatment of adolescent psychopathology, including highly selected clinicians and participants (Weisz et al., 1992, Weisz et al., 1995). Additionally, practitioners in community settings may not have the same level of intensive training, monitoring and supervision compared to therapists in research settings. Adolescents seeking treatment in community settings are often more diverse compared to those in research settings both sociodemographically and clinically (Ehrenreich-May, Southam-Gerow, Hourigan, Wright, Pincus, & Weisz, 2011); such differences mean that the therapists might have to deviate from the intervention protocol in order to meet the adolescent’s clinical needs (Smith, McLeod, Southam-Gerow, et al., 2017). The differences between research and community settings have raised concern about the generalizability of findings of previous studies to regular community settings.

Understanding the level and role of treatment adherence is important for the following reasons: treatment adherence shapes or facilitates focused training, improves the quality of the intervention manuals, identifies areas of supervision, and increases adherence and competence levels (Dimitropoulos, Lock, Agras, et al., 2020). To our knowledge, there is no research to have examined adherence to family-based intervention among practitioners in community settings in low-income countries. To close this gap, the main aim of the present study, implemented in the framework of a United Nations Office on Drugs and Crime (UNODC) global programme on drug dependence treatment and care, was to examine the fidelity (adherence) of an evidence-informed family-based intervention (Treatnet Family; TF) among practitioners in community settings in Jakarta, Indonesia that offers outpatient interventions to adolescents with SUD.

Method

The study protocol, informed consent, and all study materials were reviewed and approved by the Committee on Research Ethics at Atma Jaya Indonesia Catholic University, and at the Universitas Indonesia, Jakarta, Indonesia. All the participants (i.e., practitioners, national TF supervisors, adolescents and each family member/caregiver) provided a written consent to participate in the study. The practitioners, adolescents and each family member/caregiver additionally provided active consent for their sessions to be observed for adherence and competence.

Participants

Practioners: Twenty-three practitioners (60.9% were male; mean age: 37.5 years) who work in five community-based counselling centres that offer psychological/counselling services to people with SUD, including young people with SUD, were recruited for the present study. They all participated in a one-week intensive training in delivering TF from three international experts in TF.

All, except one, had experience in working with adolescents, with a mean number of years of working experiences being 5.2 years. Over 40% of the practitioners had a bachelor’s degree and all of them had received training in drug use disorder counselling. Most of them were male (60.9%) and had a history of drug use themselves (60.9%); having people with lived experience in drug use as practitioners will help to reduce stigmatisation against people with SUDs and could serve as models of hope for adolescents with SUDs and their families. Furthermore, human resources for the treatment of drug use disorders are limited and in many places in the word, people with a history of drug use disorder in recovery are involved in the provision of services. It is of key importance to build their professional capacity through initiatives such as Treatnet Family among other trainings. Participants were also selected based on the region that they provide their services, which are low-income community in North and East Jakarta where the National Narcotic Board recently established its out-patient post-rehabilitation services.

National TF Supervisors: Five counsellors who have been previously trained in TF served as national supervisors during this project. All of them had participated in two week-long TF workshops. All the supervisors were female, with a mean age of 43.40 years. They all had a master’s degree in psychology and were currently working as a family and child/adolescent psychologist (N = 2) and as a clinical psychologist (N = 3). They all had experience working with families (mean = 13.8 years) and young people (mean = 14.8 years).

Their main role was to provide supervision and support to the practitioners throughout this research project; they also observed a randomly selected TF session to directly examine the fidelity level in their implementation of TF. The supervisors were given the opportunity to discuss any of the issues raised by the practitioners with the local United Nations Office on Drugs and Crime (UNODC) staff and the international experts in TF.

Adolescents and their family members: Nineteen adolescents and their families participated in the TF and completed the questionnaires at pre-, post-intervention as well as at a month-follow-up assessment. Most of the adolescents (mean age = 16.1 years) were males (84.2%), living with both parents (63.2%), and were still going to school (68.4%).

Family members who completed the questionnaires were mostly mothers, with a mean age of 42.5 years. Most of them were married (78.9%) and were in employment (84.2%). About half of the family has a family monthly income of Rp 6–10 million (approximately 400 – 700 USD) (52.6%); another 42.1% had a family monthly income of above Rp 10 million (approximately above 700 USD).

Treatnet family (TF)

TF contains common elements of evidence-based family therapies and has been developed with a specific view for adolescents with SUD and their families in low and middle-income countries (LMIC) (Hogue et al., 2019, United Nations Office on Drugs and Crime (UNODC), 2019). TF focuses on family interactions and uses elements of family therapy to, among other things, change ineffective communication patterns within the family and learn more effective ones. It has six sessions, with each session lasting between 90 and 120 min. Each session was attended by the adolescent with SUD and his/her family members. The sessions follow a cycle starting with engaging the family, using the core skills in family-based intervention such as positive reframing, positive relational reframing; perspective taking, relational questions; going with resistance; family assessment; creating a motivational context for change and finally termination of the intervention. However, these skills can be applied with some flexibility throughout the intervention. It is important to emphasise that structure and flexibility go hand in hand. While the structure of TF provides the practitioners with a strong sense of direction, it is critical that they sensitively apply TF in a way that fits the families they work with and unlocks their resources to help themselves.

Measures

(a) Practitioners: After each session with the adolescents and their families, the practitioners completed the following questionnaires:

Treatment Integrity Scale was designed specifically for this study to measure the extent to which the practitioners felt the session was successfully implemented by indicating the following rating scale: “Not at all “Successful”, “A little successful”, “Moderately successful”, “Very successful”, and “Extremely successful”. Successful implementation was defined as practitioner’s ability to apply specific core skills of TF (e.g., positive relational reframing; perspective taking, relational questions), time management, and in establishing working alliance with the clients. TF contains “elements of family therapy” and as such it involves the implementation/integration of skills into the practitioner’s existing practice. This scale also asked about the length of the session and the number of family members (and their relationship to the adolescents) who attended the session with the adolescents.

Inventory of Therapy Techniques (ITT; Hogue & Dauber, 2013) was used to measure the extent to which practitioners used specific intervention strategies/skills in each of the TF session. The ITT contains 27 items which described four main therapeutic approaches: cognitive-behavioral therapy, family therapy, motivational interviewing, and drug counselling. The items were rated on a 5-point Likert scale: “A little bit”, “Moderately”, “Considerably”, and “Extensively”.

(b) National Supervisors evaluated the fidelity level by observing a randomly chosen TF session. At the end of the session, each supervisor was asked to indicate whether or not the practitioners delivered the skills as outlined in the TF manual using the ITT (Hogue & Dauber, 2013). In addition to the ITT, the supervisors also completed the Session Quality Scale.

Session Quality Scale was developed for the present study, where national supervisors were asked to indicate their judgment regarding the three aspects of the session: (i) Client Difficulty (“What is the level of difficulty presented in this session by the client?”), which is defined as a high level of resistance (i.e., oppositional, reactionary behavior, noncompliant, intractable, and unmotivated) displayed by the adolecents and their family members.

(ii) Practitioner Competence (“How competent do you think the practitioner performed in this session?”) was defined by the practitioner’s ability to apply the core skills of TF (family-based intervention). These two questions were to be rated on a 5-point Likert Scale, ranging from “Not at all” to “Extensively”.

(iii) Session Success (How successful was the practitioner in meeting apparent or presumed goals of the session?) was defined in terms of practitioner’s ability in applying the session’s goal (e.g., section 3 involves “creating motivational context for change”), and in establishing working alliance, and in managing time. This question was to be rated on a 5-Likert scale, ranging from “Not at all” to “Extremely successful”.

(c) Adolescent and family members were each asked to complete a short questionnaire about their experience in the TF session that was observed by the supervisor (i.e., the same TF session that was observed by the national supervisor). They were asked about how interactive the sessions were, and whether the practitioner gave them much opportunity to talk and to take each other’s viewpoint.

Results

Supervisor’s rating of TF session

The most commonly observed sessions were sessions 3 and 5. According to the supervisor’s observation, all the practitioners covered the TF core skills such as positive reframing, positive relational reframing; perspective taking, relational questions; and going with resistance. The less commonly used skills were “reflecting” (8.8%) and “paraphrasing” (8.8%).

The level of difficulty presented by the client in this session was rated by the supervisor as mostly at “moderate level” (58.8%). Furthermore, in 29.4% of the cases, this was rated as having “considerable” difficulty. Most practitioners were reported to have handled the session with “moderate competence” (70.6%) and “considerable competence” (20.6%). Overall, the session was evaluated by the supervisor as a success in terms of the quality in which the practitioner carried out the session (50% moderate and 47.1% considerable success).

Practitioner’s use of specific skills during TF sessions

Following Hogue and Dauber (2013), the ITT skills were grouped under: cognitive behaviour therapy (CBT), family therapy, motivational interviewing and counselling. Table 1 shows the means of ITT subscales based on the practitioner’s self-report and as observed by the supervisor for sessions 3, 4, and 5, respectively. Except for the CBT scale for session 5, there is no significant differences in the practitioner’s and supervisor’s rating. In session 5, the practitioner’s report in terms of the use of CBT skills was significantly higher than those observed by the supervisor, F = 17.19, p < 0.05.

Table 1

Means of ITT scale for Sessions 3, 4, 5.

Session 3	PractitionerMean (SD)	SupervisorMean (SD)
• Cognitive Behavior Therapy scale	3.45 (0.59)	3.13 (0.91)
• Family Therapy scale	3.82 (0.50)	3.42 (0.85)
• Motivational Interviewing scale	3.38 (0.58)	3.42 (0.85)
• Drug Counselling scale	2.18 (1.08)	2.55 (1.21)
Session 4
• Cognitive Behavior Therapy scale	3.65 (1.01)	3.31 (0.57)
• Family Therapy scale	3.83 (1.09)	3.69 (0.69)
• Motivational Interviewing scale	3.61 (0.94)	3.42 (0.76)
• Drug Counselling scale	2.71 (1.38)	2.57 (0.98)
Session 4
• Cognitive Behavior Therapy scale	3.66 (0.78)	3.09 (0.84)
• Family Therapy scale	3.78 (0.61)	3.33 (0.98)
• Motivational Interviewing scale	3.77 (0.58)	3.32 (0.89)
• Counselling scale	3.27 (1.35)	2.27 (1.48)

Practitioners with a history of drug use and those without a history of drug use did not differ significantly in their use of specific ITT skills during the TF, F = 0.71, p > 0.05.

Table 2 shows the level of agreement between practitioners’ and supervisors’ ratings on the practitioner’s use of specific therapeutic skills as measured by the ITT scale. The kappa values (unweighted and weighted) were calculated based on the original rating. The Maxwell value was based on a dichotomised value (1 = skills were used; 0 = skills were not used). The high Maxwell value illustrated a high level of agreement between the two informants.

Table 2

ITT – Level of Agreement between Practitioners and Supervisors.

	Kappa (unweighted)	p-value	Kappa (weighted)	p-value	Maxwell (dichotomised)
Cognitive Behavior Therapy
• Sets agenda	0.3	0.143	0.65	0.022	1.00
• Cravings, triggers, and high-risk situations	0.02	0.861	0	1	0.64
• Coaches interaction	−0.1	0.503	0.17	0.334	1.00
• Behavioral interventions	0.04	0.769	0.03	0.905	0.27
• Teaches new skills	0.19	0.281	0.54	0.04	0.45
• Non-drug activities	0.11	0.442	−0.09	0.644	0.82
• Homework assignment	0.36	0.037	0.5	0.081	1.00
Family Therapy
• Parental monitoring	0.44	0.014	0.62	0.029	1.00
• Family attachment	0.29	0.026	0.52	0.026	1.00
• Family intervention (relational issues)	−0.06	0.737	0.21	0.461	1.00
• Deal with presenting problems	0.19	0.281	0.54	0.04	0.45
• Core relational themes	0.25	0.093	0.29	0.126	0.64
Drug Counselling
• Confront denial	0.02	0.896	0.28	0.322	0.45
Motivational Interviewing
• Affirmed self-efficacy	0.44	0.034	0.49	0.084	1.00
• Reflective statements	0.17	0.39	0.32	0.2	1.00
• Promote equality	0.06	0.752	0.03	0.903	0.82
• Motivation to change	0.1	0.419	0.34	0.215	0.82
• Heightens discrepancies	−0.01	0.945	0	1	0.64
• Drug use and the pros and cons	−0.09	0.451	0.23	0.376	0.64
• Change planning	0.13	0.388	0.34	0.124	0.82

Skills with the high levels of agreement included those related to the Cognitive Behavior Therapy (i.e., sets agenda, coaches interaction, homework assignment), Family Therapy (i.e., parental monitoring, family attachment, family intervention (relational issues)), and Motivational Interviewing (i.e., affirmed self-efficacy, reflective statements).

Interaction during the TF session

There was a good level of agreement between the adolescents and their family members in terms of how interactive the session was. As shown in Table 3, 65.8% and 71.1% of the adolescents and their family members, respectively, reported the session as very interactive. However, the supervisor rated the level of interaction as mostly “somewhat interactive” and “interactive”. Similarly, the level of agreement between adolescents and their family members in terms of the level of opportunity that the practitioner gave them to talk to each other and to express their views was high.

Table 3

Interaction during the observed session.

	Adolescent	Family member	Supervisor
	%	%	%
How interactive was this session?
- Somewhat interactive	15.8	15.8	47.1
- Interactive	18.4	13.2	50.0
- Very interactive	65.8	71.1	2.9
How much opportunity did the practitioner give you and your family members to talk?
- A little	2.6	–	8.8
- Some	13.2	13.2	55.9
- A Lot	84.2	86.8	35.3
How much opportunity did the practitioner encourage you/the adolescents and your family members/their family members to talk to each other’s viewpoint?
- Not at all	2.6	–	20.6
- Some	18.4	26.3	47.1
- A Lot	78.9	73.7	32.4

Discussion

As in many family-based interventions, TF includes a combination of relational and structuring skills which are needed to manage the interactions that occur in therapy sessions (Szapocznik, Muir, Duff, Schwartz, & Brown, 2015). TF begins with engagement which involves creating an environment where the family believes that meeting together with the practitioner will help them address their problems and improve their life and relationships. The subsequent sessions include assessment and leading to behavior change. The present results indicated that the practitioners from community settings in a lower-to-middle-income country who participated in a one-week workshop in TF used the core skills (e.g., positive reframing, positive relational reframing; perspective taking, relational questions; and going with resistance) of a family-based intervention. Overall, our findings support previous studies that practitioners from community settings can be trained to deliver the family-based interventions in an adherent manner.

Our findings also showed a high level of agreement between practitioners’ and supervisors’ rating on the practitioners’ use of specific therapeutic skills as measured using the Inventory of Therapy Techniques scale. These adherence levels are in agreement with levels reported in previous studies that used the ITT (Hogue & Dauber, 2013). The skills with the high levels of practitioner-supervisor agreement included those related to the Cognitive Behavior Therapy (i.e., sets agenda, coaches interaction, homework assignment), Family Therapy (i.e., parental monitoring, family attachment, family intervention (relational issues)), and Motivational Interviewing (i.e., affirmed self-efficacy, reflective statements); it should be noted that there are overlaps in some of the skills that are categorised under Cognitive Behavior Therapy, Family Therapy, and Motivational Interviewing. Skills with the lowest level of agreement were related to those related to behavioral interventions. While it is beyond the scope of this study to explore reasons for this lack of agreement, it can be speculated that it might be related to the fact that behavioral interventions may take the form of homework assignments that the adolescents and their family members need to do outside of the therapeutic sessions.

According to the supervisor’s observation, most practitioners handled the session with “moderate competence” (70.6%) and “considerable competence” (20.6%). The level of competence as measured in the present study was a weak proxy for therapeutic competence in implementing TF. As competence is an important aspect of fidelity, it ideally should have been measured with more rigorous methods such as using observation approach by expert judges (Waltz et al., 1993). However, such an approach is resource intensive (Hogue & Dauber (2013). Furthermore, as argued by Hogue and Dauber (2013), the operationalisation of competence reliably is difficult even when manuals are available that provide specific guidelines for skillful implementation of the family-based interventions.

Our results should be interpreted in light of several limitations. First, the practitioners were recruited from five community centers in East and North Jakarta and may not be representative of all practitioners in Indonesia. Second, only one TF session was being observed which might not reflect the way in which the other sessions were being delivered by the practitioners. Furthermore, it is widely acknowledged that practitioner reactivity to observation may provide less accurate estimates of implementation fidelity as some practitioners tend to adhere more closely to the protocol while being observed; some other practitioners may on the other hand become anxious when being observed, leading to low level of adherence (Breitenstein, Gross, Garvey, Hill, Fogg, & Resnick, 2010). Third, this study only focused on the intervention fidelity (i.e., intervention adherence) in the delivery of TF; furthermore, only a broad measure of practitioner’s competence was used in the present study. As argued by several authors, although manualised intervention protocols do offer guidelines on how to skillfully deliver the intervention, the way to measure competence reliably has proven difficult Hogue et al. (2008).

These limitations notwithoutstanding, the results of the present study indicate that it is possible for practitioners to use a family-based intervention (i.e., Treatnet Family) with fidelity in their routine clinical work in community settings. These results have important implications for practitioners training especially in low-and middle-income countries.

CRediT authorship contribution statement

Anja Busse: Conceptualization, Validation, Writing - original draft, Writing - review & editing, Supervision, Funding acquisition. Wataru Kashino: Conceptualization, Writing - review & editing, Supervision, Funding acquisition. Sanita Suhartono: Conceptualization, Writing - review & editing. Narendra Narotama: Writing - review & editing. Giovanna Campello: Validation, Funding acquisition. Irwanto: Data curation, Formal analysis, Writing - review & editing. Dicky Pelupessy: Data curation, Formal analysis, Writing - review & editing. Fred P. Piercy: Supervision, Writing - review & editing. Cecilia A. Essau: Conceptualization, Methodology, Data curation, Formal analysis, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.