Advancing implementation frameworks with a mixed methods case study in child behavioral health.

Despite a growing policy push for the provision of services based on evidence, evidence-based treatments for children and youth with mental health challenges have poor uptake, yielding limited benefit. With a view to improving implementation in child behavioral health, we investigated a complementary implementation approach informed by three implementation frameworks in the context of implementing motivational interviewing in four child and youth behavioral health agencies: the Active Implementation Frameworks (AIF) (process), the Consolidated Framework for Implementation Research (factors), and the Implementation Outcomes Framework (evaluation). The study design was mixed methods with embedded interrupted time series and motivational interviewing (MI) fidelity was the primary outcome. Focus groups and field notes informed perspectives on the implementation approach, and a questionnaire explored the salience of Consolidated Framework for Implementation Research (CFIR) factors. Findings validate the process guidance provided by the AIF and highlight CIFR factors related to implementation success. Novel CFIR factors, not elsewhere reported in the literature, are identified that could potentially extend the framework if validated in future research. Introducing fidelity measurement in practice proved challenging and was not sustained beyond the study. A complementary implementation approach was successful in implementing MI in child behavioral health agencies. In contrast with the typical train and hope approach to implementation, practice change did not occur immediately post-training but emerged over a 7 month period of consultation and practice following a discrete interactive training period. The saliency of CFIR constructs aligned with findings from studies conducted in other contexts, demonstrating external validity and highlighting common factors that can focus planning and measurement.

Implications

Practice: Implementation of evidence-based treatments (EBTs) must consider the manner in which consultation and clinical supervision facilitate practitioner behavior change, the need to develop fidelity assessment as a common practice, and a focus on pretraining implementation preparedness.

Policy: Policymakers who want to support evidence implementation must understand the process, time, and cost implications backed by the growing evidence based for this emerging field.

Research: Future research should be aimed at testing the newly proposed Consolidated Framework for Implementation Research constructs, exploring how best to develop a fidelity practice within provider organizations, and studying the subtle implications of implementing manualized versus discrete EBTs within differing initiating contexts.

INTRODUCTION

Research has led to numerous evidence-based treatments (EBTs) for children and youth with mental health challenges [1]; yet, we continue to see poor uptake of EBTs beyond controlled research settings. To address this, governments in many countries are creating policies, plans, and in some cases, passing laws to accelerate the use of EBTs and improve the quality of care [2-6]. In Canada, for example, the government’s report on mental health and addictions transformation points to the importance of improved implementation for quality care [5]. This type of healthcare reform ensures greater attention to establishing quality standards, accountability for treatment services, and monitoring of client-level outcomes. Poor or unsustainable implementation results in limited access and suboptimal benefits for children and youth receiving behavioral health services in real-world, community-based programs [7, 8]. EBTs and policy drivers are necessary but insufficient for facilitating uptake and improving client outcomes. Quality care occurs when EBTs are implemented effectively, placing research that can reveal the most successful mechanisms for effective implementation in high demand [9].

To this end, the current study used an implementation approach informed by complementary frameworks to guide the implementation of motivational interviewing (MI) [10-11] in the child and youth mental health context, including the Active Implementation Frameworks (AIF) [12], the Consolidated Framework for Implementation Research (CFIR) [13], and the Implementation Outcomes Taxonomy [14]. We also explored the salience of CFIR constructs for implementation process to better understand which determinant factors providers believed had any influence on implementation. The CFIR provides a comprehensive, standardized list of constructs that can help identify variables that are most salient to implementation of a particular innovation and in a particular context. How CFIR factors perform for this EBT and context (MI in child and youth mental health) when compared with how these factors perform for other EBTs and in different contexts also provides evidence of CFIR’s external validity.

MI [15] is an effective client-centered approach to elicit client motivation for positive change that is highly flexible and can be delivered as a stand-alone intervention or as a motivational component integrated with other treatments. Format flexibility adds a level of complexity to studying its implementation, as we shall report further on. In this study, MI was implemented as an evidence-based element [16] rather than as a manual-based intervention; a preference requested by our practitioner partners to better fit their therapeutic style.

Implementation difficulties in behavioral health

A strong knowledge base requires studies grounded in conceptual frameworks that can document challenges emerging in real-world circumstances. This challenge is fitting with the current stage of implementation science, which is progressing from studies identifying barriers and facilitators to those exploring optimal processes, factors, strategies, and implementation outcomes [17].

Several issues make it inherently difficult to reconcile findings across studies and to advance the field, including models that provide limited “how-to” knowledge, a lack of practical implementation evaluation tools, measures and metrics [18], and poor reporting of implementation studies [19-21]. Another difficulty is that the typical approach to implementing EBTs in community settings, often referred to as “the train and hope approach,” is characterized by pairing didactic workshops and educational materials [22] that may result in moderate knowledge improvements but effect little or no practice change [7, 22, 23]. Ineffective and inefficient fidelity measurement [24] and inadequate supervision models [25] also impede implementation. In the case of MI, fidelity monitoring tools are predominantly developed for and used in efficacy trials (i.e., Motivational Interviewing Skills Code [26]; Motivational Interviewing Treatment Integrity Code [27]; and Motivational Interviewing Supervision and Training Scale [28]) and are too complicated and impractical for use in community mental health settings [29]. Useful MI-related implementation strategies include fidelity consultation and feedback [30] that are also recommended in other implementation studies [29, 31]. However, several characteristics of Miller’s MI training study [30] limit the applicability of the findings to community-based clinical settings [29]. For instance, Miller’s therapists were not typical of community-based practitioners, as they were highly motivated to learn MI (i.e., pursued training at their own expense) and showed high MI fidelity at baseline. They were also free to select their individual clients for MI work, which is typically associated with higher MI performance. By comparison, community-based therapists likely demonstrate more variable motivation to learn MI and preimplementation MI abilities. They are also less likely to have control over which clients they see than what is typical for a clinical trial [29, 32, 33]. Moreover, although competence and fidelity monitoring are standard practice in efficacy trials and purveyor-assisted implementation models, supervision, fidelity monitoring, and consultation support vary tremendously in routine clinical practice [12]. Consequently, we know significantly less about how practitioners experience implementation in community-based settings [25, 29].

Frameworks informing our implementation approach

Different types of implementation frameworks provide promising evidence about the processes (steps and stages) required for effective and sustained implementation under real-world conditions, the determinant factors that influence implementation outcomes, and the evaluative frameworks that guide our understanding of whether implementation approaches were successful [34]. Because these models and frameworks focus on different aspects of implementation, it is useful to integrate them into a complementary approach, as we have done in this study.

The present study was informed by elements of three complementary implementation frameworks. The CFIR [13] is a determinant framework comprised of 39 constructs organized into five major domains: intervention characteristics, outer setting, inner setting, characteristics of individuals, and process. See Supplementary Material 1. At the macro level, the CFIR provides a standardized structure for building on findings across studies (see systematic review, ref. 35).

The AIF [12] is a group of process frameworks that promote implementation stages (exploration, program installation, initial implementation, and full implementation), drivers (competency: staff selection and coaching; organization supports: facilitative administration and decision support data systems; and leadership: technical and adaptive), and process innovations (implementation teams). AIF also highlights the importance of clinical consultation as an aspect of training [36-37]; see https://implementation.fpg.unc.edu/.

The Implementation Outcome taxonomy (IO) [14] addresses the measurement of implementation outcomes, including acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability. Together, these frameworks and models informed the objectives of the present study: (a) to explore the usefulness of a complementary approach for MI implementation in child behavioral health agencies and (b) to validate the saliency of the CFIR constructs for implementation processes.

METHOD

This study used a mixed-methods case study design with embedded interrupted time series to analyze how clinician fidelity data are affected by implementation consultation over time. Practitioner MI fidelity was the primary implementation outcome. Other implementation outcomes were considered in the planning phase (acceptability) and the qualitative analysis (appropriateness, feasibility, and sustainability). No clinical outcome data were collected given considerable evidence for MI effectiveness in youth and adults with mental health problems [37-40]. MI was delivered as one component of therapy or as an evidence-based element [16] rather than as a discrete EBT, rendering it impossible to attribute clinical outcomes to MI specifically. Both quantitative and qualitative data on implementation process and outcomes were captured at the practitioner and organizational levels, collected sequentially (quantitative followed by qualitative), and integrated during analysis and interpretation to identify patterns of results [41].

StaRI standards guided comprehensive reporting for this implementation research [19-21]. The research protocol was approved by the Hospital for Sick Children’s Research Ethics Board and the review committees of participating organizations.

Setting

Purposeful sampling identified five child behavioral agencies from among 120 known to our team. We selected agencies that were as follows: (a) Ontario providers of mental health services to children and youth under the age of 18; and willing to (b) participate in consensus-based decision making to identify a target EBT for implementation, (c) make staff available for training in the target EBT, and (d) engage staff in fidelity monitoring, consultation, and participation on the site’s implementation team. Each organization identified up to 10 practitioners providing clinical service to children 6 to 18 years of age and their families. We looked for practitioners within the same program to leverage peer support for practice change. Including all agency practitioners was deemed too disruptive and costly; research funds were available to train and coach 40 practitioners and their supervisors. To guard against practitioner attrition and maintain focus on building sustainable implementation capacity, we excluded contract and temporary staff, including student interns. One organization withdrew from the study before training, deeming MI to be incompatible with their Collaborative Problem Solving approach [42]. The four remaining sites varied in size (i.e., operating budgets ranged from CAD$6,768,000 to CAD$16,211,000) and location (i.e., one rural and three urban) and offered a range of psychoeducational and counseling services to children and parents, with two sites also providing residential care for boys. All agencies were members of a provincial association representing Ontario’s publicly funded Child and Youth Mental Health agencies that advocate for government investments, policies, and programs that are responsive to the needs of children, youth, and families seeking mental health services in Ontario, and all complied with Quality Accreditation Standards.

Participants

The final sample included 22 practitioners and nine supervisors (organization A: six practitioners, three supervisors; organization B: two practitioners, two supervisors; organization C: six practitioners, two supervisors; and organization D: eight practitioners, two supervisors). All practitioners had variable experience with practice change and implementing new EBTs prior to this study, with some trained in cognitive behavioral therapy (n = 17), Triple P (n = 4), Multi-Systemic Therapy (n = 2), and trauma-focused cognitive behavioral therapy (n = 2). Two supervisors and seven practitioners were initially part of the project but withdrew due to job change, illness, or difficulties with audiotaping sessions and attending consultation calls; we did not capture when they withdrew. Across all organizations, an additional 10 people with roles in information technology, human resources, research, or management were members of the onsite implementation teams, as informed by the AIF [12].

Implementation process

Preparation

In alignment with AIF’s exploration stage [12], we contacted each agency before research funding was secured to elicit their interest and input on needs, methods, and logistics. Post funding, we reconnected to identify a target EBT through consensus dialogue. This process culminated in a shared interest in implementing an EBT that could improve client engagement. MI [15] was proposed and unanimously supported.

Before conducting MI training, two planning visits were made to each agency to orient them to study aims and procedures and assess implementation readiness via baseline measures. Implementation guidance was shared via e-learning modules developed previously by the primary author (see https://tinyurl.com/ychral9n) and in a reading assignment on the theoretical foundations of MI use with children [43]. Consequently, our research team provided no implementation guidance and limited our contact to research evaluation activities to avoid supporting implementation in ways that would be difficult to replicate in a nonresearch implementation context.

To situate ownership of implementation internally, we invited agencies to form an implementation team (AIF) [12] and to meet monthly to support implementation tasks. Teams also participated in a monthly call with the investigators to track their implementation efforts.

Training

Four, highly experienced MINT (Motivational Interviewing Network of Trainers)–certified clinical social workers provided training and post-training consultation, under contract to the research team. Research investigators met with MI trainers twice, before the training, to discuss a training approach that blended core MI principles with best practices in implementation science (i.e., consultation, modeling, role-play, and feedback informed the training approach). We held separate 2 day MI workshops, led by two randomly assigned MI trainers, for two groups of trainees, each from two of the four participating agencies. Backfill funds provided to each agency accommodated staffing absences and service disruption incurred on training days. Additional training resources were provided to facilitate learning and application of MI in practice, including a two-page MI pocket guide and a summary of central MI concepts in bookmark format. We evaluated participant training satisfaction following training.

Practice and consultation

Following MI training, practitioners were encouraged to use MI in practice for a period of 7 months. During this time, MI trainers led monthly phone consultations for each of the two training cohorts: practitioners and supervisors. MI trainers worked with the agencies they had trained throughout the study, allowing for trust building, consistency, ease of communication, and coordination concerning agenda setting for consulting calls and postconsulting debriefing. Separate consulting calls were held for practitioners and supervisors to address specific needs. Consultation calls were 1.5 hr long for practitioners and 1 hr for supervisors, and followed a similar discussion format (e.g., updates on MI use in practice, discussion of barriers and facilitators, the practice of MI case scenarios, and specific MI skills; see [44] for more on the consultation component of this study). Because supervisors did not have a clinical caseload, their consultation calls focused on core MI skills, facilitators and barriers to MI use by practitioners, and, more rarely, the role-play of MI skills.

Fidelity assessment

As both the AIF and IO taxonomy stress the importance of fidelity, we assessed MI fidelity with monthly audiotaped therapy sessions that featured MI use. Each practitioner audiotaped three sessions before MI training occurred, seven sessions during the consultation stage, and three sessions postconsultation. We assessed MI fidelity in two ways: (a) the Behavior Change Counseling Index (BECCI) [45-46] provided a pragmatic and easy to use measure that we hoped would introduce a practice of fidelity assessment within organizations that did not have one. BECCI feedback was shared with each practitioner confidentially, by email, on three occasions over the 7 month consultation period. (b) Fidelity was also assessed by a Motivational Interviewing Treatment Integrity (MITI) expert coder using the more rigorous MITI on audiotapes submitted by 13 randomly chosen practitioners. The MITI was used to provide rigor and to establish concordance with the BECCI, but given its cost and low practical utility, we scored only a portion of tapes with the MITI.

Sustainability

We undertook five strategies to support sustainability of MI practice: selection of an EBT needed and endorsed by the agencies; adequate implementation preparation before training; agency-led implementation teams; consultation provided by MI trainers only, with no technical assistance provided by the investigators; and introduction of a practical fidelity measure that could be easily used in a practice setting. A final follow-up visit was made to each agency 2 years following the end of the consultation phase to share study findings and explore the sustainability of MI practice and fidelity assessment with the BECCI.

Quantitative measures

Training satisfaction

We assessed MI training satisfaction (i.e., design, content, relevance to participants, and facilitation) with a Training Satisfaction Questionnaire having 17 items rated on a 5-point scale (1, “disagree strongly” to 5, “agree strongly”). Seven additional items (multiple choice answer format or open-ended questions) captured trainer strengths and weaknesses and supports needed to apply new learning. Results were shared with MI trainers in real-time to inform ongoing training.

Readiness, attitudes, and self-efficacy

Four questionnaires were administered at baseline and at postconsultation to evaluate changes in readiness, attitudes, and self-efficacy (i.e., 3 months postconsultation). The Organizational Readiness for Change (ORC) [47-48] captured three broad dimensions of change: motivation, program resources, and organizational dynamics, with 115 items rated on a 5-point Likert scale (1, “disagree strongly”; 5, “agree strongly”). Several questions were adapted to suit a child behavioral health context and some questions omitted, resulting in 60 items. The Brief Individual Readiness for Change Scale (BIRCS) [49] assessed individual readiness for change based on ORC constructs using five questions rated on a 5-point Likert scale (0, “disagree strongly”; 4, “agree strongly”). The Evidence-Based Practice Attitude Scale (EBPAS) [50-51] assessed attitudes toward the adoption of innovations in behavioral health. Fifteen items divided into four subscales are rated on a 5-point Likert scale (0, “not at all”; 4, “to a very great extent”): Requirements (whether a practitioner would adopt an EBP if the agency required it, supervisor, or state); Appeal (the likelihood of EBP adoption if it was intuitively appealing, could be used correctly, or was being used by colleagues who were satisfied with it); Openness (practitioner willingness to try new interventions and use more structured or manual-based interventions); and Divergence (whether practitioners perceive EBPs as not clinically useful and less important than experience). The Personal Efficacy Scale (PES) [52-53] is composed of two scales comprising 17 items from the original Case Management Personal Efficacy Scale [52] rated on a 5-point scale (1, “rarely or never”; 5, “nearly all of the time”), and 11 items from the Counseling Self-Estimate Inventory (CSEI) [53] that form two scales: Difficult Client Behaviors (7 items) and Cultural Competence (4 items).

Fidelity

The primary implementation outcome was MI fidelity, assessed by the MITI and BECCI; primacy is given to BECCI data in this study, as it was available for a larger sample. The 11 items of the BECCI [45, 46] tap various domains of behavior change counseling and MI processes, including permission seeking, evocation, empathy, summarizing, respect for autonomy/support, and thoughts and feelings about behavior change. Items are rated on a 5-point Likert scale (0, “not at all”; 4, “a great extent”), with the total score comprising the average of all items.

Determinant factors

A questionnaire on CFIR constructs was administered following the 7 month consultation period to explore salient factors related to implementation. The CFIR Constructs Questionnaire developed by the authors captured specific aspects of MI implementation related to intervention characteristics (10 items), inner setting (9 items), outer settings (6 items), process (4 items), practitioner characteristics (3 items), and sustainability (9 items). Respondents rated factor importance relative to implementing new programs in behavioral health agencies using a 5-point Likert scale (1, “very unimportant”; 5, “very important”). See Supplementary Material 1.

Qualitative measures

Participant experience with implementation

Eight focus groups (FGs) explored participants’ experiences with MI implementation 3 months following the end of the consultation calls (two per agency, with similar protocols for practitioners and supervisors). FGs lasting between 60 and 90 min were conducted by the same investigators, audio-recorded, and transcribed verbatim. FGs were selected over interviews because they harness individual and collective perspectives, elicit shared experiences, encourage conversational depth and breadth through group interaction, provide a sense of comfort and support for shared and divergent experiences, and engender motivation to contribute to and be a part of the “collective” conversation [54]. FGs elicited shared and divergent experiences from individuals embedded within our four participating agencies. Implementation experiences were also captured with observational notes taken during consultation calls and postconsultation FGs. One investigator present at each of these events was responsible for note taking.

Two year follow-up

A follow-up visit to each agency occurred 2 years following the end of the consultation phase. Visits were conducted by the same two investigators (M.B. and R.B.) who met with the agency’s MI implementation team. Observational notes captured discussions exploring the sustainability of MI practice and fidelity assessment with the BECCI.

Quantitative data analysis

Training satisfaction and readiness, attitudes, and self-efficacy questionnaires

Descriptive statistics summarize data from the questionnaires administered at a single time point. Questionnaires administered at baseline and end line were analyzed using paired t-tests (readiness, attitudes, and self-efficacy).

Research assistants rated the audiotapes of 17 of 22 practitioners (77%) using the BECCI [45-46]. Five practitioners did not have tapes because they were on intake assignment, carried no clinical caseload, or worked in the residential clinic with a group milieu setting and individual session therapy was not possible. Two authors rated randomly selected 20 min audio segments of therapy for each tape; rater 1 coded all audiotapes and rater 2 rated a random selection of 50% of submitted tapes. Independent rating of 30 audio segments and discussion on discrepancies established inter-rater reliability. With a plan for achieving consensus on discrepancies, raters independently rated another 30 audiotapes. All inter-rater assessments measured by ICCs were above .80, and intra-rater reliability for rater 1 was above .70.

Repeated measures ANOVA and paired t-tests data were used to analyze fidelity data collected at multiple time points. Missing information was imputed using multiple imputation techniques in SPSS 22 [55]. We used expectation–maximization methods to impute 100 data files, with one of these imputed datasets randomly selected (Dataset 22) for analysis. Repeated measures ANOVA explored overall change in practitioner fidelity to MI (on BECCI) for 12 of 17 practitioners (71%). Paired t-tests assessed change in practitioner MI fidelity from baseline (Time 1) to post-training (Time 3), from post-training (Time 3) to the end of clinical consultation (Time 11), and from the end of the clinical consultation period (Time 1) to postimplementation (Time 13).

Intraclass correlation coefficients were computed to explore consistency between the MITI and BECCI for 115 tapes from 13 of 17 practitioners (76%), including four men and nine women. Each practitioner submitted 13 audiotapes: three before implementation, one audiotape per month following training and during 7 months of coaching, and three audiotapes postcoaching, one per month. Given that behavior change counseling is considered a tempered version of the MI model and that BECCI and MITI measures assess conceptually related but distinct features of MI practice, our agreement analysis focused on the most salient aspects of MI which are captured by each of the respective measures, namely, practitioner MI spirit and empathy. We computed the intraclass correlation coefficient [(3, k); two-way mixed] for MI spirit and empathy scores between the BECCI and MITI measures, estimating the extent to which these measures of fidelity to MI spirit and MI empathy provide consistent values (see also [56] for companion fidelity paper).

Average ratings and internal consistency were calculated for the six domain scales of the CFIR questionnaire.

Qualitative data analyses

Participant experience with implementation and 2 year follow-up visits

Qualitative data were analyzed inductively using interpretive description (i.e., bottom-up, emerging from the raw data) [57-58] and deductively (i.e., top-down, guided by the CFIR) [13] using QSR NVivo version 10 software. We used the CFIR to guide deductive qualitative coding because it is a determinant framework that captures the relative importance or salience of factors according to the respondents who were part of the implementation process. The present paper reports on the inductive analyses of field notes from the 3 month postconsultation FGs and the 2 year follow-up visits, and deductive analyses of field notes from consultation calls and the 3 month postconsultation FGs (see Supplementary Material 2 for qualitative coding details). Inductive analysis of the consultation call field notes is reported in a companion paper with an exclusive focus on the role of consultation in implementation [59].

RESULTS

Participant characteristics

Seventeen women and five men practitioners received MI training: most were 35–44 years of age (40.9%), with others 18–24 years (13.6%), 25–34 years (18.2%), 45–54 years (13.6%), and over 55 years (13.6%). Slightly more than half (59.1%) held a college diploma, 18.2% held undergraduate degrees, and 22.7% were masters trained. Work experience in their present role ranged from 4 months to 32 years (M = 7.2 years, SD = 6.6 years), whereas overall work experience in service provision ranged from 2 to 35 years (M = 13.9 years, SD = 9.5 years). Queries of prior MI knowledge and training identified that one-third of practitioners (n = 7) had undertaken a previous introductory session or didactic training as part of a larger strength-based training endeavor. Supervisors included seven women and two men: most were 35–44 years (55.6%), 11.1% were 25–34 years, and 33.3% were 45–54 years of age. More than half (55.6%) held a college diploma, 11.1% had an undergraduate degree, and 33.3% were masters trained. Supervisors’ work experience ranged from .92 to 15 years (M = 6.4 years, SD = 4.2 years) in the present role, with overall work experience in service provision ranging from 9 to 28 years (M = 17.9 years, SD = 5.9 years).

MI training

Table 1 presents results of training evaluations and shows good internal consistency and high satisfaction with the trainers and the training approach.

Table 1

Descriptive statistics for participants’ satisfaction with MI training and trainers

	Day 1Mean (SD)	Day 2Mean (SD)
Satisfaction with MI Training*Cronbach’s alpha = .78	3.98 (.42)	4.25 (.37)
Satisfaction with MI Trainers*Cronbach’s alpha = .88	4.14 (.53)	4.36 (.46)

*5-point scale: 1 is “strongly disagree” and 5 is “strongly agree.”

Readiness, attitudes, and self-efficacy

Table 2 provides means and standard deviations for the four measures administered pre- and post-implementation. A series of repeated measures t-tests were performed to determine shifts in participants’ attitudes and beliefs from the beginning to the end of the MI implementation. Most measures (EBPAS, PES, and most of ORC) show no pre-post differences; yet, motivation for practice change on the ORC and individual readiness for change on the BIRCS decrease from the implementation start to the end of clinical consultation and support, 7 months later.

Table 2

Descriptive statistics and t-tests for pre- and post-implementation questionnaire data

	Pre-implementationMean (SD)	Post-implementationMean (SD)	t-test	p value
Organizational Readiness for Change (ORC; scale 1 to 5; n = 27)
Motivation for Change	3.28 (.51)	2.83 (.54)	3.35	0.003*
Program Resources	3.48 (.32)	3.46 (.38)	0.34	0.74
Organizational Dynamics	3.52 (.24)	3.49 (.28)	0.65	0.52
Personal Efficacy Scale (PES; scale 1 to 5; n = 20)
Personal Efficacy	4.29 (.38)	4.33 (.68)	−.28	0.79
Difficult Client Behavior	4.04 (.39)	4.06 (.39)	−.17	0.87
Cultural Competence	4.39 (.56)	4.34 (.48)	.34	0.74
Evidence-Based Practice Attitude Scale (EBPAS; scale 0 to 4; n = 20)
Requirements	1.93 (.59)	1.53 (.93)	1.94	0.067
Appeal	2.24 (.40)	1.99 (.56)	1.78	0.091
Openness	1.89 (.60)	1.70 (.46)	1.45	0.163
Divergence	3.67 (.29)	3.65 (.37)	0.26	0.801
Total	2.45 (.27)	2.26 (.44)	1.99	0.060
Brief Individual Readiness for Change Scale (BIRCS; scale 0 to 4; n = 19)
Total	2.51 (.42)	2.16 (.37)	2.99	0.008*
Total (excluding item 3)	2.18 (.47)	1.79 (.42)	2.80	0.012*

* specifies statistically significant difference (p < .05).

Fidelity

For our interrupted time series analysis, repeated measures ANOVA with a Greenhouse–Geisser correction indicated a significant mean change in total BECCI fidelity scores over the course of implementation [F(1.54, 26.12) = 5.77. p = .01]. Although paired t-tests showed a nonsignificant change in mean total BECCI fidelity scores from baseline (M = 0.97, SE = 0.13) to post-MI training (M = −0.91, SE = 0.99), t(17) = 1.92, p = .07, we saw a significant mean change in BECCI scores from post-MI training to the completion of MI consultation phase (M = 1.75, SE = 0.10), t(17) = −2.568, p = .02 and a nonsignificant change in mean BECCI scores from the completion of MI consultation to 2 months postimplementation intervention (M = 1.81, SE = 0.06), t(17) = −0.44, p = 0.67.

The intraclass correlation coefficient [(3, k); two-way mixed] between BECCI Total Scores and MITI MI Spirit Scores demonstrated adequate consistency [ICC (3, k) = 0.70, p < .001; 95% CI (0.56, 0.79)]. The intraclass correlation coefficient [(3, k); two-way mixed] between BECCI and MITI Empathy measures also demonstrated a significant level of consistency [ICC (3, k) = 0.64, p < .001; 95% CI (0.48, 0.75)].

Determinant factors: CFIR

CFIR questionnaire

Average ratings for the six domain scales of the CFIR questionnaire were centered on the middle point of the scale: intervention characteristics, M = 3.41, SD = .62; outer setting, M = 2.48, SD = .52; inner setting, M = 3.30, SD = .63; individual characteristics, M = 3.56, SD = .68; process, M = 3.30, SD = .82; and sustainability, M = 3.23, SD = .56.

Internal consistency

Internal consistency coefficients for six scales of the CFIR questionnaire were good for inner setting (Cronbach’s alpha = .83) and intervention characteristics (Cronbach’s alpha = .82), acceptable for practitioner characteristics (Cronbach’s alpha = .70) and sustainability (Cronbach’s alpha = .70), weak for process (Cronbach’s alpha = .62), and unacceptable for outer setting (Cronbach’s alpha = .30).

Salience of CFIR constructs in consultation calls and FG

Nine CFIR constructs were particularly salient and mentioned with high frequency during consultation calls and postconsultation FGs (i.e., the top 25th percentile of frequency): structural characteristics; networks and communication; relative priority; leadership engagement, access to knowledge and information (Inner Setting); knowledge and beliefs about the intervention; self-efficacy; other personal attributes (Characteristics of the Individual); and planning (Process construct).

Ten CFIR constructs demonstrated low saliency (i.e., bottom 25th percentile of frequency): tension for change, organizational incentives and rewards (Inner Setting constructs); identification with the organization (Individual Characteristics constructs); formally appointed implementation leader, opinion leader (Process constructs); cosmopolitanism, peer pressure (Outer Setting constructs); and cost, intervention source, and trialability (Intervention Characteristics constructs). The remaining 20 constructs were in the median range. See Table 3 for a joint display detailing the frequency of existing CFIR constructs identified within and across the consultation calls and postconsultation FGs, as well as the saliency of these and our newly proposed CFIR constructs.

Table 3

Joint display depicting the frequency of CFIR constructs from focus groups and consultation calls and proposed new CFIR factors

CFIR CONSTRUCT*	SUMMARY
INTERVENTION CHARACTERISTICS
Intervention Source (0; L)Perception of key stakeholders about whether the intervention is externally or internally developed	No mention of this construct
Evidence Strength & Quality (21; M)Stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes	All practitioners shared multiple examples of successful use of MI during consultation calls. Their examples were all based on direct observation of the effects of using MI skills, such as asking permission and scaling questions on clients. Practitioners expressed surprise at how well MI worked with resistant clients, and how MI spirit empowered their clients but also noted its limitations when used with particular subpopulations of youth (i.e., fetal alcohol spectrum disorder)
Relative Advantage (29; M)Stakeholders’ perception of the advantage of implementing the intervention versus an alternative solution	All practitioners found MI to be a good fit with their clients’ needs, and, in some cases, more effective than a directive approach which can create tension between practitioners and youth. Practitioners also noted that MI could not be used exclusively, requiring integration with other treatments such as cognitive behavioral therapy and brief solution–focused therapy
Adaptability (28; M)The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs	All practitioners talked about their experience of using MI flexibly in practice, not as a full, stand-alone treatment but selectively, weaving elements of MI in and out during a session, as they saw fit (i.e., “MI moments”). Practitioners recognized the need to combine MI with other therapeutic approaches whenever they could not progress with MI, but typically maintained the MI spirit regardless of the therapeutic approach used
Trialability (4; L)The ability to test the intervention on a small scale in the organization, and to be able to reverse course (undo implementation) if warranted	Some of the organizations viewed the implementation of MI as a trial, involving a small part of the organization and providing implementation lessons for an organization-wide implementation. Other organizations intended to use MI for single, walk-in therapy sessions
Complexity (7; M)Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement	None of the four organizations perceived the MI implementation as a difficult and disruptive process. The work did not require of human resource of IT staff and no significant infrastructure or workflow changes were required. At the level of clinical practice, practitioners did not find it difficult to incorporate elements of MI in their routine practice, with notable exceptions related to audiotaping sessions for fidelity assessment and abstaining from being directive and offering solutions to clients, and instead, helping the youth work through the ambivalence
Design Quality & Packaging (23; M)Perceived excellence in how the intervention is bundled, presented, and assembled	Practitioners and supervisors expressed appreciation for the resources received at the MI training and throughout the consultation (e.g., MI pocket guide, MI book, and practitioner checklist) and commented on their quality, usefulness, and appropriateness
Cost (5; L)Costs of the intervention and costs associated with implementing that intervention including investment, supply, and opportunity costs	The cost of implementing MI at the four organizations was supported by the research grant, but supervisors recognized cost as a fundamental challenge if implementation were to be fully financed internally. Cost was noted as the main reason why EBT training typically consists of a 1 day workshop, and why consultation is not provided post-training
**Type of EBT (50; H)This construct is related to whether the intervention is seen as a stand-alone (typically manualized) intervention or an “element” of an intervention, which has implications for how it is learned, delivered and for measuring fidelity	Practitioners and supervisors perceived MI as familiar and not requiring a big learning curve or changes to organizational infrastructure or workflow. They viewed MI as an element, a skill or technique in their toolkit to use as needed. MI was contrasted with Multi-systemic Therapy which is a manualized treatment approach that requires greater implementation effort. Some practitioners viewed MI as the most recent addition to their practice and they used it in combination with other techniques, as part of an eclectic therapeutic approach. The implications for implementation are that more structured, manualized EBTs are typically supported by purveyors who provide significant oversight, coaching, guidance, and evaluation. Fidelity assessment and program evaluation are easier for EBTs that are contained when compared with elements that are integrated alongside other approaches
OUTER SETTING
Patient Needs & Resources (10; M)The extent to which patient needs, as well as barriers and facilitators to meet those needs are accurately known and prioritized by the organization	MI was selected as the EBT of choice by consensus among the four organizations at the outset of the study with an intent to promote client engagement. The fit between MI and client needs was a common topic of discussion on consultation calls (i.e., appropriate and optimal use of MI when working with parents and clients with different needs, such as youth on probation or youth with no parental figures)
Cosmopolitanism (2; L)The degree to which an organization is networked with other external organizations	Networking with other organizations and the value of sharing common experiences was rarely mentioned in the context of MI implementation
Peer Pressure (1; L)Mimetic or competitive pressure to implement an intervention; typically because most or other key peer or competing organizations have already implemented or in a bid for a competitive edge	Only one practitioner made mention of peer pressure from supervisors to implement MI and to audiotape sessions. There was a sense that audiotaping could diminish the quality of the therapy session
External Policy and Incentives (9; M)A broad construct that includes external strategies to spread interventions including policy and regulations (governmental or other central entity), external mandates, recommendations and guidelines, pay-for-performance, collaboratives, and public or benchmark reporting	Most supervisors sensed government pressure to reduce service wait lists and create new programs, which affected the time and energy they could dedicate to implementation. Participants from one organization talked about government pressure to adopt a brief service model (i.e., walk-in service), in line with new funding investments
**Initiating Circumstance for Implementation Endeavor (24; M)This construct relates to the source of implementation support, in other words whether implementation is initiated, driven and/or supported by (i) research funding, (ii) government mandate, (iii) under contract with an EBP purveyor, or (iv) from within the organization	Staff at the four organizations commented on what it meant to participate in an implementation project initiated and facilitated by external research, such as the provision of resources and structure for fidelity measurement, audiotaping, and process. Both positive and negative aspects of research as an implementation source were noted. Negative aspects included a need for an even longer preparation stage (something that was hampered by grant timelines), and a perceived artificiality of process resulting from externally driven decisions. Positive aspects included gaining general knowledge about EBP implementation beyond MI, and motivation derived from working with a highly credible research institution and team. The implications for sustainability were evident in one organization where staff sensed the implementation activities were largely “owned” by the research team, and struggled with how best to take ownership of MI implementation as the research team’s involvement came to an end
**Sector Context (16; M)The sector (health; mental health; education; global health) in which implementation occurs may have idiosyncratic impacts on the implementation process	Staff commented on the several ways in which the children’s mental health sector works (i.e., funding opportunities, resources and implementation facilitation provided by government, and performance expectations) that shape EBT implementation in a way that is likely different from other sectors
INNER SETTING
Structural Characteristics (41; H)The social architecture, age, maturity, and size of an organization	Practitioners and supervisors at all organizations talked about organizational instability resulting from multiple organizational changes occurring throughout the implementation initiative, including amalgamation of two organizations, creation of new clinical roles, and elimination of older roles, and MI learners who were dispersed across different sites. Organizations and individual lives are dynamic, and shifts sometimes led to team members moving sites and becoming isolated from one another or added pressures of assuming new role responsibilities in the midst of adopting a new EBT. Organizational shifts sometimes resulted in practitioners having fewer opportunities to use MI in their new roles (i.e., moving to intake or walk-in counseling services) and becoming distanced from the support of their former learning or service team. In addition, the nonstructured (i.e., milieu therapy; sessions held in transit or in a coffee shop vs. a private office) nature of working therapeutically with youth made audiotaping for fidelity assessment difficult
Networks & Communications (51; H)The nature and quality of webs of social networks and the nature and quality of formal and informal communications within an organization	All four organizations had multiple formal and informal communication channels and structures in place. Formal communication often took place during peer and individual supervision, consultation calls, reminder emails and implementation team meetings. All organizations found the consultation calls and reminder emails helpful and effective in keeping MI “alive” and as a mechanism for sharing the burden of learning new skills. With our aim to maintain some distance from the facilitation efforts (to preserve sustainability options), each organization was tasked with developing an internal implementation team. We learned that team meetings rarely took place in the absence of external (i.e., researcher) facilitation, as their functions and aims were unclear to internal implementation leaders. Failure of internal facilitation proved to be a missed opportunity for facilitating communication and solving emerging implementation issues. Staff in some organizations held frequent, sometimes daily, informal conversations about MI use and client cases. Although this benefitted those who were colocated, those working in other sites felt isolated and unsupported in the change process
Culture (17; M)Norms, values, and basic assumptions of a given organization	Two organizational culture issues were identified in all four organizations. First was a pressure to participate in multiple EBT implementations and training days, which led to “training overload” and made it difficult for staff to consolidate new knowledge and develop competence in any one of the EBTs being implemented. Second was the challenge of creating a fidelity culture in which video- or audio-taping of therapy sessions for fidelity assessment was normative. There was some variability among the four organizations in how fidelity was assessed, but audiotaping—and fidelity assessment generally—was rarely done
Implementation ClimateThe absorptive capacity for change, shared receptivity of involved individuals to an intervention and the extent to which use of that intervention will be rewarded, supported, and expected within their organization
Tension for Change (5; L)The degree to which stakeholders perceive the current situation as intolerable or needing change	Staff talked mostly about the need to change the way they judged their competence because of the pressure of being in the study. The change was from self-report to taping sessions with clients monthly and submitting the tapes to be rated for fidelity to MI by external raters
Compatibility (34; M)The degree of tangible fit between meaning and values attached to the intervention by involved individuals, how those align with individuals’ own norms, values, and perceived risks and needs, and how the intervention fits with existing workflows and systems	Overall MI was perceived as well aligned with the needs of the clients, practitioners’ philosophical approaches and the existing systems, particularly because of its flexible nature, but there were notable exceptions and variations. For instance, practitioners who worked with children with developmental delays, Asperger syndrome, fetal alcohol spectrum disorder, or parents with significant parenting skills, found it difficult to use certain elements of MI and reverted to a more directive approach. In some other cases, when practitioners were moved to doing intakes because of the internal organizational shifts, MI became a worse fit with the new workflow. And finally, part of the perceived incompatibility was related not to the MI practice itself, but to the implementation infrastructure built around it such as taping sessions to assess fidelity to MI, made difficult in situations of crisis, or when dealing with Children’s Aids Society cases
Relative Priority (59; H)Individuals’ shared perception of the importance of the implementation within the organization	For all participating staff, implementation of MI was one of the many tasks occurring simultaneously, which led to reaching absorptive capacity both at personal and organizational level. In addition to being trained in MI, for the duration of the present study practitioners participated in training in other evidence-based treatments, and consequently MI was not always top of the list (i.e., practitioners talked about not finding the time to review note, do readings, listen to their tapes, and reflect on their practice). Although consultation calls were reported as crucial for maintaining the momentum for MI implementation, staff did not perceive MI as a priority because: it did not lead to any significant changes in the organization; there were many other competing projects unfolding simultaneously such as accreditation, other trainings, work strikes, program restructuring related to the newly received funding from the government, etc.; in residential settings, the pace of work was fast with no time for reflective moments, which are essential when implementing a new evidence-based treatment. Some supervisors expressed that they did not do sufficient pull and sell of MI because of these competing priorities
Organizational Incentives & Rewards (2; L)Extrinsic incentives such as goal-sharing awards, performance reviews, promotions, and raises in salary and less tangible incentives such as increased stature or respect	Organizational incentives and rewards for MI delivery in practice were rarely mentioned, and were related to considering MI knowledge and skills when hiring new staff, as well as being part of the performance evaluation for some practitioners at one of the four organizations
Goals & Feedback (34; M)The degree to which goals are clearly communicated, acted upon, and fed back to staff and alignment of that feedback with goals	Practitioners received feedback on their fidelity to MI from the research team and overall found this specific feedback helpful in motivating them to improve the weaker skills and set sub-goals in their process of learning MI. Although limited in that it looked at a randomly selected 20 min segment of the audiotape, this feedback was valuable particularly because supervision often focused on administrative and not clinical issues
Learning Climate (34; M)A climate in which (a) leaders express their own fallibility and need for team members’ assistanceand input; (b) team members feel that they are essential, valued, and knowledgeable partners in the change process; (c) individuals feel psychologically safe to try new methods; and (d) there is sufficient time and space for reflective thinking and evaluation	Participants at all organizations talked about a constant struggle to find time to do readings, practice new skills or listen to their tapes, which made the monthly consultation calls and the support they received from the MI trainers and the research team essential for their professional development. On the consultation calls, the majority of practitioners felt safe to practice the newly acquired MI skills, with a few exceptions of practitioners who preferred to observe others do role-plays instead of engaging in role-plays themselves. Apart from the support and recognition received from the research team and the MI trainers, there was variability in practitioners’ experience of having their efforts to implement MI acknowledged and valued by the supervisors
Readiness for ImplementationTangible and immediate indicators of organizational commitment to its decision to implement an intervention
Leadership Engagement (36; H)Commitment, involvement, and accountability of leaders and managers with the implementation	Practitioners and supervisors at the four organizations discussed various degrees of commitment and involvement with MI implementation from the leadership (i.e., frontline supervisors): some practitioners felt unsupported, in part because the supervisors were not always competent in MI themselves; at some organizations supervisors were largely disengaged and their role limited to reminding practitioners to submit tapes monthly, without helping them to sort through the taping difficulties they were experiencing; in some cases MI was a consistent part of the supervision meetings, MI even becoming the style of supervision, whereas other supervision meeting focused exclusively on administrative issues and paperwork. Supervisors themselves acknowledged that they gave priority to other organizational work that appeared as more urgent than MI, particularly because there was no clear accountability for supervisors
Available Resources (33; M)The level of resources dedicated for implementation and on-going operations including money, training, education, physical space, and time	The present implementation project, being largely supported by research funding, put fewer demands on the organizational resources, and created a relatively unique situation in which staff were satisfied with the resources available for implementation. Supervisors noted that this was in contrast with the typical implementation model with funding from the government, which consistently underestimated the resources needed for an effective implementation. Most comments referred to time, a scarce resource for everyone involved in the implementation but especially for staff in the residential treatment programs. In addition, practitioners noted a mismatch between the more difficult profiles of the clients they were serving and the straightforward case examples used in training and consultation
Access to Knowledge and Information (45; H)Ease of access to digestible information and knowledge about the intervention and how to incorporate it into work tasks	Consultation calls were perceived by all staff as an easy and systematic way of accessing information about MI, practicing skills, and asking questions about its delivery in practice with various client populations. Some practitioners noted that consultation compensated for the lack of clinical supervision they received within their organizations, which was consistent with supervisors acknowledging feeling unprepared to supervise MI practice. Similarly, staff working in isolation at remote sites made it “difficult to utilize each other as resources”
CHARACTERISTICS OF INDIVIDUALS
Knowledge and Beliefs about the Intervention (56; H)Individuals’ attitudes toward and value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention	Overall, both practitioners and supervisors valued the positivity of the MI approach, the way in which MI spirit empowered the clients and allowed the practitioners to walk alongside the clients, as opposed to stepping ahead of them. Some practitioners perceived an immovable difference between their personal therapeutic style (e.g., “go in and take over”) and MI principles (e.g., permission-seeking), which made it difficult to adopt MI in their practice. In some cases, there was a lack of clarity or misconceptions about the appropriateness of MI when working with clients who needed clear direction, talked a lot, or were at the beginning of their therapy
Self-Efficacy (45; H)Individual belief in their own capabilities to execute courses of action to achieve implementation goals	Most practitioners rated their MI competence as average or above average, and noted the discrepancy between knowledge and use, as well as MI spirit and MI skills, with MI knowledge and spirit exceeding use of MI, in particular of specific MI skills. There was a range of perceived confidence in applying MI to practice: practitioners who were used to giving solutions to youth did not feel confident in their MI skills and their ability to “sit back”; other practitioners expressed confidence in their ability to deliver MI, especially with a segment of clients. Supervisors generally, did not feel sufficiently competent to assess their practitioners’ use of MI and support their clinical practice
Individual Stage of Change (14; M)Characterization of the phase an individual is in, as he or she progresses toward skilled, enthusiastic, and sustained use of the intervention	Supervisors selected staff who were highly motivated, able to manage new challenges and at the appropriate stage in their professional development; however, as a results of internal organizational shifts and changes in clinical roles, for some practitioners other tasks took priority and learning MI did not fit well anymore with their professional trajectory. At one organization, supervisors who had the whole team trained in MI, expressed initial apprehension related to the outcomes of this implementation project at the organizational level. Many practitioners characterized participation in the MI project as a step forward which allowed them to gain new clinical skills and reflect on their own clinical competence, although most of them felt at the beginning stages with MI
Individual Identification with Organization (4; L)A broad construct related to how individuals perceive the organization and their relationship and degree of commitment with that organization	There were very few references to staff’s identification with their organization, largely related to a supervisor perceiving one practitioner as not being fully committed to the MI implementation (i.e., not taping any sessions for the duration of the implementation project) because she was in the process of exploring other career options
Other Personal Attributes (62; H)A broad construct to include other personal traits such as tolerance of ambiguity, intellectual ability, motivation, values, competence, capacity, and learning style	Staff commented on their learning style and noted selected aspects of the training and consultation: staff valued the use of practical resources such as the MI pocket guide, which served as an effective and quick reminder of the main MI concepts and could be used during the therapeutic sessions; the opportunity to apply MI and practice weaker skills and expressed their preference for either watching other practitioners do role-playing or actively practising skills; instead of doing readings on MI, staff preferred watching videos with MI experts demonstrating MI skills. Overall staff were motivated to participated in this implementation project (i.e., no one dropped out despite many changes in the roles within the organization; motivation was a selection criterion used for participation in this project). Staff expressed their excitement to be part of the project, particularly because it was facilitated by a research team, but at the same time, they felt overwhelmed and immobilized in their work because of the magnitude of their clients’ issues
**Clinical Paradigm (7; M)This construct is related to characteristics of the practitioners delivering an intervention such as their professional formation and training and they fit with the intervention	Overall MI was perceived to work well and be a good fit with both new and experienced practitioners, but practitioners’ educational background appeared as a significant factor determining the appropriateness and effectiveness of MI: for social workers, MI style matched their educational training and clinical approach, whereas for child and youth workers who were trained to be directive, intervene and tell youth what they needed to do – “go in and take over”—MI appeared as a misfit
PROCESS
Planning (40; H)The degree to which a scheme or method of behavior and tasks for implementing an intervention are developed in advance and the quality of those schemes or methods	The present implementation was different than the typical implementation experiences that organizations had because it was funded, planned and facilitated as part of a research project. Some supervisors commented on the artificial nature of the implementation given that it was not internally driven, whereas others found relief not having MI experts and an implementation plan set by the research team. Overall, participating staff valued the attention given by the research team to implementation drivers and strategies (i.e., consultation, feedback, implementation team) and contrasted it to the typical model of sending people off for training or holding in-house training. At the same time, participating staff noted several aspects of the implementation that were insufficiently planned: brief pre-implementation period, staff felt unprepared and disengaged in the initial phase of the implementation; examples used in training and consultation were sometimes not relevant to the client populations served by the four organizations, and, similarly, MI trainers/ consultants were not sufficiently familiar to the organizational conditions and procedures at the four organization; the plan to have supervisors learn MI without carrying a clinical caseload and at the same time supervise the practitioners’ MI practice led to supervisors feeling unprepared for the task
EngagingAttracting and involving appropriate individuals in the implementation and use of the intervention through a combined strategy of social marketing, education, role modeling, training, and other similar activities
Opinion Leaders (2; L)Individuals in an organization who have formal or informal influence on the attitudes and beliefs of their colleagues with respect to implementing the intervention	The role of opinion leaders in the uptake of MI was briefly discussed only at one organization. Supervisors talked about the influence of one practitioner, who had strong credibility in the team (i.e., a peer opinion leader), on persuading the rest of the team about the benefits of MI
Formally Appointed Internal Implementation Leaders (1; L)Individuals from within the organization who have been formally appointed with responsibility for implementing an intervention as coordinator, project manager, team leader, or other similar role	The construct of formally appointed internal implementation leaders was mentioned only at one organization: supervisors noted their failure to appoint an MI implementation leader, along with implementation team not meeting for the duration of the project, as a significant gap in the implementation process
Champions (6; M)“Individuals who dedicate themselves to supporting, marketing, and ‘driving through’ an [implementation]”, overcoming indifference or resistance that the intervention may provoke in an organization	At one of the organizations, supervisors talked about the emergence of a MI champion among the practitioners, who took initiative in finding additional MI resources and organizing internal MI training sessions for new hires. Champions were mentioned at a second organization as a necessary condition for MI to be sustained in the organization beyond the life of the research project
External Change Agents (31; M)Individuals who are affiliated with an outside entity who formally influence or facilitate intervention decisions in a desirable direction	Staff at all organizations talked about the facilitating effect that the MI consultants and members of the research team had on the uptake of MI in their practice and continued motivation for the implementation project. Being part of the consultation calls post-training “made the experience” for the practitioners because they improved their skills and knowledge and felt what was like to be at the receiving end of MI, because consultants used MI as a consultation style. Similarly, receiving feedback on MI competence from members of the researchers who were perceived as unbiased was preferred to receiving feedback from the peers or supervisors within the organization
Executing (33; M)Carrying out or accomplishing the implementation according to plan	In terms of carrying out the implementation of MI according to the plan largely set by the research team, staff at the four organizations talked about several types of barriers and difficulties they experienced: (a) barriers related to taping, which impacted the timeliness of tape submission for feedback—these were related to therapy sessions happening on the go, in the car or at coffee places; clients often refusing to be taped; the audio-recorder altering the nature of the session; (b) difficulties filling out some of the paperwork required by the research team because of a perceived misfit between the practitioners’ goals/ therapeutic approach and the approach to therapy implied by the paperwork; (c) the perception that the implementation team did not fulfill its intended role at any of the four organizations
Reflecting and Evaluating (34; M)Quantitative and qualitative feedback about the progress and quality of implementation accompanied with regular personal and team debriefing about progress and experience	Although there was variability between the four organizations in the use of audiotapes to reflect on and evaluate clinical practice, this was not done routinely by any of the participating staff. For this reason, the opportunities for reflection and evaluation of the implementation of MI opened by the present project were perceived as both valuable and anxiety-provoking by the practitioners. The fidelity to MI feedback was helpful at the individual level in keeping practitioners accountable and orienting them to areas that needed improvement, but staff felt that it was not part of supervision, group meetings or a larger infrastructure that would support learning and competence. All practitioners valued the consultation calls for the opportunity to take time from the hectic pace of everyday work and reflect on what works and what does not work well. Overall staff perceived the way the implementation team worked as a failure because team members did not take the time to meet and assess the progress and quality of the implementation

*Construct Frequency and Salience Level: H/L/M = high/low/medium saliency based on percentile (H = top 25th and L = bottom 25th).

** specifies constructs emerging from the study data that are proposed as additional to the original CFIR framework [11].

Proposed CFIR constructs

Our deductive analysis suggested the addition of four new CFIR constructs: (a) Professional training (Individual Characteristics) captures the fit between the practitioner’s foundational clinical paradigm and that underlying the EBT being implemented. For example, practitioners working in residential care functioned in a psychoeducational paradigm, whereas MI is based on a less prescriptive paradigm. (b) Initiating circumstance for the implementation endeavor (Outer Setting) refers to the entity initiating and driving the implementation endeavor, the source and type of implementation support, the timeline, approach, pacing, and endorsement of implementation outcomes. We propose four common initiating circumstances for implementation work: (i) researcher-initiated; (ii) government initiated, typically with some provision of technical assistance; (iii) organizationally self-initiated; and (iv) EBT developer, intermediary, or purveyor initiated (e.g., MST, Triple P). (c) Sector (Outer Setting) refers to the context in which the implementation endeavor occurs (e.g., health, mental health, education, and global health). (d) Type of evidence-based treatment (Intervention Characteristic) relates to whether the EBT is delivered in a stand-alone manual–based fashion or as an evidence-based element [16], which has implications for training, consultation, fidelity measurement, and sustainability.

Themes emerging from the implementation experience

Seven themes emerged from the inductive analysis and are briefly discussed below, and more fulsomely in relation to CFIR factors in Supplementary Material 3.

Changes in clinical and supervisory practice resulting from MI implementation

Practitioners discussed acquiring MI spirit and increasing MI competence. Although most practitioners did not feel highly competent in their MI skills postconsultation, they acknowledged their learning progress, its impact on the clients, and their intention to continue improving their skills. Not all MI skills were valued from the beginning of the learning process. This suggests that limiting EBT skill acquisition to a discrete training workshop, without additional ongoing consultation, may result in missed learning opportunities. Certain MI skills evolved over time, such as the MI technique of asking permission. Practitioners were initially quite resistant to this technique during training, but shifted toward ambivalence and then full appreciation after observing the effects on clients and practicing the technique during consultation calls.

Supervisors focused on their lack of competence in supervising MI practice. They found MI consultation helpful for strengthening their MI knowledge base and exploring how MI could be used as a supervision style, but this did not compensate for the lack of opportunity to practice MI skills in practice. This feeling was further compounded by not having access to the MI fidelity feedback that was confidentially shared with practitioners. Supervisors felt like impostors because they did not carry a caseload, could not practice MI, and perceived practitioners’ MI competence as more advanced than their own.

Space for reflection and the possibility of intervention fidelity

Practitioners noted a change in attitude toward their clinical practice, specifically a heightened practice consciousness. They reported that various elements of the MI implementation experience (e.g., MI training, monthly consultation calls, regular feedback, audiotaping therapy sessions, and access to the MI fidelity tool) slowed their practice pace to make room for self-reflection. They learned about their MI technical strengths and weaknesses, tracking their learning progress, selectively focusing their readings to develop their weaker MI skills, and, more rarely, reviewing the BECCI before the therapeutic session to prepare their MI techniques methodically. They found audiotaping of therapy sessions challenging but noted it enabled observations and reflections that were facilitative of creating a fidelity culture within the organization. Many felt discomfort and anxiety about audiotaping therapy sessions, perceiving it as unnatural. Consequently, fidelity assessment was viewed as artificial. With one exception, practitioners reportedly never listened to their audiotapes throughout the duration of the study. Although this would have provided a means to reflect on their performance, they did not do it, either due to time constraints or anticipated discomfort.

Moving beyond learning MI techniques toward learning how to implement EBTs more generally

All participants talked about acquiring dual gains from implementing MI, specifically, acquiring a new clinical practice while learning how to implement EBTs. They contrasted this implementation with their previous implementation experiences that were limited to a training component typical of the train and hope approach. Both practitioners and supervisors felt that the MI experience enhanced the salience of specific implementation features (e.g., consultation) and facilitated reflections on implementation lessons for the future.

Implementing EBTs as evidence-based elements versus manual based

Participants identified MI as a tool in their clinical toolkit that was accessible with particular clients, as needed. MI was perceived as a clinical skill that could be combined with other therapeutic practices, but “not enough of an EBP,” stand-alone treatment, or “entity in and of itself.” MI was not regarded as a new practice but rather as a refresher of core skills already within their clinical repertoires. They acknowledged that some MI skills were new (e.g., guiding clients in weighing pros and cons to changing or maintaining the status quo; asking scaling questions), but their perception overall was one of “increasing the skill set that they already had rather than implementing something brand new.”

Factors hindering MI implementation

Implementation challenges related to organizational factors, practitioner educational background, client characteristics, and EBT characteristics. Although we endeavored to include practitioners who were part of stable clinical teams, all of the agencies experienced several significant organizational shifts that affected their clinical teams during active implementation (e.g., agency amalgamations and new funding for new programs).

Some child and youth workers perceived MI as discrepant from their therapeutic style and educational training, and this presented an implementation barrier. Other challenges related to MI characteristics, such as the fit between MI and client characteristics, and difficulty with fidelity assessment. Practitioners experienced difficulty in using MI with youth presenting with developmental delay, Asperger’s syndrome, fetal alcohol syndrome, youth in crisis or with a lifelong history of neglect, or where there was a sense of urgency for immediate intervention. Some practitioners found audiotaping therapy sessions for fidelity assessment to be “the most challenging part of the implementation.”

Factors facilitating MI implementation

Participants valued and were motivated by the MI philosophy, noting its positivity, universality, slow pace, and empowering effect it had on clients. There was a universal acknowledgment that the primary driver of their implementation experience, what kept it alive despite the challenges, was a staged AIF-informed implementation approach comprised of attention to pretraining preparation, use of didactic educational materials, interactive training, ongoing clinical consultation from MI experts, and fidelity feedback. They perceived this approach as having been the most in-depth, comprehensive, prolonged learning experienced throughout their professional careers.

Implementation ownership and the intention to sustain

Only at the end of the study did participants recognize the importance of taking ownership of the implementation work. Although we were as hands-off as possible, our guidance of the research evaluation was perceived as leading the implementation process. By the end of the study, however, participants recognized the need to assume control of implementation decisions, but they lacked a plan for achieving this transition. All participants expressed an intention to sustain MI in their organizations and discussed possible ways they might do so.

Implementation success and sustainability

Implementation success was operationalized in two ways. First, we deemed implementation to be successful if clinicians demonstrated improved MI fidelity over time; which they did. Second, follow-up discussions conducted with each agency 2 years postimplementation revealed that MI remained in use in the four agencies. MI implementation was perceived to have left a small clinical legacy such that elements of MI spirit, permission seeking, weighing pros and cons, and scaling questions were commonly used by practitioners as they had proven effective in maintaining client engagement; this had, in fact, been the identified clinical need driving the initial selection of MI. Participants discussed the importance of consultation for consolidating new clinical skills and made efforts to incorporate consultation into their implementation approaches whenever possible. None of the agencies maintained the practice of assessing MI fidelity with the BECCI, or otherwise.

DISCUSSION

A complementary implementation approach informed by the AIF [12], CFIR [13], and IO taxonomy [14] was successful in supporting the implementation of MI in four child behavioral health agencies as evidenced by improved clinician MI fidelity over time and moderate sustainability of MI skills beyond the study. The implementation strategy included targeted pretraining preparation, focused interactive training, and 7 months of follow-up consultation and fidelity feedback. Our findings are consistent with approaches and outcomes described for implementation of Applied Behavior Analysis treatment in community organizations serving children with autism [60] and for motivational enhancement therapy in community addiction treatment programs [29].

Demonstration of improved fidelity following 7 months consultation rather than immediately following MI training complements participants’ reflections on the value of consultation and feedback for skill building, consolidating knowledge, and addressing implementation barriers. The comprehensive elements of our complementary implementation approach and gradual improvement in MI fidelity highlight the importance of a staged implementation approach that utilizes multiple active training methods beyond didactic training and illustrates why a train and hope approach rarely leads to successful implementation. Clinicians and supervisors perceived consultation as effective because it helped us to “keep motivational interviewing alive,” fulfilled a profound learning role through collaboration and connection with others, and offered protected time for reflection on practice change.

No changes were evident over time in practitioners’ attitudes toward evidence-based practices (EBPAS), their perceptions of personal efficacy (PES), or for most organizational readiness for change factors. Lack of change on these characteristics and observed decreases in practitioner motivation and individual readiness for practice change could reflect late onset realization of the complexities inherent in practice change; practitioners may be unaware of how implementing a new EBT will affect their service provision or practice until they are in the midst of the process.

Saliency of CFIR constructs

CFIR questionnaire data aligned well with qualitative findings on strengths and weaknesses for implementing MI, the discrepancy in the pace of advancing MI knowledge versus skills, and emerging realization of the extensive time needed to automatize skills, make fast decisions, and reach a high level of MI competence. Overall, there was general congruence between quantitative and qualitative CFIR data, with qualitative data adding rich detail on the how and why of observed quantitative results related to training and perceived competence. The study provided external validation for CFIR and extends it by proposing new factors that are not addressed in the framework nor elsewhere in the literature and that could be explored in future research.

Nine constructs emerged as highly salient in consultation calls and postconsultation FGs. Five Inner Setting constructs were particularly salient in the implementation experience, including structural characteristics; networks and communication; relative priority; leadership engagement; and access to knowledge and information. These factors are consistent with previous research highlighting the importance of focusing pretraining preparation on inner setting factors to foster optimal organizational conditions for change [61-64].

An additional four highly salient constructs aligned with Practitioner Characteristics (knowledge and beliefs about the intervention; self-efficacy; other personal attributes) and process (planning). In this study, practitioner knowledge of MI stemmed from training and consultation, but their beliefs were primarily informed by perceptions of how well MI worked for their clients and the goodness of fit between MI and client needs. Although less commonly examined (e.g., [62] did not collect data for Individual Characteristics), studies that have looked at this domain also highlight the importance of practitioner knowledge and beliefs about the intervention [65]. Moreover, the emergence of self-efficacy as a salient construct is consistent with research showing associations between practitioner self-efficacy and higher practitioner skill and performance, and a higher likelihood of investing effort in and sustaining a new practice [66].

Low saliency CFIR constructs included tension for change and organizational incentives and rewards (Inner Setting), identification with the organization (Individual Characteristics), formally appointed implementation leader and opinion leaders (process), cosmopolitanism and peer pressure (Outer Setting), and cost, intervention source, and trialability (Intervention Characteristics). The relative low saliency of these constructs may be related to the fact that a research context served as the initiating circumstance for the implementation endeavor. Research funding covered implementation costs (clinical backfill, training, and fidelity assessment) and removed the option for trialability and the need for networking and peer pressure. Participants viewed the research team as the formally appointed implementation leaders, despite our efforts to “intervene” as little as possible and to situate ownership within the internal implementation teams. Late stage ownership of the implementation endeavor emerged as a central qualitative theme, as did recognition that lack of practitioner buy-in negatively affected sustainability and motivation.

Proposed new CFIR constructs

Four new factors emerged to potentially extend the CFIR that are not discussed in the literature nor included in the framework. Professional training (Individual Characteristics) addresses the fit between the practitioner’s professional paradigm and that underlying the EBT being implemented; this is not captured in the CFIR construct of other personal attributes (personal traits of ambiguity, intellectual ability, motivation, competence, and learning style). The issue of EBT fit with the practitioner’s dominant clinical paradigm was particularly evident in residential settings staffed by psychoeducationally oriented child and youth care practitioners who were inclined to be prescriptive and provide behavioral scaffolding for youth, and who struggled with an EBT that was nonprescriptive by design.

Two newly proposed constructs align with the Outer Setting domain to capture the initiating circumstance of the implementation endeavor (i.e., the context or entity initiating and driving the implementation initiative) and the sector in which the implementation occurs (i.e., health, mental health, education, and global health). Both of these factors have implications for implementation approach, ownership, buy-in, sustainability, allocation of resources, and cost.

Finally, the type of evidence-based treatment (Intervention Characteristics) to be implemented—manual based or an evidence-based element, was identified as having implications for training, delivery, and fidelity measurement. EBTs that function as a therapeutic element [16] and that can be combined with other approaches can make it challenging to measure penetration, level of institutionalization, and service access. Although this proposed factor is relevant to behavioral health care, it may conceivably extend to other health domains where quality or clinical standards are as prescriptive as manuals. This is a question for future investigation.

Findings contribute to the convergence of support for the CFIR across several studies using a similar methodological approach in different contexts ( M. Barwick, R. Barac, S. Zlotkin, unpublished data, 2015) [61, 65, 67]. Knowing which CFIR constructs are commonly associated with implementation success across settings helps us to refine the model and better inform an implementation approach that can optimize the selection of implementation strategies.

Practice implications

As with previous research [29-31], the manner in which consultation and clinical supervision facilitate practitioner competencies for a new EBT is an essential consideration for implementation. At issue is how clinical supervisors and others can best provide clinical supervision for a new EBT when they may have little or no opportunity to practice it. New clinical roles, such as clinical practice lead and clinical coach, have been used in some settings to support implementation [68, 69] and should be considered in the first phase of implementation work.

Qualitative findings support the argument that achieving and assessing fidelity in usual care are challenging [70-73]. In the present study, implementation outcome was evaluated by MI fidelity and informed by qualitative data on efforts to sustain MI practice following the consultation phase. Although our measure of MI fidelity, the BECCI, was easy to use, quick, and practical, practitioners did not sustain fidelity assessment beyond the research study, primarily because they felt that it was unnecessary to do so in the face of their clinical experience. The practical nature of fidelity assessment is a common challenge [73, 74], but this is not the only obstacle. We learned that establishing a regular practice of fidelity assessment requires an organizational culture that recognizes fidelity as essential for achieving high clinical competence and quality outcomes. Moreover, there is added complexity in assessing fidelity for EBTs that are not manual based. Difficulty in establishing a practice of fidelity assessment is a pervasive issue among community mental health provider organizations that, to date, has not received much attention in the literature.

We learned that a preimplementation period, as prescribed by the AIF, is useful before EBT training, to prepare the organizational conditions for practice change. This phase involves building consistent communication and engagement at all levels of the organization to ensure readiness and buy-in. Implementation teams need guidance and clarity concerning roles and tasks, without which they do not attend implementation meetings, and the process slows or fades out entirely. Recruiting practitioners from stable teams at the onset and minimizing staff turnover are essential to successful and sustained implementation, albeit challenging in dynamic work environments. The fit between the practitioners’ clinical paradigm and that underlying the targeted EBT is also an important consideration [31]; a psychodynamically influenced EBT may not fit well in a psychoeducational environment, for instance.

Limitations

Research funding was the initiating circumstance for this implementation endeavor, potentially influencing the implementation process in ways that may prove different from an alternative initiating circumstance. How different initiating circumstances may affect implementation requires further study. Second, this study focused on MI fidelity and only lightly touched on other implementation outcomes in planning (i.e., acceptability) and qualitative analyses (i.e., appropriateness, feasibility, sustainability, and cost). Third, sustainability data collected 2 years postconsultation were not part of the initial research plan. However, given the lack of sustainability evidence in the literature, our findings offer some insight into the long-term survival of an EBT in community behavioral health organizations and the difficulties of introducing and sustaining fidelity assessment within usual practice. Finally, the small number of organizations and practitioners and the child behavioral health context limits the generalizability of our conclusions.

CONCLUSIONS

This case study demonstrated the usefulness of an implementation approach informed by several complementary models in the context of implementing MI in child behavioral health agencies. In contrast with the typical train and hope approach to implementation, we demonstrated that practice change does not occur immediately post-training but rather emerges over a period of consultation support and use in practice, following a discrete interactive training period.

The saliency of CFIR constructs in this study aligned well with evidence from similar studies in other contexts, building external validity for the framework and highlighting common or universal constructs that can inform planning and measurement. Novel CFIR factors, not elsewhere reported in the literature, were identified that could potentially extend the framework if validated in future research.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.