The case for prioritizing implementation strategy fidelity measurement: benefits and challenges.

Implementation strategies are systematic approaches to improve the uptake and sustainability of evidence-based interventions. They frequently focus on changing provider behavior through the provision of interventions such as training, coaching, and audit-and-feedback. Implementation strategies often impact intermediate behavioral outcomes like provider guideline adherence, in turn improving patient outcomes. Fidelity of implementation strategy delivery is defined as the extent to which an implementation strategy is carried out as it was designed. Implementation strategy fidelity measurement is under-developed and under-reported, with the quality of reporting decreasing over time. Benefits of fidelity measurement include the exploration of the extent to which observed effects are moderated by fidelity, and critical information about Type-III research errors, or the likelihood that null findings result from implementation strategy fidelity failure. Reviews of implementation strategy efficacy often report wide variation across studies, commonly calling for increased implementation strategy fidelity measurement to help explain variations. Despite the methodological benefits of rigorous fidelity measurement, implementation researchers face multi-level challenges and complexities. Challenges include the measurement of a complex variable, multiple data collection modalities with varying precision and costs, and the need for fidelity measurement to change in-step with adaptations. In this position paper, we weigh these costs and benefits and ultimately contend that implementation strategy fidelity measurement and reporting should be improved in trials of implementation strategies. We offer pragmatic solutions for researchers to make immediate improvements like the use of mixed methods or innovative data collection and analysis techniques, the inclusion of implementation strategy fidelity assessment in reporting guidelines, and the staged development of fidelity tools across the evolution of an implementation strategy. We also call for additional research into the barriers and facilitators of implementation strategy fidelity measurement to further clarify the best path forward.

Implementation strategy fidelity is under-developed and under-reported, and the quality of reporting is decreasing over time.

This position paper describes the costs and benefits of implementation strategy fidelity. We ultimately call for the continuation and improvement of implementation strategy fidelity measurement while offering pragmatic solutions to noted challenges.

Future research is needed regarding the barriers and facilitators to implementation strategy fidelity measurement/reporting, the costs and cost–benefits of implementation strategy fidelity measurement, and the extent to which implementation strategy fidelity moderates the relationship between an implementation strategy and implementation outcomes.

BACKGROUND

This paper examines the state of implementation strategy fidelity measurement and argues for its improvement. We begin by framing the importance of implementation strategy fidelity by first defining fidelity as it is classically understood, in relation to intervention fidelity measurement, before expanding that definition to consider fidelity of implementation strategies. We then describe the benefits and challenges related to its measurement and suggest action steps implementation researchers might take to overcome them. We ultimately conclude that the benefits of implementation strategy fidelity measurement outweigh the costs, and call for changes at multiple levels and future research that might facilitate better measurement.

Intervention fidelity

Fidelity to an intervention represents an important implementation outcome in both research and practice settings [1-3]. Defined as the extent to which an intervention is implemented as originally intended, fidelity plays a central role in the assessment of a Type-III research error [2-5]. A Type-III error is defined as failure to implement an intervention as planned, leading to an erroneous conclusion that null results are due to attributes of the intervention itself, rather than to its mal-implementation [5]. Intervention fidelity also operates as a moderator of main effects pathways, such that efficacious interventions carried out with higher fidelity tend to yield better clinical outcomes compared to the same interventions delivered with lower fidelity [2, 6]. Intervention fidelity remains an important means of quality assurance in practice settings and in implementation research. Poor fidelity often explains why interventions that perform well in controlled research settings show worse outcomes in practice settings [7, 8]. Reviews of implementation outcome measurement also describe intervention fidelity as one of the most common targets of implementation strategies [9-11].

Implementation strategies

Implementation strategies are “deliberate and purposeful efforts to improve the uptake and sustainability of [evidence-based interventions (EBIs)],” to proximally affect implementation outcomes (e.g., adoption, appropriateness, feasibility, acceptability) that in turn distally affect service system and clinical outcomes through the successful implementation of an EBI [12]. Implementation strategies span multiple levels and categories [13, 14]. Examples include the restructuring of physical or virtual spaces, financial incentives, training, supervision, etc. Strategies address specific barriers to implementation of an EBI. For instance, if a team of providers newly trained in cognitive behavioral therapy (CBT) are struggling to maintain fidelity in their sessions with clients, implementation strategies like promoting supervision or audit-and-feedback might re-enforce the counselors’ quality of CBT delivery, in-turn improving CBT fidelity, and ultimately lead to better client outcomes. The relationship among implementation strategies, implementation outcomes, and clinical or service outcomes are depicted in the Conceptual Model of Implementation Research developed by Proctor et al. (2009) (Fig. 1).

Fig. 1

Conceptual model of implementation research. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34. https://doi.org/10.1007/s10488-008-0197-4.

Because implementation research regularly aims to improve clinical practice (e.g., clinical guideline adherence), some of the most common implementation strategies are inextricably linked to human behavior change (e.g., training, coaching, audit-and-feedback) [15-17]. A recent review by Lewis and colleagues (2020) found that implementation strategies were most commonly: (1) utilized in behavioral/community mental health settings, (2) targeted at improving behavioral EBIs, and (3) informed by behavior change theory [18]. Because behavior change efforts in healthcare are often complex, Lewis et al. (2018) focus on the importance of identifying mechanisms within implementation strategies most responsible for causing change. The authors ultimately call for improvements in several areas related to implementation strategy specification and measurement, including the measurement of moderating variables like implementation strategy fidelity [19].

Implementation strategy fidelity

Similar to intervention fidelity, fidelity to an implementation strategy is defined as carrying out the strategy as it was designed [20]. Despite its importance, measurement of implementation strategy fidelity is both under-developed and under-reported [20, 21]. In their scoping review of fidelity measurement to implementation strategies, Slaughter et al. (2015) concluded that fidelity domains were on average inadequately measured across trials, and that few reports of implementation strategy fidelity existed [20]. The authors also found that the quality of reporting on fidelity to implementation strategies demonstrated a statistically significant decline over time and suggested their results may be due to a lack of measurement tools [20]. The poor reporting on descriptions of implementation strategies, combined with the lack of fidelity description regarding those same strategies, compound to create an environment where it is not always clear which strategies were performed nor how well they were performed. Given these challenges, it is understandable that implementation strategy fidelity has yet to take on a consistent focus within trials of implementation strategies.

In this position paper, we describe a confluence where a desire for conceptual and methodological rigor (e.g., fidelity instruments with demonstrated psychometric properties), runs into common logistical research challenges (e.g., the measurement of a complex variable and data collection costs). In the sections below, we carefully consider the benefits and challenges regarding the value of implementation strategy fidelity measurement and reporting. We ultimately argue that both measurement and reporting need to improve, and we offer pragmatic short-term solutions and structural long-term remedies.

DISCUSSION

Benefits of implementation strategy fidelity measurement

Like fidelity measurement of intervention delivery, fidelity measurement of implementation strategy delivery strengthens the quality of implementation trials. Fidelity measurement facilitates the assessment of a Type-III research error in clinical trials, a paradigm that extends to implementation research [2-5]. Without implementation strategy fidelity assessment, researchers cannot be assured that implementation strategies (or the mechanisms most responsible for change within them), are responsible for impacts on implementation or clinical outcomes. Or conversely, that lack of impact on clinical outcomes is due to limitations of an implementation strategy’s mechanisms as opposed to limitations in its fidelity.

A lack of implementation strategy fidelity measurement may in part explain variations in effectiveness reported across trials of implementation strategies. Although reviews of implementation strategies tend to show significant and positive impacts on outcomes in expected directions, such reviews often note wide variations in effectiveness among studies. Prior et al. (2008) and Hakkennes et al. (2008) describe this phenomenon in the context of their reviews focused on medical guideline implementation strategies. Prior et al. (2008) found that multifaceted and reminder-system strategies displayed the most positive results on guideline implementation, however their effectiveness ranged from 0% to 60% and 0% to 56%, respectively [22]. Hakkennes et al. (2008) found that reviewed studies showed only small to moderate effects with wide variations as well [23]. The authors ultimately call for the use of process measures in future trials of implementation strategies. Similarly, Powell et al. (2014) reviewed trials of implementation strategies to improve mental health EBI implementation. The authors concluded that 64% of the reviewed strategies resulted in statistically significant positive impacts, but similarly found wide variations in effectiveness among studies [24]. The authors note their inability to assess the fidelity of implementation strategies as a limitation and call for further research into fidelity measurement [24].

Implementation strategy fidelity measurement provides an opportunity to explore a moderation pathway. Intervention fidelity moderates the positive relationship between an EBI and its clinical outcomes, such that EBIs with higher fidelity tend to yield better clinical outcomes compared to EBIs with lower fidelity [2, 6]. It is plausible that implementation strategy fidelity similarly moderates the relationship between implementation strategies and implementation outcomes (and thereby clinical outcomes as well).

The quality and clarity of implementation research may additionally benefit from increased measurement and reporting of implementation strategy fidelity. Implementation strategy fidelity measurement requires a detailed account of implementation strategies themselves and their adaptations [20]. The quality of such descriptions has been criticized in the past and is discussed later in this article [25, 26]. Improving implementation strategy fidelity measurement may provide an additional benefit by nudging researchers toward improved implementation strategy specification.

Challenges in implementation strategy fidelity measurement

While implementation strategy fidelity holds both conceptual and methodological importance, it also introduces an additional, complex, and potentially costly variable to measure. In their description of intervention fidelity, Carroll et al. (2007) detail four main fidelity domains (details of content, coverage, frequency, and duration) along with four additional domains that affect the level of fidelity (comprehensiveness of component description, implementation strategies, quality of delivery, and participant responsiveness) [27]. While the domains of content, coverage, frequency, and duration may be straightforward to assess, domains focused on measuring behavior like quality of delivery, or on attitudes like participant responsiveness, may require more complex scale development like the creation of items, item-responses, or reliability/validity evaluation. Comprehensive fidelity measurement requires an assessment of each domain.

Fidelity domains vary regarding their level of effort and cost to measure, some often more challenging than others. Intervention fidelity measurement reviews commonly describe how “structural” fidelity domains like coverage, frequency, and duration are measured and reported more readily compared to elements that required the development or use of additional scales, like quality or participant responsiveness which often require more labor-intensive data collection (e.g., observation by trained fidelity raters) [21, 28–30].

Despite its associated costs and effort regarding measure development and subsequent inter-rater reliability testing, observation is often viewed as a gold standard data collection modality for domains like quality and participant responsiveness [31, 32]. Clinician self-report, or client report on clinician behavior, is often less costly but can also introduce positive-response bias [21]. In their work on fidelity measurement in behavioral research, Ledford et al. (2014) highlight how implementers regularly overestimated their own adherence to implementation procedures. To mitigate these challenges, recommendations have been made to utilize multiple data sources when measuring fidelity domains (e.g., pairing self-report with observation) [33, 34]. While an all-encompassing approach to fidelity measurement may appeal to researchers theoretically, the costs and stakeholder burden associated with intensive fidelity measurement may be prohibitive in the context of some implementation trials. The differing degrees between costs and effort to measure various fidelity domains may result in no or partial fidelity measurement, leading to results that may be challenging to interpret.

Similar to interventions, implementation strategies regularly undergo adaptations to fit different contexts and require a similar account of each adaptation [35, 36]. In fact, numerous authors have noted the importance of prospectively tracking implementation strategies carefully to document changes over time and report them in ways that are consistent with reporting recommendations for implementation strategies [18, 29, 37–40]. This is particularly important for strategies like facilitation, that are tailored to address site-specific needs. While tracking adaptations improves our understanding of implementation strategies’ inner-workings, adaptation necessitates change to the fidelity measurement of each adapted implementation strategy, potentially posing additional effort and costs. For example, Haley et al. (2021) describe the tracking of a facilitation strategy meant to increase the adoption of social determinants of health screening and referral activities. The authors utilize several implementation frameworks to specify implementation strategies, describe implementation barriers, and the modifications to strategies used to overcome them. They ultimately describe several adaptations including (1) the reduced frequency of peer support meetings, (2) additional information sharing between study clinics, and (3) additional items included in a data collection tool [40]. Adaptations of implementation strategy fidelity assessment could include (1) participant responsiveness to the new frequency of peer support meetings, (2) the receipt and additional content of the new information shared between study clinics, and (3) the receipt and content of the new questions added to the data collection tool.

The transition from implementation strategy fidelity measurement in research to its measurement in practice represents additional challenges as well. Bond et al. (2011) describe this challenge in-part through their case study measuring fidelity of Individual Placement and Support (IPS), an intervention that improves employment outcomes for individuals with mental health disorders. The authors describe the multiple purposes of fidelity measurement: (1) as a latent, unobservable variable assessing intervention receipt in the context of a trial, (2) as a quality assurance metric that healthcare agencies use to ensure they are “getting the intervention they paid for” in practice, and (3) as a tool for supervisors to track and build the treatment capacity of individual clinicians delivering the intervention [41]. Bond et al. describe how a fidelity tool developed to assess intervention receipt during a research phase, may require adaptation when used as a means of ensuring the quality of individual clinicians delivering the intervention in practice [41]. In addition to the knowledge, time, and effort required to adapt a fidelity tool from a research setting to a practice setting, the costs associated with fidelity measurement are also likely be transferred from a research team to the organization that takes-up and sustains that implementation strategy.

Moving implementation strategy fidelity measurement forward

Despite the challenges inherent in implementation strategy fidelity measurement and reporting, we believe that the pursuit of improving both are not only worthwhile, but necessary to enhance the quality of implementation research. The recommendations herein include pragmatic actions that implementation researchers can take immediately, calls for future research to further develop implementation strategy fidelity measurement, and structural changes at the funding and publication levels.

Table 1 depicts the current state of implementation strategy fidelity measurement challenges alongside our recommendations for improvements. To give readers a sense of the state of implementation strategy fidelity measurement compared to intervention fidelity measurement, these challenges and recommendations are shown side-by-side with Toomey et al.’s (2020) work on challenges of and recommendations to intervention fidelity measurement improvements [30]. Overall, the state of implementation strategy fidelity is not at the level of intervention fidelity, and more work is required to understand the specific barriers and facilitators to implementation strategy fidelity measurement. One similarity is shared between the two regarding challenges in defining and conceptualizing fidelity, which may serve as a good starting place for implementation strategy fidelity efforts.

Table 1

A comparison of challenges and recommendations for improving intervention fidelity (Toomey et al., 2020) and implementation strategy fidelity

Toomey et al. 2020 (intervention fidelity)		Implementation strategy fidelity
Overarching issue	Specific recommendations	Overarching issue	Specific recommendations
Lack of standardization regarding how fidelity is conceptualized and defined	Clarify how fidelity is defined and conceptualized	Lack of standardization regarding how fidelity is conceptualized and defined	Build consensus definition and conceptualization of implementation strategy fidelity
Limited focus beyond assessing of fidelity of delivery	Consider fidelity beyond intervention delivery 2b. Consider both enhancement and assessment strategies explicitly	Limited focus on assessing implementation strategy fidelity	Increase understanding of barriers and facilitators to implementation strategy fidelity assessment
Limited use of existing fidelity frameworks or guidance	Make use of existing frameworks		Increase focus on implementation strategy fidelity incrementally throughout the strategy’s evolution
Lack of focus on quality and comprehensiveness of fidelity assessment strategies	Consider the psychometric and implementation properties of mixed method fidelity assessment strategies		Utilize mixed methods approaches to fidelity assessment
			Consider the use of the Implementation Strategy Fidelity Checklist (throughout study timeline)
Lack of explicit focus on the balance between fidelity and adaptation	Consider the need for balance between fidelity and adaptation a-priori	Poor reporting on implementation strategy and mechanism specification	Make use of existing implementation strategy specification frameworks
Poor reporting of how intervention fidelity is addressed	Comprehensively report use of strategies to enhance and assess fidelity and results of fidelity assessments	Poor reporting on implementation strategy and mechanism adaptation	Make use of adaptation tracking techniques, assess fidelity to adapted implementation strategies
		Cost of implementation strategy fidelity measurement	Develop innovations to facilitate less costly implementation strategy fidelity measurement

While we note the challenge of quantitative fidelity measure development and assessment, mixed methods approaches may facilitate measurement more readily while possibly increasing the quality of fidelity measurement overall. Techniques like triangulation, sequential analysis, or ethnography can be used to describe fidelity domains in combination with quantitative results [42-44]. For example, Williams et al. used mixed methods to evaluate fidelity of a physical activity intervention delivered by primary care physicians. The authors measured the structural domains quantitatively, and participant responsiveness and quality of delivery qualitatively [45]. Williams et al. conclude that the use of mixed methods helped facilitate a more holistic understanding of fidelity than would have been provided by quantitative or qualitative measurement alone [45]. Mixed methods have also been used in the assessment of implementation strategy fidelity. When assessing fidelity to a practice facilitation implementation strategy meant to improve provider guideline adherence, Berry et al. measured the fidelity domains of content, frequency, duration, coverage, and quality quantitatively, but assessed participant responsiveness qualitatively [46]. The use of qualitative and mixed methods is common among implementation trials, and opportunities may exist to add questions that tap specific implementation strategy fidelity domains within already-planned qualitative activities [47].

Innovations in fidelity measurement may additionally facilitate less costly or labor-intensive data collection efforts. Beidas et al. describe their trial comparing different types of fidelity measurement to youth CBT, a therapist delivered EBI addressing a range of mental health outcomes. Their protocol outlines a four-group trial where various fidelity data collection modalities are compared to the gold standard of direct observation [48]. The results of this study, and others like it, can help determine the costs and cost-effectiveness of different modalities. It also might not always be necessary to rate an entire data set to accurately measure fidelity. Caperton et al. examined the amount of raw data required to measure fidelity to therapist delivered motivational interviewing (MI) sessions, an EBI to improve substance abuse outcomes. The authors found that rating just one-third of an MI session was sufficient to determine the fidelity of the entire session [49]. Due to their focus on inter-personal relationships, many implementation strategies (e.g., training, facilitation, coaching, audit-and-feedback) share similarities with the interventions described above. Implementation strategy fidelity measurement could, therefore, draw from fidelity measurement of behavioral interventions like MI or CBT in terms of how to conceptualize and measure fidelity of behavior implementation strategies [50-52].

While improvements in measurement are necessary, improvements in reporting are equally important. Slaughter et al. call for researchers to apply the Implementation Strategy Fidelity Checklist, a tool that assesses the quality of fidelity reporting in implementation trials, to their trial manuscripts prior to publication so that under-reported fidelity domains might be highlighted and improved [20]. While researchers may find that some fidelity domains have not been fully measured during their trials, the application of the checklist may still prompt researchers to improve reporting on what fidelity data they might have available.

We further suggest that the Implementation Strategy Fidelity Checklist could be applied at the study design phase when measures are first selected or developed. A preassessment of a study’s ability to report on implementation strategy fidelity may highlight measurement gaps for certain fidelity domains at the outset. We recognize that while comprehensive fidelity measurement exists as a theoretical ideal, it is often unfeasible in the context of many study budgets or timelines and may not always be necessary. Bond et al. reviewed the fidelity of several interventions to improve mental health outcomes. They found that team-based interventions, whose fidelity were determined mostly by organizational factors like staffing or receipt of services, achieved higher levels of fidelity compared to interventions that focused more on individual clinician behavior (like adherence to a counseling intervention) [53]. It could be that implementation strategies that focus more on organizational structural change require less attention to fidelity domains like quality or participant responsiveness compared to strategies that rely more on interpersonal interactions. Hankonen’s work on participant reception and enactment, or the extent to which participants connect with and enact the knowledge/skills learned through a health intervention, help highlight the importance of comprehensive fidelity measurement when EBI components focus on changing individuals’ behavior. The author notes that when the target of an EBI encompasses individual behavior change, the need to measure participants’ reception and enactment of intervention content becomes critical [54]. Fidelity indicators like “number of sessions delivered” or “adequate content delivered” may tell us that an actor delivered structural intervention components as intended, but such indicators may not tell us about how well the components were received by their targets, nor how able the targets were to utilize what they learned [54]. Relating back to implementation strategies and their fidelity, this point highlights again the need to adequately describe implementation strategies, their action targets, and their mechanisms most responsible for change in implementation outcomes [19]. A clear understanding of the inner workings of a strategy help to define the fidelity domains most essential to their success.

Recent reviews of intervention fidelity call for the development and utilization of higher quality fidelity measures [55, 56]. We share these sentiments as they relate to implementation strategy fidelity. However, while high quality fidelity tools may serve as an end goal, we recognize that implementation strategy fidelity tools may be best suited for development in stages given the challenges outlined above. For example, researchers piloting a new implementation strategy may only have the ability to measure if their strategy was delivered and to what extent. The burden of collecting, analyzing, and reporting this information is likely to be low given such that data are often collected during the execution of the implementation strategy itself [57]. The later stages of an implementation strategy’s evolution may be best suited for more robust fidelity measurement, including psychometric evaluation. Stockdale et al. suggest that implementation strategy fidelity measures might be validated in the context of Hybrid 3 trials given their tendencies to utilize larger sample sizes [58]. Incremental improvements over time likely pose the most realistic path toward improved measurement. The addition of an implementation strategy fidelity measurement step to Proctor et al. (2009)’s Conceptual Model of Implementation Research (Fig. 1) may help facilitate further measurement [12].

CONCLUSION

In this position article, we describe the benefits and challenges inherent in implementation strategy fidelity measurement. The most recent review of implementation strategy fidelity describes measurement as under-developed, under-reported, and decreasing in quality over-time; recent guidance on implementation trials also calls for the inclusion of implementation strategy process evaluation (inclusive of fidelity measurement) [20, 59]. Explanations regarding the state of implementation strategy fidelity measurement are not yet fully understood, and more research into the specific barriers and facilitators that influence implementation researchers are necessary. In the conclusion of their review on the history of intervention fidelity, Bond et al. ultimately determine that the benefits of fidelity measurement outweigh the costs. The authors describe research and clinical care that lack fidelity measurement as a prescientific “black box” era where an intervention, its components, and their impact on clinical outcomes are not made explicitly clear [28]. We share this sentiment as it relates to implementation strategy fidelity measurement and describe the need to better understand the inner workings of causal chains under examination in implementation trials. Clearer specifications of implementation strategies and their mechanisms most responsible for change, the degrees of fidelity they achieve, and the extent to which implementation strategy fidelity acts as a main effects moderator will help to move implementation research moving forward.