Translating evidence in complex systems: a comparative review of implementation and improvement frameworks.

PURPOSE: An increasing number of implementation and improvement frameworks seek to describe and explain how change is made in healthcare. This paper aims to explore how existing frameworks conceptualize the influence of complexity in translating evidence into practice in healthcare. DATA SOURCES: A database was interrogated using a search strategy to identify publications that present frameworks and models for implementation and improvement. STUDY SELECTION: Ten popular implementation and improvement frameworks were purposively selected. DATA EXTRACTION: Comparative analysis was conducted using an analytical framework derived from SHIFT-Evidence, a framework that conceptualizes complexity in implementation and improvement initiatives.
RESULTS: Collectively the frameworks accounted for key concepts of translating evidence in complex systems: understanding the uniqueness of each setting; the interdependency of practices/processes and the need to respond to unpredictable events and emergent learning. The analysis highlighted heterogeneity of the frameworks in their focus on different aspects of complexity. Differences include the extent to which problems and solutions are investigated or assumed; whether endpoints are defined as the uptake of interventions or achievement of goals; and emphasis placed on fixed-term interventions versus continual improvement. None of the individual frameworks reviewed incorporated all the implications of complexity, as described by SHIFT-Evidence.
CONCLUSION: This research identifies the differences in how implementation and improvement frameworks consider complexity, suggesting that SHIFT-Evidence offers a more comprehensive overview compared with the other frameworks. The similarity of concepts across the frameworks suggests growing consensus in the literature, with SHIFT-Evidence providing a conceptual bridge between the implementation and improvement fields.

Introduction

There is a growing need for guidance to support implementation and improvement initiatives and programmes which aim to bridge the gap between the production of research and its use in delivering routine evidence-based care [1-5]. This need has prompted an increase in the number of theories, models and frameworks that explicitly seek to support this endeavour [6].

Our experience of designing, conducting and evaluating a programme to translate evidence into practice (the National Institute of Health Research (NIHR) Collaboraton for Leadership in Applied Health Research and Care (CLAHRC) Northwest London) [7] provided us with first-hand practical knowledge of the challenges of intervening in healthcare settings. Through this work, we explored the existing literature and identified a gap in how frameworks deal with practical reality of working in complex systems. This has led to the development of the conceptual framework Successful Healthcare Improvements From Translating Evidence in complex systems (SHIFT-Evidence) [8, 9]. The framework has been developed from extensive research on initiatives with a common goal of translating existing evidence into practice to improve patient care, with empirical examples covering diverse topics such as chronic obstructive pulmonary disease in acute care [10], sickle cell disease in primary care [11], financial incentives for GP alcohol-dependency screening [12] and pulmonary rehabilitation in community settings [13].

This novel framework draws on complexity science to reconceptualize the challenge of evidence-translation and improvement in healthcare. Complex systems are shaped through the agency of system actors and the interconnectedness and interdependency of system components, which can lead to unpredictability and surprise as systems evolve dynamically overtime [14-16]. Complex systems can, therefore, defy deliberate attempts to intervene and achieve system change. SHIFT-Evidence provides three strategic principles and 12 ‘simple rules’ (Table 1) that aim to support practitioners and academics to navigate and negotiate the process of change in complex systems [8, 9].

Table 1

Summary of strategic principles of the Successful Healthcare Improvements From Translating Evidence in complex systems framework (SHIFT-Evidence) and associated ‘simple rules’

Principle	Rationale	Simple rules for complex systems
Act scientifically and pragmatically	Knowledge of existing evidence needs to be combined with knowledge of the unique initial conditions of a system. Interventions need to adapt as the complex system responds and learning emerges about unpredictable effects	Understand problems and opportunities Identify, test and iteratively develop potential solutions Assess whether improvement is achieved, capture and share learning Invest in continual improvement
Embrace complexity	Evidence-based interventions only work if related practices and processes of care within the complex system are functional. Evidence-translation efforts need to identify and address existing problems with usual care, recognizing this typically includes a range of interdependent parts of the system. This emphasizes the need to investigate and understand the uniqueness of each local system and respond to complexity from the micro- to macro-system	Understand processes and practices of care Understand the types and sources of variation Identify systemic issues Seek political, strategic and financial alignment
Engage and empower	Evidence translation and system navigation require commitment and insights from staff and patients with experience of the local system. Changes need to align with their motivations and concerns. The principle reflects factors that influence engagement at an individual and team level through to supporting infrastructure and organizational level	Actively engage those responsible for and affected by change Facilitate dialogue Foster a culture of willingness to learn and freedom to act Provide headroom, resources, training and support

Comparing SHIFT-Evidence with existing frameworks helps situate it within the growing library of published frameworks and models, and clarify how it builds on existing concepts. This paper presents a comparative analysis of existing ‘popular’ frameworks to consider commonalities and differences between these frameworks and SHIFT-Evidence, and comment on the degree to which complexity is covered by the different frameworks and the insight this provides.

Methods

Literature search

The comparative analysis set out to compare and contrast the conceptualization of complexity across a number of popular implementation and improvement frameworks. As such a search strategy was developed to identify an illustrative range of implementation and improvement frameworks and models. It is not intended to provide a comprehensive review of all available frameworks. The structured search methodology used terms including ‘knowledge translation’ or ‘implementation’ or ‘quality’ or ‘improvement’ and ‘framework’ or ‘model’ using the Web of Science database. Inclusion criteria required a framework or model with general relevance to healthcare and published in English in a peer-review journal that was highly cited, as an indication of the level of engagement or ‘popularity’ of the framework. Papers not relevant to healthcare, limited to a specific setting or clinical problem were excluded. No date limitations were set. Where multiple papers by the same research group were identified, the paper that best met the inclusion criteria was selected. Frameworks were purposively selected to reflect diverse perspectives on evidence translation, implementation and improvement in healthcare and avoid duplication of similar frameworks.

Data extraction and analysis

A data extraction table was developed from the SHIFT-Evidence framework, comprising 12 separate fields aligned to the principles or ‘simple rules for complex systems’ (Table 1). Two reviewers extracted data from the original manuscripts for each published framework, one with no prior explicit involvement in developing SHIFT-Evidence (J.R. and S.G.). Framework content was assessed against each of the SHIFT-Evidence framework simple rules and recorded as extensively covered (††), partially covered (†), minimally or not covered (-). Assessment considered both the number of related concepts and the breadth of issues they addressed. Disagreements were discussed and resolved, producing a final assessment and rationale. Analysis was also undertaken to classify frameworks using a published taxonomy [6].

Results

Implementation and improvement frameworks

The literature search identified 10 popular highly cited implementation and improvement frameworks that support evidence-translation, each with more than 100 citations (Table 2).

Table 2

Citation list of frameworks included in analysis

	Analytical framework
	Reed JE, Howe C, Doyle C, et al. Simple rules for evidence translation in complex systems: a qualitative study. BMC Medicine 2018 (in press).
	Implementation and improvement frameworks identified by search
1	Batalden PB, Stoltz PK. A framework for the continual improvement of health care: building and applying professional and improvement knowledge to test changes in daily work. Jt Comm J Qual Improv 1993;19:424–7.
2	Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research.Implement Sci 2012;7:37.
3	Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.Implement Sci 2009;4:50.
4	Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof 2006;26:13–24.
5	Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82:581–629.
6	Kilo CM. A framework for collaborative improvement: lessons from the Institute for Healthcare Improvement’s Breakthrough Series. Qual Manag Health Care 1998;6:1–4.
7	May C, Finch T, Mair F, et al. Understanding the implementation of complex interventions in health care: the normalization process model. BMC Health Ser Res 2007;7:148.
8	Rycroft-Malone J. The PARIHS framework—a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual 2004;19:297–304.
9	Stetler CB, Mittman BS, Francis J. Overview of the VA Quality Enhancement Research Initiative (QUERI) and QUERI theme articles: QUERI series. Implement Sci 2008;3:8.
10	Wandersman A, Duffy J, Flaspohler P, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol 2008;41:171–81.

Initially, frameworks were classified according to their main purpose, which was either explicitly outlined in the paper or a judgement by the authors (Table 3). Implementation frameworks were defined as those that primarily aimed to support the translation of evidence into practice and improvement frameworks those that primarily aimed to support improving healthcare quality. Most frameworks were either classified as implementation [17-23] or improvement [24, 25] frameworks, with the exception of Stetler et al., which was identified as both [26]. In addition, SHIFT-Evidence, the analytical framework is included, which was also classified as an implementation and improvement framework [9].

Table 3

Categorization of frameworks: summary of classification of 10 implementation and improvement frameworks identified for comparative analysis [plus SHIFT-Evidence—Reed et al. [9]]

	Batalden and Stoltz [24]	Cane et al. [19]	Damschroder et al. [20]	Graham et al. [21]	Greenhalgh et al. [22]	Kilo [25]	May et al. [23]	Rycroft-Malone [17]	Stetler et al. [26]	Wandersman et al. [18]	Reed et al. [9]
Purpose
Implementation framework		X	X	X	X		X	X	X	X	X
Improvement framework	X					X			X		X
Categorization
Describing and/or guiding the process of translating research into practice (or guiding improvement processes for improvement)	X			X		X			X	X	X
Understanding and/or explaining what influences implementation outcomes (or improvement)		X	X		X		X	X		X	X
Evaluating implementation (or improvement)		X	X	X	X		X	X			X

Further categorization of the frameworks using the taxonomy proposed by Nilsen [6], demonstrates that the frameworks included those that principally aim to guide the process of translating evidence into practice (or guiding improvement processes for improvement) [21, 24–26], those that principally aim to explain what factors influence implementation (or improvement) i.e. barriers and enablers [17, 19, 20, 22, 23], and those that combine these functions [18]. Although no framework was deemed an evaluation framework explicitly, most were perceived to fulfil this role as a secondary function [17, 19–21, 23, 27].

Whilst this taxonomy was helpful in understanding the main function of each framework, there did appear to be some overlap between these functions across frameworks, for example, Wandersman et al. propose to not only offer a guide in translating evidence but also an understanding of potential challenges and barriers [18].

Comparative analysis

The comparative analysis across multiple frameworks allowed exploration of the principles proposed within the SHIFT-Evidence framework (Tables 4, 6 and 7). The analysis is presented across each of the high-level strategic principles: act scientifically and pragmatically, embrace complexity and engage and empower. Each section contains a summary of the principle followed by detailed analysis of the presence or absence of each of the ‘simple rules’ across the frameworks.

Table 4

Comparative analysis of SHIFT-Evidence principle act scientifically and pragmatically with 10 implementation and improvement frameworks

Act scientifically and pragmatically	Batalden and Stoltz [24]	Cane et al. [19]	Damschroder et al. [20]	Graham et al. [21]	Greenhalgh et al. [22]	Kilo [25]	May et al. [23]	Rycroft-Malone [17]	Stetler et al. [26]	Wandersman et al. [18]
Understand problems and opportunities	††	–	–	††	–	†	–	–	††	–
Identify, test and iteratively develop potential solutions	†	†	††	††	††	††	†	†	†	†
Assess whether improvement is achieved, capture and share learning	†	–	†	†	†	††	†	†	††	†
Invest in continual improvement	††	–	–	†	–	†	–	–	–	–

Key: extent to which simple rule covered †† extensively covered; † partially covered; – minimal or not covered. Full details of constructs and rationale for decisions can be found in Supplementary materials 1 and 2, respectively.

Table 6

Comparative analysis of SHIFT-Evidence principle embrace complexity with 10 implementation and improvement frameworks

Embrace complexity	Batalden and Stoltz [24]	Cane et al. [19]	Damschroder et al. [20]	Graham et al. [21]	Greenhalgh et al. [22]	Kilo [25]	May et al. [23]	Rycroft-Malone [17]	Stetler et al. [26]	Wandersman et al. [18]
Understand processes and practices of care	††	†	†	–	†	†	††	–	††	–
Understand the types and sources of variation	††	–	–	–	–	††	†	–	††	–
Identify systemic issues	††	–	–	–	–	†	–	–	†	†
Seek political, strategic and financial alignment	†	†	††	–	††	†	††	†	†	†

Key: extent to which simple rule covered †† extensively covered; † partially covered; – minimal or not covered. Full details of constructs and rationale for decisions can be found in Table 5 and Supplementary material 2, respectively.

Table 7

Comparative analysis of SHIFT-Evidence principle engage and empower with 10 implementation and improvement frameworks

Engage and empower	Batalden and Stoltz [24]	Cane et al. [19]	Damschroder et al. [20]	Graham et al. [21]	Greenhalgh et al. [22]	Kilo [25]	May et al. [23]	Rycroft-Malone [17]	Stetler et al. [26]	Wandersman et al. [18]
Actively engage those responsible for and affected by change	††	††	††	††	††	††	†	††	†	††
Facilitate dialogue	††	†	†	–	†	–	†	††	–	–
Foster a culture of willingness to learn and freedom to act	††	†	††	–	††	†	–	††	–	†
Provide headroom, resources, training and support	†	†	††	–	††	†	†	††	†	††

Key: extent to which simple rule covered †† extensively covered; † partially covered; – minimal or not covered. Full details of constructs and rationale for decisions can be found in Supplementary materials 1 and 2, respectively.

Act scientifically and pragmatically

The first strategic principle of SHIFT-evidence proposes that it is necessary to act scientifically and pragmatically within a complex system to understand local problems and opportunities prior to identifying and adapting potential solutions informed by the evidence base. An assessment of whether local improvements are achieved can inform learning, which can be shared. To sustain improvements, it is necessary to invest in continual improvement, as the system does not remain static and continues to evolve.

All of the concepts represented by the strategic principle act scientifically and pragmatically were identified in the frameworks reviewed, as elaborated in more detail in Supplementary materials 1 and 2. However, no individual framework reflected all concepts, and interesting patterns were noticed in the commonality of different concepts (Table 4).

Four frameworks explicitly acknowledged the need to understand local problems and opportunities, which recognizes that just because an intervention exists and has been identified as a local solution does not mean that the local problem has been truly understood. The other frameworks have a tendency to ‘jump’ to a solution to a perceived problem before taking the time to explore the problem in depth. The principle values the importance of local context and the need to contextualize and adapt potential solutions, as well as engage with system actors – all of which are necessary to successfully intervene in complex systems. While all frameworks supported the need to identify and implement interventions/solutions, not all recognized the iterative nature of the testing that may be required. For example, while Wandersman et al. acknowledge the need for local adaptation, this is proposed upfront in the design of the intervention(s) rather than iterative development during implementation phase [18]. This suggests limited recognition of the unpredictable nature of complex systems nor does the framework promote use of emergent learning. Most of the frameworks promote the monitoring of implementation and subsequent assessment of improvements, often through defined indicators or measures, allowing learning to be captured and shared. However, only three frameworks included investing in continual improvement, exemplified by Batalden and Stoltz [24], which explicitly highlighted the main purpose of the framework was to promote the embedding of continual improvement within healthcare. The need for continual improvement relates to the dynamic nature of complex systems and recognition of the necessity to respond to changes over time rather than rely on one-off solutions.

Embrace complexity

The second strategic principle of SHIFT-evidence proposes that it is necessary to embrace complexity within healthcare systems and resist over-simplification. The processes and practices of care and their inherent variation need to be identified and the underlying causes understood in order to enact effective change. This encourages interventions to be seen as part of the complex system they are dependent on, rather than in isolation. Through reviewing practices and processes, systemic issues may be identified requiring an assessment of political, strategic and financial factors, which need to be aligned to the specific improvements to ensure the successful translation of evidence.

All of the concepts represented by the strategic principle embrace complexity were identified in the frameworks reviewed, as elaborated in more detail in Table 5 and Supplementary material 2. The most common concept is that of political, strategic, and financial alignment, whereas identifying systemic issues was rarely covered by the frameworks (Table 6).

Table 5

Academic, theoretical and methodological concepts mapped against SHIFT-Evidence principles for scoring decisions in comparative analysis of frameworks: embrace complexity (see Supplementary material 1 for equivalent tables for act scientifically and pragmatically and engagement and empower)

Framework	Understand processes and practices of care	Understand the types and sources of variation	Identify systemic issues	Seek political, strategic and financial alignment
Batalden and Stoltz [24]	†† Knowledge of system; Process and system tools	†† Knowledge of variation; Statistical thinking (Common cause and special cause variation); Tampering (reacting inappropriately to causes of variation)	†† Knowledge of system	† Mission, vision and quality definition; Shared sense of purpose; Understanding ‘customer’ and social/community needs
Cane et al. [19]	† Procedural knowledge; Professional roles	–	–	† Environmental stressors; Organizational commitment; Reinforcement (rewards, incentives, punishment, consequences, sanctions); Salient events/critical incidents
Damschroder et al. [20]	† Compatibility (fit with processes)	–	–	†† Cost; External Policies and incentives; Organizational incentives and rewards; Outer context; Relative priority; Structural, political and social context; Leadership engagement
Graham et al. [21]	–	–	–	–
Greenhalgh et al. [22]	† System readiness for innovation (Innovation-System Fit)	–	–	†† Outer context (socio-political climate, incentives and mandates, inter-organizational norm-setting and networks, environmental stability); System antecedents for innovation (receptive context for change)
Kilo [25]	† Outcomes are the results of processes; understand system through first-hand experience	†† Variation in care; statistical process control	† Topic selection (identifying area for improvement); High leverage change ideas	† System leader engagement
May et al. [23]	†† Contextual integration (realization); Interactional Workability (congruence, disposal) Relational integration (accountability); Skill-set workability (allocation)	† Variation in outcome of implementation process	–	†† Skill-set workability (performance); Contextual integration (execution)
Rycroft-Malone [17]	–	–	–	† Context (Culture, strategic fit)
Stetler et al. [26]	†† Measure and diagnose quality and performance gaps	†† Identify variations from evidence-based practices (‘quality/performance gaps’)	† Select conditions per patient populations associated with high risk of disease and/or disability and/or burden of illness	† Regional and national roll out (stakeholder engagement)
Wandersman et al. [18]	–	–	† General capacity-building (e.g. infrastructure stabilization)	† Delivery system (organization factors)

Some of the frameworks, for example May et al., recognize the need to focus on the practices and processes of care, and how interventions align to the local context, but do not include the potential need to address wider organizational or systems issues that could arise due to system interconnectivity [23]. Furthermore, practices and processes of care were often considered as variables or explanatory factors rather than areas for local inquiry and action. Despite this, a number of frameworks valued unpacking local systems to understand current practices and processes of care, in particular Batalden and Stoltz’s improvement framework, demonstrate the key role and importance of understanding variation and identifying systemic issues [24]. This alludes to the ability to respond to problems and navigate complexity through modifications in the design and implementation process, which may pose particular challenges where the intervention and/or outcomes are fixed. The need for strategic, financial and political alignment both within and outside the organization represents the inherent links/influences across parts of the system that need to be aligned to support improvement efforts. This was recognized across nearly all frameworks to some extent, although was usually conceptualized as static factors that comprise ‘context’ rather than dynamic positions that can be deliberately targeted and influenced through intervening within systems.

Engage and empower

The third strategic principle of SHIFT-Evidence proposes that it is necessary to actively engage those responsible for and affected by the proposed changes. This principle recognizes that the motivation, knowledge, agency and authority necessary for translating evidence in complex systems resides within local organizations, their staff and patients.

All of the concepts represented by the strategic principle engage and empower were identified in the frameworks reviewed, as elaborated in more detail in Supplementary materials 1 and 2. Active engagement was the most commonly included out of all concepts, and the need to provide support, resources and training was also widely recognized. The need for facilitated dialogue received less recognition, although in frameworks that included this concept it was perceived as critical to successful active engagement (Table 7).

All frameworks recognize the value of engaging system actors, including those that effect and are affected by change. However, the level of active engagement varied. Some frameworks promoted collaboration between different professional groups and, in some cases patients, and others suggested more passive engagement or emphasized the role of external change agents as motivators for change. Frameworks that promoted creating partnerships with system actors recognized their role as fundamental in understanding the local system, aligning with local motives, and also in building capacity to effect change from within the system. The principle facilitate dialogue was poorly attended to with four frameworks not discussing this at all, while others, like Rycroft-Malone [17], promoted the value of facilitation to create a dialogue for change. They recognize the need for explicit mechanisms for facilitating discussion to ensure system actors are engaged and efforts are made to understand their perspectives of the system and their drivers and incentives. The simple rule willingness to learn and freedom to act recognizes that the autonomy and agency of individuals are influenced by the wider organizational and system context, which can create a permissive culture where people are willing to accept that current practice might be sub-optimal, and are able to take action and learn from improvement. However, few frameworks presented key concepts related to this. Frameworks varied in the extent to which headroom, training and resource were explicitly recognized. Of the frameworks that did consider it, many considered aspects such as resource as an explanatory factor rather an area to actively influence. In addition, many frameworks did not consider the headroom, training and support required for people within the system (e.g. healthcare professionals and patients) to empower them with the skills and competencies needed to make sense of the systems they work in and understand how to effectively intervene.

Table 8 provides an overview of the degree to which the main SHIFT-Evidence principles (act scientifically and pragmatically, embrace complexity and engage and empower) are covered within each of the 10 implementation and improvement frameworks.

Table 8

Comparative analysis of SHIFT-Evidence principles with 10 implementation and improvement frameworks

	Batalden and Stoltz [24]	Cane (2011)	Damschroder et al. [20]	Graham et al.[21]	Greenhalgh et al. [22]	Kilo [25]	May et al. [23]	Rycroft-Malone [17]	Stetler et al. [26]	Wandersman et al. [18]
Act scientifically and pragmatically
Embrace complexity
Engage and empower
All principles

Key: extent to which strategic principle covered; - complete coverage; - extensive coverage; - moderate coverage; - slight coverage, - minimal or no coverage. Full details of constructs and rationale for decisions can be found in Supplementary materials 1 and 2, respectively.

Discussion

The comparative analysis demonstrates the differences in the degree to which implementation and improvement frameworks consider complexity, suggesting that SHIFT-Evidence offers a more comprehensive overview compared with the other frameworks considered. The analysis also shows the differences between each of the 10 frameworks reviewed, demonstrating their heterogeneity, with each offering a particular area of focus and strengths. This analysis can guide considered selection of which framework(s) might be the most appropriate for use in different situations.

Nonetheless, elements of all the three strategic principles of SHIFT-Evidence: act scientifically and pragmatically, embrace complexity and engage and empower were found across the frameworks and within the broader academic literature, indicating some level of consensus about how evidence translation and improvement should be approached [28, 29].

Of the frameworks we reviewed, Batalden and Stoltz’s framework for continual improvement in healthcare most comprehensively covered the SHIFT-Evidence principles and ‘simple rules’ [24]. This framework was based on a Deming’s ‘System of Profound Knowledge’ (1986, 1993), which was originally developed in industry settings [30, 31]. While Deming and Batalden’s frameworks have since been widely promoted in healthcare, there is a lack of empirical or theoretical grounding in healthcare settings [32]. SHIFT-Evidence reinforces the relevance of this framework to healthcare. However, healthcare systems tend to involve more complex processes and problem-solving than industry and are managed by more highly skilled and autonomous professionals with more complex political and policy drivers (addressed in part in Batalden’s framework for continual improvement in healthcare but not explicit in Deming’s work) [33, 34]. SHIFT-Evidence also extends Batalden and Stoltz’s framework, and Deming’s work, by emphasizing the need to utilize existing evidence and to capture and produce new knowledge [24]. Therefore, a novel contribution of the SHIFT-Evidence framework is that it provides an explicit connection between the fields of industrial quality improvement and evidence translation in the healthcare context.

While Batalden and Stoltz’s framework has broad coverage of SHIFT-Evidence principles many of the other frameworks have greater depth and provide valuable insights into aspects of the ‘simple rules’. Collectively, the reviewed frameworks provided a wealth of academic, theoretical and methodological concepts that can contribute to a more in-depth understanding. In mapping the concepts and constructs from each framework to the 12 ‘simple rules’ (Table 5 and Supplementary material 1), this paper demonstrates a conceptual connection between SHIFT-Evidence and existing frameworks. The analysis also highlights which of these popular frameworks provides greater insights into specific concepts.

Implementation versus improvement?

In analysing the frameworks, we identified common patterns, reflecting the extent to which each framework has a primary focus on implementation or improvement. The comparative analysis has highlighted that improvement frameworks, such as Batalden and Stoltz [24] and Kilo [25] are more clearly aligned to many of the challenges associated with complexity in translating evidence in practice. This is especially true in direct comparison with some of the implementation frameworks, such as Rycroft-Malone [17] and Cane et al. [19], which, while acknowledging some levels of complexity such as agency and systems perspective, have less focus on understanding problems and local contexts. The extent to which complexity was considered in the frameworks had a major influence on how the challenge of evidence translation was represented, including the extent to which; problems and solutions were investigated or assumed; endpoints were defined as the uptake of a particular intervention or the achievement of an improvement goal; and emphasis was placed on fixed-term interventions versus continual improvement.

The very nature of the implementation paradigm is about the uptake of known solutions into routine practice, with a focus on evidence-based interventions. As such, implementation frameworks tend to start with a ‘solution’ in mind, and demonstrate little interest in understanding local problems. Improvement frameworks placed greater emphasis on understanding the local problem, recognizing that the problem and solution are either unknown or not fully understood at the start of an initiative. System interconnectedness means that the successful implementation of an evidence-based intervention may be dependent on other practices and processes working effectively, meaning that interventions need to be adapted or additional interventions deployed [15, 35]. This suggests that failing to sufficiently understand local problems and opportunities could undermine the effective deployment of interventions and their ability to achieve the desired outcome. While improvement frameworks aim to provide guidance on how to understand local problems and opportunities prior to the selection of solutions, there tends to be a less explicit connection to drawing on the existing evidence base to design and select interventions. This predicament highlights the value of combining knowledge (and subsequent practice) from both implementation and improvement frameworks.

Implementation frameworks tend to focus on finite study periods during which an intervention is introduced. The focus on defined study periods reflects a lack of conceptualization of the temporal nature of change and the need to continually adapt and respond to emerging opportunities and threats over time as a system dynamically evolves [14]. In comparison, improvement frameworks tend to highlight the need for continual investment in improvement as an ongoing process to achieve and maintain improvement goals. As a consequence, improvement frameworks also emphasize the need to build capability and capacity of people within the system to continue learning and improving overtime, rather than seeing change as something that is done by people external to the system.

Typically, the end-point of implementation is defined as the uptake of a particular intervention (which may include evidence-based practices or other interventions or innovations) [17, 20, 22, 23]. Conversely, the end-point in improvement frameworks is typically the achievement of a particular improvement goal, often relating to care quality or patient outcomes [24-26]. While this difference may appear semantic, the choice of academic lens alters what is of interest for investigation and explanation, with implementation focused on the intervention (and objectively assessing what outcome it has achieved), and improvement focused on the outcome (and learning how best to intervene to achieve it). Frameworks that usually define their end-point as the implementation of a specific intervention or evidence-based practice, tend not to recognize the need to make wider system improvements. Conversely, frameworks that define improvement as their end-point recognize the need to ‘work with’ the system to maximize achievement. In the SHIFT-evidence framework we suggest that terminology shifts from the use of the noun ‘intervention’ to the verb ‘intervening’, in recognition of the fact that the concept of ‘intervening to achieve an improvement’ better reflects the iterative and negotiated process required to understand and influence complex systems [8, 9].

SHIFT-Evidence provides a clear explanation as to why the successful implementation of evidence-based interventions is dependent on wider system improvements: to ensure that ‘usual care’ is functioning well to support evidence-based care provision, and to address wider concerns of importance to patients and staff. SHIFT-Evidence, therefore, adds to the literature by providing a conceptual bridge between the fields of evidence translation (including implementation, knowledge mobilization and other related studies) and improvement [36].

Analysis of the frameworks highlighted the strong empirical or theoretical grounding of implementation frameworks, which focus on understanding or explaining implementation outcomes, specifically through identifying facilitators and barriers (determinant frameworks), or explaining aspects of implementation using existing theories (theoretical frameworks) [6]. However, despite this robust underpinning the majority of implementation frameworks are intended to guide research and evaluation rather than to provide practical guidance to practitioners or policy-makers. Conversely, the improvement frameworks, although lacking in an empirical or theoretical grounding, tend to provide greater practical guidance. SHIFT-Evidence, therefore, also has potential to add to the literature by providing a common framework to guide both research and practice.

Strengths, limitations and future work

This work has aimed to compare the recently published SHIFT-Evidence framework with a sample of existing implementation and improvement frameworks. In focusing this comparison on the constructs and concepts included in each framework, analysis is intended to go beyond the superficial or descriptive elements and unpack the underpinning theories and rationale within these frameworks. We think this methodological approach is a valuable way of conducting a comparative framework analysis and also establishes a level of transparency in doing so (Supplementary materials 1 and 2) to support critical reflection and genuine understanding of the field(s) of implementation and improvement. However, we recognize there are limitations to the approach. Analysis was not intended to be a systematic review of implementation and improvement frameworks, nor intended to go beyond the fields of implementation and improvement. The frameworks were purposively selected to allow an illustrative comparative analysis of ‘popular’ frameworks with the intention to be exploratory rather than comprehensive. We recognize the bias introduced by using SHIFT-Evidence as the comparative lens for assessing the frameworks; while this was a deliberate choice to both place the new framework in context and to explore the breadth of concepts that relate to a complex system worldview, we recognize that if viewed through another lens, different implications could be drawn.

Further work is required to continue the comparative review of other frameworks, across the emerging literature within the fields of implementation and improvement sciences, and in consideration of how principles transfer between industrial settings and healthcare. There is also scope to increase the review and inclusion of learning from other related disciplines including, but not limited to, psychology, organizational studies, education, human ergonomics and complexity science. We believe this will help advance interdisciplinary working between academic fields, and increase the transparency and accessibility of information to practitioners, patients and policy-makers involved with implementation and improvement initiatives.

Conclusion

The comparative analysis demonstrates the heterogeneity of the 10 implementation and improvement frameworks in their focus on different aspects of complexity, with some providing a greater depth of insight into individual concepts. Analysis highlights that SHIFT-Evidence offers a more comprehensive overview compared with these frameworks. The identification of similar concepts across the other frameworks suggests the principles in SHIFT-Evidence represent a growing consensus in the literature about how implementation and improvement in healthcare should be approached to increase chances of success.

Other than SHIFT-Evidence, Batalden and Stoltz’s framework for continual improvement in healthcare covered the greatest breadth of concepts related to intervening in complex systems. This suggests that SHIFT-Evidence provides empirical grounding and validation of the relevance of the industrial approach developed by Deming (and interpreted in Batalden’s framework) to healthcare contexts.

By combining features that are recognized as important from implementation and improvement perspectives, SHIFT-Evidence provides a conceptual bridge between the fields of implementation and improvement science. The framework, therefore, has potential to facilitate collaboration between those involved in diverse academic fields. In addition, the SHIFT-Evidence framework has the potential to provide a common platform for practitioners, patients, policy-makers and researchers involved with change efforts.

Click here for additional data file.

Click here for additional data file.