How to build up the actionable knowledge base: the role of 'best fit' framework synthesis for studies of improvement in healthcare.

Increasing recognition of the role and value of theory in improvement work in healthcare offers the prospect of capitalising upon, and consolidating, actionable lessons from synthesis of improvement projects and initiatives. We propose that informed use of theory can (i) provide a mechanism by which to collect and organise data from a body of improvement work, (ii) offer a framework for analysis and identification of lessons learnt and (iii) facilitate an evaluation of the feasibility, effectiveness and acceptability of improvement programmes. Improvement practitioners can benefit from using an underpinning external structure as a lens by which to examine the specific achievements of their own projects alongside comparable initiatives led by others. We demonstrate the utility of a method known as 'best fit framework synthesis' (BFFS) in offering a ubiquitous and versatile means by which to collect, analyse and evaluate improvement work in healthcare. First reported in 2011, BFFS represents a pragmatic, flexible approach to integrating theory with findings from practice. A deductive phase, where a review team seeks to accommodate a substantial part of the data, is followed by an inductive phase, in which the team explores data not accommodated by the framework. We explore the potential for BFFS within improvement work by drawing upon the evidence synthesis methodology literature and practical examples of improvement work reported in BMJ Quality and Safety (2011-2015). We suggest four variants of BFFS that may have particular value in synthesising a body of improvement work. We conclude that BFFS, alongside other approaches that seek to optimise the contribution of theory to improvement work, represents one important enabling mechanism by which to establish the rigour and scientific credentials of the emerging discipline of 'improvement science'. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Background

In the quest to build an actionable knowledge base,1 improvement practitioners need to be able to move away from the specific characteristics of their own projects towards an understanding of generalisable factors that influence the implementation and impact of improvement interventions. Insights from theory may help complete such a transition, particularly in helping practitioners understand what works for whom under what circumstances. One method for achieving such insights is the ‘best fit’ framework synthesis (BFFS).2 3

BFFS is an innovative methodology, having received 45 citations (April 2015) since we first described the approach in 2011. We first devised the best fit framework approach as a pragmatic methodology for research synthesis. Research synthesis, the science of systematic reviews, is one route by which improvement work in healthcare may flourish. Synthesis is the process of combining or ‘pooling’ relevant evidence from multiple similar studies in order to develop more robust, generalisable conclusions than are possible from the findings of a single study. Experience from several fields suggests that research synthesis stimulates the demand for good quality reporting of initiatives, programmes and policies.4 Evidence synthesis offers an opportunity to demonstrate repeatability, a pre-requisite if improvement practitioners are to learn from the experience of others.5

Framework synthesis derives its main operating principles from framework analysis of primary data.6 Framework analysis was developed by two qualitative researchers, Ritchie and Spencer, in 19947 and offers a qualitative method suited for analysis of organisational policies or procedures.6 It is best adapted to research with specific questions, a limited time frame and issues that have been identified a priori. Although framework analysis may generate theories, the prime concern is to describe and interpret what is happening in a particular setting.8 As such, its principles are well suited for analysis of improvement studies.

Improvement studies pose numerous challenges to evidence synthesis. They incorporate a diversity of perspectives, approaches and contexts.9 The resultant heterogeneity in quality improvement (QI) publications (eg, statistical process control charts, before-and-after studies, qualitative designs) requires synthesis approaches that are flexible and yet underpinned by sound organising principles. Differences between improvement strategies make it correspondingly more difficult to synthesise across studies and to draw appropriate conclusions.10 At the same time, the field benefits from a rich range of strategies from which to choose. Heterogeneity in outcomes resists formal mechanisms for pooling quantitative data while, at the same time, offering multiple different lenses through which to view quality of care. Notwithstanding such challenges, commentators have detected improvements in the methodological quality of the evidence base.5 Evidence, while not necessarily of robust design, is sufficiently plentiful to recommend synthesis. A rich variety of study types provide additional supporting detail to supplement findings from trial reports. Framework synthesis offers a flexible vehicle for:

analysis and synthesis of secondary data;

analysis and synthesis of primary data;

integration of primary data with secondary data;

integration of quantitative and qualitative data, of either primary or secondary origin.

Before we developed BFFS, reviewers of qualitative evidence faced a choice between descriptive approaches separated from theory, such as thematic synthesis, and more time-consuming interpretive approaches.11 A descriptive approach misses an opportunity to make an early connection between data from the included studies and the relevant theory. In contrast, an interpretive approach, such as meta-ethnography, is not best suited to a key purpose in QI, namely production of actionable messages to inform practice. Such an interpretive approach is further limited by an exacting requirement for rich and thick conceptual and contextual data.12 BFFS seeks to capitalise on the strengths of each approach: It reduces the time taken to generate theory de novo; it also enriches the process of theory generation by engaging with existing theory and is specifically geared to produce actionable messages.

Damschroder acknowledges that use of a framework in guiding research

… promotes generalization of findings through use of theory. Specifically, a theoretical framework will enable systematic identification and understanding of drivers that predict success in different settings, guide adaption of targeted practice changes and implementation strategies, and more quickly and confidently build the scientific knowledgebase.13

Such inherent advantages broadly transfer from research to QI and evaluation. However, in the context of QI the emphasis is on actionable messages to enhance performance rather than on an imperative to discover and disseminate new knowledge.5

If improvement practitioners are to harness the potential of theory, as recommended in this journal,14 they need to identify transparent, workable and pragmatic approaches to integrating their data and engaging with wider theory. A framework synthesis approach allows a team to go beyond insights from isolated case studies by seeking to identify what is generalisable across multiple settings. By identifying patterns and themes from the synthesis, an improvement practitioner is able to formulate a well-conceived action plan to address system-wide considerations.

This paper describes the best fit method and its variants, considers the potential role for the method in improvement work and acknowledges some challenges raised by the technique.

BFFS: the basic approach

The BFFS technique follows seven steps (figure 1). We shall illustrate these by following a hypothetical example based on a real-world improvement project.15 First, a team defines an area for improvement using standard systematic review approaches (Step 1). So, for example, a team may seek to improve prevention of deep vein thrombosis (Outcome) in postoperative surgical patients (Population) by introducing a computerised decision support (Intervention). The team then conducts comprehensive searches to identify systematically as many relevant publications on the topic as is feasible (Step 2a). In a parallel process, the team identifies examples of theories or methods, either retrieved opportunistically from within the topic-relevant searches or purposively from supplementary search methods (Step 2b). Compendia of behavioural theories16 or evidence-based practice models17 may offer a useful starting point for a best fit framework. In our illustrative example, the FITT framework18 (‘fit between individuals, task and technology’) could be used to inform the a priori framework for the synthesis because it conceptualises adoption of information technology in a clinical environment as a ‘fit’ between users of the system, the technology itself and the clinical activities which it is designed to support. Significantly, the theoretical framework need not represent a perfect match for the question or evidence—it only needs to offer a ‘good enough’ starting point as designated by the label ‘best fit’. Once the team has identified one or more existing, relevant theories or models, they ‘deconstitute’ the themes, factors or concepts from the model or framework into a single pragmatic framework (Step 3b). So, in our example, data extraction would be constructed around three main concepts: attributes of the users (deconstituted into such subcategories as computer anxiety, motivation), attributes of the technology (deconstituted into such subcategories as usability, functionality, performance) and attributes of the clinical tasks and processes (again deconstituted into such subcategories as organisation, task complexity).

Figure 1

Overview of the best fit framework synthesis process.

Figure 2 shows the interconnected parts of the original FITT framework deconstituted into a framework for data extraction. Data may be derived from published studies or unpublished reports. They may be expressed as quantitative data (eg, responses to a survey on computer anxiety) or qualitative data (perceived characteristics of a usable clinical decision support system). A team may even synthesise secondary data from published reports with primary study data. The team then extracts study data into the concepts and subcategories of the framework (Step 4). Where the team identifies data not accommodated by the original a priori framework they can modify the framework by adding additional concepts, as they emerge from the data using thematic analysis to make the framework more specific to the question, intervention and setting (Step 5). So, in our example, if the team identified a sizeable body of data relating to the importance of organisational culture then this could be added as a new concept, again subdivided into appropriate subcategories.

Figure 2

Selected a priori themes deconstituted from the FITT (Fit between Individuals, Task and Technology) framework 7.

Concepts generated from either the a priori framework or the subsequent thematic analysis are amalgamated to create a new expanded thematic framework (Step 6), as illustrated in figure 3. This process makes transparent which concepts in the final synthesis are based on pre-existing theories and frameworks and are thus confirmed by extracted data and which are new, emerging from the evidence. This transparency represents a significant and original benefit of the BFFS method, locating it within systematic approaches to evidence synthesis. Finally, the team completes the synthesis by exploring any relationships that may exist between the individual concepts with reference to relevant theories and the evidence from the primary research studies. In effect, they ‘reconstitute’ the concepts identified from the revised framework into a revised and expanded conceptual model that describes the processes or mechanisms at work in an intervention or behaviour (Step 7). (See the hypothetical revised framework in figure 4.) The robustness of this revised model can be ‘tested’ qualitatively using sensitivity analysis, based on variables such as study quality or setting (data generated in Step 3a), to establish the extent to which concepts in the new model are underpinned by findings from good quality research studies.19

Figure 3

Building on the a priori framework.

Figure 4

Step 7—Hypothetical revised FITT (Fit between Individuals, Task and Technology) framework 7 incorporating new concepts.

Source of the a priori framework

The ‘best-fit’ approach differs from other methods of synthesis in two ways. First, it creates an a priori framework for data extraction and analysis. Second, the method combines deductive (framework) and inductive (thematic) analysis approaches, not reported explicitly for any other method of synthesis. We have identified four variants that relate to how the initial framework is identified, derived and subsequently used. These variants are described briefly here.

Single framework

In some cases, a single theory or framework will be rich and comprehensive enough to generate themes for the a priori framework. The choice of theory or model depends on the question or topic. For some topics, there will be an obvious choice for the framework, such as the Health Belief Model20 or a framework for monitoring patient safety.21 Frameworks do not necessarily have to be based on theory. A policy-orientated framework such as, in the UK, the Quality and Outcomes Framework (QOF)22 may be equally informative as a structure against which to extract data.

Meta-frameworks

In other well-theorised areas, as in our own case of workplace smoking prevention,23 multiple relevant theories, with their combination of common and unique elements, are used to generate the a priori framework and its themes. The presence of a choice of candidate theories is especially likely to be the case within improvement work given its cross-disciplinary nature. From these, the review team is able to create a customised meta-framework (framework of frameworks) within which they can reconcile overlapping elements and recognise unique elements contributed by each theory. One such example is the Consolidated Framework for Implementation Research (CFIR), created from 19 frameworks,24 which subsequently contributed a ‘best-fit’ a priori framework to a QI synthesis in this journal.25 An initial task for a review team is to determine whether use of either a single theory or a meta-framework is indicated by the review topic. No other synthesis method overtly combines multiple theories or frameworks in this way.

Logic models

In other cases, it may be helpful for a review team to use a logic model variant as a framework. A logic model is ‘a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve’.26 Logic models are increasingly used within systematic reviews to highlight key contextual factors and expected outcome variables.27–29 The logic model variant is indicated where theorising is relatively immature as it offers a ‘scaffolding’ framework while focusing on programme theory. Programme theories seek to explain how a particular improvement programme is conceived to work. This is in contrast to more abstract representations of what are termed ‘grand theory’ and ‘mid-range theory’. A logic model framework synthesis is appropriate when a team has identified key elements of an intervention but not necessarily how these are interrelated. Elements of a logic model are ‘deconstituted’ to become fields in a data extraction form. Once extraction is completed, relationships identified from the data are depicted and a revised, expanded and tested logic model is ‘reconstituted’.

Test and retest frameworks

The best fit approach implicitly involves a test–retest function (the team tests the a priori framework against the evidence and then retests the robustness of the revised framework or model qualitatively through sensitivity analysis). However, the approach explicitly offers an additional opportunity to test and then retest a theoretical framework. For example, a team could divide a dataset into two subsets using the first subset to generate the framework and the second to test it. They could divide the dataset chronologically to document improvement over time. Alternatively, they could divide the dataset by separating ‘low-quality’ studies from high-quality studies to examine similarities or differences.18 30 They could even divide the dataset according to theoretical richness to examine whether projects that explicitly reference theory demonstrate more complete fulfilment of important concepts than those that have been developed in the absence of theory. A framework can also be used to highlight contradictions or disparities between data sources. For example, a framework that documents desired characteristics of a service, based on the published literature, may match imperfectly to qualitative data from stakeholders. Data not explained by this aspirational framework may be used to highlight shortcomings, expand the evaluation criteria or justify a mixed-method evaluation design. Test and retest frameworks are therefore indicated when a team seeks to explore or explain a programme or intervention, and its supporting evidence base, beyond merely itemising its characteristics.

We illustrate the potential role for the best fit framework synthesis method within improvement work from two types of source given that few published examples of its practical use exist in the literature:

by defining the practical contribution of the method more generally within the wider literature. It should be possible to extrapolate from these syntheses to how the method might be applied to studies of improvement (ie, via cross-fertilisation from other fields);

by identifying published examples of the use of theory and theoretical frameworks within reviews of improvement studies. It should be possible to demonstrate the potential added value in enhancing current improvement work (ie, via extension within the improvement field).

We start, in the next section, by examining how existing syntheses have used the methods referenced in our two seminal best fit framework methodology articles.2 3 In the following section, we examine illustrative examples31 of the use of frameworks, models or theories published in BMJ Quality and Safety between 2011 and 2015.

Defining the contribution of the best fit synthesis approach

Recently, much attention has focused on how theory makes a conceptual contribution to the QI process.14 32 While there is reason to believe that examining theory through established good practice in theory development or through synthesis-specific methods can enhance understanding, particularly of complex interventions or programmes,33 this contribution remains to be demonstrated empirically.

BFFS focuses instead on the instrumental contribution of theories or frameworks—that is, how a model, framework or theory makes it easier to conduct a meaningful and insightful review. In particular, it represents a systematic, explicit and auditable way of engaging with theory. Identification and selection of theory is made explicit, while acknowledging that the selected theory/theories are contingent, that is, they do not claim to be ‘ideal’ but only fit for purpose. In comparison with logic models, use of external theoretical frameworks extends beyond the internal case-specific logic of an individual programme or intervention to engage with a wider body of potentially more generalisable theory. As with logic models, best fit frameworks may further offer a role that extends, beyond their original context of qualitative synthesis, to ‘scaffold’ the synthesis of primary as well as secondary data and quantitative as well as qualitative data.

Next, the review team can use the selected framework, or meta-framework, as a structure within which to organise subsequent data extraction. This ensures that the review team is self-consciously engaging with existing theory throughout the whole process of data extraction, that is, earlier in the theory generation/validation process.

Finally, the use of an existing framework deductively, and the subsequent inductive phase to accommodate extraneous data, offers a transparent mechanism by which a team might document any added, and potentially innovative, contribution made by data derived for the review. Review teams may find BFFS particularly valuable in improving the clarity of both description and interpretation.

Potential within improvement work

BFFS appears well suited to improvement work as an activity rich in theories, where behavioural, social, organisational and implementation theories and frameworks might all be considered relevant.32 Frameworks are already common within the improvement literature15 34–37 and may be used at all stages of the improvement chain, from identification of intervention components through to criteria for evaluation and as vehicles for subsequent dissemination. Theory offers researchers a head start in understanding processes and relationships at play within a given context.38 39 For example, Taylor et al40 used a theoretical framework to highlight the inconsistent application of Plan-Do-Study-Act principles together with the associated complexity of such cycles. Potentially, such a framework may help understand why such approaches are successful in some contexts but not in others.

BFFS enables a review team to explore factors affecting the development and implementation of QI interventions. Using documentary evidence as well as other literature, Dixon-Woods et al describe how they ‘analysed evaluation reports relating to five Health Foundation improvement programmes using a form of ‘best fit’ synthesis’.25 They used a pre-existing framework, the CFIR, essentially a meta-framework of themes from a synthesis of existing theories,24 for initial coding and then ‘updated this in response to the emerging analysis’.25 A meta-framework appears to make it less likely that the review team will overlook important themes. It also makes it more likely that the framework will be able to accommodate a significant amount of the data. The resultant practitioner/policy evaluation framework identified 10 key challenges to be addressed by future initiatives. This transformative process turned a theoretically informed academic meta-evaluation framework into an authentic knowledge translation product. Dixon-Woods et al do not discuss in detail their experience of using the BFFS method, it being, essentially, a means to the end of integrating ‘lessons from evaluations of the Health Foundation's improvement programmes with relevant literature’.25 However, they claim several advantages in passing. The framework facilitated ‘rapid preliminary classification of the material’25 and ‘while neither representing the only way of presenting the data nor imposing a hierarchy for the themes, did offer an opportunity for clarity of representation’.25

Elsewhere, theory has been used to identify barriers and facilitators to QI work15 and to explore interconnected elements crucial to hospital reliability.35 The most extensive framework approach within BMJ Quality and Safety reviewed 95 papers (83 studies) and coded 1676 contributory threats within 20 domains of patient safety.35 The resultant framework sought to improve prevention of factors that cause harm to patients. Kaplan et al combined a systematic literature review with opinions of stakeholders to develop a conceptual model, Model for Understanding Success in Quality (MUSIQ), of contextual factors affecting the success of a QI project.36 Dy et al used a consensus approach to develop a best fit model as ‘a way to classify and compare patient safety practices and thereby to interpret the patient-safety literature’.37

Three scoping reviews sought to generate ‘five dimensions of safety’21 to capitalise on learning from the Mid-Staffordshire Inquiry41 and the Berwick report,42 achieving political salience in preference to theoretical adequacy. BFFS represents one method of using theoretical or policy-derived frameworks pragmatically to advance specific improvement programmes. It also offers transparency and auditability when selecting an appropriate analytical structure. The production of such theory-based evidence syntheses can help in part to meet the call for ‘developing a compendium of well-constructed reports that demonstrate the effective use of theory in actual improvement projects’.14

Outstanding challenges in using the best fit framework approach

Although BFFS has been tested in the context of studies of improvement, more exemplars are needed to help decide which method is most appropriate under which circumstances. Our starting assumption, based on the characteristics of improvement studies, is that a meta-framework variant is an appropriate default. However, we need more plentiful instances of how frameworks have been used to integrate different types of evidence, whether these be primary/secondary, qualitative/quantitative or documentary/routine data.

Implicit within ‘best fit’ is that a framework should be ‘fit for purpose’. This requires assessment of the quality of the theoretical framework for rigour and relevance. Theory can be produced empirically, through engagement with the literature or through non-transparent internalised processes. It may be tested with primary or secondary data, subject to Delphi or other normative group processes or explored with stakeholders. To what extent does the rigour of such processes result in a ‘better’ theory? Indeed, what is the best source of underpinning theory or theories? Such issues need further exploration.

How do we judge whether a best fit framework is useful? Damschroder et al24 identify three criteria, used to evaluate implementation frameworks, that translate as generic requirements when using any framework approach in synthesis. First, is the terminology used by the framework coherent?24 A glossary or coding guidance is typically required when operationalising a best fit framework. Associated considerations include (i) whether the concepts are readily understandable to the team (external validity) and (ii) whether the concepts are operationalised consistently by different coders (internal validity). Both criteria must be satisfied if a framework is to be a window, not a gallows. An associated requirement is that ease of use must not encourage the oversimplification, misapplication or abuse of already existing theories. Use of existing definitions, from the creators of a theory or other critical commentators, makes it easier for the team to categorise the data consistently. Where multiple frameworks are used then the resultant meta-framework must ensure that concepts translate across frameworks, that different domains from different frameworks are integrated with one another and that different concepts are designated by different labels.

Second, does the framework promote comparison of results across studies (meta-study)?24 If a framework fulfils such a criterion of ‘transferability,’ this enhances confidence that the chosen framework fits a team's requirements. A third and final consideration is whether the framework stimulates new theoretical development24—if the team cannot readily identify data beyond the original framework then its usefulness as a ‘window’ is constrained. However, even a meta-framework derived from 19 different source documents was found to require additional granularity when accommodating data at an individual project level.43

What is the added value of using theoretical frameworks in evidence synthesis? It is not sufficient to highlight ease of data extraction and associated efficiency savings.44 Ultimately, the contribution of theoretical insights to the quality of the resultant synthesis determines whether time spent in identifying potential frameworks, developing tools for analysis and evaluating data against the framework proves worthwhile. In addition, where the added generalisability of findings advances the discipline as a whole then this, too, should be factored into the overall evaluation.

This exploratory paper seeks to identify potential applications for a synthesis approach that is still in its infancy. It is limited by the relatively small number of published examples and by drawing upon a purposive intensity sample, rather than a more comprehensive sampling approach. Nevertheless, we believe that we have offered sufficient cause to justify further evaluation and testing of the method, particularly given that its equivalent for primary data, framework analysis, has an extensive pedigree.

Conclusion

This paper seeks to promote cross-fertilisation from evidence synthesis to improvement studies. BFFS offers a systematic methodology as a route towards creation of a more rigorous evidence base. Considerable endeavour, showcased within this journal and elsewhere, has targeted methodologies to generate ‘local wisdom and generalisable or transferable knowledge’.45 These objectives translate well to the BFFS approach, which offers a pragmatic response to time-critical projects that integrate large quantities of data.

This paper describes four variants of the BFFS approach to offer a toolkit for future use of theories, models and frameworks in improvement work. Most importantly, BFFS offers potentially replicable methods of data synthesis that, by yielding insights at a ‘meta-level’, may contribute to the advancement of the discipline. For the moment, we only claim that the method possesses relative advantage in terms of efficiency, transparency and engagement with theory at all stages of the review process. We look forward to engaging with the QI community in extending and evaluating this pragmatic method of evidence synthesis.