Understanding the conditions for improvement: research to discover which context influences affect improvement success.

Context can be defined as all factors that are not part of a quality improvement intervention itself. More research indicates which aspects are 'conditions for improvement', which influence improvement success. However, little is known about which conditions are most important, whether these are different for different quality interventions or whether some become less or more important at different times in carrying out an improvement. Knowing more about these conditions could help speed up and spread improvements and develop the science. This paper proposes ways to build knowledge about the conditions needed for different changes, and to create conditional-attribution explanations to provide qualified generalisations. It describes theory-based, non-experimental research designs. It also suggests that 'practical improvers' can make their changes more effective by reflecting on and revising their own 'assumption-theories' about the conditions which will help and hinder the improvements they aim to implement.

Introduction

Does a proven quality improvement (QI) vary in its effectiveness in different settings and if so, how and why? If there are variations, should we only try to implement it in some settings? These are questions I had of the Michigan Keystone programme, as many of us in Europe and elsewhere are trying to implement similar programmes.1 This programme implemented a similar set of changes across 108 different ICUs in the state. What caught the headlines was the dramatic overall reductions in infections. But what was noticed by implementers elsewhere was the variation of results between the projects. Intriguingly, one of the studies reported that, ‘the intervention was modestly more effective in small hospitals’. Was it due to differences in how thoroughly different ICUs implemented the changes? The study design did not provide for data collection to give details of the variations or allow explanation of the variations.

For other QIs there also is evidence of variations in their effectiveness in different settings.2 These variations may be due to differences in implementation: many improvements are not discrete single ‘before–after’ changes but ‘facilitated evolution’.3 Differences in implementation may be due to differences in context: implementers often cite differences in resources for fewer results than other settings or in their access to expertise, or other ‘context factors’.4 5

Context can be defined as all factors that are not part of a quality improvement (QI) intervention itself.

Only some of these ‘surroundings’ may influence improvements and their effectiveness—termed here the ‘conditions for improvement’.

‘Conditions for improvement’ are those internal to the implementing organisation (eg, information technology) and those external to it (eg, payment and regulation systems), and are made by, and operate on, different levels of the health system.

The definition of a boundary between the improvement ‘intervention’ and the ‘context’ is relatively arbitrary. To be useful to others, reports need to describe precisely the intervention implemented and any evidence of the conditions which influenced the intervention.

The aim of some QI research is to understand which conditions influence improvement and how they do so.

Knowing whether or how much context explains differences in implementation and effectiveness would help others to make changes and speed up the spread of improvements proven in other settings. Part of building this knowledge and the science of improvement is discovering whether context affects the implementation and outcomes of some or of all types of QI. If it does, exactly which context influences are important for which improvements?

Why describe or study context?

To decide whether to carry out a QI, policy makers or managers and clinician say they need to know if the change contemplated is effective in improving outcomes. For some improvements, research provides knowledge with a high degree of certainty from using controlled trials, such as trials of whether antibiotic prophylaxis before surgery reduces infection.6

But to make use of this ‘efficacy knowledge’, decision makers say they also need to know if it is likely to be effective in their setting, and how to implement it. This knowledge can come from a number of controlled trials of the same intervention in different settings, and this can help decision makers to answer the ‘will it work here?’ question: it can help discover how ‘context-sensitive’ or ‘robust’ an improvement change is. A recent AHRQ-funded study has reviewed some studies of such research in the patient safety field and used an expert panel process to identify evidence from this type of research and suggest which factors are important. These covered organisational characteristics, external factors, culture, teamwork, and leadership at the level of the unit, and management tools.7

However, using traditional controlled trial efficacy research designs to explore effectiveness in different settings is an expensive and time-consuming strategy. It gives limited help with the ‘how do we implement it?’ question, which controlled trials are not designed to answer. Neither does it answer the question about why an intervention varies by setting, because many features of the setting-context are ‘controlled-out’ in order to answer the efficacy question. “Strength of evidence” in the traditional hierarchy only refers to the certainty of efficacy provided by the design of the study – the internal validity. Implementers need effectiveness “strength of evidence” assessments about external validity, and need to know how to implement the intervention in their setting. An answer to the ‘why does it work or not work?’ question would help decision makers to decide better whether and how to try the change in other settings. It would show whether exact replication is critical, and, if not, help them to design a change which follows the principles in a way which is adapted to their setting, rather than repeating a detailed prescription.

Some QIs, like antibiotic prophylaxis, are treatment-type changes found effective through controlled trials. To ensure many patients benefit from proven changes like this, provider behaviour change is needed. Decision makers have less knowledge about the effectiveness of actions to change provider behaviour or organisation to ensure this change. One stream of QI research is concerned with evaluating the effectiveness of these interventions to provider behaviour and organisation (changes to providers, rather than changes to patient treatment), and with the variety of research methods which could be used.

There are also streams of QI research to assess effectiveness of organisational models, such as those for chronic care or team-based care, or of support systems such as computer physician order entry. These complex interventions are more difficult to define as discrete and bounded interventions, independent of their surroundings, and then to subject them to controlled trial designs.8 The knowledge produced using this approach is also less useful because implementation and effectiveness is likely to depend on organisational features—for example, the type of electronic medical record in use.9 It is also possible that some complex interventions change their surroundings and interact with them over time, as was found in one study of a safety programme in a surgical unit.10 Additional understanding of conditions for improvement may come from studying improvement changes, less as discrete and bounded ‘interventions’ and more as evolving actions by conscious actors interpreting and responding to their surroundings.

Research into, and theory about, context influences on improvement can thus help speed up and spread improvement. First, it helps answer efficacy questions where controlled trials are not possible. In these studies, knowledge about context influences can help to assess how much the quality intervention and how much the context influences affected the outcomes. Second, it helps answer the ‘will it work here?’ and ‘how do we implement it?’ questions. This is because a theory about context could show which context factors influenced implementation at the study site, and thus help others to assess how similar they are and make their own judgement about likely implementation success. Such context theory allows generalisation to settings other than those in which the study was undertaken, through analytical rather than statistical generalisation.

Infrastructure and programme intervention: changes to encourage provider interventions (eg, finance, or regulations)

Implementation intervention: actions taken to change provider behaviour or organisation (eg, training, computer prompts)

Clinical intervention: a change in how patients are treated (eg, antibiotics before surgery)

Higher level interventions set the context for lower level interventions

No need to, if the aim is to answer efficacy questions and a controlled trial is possible, apart from describing the patients or units in the trial (to ‘contextualise’ the study).

In an uncontrolled experimental study, data about context can help to assess the influence of context on outcomes and increase certainty of attribution of outcomes to the intervention.

If the aim is to answer questions about implementation in an observational study, then data about context are needed to assess how context influences implementation in the selling, starting, sustaining and spreading phases (to develop ‘context theory’).

How to study context?

Randomised controlled trials assume that context could influence outcomes. The design aims to exclude many context influences by using comparison groups which are the same, apart from the fact they do not get the intervention. However, if controls are not possible, or if the aim is to discover which aspects of context influence implementation and outcomes, which research designs and methods are best?

Context in uncontrolled experimental trials and PDCA tests

Collecting and reporting data about context is an added burden to research, and may not be possible for simple before/after Plan-Do-Study-Act (PDSA) practitioner reports, although two case report repositories do provide some of this ‘background’.11 12 Any such data collection in these designs needs to be focused on documenting the aspects which are most likely to affect the outcomes, and analysis focused on assessing their relative influence. Reviews of research to discover which aspects have been reported previously would help researchers and practitioners to select which aspects to collect data about and how to collect the data. For example, effectiveness studies of health information technology have found aspects of the host organisation's staffing, size, previous experience and financing to be important.9

Context in ‘naturalistic studies’

“Naturalistic studies” are a broad category of designs for studying QI change in natural settings and which are more suited to developing theory about context. Some designs describe implementation only (eg, case study),13 14 but some may also assess intermediate outcomes (eg, some types of programme evaluation).15 16 Such designs are more often used for studies of large-scale programmes, policies or regulatory changes than for studying smaller projects, although many such large-scale changes include local projects. Two recent examples are Benn et al's 2009 mixed method study of a UK patient safety initiative17 and Greenhalgh et al's 2009 study of large-scale complex improvement programme in London, UK.18

One approach to develop theory of context is to collect a cross section of informants' views about aspects of context which they suggest were important at different stages of implementation. Validity can be enhanced by selecting a sample of informants, from different organisational levels and perspectives, who are knowledgeable about the change and may be able to cite evidence to support their insights. Observers' views can then be crosschecked against documentary or other data to assess the influence of context factors repeatedly mentioned, thus building inductively an understanding of context influences.19 However, informants may not be aware of some influences, and their ‘observer theories’ need to be supplemented with scientific theory, which may direct attention to other data to test hypotheses about possible influences. This approach starts with a model or theory, preferably based on previous research into similar QI changes, which suggested aspects of context which were important. This can then be used to plan data collection about these aspects of context or specify these as hypotheses to be tested.

One of the challenges of this type of research is to capture changes over time and the dynamics of intervention–condition interactions. To do so requires documenting how any influencing conditions change over the period of implementation (eg, a senior manager who was supportive of the change leaves or there are sudden cuts to budget). There are also challenges in understanding exactly how these changes influence the QI implementation. Generalisation in these types of studies could be helped by better understanding exactly how a condition influence interacts with parts of the intervention. For example, why what a leader does affects how personnel respond to training about use of a checklist. This approach moves towards examining how an idea is adapted and evolves in a setting through ‘enabled adaptive change’. In some action- or collaborative-research studies, this may involve the researchers contributing to the change, for example, by giving feedback to implementers.

These types of studies can be enhanced by specifying the ‘change theory’ of the implementers—the assumptions about which actions lead to which results through which steps20—or by defining the researchers' ‘programme theory’ before and after data gathering.15 These ‘theories’ could also include ideas about which aspects of context help and hinder the implementation. It is also possible that pragmatic testing, using PDSA cycles21 would be enhanced by implementers stating their assumptions about the conditions they need and the steps through which changes might affect outcomes. By making explicit their assumptions (theories ‘T’) before testing, and revising these after testing (‘T-PDSA-T’), improvers could learn not just whether a change affected outcomes, but why.

Understanding the conditions influencing an improvement is important to implementers, for deciding whether and how to implement many QIs. Both implementation and effects may depend on aspects of the context ‘surrounding’ the QI change at the study site, which may be very different from those at the implementer's site.

Context influences form part of some theories about how QIs have their effects.

Understanding the ‘implementation action theory’ can help implementers adapt the improvement to their situation, by reproducing the principle rather than copying exactly what others did.

Guidance for deciding which data to gather about context can come from previous research into similar interventions or generic theories about conditions for QI.

A better research base for quality improvement

What could best help develop more knowledge about how context influences implementation and outcomes? Two research questions to be addressed are which specific details about context influences to collect and whether different influences affect different types of improvement? Is, for example, a strong safety culture a much stronger condition for improving hand hygiene interventions than for medication reconciliation, or for computer decision support for prescribing?

Learn from other disciplines

A number of disciplines have developed methods to study ‘implementation’, including public health22, sociology,23 educational studies,24 business studies,25 programme evaluation26 and implementation and innovation science.27 28 Some approaches draw on critical realist philosophy which emphasises the study of mechanisms which trigger effects under certain circumstances rather than using other types of cause–effect understanding.29 These bodies of knowledge provide methods and frameworks for the description of intervention actions, and also seek to assess intermediate changes to behaviour or organisation, or even final outcomes using a theory about how the actions taken then influence outcomes. Such methods are being applied to understand improvements in healthcare. However, research including context factors in these ‘causal’ theories is at the early stages.

More specific assessments of context are required

Research would be more useful and cumulative if it moved beyond general statements about variations in implementation and outcomes being due to ‘leadership’ or ‘culture’. More specific data are needed about, for example, which aspects of leadership by which leaders are important for which QI changes and how what leaders do has this influence. In the case of culture as a context factor, there are different data gathering instruments which can be used.30 Helfrich et al describe a method to assess ‘organisational readiness to change’, which may be an important context factor.31 Some context factors are difficult to operationalise as measures, but the discipline of seeking measures or defining concepts more precisely is necessary for data gathering and for understanding the influence of these factors.

There are frameworks and research from different knowledge domains which can provide a starting point for researchers, ideally drawing on previous research into changes similar to the particular QI being studied. These frameworks often distinguish context influences external to the implementing organisation, and those internal to it,28 or separate context influences which originate at different organisational levels.32 French et al provide a useful general overview of frameworks from different knowledge domains,33 and Rycroft-Malone et al describe a study protocol for studying context in one study.34 Context factors found to affect implementation of evidence-based practices are described by Dopson and Fitzgerald35 and Rycroft-Malone et al.36 Other relevant frameworks include those described by Pettigrew et al,37 Kitson et al38 and Lukas et al.39

Categorising QI changes according to the impact of context

It is possible that the implementation of some types of QI depends more on context than others.7 Furthermore, similar context factors are important for certain groupings of QI (eg, the groupings proposed by Shojania et al40). For example, equipment or automation changes may form one grouping of improvements, with computer-based changes forming a sub-grouping. Taxonomies of improvement interventions and context influences are underdeveloped and the lack of a common language is hindering scientific progress. At present, researchers have little guidance from previous research about which aspects of context to document, and there is possibly an assumption that all types of QI are equally affected by the same types of context influences. Theoretical research is needed to produce groupings of QI interventions according to which aspects of context are important for their implementation and which are different from other groupings of QI. The synthesis of implementation frameworks by Damschroder et al might provide one starting point.41 Such a categorisation could then be the basis for specifying the context elements for reporting of uncontrolled studies (eg, the SQUIRE reporting guidance).42

Interventions, multiple component changes or system changes?

QIs have traditionally been conceptualised as interventions—discrete changes separated from their surroundings—in order to assess whether they cause other changes in outcome variables such as patient outcomes. If the change is an improvement, the assumption is they can be repeated elsewhere to cause the same outcome changes. Some limitations of this way of understanding QI have been noted when applied to complex interventions such as QI ‘bundles’.10 If an intervention is separated conceptually from its surroundings, then one research agenda is to explore how the intervention changes context, as well as vice versa.

Another research agenda is not to conceptualise such a sharp separation, and to view improvement less as a specific change but as an interdependent set of actions which result in many types of changes, which in turn may result in better patient outcomes. Including context in understanding implementation and in improvement theory can advance improvement science and practice. It allows exploration of whether and how aligned changes at different levels may result, through complex influences, in better outcomes, and how these can be sustained. It moves from cause–effect understanding to conditional attribution, which allows qualified generalisations by discovering which conditions were necessary for the improvement to be carried out and its effects. This in turn allows decision makers to assess better likely results locally and how to adapt the change.

Conclusions

In QI, nothing ever happens for one reason or cause. It would be convenient to package changes as a QI which could work anywhere, like an effective drug. But for social interventions to changing social systems, a number of factors influence the implementability and success of many. Some useful knowledge can be generated using medical treatment research designs like randomised controlled trials, some using well-documented pragmatic PDSA testing, but some also require non-experimental naturalistic methods more often used in the social sciences. An understanding of the conditions influencing an improvement in one place is important for spreading this change if it proves effective in this place. Such an understanding can be advanced by better descriptions of implementation and likely significant context factors. It can also be advanced by developing theory in different ways about how specific changes are best implemented in different environments, and how they work through a pathway of influences to change outcomes. More improvement research could usefully aim for ‘conditional-attribution’ explanations and ‘qualified generalisations’: showing the conditions under which improvement changes are likely to be successful, and thus integrating, rather than trading-off, internal and external validity.