Telling the story of complex change: an Impact Framework for the real world.

BACKGROUND: In the National Health Service (NHS) in England, traditional approaches to evidencing impact and value have an important role to play but are unlikely to demonstrate the full value of national quality improvement programmes and large-scale change initiatives in health and care. This type of work almost always takes place in complex and complicated settings, in that it involves multiple players, numerous interventions and a host of other confounding factors. Improvement work is usually emergent, with cause and effect only understood in hindsight; challenges around contribution and attribution can lead the key players to question how they can be certain that the described or observed changes are due to their intervention and would not have happened without them. In this complex environment, there is a risk of oversimplifying the observed impact and focusing instead on those things that are easier to measure, missing that which is important but more difficult to evidence.
METHODS: Between 2016 and 2019, an action-orientated approach, drawing on realist and development evaluation approaches, was taken to designing and testing the Impact Framework. First, we undertook a pragmatic review of tools and approaches used by others to capture and demonstrate their impact both within and outside the health and care environment. Following the identification and review of these tools and approaches, and in consultation with national improvement teams in England about their evaluation challenges and aspirations, we developed a set of underpinning principles to inform the design and build of the framework. The principles were informed and finessed following conversations with improvement teams and programme leads in NHS England with respect to the challenges that they were facing and their aspirations in terms of demonstrating their impact and learning as they worked.
RESULTS: The 'Impact Framework' described in this article offers a practical approach to capturing the impact of improvement work at any scale, taking account of unintended outcomes, considering attribution and contribution, and using a narrative approach to uncover the difference made by improvement initiatives in rich detail. In this article, we describe how the Impact Framework has been used with one of NHS England's national programmes, Time for Care, which was delivered between 2016 and 2020.
CONCLUSIONS: The Impact Framework continues to be used, developed and further tested by national improvement programmes being delivered by NHS England and NHS Improvement and is updated regularly. The framework has been developed to be accessible to frontline teams and is supported by a set of resources to help improvement teams and individuals to use by themselves (https://www.england.nhs.uk/sustainableimprovement/impact-framework/).

Introduction

Improvement programmes have been undertaken in the National Health Service (NHS) for decades, but there has been wide variation in how effectively these programmes have been able to capture and describe the difference they have made. In 2016, NHS England identified the need to develop a robust but pragmatic approach to evaluating the impact of its improvement programmes. This led to the creation of the Impact Framework.

The NHS must be able to demonstrate that it uses its resources wisely. This is true for all aspects of the NHS, including within the field of quality improvement, which is the focus of this paper. There is a need to be able to demonstrate with a high degree of certainty that a specific improvement initiative contributed (or failed to contribute) to an outcome or a set of outcomes. There is a need to understand the mechanism by which improvement happens to maximize the success of the initiative, to share learning with colleagues and others and to inform future work in the area. It is also vital to identify and explore the unintended consequences of any changes made.

However, evaluating quality improvement in healthcare is challenging [1, 2]. The NHS in England operates in a highly political and frequently changing environment. There are often limited evaluation resources on the front line, which coupled with political pressure to report impact quickly can make it tempting to overlook changes that are challenging to evidence and instead focus on what can most readily be measured. Often, the focus is on a small number of priority outcomes, determined by commissioners before the work begins, which can be evidenced within the short reporting cycles of complex, highly political environments.

This type of work usually takes place in complicated or complex settings and involves multiple players, numerous interventions and often a host of confounding factors. Snowden and Boone [3] describe a complicated setting as one where the relationship between cause and effect requires analysis, some form of investigation or the application of expert knowledge. In a complex setting, they suggest that there is no one right answer to a problem or challenge and that in such settings cause and effect can only be determined in hindsight. In practice, there is a real risk of oversimplifying the observed impact and missing important learning by focusing on the things that are easiest to measure and ignoring what is important but more difficult to evidence.

Traditional approaches to evaluation have a vital role to play but, on their own, are unlikely to demonstrate the full impact or value added of an improvement intervention. Some approaches such as the use of control groups or other counterfactuals can be cumbersome and impractical for multifaceted improvement initiatives taking place in complex settings [4]. The many influencing variables in complex contexts cannot always be known or controlled for and trying to do so risks oversimplifying our understanding of the change mechanisms that this type of evaluation seeks to describe. Greenhalgh and Papoutsi [5] argue that not evaluating the mechanisms or processes of how change occurs in complex environments risks missing the ‘active ingredients’ of the models we seek to understand. They describe how the model of care under scrutiny can ‘change over time, operate at multiple levels and be context specific’, making attempts at meaningful evaluation particularly challenging.

Also, contexts tend to be ‘busy’ in so far that a whole range of players can be working towards the same desired outcome; this leads to challenges around attribution and contribution with key stakeholders asking how described or observed changes can be attributed to their intervention and their intervention alone.

Shah and Course [6] outline the challenges experienced by healthcare leaders in trying to articulate the return on investment from applying quality improvement at scale. They describe a framework to articulate and assess return in investment at multiple levels: this includes improved outcomes and experiences for those that receive services; enhanced engagement and motivation of staff; improved productivity and efficiency of teams; cost avoidance; and cost reduction and increased revenue.

The Impact Framework was developed as a pragmatic approach to help teams working in quality improvement to overcome the challenges set out here.

What is the Impact Framework?

The Impact Framework (Figure 1) is an approach designed to support teams who are delivering quality improvement projects or programmes in health and care systems with the skills and confidence to demonstrate the impact and value of their work, Underpinned by eight principles (Box 1, see section ‘How we developed the Impact Framework’), the framework draws heavily on improvement methodology itself, by promoting the use of small tests of change, reviewing incoming evidence and revising the programme theory in real time of what will change, how and why. The framework consists of four core steps that form a cyclical process; the cycle may be completed once for small-scale projects or several times for large, more extended programmes. Two additional optional steps can be added to the core process to take account of unintended outcomes or consequences and to consider attribution and contribution. Stakeholders are involved at every stage in the framework, and a narrative approach is used to describe in rich detail the difference that the project or programme makes.

Figure 1

The underpinning principles

Principle 1: be clear about what you are trying to achieve and understand what changes

Principle 2: there is no one absolute measure of value so the focus is on creating a good enough account of value at a point in time

Principle 3: judgements of value can and should be made by those involved in the work, not just third-party ‘objective’ evaluators

Principle 4: involve stakeholders in defining and evaluating at meaningful intervals

Principle 5: be honest about contribution and attribution

Principle 6: do not dismiss the elements of value that are hardest to capture

Principle 7: keep it simple and transparent

Principle 8: focus on learning not just accountability

Step 1: Articulate what will change, how and why

This step sets out the broad aim of the work, accepting that in complex settings there is unlikely to be a clear and detailed understanding of ‘the answer’ or how this will be achieved at the start of a project or programme. The aim itself and how it will be achieved is likely to develop and evolve as evidence is captured about what works, what doesn’t and what has changed. The articulation of change should therefore ‘describe’ what is expected to change and how the change might be achieved, but not ‘prescribe’ it. A range of tools can be used to capture the articulation of change, including logic models, driver diagrams, visual approaches and narratives. The Impact Framework does not prescribe a specific tool for this purpose, instead allowing teams to select the best approach for them. Key stakeholders (for example, commissioners, sponsors, patients and carers and staff members) should be engaged to build a comprehensive and shared understanding of the logic behind the intended change.

Step 2: Identify and develop your evidence of change and don’t ignore the things that are harder to evidence

This second step takes a practical approach to identifying and prioritizing sources of evidence needed to demonstrate that the changes articulated in Step 1 have been achieved or that progress is being made towards achieving them. Again, stakeholders will be vital in informing the choice and source of evidence. The Impact Framework advocates a mixed method approach using a combination of qualitative and qualitative evidence sources as appropriate, as well as the triangulation of evidence to counter the fact that almost all data available will have limitations. For example, Wolpert and Rutter [7] discuss the value and usefulness of routinely collected datasets in the evaluation of improvement work in complex health systems. These data sets can often be Flawed, Uncertain, Proximate and Sparse and as a result can be over-interpreted or alternatively dismissed as incomplete. Wolpert and Rutter assert that these data, although incomplete has the potential to increase our knowledge and understanding by exposing issues for interrogation and by encouraging triangulation of different data sources and types, which can also encourage teams to think about the acceptable standard of evidence. This point relates to one of the key principles of the Impact Framework; there is no one absolute measure of value and it is important to focus on creating a good enough account of value at a point in time.

The Impact Framework promotes two additional and optional tools that can be used to capture and explore unintended consequences, and the specific contribution of the work, as opposed to ‘proof’ of attribution. Most Significant Change [8] describes the technique of collecting, reviewing and sifting stories from the front line of the difference the intervention is making and, in this way, can surface unintended consequences. Contribution Analysis [9] describes a robust process of exploring and capturing the contribution of an intervention to an outcome or set of outcomes, where casualty is challenging to attribute in a complex setting or where many players contribute to the same outcome.

Step 3: Review the evidence

At the heart of the Impact Framework is the regular review of the evidence captured at Step 2 and how it facilitates the understanding of what will change, how and why, as set out at Step 1. Reviewing evidence as it emerges can help understanding of progress and successes as well as challenges and unexpected outcomes. Doing this regularly and collaboratively is a core step in the Impact Framework, in line with the idea that we are conducting small tests of change as part of an overall programme of change. It is vital that stakeholders (including those delivering the work) should be involved in such evidence reviews. Key questions, framed around the articulation of change from Step 1 and considered as part of an evidence review include; Is what we thought would happen happening? Are there any gaps in what we know? Do we need to make any changes to the programme? Do we need to make any changes to the way we capture evidence? Have there been any unexpected outcomes or setbacks?

The answers to these questions may, in turn, bring about amendments in the articulation of change and the evidence that is collected and reviewed in future.

Step 4: Share the impact of your work and key learning with others

An ‘Impact Story’ brings together the ‘articulation of change’ from Step 1 and information from the evidence reviews in Step 3, to tell the story of the work, the difference it has made and what has been learned along the way.

There is no predefined format, it can vary depending on the work and the audience, but there are some common features: it should include a rich range of triangulated evidence, which relates to the ‘articulation of change’; there should be ‘no narrative free data’; it should be accessible whilst capturing the complexity of the work; it should focus on learning, not just accountability and it should reflect the most significant changes as reported by people on the ground. Whilst trialling the Framework, this step was undertaken using the Disney Pixar framework as described in a blog post by Allen [10]; this helped to structure the impact story in a format that captured the imagination and emotional engagement of the stakeholder, bringing to life, the difference that the work made.

How we developed the Impact Framework

Our understanding of what works and how it works will change over time, based on small tests of change and taking account of a rich range of evidence. The idea of small tests of change is set out in the NHS England Large Scale Change Guide that informs much of the quality improvement work at the national level across NHS England and NHS Improvement [11]. Ramaswamy et al. [12] describe how quality improvement projects implemented in complex settings necessitate the use of an approach defined as ‘probe-sense-respond’ where teams experiment, learn and adapt their changes to the local setting. Quality improvement professionals instinctively use the probe-sense-respond approach in their work but tend to approach evaluation as a linear path between the intervention and the desired outcome.

The guiding principle in developing a robust and pragmatic approach to demonstrating impact in the improvement world was that quality improvement required a broad framework that would be useful and practical, would scale up or down depending on the context and would acknowledge and consider the complex environment that much improvement work in the NHS in England takes place.

Between 2016 and 2019, an action-orientated approach, drawing on realist [4] and development evaluation approaches [13], was taken to designing and testing the Impact Framework. Initially, we undertook a pragmatic review of tools and approaches used by others to capture and demonstrate their impact both within and outside the health and care environment (Supplementary Appendix A sets out a list of the 55 resources that were reviewed during this development phase).

Following the review of these tools and approaches, and in consultation with national improvement teams about their evaluation challenges and aspirations, we developed a set of underpinning principles to inform the design and build of the framework (Box 1). The principles were the building blocks of the framework and were informed and finessed following conversations with improvement teams and programme leads in NHS England and NHS Improvement with respect to the challenges that they were facing and their aspirations in terms of demonstrating their impact.

A case study of how we used the Impact Framework

Time for Care was an NHS England and NHS Improvement national programme, delivered between 2016 and 2020. It was designed to help general practice teams manage their workload, adopt and spread innovations that free-up clinical time for care, and develop the skills and confidence to lead local improvement.

The route to impact

Logic models had been developed to underpin evaluation activities within the Time for Care programme but were not generally utilized by the programme team more widely and were felt to be inaccessible and overly technical. As a result of this experience, narrative approaches were used to co-design an articulation of change with key stakeholders from across the team. Small groups were able to use the Disney Pixar framework to quickly articulate what would change, as well as how and why for their own areas of focus within the programme. Review of these narratives as a programme team allowed the identification of shared outcomes, interdependencies between projects and areas they were likely to reinforce each other’s impact. Once a degree of consensus was reached around a programme level narrative, this was used to inform the development of a one-page visual representation, referred to within the team as its Route to Impact (Figure 2).

Figure 2

The Route to Impact was used to prioritize the collection of evidence around key processes and outcomes and also acted to identify key messages for effective and consistent communication about the programme with both commissioners and those the team sought to work with in general practice.

Gathering and reviewing evidence

A wide range of qualitative and quantitative evidence was collected for each priority identified in the Route to Impact. Evidence reviews were undertaken on a quarterly basis with key stakeholders from across the team where all available evidence was considered and key questions included; Is what we thought would happen actually happening? Do we need to make any changes to the programme? Are there any gaps in our evidence? Do we need to make any changes to the way we capture evidence?

The impact story

At the end of the 2016–2020 commission for Time for Care, an Impact Story was developed using the Route to Impact and the wide range of qualitative and quantitative data collected and reviewed throughout the commission. This was tailored for different audiences, with an in-depth 50 slide presentation deck for use within the team, and a shortened version for busy commissioners. Both versions used the Disney Pixar framework (see Box 2).

The Time for Care narrative, using the Disney Pixar framework (March 2020)

Using a narrative approach at this point brought the work full circle, the original articulations of change could be considered as describing what had been intended to happen, with the Impact Story telling the tale of what actually happened. This proved to be an engaging and compelling approach. Delivery teams and other stakeholders from across the Time for Care programme felt proud of the story of what had been achieved, and commissioners were able to easily consider the difference the programme had made and went on to recommission the work.

Changes made to the programme because of the use of the framework

The programme team recognized the successes claimed in the story, but they wanted more ‘focus on the dragons’ to reflect the challenges they had overcome along the way. As a result, there has since been a much more systematic focus on reflective practice and capturing and applying actionable learning.

The Time for Care team was recommissioned in 2020 and became responsible for delivering the NHS England and NHS Improvement Access Improvement Programme. The continued value placed on this team was felt by senior managers in the programme team to be due in no small part to the systematic and robust approach given to evidencing the impact of the programme, a result of the application of the Impact Framework. The Time for Care team continues to apply the Impact Framework to all its work, including the COVID-19 support work it undertook between May and October 2020, which required the rapid development of a theory of change and monthly evidence reviews to keep pace with the emergent nature of the work.

Reflection and conclusion

As outlined, the challenges of evidencing the impact and value of improvement programmes in the NHS reflect the fact that the work takes place in complex and often highly politicized contexts, with teams under pressure to evidence impact quickly. The value or impact is often narrowly expressed via a handful of priority outcomes and proving attribution in relation to these outcomes cannot be achieved with any certainty. Traditional methods of evaluation are likely to miss vital aspects of value.

The Impact Framework described in this paper aims to offer a practical approach to capturing the impact of improvement work at any scale, taking account of unintended consequences, considering contribution and attribution and using a narrative approach to describe changes made by improvement initiatives in rich detail. As outlined, it was developed after reviewing and taking account of evaluation tools and approaches which work in complex settings in health and care. The framework has been developed to be accessible to frontline teams and is supported by a set of resources to enable others to use it in their work [14].

The iterative way in which the framework was developed has enabled refinement of the framework based on some key observations from practice as outlined below.

A prescriptive step-by-step approach that calls for the use of specific templates was deemed to be unhelpful from our experiences of applying the Impact Framework with real teams. What is often valued about the Impact Framework is that it offers an accessible framework that can guide a team’s thinking but can be applied in a wide range of contexts. It can be flexed to reflect the evaluation capability within the team. We have found that those who do not have evaluation experience can apply the Impact Framework for themselves with little or no support, particularly where they have been able to access the offered training or capability building support before starting.

Teams often find narrative approaches more accessible. Developing a ‘story’ at Step 1 helps to engage a wide range of stakeholders in articulating the change and building consensus around it. Terms like ‘logic model’ or ‘theory of change’ can be off-putting to some, but we have observed individuals and groups to be very quickly engaged in telling their story.

As part of the development and refinement of the Impact Framework, a retrospect was facilitated by the Knowledge and Intelligence Team within NHS England and NHS Improvement in April 2018. The spirit of the retrospect review is one of openness and honesty, asking ‘how do we avoid problems and ensure repeated successes in future, similar projects?’. Recommended actions that are worth sharing include the following:

Commissioners need to give programme teams the time and space to articulate their desired change and potential impact; this will change and evolve as the programme develops. It is essential that programmes have clarity in their commission and are clear where their accountability stops (what we term the ‘accountability threshold’). Beyond this point, programmes may well have a significant contribution to make. Exploring contribution rather than attribution becomes essential at this point, in order to avoid the missing aspects of value whilst accepting attribution is unlikely to be proven with any degree of certainty because of the likely number of players effecting the same outcomes.

Programme teams must consider system drivers and strategic alignment of their work when developing their theory of change and regularly check the alignment of the work with wider strategic plans. Political dimensions of the work and indeed its evaluation need to be understood and reflected.

Programme teams need to be aware that applying the Impact Framework is not a separate activity to the work of the programme, rather it can help shape the shared purpose and understanding of what is working or not working.

Clarity of the role in relation to data collection is critical, with the need to be clear as to who is responsible for evaluation, helping ensure that is becomes embedded as ‘business as usual’ for the programme. The teams that were most successful in using the Impact Framework to support their work understood that evaluation ‘was not a separate industry.’

Finally, language needs to be considered carefully, because of potential political ramifications in the highly politicized health and care system; it is important to appropriately adapt the language that is used with different audiences.

Click here for additional data file.

Once upon a time…	The general practice workforce was under tremendous pressure. There was more work, increasing costs, increasing patient expectations, tighter financial constraints, low staff morale and difficulties in staff recruitment and retention.
Every day…	Many practices recognised there was waste and inefficiency in the system, but the workforce lacked the hope and ambition that they could make things better. There had been limited opportunities to develop improvement skills and knowledge in primary care, as well as inadequate investment in developing change leadership skills.
One day…	In response to these challenges, the General Practice Development Programme (GPDP) was launched. At the heart of this, NHS England was tasked with improving quality, collaboration, access, safety and staff morale by releasing time and increasing improvement capability. We called this the Time for Care programme. It was a national programme, with an ambition to reach every Clinical Commissioning Group (CCG) and associated general practice within the NHS in England.
Because of that…	A number of offers were made to CCGs with various support options. CCGs, practices and individuals began to engage in the programme and overall 3622 practices (53%) in England took part. People who got involved had a positive experience and recommended it to others.
Because of that…	Our evidence tells us that as a result of taking part, people developed quality improvement skills and gained confidence in applying them. As a result, we saw improved processes, improved team dynamics and an increase in the capability to improve safety, patient experience and quality. Time was reportedly being saved or was very likely to be saved in practices as a result of taking part in the programme, amounting to an estimated 850, 198 annual hours of clinical and admin time. Participants were able to work together at scale with increased learning and sharing, and the programme’s Primary Care Improvement community grew to over 5,000 members.
Until finally…	Efforts by the Time for Care team have been recognised and valued as an effective means of supporting general practice in England to release capacity and secure development to improve in areas such as quality, collaboration, access, safety, and staff morale. The Time for Care team continues to work in this area and has gone on to deliver NHS England and NHS Improvement’s Access Improvement Programme.