Mapping outcomes in quality improvement and system design activities: the outcome identification loop and system impact model.

BACKGROUND: Whether explicit or implicit, models of value are fundamental in quality improvement (QI) initiatives. They embody the desirability of the impact of interventions-with either foresight or hindsight. Increasingly impact is articulated in terms of outcomes, which are often prescribed and sometimes inappropriate. Currently, there is little methodological guidance for deriving an appropriate set of outcomes for a given QI initiative. This paper describes a structured approach for identifying and mapping outcomes. OVERALL APPROACH: Central to the approach presented here is the engagement of teams in the exploration of the system that is being designed into. This methodology has emerged from the analysis and abstraction of existing methods that define systems in terms of outcomes, stakeholders and their analogues. It is based on a sequence of questions that underpin these methods. OUTCOME ELICITATION TOOLS: The fundamental questions of outcome elicitation can be concatenated into a structured process, within the Outcome Identification Loop. This system-analysis process stimulates new insights that can be captured within a System Impact Model.The System Impact Model reconciles principles of intended cause/effect, with knowledge of unintended effects more typically emphasised by risk approaches. This system representation may be used to select sets of outcomes that signify the greatest impact on patients, staff and other stakeholders. It may also be used to identify potential QI interventions and to forecast their impact. DISCUSSION AND
CONCLUSIONS: The Outcome Identification Loop has proven to be an effective tool for designing workshops and interviews that engage stakeholders, critically in the early stages of QI planning. By applying this process in different ways, existing knowledge is captured in System Impact Models and mobilised towards QI endeavours.

Introduction

Through innovation and decision-making, health professionals collectively shape the health system’s impact on its many stakeholders. However, healthcare system designers do not always engage intentionally with design or systems principles, leading to suboptimal and/or dangerous outcomes.1 Failings like these are the target of interventions that apply systems approaches,2–4 and outcome-based quality improvement (QI) programmes.

There is a growing acceptance that outcomes are a powerful tool for QI,5 6 and for the planning of robust services that deliver value.7 Despite this, there are few tools to help to identify holistic sets of outcomes that are suitably tailored to the context of QI interventions.

Outcomes and quality measures that are prescribed at a corporate level8 represent a strategic mandate, but can fail to represent the context of interventions and ‘need to be translated into locally implementable structure, process and intermediate outcome measures’.9 Even outcome sets that are tailored to particular conditions10 can benefit from further development because, as argued by Nagendran et al,9 an ‘exclusive focus on healthcare outcomes neglects critical dimensions of healthcare performance, such as timeliness, access and efficiency’. Elsewhere, Poksinska et al11 find that other initiatives ‘primarily target internal efficiency, and do not focus on activities to improve patient satisfaction.’ They surmise that ‘value needs to be defined … based on both the knowledge and clinical expertise of care providers, and the preferences and needs of patients’. In order to achieve this, the methods used to define value must engage stakeholders in the identification of locally relevant and holistic sets of outcomes.

Beyond their usefulness in the evaluation of QI, outcomes have the commonly unrealised potential to be used proactively, in system design and QI. This use of outcomes requires an understanding of how they emerge from the often complex system of factors that affects them.12 That is, system designers need to appreciate which factors and potential interventions affect outcomes, and how; and by contextualising outcomes in a model of the system that produces them, they can achieve this. The contextualisation of outcomes also has the advantage of reconciling ‘internal efficiency’ and other process measures with outcome measures.11

This paper describes an approach for defining a shared understanding of outcomes, mapped within the context of a system model. This system analysis approach is based on a sequence of questions, each of which can be posed directly or using methods available in the QI literature. The resulting model provides a means for individuals and teams to act with more cognisance and purpose, improving design and decision-making in QI. This is the premise that underpins the approach presented here, and the value-based decision/design framework within which the approach sits.12

Background to method development: the value modelling project

The outcome identification loop (OIL) and other approaches described in this paper sit within a wider ‘value-based decision-making/design’ framework. This framework was developed as part of a collaborative project between the University of Cambridge’s Engineering Design Centre and Guy’s and St Thomas’ NHS Foundation Trust.12 The goal of the project was to develop an understanding of how models of value and forecasted system behaviour can drive system optimisation.

The wider framework that was developed went beyond the elicitation and mapping of outcomes. It addressed quantitative modelling of value perspectives and considers how they might be reconciled within corporate value models that equitably represent the will of all patients and individuals served by the system. During the course of the project, however, it became clear that the process of developing a system description and identifying coherent sets of contextualised outcomes is of intrinsic value in itself.

The outcomes that are elicited using the approach described here can be used to develop statements of shared purpose and to systematically evaluate QI and other design activities. Furthermore, the system impact models (SIMs) that are created can be used to develop existing improvement activities or identify new ones. On the basis of these benefits, the OIL process and SIM are presented here as valuable tools for QI support.

Identifying ‘outcomes’

‘Outcomes’ can be defined as emergent attributes of a system that may be influenced by intentional design activity in QI initiatives. They are the currency and instrument of value; hypothetically, changes in some set of outcomes could fully describe the value (meaningful impact and desirability) of a given QI activity. In Akinluyi’s ‘value ontology’,12 outcomes are considered to take the form of ‘amounts’ or ‘degrees’ that may be increased or decreased. For example, when considering discharge efficiency,13 rather than describing the event ‘patient is discharged promptly’, the outcome might be articulated as ‘time-to-discharge’. In this framework, outcomes are strictly quantities, despite sometimes being difficult to quantify—for example, ‘long-term satisfaction’ would qualify as an outcome by this definition.

Outcome identification is typically a rather piecemeal process involving primary and secondary research in areas, such as risk assessment, market analysis and product design.

Primary research

Primary research involves synthesising data, observations and predictions contributed from selected participants, based on their knowledge and perspectives of the context. This may be supported using methods, such as reflective practice,14 action learning15 16 and other forms of questioning. It is around a sequence of key questions that the OIL process described in this paper is structured.

Secondary research

Secondary research into outcomes, including product design, often involves designers adapting a priori lists of generic outcomes, such as the National Health Service (NHS) outcomes framework,8 or other standard outcome sets.10 As Nagendran et al9 highlight in their critique of this approach, “[this] framework will need considerable ‘deconstruction’ in its application locally into actionable structure and process measures that will support quality and outcomes improvement”. The tools presented in this paper support this local adaptation.

Overview of support: the SIM and supporting tools

An ‘impact model’ describes a network of outcomes that act as influences within a given situation, problem or opportunity. It is a ‘directed graph’ whose vertices represent outcomes and whose arrows show causation. Blessing et al17 distinguish between ‘reference’ and ‘impact’ models, as describing understanding about a situation as-is or as-desired, respectively. The impact models described here combine the two; embodying both the current situation, and the introduction of intended changes in QI.

In a complete SIM, a boundary is drawn around outcomes that are of interest from a stakeholder’s perspective in a particular QI initiative or system design activity (see figure 1 for a generic example). These particular outcomes form the basis of the value model.

Figure 1

A generic picture of a system impact model. QI, quality improvement.

The development of the SIM is supported by tools and methods presented in this paper. These include the OIL process for building impact models, and criteria for setting the system boundary—that is, determining the set of outcomes used for the value model. Figure 2 illustrates how these methods can be embedded within the design of a workshop for eliciting outcomes and building a SIM. Both the methods and the overarching process are described in subsequent sections.

Figure 2

A suggested outcome identification loop (OIL) workshop design. SIM, system impact model.

The OIL process for building impact models

Initiation

It is useful to seed discussion around prior statements of purpose. This can be done by simply asking, ‘What are we hoping to achieve (through the situation being considered)?’, and gathering the resulting statements of core purpose. This can be an engaging way to open a workshop, particularly if starting with real-life scenarios. The resulting statements of core purpose are then articulated as outcomes—that is, in the form of attributes that may be increased or decreased. The translation of process measures (and other types of goals that participants tend to identify) into outcomes may be supported by methods developed by Akinluyi.12

In recording the initial set of outcomes, it is important to observe the left–right convention of the impact diagram.12 The initial set of outcomes are drawn towards the right-hand side of the impact model and arrows generally point from left to right such that outcomes relating to the objectives of QI are placed towards the right. This convention serves a practical purpose in improving clarity and helping to organise ideas as they are generated. Additionally, the overall direction of the diagram reflects the design-orientated intent of the QI programme.

Iteration of the OIL process

The OIL questions

The knowledge and understanding required to produce the impact model lie with those people who interact with it. In healthcare, this means patients, staff, managers and others. ‘Mapping out’ the system relies on asking participants the right questions in the right way, or, alternatively, asking these questions in the thematic analysis of transcript data, reviews and other sources of insight.18–20

The questions that form the outcome identification process described here have emerged from the analysis and abstraction of existing methods,21–37 that define systems in terms of outcomes, stakeholders and their analogues (these methods are reframed within the OIL process, as described later). The questions are:

The stakeholder identification question. Referring to an outcome:‘Who (else) might this affect?’.

The stakeholder impact question. Referring to a stakeholder:‘What (else) might affect them?’.

The system analysis/design question. Referring to an outcome:‘What (else) might affect this?’.

The consequence (intent/risk) question. Referring to an outcome:‘What (else) might this affect?’.

Rather than attempting to consider the system as a whole, these questions subdivide the process, focusing participants on each outcome or stakeholder in sequence. Focus can be further subdivided by emphasising analysis or intent in each question. For example, question 3 can be posed as either a design question: ‘How might we affect this?’ or as a system analysis query: ‘What do we know to affect this?’. Similarly, question 4 can elicit intended consequences alongside perceived risks. Both intent and analysis are valid in the impact diagram.

As these questions are answered, outcomes and relationships may be added to the impact model and stakeholders can be identified. While question 3 elicits antecedent outcomes (branching to the left on the impact model) and question 4 elicits consequent outcomes (drawn to the right), questions 1 and 2 help to identify accompanying outcomes. Figure 3 illustrates how these questions can be used to grow an impact diagram from any given starting point.

Figure 3

Impact model proliferation during questioning.

Concatenating these questions in iteration

At any point, at least one of these questions can be applied to the content of the nascent impact diagram. For a more balanced and coherent process, it is possible to concatenate the questions (as illustrated in figure 4). Akinluyi determined that by answering these questions in sequence, considerations of the system and its stakeholders could be interlaced, unifying multiple outcome-identification activities within one process.12

Figure 4

The outcome identification loop—Adapted from Akinluyi’s outcome identification process12 (with permission).

Using OIL to frame the use of other techniques

The OIL questions may simply be posed to the participant(s) as they are. To enrich the process, however, one can use them as a basis to select and integrate techniques that pose questions in different ways.

Questions 1 (‘Who (else) might this affect?’) and 2 (‘What (else) might affect them?’) are a paired abstraction of core questions from stakeholder identification and analysis techniques. Literature on this topic in several fields emphasises both its importance21 22 and its need for improvement.23 24 Methods to support these questions include participatory techniques, such as contextual inquiry,25 26 and visualisation techniques, such as user stories or personas.27 Stakeholders are otherwise commonly identified through ‘brainstorming’—a process facilitated here by the piecewise consideration of a system’s outcomes rather than the less-tractable picture of the overall system.

The checklist is another popular tool for stakeholder identification. A priori lists of stakeholder groups can be adopted or adapted to generate more specific lists. Some QI approaches are highly focussed on particular stakeholder groups—consider, for example, singular emphasis on ‘the customer’ in Lean thinking and analogously on ‘the patient’ in some value-based healthcare approaches.28 Leviton and Melichar,22 however, ‘urge QI evaluators to think more broadly about the stakeholders they consult’. Less-constrained approaches like theirs or that of Concannon et al29 support more complete system exploration.

Question 3 (‘What (else) might affect this?’) relates to system analysis and design. It performs a function analogous to that of the ‘(solution-neutral) problem statement’30 which seeds many design exercises. This question is also the basis of the ‘driver diagram’ and its analogues.31 32 Within an impact model, the content of these diagrams can be combined with chains of outcomes elicited using analysis methods, such as process modelling,33 and root cause analysis.34 35

Question 4 (‘What (else) might this affect?’) can be answered using analysis methods, including those intended for proactive hazard/risk analysis, such as Failure Modes and Effects Analysis and the ‘Structured What if Technique’.36 37 However, it is important to additionally consider desirable-yet-unintended consequences, which some of these techniques are less attuned to.

Filtering, converging, iterating and stopping

As figure 4’s loop implies, it is possible to continue elaborating the impact model almost indefinitely—stopping criteria are required. As a minimum, the process should ensure that stakeholders and outcomes of the greatest concern have been identified. The impact model should contain, on the left-hand side, those outcomes directly influenceable through QI intervention and on the right-hand side, outcomes experienced directly by stakeholders. Beyond this, OIL can be used to elaborate the impact model and test its validity until the end user judges it to be sufficient.

To focus the OIL process on the outcomes of greatest concern, it is important to prioritise the content of the impact model periodically, filtering out unnecessary detail. This means assessing the relative importance of outcomes as they are identified. Outcome valuation in complex systems is not a trivial task and it is subject to the limitations of ‘bounded rationality’.38 To compensate for this, it might be necessary to revisit and update impact models in multiple outcome-identification exercises or workshops. It is possible to accommodate impact model iterations within QI cycles, such as Plan-Do-Study-Act iterations.39

Completing the SIM: criteria for setting the system boundary

Healthcare is complex, with outcomes emerging in different contexts, to various stakeholders. In the UK NHS, for example, where services are free at the point of access: costs, clinical outcomes and patient experience might appear quite removed from one another, but they all constitute value.

While some outcomes emerge within a system as intermediate factors and others emerge coincidentally, the most meaningful outcomes are those which we purposefully deliver or avoid. They are located at the boundary of the system being designed in QI and collectively model the value of the system to a given stakeholder group. As shown in figure 5, boundary definition in an impact model literally involves connecting sets of these ‘boundary outcomes’ of interest. This process completes the SIM for a stakeholder group.

Figure 5

Flexibility in boundary outcome combination in system impact modelling. QI, quality improvement.

As described by Akinluyi,12 boundary outcomes are ideally:

Directly experienced by stakeholders.

Mutually exclusive; collectively exhaustive.40

Quantifiable.

Commensurable.

Boundary outcomes are directly experienced by particular stakeholder groups and different groups might require separate boundaries to reflect their interests. This underlines the importance of including and prioritising among stakeholder groups with an interest in the outcomes of a QI exercise.

Sets of boundary outcomes that are mutually exclusive are easier to resolve from one another—though this is not a strict requirement, it makes value models easier to understand. It is more important, however, that boundary outcomes should be collectively exhaustive, in the sense that they should fully account at some level for the impact of the QI initiative. It is worthwhile checking an impact diagram to ensure that meaningful outcomes have not been omitted. In figure 5, collective exhaustiveness is represented by boundaries that span the impact model from top to bottom, though this may depend on the arrangement of outcomes in the diagram.

For quantitative value modelling, or in QI monitoring, the outcomes that affect stakeholders are ideally directly quantifiable and also comparable with one another in terms of their desirability (ie, commensurability41). Boundary outcomes may not always be measurable directly, however, and this can be problematic in some QI approaches. An alternative approach, and one that is advocated here, is to strictly maintain that boundary outcomes are experienced by stakeholders, whether measurable or not, and that where an outcome cannot be measured directly, associated outcomes are used as a proxy. These proxy outcome measures can be seen as ‘indicators’ of the boundary outcome and can then be interpreted more honestly in the context of the SIM.

As a general rule, outcomes closer to the left-hand side of the impact diagram are more measurable. They are also relatively context-specific and less generalisable. Conversely, outcomes on the right-hand side are more relevant to stakeholders and support more general, transferrable models of value linked to corporate shared purpose; it is here that the system boundary lies. While it also helps if these right-hand side outcomes are mutually exclusive, quantifiable, measurable and commensurable, it is more important that the value model retains its overall integrity and relevance.

Applying the SIM

The primary purpose of the SIM is to identify outcomes that form objectives in QI. Subsequently, these outcomes form the building blocks of more sophisticated value models and QI evaluation criteria.12

The diagram also captures valuable strategic information that can be used when initiating QI. For example, if a diagram exposes several factors that systematically confound the delivery of value to patients or other stakeholders, this can trigger a QI intervention to mitigate this effect. Where QI projects are initiated to address a particular boundary outcome, such as ‘reducing number of attendances’,42 the SIM provides a repository of knowledge about current and future mechanisms for effecting this change.

Another major benefit of the SIM is that it contextualises operational activity. As well as giving participants an opportunity to be heard, the SIM engages staff by demonstrating their contribution to the system. This is the case in the example produced by a radiation safety team described in box 1 and shown in figure 6.

Figure 6

System impact model produced by a radiation safety team.

A 90 min workshop was conducted with nine members of the radiation safety team at Guy’s and St Thomas’ NHS Foundation Trust.12 This team is a group of healthcare scientists (specifically, physicists) whose work includes validation, optimisation and risk management in X-ray diagnosis and treatment. Here, participants developed a SIM (figure 6) to initiate quality improvement, while also testing the validity and usability of OIL.

Although only the example of box 1 is presented here, in subsequent adaptations of this method, similar benefits have been noted. SIMs have been produced in workshops at Guy’s and St Thomas’ NHS Foundation Trust to explore the impact of its clinical engineering service and, in other work, to develop a strategy to improve nursing staff retention. SIMs have also been produced in workshops with King’s Health Partners’ ‘Education Academy’ to explore the impact of training and development initiatives.

Workshop overview

The SIM was seeded using statements of core purpose that the participants had produced: they initially identified ‘(increase) image quality’, ‘(lower) patient dose’ and ‘(lower) staff dose’, alongside a list of activities that nominally fall within the remit of the radiation safety team (see left-hand side, figure 6).

Participants were then divided into small groups undertaking one of two tasks: some used driver diagrams and other methods to identify antecedent outcomes, and others used communication diagrams to identify consequent outcomes. Antecedent outcomes were elicited during a discussion about encounters with patients and staff.

Insights derived immediately from OIL and the SIM

‘Patient assurance’ was identified as a key outcome during the course of the workshop—and this triggered a discussion about how ‘quality of patient information’ might be the key to addressing patients’ concerns. This exchange demonstrated that, even in the initial draft, the SIM could be used to inspire new QI interventions.

The SIM also highlighted some dependencies that the team decided they would like to better understand or influence. For example, one senior physicist made the observation:

‘We would like to think that by adding quality control processes, we optimise patient (radiation) dose. Actually, it’s the radiographers that can apply or ignore our recommendations, and so there’s a big dependency there—it’s important that we let the radiographers know why changes are being made. The diagram perhaps shows the complexity that we have to deal with.’

This comment also reflects the benefit of considering a range of stakeholder outcome boundaries.

Analysis of the SIM

Figure 6’s SIM illustrates the fact that the team is driven, to a degree, by high-level organisational goals. It is also apparent that their service impacts patients in numerous ways that often depend on how they engage and interact with internal stakeholders, such as radiologists and radiographers. Reconciling stakeholder perspectives with an overarching mission can help to identify opportunities for QI through improved collaboration—as was exemplified here, in the discussion on patient information.

Reception of the SIM

The utility of the SIM was positively received:

‘This representation is like a Swiss-army knife—we can work backwards from the right side to understand how we can make changes … starting on the left, we can think about what we’re doing now’ (where ‘the right side’ corresponds to the organisational outcome boundary in figure 6 and ‘the left side’ corresponds to radiation safety interventions).

The process empowered the team to challenge what they were doing and to identify opportunities to strengthen the evidence base for their decisions and improve outcomes (by identifying poorly understood relationships within the SIM). Although it could stimulate reflection in this way, the SIM did, however, lack detail in its representation of causal relationships between outcomes:

‘It would have been useful to apply some sort of weighting system to know the strengths of these factors’.

The quantitative analysis of outcomes features in the wider value-modelling framework by Akinluyi,12 but was decidedly excluded from the scope of the SIM—however, it was acknowledged that it is possible and sensible to add more information to SIMs in future, by at least assigning +/− signs (to indicate whether causal relationships are incremental/decremental, respectively).

A major benefit perceived by participants was that the diagram linked procedure to purpose, moving their focus away from the operational task and desk/workshop, towards the interests of, and interaction with, other stakeholders. The process of constructing the SIM helped them to articulate their contribution.

All of this was achieved by asking the OIL questions in such as a way as to access and mobilise the considerable knowledge of the team.

Discussion

As set out by Clarkson et al in their report Engineering Better Care, engineers ‘routinely use a systems approach … they consider the layout of the system, defining all the elements and interconnections’.4 The purpose of the SIM and methods presented here is to enable healthcare professionals, service users and others involved in QI to do just this. In doing so, this approach can engage participants, giving them ownership43 and making the most of their insights, to enhance initiatives.

The SIM simultaneously presents potential QI interventions, contextual factors and hazards, consequences (intentional and unintentional) and the relationships that connect them. In doing so, a SIM reconciles knowledge of how things are with the intent to influence outcomes. It also reconciles process measures (as intermediate outcomes) with outcome measures,9 providing context for their interpretation within the wider system. The questions of the OIL process provide a vehicle for existing methodologies, such as the contextual inquiry and25 26 techniques of experience-based (co-)design,44 and the SIM provides a repository for the resulting insight.

Perhaps the main shortcoming of the process described in this paper is the effort required to transcribe the contributions of many participants or groups into a single SIM. In the example in box 1, the group size of nine was small enough, and the participants’ specialisms were homogenous enough for the workshop to take the form of a single, coordinated discussion—more so because the topic had a narrow focus on just one area of the radiation safety team’s work. This meant that the proliferation of the impact model (as per figure 3) could be captured directly during the workshop. In subsequent work, the author conducted larger, multidisciplinary workshops (eg, workshops with King’s Health Partners’ ‘Education Academy’ to explore the impact of training and development initiatives) where multiple impact models had to be reconciled and combined post hoc. This combination process, though enlightening, is intensive and doesn’t scale easily.

During development, these diagrams can appear complex or chaotic, but the authors would maintain that final versions need be no more chaotic than any situation they model. Even so, patterns emerge within these diagrams according to the left–right convention of SIMs and the specific considerations of risk, design and stakeholder perspectives.

Whereas outcomes towards the left-hand side contain information about ‘what we could/will do’, they are soon joined with outcomes and relationships that communicate ‘how things work’—the realities, hazards and opportunities of a QI intervention. The combination of these views emerges towards the right-hand side of the SIM in higher-level considerations that are transferrable and relevant within organisations; these ‘bottom-line’ outcomes connect more closely to the shared purpose than those application-specific outcomes found towards the left-hand side (as exemplified in figure 6).

For simplicity, the outcome-identification process described here only requires that the arrows signify the direction of effects. It is possible to add, yet, more information to SIMs: by assigning +/− signs, weightings or other descriptors to indicate the strength of causal relationships; or by modelling complex non-linear relationships between outcomes; or by modelling feedback from outcomes to causes.12 Such approaches are likely to refine the link between outcome and value, guiding investment in system improvement.

Conclusions

The way we define, select and fixate on outcomes affects the way we shape services systematically through our decisions and designs, and this can be instrumental in any pursuit of strategic clarity and coherence. The use of SIMs, the OIL process and other aspects of the methodology described here guide the identification of outcomes and value in a way that is entirely compatible with a ‘true systems approach … successfully integrating people, systems, design and risk perspectives in an ordered and well-executed manner’.4