Process evaluation of complex cardiovascular interventions: How to interpret the results of my trial?

Complex interventions of varying degrees of complexity are commonly used and evaluated in cardiovascular nursing and allied professions. Such interventions are increasingly tested using randomized trial designs. However, process evaluations are seldom used to better understand the results of these trials. Process evaluation aims to understand how complex interventions create change by evaluating implementation, mechanisms of impact, and the surrounding context when delivering an intervention. As such, this method can illuminate important mechanisms and clarify variation in results. In this article, process evaluation is described according to the Medical Research Council guidance and its use exemplified through a randomized controlled trial evaluating the effectiveness of a transition program for adolescents with chronic conditions.

Understand the different elements of complex interventions and why effectiveness evaluations alone are not enough to interpret their effects.

Recognize the key features of a process evaluation study and how this can be carried out alongside a trial or experimental study, in order to understand implementation, mechanisms of impact, context, and outcomes.

Describe common issues that can arise during a process evaluation related to the planning, conducting, and reporting phases.

The problem: complex interventions = ‘black boxes’?

Complex interventions are widely used within health services to mitigate health problems at the individual, community, or population level. The Medical Research Council (MRC) describes complex interventions as those that contain several interacting components. However, the level of complexity of these interventions depends on a variety of dimensions, such as the range of possible outcomes, the degree of flexibility needed in delivering the intervention, the number of behaviors needed to deliver or receive the intervention, or the causal pathways leading to the desired or undesired outcomes.[1] Following this definition, it is clear that a wide range of interventions within the cardiovascular field can be considered as complex interventions with a varying degree of complexity.

In the European Journal of Cardiovascular Nursing there have been several studies investigating the effects of complex interventions, such as self-management programs for chronic conditions,[2-4] tele-rehabilitation after hospital discharge,[5,6] complementary methods for symptom management,[7] and interventions utilizing peer support for postoperative recovery.[8] The ultimate goal of these complex interventions is to achieve a meaningful and sustainable change of clinical practice. In order to achieve this goal, the intervention should be reproducible in other contexts and in some cases for other populations. However, reproducibility of complex interventions is possible only when several important factors are met: (a) the intervention must be described in detail, with a clear linkage between program theory and outcomes, (b) the intended or unintended mechanisms of impact must be investigated within the trial, and (c) researchers must investigate and describe how successful implementation of the intervention was achieved within the trial. If researchers neglect to focus on understanding these crucial aspects of how complex interventions are delivered, the conclusions assume that the trial was implemented perfectly.[9] Unfortunately, how a complex intervention was delivered within a trial is rarely investigated or reported in cardiovascular nursing or in other healthcare fields. Complex interventions therefore remain as ‘black boxes’ in which the active ingredients are unknown.

A solution: process evaluation to shed light on the black box

Process evaluations are essential to understanding how complex interventions work in producing change to a problem under study. Nested within a trial or experimental study, process evaluations can be used to assess fidelity and quality of implementation, clarify mechanisms of impact, and identify contextual factors that are associated with variation in how the intervention is delivered between sites, which has an impact on participant outcomes. Moreover, when trials fail to achieve the intended outcome, a process evaluation can provide knowledge of what in the intervention or where in the delivery process the intervention failed.[10] As an example, a randomized controlled trial (RCT) evaluating a falls prevention program for older people presenting at the emergency department was delivered with ambiguous results. The trial reduced the rate of falls and fractures but not fall-related injuries or hospitalizations. The process evaluation was performed alongside the RCT through audio-recordings of the visits at the emergency department, interviews with healthcare providers delivering the intervention and patients receiving it, and questionnaires concerning providers’ and patients’ adherence to the intervention. The results showed that even though the intervention was delivered according to the program theory, the dose delivered in practice was too low to achieve an impact on all the outcomes of the trial. In this study, the dose was evaluated by quantitatively measuring providers’ adherence to the intervention by using questionnaires and scoring systems in combination with participants’ intervention uptake through questionnaires.[11] Indeed, this example highlights the important role of process evaluation alongside a trial in order to answer pivotal questions about where the pitfalls and barriers are to delivering a successful complex intervention.

A number of different frameworks have been developed to guide researchers in structuring and carrying out a process evaluation.[12,13] However, many of these frameworks lack a clear description of how to actually perform the evaluation and which methods to choose. The MRC guidance on process evaluation from 2015 provides a solution to this problem, stressing three core aspects that need to be evaluated within the process of delivering a complex intervention: implementation, mechanisms of impact and context (Figure 1).[9] Nevertheless, before conducting the actual process evaluation there are certain things that need to be considered. Firstly, the person(s) conducting the process evaluation should be experienced in both qualitative and quantitative research methods. As complex interventions are events within complex social systems, mixed methods are necessary to understand the full extent of how the intervention works.[14] Secondly, the intervention and the components must be described in detail, preferably through a logic model where (anticipated) causal relationships are depicted, along with intended delivery of components.[15] As many previous studies on complex interventions have left out important information on how the intervention was intended to work,[16] a logic model of the intervention is a pivotal part of planning the process evaluation. This model will guide in choosing key objectives of the process evaluation and where the major uncertainties lie.[9] Thirdly, as process evaluations are inherently evaluations of the interventions’ ability to create change, a good relationship between intervention developers and implementers is fundamental. The relationship must allow for close observation, as well as honest feedback, since process evaluations might reveal problems with implementation.[9] As with most qualitative methods, access to the field (i.e. being allowed to observe and take part of the complex intervention’s implementation phase) is fundamental for credibility of the findings.[17] Fourthly, the degree of separation or integration of process and outcome data must be decided. Will the two strands be reported separately or together? Will process data be analyzed before knowing outcome data or after? These issues need to be decided early on and are associated with benefits and drawbacks respectively.[9]

Figure 1.

Features of a process evaluation according to the MRC guidance.[9] Colored boxes represent the core components of the process evaluation, which is informed by the program theory, and inform how to interpret the outcomes of the trial.

When the aforementioned considerations have been reflected and decided upon, designing and conducting the process evaluation can begin. It is not uncommon to feel overwhelmed by the extent to which a process evaluation has grown and therefore abstain from performing it. However, all interventions are different, so not all process evaluations need be extensive. By structuring the data collection around the logic model of the intervention and the three core components described in the MRC guidance (Figure 1), process evaluators can be confident that the most important aspects will be investigated. The most commonly used data collection sources to investigate how implementation is achieved are through qualitative methods such as interviews, observations and audio-recordings. Furthermore, self-reported data such as quantitative questionnaires can provide data on fidelity of delivery.[18] Mechanisms of impact can be investigated by interviewing intervention participants, as well as those responsible for implementation (e.g. healthcare providers). Context, which is the sum of all the social and organizational systems surrounding the intervention, can be investigated through participatory observations of how the intervention is being implemented in practice,[9] or by surveys assessing context.[19] In addition, an assessment of usual care to understand what existing factors affect implementation can in some cases be relevant and provide important data.[20]

Software

It is not uncommon for process evaluations to generate large amounts of both quantitative and qualitative data.[9] As quantitative data from these types of studies are generally analyzed with descriptive and inferential statistics, a common statistical software such as the Statistical Package for Social Sciences (SPSS) covers the general purpose. Qualitative data tends to be extensive, as it is generated from various sources, such as participatory observations, interviews and documents. A suitable computer-assisted qualitative data analysis software, such as NVivo, can be helpful in sorting and analyzing these data in a comprehensive way.

Example of process evaluation: the Stepstones project

Transition programs for adolescents with chronic conditions in transition to adulthood are complex interventions due to their numerous interacting components, organizational levels targeted, and behaviors required from the adolescents to achieve the outcome. The effectiveness of these programs along with the causal mechanisms that lead to empowered and independent individuals in the adult healthcare system are yet to be known.[21] The Stepstones (Swedish Transition Effects Project Supporting Teenagers with chrONic mEdical conditionS) project was established to bring evidence to this knowledge gap. An extensive process evaluation alongside an RCT was developed following the MRC guidance and is currently being carried out. The design of the effectiveness evaluation has been reported in a study protocol.[22] In short, the intervention consists of eight key components delivered in five steps, with the primary outcome being patient empowerment.[23] The research questions for the process evaluation were based on the logic model of the intervention, which was developed through the protocol of intervention mapping.[24] The logic model depicts intended input (i.e. components and implementation steps), output and outcomes, and is described in Figure 2. From that point, data sources to answer these research questions were selected according to each of the components of the MRC guidance: implementation, mechanisms of impact, and context (Figure 3). As seen in Figure 3, multiple data sources are used through a mixed methods approach to capture different aspects of program delivery, which strengthens the methodological rigor of the evaluation. However, one data source (e.g. participatory or non-participatory observations) can be used to assess fidelity (i.e. implementation), participants’ responsiveness (i.e. mechanism of impact) and the surrounding context, which makes this method a cost-effective choice in collecting comprehensive data.[9,25]

Figure 2.

Logic model of the intervention of the Stepstones project.

Figure 3.

Overview of the Stepstones process evaluation components, research questions and data sources.

Qualitative data sources in orange, quantitative in blue.

Source: reproduced with permission from BMJ Publishing Group Ltd.[26]

Reporting process evaluation – a challenge

One of the inevitable challenges of process evaluation studies is the reporting phase. As previously mentioned, process evaluations can generate an abundance of data which might be time consuming to analyze. Furthermore, process evaluations are generally mixed method studies and require researchers with skills in analyzing and integrating insights from both methods. In general, qualitative data are used to shed light on and describe quantitative findings.[9] For instance, in a complex intervention promoting psychosocial well-being following stroke, a mixed methods sequential explanatory design was used. In this study, several components were delivered at a low dose, which were described in absolute numbers and percentages. Qualitative data, through interviews and focus groups, were used to explain reasons and factors that affected the delivery of the components.[27]

It is sometimes favorable to report process evaluation findings in several publications. For instance, the ASSIST project published three articles before knowing outcome data of the effectiveness trial,[28] and several publications afterwards where outcomes were linked to process findings.[9] A challenge with this approach may be unclear linkage between different parts of the process evaluation. To deal with this issue in the Stepstones project, we have published a separate study protocol for the process evaluation connecting the different parts of the overall study.[26]

As process evaluations differ depending on the type of study, several reporting guidelines can be used to assure quality and transparency. Some commonly used reporting guidelines are CReDECI 2,[29] which is built on the MRC framework for development and evaluation of complex interventions,[10] the TIDIeR checklist,[30] and Grant et al.’s framework for cluster-RCTs.[20] Furthermore, since process evaluation is commonly used in implementation science, the StaRI checklist has been developed to enhance the reporting of these studies.[31]

Conclusion

Complex interventions in cardiovascular nursing and allied professions are commonly used to mitigate clinical health problems, but these studies rarely incorporate data on implementation processes and potential mechanisms of impact. Process evaluations are imperative in understanding how complex interventions work in producing the (un-)intended outcomes. The MRC guidance stresses three core components which can be used by researchers in cardiovascular nursing and allied health to design and conduct a mixed-methods process evaluation of a complex intervention: implementation, mechanism of impact, and context. By doing so, the active ingredients of these interventions are made visible, therefore increasing the chance of successfully reproducing complex interventions in other contexts.