Taking a systems approach to explore the impacts and outcomes of a research and evaluation capacity building partnership: a protocol.

INTRODUCTION: Partnership models that bring researchers, policymakers and service providers closer together are gaining traction as a strategy to improve public health practice. Yet, there is little evidence of how these models work, or indeed if they do work. The Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN) is one such model. SiREN is a partnership between researchers, policymakers and service providers that aims to develop the research and evaluation capacity and evidence-informed decision making capability of professionals working to address sexual health and bloodborne virus issues in Western Australia. This study will use a systems approach to identify the mechanisms of action, impacts and outcomes of SiREN and inform the development of evaluation tools. METHODS AND ANALYSIS: Data will be collected from organisational documents, surveys, in-depth interviews and a workshop. It will be analysed using a complex adaptive systems lens and findings will be used to inform the development of a type of qualitative systems model called a causal loop diagram. The causal loop diagram will illustrate the: contextual factors influencing engagement; mechanisms of action; and impacts and outcomes of SiREN. Evaluation tools will then be developed that can be used to assess the indicators identified in the causal loop diagram. ETHICS AND DISSEMINATION: Ethics approval was obtained from the Curtin University Human Research Ethics Committee (approval number: HRE2017-0090). Participants will be free to withdraw from the study at any point and confidentiality will be maintained by de-identifying participant responses in any published or shared data. The findings from this study will be shared in conference presentations, reports, peer-reviewed journals and online through websites and social media. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The use of causal loop diagramming is a novel approach to programme evaluation and this study will explore the practicality and usefulness of this technique to evaluate a partnership model.

A nuanced description of the influence of Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN) on research and evaluation practices will be achieved through using a variety of data collection methods and providing rich descriptions in the causal loop diagram and accompanying textual narrative.

The process of refining the causal loop diagram through a workshop will provide an opportunity for stakeholders to develop a shared understanding of what SiREN can achieve.

Some of the study researchers are involved in the partnership being explored and therefore participants may wish to portray experiences positively and/or avoid focusing on negative aspects.

Data collection will occur up to 24 months after participants engaged with SiREN which may lead to recall bias.

Introduction

Public health challenges benefit from evidence-informed and partnership-based responses.1 2 This has been demonstrated through Australia’s response to HIV/AIDS, where evidence-informed and partnership-based strategies led to far fewer new infections and deaths when compared with other developed countries.3 Despite the successes attributed to this response, similar inroads have not been made across all sexual health and bloodborne virus challenges. In order to achieve national objectives for improvements in health outcomes, public health programme and policies need to be supported by evidence.4

Three critical success factors support evidence-informed decision making in public health. First, the development of research and evaluation capacity is essential to enable the public health workforce to generate evidence on what works.1 Research and evaluation capacity can be defined as the skills, knowledge, commitment and resources to engage in effective and sustained research and evaluation practice.5 6 Second, researchers, service providers and policymakers need to work in partnership to create evidence that is both timely and relevant.7 Third, effective processes to increase sharing of, and access to, evidence are required.8

Acknowledging these factors, the Western Australian Department of Health funded the Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN) project in 2012. SiREN is a partnership between sexual health and bloodborne virus researchers, service providers and policymakers in Western Australia. This project aims to: (1) build the capacity of professionals working to address sexual health and bloodborne virus issues to engage in research, evaluation and evidence-informed decision making; (2) support collaboration between policymakers, researchers and service providers; and (3) raise the profile of sexual health and bloodborne virus concerns that are relevant to Western Australia through building relationships with national research centres and contributing to national research agendas.

SiREN is governed by a project steering group and a management team. The project steering group members are representatives from research, not-for profit, professional and government organisations working to address sexual health and bloodborne virus issues. The project steering group has a strategic role in determining the direction of SiREN and ensure that SiREN’s activities reflect the needs of researchers, service providers and policymakers working to address sexual health and bloodborne viruses. The management team are university-based staff with expertise in research, evaluation, sexual health and bloodborne viruses, who play a strategic and operational role in SiREN.

To achieve its aims, SiREN uses a diverse range of strategies. SiREN facilitates and participates in partnerships between research, government and non-government organisations to apply for competitive grant funding and undertake research and evaluation projects; provides tailored project planning, research and evaluation support; provides web-based resources to support evidence-informed policy and practice; hosts a biennial research and evaluation symposium that brings together those working in sexual health or bloodborne viruses across Australia; and maintains a network (database) of approximately 300 professionals with whom it shares relevant research and evaluation evidence, news, funding opportunities and events via electronic mail. A detailed description of the SiREN model has been published elsewhere.9

Partnerships, like SiREN, are based on the theory that more can be achieved by working together than by working alone.2 Research shows that partnerships between researchers, policy makers and service providers can lead to knowledge and skill acquisition, the increased generation and use of evidence, improvements to service delivery and greater efficiencies in the use of limited health resources.10–14 Despite their potential, there is scant evidence describing the kinds of long-term changes that can be achieved by these partnerships, and of the processes that lead to these changes.15 Evaluations of SiREN conducted 2 and 3 years after its inception established that SiREN had contributed to improving research, evaluation and evidence-informed practice, attitudes, skills and knowledge.9 16 While these short-term impacts are promising, exploration of long-term outcomes and the processes by which these are achieved is important given the potential of partnership models, like SiREN, to support the practice of public health.

Evaluating partnerships is challenging. This is because of their dynamic nature; they adapt and evolve in response to the complex relationships between partners and the contexts in which they are embedded.13 15 17 Furthermore, establishing a causal link between partnership processes and outcomes can be difficult as: partnerships may attempt to bring about change in various areas and across various levels (individual, organisational and system); there can be a long lag time between the establishment of the partnership and outcomes occurring; and outcomes are often influenced by contextual factors outside of the partnership such as organisational, policy and funding environments.14

A systems approach offers a way to overcome some of these evaluation challenges. A systems approach is a way of thinking that brings together components of a bounded system, illuminates the relationships between them and how they interact as part of a whole.18 Traditional approaches to evaluation are very linear, often with direct causal links made between strategies and outcomes without consideration of external factors that may influence outcomes. When systems thinking is applied to evaluation, it considers the programme within the context in which it works with all its variables and relationships, and then refocuses attention on the most relevant points for evaluation purposes.19 20

Aims

The overall objective of this mixed methods study is to use a systems approach to explicate the influence that SiREN has on research, evaluation and evidence-informed decision making practices. This study aims to: (1) identify the impacts and outcomes of SiREN and the mechanisms by which these are achieved using a type of systems map called a causal loop diagram and (2) develop and pilot test evaluation tools to measure the indicators identified through the process of developing the causal loop diagram.

Methods and analysis

This study will have two stages. Stage 1 will involve collecting data from organisational documents, surveys and in-depth-interviews and a workshop to inform the development of the causal loop diagram. The second stage of this study will be the development and testing of evaluation tools that can be used to evaluate SiREN.

Participants

The study will recruit three groups of purposefully selected participants: (1) the entire SiREN member network. This is SiREN’s database of individuals with an interest in sexual health and bloodborne viruses who have voluntarily subscribed to receive regular electronic mail from SiREN containing summaries of the latest evidence, news, grants and events; (2) SiREN partners and service recipients. This group will be individuals who have actively engaged with SiREN at least once in the last 24 months. Active engagement will be defined as: participated in the SiREN project steering group or other advisory groups; partnered with SiREN to apply for research grants, or undertake research or evaluation; or received project planning, evaluation or research support. These participants will be from: government, non-government and research organisations across Australia; and employed in a variety of positions including managers, project officers and researchers; (3) the SiREN management team which consists of university-based staff who are involved in the strategic and operational management of SiREN.

Stage 1: causal loop diagram development

In this stage, a causal loop diagram will be developed. Causal loop diagrams provide a visual representation of systems, including their boundaries, component parts and the relationships within them.21 For the purposes of this research, the bounded system is SiREN and the influence it has had on the thinking and practice of individuals it has interacted with.

A causal loop diagram comprises of variables (system factors) and links (relationships). Links are depicted by arrows and the type of relationship indicated by a plus (positive) or minus (negative) symbol. The system variables and links are matched together to form feedback loops, which tell the story of the relationship between them. Feedback loops can either be reinforcing or balancing. A reinforcing loop represents a change that creates more of the same change, resulting in an increase or decrease in growth.22 Balancing loops illustrate when a change in one direction is balanced by a change in the opposite direction.22

Figure 1 shows an example of a reinforcing and balancing loop. In the reinforcing loop (indicated by an R in the centre of the loop), a service provider engages with SiREN, which increases their motivation to engage in evidence-informed decision making, leading to an increase in evidence-informed decision making and further engagement with SiREN. In the balancing loop (indicated by a B in the centre of the loop), service providers engage in evidence-informed decision making, which increases their workload, and subsequently decreases their available time to engage in further evidence-informed decision making.

Figure 1

An example of a reinforcing loop and a balancing loop.

Data collection to inform the development of the causal loop diagram

Data collection will be undertaken by the lead author. Data will be collected from organisational documents, surveys, in-depth interviews and a workshop and used to inform the development of the causal loop diagram.

SiREN organisational documents will be collected to establish an understanding of the structure, processes, impacts and outcomes of SiREN. These will include documents such as past evaluations, funding agreements and activity reports dating back to SiREN’s inception in 2012.

The SiREN member network (n=204) will be asked by email to complete an online survey (see online supplementary appendix 1) designed using Qualtrics survey-building software.23 The findings from this survey will provide insight into the broader context in which SiREN operates. The survey aims to: understand contextual factors that influence research, evaluation and evidence-informed decision-making, as well as engagement with SiREN, and identify how they have engaged with SiREN and the influence this had on their practice.

Semistructured qualitative interviews will be undertaken with SiREN partners and service recipients (n~35), and the management team (n=5). The interviews will explore the impacts, outcomes and mechanisms of action of SiREN in detail. The interview guide for SiREN partners and service recipients (see online supplementary file 2) is flexible and will be modified based on findings from past interviews and the type of engagement the participant has had with SiREN. The interview will be approximately 1 hour in length and will cover: contextual factors influencing evidence-informed practice, research and evaluation; details of engagement with SiREN; and how and in what ways engagement with SiREN has influenced practice. The interview guide for the management team (see online supplementary file 3) will explore their perspectives of SiREN and how factors such as governance, staffing and resourcing influence the ability of SiREN to achieve its aims.

In-depth interview participants will be recruited via direct email. Interviews will be undertaken face-to-face with Perth-based participants and via webcam or telephone with regional or remote participants. Interviews will be digitally recorded and transcribed verbatim, and checked for accuracy.

Approximately 40 people meet the criteria of actively engaging with SiREN in the last 24 months and are eligible to participate in an in-depth interview. Recruitment will continue until the sample is exhausted or saturation of the findings from data analysis is reached. Saturation is the point where no new information arises from the analysis.24

Data analysis and development of the causal loop diagram

All data collected will be entered into QSR NVivo V.11 data management software25 for analysis. Data will be coded to identify contextual factors that influence research and evaluation practices and engagement with SiREN, mechanisms of action, impacts and outcomes. Following this, a complex adaptive systems lens will be applied to identify tenets common to complex adaptive systems (table 1).26–28 This lens has been chosen as the sexual health and bloodborne virus system in which SiREN operates is a complex adaptive system; it has many actors (individuals, organisations or groups), who are constantly interacting, learning and evolving in response to changes within the system.21 29 Applying a complex adaptive systems lens will enable a more complete representation of how and why system elements influence each other.

Table 1

Elements of complex adaptive systems

Element	Description
Path dependence	History is important. Processes can have similar starting points and lead to different outcomes.
Feedback loops	An output within the system is fed back as an input.
Emergent behaviours	Spontaneous creation of order. Occurs when elements within the system interact to bring about change within the system.
Time delays	Time delays between events occurring within the system.

Once the draft causal loop diagram is developed, SiREN partners and service recipients and the management team will be invited to attend a workshop to refine it. The workshop will be run by two facilitators, one to lead the discussion and a second to observe and record the discussion. Workshop participants and facilitators will work together to clarify: the system variables; the nature of the relationships between them; and if there are any factors that had not been considered. This consensus building process increases the credibility of the causal loop diagram and is similar to approaches taken in other systems modelling studies.29 30

Stage 2: identification and development of evaluation tools

In the second stage of this study, evaluation tools that can measure the process and outcome indicators identified in the causal loop diagram will be identified and developed. To identify existing tools, a rapid review of the literature will be undertaken. A rapid review is a streamlined evidence synthesis approach.31 The indicators identified in the causal loop diagram will be used, along with their synonyms, as search terms to systematically search the literature for existing tools that can measure them. Tools identified through this process will be appraised for design, content and methodological quality. Suitable tools will either be modified or used in their current form.

Where a suitable evaluation tool to measure an indicator cannot be located, a new one will be developed. The process of identifying and developing tools will be undertaken in consultation with the SiREN management team and the SiREN project steering group. They will be asked to comment on practical aspects of the tool such as the implementation processes, usefulness of data collected and level of administrative and respondent burden. The specific aspects of the tools to be developed cannot be described in detail at this stage, as it will depend on the indicators identified. However, it is anticipated that tools may assess factors such as: changes in research and evaluation skills and confidence, increased application of evidence in programme planning and changes to how services and programme are delivered.

To increase content validity, questions for the tools will be primarily sourced from existing tools with predetermined reliability and validity. Once the tools are developed, they will be pilot tested with SiREN partners and service recipients. This will involve participants reviewing the tools and undertaking an online survey using Qualtrics Survey Software.23 The survey will assess participants’ perceptions of the tools, specifically: (1) Usability: Appropriateness of language, ease of use and clarity of content. (2) Acceptability: If the time, resources and skills required to complete the tools were acceptable. (3) Sustainability: The likelihood that the tool will continue to be used and suggestions to improve it for future use. The tools will be refined based on the feedback from these processes; if a major revision is required further testing will be undertaken.32

Patient and public involvement

No patients or members of the public were involved in this study. Input into the methods will be sought through meetings with the SiREN management team and the SiREN project steering group.

Discussion

SiREN is a partnership between researchers, policymakers and service providers that aims to develop the research and evaluation capacity and evidence-informed decision making capability of professionals working to address sexual health and bloodborne virus issues. This study will explore the processes, impacts and outcomes of SiREN and develop and test evaluation tools. This will enhance understanding of how and in what ways partnership approaches support research, evaluation and evidence-informed decision making practices in public health. The findings from this research are intended to provide insight for researchers, service providers and policymakers seeking to establish or evaluate a similar capacity building partnership.

Approaches to evaluating partnerships need to be sensitive to their dynamic nature, the influence of contextual factors, and able to show causal links between partnership processes, impacts and outcomes.13 14 Causal loop diagramming is a qualitative systems modelling technique that is well suited to this as it enables understanding of how elements within the system in which the partnership is embedded interact and lead to change.33 Furthermore, the process of developing the causal loop diagram provides an opportunity for stakeholders to develop a shared understanding of how the partnership functions and what it can achieve which can enhance partnership functioning.33 34 While causal loop diagramming can strengthen an evaluation, it is not without its draw backs. It involves the collection and analysis of large amounts of data which can be time consuming, and as the partnership evolves, the causal loop diagram may need to be updated which will require expertise. Additionally, presenting findings as a causal loop diagram can be difficult for those who are not experienced in the technique to understand.35

While there have been calls for greater application of systems thinking to evaluation, there are limited examples of systems approaches to evaluation in the literature,33 35 particularly for causal loop diagramming. This research will contribute to building understanding of how to use causal loop diagramming for evaluation purposes, as well as provide insight into the usefulness of this approach.

Results from this research will be shared in conference presentations, reports, peer-reviewed journals and online. Results will be made available to those who participated in the study as well as those with an interest in SiREN’s approach.