Applying a logical theory of change for strengthening research uptake in policy: a case study of the Evidence Informed Decision Making Network of the Caribbean.

OBJECTIVE: Health policymakers in the Caribbean face challenges with research use in decision-making. Although copious approaches to strengthen evidence-informed policy can be found in the literature, these strategies should be applied and evaluated in specific settings. We developed a theory of change for strengthening research uptake in health policy, and the interventions were implemented as the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe). We assessed the model's logic and evaluated whether the expected outcome was achieved.
METHODS: The model was mapped in three stages: problem identification; goal determination; and backward linking of interventions. Beneficiaries were surveyed to assess the design logic and to evaluate the main outcome.
RESULTS: A total of 137 respondents completed evaluation questionnaires. The inclusion of evidence briefs, stakeholder dialogues, a research database, and training programs for policymakers in the model was validated. Respondents also reported their intention to act on research evidence to which they were exposed. After respondents had participated in stakeholder dialogues, the mean intention-to-use score was 6.4 on a scale of 1 (strongly disagree) to 7 (strongly agree), and 6.3 on the same scale, after exposure to training.
CONCLUSIONS: This work provides initial validation of EvIDeNCe as a consolidated strategy to strengthen the application of research in policy in the Caribbean. To our knowledge, it is the first study to develop and apply a comprehensive model of this type to the Caribbean. The findings support results from similar initiatives in other countries, but additional work is needed to evaluate the overall impact of the initiative.

Evidence-informed health policy aims to utilize the best available research evidence to inform policy-making (1). This approach gained increased attention globally after publication of the World Health Organization's seminal report on strengthening health research systems in 2004 (2). The real-world practice of evidence-informed policy-making is, however, less than optimal, with many reasons advanced for persistence of the research-policy gap in health systems (3–7). In the Caribbean region, these explanations include poor access to published literature; limited capacity for knowledge translation and dissemination; and incongruity between the type of research that is produced and the type required for decision-making (8–10).

Many approaches have been suggested for strengthening the application of research in health decision-making, with interventions having varied effectiveness, depending on the context in which they are applied (11–14). Models to guide country selection of strategies also exist. For example, one makes a distinction among four types of efforts: 1) those by research users to pull for evidence; 2) those made by intermediary groups to push evidence towards users; 3) efforts to facilitate user pull; and 4) efforts to engage the producers and users of research in collaborative activities (15). This model has been implemented as an integrated strategy in settings such as Canada (16), Europe (17), Africa (18–20), and Latin America (21). Johnson and Lavis (22) developed a common logical framework for evaluating implementation of this strategy in low- and middle-income countries. They noted, however, that more needs to be known about the interrelationships in the framework, specifically how different activities and outputs can be combined, and how the framework can be applied in different jurisdictional and institutional contexts. This paper seeks to address this knowledge gap.

The Caribbean Public Health Agency (CARPHA) is a regional public health institution serving 24 Member States in the Caribbean. Public policymakers are the primary clients of CARPHA. One of the challenges faced by these policymakers is research utilization in policy. A framework to explain the mechanisms of change expected to result from implementation of interventions to address this challenge was needed. According to Breuer and colleagues (23), logical frameworks are limited in their ability to explain how various components of a program work together in a causal manner to achieve impact. Theory of change (ToC), on the other hand, is a more flexible format. ToC describes how a program, through a logical sequence of intermediate outcomes, can bring about specific long-term outcomes. The pathways through which interventions work to achieve a desired impact are made explicit. ToC also allows for depiction of multiple pathways and feedback loops, which is more reflective of the complexities involved in real-world settings (23, 24). The objectives of the study reported in this paper were: i) to develop a ToC to narrow the research-policy gap in the Caribbean; ii) to assess its design logic; and iii) to evaluate the extent to which the main expected outcome was achieved.

METHODS

Development of the theory of change

Using the approach suggested by the World Bank (25) and Morra and Rist (26), we constructed the ToC using the following steps: i) identification of long-term goals; ii) backward mapping and connecting the preconditions to achieve goals; and iii) identifying interventions to generate desired change. Prior to goal determination, we also included a problem identification step, to ensure that the model's expected results were directly linked to the problems faced by Caribbean policymakers.

To identify long-term goals and map preconditions, we searched for published and gray literature that investigated application of evidence in decision-making in the Caribbean. Two pertinent studies were identified (8, 9). Seven challenges were common across these studies: 1) access to research was not systematic or widespread; 2) the dissemination of research findings to users was less than ideal; 3) the type of research that was being conducted was not the type needed to inform decisions; 4) national planning and policy-making processes provided limited avenues for the systematic review of evidence; 5) there was little interaction between producers and users of research; 6) research had a low standing in the list of factors deemed important to decisionmakers; and 7) policymakers generally lacked expertise to apply research in decision-making. We therefore posited that three broad changes were needed to address these shortcomings: i) relevant evidence should be made available to policymakers when needed; ii) the skills of policymakers to apply evidence in the policy-making process should be enhanced; and iii) policymakers and other knowledge users who could inform health system changes should be well equipped to make, or to advocate for, these changes. We further hypothesized that these adjustments would stimulate knowledge users to act on the available evidence. Interventions that could generate these outcomes were then identified.

Selection of interventions to generate desired change was informed by analysis of published literature on the topic. We found that providing research databases was a recognized strategy to facilitate user pull, as these resources assisted policymakers to efficiently find research and increased the likelihood of research use (27–29). A promising push strategy to answer policymakers’ questions was summarizing evidence and highlighting messages from systematic reviews (28, 30, 31). We found that deliberative dialogues were used to foster knowledge exchange between policymakers and researchers (30, 32–34). Communities of practice (18, 35) and capacity- building programs (36–38) were also used in several settings to support users in their pull for research. Based on these findings, the following five interventions were identified as appropriate for inclusion in the model: i) an electronic library of clinical, population health, and health systems research (the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe) portal); ii) preparation and dissemination of research syntheses (evidence briefs); iii) deliberative dialogues using the best available research evidence (stakeholder dialogues); iv) CARPHA-sponsored training courses for policymakers; and v) a community of practice for evidence-informed policy.

Assessment of design logic and evaluation of expected outcome

Four of the interventions in the initiative were implemented by CARPHA during 2014–2017. Due to resource constraints, start-up of the community of practice was not realized.

We surveyed beneficiaries exposed to each intervention, to collect data about the plausibility of the hypothesized relationships in the model. We also evaluated whether two interventions (stakeholder dialogues and training) achieved the main anticipated outcome, that is, the intention to act on presented evidence.

For the EvIDeNCe portal survey, beneficiaries were purposefully selected from CARPHA's stakeholders who reflected the portal's main target audiences (i.e., policymakers at ministries of health, CARPHA staff, professionals in health-related civil society organizations, and researchers). They were sent a link to an online survey, which included questions about the value of the portal, the usefulness of its search features in assisting users to build a search relevant to their needs, and users’ perception of its ability to improve job efficiency and effectiveness. Feedback was measured on a 5-point Likert scale. Development, pilot testing, and refinement of the data collection instrument was done under guidance of the EvIDeNCe portal project team, which consisted of technical staff from the Pan American Health Organization/World Health Organization (PAHO/WHO), the University of the West Indies, and CARPHA.

For surveys on evidence briefs and stakeholder dialogues, we followed the methodology used by Moat et al. (30) and Lavis et al. (39). Three stakeholder dialogues were convened using evidence briefs that were circulated to participants approximately two weeks in advance of the dialogue. Each dialogue included 17 to 19 participants, chosen from policy, research, and other stakeholder communities. A steering committee comprised of policymakers, researchers, and other stakeholders selected the participants based on two criteria. The first was that they would be able to articulate views and experiences about the issue under discussion, while constructively engaging with others. The second was that they would be able to champion, within their respective constituencies, actions to address the policy challenges.

All participants attending stakeholder dialogues received evidence briefs and evidence brief evaluation questionnaires prior to the dialogue. They also received stakeholder dialogue evaluation questionnaires at the end of each deliberation. We used the instruments developed and tested by Moat et al. (30) and Lavis et al. (39). Three types of questions were included: i) how well beneficiaries felt briefs or dialogues achieved their intended purpose; ii) how helpful the beneficiaries found the design characteristics of these two mechanisms; and iii) a question about beneficiaries’ intention to use the evidence presented. All responses were measured on a 7-point Likert scale.

For the training component, CARPHA Member States were invited to nominate individuals to participate in a combined online and face-to-face training workshop. Eleven individuals whose main responsibility was health policy and planning were nominated. All beneficiaries of the training received evaluation questionnaires that rated: i) their overall assessment of the course; ii) whether the course enhanced their skills in finding and using research evidence; and iii) their intention to use the type of research presented during the training. Responses were measured on a 7-point Likert scale.

Data analysis

In total, 164 respondents received evaluation questionnaires: 49 evidence briefs; 50 EvIDeNCe portal; 11 training evaluation; and 54 stakeholder dialogue. The respective response rates were 85.7%, 72.0%, 100.0%, and 88.9%.

We calculated the mean scores on the Likert scales for all questions. To assess the model's design logic, we analyzed responses to questions about the intended objectives of each intervention. Since the EvIDeNCe portal was expected to increase availability of relevant evidence, we analyzed mean scores for five questions about the usefulness of the portal's search features in helping to build a suitable search. Evidence briefs were expected to increase availability of relevant evidence, and to contribute to stakeholder dialogues. Beneficiaries’ feedback about how well the briefs achieved this purpose was examined. We also considered views about the design features of the briefs. Similarly, stakeholder dialogues were expected to equip participants to make or to advocate for evidence-informed change. How well the dialogue supported a full discussion of issues was considered, along with feedback about the dialogue's design features. Finally, the training intervention was intended to enhance participants’ knowledge and skills. Responses to three questions about finding and applying evidence in policy-making were investigated.

The key expected outcome from implementation of the EvIDeNCe initiative was the intention to act. We evaluated whether this was achieved after participation in stakeholder dialogues and exposure to training, by examining the responses to the “intention to use” question. For stakeholder dialogues, we also compared the mean intention-to-use scores, for public policymakers working in government and in regional agencies.

RESULTS

The ToC diagram (Figure 1) shows the relationships among inputs, activities, and outputs for each intervention, and across interventions. These interventions were collectively implemented by CARPHA, under the umbrella of the EvIDeNCe. Table 1 summarizes the survey results.

FIGURE 1

Theory of change model for bridging the research-policy gap in the Caribbean, depicting the relationships in the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe) initiative.

TABLE 1

Beneficiaries’ views of interventions in the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe) initiative, implemented in 2015–2017, with Likert-scale ratings (mean score and standard deviation (SD))

Strategy			Mean score (SD)
EvIDeNCe portal (n = 36)
	Usefulness of advanced search features in helping to build a search suited to users’ needsa
		Type of document	4.1 (1.2)
		Country focus	4.3 (1.3)
		Author location	3.6 (1.3)
		Publication year	4.1 (1.2)
		Full text availability	4.4 (1.2)
		Language	4.5 (1.1)
	As a tool to enhance effectiveness on the jobb		4.3 (1.0)
	As a tool to improve job efficiency (make it easier to do job)b		4.1 (0.9)
	Would use the portal frequentlyb		3.8 (0.9)
Evidence briefs (n = 42)
	As a product to present available research evidence on a priority issue and to inform a stakeholder dialoguec		6.6 (0.7)
	Design features of the evidence briefd
	Describing the context for the health issue		6.6 (0.9)
		Describing different features of the problem	6.3 (1.0)
		Describing options for addressing the issue	6.3 (1.1)
		Describing what is known based on synthesized research evidence	6.6 (0.6)
		Describing implementation considerations	6.5 (0.7)
		Employ systematic and transparent methods to identify, select, and assess and synthesize research evidence	6.3 (0.9)
		Did not conclude with recommendations	5.7 (1.3)
		Employed a graded-entry format	6.5 (0.8)
		Included a reference list	6.8 (0.6)
		Was subjected to a merit review	6.2 (1.3)
Stakeholder dialogues (n = 48)
	As a mechanism to support a full discussion of priority health issues, including research evidencec		6.6 (0.6)
	Design features of the stakeholder dialogued
		Addressed a high-priority policy issue	6.7 (0.5)
		Provided opportunity to discuss different features of the problem, including (where possible) how it affects particular groups	6.4 (0.8)
		Provided opportunity to discuss three options for addressing the problem	6.5 (0.7)
		Provided opportunity to discuss key implementation considerations	6.5 (0.8)
		Informed by a precirculated evidence brief	6.7 (0.6)
		Informed by discussion about a full range of factors that can inform how to approach a problem	6.5 (0.7)
		Brought together many parties who could be involved in or affected by future decisions related to the issue	6.6 (0.6)
		Engaged a facilitator to assist with the deliberations	6.9 (0.3)
		Allowed for frank, off-the-record deliberations by following the Chatham House Rule	6.7 (0.8)
		Did not aim for consensus	6.2 (1.1)
	Intention to use evidence from the dialogue for briefings, advocacy, and decision-makinge (all participants)		6.4 (0.6)
	Intention to use evidence, by role category
		Policymaker in government	6.6 (0.5)
		Policymaker in regional agency	6.6 (0.5)
Training (n = 11)
	Overall assessment of trainingf		5.9 (0.5)
	As a mechanism to enhance professional skillse
		Enhanced knowledge of appropriate sources of key types of research evidence	6.7 (0.5)
		Enhanced skills in searching appropriate sources of research evidence	6.3 (0.8)
		Enhanced skills in finding and using research evidence to clarify a problem and its causes	6.3 (0.8)
		Intention to use the type of research evidence discussed in the traininge	6.3 (0.8)

On a scale of 1 (very unhelpful) to 5 (very helpful).

On a scale of 1 (strongly disagree) to 5 (strongly agree).

On a scale of 1 (fail) to 7 (achieved).

On a scale of 1 (very unhelpful) to 7 (very helpful).

On a scale of 1 (strongly disagree) to 7 (strongly agree).

On a scale from 1 (very poor) to 7 (excellent).

All interventions were assessed in relation to their intended objectives. Evidence briefs achieved the purpose as a product to present available research evidence, and for use in stakeholder dialogues. The mean score was 6.6 on a scale of 1 (fail) to 7 (achieved). These briefs summarized findings from systematic reviews and Caribbean research, contextualized the evidence by outlining the political and socioeconomic relevance, and included considerations for implementation to guide users. These characteristics were among the highest rated by the beneficiaries, in terms of their helpfulness. Stakeholder dialogues, when combined with evidence briefs, achieved the purpose as a mechanism to support a full discussion of policy issues. The mean score was 6.6 on a scale of 1 (fail) to 7 (achieved). Providing an opportunity for frank deliberation and engaging a facilitator to assist with the deliberation were prominent features contributing to the usefulness of this mechanism. Beneficiaries agreed that the EvIDeNCe portal would enhance job effectiveness and efficiency. Three key features of the portal, designed to assist beneficiaries to build a relevant search, received the highest ratings: i) the ability to search for research in different languages spoken in the Caribbean; ii) the availability of full text; and iii) the ability to narrow a search by Caribbean country of interest. Policymakers also agreed that the training course enhanced knowledge of sources of research evidence, with a mean score of 6.7 on scale of 1 (strongly disagree) to 7 (strongly agree). The course also enhanced skills in searching for research, with a mean score of 6.3 on the same scale.

Beneficiaries’ intention to act on presented research was a key expected outcome. Stakeholder dialogues resulted in the production of three outputs (a dialogue summary, a topic overview, and a commitment-to-act form), which included clear statements of the participants’ intention to use the evidence in policy briefings, policy decisions, or advocacy. (These qualitative findings are not reported here.) The intention to act was also reported quantitatively. Overall, the mean response for all dialogue participants was 6.4 on a scale of 1 (strongly disagree) to 7 (strongly agree). Participants who were policymakers in government or in regional agencies reported a slightly higher preference, with a mean score of 6.6 for both groups. Beneficiaries of the training course also signaled their intention to act on the research evidence to which they were exposed, with an overall mean score of 6.3 on a scale of 1 (strongly disagree) to 7 (strongly agree).

DISCUSSION

This study has shown that the barriers faced with research utilization in decision-making in the Caribbean can be precisely addressed through a portfolio of interventions incorporated into a single cohesive program. The logical pathway linking each intervention to the desired change in decision-making was demonstrated. Surveys of beneficiaries validated the inclusion of four of the interventions in the model: i) EvIDeNCe portal research database; ii) evidence briefs; iii) stakeholder dialogues; and iv) training for policymakers. Two interventions (stakeholder dialogues using evidence briefs, and training) achieved the key desired outcome, namely, the intention to act on available research.

The design characteristics of the included interventions were instrumental. Policymakers need context-specific evidence to support their decision-making. The EvIDeNCe portal database allowed users to focus a search on a specific Caribbean country or group of countries and a particular language spoken in the region. These features were viewed by beneficiaries as helpful. Evidence briefs presented evidence on important topics while also considering the political and socioeconomic environment. This, along with the opportunity to hold a frank off-the-record deliberation, was also viewed by beneficiaries as very useful.

The findings of this study are similar to those conducted in other countries. Evaluations of evidence briefs and deliberative dialogues used in Africa (20, 30) and Canada (39) have shown that participants respond in a similar manner when asked about the usefulness of these mechanisms. The integrated approach adopted by the EvIDeNCe initiative is also in alignment with knowledge translation platforms (KTPs), such as the Evidence-Informed Policy Network (EVIPNet), which is implemented in Africa, Europe, and Latin America. The activities included in the EvIDeNCe initiative are similar to those undertaken by EVIPNet teams (20, 21, 40), with the exception of a Rapid Response Service, which was not included in EvIDeNCe. Two main institutional differences can also be observed. Firstly, KTPs are usually hosted in universities, hospitals, or ministries of health (41). The EvIDeNCe initiative is, however, located in a regional public health agency. Secondly, the work of KTPs tends to be country-specific, while CARPHA's EvIDeNCe addresses issues relevant to the health systems across an entire region. It would therefore be important to understand, through future research, how this difference in governance arrangement and activity scope impacts operations and sustainability.

Our study has several strengths. It is the first to develop and apply a comprehensive framework for development and evaluation of interventions to strengthen the use of evidence in decision-making in the Caribbean region. It contributes theoretical and empirical knowledge on the activities and outputs that can be combined to support evidence-informed decision-making in this geographical and institutional context. It links the literature about the specific problems faced in the Caribbean region to program design. By using ToC instead of the logical framework, it depicts how the required change process can be logically brought about through the interrelated elements of the program.

Despite these strengths, the limitations of the work should also be acknowledged. Firstly, the study is a formative assessment that signals the change process possible, using the strategies identified. We did not evaluate all outcomes included in the model. Although the intention to use evidence in decision-making is an important and measurable outcome, some attempt should be made to determine whether the other expected outcomes were also achieved. This would include determining whether the portal actually increased evidence availability, and whether policymakers and stakeholders were indeed better equipped to make evidence-informed decisions. We did not evaluate the consolidated initiative by using a composite data collection instrument. Therefore, until this further evaluative work is done, we are unable to draw conclusions about the overall impact. Secondly, the model rests upon several critical assumptions. For example, we have assumed the absence of access barriers to electronic resources. The validity of these assumptions would have to be verified.

Conclusion

This work developed a theory of change to narrow the research-policy gap in the Caribbean. The model's design logic was validated by beneficiaries, who gave positive ratings to the overall purpose and usefulness of four of the included interventions. The ability of two of these interventions (stakeholder dialogues using evidence briefs, and capacity-building workshops) to contribute to the main desired outcome was also confirmed.

Policy recommendations

While the results are promising, consideration could be given to incorporating a Rapid Response Service in the EvIDeNCe model. Such a service would provide research syntheses to policymakers, at their request, in a relatively short time frame. It was reported in Chile that this service is widely used and highly valued by ministry of health officials (21). However, because the EvIDeNCe initiative serves a region and not a country, the operation of such a service would have to be carefully managed and the required resources made available. One possible approach could be to build the capacity of country teams to support implementation.

A consolidated indicator framework should also be developed to monitor the reality of progress in research utilization in policy. We recommend a high-level framework for use in the Caribbean (Table 2), although it could be applied elsewhere. The indicators in this framework span the four strategic areas included in the ToC model. Monitoring them would require collection of baseline data and intermittent surveys of policymakers. This framework can be implemented by the Caribbean Community Secretariat, as part of the monitoring and evaluation process for the Caribbean Cooperation in Health (CCH). CCH facilitates coordinated action and collaboration among partners for health in the region, and that initiative clearly articulates the desire to enhance regional capacity to generate and use quality evidence for effective policy and programming.

TABLE 2

Recommended consolidated indicator framework for monitoring research uptake in health policy in the Caribbean

Strategy type	Outcomes	Outcome indicator	Impact	Impact indicator
Push strategy	Increased availability of relevant evidence when needed	Proportion of policymakers reporting access to synthesized evidence to answer policy questions	Increased use of evidence in decision-making	Number of health policies and programs developed based on high-quality evidence
Facilitated user pull	Increased availability of relevant evidence when needed	Proportion of policymakers who utilize the EvIDeNCe portal, reporting acquisition of relevant evidence	Increased use of evidence in decision-making	Number of health policies and programs developed based on high-quality evidence
User pull	Policymakers’ skills to find and apply evidence in policy-making process strengthened	Proportion of trained policymakers utilizing available evidence in policy	Increased use of evidence in decision-making	Number of health policies and programs developed based on high-quality evidence
Exchange	Policymakers and stakeholders better equipped to make or advocate for policy change	Proportion of policymakers using the EvIDeNCe portal/participating in dialogues/training who apply available evidence in policy, programming, and advocacy	Increased use of evidence in decision-making	Number of health policies and programs developed based on high-quality evidence