A framework of evidence-based decision-making in health system management: a best-fit framework synthesis.

BACKGROUND: Scientific evidence is the basis for improving public health; decision-making without sufficient attention to evidence may lead to unpleasant consequences. Despite efforts to create comprehensive guidelines and models for evidence-based decision-making (EBDM), there isn`t any to make the best decisions concerning scarce resources and unlimited needs. The present study aimed to develop a comprehensive applied framework for EBDM.
METHODS: This was a Best-Fit Framework (BFF) synthesis conducted in 2020. A comprehensive systematic review was done via six main databases including PUBMED, Scopus, Web of Science, Science Direct, EMBASE, and ProQuest using related keywords. After the evidence quality appraisal, data were extracted and analyzed via thematic analysis. Results of the thematic analysis and the concepts generated by the research team were then synthesized to achieve the best-fit framework applying Carroll et al. (2013) approach.
RESULTS: Four thousand six hundred thirteen studies were retrieved, and due to the full-text screening of the studies, 17 final articles were selected for extracting the components and steps of EBDM in Health System Management (HSM). After collecting, synthesizing, and categorizing key information, the framework of EBDM in HSM was developed in the form of four general scopes. These comprised inquiring, inspecting, implementing, and integrating, which included 10 main steps and 47 sub-steps.
CONCLUSIONS: The present framework provided a comprehensive guideline that can be well adapted for implementing EBDM in health systems and related organizations especially in underdeveloped and developing countries where there is usually a lag in updating and applying evidence in their decision-making process. In addition, this framework by providing a complete, well-detailed, and the sequential process can be tested in the organizational decision-making process by developed countries to improve their EBDM cycle.

Background

Globally, there is a growing interest in using the research evidence in public health policy-making [1, 2]. Public health systems are diverse and complex, and health policymakers face many challenges in developing and implementing policies and programs that are required to be efficient [1, 3]. The use of scientific evidence is considered to be an effective approach in the decision-making process [3-5]. Due to the lack of sufficient resources, evidence-based decision-making (EBDM) is regarded as a way to optimize costs and prevent wastes [6]. At the same time, the direct consequence of ignoring evidence is poorer health for the community [7].

Evidence suggests that health systems often fail to exploit research evidence properly, leading to inefficiencies, death or reduced quality of citizens’ lives, and a decline in productivity [8]. Decision-making in the health sector without sufficient attention to evidence may lead to a lack of effectiveness, efficiency, and fairness in health systems [9]. Instead, the advantages of EBDM include adopting cost-effective interventions, making optimal use of limited resources, increasing customer satisfaction, minimizing harm to individuals and society, achieving better health outcomes for individuals and society [10, 11], as well as increasing the effectiveness and efficiency of public health programs [12].

Using the evidence in health systems’ policymaking is a considerable challenging issue that many developed and developing countries are facing nowadays. This is particularly important in the latter, where their health systems are in a rapid transition [13]. For instance, although in 2012, a study in European Union countries showed that health policymakers rarely had necessary structures, processes, and tools to exploit research evidence in the policy cycle [14], the condition can be worse among the developing and the underdeveloped ones. For example, evidence-based policy-making in developing countries like those located in the Middle East can have more significant impacts [15, 16]. In such countries resources are generally scarce, so the policymakers' awareness of research evidence becomes more important [17]. In general, low and middle-income countries have fewer resources to deal with health issues and need quality evidence for efficient use of these resources [7].

Since the use of EBDM is fraught with the dilemma of most pressing needs and having the least capacity for implementation especially in developing countries [16], efforts have been made to create more comprehensive guidelines for EBDM in healthcare settings, in recent years [18]. Stakeholders are significantly interested in supporting evidence-based projects that can quickly prioritize funding allocated to health sectors to ensure the effective use of their financial resources [19-21]. However, it is unlikely that the implementation of EBDM in Health System Management (HSM) will follow the evidence-based medicine model [10, 22]. On the other hand, the capacity of organizations to facilitate evidence utilization is complex and not well understood [22], and the EBDM process is not usually institutionalized within the organizational processes [10]. A study in 2005 found that few organizations support the use of research evidence in health-related decisions, globally [23]. Weis et al. (2012) also reported there is insufficient information on EBDM in local health sectors [12]. In general, it can be emphasized that relatively few organizations hold themselves accountable for using research evidence in developing health policies [24]. To the best of our knowledge, there isn`t any comprehensive global and practical model developed for EBDM in health systems/organizations management. Accordingly, the present study aimed to develop a comprehensive framework for EBDM in health system management. It can shed the light on policymakers to access a detailed practical model and enable them to apply the model in actual conditions.

Methods

This was a Best Fit Framework (BFF) synthesis conducted in 2020 to develop a comprehensive framework for EBDM in HSM. Such a framework synthesis is achieved as a combination of the relevant framework, theory, or conceptual models and particularly is applied for developing a priori framework based on deductive reasoning [25]. The BFF approach is appropriate to create conceptual models to describe or express the decisions and behaviors of individuals and groups in a particular domain. This is distinct from other methods of evidence synthesis because it employs a systematic approach to create an initial framework for synthesis based on existing frameworks, models, or theories [25] for identifying and adapting theories systematically with the rapid synthesis of evidence [25, 26]. The initial framework can be derived from a relatively well-known model in the target field, or be formed by the integration of several existing models. The initial framework is then reduced to its key components that have shaped its concepts [25]. Indeed, the initial framework considers as the basis and it can be rebuilt, extended, or reduced based on its dimensions [26]. New concepts also emerge based on the researchers' interpretation of the evidence and ongoing comparisons of these concepts across studies [25]. This approach of synthesis possesses both positivist and interpretative perspectives; it provides the simultaneous use of the well-known strengths of both framework and evidence synthesis [27].

In order to achieve this aim the following methodological steps were conducted as follows:

Searching and selection of studies

In this step, we aimed to look for the relevant models and frameworks related to evidence-based decision-making in health systems management. The main research question was “what is the best framework for EBDM in health systems?” after defining the research question, the researchers searched for published studies on EBDM in HSM in different scientific databases with relevant keywords and constraints as inclusion and exclusion criteria from 01.01.2000 to 12.31.2020 (Table 1).

Table 1

Search strategy for the review

Databases:ISI web of science, PubMed (PMC, MEDLINE), Scopus, Science Direct, ProQuest, EMBASE
Limits: Language (English), In Title/Abstract (keywords), Full text Available,Document type: Article, Review, Dissertation & Thesis
Publication date: 2000 up to 2020
Search strategy: #1 AND #2 AND #3
#1	"Evidence-Based Decision-Making" OR "Evidence-Based Management" OR "Evidence-Based Policy-Making" OR "Evidence-Informed Decision-Making" OR "Evidence-Informed Policy"
#2	Criteria OR Factor* OR Component* OR part* OR element* OR segment* OR item* OR determinant* OR section* OR Process OR Model OR Framework
#3	Health OR Hospital*
Example (Scopus database)	( TITLE-ABS-KEY ( "Evidence-Based Decision-Making" OR "Evidence-Based Management" OR "Evidence-Based Policy-Making" OR "Evidence-Informed Decision-Making" OR "Evidence-Informed Policy") AND TITLE-ABS-KEY ( Criteria OR Factor* OR Component* OR part* OR element* OR segment* OR item* OR determinant* OR section* OR Process OR Model OR Framework) AND TITLE-ABS-KEY ( health OR hospital*) AND LANGUAGE ( english)) AND DOCTYPE ( ar OR re) AND PUBYEAR > 2000

Inclusion and exclusion criteria

Inclusion criteria were determined as the studies that identify the components or develop a model or framework of EBDM in health organization in the form of original or review articles or dissertations, which were published in English and had a full text. The studies like book reviews, opinion articles, and commentaries that lacked a specific framework for conducting our review were excluded. During the search phase of the study, we attempted as much as possible to access studies that were not included in the search process or gray literature by reviewing the references lists of the retrieved studies or by contacting the authors of the articles or experts and querying them, as well as manually searching the related sites (Fig. 1).

Fig. 1

The PRISMA flowchart for selection of the studies in scoping review

Quality appraisal

The quality of the obtained studies was investigated using three tools for assessing the quality of various types of studies considering types and methods of the final include studies in systematic review. These tools were including Critical Appraisal Skills Program (CASP) for assessing the quality of qualitative researches [28], Scale for the Assessment of Narrative Review Articles (SANRA) [29], and The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers [30] (Table 3-Appendix).

Data extraction

After searching the studies from all databases and removing duplicates, the studies were independently reviewed and screened by two members (TS and MRAM) of the research team in three phases by the title, abstract, and then the full text of the articles. At each stage of the study, the final decision to enter the study to the next stage was based on agreement and, in case of disagreement, the opinion of the third person from the research team was asked (PB). Mendeley reference manager software was used to systematically search and screen relevant studies. The data from the included studies were extracted based on the study questions and accordingly, a form of the studies’ profile including the author's name, publication year, country, study title, type of study, and its conditions were prepared in Microsoft Excel software (Table 4-Appendix).

Synthesis and the conceptual model

In this step, a thematic analysis approach was applied to extract and analyze the data. For this purpose, first, the texts of the selected studies were read several times, and the initial qualitative codes or thematic concepts, according to the determined keywords and based on the research question, were found and labeled. Then these initial thematic codes were reviewed to achieve the final codes and they were integrated and categorized to achieve the final main themes and sub-themes, eventually. The main and the sub-themes are representative of the main and sub-steps of EBDM. At the last stage of the synthesis, the thematic analysis was finalized with 8 main themes and all the main and the sub-themes were tabulated (Table 5-Appendix).

Creation of a new conceptual framework

For BFF synthesis in the present study, we compared the existing models and tried to find a model that fits the best. Three related models that appeared to be relatively well-suited to the purpose of this study to provide a complete, comprehensive, and practical EBDM model in HSM were found. According to the BFF instruction in Carroll et al. (2013) study [25], we decided to use all three models as the basis for the best fit because any of those models were not complete enough and we could give no one an advantage over others. Consequently, the initial model or the BFF basis was formed and the related thematic codes were classified according to the category of this basis as the main themes/steps of EBDM in HSM (Table 5-Appendix). Then, the additional founded thematic codes were added and incorporated to this basis as the other main steps and the sub-steps of the EBDM in HSM according to the research team and some details in the form of sub-steps were added by the research team to complete the synthesized framework. Eventually, a comprehensive practical framework consisting of 10 main steps and 47 sub-steps was created with the potentiality of applying and implementing EDBM in HSM that we categorized them into four main phases (Table 6-Appendix).

Testing the synthesis: comparison with the a priori models, dissonance and sensitivity

In order to assess the differences between the priori framework and the new conceptual framework, the authors tried to ask some experts’ opinions about the validity of the synthesized results. The group of experts has included eight specialists in the field of health system management or health policy-making. These experts have been chosen considering their previous research or experience in evidence-based decision/policy making performance/management (Table 2). This panel lasted in two three-hour sessions. The finalized themes and sub-themes (Table 6-Appendix) and the new generated framework (Fig. 3) were provided to them before each session so that they could think and then in each meeting they discussed them. Finally, all the synthesized themes and sub-themes resulted were reviewed and confirmed by the experts.

Table 2

The demographic characteristic of the experts that participated in the synthesis

Variables		Frequency (percent)
Expertise of experts	health system management	4 (50)
	health policy-making	4 (50)
Gender	Male	6 (75)
	Female	2 (25)
Workplace	Tehran University of medical sciences	2 (25)
	Iran University of medical sciences	2 (25)
	Shiraz University of medical sciences	2 (25)
	Esfahan University of medical sciences	2 (25)
Age (Mean)		47
Work experience (year) (Mean)		10

Fig. 3

The main steps and sub-steps of the framework of EBDM in health system management

Ethical considerations

To prevent bias, two individuals carried out all stages of the study such as screening, data extraction, and data analysis. The overall research project related to this manuscript was approved by the medical ethics conceal of the research deputy of Shiraz University of Medical Sciences with approval number IR.SUMS.REC.1396–01-07–14184, too.

Results

The initial search across six electronic databases and the Cochrane library yielded 4613 studies. After removing duplicates, 2416 studies were assessed based on their titles. According to the abstract screening of the 1066 studies that remained after removing the irrelevant titles, 291 studies were selected and were entered into the full-text screening phase. Due to full-text screening of the studies, 17 final studies were selected for extracting the components and steps of EBDM in HSM (Fig. 1). The features of these studies were summarized in Table 4-Appendix (see supplementary data). Furthermore, according to the quality appraisal of the included studies, the majority of them had an acceptable level of quality. These results have been shown in Table 3-Appendix.

Results of the thematic analysis of the evidence (Table 5-Appendix) along with the concepts proposed and added by the research team according to the focus-group discussion of the experts were shown in Table 6-Appendix. Accordingly, the main steps and related sub-steps of the EBDM process in HSM were defined and categorized.

After collecting, synthesizing, and categorizing thematic concepts, incorporating them with the initial models, and adding the additional main steps and sub-steps to the basic models, the final synthesized framework as a best-fit framework for EBDM in HSM was developed in the form of four general phases of inquiring, inspecting, implementing, and integrating and 10 main steps (Fig. 2). For better illustration, this framework with all the main steps and 47 sub-steps has been shown in Fig. 3, completely.

Fig. 2

The final synthesized framework of evidence-based decision-making in health system management

Discussion

In the present study, a comprehensive framework for EBDM in HSM was developed. This model has different distinguishing characteristics than the formers. First of all, this is a comprehensive practical model that combined the strengths and the crucial components of the limited number of previous models; second, the model includes more details and complementary steps and sub-steps for full implementation of EBDM in health organizations and finally, the model is benefitted from a cyclic nature that has a priority than the linear models. Concerning the differences between the present framework and other previous models in this field, it must be said that most of the previous models related to EBDM were presented in the scope of medicine (that they were excluded from our SR according to the study objectives and exclusion criteria). A significant number of those models were proposed for the scope of public health and evidence-based practice, and only a limited number of them focused exactly on the scope of management and policy/decision making in health system organizations.

Given that the designed model is a comprehensive 10-step model, it can be used in some way at all levels of the health system and even in different countries. However, there will be a difference here, given that this framework provides a practical guide and a comprehensive guideline for applying evidence-based decision-making approach in health systems organizations, at each level of the health system in each country, this management approach can be applied depending on their existing infrastructure and the processes that are already underway (such as capacity building, planning, data collection, etc.), and at the same time, with a general guide, they can provide other infrastructure as well as the prerequisites and processes needed to make this approach much more possible and applicable.

It is true that evidence-based management is different from evidence-based medicine and even more challenging (due to lack of relevant data, greater sensitivity in data collection and their accuracy, lack of consistency and lack of transparency in the implementation of evidence-based decision-making in management rather than evidence-based medicine, etc.). Still, the general framework provided in this article can be used to help organizations that really want to act and move forward through this approach.

Furthermore, based on the findings, most of the previous studies only referred to some parts of the components and steps of the EBDM in health organizations and neglected the other parts or they were not sufficiently comprehensive [31-40]. Most of the previous models did not mention the necessary sub-steps, tools, and practical details for accurate and complete implementation of the EBDM, which causes the organizations that want to use these models, will be confused and cannot fully implement and complete the EBDM cycle. Among the studies that have provided a partly complete model than the other studies, were the studies by Brownson (2009), Yost (2014), and Janati (2018) [3, 41, 42]. Consequently, the combination of these three studies has been used as the initial framework for the best-fit synthesis in the present study.

Likewise, the models presented by Brownson (2009) and Janati (2018) were only limited to the six or seven key steps of the EBDM process, and they did not mention the details required for doing in each step, too [3, 4, 42]. Also, the model presented in the study of Janati (2018) was linear, and the relationships between the EBDM components were not well considered [42, 43]; however, the model presented in this study was recursive. Also, in Yost's study (2014), despite the 7 main steps of EBDM and some details of each of the steps, the proposed process was not schematically drawn in the form of a framework and therefore the relationships between steps and sub-steps were not clear [41]. According to what was discussed, the best-fit framework makes the possibility of concentrating the fragmented models to a comprehensive one that can be fully applied and evaluated by the health systems policymakers and managers.

In the present study, the framework of EBDM in HSM was developed in the form of four general scopes of inquiring, inspecting, implementing, and integrating including 10 main steps and 47 sub-steps. These scopes were discussed as follows:

Inquiring

In the first step, “situation analysis and priority setting”, the most frequently cited sub-step was identifying and prioritizing the problem. Accordingly, Falzer (2009), emphasized the importance of identifying the decision-making conditions and the relevant institutions and determining their dependencies as the first steps of EBDM [44]. Aas (2012) has also cited the assessment of individuals and problem status and problem-finding as the first steps of EBDM [34]. Moreover, the necessity of identifying the existing situation and issues and prioritizing them has been emphasized as the initial steps in most management models such as environmental analysis in strategic planning [45].

Despite considering the opinions and experience of experts and managers as one of the important sources of evidence for decision-making [42, 46–50], many studies did not mention this sub-step in the EBDM framework. Hence, the present authors added the acquisition of experts’ opinions as a sub-step of the first step because of its important role in achieving a comprehensive view of the overall situation.

In the second step, “quantifying the issue and developing a statement”, “Developing the conceptual model for the issue” was more addressed [37, 41, 47]. In addition, the authors to complete this step added the fourth sub-step, “Defining the main statement of issue”. This is because that most of the problems in health settings may have a similar value for managers and decision-makers and quantifying them can be used as a criterion for more attention or selecting the problem as the main issue to solve.

The third step, “Capacity building and setting objectives”, was not seen in many other included studies as a main step in EBDM, however, the present authors include this as a main step because without considering the appropriate objectives and preparing necessary capacities and infrastructures, entering to the next steps may become problematic. Moreover, in numerous studies, factors such as knowledge and skills of human resources, training, and the availability of the essential structures and infrastructures have been identified as facilitators of EBDM [51-55]. According to this justification, they are included in the present framework as sub-steps of the third step.

Considering the third step and based on the knowledge extracted from the previous studies, the three sub-steps of “understanding context and Building Culture” [56, 57], “gaining the support and commitment of leaders” [39, 57, 58], and “identifying the capabilities required by employees and their skills weaknesses” [58-60] were the most important sub-steps in this step of EBDM framework. In this regard, Dobrow (2004) has also stated that the two essential components of any EBDM are the evidence and context of its use [32]. Furthermore, Isfeedvajani (2018) stated that to overcome barriers and persuade hospital managers and committees to apply evidence-based management and decision-making, first and foremost, creating and promoting a culture of "learning through research" was important [61].

The present findings showed that in the fourth main step, “evidence acquisition and integration”, the most important sub-step was “finding the sources for seeking the evidence” [39–41, 60, 62, 63]. Concerning the sources for the use of evidence in decision-making in HSM, studies have cited numerous sources, most notably scientific and specialized evidence such as research, articles, academic reports, published texts, books, and clinical guidelines [39, 64, 65]. After scientific evidence, using the opinions and experiences of experts, colleagues, and managers [42, 46, 49, 66] as well as the use of census and local level data [49, 66, 67], and other sources such as financial [67], political [42, 49] and evaluations [49, 68] data were cited.

Inspecting

The fifth step of the present framework, “evidence appraising”, was emphasized by previous literature; for instance, Pierson (2012) pointed to the use of library services in EBDM [69]. Appraising and selecting the evidence according to appropriate appraisal tools/methods was cited the most. International and local evidence is confirmed that ignoring these criteria can lead to serious faults in the process of decision and policy-making [70, 71].

Furthermore, the sixth step, “analysis, synthesis, and interpretation of data”, was mentioned in many included studies [36, 39, 41, 42, 57, 59, 72]. This step emphasized the role of analysis and synthesis of data in the process of generation applied and useful information. It is obvious that the local interpretation according to different contexts may lead to achieving such kind of knowledge that can be used as a basis for local EBDM in HSM.

Implementing

The third scope consisted of the seventh and eighth steps of the EBDM process in HSM. In the seventh step, “developing evidence-based alternatives”, the issue of involving stakeholders in decision-making and subsequently, planning to design and implementation of the process and evaluation strategies had been focused by the previous studies [58, 60, 62, 63, 73]. Studies by Belay (2009) and Armstrong (2014) had also emphasized the need to use stakeholder and public opinion as well as local and demographic data in decision-making [49, 67].

“Pilot-implementation of selected alternatives” was the eighth step of the framework. Some key sub-steps of this step were resources allocation [58], Pre-implementation and pilot change in practice and assessing barriers and enablers for implementation [40] that indicated the significance of testing the strategies in a pilot stage as a pre- requisition of implementing the whole alternatives. It is obvious that without attention to the pilot stage, adverse and unpleasant outcomes may occur that their correction process imposes many financial, organizational, and human costs on the originations. In addition, a study explained that one of the strategies of the decision-makers to measure the feasibility of the policy options was piloting them, which had a higher chance of being approved by the policymakers. Also, pilot implementation in smaller scales has been recommended in public health in cases of lack of sufficient evidence [74].

Integrating

This last scope consists of the ninth and tenth steps. The main sub-step of the ninth step, “evaluating alternatives”, was to evaluating process and outcomes and revise. After a successful implementation of the pilot, this step can be assured that the probable outcomes may be achieved and this evaluation will help the decision and policymakers to control the outcomes, effectively. Also, it impacts the whole target program and proposes some correcting plans through an accurate feedback process, too. Pagoto (2007) explained that a facilitator for EBDM would be an efficient and user-friendly system to assess utilization, outcomes, and perceived benefits [55].

Also, the tenth step, “integrating and maintaining change in practice”, was not considered as a major step in previous models, too, while it is important to maintain and sustain positive changes in organizational performance. In this regard, Ward (2011) also suggested several steps to maintain and sustain the widespread changes in the organization, including increasing the urgency and speed of action, forming a team, getting the right vision, negotiating for buy-in, empowerment, short-term success, not giving up and help to make a change stick [35]. Finally, the most important sub-steps that could be mentioned in this step were the dissemination of evidence results to decision-makers and the integration of changes made to existing standards and performance guidelines. Liang (2012) had also emphasized the importance of translating existing evidence into useful practices as well as disseminating them [47]. In addition, the final sub-step, “feedback and feedforward towards the EBDM framework”, was explained by the authors to complete the framework.

Some previous findings showed that about half and two-thirds of organizations do not regularly collect related data about the use of evidence, and they do not systematically evaluate the usefulness or impact of evidence use on interventions and decisions [75]. The results of a study conducted on healthcare managers at the various levels of an Iranian largest medical university showed that the status of EBDM is not appropriate. This problem was more evident among physicians who have been appointed as managers and who have less managerial and systemic attitudes [76]. Such studies, by concerning the shortcomings of current models for EBDM in HSM or even lack of a suitable and usable one, have confirmed the necessity of developing a comprehensive framework or model as a practical guide in this field. Consequently, existing and presenting such a framework can help to institutionalize the concept of EBDM in health organizations.

In contrast, results of Lavis study (2008) on organizations that supported the use of research evidence in decision-making reported that more than half of the organizations (especially institutions of health technology assessment agencies) may use the evidence in their process of decision-making [75], so applying the present framework for these organizations can be recommended, too.

Limitations

One of the limitations of the present study was the lack of access to some studies (especially gray literature) related to the subject in question that we tried to access them by manual searching and asking from some articles’ authors and experts. In addition, most of the existing studies on EBDM were limited to examining and presenting results on influencing, facilitating, or hindering factors or they only mentioned a few components in this area. Consequently, we tried to search for studies from various databases and carefully review and screen them to make sure that we did not lose any relevant data and thematic code. Also, instead of one model, we used four existing models as a basis in the BFF synthesis so that we can finally, by adding additional codes and themes obtained from other studies as well as expert opinions, provide a comprehensive model taking into account all the required steps and details. Also, the framework developed in this study is a complete conceptual model made by BFF synthesis; however, it may need some localization, according to the status and structure of each health system, for applying it.

Conclusions

The present framework provides a comprehensive guideline that can be well adapted for implementing EBDM in health systems and organizations especially in underdeveloped and developing countries where there is usually a lag in updating and applying evidence in their decision-making process. In addition, this framework by providing a complete, well-detailed, sequential and practical process including 10 steps and 56 sub-steps that did not exist in the incomplete related models, can be tested in the organizational decision-making process or managerial tasks by developed countries to improve their EBDM cycle, too.

Table 3

Quality assessment of the included studies

Study title	Critical Appraisal Skills Program (CASP) for assessing the quality of qualitative researches (yes/no/can’t tell)
	Was there a clear statement of the aims of the research?	Is a qualitative methodology appropriate?	Was the research design appropriate to address the aims of the research?(Appropriate research design)	Was the recruitment strategy appropriate to the aims of the research? (Sampling)	Were the data collected in a way that addressed the research issue? (Data collection)	Has the relationship between researcher andParticipants been adequately considered? (Reflexivity)	Have ethical issues been taken into consideration? (Ethical Issues)	Was the data analysis sufficiently rigorous? (Data Analysis)	Is there a clear statement of findings? (Findings)	How valuable is the research? (Value of the research)
Public Health Decision-Makers’ informational Needs and Preferences for Receiving Research Evidence	Yes	Yes	Yes	Can’t tell	Yes	Yes	Yes	Yes	Yes	Yes
Challenging Evidence-based Decision-making: A Hypothetical Case Study about Return to Work	Yes	Yes	Can’t tell	No	Yes	Can’t tell	Can’t tell	Can’t tell	Yes	Yes
Factors Affecting Evidence-Based Decision Making in Local Health Departments	Yes	Yes	Yes	Yes	Yes	Yes	Can’t tell	Can’t tell	Yes	Yes
Organizational impact of evidence-informed decision making training initiatives: a case study comparison of two approaches	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Tools to support evidence-informed public health decision making	Yes	Yes	Yes	Can’t tell	Yes	Can’t tell	Yes	No	Yes	Yes
Evidence-based Management in Practice: Opening up the Decision Process, Decision-maker and Context Evidence-Based	No	Can’t tell	Yes	Yes	Yes	Yes	Can’t tell	Yes	Yes	Yes
Perception of Nursing Middle Managers about the Evidence-Based Management	Yes	Yes	Yes	Can’t tell	Yes	Yes	Yes	Yes	Yes	Yes
Evidence-based management of Caribbean health systems: barriers and opportunities	Yes	Yes	Yes	Yes	Yes	Yes	Can’t tell	Yes	Yes	Yes
An Evidence-Based Framework for Evidence-Based Management in Healthcare Organizations: A Delphi Study	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes
Evidence-Based Management Competency Model for Managers in Hospital Settings	Yes	Yes	Yes	Yes	Yes	Yes	Can’t tell	Yes	Yes	Yes
Study title	Scale for the Assessment of Narrative Review Articles (SANRA) (score from 0 to 2)
	Justification of the article’s importance for the readership	Statement of the concrete aims or formulation of questions	Description of the literature search	referencing	Scientific reasoning	Appropriate presentation of data
Evidence-Based Public Health: A Fundamental Concept for Public Health Practice	2	1	0	2	1	2
A framework to improve evidence-informed decision-making in health service management	2	2	1	2	2	2
Study title	The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (yes/no/can’t tell)
	Screening questions		Questions for Mixed methods studies
	Are there clear research questions?	Do the collected data allow to address the research questions?	Is there an adequate rationale for using a mixed methods design to address the research question?	Are the different components of the study effectively integrated to answer the research question?	Are the outputs of the integration of qualitative and quantitative components adequately interpreted?	Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?	Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?
Understanding evidence: a statewide survey to explore evidence-informed public health decision-making in a local government setting	Yes	Yes	Yes	Yes	Yes	No	Can’t tell
Sustainability in Health care by Allocating Resources Effectively (SHARE) 7: supporting staff in evidence-based decision-making, implementation, and evaluation in a local healthcare setting	Yes	Yes	Yes	Yes	Yes	Can’t tell	Can’t tell
Sustainability in Health care by Allocating Resources Effectively (SHARE) 8: developing, implementing, and evaluating an evidence dissemination service in a local healthcare setting	Yes	Yes	Yes	Yes	Yes	Can’t tell	Can’t tell
Evaluation of the performance and achievements of the WHO Evidence informed Policy Network (EVIPNet) Europe	Yes	Yes	Yes	Yes	Yes	Can’t tell	Yes
Promoting the use of evidence in health policymaking in the ECOWAS region: the development and contextualization of an evidence-based policymaking guidance	Yes	Yes	Yes	Yes	No	No	Can’t tell

Table 4

Summary of characteristics of included studies

Study number	First Author (Year)	Country	Study Title	Study Design	Setting
1	Maureen Dobbins (2007) [72]	Ontario, Canada	Public Health Decision-Makers’ informational Needs and Preferences for Receiving Research Evidence	Qualitative research (semi-structured interviews)	16 participants (nine program managers, six directors, and one Medical Officer of Health)
2	Ross C. Brownson (2009) [3]	US	Evidence-Based Public Health: A Fundamental Concept for Public Health Practice	Descriptive review	––
3	Zhanming Liang (2012) [47]	Australia	A framework to improve evidence-informed decision-making in health service management	literature review	46 studies were included
4	Randi W. Aas (2012) [34]	Norway	Challenging Evidence-based Decision-making: A Hypothetical Case Study about Return to Work	Qualitative research (case study/ Interviews)	30 participants
5	Collette D. Sosnowy (2013) [54]	New York, USA	Factors Affecting Evidence-Based Decision Making in Local Health Departments	Qualitative research (Interviews, focus groups)	47 participants (7 commissioners, 21 health directors, and 19 other upper-level staff)
6	François Champagne (2014) [36]	6 province of Canada	Organizational impact of evidence-informed decision making training initiatives: a case study comparison of two approaches	Qualitative research (Interviews, documents analysis)	84 participants, organizational documents and organizations’ web sites
7	Rebecca Armstrong (2014) [49]	Victoria, Australia	Understanding evidence: a statewide survey to explore evidence-informed public health decision-making in a local government setting	Mixed-method (cross-sectional survey and interviews)	135 survey respondents and 13 interviews
8	Jennifer Yost (2014) [41]	Ontario, Canada	Tools to support evidence-informed public health decision making	Qualitative research (Interviews)	37 participants
9	April L. Wright (2016) [37]	Australia	Evidence-based Management in Practice: Opening up the Decision Process, Decision-maker and Context Evidence-Based	Qualitative research (case study/ semi-structured interview)	24 emergency physicians and registrars, 4 hospital executives, and 1 nurse
10	Wilza Carla Spiri (2017) [65]	São Paulo, Brazil	Perception of Nursing Middle Managers about the Evidence-Based Management	Qualitative research (Case Study/ individual structured interview)	9 nurse managers
11	Damian Eisenghower Greaves (2017) [77]	Caribbean island	Evidence-based management of Caribbean health systems: barriers and opportunities	Qualitative research (semi-structured interviews)	20 senior managers/leaders (7 ministers for health, 7 permanent secretaries and 6 chief medical officers from 7 countries)
12	Claire Harris (2017) [39]	Australia	Sustainability in Health care by Allocating Resources Effectively (SHARE) 7: supporting staff in evidence-based decision-making, implementation, and evaluation in a local healthcare setting	Mixed methods (Literature reviews, surveys, interviews)	literature review, 178 survey responses and 68 interviews
13	Claire Harris (2018) [40]	Australia	Sustainability in Health care by Allocating Resources Effectively (SHARE) 8: developing, implementing, and evaluating an evidence dissemination service in a local healthcare setting	Mixed methods (Literature reviews, surveys, interviews)	literature review, 164 survey responses and 27 interviews
14	Ali Janati (2018) [42]	Iran	An Evidence-Based Framework for Evidence-Based Management in Healthcare Organizations: A Delphi Study	Qualitative research (Delphi study)	66 participants
15	Louise Lester (2020) [58]	Europe	Evaluation of the performance and achievements of the WHO Evidence informed Policy Network (EVIPNet) Europe	Mixed-method research	21 current Evidence informed Policy Network (EVIPNet) Europe member countries
16	Lina Daouk-Öyry (2020) [60]	Lebanon	Evidence-Based Management Competency Model for Managers in Hospital Settings	Qualitative research (semi-structured interviews)	36 executive managers from 11 hospitals
17	Chigozie Jesse Uneke (2020) [63]	West Africa	Promoting the use of evidence in health policymaking in the ECOWAS region: the development and contextualization of an evidence-based policymaking guidance	Mixed-method research (review and consultations)	15 countries of West African sub-region

Table 5

The steps and sub-steps of the EBDM framework resulted from thematic analysis

Steps(frequency of references)	Sub-Steps(frequency of reference)
Situation analysis and priority setting (7)	Identifying and prioritizing the problem (4)
	Surveying the results of previous interventions (4)
	Determining information gaps (2)
Quantifying the issue and developing a statement (2)	Developing the conceptual model for the issue (3)
	Experts’ opinions/experiments (1)
Capacity building (2)	Understanding the context (2) and building an evidence-based culture (1)
	Gaining leadership support & commitment (3)
	Identifying the capabilities required by employees and their skills weaknesses (3)
	Training (1)
	Developing the necessary infrastructures and structures (1)
	Assigning mandates (1) and determining incentives (2)
Evidence acquisition and integration (10)	Developing an efficient search strategy (2)
	Finding the sources for seeking the evidence (6) according to 6S Pyramid (1) including: Scientific literature (2), Rapid Reviews (1), Expert panels (1), Patient’s experience (1), Professional expertise (1), Consultation (1), case studies (1)
	Keeping track of search results (1)
Evidence appraising (7)	Sourcing the evidence (1), library services and reference managers (1)
	Appraising and selecting the evidence according to appropriate appraisal tools/methods such as: AGREE II instrument, AMSTAR Tool, Critical Appraisal Skills Program (CASP) Tools, Scottish Intercollegiate Guidelines Network, Quality Assessment Tool for Quantitative Studies (1), Benefits & risks, feasibility, applicability, and transferability data (5)
Analysis, synthesis and interpretation of data (7)	Data extraction (1)
	Data analysis and synthesis (3) according to: evidence format, style of presentation, accessibility, validity, context sensitivity, applicability, timeliness
	Determining potential features (scope, components, knowledge brokers, target audience, methods) (1)
	Translation of data into user-friendly formats (1) using knowledge translation planning tools (1)
Developing evidence- based alternatives (8)	Engaging community and stakeholders (3) and participatory decision making (3)
	Developing program logic (1) and defining proposed change alternatives (1)
	Identifying needed resources (1)
	Planning implementation and evaluation process and strategies (1)
	Defining outcomes to be achieved (29, 30) and develop indicators (1)
Pilot implementation of selected alternatives (10)	Resources allocation (1)
	Pilot change in practice (1)
	Assessing barriers and enablers for implementation (1)
Evaluate alternative (8)	Collecting baseline data (1)
	Evaluating processes and outcomes (2) and revise (1)
	Deciding to adapt, adopt, or reject practice change ((1)
	Assessing factors for success and sustainability (1)
	Checking the program checklist (1)
Integrate and maintain change in practice (1)	Disseminating evidence results to decision makers (2), Essential information conveyed effectively to target audiences/stakeholders (2)
	Integrating change into standards of practice (1) or discontinue program or policy (1)
	Thoughts for future and adaptions (1)

Table 6

The finalized steps and sub-steps of the EBDM framework resulted from evidence synthesis and the research team analysis

Steps	Sub-Steps
Situation analysis and priority setting (7)a	Identifying and prioritizing the problem (4)
	Surveying the results of previous interventions (4)
	Acquisition of experts’ opinions (RTS)b
	Determining information gaps (2)
Quantifying the issue and developing a statement (2)	Developing the conceptual model for the issue (3) and defining the main statement of issues (RTS)
	Determining data resources like: surveillance data or clinical problems data (RTS), process improvement or risk-management data (RTS), internal/external benchmarking data (RTS), financial data (RTS), national agencies or organizational standards and guidelines (RTS), new researches and other literature (RTS), and experts’ opinions/experiments (1)
Capacity building (2) and setting objectives (RTS)	Understanding the context (2) and building an evidence-based culture (1)
	Forming a team (RTS)
	Gaining leadership support & commitment (3)
	Setting objectives (RTS)
	Identifying the capabilities required by employees and their skills weaknesses (3)
	Training (1)
	Developing the necessary infrastructures and structures (1) like improving health information systems (RTS)
	Assigning mandates (20) and determining incentives (2)
Evidence acquisition and integration (10)	Developing an efficient search strategy (2)
	Finding the sources for seeking the evidence (6) according to 6S Pyramid (1) including: Scientific literature (2), Meta-analysis or meta-synthesis (RTS), Rapid Reviews (1), Other types of evidence (case-report, expert opinion, scientific principles, theory (RTS), Expert panels (1), Patient’s experience (1), Professional expertise (1), Consultation (1), Risk assessments (RTS), Economic data (RTS), case studies (1)
	Keeping track of search results (1)
	(If necessary) designing toa conduct research (RTS)
Evidence appraising (7)	Sourcing the evidence (1) using library services and reference managers (1) and Removing duplicates (RTS)
	Appraising and selecting the evidence according to appropriate appraisal tools/methodsa such as: AGREE II instrument, AMSTAR Tool, Critical Appraisal Skills Program (CASP) Tools, Scottish Intercollegiate Guidelines Network, Quality Assessment Tool for Quantitative Studies (1), Benefits & risks, feasibility, applicability, and transferability data (5)
Analysis, synthesis and interpretation of data (7)	Data extraction (1) and discarding mismatched alternatives (RTS)
	Data analysis and synthesis (3) according to: evidence format, style of presentation, accessibility, validity, context sensitivity, applicability, timeliness
	Determining potential features (scope, components, knowledge brokers, target audience, methods) (1)
	Translation of data into user-friendly formats (1) using knowledge translation planning tools (1)
Developing evidence- based alternatives (8)	Engaging community and stakeholders (3) and participatory decision making (3)
	Developing program logic (1) and defining proposed change alternatives (1)
	Identifying needed resources (1)
	Planning implementation and evaluation process and strategies (1) /design EBP guideline(s) (RTS)
	Defining outcomes to be achieved (2) and develop indicators (1)
Pilot implementation of selected alternatives (10)	Document and investigate the change process in an action research approach (RTS)
	Training the trainers and target audience (RTS) and empowerment staff according to the specific needs assessments (RTS)
	Resources allocation (1)
	Pre-implementation (RTS) and pilot change in practice (1)
	Assessing barriers and enablers for implementation (1)
	Developing recommendations and reporting template (RTS)
	Modifying the practice guidance (RTS)
Evaluate alternative (8)	Collecting baseline data (1) and Measuring at baseline mid-project and one-year post-project (RTS)
	Evaluating processes and outcomes (2) and revise (1)
	Deciding to adapt, adopt, or reject practice change (1)
	Assessing factors for success and sustainability (1)
	Checking the program checklist (1) [and if necessary] modifying the assessment criteria (RTS)
	Feedback/correction (RTS)
Integrate and maintain change in practice (1)	Making decisions based on the best available scientific and rigorous program evaluation evidence (RTS)
	Disseminating evidence results to decision makers (2)/ [making sure that] Essential information conveyed effectively to target audiences/stakeholders (2)
	Integrating change into standards of practice (1) or discontinue program or policy (1)
	Presenting staff in-service education on change in practice (RTS)
	Thoughts for future and adaptions (1)/ Feedback and feedforward to evidence-based decision-making model (RTS)

a The numbers in parentheses indicates the frequency of references that include the concept

b RTS stands for the concepts synthesized, proposed and added by the research team and confirmed by the experts