Defining research to improve health systems.

Robert Terry and colleagues present working definitions of operational research, implementation research, and health systems research within the context of research to strengthen health systems.

Research has an important role to play in strengthening health systems to improve system performance and public health impact.

The multiple definitions of operational research, implementation research, and health systems research creates confusion and negatively affects the credibility and progress of the research.

The aim of this paper is to present working definitions of operational research, implementation research, and health systems research to provide greater clarity for non-specialists, scientists, policymakers, and donors working to strengthen health systems.

Introduction

A major obstacle to achieving the health-related Millennium Development Goals (MDGs) is the weakness of the health systems in many low- and middle-income countries, and their struggle to effectively provide health care to populations in need [1],[2]. Several global health initiatives have been created over the last decade to support the delivery of available interventions for priority health problems, and in recent years there have been some major new initiatives to support health system strengthening [1],[3],[4]. These developments have been accompanied by a growing recognition of the role of research in improving health systems and health care delivery. The ministerial summit on health research that was held in Mexico in 2004 concluded that research has a crucial part to play in strengthening health systems and in improving the equitable distribution of quality health services for populations in need, and the summit called for greater support for such research [5]. Since then, the number of research initiatives on health systems in low- and middle-income countries has increased substantially [6],[7]. This is a positive development that we would like to see expanded and accelerated to build up evidence-based knowledge to improve the effectiveness of health systems. Unfortunately, these initiatives have also led to growing confusion about what type of research is involved and at whom that research is targeted. The fact that the various research initiatives originate from different research backgrounds (biomedicine, social sciences, organization of services, health economics, etc.) has led to an inconsistent use of terminology to describe the research. Multiple definitions of operational research, implementation research, and health systems research have been proposed in recent years [2],[6]–[18], and many of these define the scope of their research very broadly, resulting in considerable overlap between definitions. Operational research and implementation research are sometimes used interchangeably in the literature, or are classified as health systems research [10]–[12],[17].

So does this matter?

We believe it does, because the resulting confusion may create duplications and inefficiencies both in the funding for different research efforts and among those seeking to understand and use the evidence. It indicates a lack of shared conceptual clarity among scientists and decision-makers about the scope, nature, methodologies, and issues to be addressed by the research involved [6]. This makes efforts to retrieve relevant evidence on particular topics even more complex than it already is, negatively affecting the credibility of the research itself.

The aim of this paper is to present working definitions of operational research, implementation research, and health systems research in the context of research to strengthen health systems, with the intention of providing greater clarity and consistency for non-specialists, scientists, policymakers, and donors.

As a starting point, the three research areas described here refer to research domains that differ at their core in the type of research questions they address, in how they are organized, and in how they interface with the health system. Below, we briefly describe the proposed three domains, indicate where they differ and overlap, how they complement each other, and how they could more effectively interact for greater impact of the overall research effort. We hope that, in this year with an unprecedented focus on research to strengthen health systems with the First Global Symposium on Health Systems Research this paper will contribute to greater clarity and more efficient approaches to fulfil the overall objective of strengthening health systems to improve population health [19].

Defining the Research Domains

Building on the numerous definitions that exist in the literature under the overall umbrella of research to improve health systems, three domains of research can be defined using their primary characteristics: the focus of the research, the users of the research outputs, and the utility of the research outputs (see Figure 1 and Table 1).

Figure 1

Research to improve health systems.

Table 1

Defining research to improve health systems.

Research Domain	Primary Characteristic
	Focus of the Research	Users of the Research Outputs	Utility of the Research Outputs*
Operational	Operational issues of specific health programmes	Health care providers programme managers	Local
Implementation	Implementation strategies for specific products or services	Programme managers, R&D managers	Local/broad
Health System	Issues affecting some or all of the building blocks of a health system	Health system managers, policy makers	Broad

*How amenable the research outputs are to adaptation, scaling up or use or in other contexts or locations.

The focus of the research within a health system [20] is explained more fully in the following sections.

The users of the research outputs (published results, findings, methodologies, etc.) fall broadly into three groups with operational research being predominantly, but not exclusively, of use to health care providers; implementation research predominantly of use to managers of programmes scaling up an intervention; and research on the health system as a whole (or one of its building blocks) of most use to those who manage or need to make policy for the health system.

In another characteristic, the importance of how amenable the research is to adaptation and use in other contexts or locations—also varies across the three domains. We define this characteristic as the utility of the research, where utility describes the fact, character, or quality of something being useful or serviceable [21]. While it is well established that all research to support health systems is context-specific, careful consideration of study design and reporting of context-specific factors generally improve the application of this research to other settings. This can be achieved in varying degrees of success across the three domains. For example, operational research tends to address a local problem, taking into account the particular context in which it occurs. Therefore, the research utility of the outputs would not be readily applicable to other settings without careful and considerable adaptation.

As it moves through implementation research and onto health systems research, the utility of the research tends to be broader and have increasingly common points of comparison with other contexts. Therefore, it is not uncommon to be able to generate lessons from this type of research that are applicable to other settings within the country or even for other countries. So, using research utility as defined here, operational research generates research outputs that are generally local in their utility, and health systems research has a primary characteristic of generally being more amenable to adaptation and application in other contexts, i.e., a broad utility. Again the degree of utility is wholly dependent on the use of an appropriate study design or protocol that takes into consideration of contextual factors.

In subsequent sections we will elaborate on these definitions using case studies and examples to illustrate them.

1. Research Domain: Operational

Operational research aims to develop solutions to current operational problems of specific health programmes or specific service delivery components of the health system, e.g., a health district or a hospital. Table 2 gives some selected examples of research questions that illustrate the local nature of the type of problems that this research addresses. These are problems that confront a local disease control programme, health district, or health clinic during the execution of its routine operations, and for which practically useful answers or solutions are urgently needed to allow operations to proceed more effectively. This research is characterized by a strong problem-solving focus and an urgency to find solutions. Its demand-driven nature and close association with health care delivery and routine health care operations ensure operational relevance of the research activities and rapid uptake and local utilization of research findings. The operational problems are often identified through routine monitoring and evaluation activities for which this research, where it exists, can be an important complement that allows a health programme to explore new ideas and experiment with potentially more effective approaches to its operations.

Table 2

Examples of research questions for the three research domains.

Research Domain	Research Question*	Reference
Operational	Can the “communication for behavioural impact” (COMBI) strategy improve the poor compliance with mass drug administration for LF elimination in Tamil Nadu, India?	[46],[47]
	Which locations should be targeted for delivering HIV prevention services in Kawempe district, Uganda?	[9]
	Which of the current ART payment strategies in use in Nairobi should be retained for the new integrated programme?	[48]
	Should the sleeping sickness programme in Equator Nord province, DRC, change its first-line drug?	[49]
Implementation	How to deliver ivermectin for onchocerciasis control and ensure sustained high treatment coverage in isolated rural communities?	[50]
	How to improve access to vaccination among children who are currently not reached by immunisation services?	[25]
	How to implement antenatal syphilis screening—one-stop versus conventional service?	[51]
	How to effectively implement a new intervention package for kala azar elimination in the Indian subcontinent?	[52]
Health system	To what extent do health services reach the poor? How can this be improved?	[32]
	Should fees be charged to clients who use health centres for curative services?	[17]
	How effective are different policies for attracting nurses to rural areas?	[53]
	What has been the impact of the rapid scale-up of HIV programmes on fragile health systems?	[54]

*As noted above, depending on how the question is phrased and the research is designed; some of these questions can be addressed in several research domains.

A wide range of study designs and research methods are used, ranging from descriptive and analytical studies to operational experiments and the use of mathematical modelling. The research often starts with exploratory studies to better define the problem and its determinants, and to identify potential solutions that can subsequently be tested under operational conditions. The research requires the involvement of, and is usually executed by, health staff who have the necessary research experience and/or by scientists from collaborating local research institutions. In their definition of operational research, Zachariah et al. [8] stress the local, programme-based focus of research that “can enhance the quality, effectiveness or coverage of programs in which the research is being done.” Similarly, WHO refers to research “for programme decision making to achieve a specific outcome” [22]; and the Global Fund refers to research that “provides decision-makers with information to enable them to improve the performance of their programs” [9]. Hence, the results of this research tend to have a local utility and, because of its design, are not generally amenable for adaptation and use in other settings. This in no way devalues the research, and useful lessons—such as methodological approaches—could be applied elsewhere if reported. However, operational research is still not commonly undertaken, and many of the lessons that could be learnt remain unpublished.

Many health programmes and health system managers do not see operational research as a priority, and it is sometimes perceived as a waste of time and resources, distracting from the need for operational action on the basis of “common sense” [23]. Such attitudes tend to soften with exposure to properly executed operational research that delivers practical results, but quality operational research does not come easily, given the general lack of research capacity and research funding at the operational level. Several global health initiatives offer additional funding for operational research but most of these funds are not taken up at country and programme level because of the lack of appreciation for this type of research and insufficient local research capacity [10].

2. Research Domain: Implementation

Implementation research aims to develop strategies for available or new health interventions in order to improve access to, and the use of, these interventions by the populations in need. Table 2 provides some examples of this type of research for which the starting point is the availability of an intervention or intervention package that has been proven efficacious in previous research, but for which major questions remain as to how to scale up the intervention and ensure effective integration within the health system. This research is characterized by a focus on the need for innovative approaches and/or ensuring the effectiveness of implemented interventions. Examples may include mass treatment with ivermectin for onchocerciasis or the introduction of new evidence-based birth practices for isolated communities where there are no formal health services or maternity clinics. This research often addresses implementation of newly developed products, such as a pharmaceutical, medical device, or vaccine, where this research represents the last phase of the product development pipeline. However, in this definition implementation research also covers such interventions as service delivery approaches, behavioural interventions, or understanding the impact of a payment mechanism.

Many promising health interventions have had only limited impact on the burden of disease in low- and middle-income countries because of implementation problems that were not identified, researched, and addressed. For example, research on the impact of insecticide-treated nets to reduce malaria was stopped too soon: phase IV effectiveness trials were not followed up by implementation research, and 15 years later the utilization of nets is still low in Africa. Hence, it is critical to include research on implementation as an extension of the development phase or R&D pipeline when testing a new intervention.

The examples of implementation research above tend to be developed as focused studies with clear research questions. Multicentre and multicountry studies are often used, as these help clarify which findings are location-specific and which are more generalisable. Large-scale implementation studies often have two phases. The first phase consists of descriptive, formative research to better understand the major implementation challenges and to design potential implementation strategies. This is often followed by a second phase, in which the most promising implementation strategies are tested and compared in large-scale experimental studies in settings at the appropriate level of the health system [24],[25]. Social science research methods are extensively used (including qualitative research methods for stakeholder analysis and process evaluation) as well as methods for determining the cost of implementation strategies at different levels of the formal and informal health system.

Implementation research is usually undertaken by multidisciplinary research groups drawn from many countries, including those where the study is located. There is often a network of such groups, supported by international research institutions and expertise as required. The necessary local research capacity in behavioural sciences, health economics, and epidemiology is still inadequate in many low-income countries, and building such research capacity remains a top priority [7],[14]. Where adequate research capacity does exist, it is often isolated, and special efforts may be needed to identify and involve such groups in research initiatives.

With regard to existing definitions, Sanders et al. refer to “research to promote the uptake and successful implementation of evidence-based interventions and policies” [12], and Allottey et al. to “evidence that informs effective, sustained and embedded adoption of interventions by health systems and communities” [13]. The Special Programme for Research and Training in Tropical Diseases (TDR) defines implementation research as “research to significantly improve access to efficacious interventions by developing practical solutions to common implementation problems” [14]. As these definitions indicate, implementation research is intervention-specific, but in contrast to operational research, it is often designed with the intention of creating outputs that can be applicable beyond the local environment in which the research is done.

The relevance of implementation research is increasingly being recognized, and several convincing examples in recent years have demonstrated the effectiveness of this type of research in enabling implementation and scale-up of priority health interventions [26],[27]. However, compared to the corresponding investment in R&D, implementation research is receiving only limited financial support; it will be important to correct this imbalance.

3. Research Domain: Health System

Health systems research addresses health system and policy questions that are not disease-specific but concern systems problems that have repercussions on the performance of the health system as a whole. It addresses a wide range of questions, from health financing, governance, and policy to problems with structuring, planning, management, human resources, service delivery, referral, and quality of care in the public and private sector. Table 2 gives a few examples of research questions that illustrate the nature of the research involved (e.g., studies on the effectiveness of different policies for attracting nurses to rural areas). Health systems issues are often highly context-specific, and many case studies try to elucidate a certain health system challenge within its specific environment.

However, with appropriate study design and planning, health systems research can not only answer policy questions relevant to the specific health system in which the research is undertaken, but can generate valuable lessons that are more amenable to adaptation and adoption in other settings. This is particularly true when a systems perspective is used, i.e., by considering all the positive and negative effects of a particular system-level intervention, this research can provide a robust and accurate understanding of health systems challenges and their potential solutions, thereby improving the utility of the findings in other settings [28]. This systems approach, in combination with stakeholder engagement, also informs the definition of priority research questions to address health systems challenges.

Health systems research by necessity is highly multidisciplinary, with a strong emphasis on social sciences, economics, and anthropological investigations, for example on community perceptions of health care. Much ongoing research consists of descriptive, comparative, and evaluation studies and secondary analytical research. Although experimental studies are less common, partly because of operational and ethical challenges in experimenting at the health system level, they can be very informative and provide convincing evidence on the benefit of innovations in health system efficiency and health impact [29]. Most research is undertaken through collaboration between academic institutions, with a major role being played by a few institutions with special expertise in health systems research or in one of its research disciplines (e.g., health economics and policy analysis). Health planners and decision-makers may contribute to defining the research questions, but are otherwise not much involved in undertaking the research itself [6].

The research in this domain falls under the general definition by the Alliance for Health Policy and Systems Research (HPSR) as: “The production of new knowledge to improve how societies organize themselves to achieve health goals.” The Alliance for HPSR further clarifies that “the prime focus of health policy and systems research is not a specific disease or service, but rather the health system as a whole. However, health systems research sometimes adopts a disease or service specific focus” (http://www.who.int/alliance-hpsr/en/ [17]). More specifically, it can address any or all of the six building blocks of health systems identified by the WHO [20]: service delivery, information and evidence, medical products and technologies, health workforce, health financing, and leadership and governance. In doing so, it should explicitly acknowledge the importance of the continuous interactions between the different building blocks of the health systems and the different sectors (including non-health sectors) involved, as well as all the other characteristics of complex health systems [28]. Another definition offered by Varkevisser et al. [30] refers to health systems research as “research that enhances the efficiency and effectiveness of the health system.”

Research on health systems addresses a huge research area that has only been marginally covered to date [6]. Because of the multitude of system challenges and their complex multidimensional environment, research prioritization is essential and some recent priority-setting initiatives are seen as being timely [20],[31]–[33]. Due to the relative scarcity of research capacity to undertake this type of research, efforts to improve the design, robustness, and applicability of the evidence generated in one setting to another would be highly desirable. Systems thinking methods and approaches can offer tremendous help and guidance on this [28]. By using a systematic, comprehensive way of examining the design and evaluation of potential health systems interventions, and ensuring involvement and ownership of all stakeholders involved, the utility and pay back from the evidence generated from this research greatly increases.

Related Research Areas and Research Terms

There are a number of related research areas that may overlap with the above research domains but are out of the scope of this paper. Some of them are briefly described here. First, monitoring and evaluation aims to track the progress of a health intervention and to determine whether it is having its intended impact. As a routine operational activity, it is usually not regarded as research by itself, although its findings are instrumental for identifying priority problems for research. However, the term evaluation can sometimes refer to the more formal evaluation designs such as process, economic, or impact evaluations, or can be used separately to answer questions related to the three research domains described in this paper [34].

Intervention science—the development of new and improved health interventions—is another important area of research that may considerably overlap with implementation research [35],[36]. This science is mainly concerned with biomedical research, where the early stages of intervention development are often far removed from the field. However, as the development phase is nearing its completion, evaluation of the intervention effectiveness is usually done under real-life conditions [37]. Hence, towards the end of the development process, intervention development and implementation research tend to closely overlap.

Some terms are very similar to those used to describe the three research domains but have been used in different ways in the literature. Operations research usually refers to the use of mathematical optimization methods for operational decision making, but this is sometimes also called operational research [34],[35],[38].

Implementation science has been defined as the study of methods to promote the systematic uptake of research findings into routine clinical practice [39], and as such is complementary to knowledge translation (another term with its own diversity of definitions!) [40]–[42].

Implementation science has also been said to be similar to translational research or defined as ‘research that identifies barriers to proven interventions and that facilitates the creation of strategies to overcome them’ and in this sense implementation science is equal to implementation research as described above [15],[43],[44].

The term health services research suggests research that focuses on the service component of the health system, but it is often defined more broadly and used interchangeably with health systems research [17],[45].

Conclusion

Definitions are meant to clarify. But if too many different definitions for the same terms abound, so much confusion results that they become an obstacle to progress. Our aim here is not to establish which of these definitions are correct or to launch an intense debate about definitions that would distract from the need to support the research itself and the use of research results to improve health. Instead, we seek to provide a simple framework that is easily understood by both experts in the field and the managers, policy makers, and donors working to improve health systems and deliver better health care. We have tried to map the three main research domains, the research targets, and the users, and to highlight the importance of context and study design in the subsequent utility of the research findings.

Research on operational problems, on implementation strategies, and on health system challenges all involve multidisciplinary research that tends to use the same type of quantitative and qualitative research methods. But these three research domains differ in the type of research questions they address, in the way they are designed, and in their expected outcomes. Operational research and implementation research are action-oriented, respond to operational problems or implementation challenges, and work towards developing targeted solutions. Research on health system challenges addresses more complex, systems problems and is geared towards improving the understanding of what works for whom and under what circumstances. It provides guidance on what might work better within the system as a whole.

The three research domains are not mutually exclusive, and there are large overlapping areas. Research on operational problems is about local problem solving, but not all the problems it addresses are truly local. Many occur in a similar manner in multiple locations and may represent implementation problems for specific interventions that might be efficiently tackled by implementation research, or are representative of a systems problem that could be effectively addressed through health systems research. New implementation strategies are often designed to overcome specific health system failures, e.g., how to improve access to vaccination among children who are currently not reached by immunization services or home treatment for malaria in communities where formal health systems are not able to effectively provide such treatment. In such situations, implementation research develops innovative solutions that are in effect improvements of the health system and that could be regarded as health systems research, especially when these innovations affect more than just a single intervention. Such overlap between the three research domains provides opportunities for cross-fertilization that should lead to greater consideration by operational and implementation researchers of the wider system implications of their research. Accordingly, this should encourage the expansion of study designs and an appreciation of the feasibility of experimentation with different health system solutions.

We hope the above helps clarify what research to improve health systems is seeking to achieve. In essence it is quite simple: it involves operational research on operational problems, implementation research on implementation strategies for available interventions, and research on health system challenges as the main focus of health systems research. To improve health care delivery to poor populations, all of these research domains are very much needed.