'Doing' health policy analysis: methodological and conceptual reflections and challenges.

The case for undertaking policy analysis has been made by a number of scholars and practitioners. However, there has been much less attention given to how to do policy analysis, what research designs, theories or methods best inform policy analysis. This paper begins by looking at the health policy environment, and some of the challenges to researching this highly complex phenomenon. It focuses on research in middle and low income countries, drawing on some of the frameworks and theories, methodologies and designs that can be used in health policy analysis, giving examples from recent studies. The implications of case studies and of temporality in research design are explored. Attention is drawn to the roles of the policy researcher and the importance of reflexivity and researcher positionality in the research process. The final section explores ways of advancing the field of health policy analysis with recommendations on theory, methodology and researcher reflexivity.

KEY MESSAGES

Little guidance exists on how to do health policy analysis, concerning low and middle income countries. This paper explores ways of developing this field.

To advance health policy analysis, researchers will need to use existing frameworks and theories of the public policy process more extensively, make research design an explicit concern in their studies, and pay greater attention to how their own power and positions influence the knowledge they generate.

Introduction

Health policy analysis is a multi-disciplinary approach to public policy that aims to explain the interaction between institutions, interests and ideas in the policy process. It is useful both retrospectively and prospectively, to understand past policy failures and successes and to plan for future policy implementation. The case for undertaking policy analysis has been made by a number of scholars (Parsons 1995) and 15 years ago, in this journal, Walt and Gilson (1994) argued it was central to health reforms. However, there has been much less attention given to how to do policy analysis, what research designs, theories or methods best inform policy analysis. Reich and Cooper (1996) designed and have updated a software tool to help researchers and policy-makers analyse the political dimensions of public policies. Others, such as Varvasovszky and Brugha (2000), have designed guidelines for undertaking stakeholder analysis, as a part of health policy analysis. Bossert (1998) developed an approach to analyse choices for the decentralization of health sectors. Sabatier (1999, 2007) has explored different theoretical frameworks of the policy process (although not focusing on health). However, it is rare to find any scholarly work that explicitly explores the methodological challenges for researchers studying the health policy process.

This paper takes off from the conclusions drawn in the literature review by Gilson and Raphaely (2008, this issue), which identifies some of the gaps and weaknesses in the field of health policy analysis in low and middle income countries. The review notes the absence of explicit conceptual frameworks, little detail on research design and methodology, and a preponderance of single case studies on particular issues. It draws attention to the limited use of relevant theory to underpin analysis and the paucity of attempts to provide an explicit, explanatory focus. As the authors say, the main question is often ‘what happened’, to the neglect of ‘what explains what happened’. Researchers also rarely reflect on how their own positions shape their research interpretations and conclusions.

In this paper, we tackle some of these issues, looking at some of the reasons for the above deficiencies identified in the Gilson and Raphaely review, and make some suggestions for advancing the field.

First, to put the paper in context we begin by looking at the health policy environment, and some of the challenges to researching this highly complex phenomenon. Second, we argue for more attention to theory and frameworks and we consider the theoretical constructs often utilized in health policy studies. From there we move to methodology and study design, exploring the implications of case study research and temporality for health policy analysis; and then discuss the roles of the policy researcher, the importance of reflexivity and researcher positionality in the research process. The final section explores ways of advancing the field of health policy analysis.

The nature of the beast: the health policy environment

It is important to contextualize the health policy environment in order to understand the challenges to methodology and theory. While drawing on ideas and concepts from general policy analysis, most of which is derived from studies on high income countries, this paper focuses on health policy, and on low and middle income countries. Much of the theory from policy analysis in high income countries has resonance for health and developing countries, and can usefully inform research in those areas. However, transferring such concepts needs to be undertaken with caution. It is generally fair to say that the health sector has specific characteristics which affect the policy environment (and that differentiate it from other social sectors). The state may be both provider and purchaser of services, but also is involved in regulation, research and training among other functions. In service provision, it may be in competition or partnership with a private sector that it is also regulating. In undertaking its health care purchasing and regulatory functions, the state is usually heavily reliant on—and may lack—essential information that can only be provided by the sectors it is over-seeing. Information asymmetry is often a bigger problem than with the other social sectors. Health issues are often high profile and demand public responses. Health interests, ranging from professionals to the pharmaceutical industry, have traditionally been perceived to influence the policy process significantly. They are uniquely placed to do so because of their knowledge, technology, access to political processes and stake in life and death issues.

However, while these characteristics are generally typical, all scholars point out that they have to be contextualized in both place and time. Health policy environments in middle and high income countries will therefore differ from those in low income countries, where, for example, there are weaker regulations, regulatory capacity and monitoring systems; lack of purchasing power as a leverage to influence types and quality of services delivered; more patronage in political systems, and more reliance on external donor funds, among many other differences.

In spite of differences between high and low income countries, however, it is increasingly recognized that policy processes are changing everywhere. Initially policy analysis focused on the state—on the public or government sector—on politicians, bureaucrats and interest groups (Hogwood and Gunn 1984; Grindle and Thomas 1991). Over the past 10 years scholars have acknowledged a shift in the nature of policy and policy-making, which points to the involvement of a much larger array of actors in the policy process (Buse et al. 2005). The private sector, for example, including for-profit and not-for-profit organizations, large and small, has become an important player in health policy. Partnerships between public and private sectors have also changed the health policy environment. Furthermore, policy is increasingly shaped and influenced by forces (such as global civil society) outside state boundaries (Keck and Sikkink 1998). The growing literature around issues of globalization which emphasize changing spatial, temporal and cognitive dimensions (Lee et al. 2002) reflects the extent to which the world is perceived to have altered. There is less geographical distance between regions, exchanges have become faster, ideas and perceptions spread rapidly through global communications and culture.

This means that the policy environment is increasingly populated by complex cross-border, inter-organizational and network relationships, with policies influenced by global decisions as well as by domestic actions. The technological revolution has facilitated communications and relationships, both between governments and their advisers as well as between many different networks of actors outside of government. While government and its hierarchical institutions remain important, all policy analysis must also take into account a range of open-ended, more ad hoc arrangements which increasingly affect decision-making. Hajer and Wagenaar (2003, p. 8) talk about ‘new spaces of politics’ where there are ‘concrete challenges to the practices of policy-making and politics coming from below’. In their view, policy analysis has to become more deliberative: less top-down, involving expanded networks, and more interpretative, taking into account people's stories, their understandings, values and beliefs as expressed through language and behaviour.

Challenges for ‘doing’ health policy analysis

These changes in the policy environment make the analysis of policy even more complex but there are conceptual and practical problems that are specific to ‘doing’ health policy analysis.

The first challenge is that ‘policy’ can itself be defined in many different ways, with consequent implications for its study. It can be useful to think of health policy as embracing ‘courses of action (and inaction) that affect the set of institutions, organizations, services and funding arrangements of the health system’ (Buse et al. 2005, p. 6). Such policy may therefore be made within government, by non-government actors, and by organizations external to the health system. However, such processes of ‘making’ policy are not necessarily overt or clearly bounded. The ways in which decisions ‘emerge’ rather than taking place at a single point in time, and which are often unobservable to the researcher, can be particularly difficult to unpack and explain (Exworthy 2007). On the practical level, there are often many hurdles to accessing the many different, geographically widespread, actors, individuals, groups and networks involved in policy processes. Decision-making processes are often opaque, and obtaining relevant documents and papers can be problematic. Or, in contrast, an excess of information—where background documentation such as large volumes of email exchanges become available—can be burdensome and difficult to analyse. Participant observation can be difficult in practice.

There is also often a tension between the long-term nature of policy development and implementation and the short-term nature both of funding for policy research and of policy-makers’ demands for quick answers and remedies. Box 1 provides an example of some of these tensions. Hunter (2003) has called this the ‘curse of the temporal challenge’. Much health policy research is motivated (and attracts funding) by practical concerns such as the evaluation of existing programmes, and policy analysts are expected to deliver easily implementable recommendations within relatively short time horizons. The imperatives of quick policy ‘fixes’ may lead to reductionism.

Box 1 Applying health policy analysis in a fast moving policy environment

Brugha et al. (2002, 2004) have conducted a number of studies on global health initiatives such as the GAVI Alliance and the Global Fund to Fight AIDS, Tuberculosis and Malaria. Designed to gather and report the views of national-level stakeholders at very early stages in their implementation, the studies were sensitive. For the Global Fund, in particular, the research was perceived as premature, enabling country stakeholders to articulate criticisms, which it feared would have a deleterious affect on the need to raise significantly greater funds globally. The Global Fund Secretariat in Geneva requested that the scope of the study be widened to report its perspective, which was beyond the capacity and resources available to the researchers. In both studies, under pressure from funding agencies, the researchers reported findings within 9 months. The researchers resisted pressure from the Global Fund Secretariat to report interim findings to the Fund in late 2003, in advance of reporting back to country stakeholders.

The policy environment was very fluid, and the researchers found they were tracking a moving target—one where the Global Fund itself was responding to difficulties, changing guidelines, and proving to be a ‘learning organization’. Despite what the researchers viewed as rapid feedback of findings, given the need for rigour, the study funders and the Global Fund responded that the findings only confirmed what they had already learned through their own channels; and that these findings were being superseded by events. The study also found that the donor landscape had become even more complex by the time of the second phase of data collection (2004), because of the negotiation or establishment of other new HIV/AIDS financing instruments at the country level, such as the World Bank Multicountry AIDS Program and the US President's Emergency Plan for AIDS Relief (PEPFAR). New sources of funds were being negotiated sequentially or in parallel, which was distracting countries from implementation.

The dynamic nature of the policy environment made data collection and analysis difficult, and created sensitivities between the global initiatives, research funders and the researchers. One lesson was that maintaining a balance between independence and engagement with the entity being studied is difficult but key; building trust is essential if findings are to be taken on board.

There are also many other conceptual challenges to ‘doing’ policy analysis. For example, capturing and measuring levels of resources, values, beliefs and power of diverse actors is difficult; also, the notion of ‘power’—fundamental to policy analysis—is a highly contested concept. Yet it is often used as if there were little difficulty in agreeing what power is, where it lies, and how it is exercised. It can also be difficult to ‘tell the story’, without getting immersed in detail. Researchers have to find ways of organizing their analysis so that it provides a lens that represents but also explains a highly complex environment. As Gilson and Raphaely (2007) have shown, most health policy analysis is relatively intuitive, ad hoc, and the assumptions on which it is based are seldom identified.

In this paper we argue that the field of health policy analysis would be advanced if researchers approached it more systematically, developing clear and testable propositions about the issue they are studying, within explicit frameworks. Scholars have proposed a number of different theoretical frameworks to help researchers organize and focus their efforts to analyse the policy-making process. In the next section we look at some of these.

Approaches to health policy analysis: frameworks and theories

Frameworks

There are a number of widely used frameworks and theories of the public policy process.1 We discuss some of the more enduring examples; those which have been utilized most in the published public policy literature (Gilson and Raphaely 2007).

Frameworks organize inquiry by identifying elements and relationships among elements that need to be considered for theory generation (Ostrom 2007). They do not, of themselves, explain or predict behaviour and outcomes (Schlager 2007). The best known public policy framework is the stages heuristic (Lasswell 1956; Brewer and deLeon 1983). It divides the public policy process into four stages: agenda setting, formulation, implementation, and evaluation. Agenda setting is the issue sorting stage during which a small number of the many problems societies face rise to the attention of decision-makers. In the formulation stage, legislatures and other decision-making bodies design and enact policies. In the implementation stage, governments carry out these policies, and in the evaluation stage impact is assessed. Analysts have criticized the stages heuristic for presuming a linearity to the public policy process that does not exist in reality, for postulating neat demarcations between stages that are blurred in practice, and for offering no propositions on causality (Sabatier 2007). Nevertheless, the heuristic offers a useful and simple way of thinking about the entire public policy process, and helps researchers situate their research within a wider framework.

Walt and Gilson (1994) developed a policy analysis framework specifically for health, although its relevance extends beyond this sector. They noted that health policy research focused largely on the content of policy, neglecting actors, context and processes. Their policy triangle framework is grounded in a political economy perspective, and considers how all four of these elements interact to shape policy-making. The framework has influenced health policy research in a diverse array of countries, and has been used to analyse a large number of health issues, including mental health, health sector reform, tuberculosis, reproductive health and antenatal syphilis control (Gilson and Raphaely 2007).

As the number of actors involved in policy processes has expanded, so has interest in network frameworks. Seen largely as a tool for describing systems of interactions and interconnectedness between groups of actors, network analysis is a contested area, and there are many definitions of what a network is (Thatcher 1998). Most agree, however, that networks are clusters of actors linked together, who may be closely connected or loosely structured but are still capable of engaging in collective action. Policy networks are clusters of actors with interests in a given policy sector, and the capacity to help determine policy success or failure (Marsh 1998). There are many different ways of classifying networks. Marsh and Rhodes (1992) treat policy networks as a generic term, with policy communities at one end of a continuum and issue networks at the other. Policy communities are tight-knit networks with few participants who share basic values and share resources. There may be a strong inner or dominant core of actors, surrounded by a number of other, more peripheral members, all of whom make up a policy community. An issue network, on the other hand, brings together many different groups and individuals for a common purpose or cause, and may have little continuity in values or participation. Network analysis reflects the phenomenon of shared decision-making and exchange of resources to achieve their goals.

There is a debate among scholars as to whether the concept of networks is merely descriptive, or whether it has explanatory value, whether it is largely a Western concept, developed by looking at policy-making in the US and UK, and whether it has legitimacy for developing countries (Thatcher 1998). For some the network approach is not really a new analytical perspective, but signals rather a change in the policy environment and the political system. There are only a few empirical studies in health in developing countries which use network analysis as a lens (Schneider 2006; Tantivess and Walt 2008).

Theories

Explicit attention to theory development could benefit public policy practice by deepening our understanding of causality, and by bringing coherence to a fragmented body of knowledge. This does not imply a positivist approach to analysis, but a more thoughtful conceptualization of the policy process, that goes beyond ‘telling the story’.

Influential theories of the public policy process include multiple-streams (Kingdon 1984), punctuated-equilibrium (Baumgartner and Jones 1993) and top-down and bottom-up implementation (Sabatier 1999). Theories are more specific than frameworks, and postulate precise relationships among variables that can be tested or evaluated empirically. Kingdon (1984), whose multiple-streams theory is concerned with agenda setting, argues that the public policy process has a random character, with problems, policies and politics flowing along in independent streams. The problems stream contains the broad problems and conditions facing societies, some of which become identified as issues that require public attention. The policy stream refers to the set of policy alternatives that researchers and others propose to address national problems. This stream contains ideas and technical proposals on how problems may be solved. Political transitions, national mood and social pressure are among the constituent elements of the politics stream. At particular junctures the streams merge, and in their confluence windows of opportunity emerge and governments decide to act.

Several health policy scholars have adapted ideas from Kingdon's theory to explain how particular health issues have emerged on policy agendas. Reich (1995) identified additional elements that fed into the politics stream—organizational, symbolic, economic, scientific and politician politics—all of which favoured child over adult health through the 1990s, explaining the higher position of the former on the international health agenda. Ogden et al. (2003) also drew on Kingdon's ideas in their research on tuberculosis. They demonstrated that the emergence of the HIV/AIDS epidemic contributed to the opening of global policy windows, facilitating advocacy networks to promote DOTS (directly observed treatment, short-course) as a treatment of choice for tuberculosis.

Baumgartner and Jones’ (1993) punctuated equilibrium theory postulates that the policy-making process is characterized by periods of stability with minimal or incremental policy change, disrupted by bursts of rapid transformation. Central to their theory are the concepts of the policy image and the policy venue. The policy image is the way in which a given problem and set of solutions are conceptualized. One image may predominate over a long period of time, but may be challenged at particular moments as new understandings of the problem and alternatives come to the fore. The policy venue is the set of actors or institutions that make decisions concerning a particular set of issues. These actors may hold monopoly power but will eventually face competition as new actors with alternative policy images gain prominence. When a particular policy venue and image hold sway over an extended period of time, the policy process will be stable and incremental. When new actors and images emerge, rapid bursts of change are possible. Thus, the policy process is constituted both by stability and change, rather than one or the other alone

Shiffman et al. (2002) examined the emergence of global political attention for the control of tuberculosis, malaria and polio, finding that patterns conformed to punctuated rather than rational or incremental models of the policy process. Baumgartner and Jones themselves apply the theory to a health concern. They show that little changed in US tobacco policy in the first half of the 20th century, as the subject generated little coverage in the US media, government supported the industry through agricultural subsidies, and the product was seen positively as an important engine for economic growth. Beginning in the 1960s, however, health officials mobilized, health warnings came to dominate media coverage, and the industry was unable to counter a rapid shift in the policy image that focused on the adverse effects of tobacco on health.

Multiple implementation theories have been dominated by a discourse as to whether decision-making is top-down or bottom-up, or a synthesis of the two (Sabatier 1999). For example, Dye (2001) argues that even in a democracy like the United States, public policy is made from the top down, not from the bottom up. In his view, public policy reflects the values, interests and preferences of the governing elite. Dye separates policy development from implementation, admitting that bureaucrats may affect policy in implementation, but suggesting that all decisions are monitored to ensure they are not altered significantly. Lipsky (1980), on the other hand, describes implementation of policy as highly influenced by ‘street level bureaucrats’—front-line staff who can change policies significantly—and others have developed this approach (e.g. Hjern and Porter 1981). Much of the literature focuses on the gap or deficit between policy objectives and actual implementation (Hill and Hupe 2002). Saetren (2005) reviewed all implementation literature published and concluded that while most of the studies focused on health and education, they were predominantly of high income, Western countries. There are a few notable exceptions (Kaler and Watkins 2001; Kamuzora and Gilson 2007).

Researchers have also applied a range of social science theory from outside of policy studies to health policy analysis, drawing these from disciplines such as sociology, anthropology and organizational management (Gilson and Raphaely 2007). Murray (2007), for instance, in the example given in Box 2 draws on sociological theory concerning consumption to understand the impact of private medical service financing mechanisms on maternity care in Chile. Others have used social construction theories to explore why public policies sometimes fail in their objectives. Ingram et al. (2007), for example, focus on how public policy-makers may construct target populations positively or negatively, leading to unfair distribution of resources that perpetuate health inequalities.

Box 2 Applying theory from the sociology of consumption in a longer term retrospective policy analysis to maternity care in Chile in the 1980s and 1990s (Murray and Elston 2005; Murray 2007)

This study was initiated after the Chilean Minister for Women's Health Services expressed concerns over rising national rates of caesarean section delivery (37% of births in the mid-1990s), and a preliminary analysis of health fund statistics revealed that caesarean section rates were twice as high in women who had private health insurance plans than in women who were receiving delivery care financed through the National Health Fund (59%:28.8% in 1994; Murray and Serani 1997).

In order to understand this problem, the study examined healthcare financing decisions in the 1980s through to experience of care up to the present. The analysis was informed by theory from the sociology of consumption. Healthcare services for pregnancy were conceptualized as a complex good that is produced and consumed in a production/consumption ‘cycle’ (Edgell et al. 1996). The cycle has four dimensions: mode of provision, the conditions of access, the manner of delivery and the experience of consumption. Implicit to such an approach is an emphasis on social processes situated in time and place. Data to inform the analysis included documentation relating to national policy change, trends data from health services and insurance funds, and interviews with policy-makers and administrators. In-depth interviews with health practitioners and service users investigated patients’ and practitioners’ perspectives on the structure, process, delivery and consumption of maternity care. A postnatal questionnaire and medical notes review provided quantifiable detail on medical care practices and on women's perceptions of them. In an approach similar to that employed in framework analysis (Pope et al. 2000), a series of general and then increasingly more specific questions were elaborated, enabling testing of alternative explanations of phenomena.

The findings traced how neoliberal financial reforms initiated at the beginning of the 1980s under a military dictatorship which aimed to reduce fiscal support for health care had led to the roll out of private health insurance organizations and new patterns of organization of medical care. These ultimately resulted in changes in service delivery and the experience of consumption (including the programming of births so that obstetricians could manage fragmented work schedules, and users could avoid payment of unsocial hours fees) which led to high rates of caesarean delivery. Using the consumption cycle framework helped to understand the interface between macro, meso and micro levels over time, and the relationship between the policy and its healthcare outcomes.

Designing health policy studies: methodology

Few health policy analyses on low and middle income countries explicitly discuss research design, and the field would benefit from more reflection on the range of approaches that could be used, and their relative benefits. Most investigations are case studies, whether or not researchers identify them as such (Gilson and Raphaely 2007). Policy decisions often have their roots in longer term processes and the choice of time frames for research is an important factor. Temporal issues thus also affect research design. Addressing these factors is an important aspect of research design.

Research design: case studies

Case studies are in-depth investigations of a single instance of a phenomenon in its real-life context (Yin 1994). They are to be distinguished from other research designs, such as controlled comparisons, formal modelling, quantitative analyses and randomized-controlled experiments. A substantial body of work offers guidance on case study methodology (Yin 1994; Brady and Collier 2004; George and Bennett 2004; McKeown 2004; Yin 2004).

Case study methodologists argue that asking just a few basic questions about the case can improve the value of the study considerably. First, what is it a case of (George and Bennett 2004)? Is it, say, an example of health policy implementation failure, of effective transfer of a health policy from one country to another, of health policy network influence on agenda setting, of the influence of political factors on health policy evaluation? Sometimes cases may be clearly identifiable by the researcher at the start of the study, sometimes they may be constructed or re-constructed during the course of the research as the analysis reveals their defining characteristics (Ragin and Becker 1992). The process of clarifying ‘the case’ enables the researcher to specify a body of knowledge to which he or she may make a contribution. Second, why is this case a useful one to study (George and Bennett 2004)? Does it offer the possibility of comparing the explanatory value of alternative theories? Is it an unusual example of policy effectiveness, potentially offering insight into factors that facilitate policy impact? It is consideration of these issues that help the researcher to select the tools and theories that might frame a study, and to determine which methods will be used.

A study on the global availability of praziquantel, a drug for the tropical disease schistosomiasis, offers clear answers to both questions (Reich and Govindaraj 1998). The researchers document how the discovery of this effective drug did not automatically result in it reaching the poor in developing countries. They identify this as a case of the gap between drug development and drug availability for the poor—an issue of concern for many tropical disease pharmaceuticals—and highlight the usefulness of this case in revealing the influence of political and economic factors on this gap. Their careful case selection and classification enable them to suggest a set of policy prescriptions on surmounting the drug development–availability gap, recommendations that apply well beyond the case itself.

Another means of facilitating generalization is increasing the number of cases. Doing so is not always easy, since investigating even a single case is a time and resource intensive process that requires careful consideration of historical and contextual influences. Comparative case studies may introduce the further challenges of working across multiple languages and cultures. It can also be difficult to find sufficient funds for undertaking such research. Yet there are several strong examples in the health policy field.

Lee et al. (1998) used matched country comparisons to investigate factors influencing the development of strong national family planning programmes. They conducted four country comparisons: Bangladesh/Pakistan, Tunisia/Algeria, Zimbabwe/Zambia and Thailand/Philippines. Each pair was matched on socio-economic characteristics, but differed on the strength of the family planning programme. The comparisons enabled the researchers to point to three factors that shaped the development of effective programmes: the formation of coalitions among policy elites, the spread of policy risk, and the country's financial and institutional stability. Walt et al. (1999) considered individual cases of donor aid coordination in the aid-dependent countries of Bangladesh, Cambodia, Mozambique, South Africa and Zambia to develop generalizations concerning the origins and effectiveness of aid coordination mechanisms. Shiffman (2007) explored agenda-setting for maternal mortality reduction in five countries: Guatemala, Honduras, India, Indonesia and Nigeria. He identified nine factors that shaped the degree to which this issue emerged as a political priority. He found that while international donors played a role, even more critical were efforts by national champions.

Collectively, these different examples highlight the value of cross-country comparative study approaches, where comparisons between similar (and different) country contexts can help disentangle generalizable from country context-specific effects in policy adaptation, evolution and implementation. Comparisons can be incorporated into study design, ad-hoc, as in the case of Lee et al. (1998) above; researchers can select as case studies several countries with a shared feature (Brugha et al. 2005), see Box 1; or comparison can be made post-hoc, as in the case of Walt et al. (1999). Clearly, multi-country studies are more time and resource intensive.

Research design: temporal issues

The extent to which policy analyses are focused on contemporary policy, or are retrospective and take a longer view of policy development, will have implications for methods and for the questions that are asked. Short horizon approaches are sometimes appropriate and necessary for responsiveness in some fast moving political circumstances; for example, work on the global health funds that was conducted early in their implementation (see Box 1). Concurrent or ‘prospective’ analysis of policy processes may be utilized in order to support and manage policy change, and this approach is explored in some detail in a companion paper (Buse 2008, this issue). Stakeholder analyses that focus on position, power, players and perception (Roberts et al. 2004) are often central to this type of work.

Policy evaluation requires a longer timeframe than political exigencies often allow. Sabatier suggests that ‘a decade or more’ is the minimum duration of most policy cycles, from emergence of the problem through sufficient experience with implementation to render a ‘reasonably fair evaluation’ of impact (Sabatier 2007, p. 3). A long span of study of the policy process may well be needed to identify unintended and unexpected consequences of policy. For example, a study may be triggered by concern over a controversial health care outcome, or an observed inequity of delivery or access, and not by a particular ‘policy event’ itself. Longer-term analysis or ‘backward working’ from a trigger statistic or social phenomenon may be necessary to reconstruct a policy implementation trajectory. This will entail mapping out its social and historical context, and how the policy unfolded over time in order to understand its eventual impact.

Such longer-term retrospective studies throw up particular challenges for data collection and analysis, including recall bias. In the case outlined in Box 2, the legislation introducing private health insurance structure to Chile was passed in 1981, the primary data collection interviews with practitioners and users took place 14 to 16 years later and trends analysis continued for some years after that. There is no simple way of knowing when is the ‘best’ time to initiate such work. In this particular case the impetus was a concern over rising caesarean section rates within the Ministry of Health, which in turn had been influenced by international debates. Multiple corroborative sources of different kinds (qualitative and quantitative) become particularly important, including different generational perspectives from interviews.

Positionality and health policy analysis

One of the issues facing health policy analysts is how they are viewed or ‘situated’ as researchers, their institutional base, perceived legitimacy, and prior involvement in policy communities. This is critical to their ability to access the policy environment and conduct meaningful research, especially in policy analyses that require engaging with policy elites (Shiffman 2007), and when investigating sensitive issues of ‘high politics’ (Box 1). Yet in contrast to other disciplines in the social sciences (e.g. Lincoln 1995; Rose 1997), the policy analysis literature seldom explicitly discusses researcher ‘positionality’ and its possible impact on the research process.

With respect to positionality, the classic distinction often made is between ‘insiders’ and ‘outsiders’, where insiders may be both participants and researchers (participant-observers) of the policy process, or alternatively, country-based rather than foreign researchers. Class, caste, gender, age, ethnicity and profession may also be highly relevant to insider/outsider status in some health policy research contexts. In seeking to unravel complex policy dynamics, insiders may see things quite differently to outsiders, with implications for the data collected and the interpretation of research findings. As explained by Merriam et al. (2001, p. 411), ‘… being an insider means easy access, the ability to ask more meaningful questions and read non-verbal cues, and most importantly, to be able to project a more truthful, authentic understanding of the culture under study. On the other hand, insiders have been accused of being inherently biased … the outsider's advantage lies in curiosity with the unfamiliar, the ability to ask taboo questions, and being seen as non-aligned with sub-groups.’ In the study cited in Box 2, an ‘outsider’ interviewer was found to be particularly useful for persuading the interviewees to give fuller explanations than they might otherwise have felt necessary.

Policy research teams that combine both insiders and outsiders and that engage all team members in active discussions of findings during data collection and analysis may therefore yield the richest and most comprehensive understanding of the policy process (as proposed by Buse 2008, this issue). However, implementation of such a model is not easy. While policy research designs may recognize the value of this team approach, the reality is that policy analysis is only emerging and has yet to establish its legitimacy as a field within developing countries, ‘insider’ policy researchers are hard to recruit and ‘outsider’ researchers may be expensive and time-constrained.

Researcher positionality has implications not only for access to data but also for knowledge construction. Research may be based on externally imposed categories and constructs. Parkhurst (2002), for example, argues that explanations of the decline in HIV prevalence in Uganda were driven at first by the need to hold up a success story on HIV in Africa, leading to an overly simplified analysis of both the extent and the ingredients of this success by UN and donor agencies. Short timeframe policy research initiated in response to external political imperatives runs a real risk of superficial and decontextualized analyses of the policy process that reveal only part of the picture.

‘Position’ can influence the issues that researchers focus on and therefore the research agendas created and the research questions asked. Scholars have noted that positionality is tied to questions of power and resistance, and in the context of health policy research the North/South dynamics need to be acknowledged. Tensions can occur between northern researchers who have the funds, and southern researchers who have insider knowledge and understanding. As Staeheli and Lawson (1995, p. 332) point out, ‘… researchers cannot escape the power relations even when they wish to do so. Western researchers are in a position of power by virtue of being able to name the categories, control information about the research agenda, define interventions and come and go as research scientists.’

National policy researchers, for their part, tend to be invested in their policy environments in some way, even if they operate from an independent research base or are not involved in the specific policy process under investigation. Researchers linked to particular policy environments will naturally be inclined to focus on specific and contemporary features of the particular policy space, rather than more universal themes that cut across policy or country contexts. They may also be more concerned with developing policy relevant conclusions than new theoretical or methodological understanding. Over time, such researchers will typically move in and out of various policy networks: sometimes directly implicated in policy communities, other times more loosely as part of issue networks or epistemic communities (Haas 1992) that provide public commentary on policy developments. Increasingly, funders are mandating researchers to engage in research translation, forcing them to become policy actors. Being an interested actor may have both advantages and disadvantages for generating new policy knowledge. Maintaining a degree of legitimacy amongst a wide range of actors may be crucial to the ability to conduct future research. In highly contested policy spaces this may involve complicated balancing acts that limit the ability to ask certain questions. The intense polarization that has characterized HIV/AIDS in South Africa is one such example (Fassin and Schneider 2003). Or where researchers take strong activist stands, they may become ideologically positioned in ways that may both open and close doors in the research process (Narayan 2007).

Conclusions: advancing health policy analysis

Schlager (1997, p. 14) observed that the field of policy studies is characterized by ‘mountain islands of theoretical structure, intermingled with and occasionally attached together by foothills of shared methods and concepts, and empirical work, all of which is surrounded by oceans of descriptive work not attached to any mountain of theory’ (cited in Sabatier 2007, p. 323). We think this statement accurately characterizes the field of health policy analysis as well. Through this discussion of theory, methodology and positionality in health policy analysis, it is clear that there are a number of ways research in this field could be strengthened:

On theory

More critical application of existing frameworks and theories of the public policy process to guide and inform health policy inquiry, while recognizing the need and potential to contribute to theory development as a goal of health policy analysis, with consequent benefits for practice.

Greater use of social science theories (for example, of organizations or street level bureaucrats) that come from outside of policy studies to inform health policy analysis.

On methodology

Making research design an explicit concern in all health policy analyses, and identifying and justifying the type of design in published articles.

Drawing on the growing body of work on case study research methods in order to enhance the quality of case study inquiry in the field.

Clearly identifying the type of ‘case’ and the unit of analysis, and considering the need for multiple cases and comparators.

Making assumptions and propositions explicit, logical and interrelated, and open to being tested empirically, so as to explain general sets of phenomena.

Making the case to funding agencies to support more comparative work in health policy analysis in order to expand the generalizability of results and develop greater certainty concerning causality. This case will be strengthened by the willingness of researchers to collaborate across institutions, countries and regions.

Exploring other approaches to synthesis (e.g. through large sample studies which employ quantitative and qualitative methods) as well as retrospective studies which draw on research from different disciplines in one policy domain or set of countries.

On positionality

In conclusion, we argue that if those who conduct, teach, commission, fund or publish public health policy research take on board some of these points, there will be a significant improvement in research approaches to health policy analysis, especially in relation to low and middle income country settings. If applied, these recommendations will also provide lessons on the evolution of policy implementation successes and failures as well as tools to assist policy-makers in evaluating and planning current and future policies.

Greater reflexivity on the part of researchers, that involves an analysis of their own institutional power, resources and positions (in much the same way they would analyse actors in the policy process) and their role in defining research agendas and generating knowledge (rather than assuming themselves to be ‘objective’ and ‘independent’).

Greater attention to policy research team composition and roles, including insiders and outsiders, which can relate to nationality but also to multiple roles. Researcher positionality may need to be negotiated and also reflected upon, considering how it may influence data collection and interpretation.

Long-term approaches to building policy analytic capacity. This would include acknowledging and providing space for different policy research agendas arising from different researcher positionalities, and building a critical mass of policy analytic capacity to enable this.