Evaluation of Health Equity Impact of Structural Policies: Overview of Research Methods Used in the SOPHIE Project.

This article briefly assesses the research methods that were applied in the SOPHIE project to evaluate the impact of structural policies on population health and health inequalities. The evaluation of structural policies is one of the key methodological challenges in today's public health. The experience in the SOPHIE project was that mixed methods are essential to identify, understand, and predict the health impact of structural policies. On the one hand, quantitative studies that included spatial comparisons or time trend analyses, preferably in a quasi-experimental design, showed that some structural policies were associated with improved population health and smaller health inequalities. On the other hand, qualitative studies, often inspired by realist approaches, were important to understand how these policies could have achieved the observed impact and why they would succeed in some settings but fail in others. This review ends with five recommendations for future studies that aim to evaluate, understand, and predict how health inequalities can be reduced through structural policies.

The “inverse evidence law” states that the more a policy could affect population health, the harder it is to convincingly demonstrate this effect. In public health, considerable evidence shows the health impact of interventions aimed at modifying behavioral factors, such as smoking and drinking, through health education programs for young people or adults. Much less is known about the impact of policies that aim to address the wider economic, social, and physical environments in which people live (called “structural policies”). As a result, even though socioeconomic and environmental policies may be expected to improve population health and reduce health inequalities, little evidence backs up that expectation and quantifies the gains.

This implies a great need for generating evidence on the health impact of structural policies. Generating such evidence calls for approaches aligned with the principles of evidence-based medicine, which state that evaluations should consist of before-and-after measurements and include control groups. These approaches, however, must be adapted for the evaluation of structural policies, where researchers do not control the execution of these policies and where randomization is only a methodological ideal. Despite such limitations, we should strive to generate evidence on the population health impact of structural policies that is sufficiently convincing to a wide audience of scientists and policymakers.

Most structural policies are inherently complex. Their impact on population health may strongly depend on the ways in which they are implemented (e.g., with strict enforcement or not), the populations that they target (e.g., rich or poor), and the wider context (e.g., economic growth or recession). If structural policies are complex and variable in practice, it is problematic to just summarize the evaluation in terms of a simple “yes or no” effect and to transfer this result from the study population to other populations. When complexity and variability are high, we need alternative evaluation designs that are able to generate insights that take into account the complexity and to yield lessons that may be transferable to other settings and populations.

Mixed Methods Are Essential to the Evaluation of Structural Policies

In the SOPHIE project, various approaches were used to assess and understand the impact of structural policies on population health and health inequalities. The author of this paper was involved in a series of papers that evaluated the impact of urban renewal policies, policies regarding the integration of migrants, and economic crises.[1-10] In these studies, like in other SOPHIE studies, it was found that a combination of quantitative and qualitative approaches was essential to ascertain and understand the impact of structural policies and related developments.

Quantitative comparative approaches were applied mostly to assess whether a structural policy has had a demonstrable impact on health-related outcomes and whether this impact differed according to socioeconomic position. The key questions were whether these policies had a demonstrable effect and how large this effect might have been. Quantitative approaches were designed and applied in such a way as to generate evidence that was as strong as possible. In some studies, the SOPHIE project partners made comparisons between places, including comparisons between countries, between regions, and between small districts. In such comparisons, spatial variations in structural policies were related to spatial variations in health outcomes. Other studies of the SOPHIE project made comparisons over time, such as comparisons before and after the introduction of a new policy measure. In time series analyses, detailed data of trends over time were used to assess whether the introduction of a policy was followed—immediately or gradually—by a change in health outcomes. Some studies combined the spatial and time dimensions in quasi-experimental approaches. For example, some SOPHIE studies evaluated a structural policy by applying the before-after design both to the population exposed to the intervention and to a control population (e.g., those living in another country).

In addition to these quantitative approaches, the SOPHIE project applied in-depth analyses that included qualitative research methods. Such approaches aimed not only to assess whether and how much a structural policy affected population health but also how and why this effect was achieved. In other words, such analyses were used mainly to identify the mechanisms through which a structural policy could have influenced population health. Several methods have been used, such as concept mapping and multiple case studies. In particular, we explored the feasibility and informative value of realist approaches, including realist reviews and realist evaluations. On the one hand, these approaches tested expectations regarding how a structural policy could affect population health and why it succeeded or failed in achieving this effect. On the other hand, these approaches evaluated under which conditions, i.e., when or where, a certain policy would set in motion the mechanisms that would lead to an effect or the lack of effect.

Quantitative Comparative Approaches Yield Novel Knowledge

Many SOPHIE studies could apply quantitative approaches in which comparisons were made over time, between countries or places, or through a combination of both approaches. The common experience was that such studies yielded novel evidence on the impact of structural policies on the health of disadvantaged groups, such as lower socioeconomic groups or ethnic minorities.

For such quantitative comparative approaches to generate strong evidence, some common challenges had to be addressed. In the SOPHIE project, much experience was gained from cross-national comparisons. It was found that the strength of evidence depended on various conditions.

The study had to control for confounding by national factors other than the structural policies studied, such as measures of national income. A particular challenge was to control for other policies that were developed parallel to the structural policies of interest. In some studies, this problem was addressed by combining related policies in the construction of country typologies, such as welfare regimes.

The study had to maximize the number of countries studied, and the statistical analysis had to take into account the limited number of observations available (which is less than about 30 for Europe). In particular, the study had to find ways to apply statistical control for many potential confounders when the number of countries was limited (e.g., multilevel analysis with control for individual-level confounders).

The study had to apply internationally comparable information on the structural policies of interest. Problems in the comparability of the information may induce systematic or random measurement bias and thus affect study outcomes. Using comparable though simplistic indicators may be a solution but may at the same time cause problems of measurement validity.

Many SOPHIE studies found ways to address these challenges in their particular case. The solutions obtained often depended on the topic of interest and the data available.

Our general experience was that the application of time-trend analysis, rather than cross-sectional comparisons, yielded stronger evidence about the impact of structural policies on the population health. Time-trend analyses were applied in the SOPHIE project either for single countries or for several countries simultaneously. In the latter case, the study applied a quasi-experimental design to evaluate the impact of a structural policy. The experience of SOPHIE studies was that stronger evidence could be obtained if the following conditions existed:

The structural policies of interest rapidly changed over time, such as with the introduction of a new law or program. Sudden policy changes generate “natural policy experiments” that can be assessed in a quasi-experimental design. In contrast, when policies were implemented gradually over a longer period (e.g., 5 to 10 years), it was harder to identify and quantify their impact.

The available data sources were continuous (e.g., monthly, yearly or two-yearly) instead of covering only a few points in time. Continuous time series increase the possibility of accurately mapping trends in health-related outcomes during a policy change and isolating the policy impact from the effect of societal changes with a different timing.

No major developments occurred in other fields. Confounding may occur by concurring changes in other policy areas. Problems of confounding may call for the use of multi-country studies that compare “experiment” to “control” countries that differ especially with regard to the policy of interest.

The trends studies in SOPHIE differed in their execution and in the type of evidence generated, mostly due to differences in the topic of interest and the data available.

Realist Approaches Help to Address New and Vital Questions

When applied in isolation, quantitative approaches may result in simple “yes or no” verdicts regarding the impact of structural policies on health inequalities. Cost-benefit analysis may call for more detail about the magnitude of the observed impacts. Yet, this how much question is not the only question to ask. Realist approaches were used in the SOPHIE project to address how and why questions. More specifically, these approaches allowed SOPHIE researchers to start opening the “black box” and reveal the mechanisms through which structural policies could have affected the health of people. By generating evidence on such mechanisms, SOPHIE studies were able to formulate policy conclusions that were more nuanced and more aligned with the reality of people’s lives. Such detailed knowledge could also help us to formulate lessons that take into account the context and to predict when and where (i.e., under which conditions) a structural policy would have the promised impact.

The experience of SOPHIE researchers was that, to fully seize the potential of realist approaches, several challenges must be faced:

The strength of the evidence strongly depends on the quality and richness of the information obtained from primary sources or from published studies. For example, in a field dominated by quantitative studies, often very little qualitative information was available on the mechanisms of interest, and realist reviews could assemble only limited evidence on how a structural policy affected population health.

A clear and efficient working protocol must be developed. Published studies using realist approaches greatly varied in the ways in which the syntheses were executed and results were presented. Further standardization (see, e.g., www.ramesesproject.org) is likely to increase the efficiency and transparency of the work in this field.

On another level, those using realist approaches should find ways to deal with the high pressure on today’s scientists to publish. While realist approaches are more time-consuming than quantitative studies, the richness of their results does not imply that they are more likely to be published in high-ranking journals. On the contrary, our experience in SOPHIE was that realist studies may be harder to get published into high-ranking public health journals. In similar ways, because of the lack of simple answers and recipes, it may be more difficult to disseminate the results to some (but not all) policymakers or professionals.

Yet, mixed methods, such as realist approaches, are an indispensable complement to “black box” studies that only aim to demonstrate and quantify the impact of a structural policy. These approaches need to be further developed, applied, and promoted in public health, especially as an essential means to understand and predict the impact of future structural policies.

Implications for Future Research on the Health Impact of Structural Policies

In future work, stronger evidence on the health equity impact of structural policies could be obtained by further development and application of the quasi-experimental design, i.e., a pre- and post-measurement, intervention-control design. This design could be applied among countries or among areas within a country. As a general rule, the strength of evidence increases with (1) a larger number of control and interventions areas, (2) a more detailed measurement of intervention exposure and of potential confounders, and (3) the inclusion of more years of observation compared to a simple pre- and post-measurement.

Mixed methods inspired by realist thinking are essential to understand and predict the impact of structural policies. They can best applied together with a primary, prospective collection of data, such that expectations regarding “mechanisms of change” can be assessed with rigor and detail. Moreover, in the ideal case, these approaches are complemented with quantitative comparative approaches that test and quantify the expected health impacts. The application of both approaches would enable the research to assess whether a structural policy had the expected impact on population health, to understand how this impact was achieved or why this impact was not achieved, and to draw generalizable lessons from this best or poor practice.

These recommendations apply to different types of structural policies. On one side, broad structural policies and “regimes” may be most relevant for the reduction of health inequalities. However, they may be hard to assess for their precise impact, as the variety of mechanisms and context dependencies may be overwhelming. On the other side, specific structural policies of interest may be assessed with greater detail and rigor. However, the demonstrated impact may turn out to be deceptively small compared to the current magnitude of health inequalities. We shall have to find, each time again, the optimal balance between the brush and the tweezers.

In the SOPHIE project, we learned that the challenges listed above are not inherent to health inequalities research but to the assessment of structural policies on population health in general. The key challenge is to get stronger evidence of the impact of such policies on the health of people. If such an impact could be assessed for a population at large, it is possible in principle to also assess this impact for subpopulations stratified according to gender, ethnicity, and socioeconomic position. In practice, however, it may often be hard to assess inequalities in impact due to lack of statistical power in the available data.

The solution to the challenges mentioned above may sometimes call for the use of novel statistical approaches such as propensity score matching and difference-in-differences methods. Yet, in many SOPHIE studies, advanced statistical methods did not appear to offer a satisfactory solution to the key challenges. Often, the key issue was not about statistical analysis but about observation and measurement. In SOPHIE, for example, most comparative studies would not have been possible without the many European-wide surveys that were initiated in the last 10 to 15 years. We strongly recommend further development of these international surveys in the next decade.