Applying economic evaluation to public health interventions: the case of interventions to promote physical activity.

BACKGROUND: This paper explores the application of alternative approaches to economic evaluation of public health interventions, using a worked example of exercise referral schemes (ERSs).
METHODS: Cost-utility (CUA) and cost-consequence analyses (CCA) were used to assess the cost-effectiveness of ERSs. For the CUA, evidence was synthesized using a decision analytic model that adopts a lifetime horizon and NHS/Personal Social Services perspective. Outcomes were expressed as incremental cost per quality-adjusted life-year (QALY). CCA was conducted from a partial-societal perspective, including health and non-healthcare costs and benefits. Outcomes were reported in natural units, such as cases of strokes or CHD avoided.
RESULTS: Compared with usual care, the incremental cost per QALY of ERS is £20 876. Based on a cohort of 100 000 individuals, CCA estimates cost of ERS at £22 million to the healthcare provider and £12 million to participants. The benefits of ERS include additional 3900 people becoming physically active, 51 cases of CHD avoided, 16 cases of stroke avoided, 86 cases of diabetes avoided and a gain of ∼800 QALYs.
CONCLUSIONS: CCA might provide greater transparency than CUA in reporting the outcomes of public health interventions and have greater resonance with stakeholders involved in commissioning these interventions.

Background

Economic evaluation is increasingly used to inform decisions on how healthcare resources are allocated. Much of the effort to date has focused on the application of economics to innovative drugs and medical technologies, under the auspices of health technology assessment. However, there is increasing interest in exploring the cost-effectiveness of a wider range of healthcare interventions, including public health programmes.[1] The rationale for this is clear: public health interventions consume health (and other public sector) resources and as such are associated with an opportunity cost. That is, the money spent on public health interventions could be allocated to other healthcare activities and it is important to determine whether public health interventions offer comparable or superior health outcomes for a similar level of expenditure.

Many of the prevailing methods of economic evaluation in health care are based on a pharmaceutical paradigm,[2] which assumes that there is high-quality evidence on their effectiveness, typically derived from randomized controlled trials (RCTs) conducted prior to widespread uptake in the health service. Those familiar with public health interventions will recognize that these methods may have severe limitations when applied to interventions such as health promotion. A number of economists have recognized the shortcomings of the prevailing methods of economic evaluation when applied to public health.[2,3] Particular concerns include the absence of robust evidence on effectiveness from RCTs which typifies many public health interventions; the fact that costs and benefits of public health interventions may be accrued well beyond the health service and the relevance of commonly used endpoints [quality-adjusted life-year (QALY)] to public health stakeholders.

The National Institute for Health and Clinical Excellence (NICE) recognizes some of these limitations in the Methods Manual that is intended to support the development of public health guidance.[1] Within this, NICE recommends that public health interventions should be subject to economic evaluation to determine their cost-effectiveness and that the NICE Reference Case should be adopted wherever possible, comprising cost-utility analysis (CUA) from a healthcare perspective with outcomes reported in the form of incremental costs per QALY. However, the guidance acknowledges that the reference case has limitations in its application to public health and also suggests alternative methods that might be adopted. One favoured method is cost-consequence analysis (CCA), which reports costs and outcomes in a disaggregated fashion, unlike CUA that adopts a composite outcome, the QALY. CCA might seem particularly attractive for evaluating public health interventions as it allows stakeholders from the health sector and beyond to identify where costs and benefits might be accrued (e.g. health sector, local authority).[1] CCA also offers greater potential to consider the impact of programmes on health inequalities by identifying particular sub-groups of the population who may benefit more. CUA typically pays little attention to inequalities as it is founded on the principle that a QALY is of equal value, regardless of the recipient.[2,4] Whilst methods exist for equity weighting QALYs, in practice these are rarely used due to insufficient evidence on the appropriate weights.[5]

This paper explores the issue of the appropriate method to apply to economic evaluations of public health interventions, using a worked example to compare the alternative approaches in an evaluation of exercise referral schemes (ERSs). ERS is a common intervention used to encourage physical activity in primary care.[6] In an ERS, people who are sedentary and/or have risk factor(s) for conditions known to benefit from physical activity are referred by a primary care professional to a third party service (often a leisure centre), which then prescribes and monitors an exercise programme tailored to the individual needs of the patients.[7]

The objective is to present information on the cost-effectiveness of ERS to promote physical activity using CUA and CCA. The original research presented herein was conducted as part of an assessment of the clinical and cost-effectiveness of ERS to promote physical activity, funded by the NIHR Health Technology Assessment Programme.

Methods

Table 1 describes the inputs to the model used for CUA. This includes the estimates of the effectiveness of ERS on physical activity levels, intervention costs associated with ERS, probability of experiencing an outcome, utility values and life years associated with each outcome.

Table 1

Estimates of the inputs to the model used for the CUA

Input	Value	Data source
Effectiveness
Probability of becoming active after exposure to ERS	0.345	Pavey et al.[8]
Probability of becoming active after exposure to usual care	0.297	Pavey et al.[8]
Intervention costs
Cost of the intervention per participant to the providers	£222	Pavey et al.[8]
Probability of experiencing an outcome associated with physical activity
Probability of experiencing CHD when active	0.014	HSE[12]; Shaper[13]
Probability of experiencing CHD when sedentary	0.027	HSE[12]; Shaper[13]
Probability of experiencing stroke when active	0.011	HSE[12]; Herman et al.[14]
Probability of experiencing stroke when sedentary	0.015	HSE[12]; Herman et al.[14]
Probability of experiencing type II diabetes when active	0.022	HSE[12]; NICE[6]
Probability of experiencing type II diabetes when sedentary	0.044	HSE[12]; NICE[6]
Inputs used in calculating QALYs/treatment costs
Utility/health state value of being in CHD state	0.55	Kind et al.[18]; NICE[6]
Utility/health state value of being in stroke state	0.52	Kind et al.[18]; NICE[6]
Utility/health state value of being in type II diabetes state	0.7	Kind et al.[18]; NICE[6]
Utility/health state value of being in a non-disease health state	0.83	Kind et al.[18]; NICE[6]
Average age of cohort (in years)	50	HSE[12]
Average age of mortality (in years)	84	ONS[19]
Assumed average age of onset of a disease health state (in years)	55	NICE[6]
Life years remaining after onset of CHD	18.41	NICE[6]; ONS[19]
Life years remaining after onset of stroke	5.12	NICE[6]; ONS[19]
Life years remaining after onset of type II diabetes	28.13	NICE[6]; ONS[19]
Lifetime treatment costs*/QALYs associated with health states (per person)
Lifetime treatment costs associated with CHD state	£17 728	NICE[6]
Lifetime treatment costs associated with stroke state	£1965	DH[20]
Lifetime treatment costs associated with type II diabetes state	£50 309	Currie et al.[21]
Lifetime treatment costs associated with non-disease health state	—	—
QALYs associated with CHD state	9.94	Kind et al.[18]; NICE[6]
QALYs associated with stroke state	5.15	Kind et al.[18]; NICE[6]
QALYs associated with type II diabetes state	14.18	Kind et al.[18]; NICE[6]
QALYs associated with non-disease health state	17.18	Kind et al.[18]; NICE[6]

*Costs are in 2010 prices.

Evidence on the effectiveness of ERS was identified from a meta-analysis[8] conducted alongside the economic evaluation. This was based on ‘intention-to-treat’ analyses and showed that people exposed to ERS were more likely [relative risk (RR): 1.11; 95% confidence interval (CI): 0.99, 1.25] to become physically active compared with those exposed to usual care. The usual care is no active intervention, the recognized alternative in a sedentary population. This acknowledges that sedentary individuals may participate in physical activity without an intervention, although the probability of doing so is assumed to increase after exposure to ERS. In line with the effectiveness literature and physical activity for health guidance,[9] physically active state is defined as doing at least 90–150min of at least moderate intensity physical activity per week.

Evidence on the costs of ERS was identified from Isaacs et al.,[10] which included a detailed, bottom-up cost of ERS; providing resource use in a health service/local authority that comprises provision of facilities, exercise trainers and administrative support for ERS. The validity of the resource use and cost estimates were assessed by an expert advisory group (including clinicians, exercise scientists and health economists) and judged to be representative of ERS in current practice. Cost estimates are inflated to 2010 prices, using the consumer price index, for this current analysis.

Outcomes associated with physical activity (Table 1) were derived from a previous analysis developed to inform NICE guidance on physical activity.[6] For the current analysis, the conditions that are associated with physical activity included coronary heart disease (CHD), stroke and type II diabetes. Although many other conditions are thought to be associated with physical activity, these three conditions were selected because robust quantifiable evidence exists on their relationship with physical activity.[11] Evidence of the effect of physical activity on these conditions is derived from systematic searches[8] and the Health Survey for England (HSE) 2006 (survey year focused on cardiovascular disease and risk factors). HSE is the main data source on morbidities in England.[12] The probability of developing CHD, stroke or diabetes among sedentary individuals is generated from the prevalence of these conditions in that population, using the HSE 2006 data. The probability of developing these conditions among active individuals are derived using RR estimates to adjust the probabilities for the sedentary individuals.[6,13,14] The physical activity levels and population used to measure the RR estimates match those of the cohort considered here. In line with the literature on ERS,[6] a number of assumptions were made in generating these estimates. First, the risk estimates were assumed to be equivalent to the risk of developing those conditions over life-time. Second, the risk of experiencing any of these conditions was assumed to be independent of the risk of experiencing other conditions. Third, individuals were assumed to experience only one condition within the model. Whilst a potential limitation of this approach is that it inadequately accounts for confounders, data constraints precluded their inclusion.

For the CUA, evidence was synthesized using a decision analytic model (Fig. 1) that adopts a lifetime horizon and NHS/Personal Social Services perspective. The model considered a cohort of sedentary adults (40–60 years) who are exposed to an ERS that is structured as leisure centre-based intervention. This age group reflects the evidence on the effectiveness of ERS.[8] Individuals enter the model as either exposed to ERS or usual care. A physically active individual is assumed to have both improved life expectancy and quality of life, due to reduced risk of developing each of the morbidities considered in the model. The probability of becoming physically active is adjusted to reflect the effectiveness of ERS.[8] The lifetime risk of developing ill-health is adjusted based on an individual's level of physical activity. The primary endpoint for the analysis was QALYs. Future costs and benefits are discounted at a rate of 3.5% per annum.[15]

Fig. 1

Diagram of the decision analytic model used in the CUA. The model adopts a lifetime horizon and NHS/Personal Social Services perspective. A cohort of sedentary individuals exposed to an ERS is considered. Label nodes () signify that the branches indicating the outcomes (i.e. CHD, stroke, diabetes and none) apply.

Deterministic sensitivity analysis (one-way, scenario and extreme values analysis) were used to address uncertainty around parameters considered to be key drivers of the cost-effectiveness of ERS. These included uncertainties around the effectiveness of ERS and changes in the cost of ERS to account for costs incurred by participants and providers.[10] The effectiveness of ERS varied according to estimates of uncertainty reflected in upper and lower limits of 95% CI of the RR estimate from the meta-analysis.[8] Sensitivity analysis also considered how a less intensive ERS might look, using evidence on a walking-based intervention (as opposed to a leisure centre-based intervention) from Isaacs et al.[10] Further sensitivity analyses considered ‘best-case’ and ‘worst-case’ scenarios which considered the combined effect of extreme values of effectiveness and cost.

For the CCA, outcomes were reported in natural units, such as the cases of CHD avoided. Further literature searches were conducted to identify outcomes that are associated with physical activity but where the magnitude of benefit is poorly defined[8]; the CCA indicates the expected direction of change but does not seek to quantify the benefits. Costs were reported in monetary units (2010 prices) and consideration was given to whether costs were incurred by the health service or the individual participating in ERS. The analysis was conducted from a partial-societal perspective, including health and non-healthcare costs and benefits. The intervention and its cost remain unchanged from the CUA.

Results

Cost-utility analysis

The estimated incremental cost-effectiveness ratio (ICER) suggests that the incremental lifetime cost of ERS is ∼£170. Total costs and outcomes are presented in per person terms. This additional investment generates ∼0.008 QALYs over the lifetime of an individual. The cost per QALY of ERS is £20 876 and can be considered cost-effective at £30 000.[1] Adopting this threshold results in ERS generating a net benefit, i.e. the value of the health gains measured in monetary terms exceeds the cost of ERS.

Table 2 shows the findings of the sensitivity analyses, indicating the impact of the variation in parameter estimates on the cost-effectiveness of ERS. Assuming a less-intensive ERS or more-effective ERS resulted in an ICER lower than the base case and <£30 000 per QALY. Conversely, including participants costs led to an ICER >£30 000 per QALY, whilst a less-effective ERS resulted in ERS being dominated by usual care (negative ICER)—that is ERS is more expensive and leads to QALY losses. The scenario analysis suggests that in the worst-case scenario, ERS was dominated by usual care. In the best-case scenario, the ICER was <£700 per QALY. Overall, the ICER was most sensitive to changes in the scenarios (best cases of cost and effectiveness).

Table 2

Cost-effectiveness results (after deterministic sensitivity analyses) comparing ERS with usual care

Parameters/scenarios	How data were adjusted for in the model	Incremental cost per person	Incremental effect per person (QALY)	ICER
Base case analysis	—	£170	0.008	£20 876
Parameters
Intervention costs to participants	Costs of intervention was varied from £222 to £342 (including costs to providers and participants)	£290	0.008	£35 652
Less intensive ERS	Costs of intervention was varied from £222 to £110	£58	0.008	£7085
Effectiveness of ERS (based on lower limit of 95% CI)	Probability of becoming active after exposure to ERS was varied from 0.336 to 0.294	£226	−0.001	Dominateda
Effectiveness of ERS (based upper limit of 95% CI)	Probability of becoming active after exposure to ERS was varied from 0.336 to 0.371	£122	0.015	£7947
Scenarios
Worst-cases of cost and effectiveness	Worst-case cost (£342) and worst-case effectiveness (0.294)	£346	−0.001	Dominateda
Best cases of cost and effectiveness	Best-case cost (£110) and best-case effectiveness (0.371)	£10	0.015	£679
Worst-case cost and best-case effectiveness	Best-case cost (£110) and worst-case effectiveness (0.294)	£242	0.015	£15 734
Best-case cost and worst-case effectiveness	Worst-case cost (£342) and best-case effectiveness (0.371)	£114	−0.001	Dominateda

aERS more costly and less effective than usual care.

Cost-consequence analysis

Table 3 summarizes the outcomes of the CCA. In an attempt to present meaningful, population-level outcomes, the analysis considers a cohort of 100 000 individuals who might be eligible for ERS. The cost of ERS for this cohort is estimated to be £22 million to the health care provider and £12 million to the participants and total cost of £33 million. This is based on a leisure centre-based intervention as defined in the CUA.

Table 3

Results of cost-consequence analysis (using a cohort of 100,000 individuals)

Measures in analysis	Potential impact of ERSs on measures
Costs
Intervention cost to providers	£22 200 000 (2010 prices)
Intervention cost to participants	£12 000 000 (2010 prices)
Benefits
Physically active state	3900 additional physically active people
Non-disease health state	152 extra people in non-disease health state
Mental health
Anxiety	Reduced anxiety in participants with the magnitude of the effect size being 0.219
Depression	Increased the success rate to 67–74% reduction in depressive symptoms
Metabolic
Diabetes	Avoided 86 extra cases of type II diabetes
	Led to small but significant reduction in glycosylated haemoglobin (0.7%). This amount is likely to reduce diabetes complications
Cancer
Colon cancer	A 30–40% reduction in the risk of developing colon cancer
Breast cancer	A 20–30% reduction in the risk of developing breast cancer
Lung cancer	A 20% reduction in the risk of developing lung cancer
Cardiovascular
Hypertension	Decreased systolic blood pressure by 3.8 mm Hg and diastolic blood pressure by 2.6 mm Hg in samples of both hypertensives and normatensives
	In hypertensives, systolic blood pressure was reduced by 4.94 mm Hg and diastolic blood pressure by 3.73 mm Hg
	In normatensives, systolic blood pressure was reduced by 4.04 mm Hg and diastolic blood pressure by 2.33 mm Hg
CHD	Avoided 51 extra cases of CHD
	Reduced all-cause mortality [odds ratio (OR): 0.80; 95% CI: 0.68–0.93] and cardiac mortality (OR: 0.74; 95% CI: 0.61–0.96)
Stroke	Avoided 16 extra cases of stroke
Musculoskeletal
Osteoporosis	A hip fracture risk reduction of 45% (95% CI: 31–56%) and 38% (95% CI: 31–44%), respectively, among men and women
Osteoarthritis	Pooled effect sizes for pain were between 0.39 and 0.52
	For self-reported disability, pooled effect sizes ranged from 0.32 and 0.46
Low back pain	Pooled mean improvement (measured on a scale of 100 points) was 7.3 points (95% CI: 3.7–10.9 points) for pain and 2.5 points (CI: 1.0–3.9 points) for function
Rheumatoid arthritis	Improved function by 0.24 (measured via the HAQ score) and pain by 0.31 (measured via the HAQ score)
Falls prevention	Beneficial effect on the risk of falls (adjusted risk ratio: 0.86, 0.75–0.99)
Absenteeism at work	Lower absenteeism at work (effect size = 0.19)
Adverse effects
Injury	Increased the risk of musculoskeletal injury by about four times
Disability	Walking (more than three city blocks) increased the risk of walking disability because of severe pain (OR: 4.1–5.0)

The benefits of ERS, compared with usual care, include an additional 3900 people becoming physically active, 51 cases of CHD avoided, 16 cases of stroke avoided, 86 cases of diabetes avoided, 152 additional people in health states devoid of CHD, stroke or diabetes and a gain of 800 QALYs [a product of QALYs gained over the lifetime of an individual exposed to ERS (i.e. 0.008) and the number of individuals (100 000)]. ERS is also expected to positively affect the prevention or/and management of mental health, metabolic disease, cancer and musculoskeletal conditions as well as productivity through reduced absenteeism at work. There are potential adverse affects in terms of injuries and pain which are considered rare[10,16] but could still negate some of the positive impacts of ERS. Based on the quantifiable costs and benefits, ERS is expected to result in a positive net benefit of 0.008 QALYs. This is expected to be further improved by the unquantified benefits.

Discussion

Main findings

Our findings suggest that ERS is cost-effective when compared with widely accepted thresholds for cost-effectiveness. However, caution should be taken in focusing on the ICER value alone. Our analysis suggests that ERS results in only a marginal benefit in terms of QALYs at a modest cost, hence the favourable ICER. It should be noted though, that the QALY gain is based on relatively poor quality evidence on the effectiveness of ERS.

Whilst the CUA provides a useful composite outcome, in the form of the ICER, stakeholders with an interest in the planning and delivery of ERS are likely to be somewhat frustrated by this approach. One might ask how meaningful a change of 0.008 in lifetime QALYs is to an individual and how this evidence can be used to (i) make the business case for investment in ERS and monitor its impact over time and (ii) convince participants of its benefits.

The CCA goes some way to addressing this issue by reporting the outcomes in natural units likely to resonate with those involved in the planning and delivery of ERS as well as participants. Identifying changes in the number of occurrences and the related resources provides a more meaningful measure of effectiveness for many stakeholders, particularly those from the non-health sector, such as local authorities and business. Furthermore, reporting the outcomes in a disaggregated fashion, including those where it might not be possible to quantify the scale of the effect, allows stakeholders to identify costs and benefits that they are likely to accrue and plan appropriately and, more importantly, put in place monitoring systems that can detect whether the benefits of ERS are being realized in practice. This disaggregated approach to evaluation might also provide information on how ERS can contribute to meeting specific priorities or addressing inequalities within a population (e.g. reduction in number of strokes or episodes of cancer). On this basis, the CCA is expected to be an attractive form of evaluation in public health settings and resonate with a broader range of stakeholders.

What is already known about this topic?

Many commentators have recognized the limitations of methods of economic evaluation when applied to public health as these methods are based on a pharmaceutical paradigm[2,3] However, to date, there is paucity of studies exploring the issue of the appropriate method to apply to economic evaluations of public health interventions.

What this study adds?

Our work contributes to filling this gap in knowledge by showing that CCA might be most suitable to public health interventions, using ERS as an exemplar. This is not to dismiss CUA. CCA and CUA are, essentially, two sides of the same coin. Much of the information reported in the CCA is captured in the composite outcomes of the CUA. However, the granularity with which the findings of the CCA are reported is expected to be desirable to stakeholders involved in planning public health interventions. Investment in public health might seem as a high risk by some stakeholders, given the level of uncertainty around the effectiveness of many of the interventions and the fact that many of the health benefits are unlikely to occur for a generation.[17] This perception might be exacerbated by CUA which report composite outcomes modelled over the lifetime of an individual, often based on fairly heroic assumptions about behaviour change. By breaking down the outcomes into more discrete units, decision-makers are better able to understand the potential costs and benefits. A further improvement to our own analysis would be to incorporate time into the outcomes, so that decision-makers can identify when costs and benefits are likely to be accrued. That would allow decision-makers to monitor short-term or proxy outcomes to ensure that interventions are delivering on their promise of long-term improvements in health outcomes.

Health economists involved in the assessment of public health interventions are encouraged to engage with health service, and increasingly, local authority commissioners to better understand their needs. CUA remains a powerful tool, providing a common currency with which the cost-effectiveness of healthcare interventions can be compared. However, this common currency has little resonance with stakeholders from the non-health sector. Given the multi-sectoral nature of public health, it is important that costs and benefits are presented transparently if the case is to be made for continued investment

Limitations

The analysis had a number of limitations. First, the CUA examined only the long-term impact of physical activity on selected morbidities. It was not possible to include other morbidities which may be affected by physical activity due to uncertainty over the relationship between physical activity, incidence and quality-adjusted life-expectancy. Secondly, the assumption that the average age of onset of a disease health states is 55 years might not be realistic as it could vary by demographics and disease conditions.[6] Thirdly, interventions which involve complex behaviour change may not be well suited to decision analytic models. Individual level simulation models which can detect changes in individual behaviours over time may better address cost-effectiveness. However, there will be a trade-off between developing a simple model, which can be populated and acknowledges its limitations versus a more complex model which may be a better representation of reality but can only be partially populated and may result in greater uncertainty. In all cases, the fundamental issue which needs to be addressed is improvement in the source data.

Conclusion

There is an increasing demand to consider the cost-effectiveness of investments in public health programmes. Prevailing methods of economic evaluation, such as CUA, which have been developed largely for the assessment of medical technologies may have limited applicability to public health interventions. CCA provides greater transparency when considering costs and consequences that might be accrued by a wide range of stakeholders in the public and private sectors and might also usefully provide a means of monitoring the short-term progress of public health programmes.

Funding

This work was supported by the NIHR Health Technology Assessment programme (project number 08/72/01). See the HTA programme website for further project information.