Estimating the Cost-Effectiveness of Implementation: Is Sufficient Evidence Available?

BACKGROUND: Timely implementation of recommended interventions can provide health benefits to patients and cost savings to the health service provider. Effective approaches to increase the implementation of guidance are needed. Since investment in activities that improve implementation competes for funding against other health generating interventions, it should be assessed in term of its costs and benefits.
OBJECTIVE: In 2010, the National Institute for Health and Care Excellence released a clinical guideline recommending natriuretic peptide (NP) testing in patients with suspected heart failure. However, its implementation in practice was variable across the National Health Service in England. This study demonstrates the use of multi-period analysis together with diffusion curves to estimate the value of investing in implementation activities to increase uptake of NP testing.
METHODS: Diffusion curves were estimated based on historic data to produce predictions of future utilization. The value of an implementation activity (given its expected costs and effectiveness) was estimated. Both a static population and a multi-period analysis were undertaken.
RESULTS: The value of implementation interventions encouraging the utilization of NP testing is shown to decrease over time as natural diffusion occurs. Sensitivity analyses indicated that the value of the implementation activity depends on its efficacy and on the population size.
CONCLUSIONS: Value of implementation can help inform policy decisions of how to invest in implementation activities even in situations in which data are sparse. Multi-period analysis is essential to accurately quantify the time profile of the value of implementation given the natural diffusion of the intervention and the incidence of the disease.

Introduction

International Adoption of Health Technologies

Problems with slow adoption of health technologies exist internationally. A study which examined the differential international diffusion of six health innovations found that rates of adoption varied significantly between innovations and countries [1]. The international OECD project found that there is widespread variation in the uptake and diffusion of healthcare technology amongst OECD countries, indicating that there are opportunities for more effective integration of such technologies into the health system. The report comments encouraging the uptake of the most efficient and effective healthcare technologies remains a significant policy challenge in many OECD countries [2].

Implementation of National Institute for Health and Care Excellence Clinical Guidelines

In England, the National Institute for Health and Care Excellence (NICE) produces clinical guidelines and technology appraisals for the UK National Health Service (NHS). Recommendations are made based on effectiveness and cost-effectiveness. Interventions (both treatments and diagnostics) which are recommended by NICE should be available to patients in England and Wales on the NHS. Indeed the NHS has an obligation to implement NICE technology assessments within three months of publication [3]. However, uptake of new guidance can be suboptimal [4], [5], [6], [7], [8], [9]. For example, heat maps for the use of medical technologies and primary care medicines show that there is wide variation between the medicines patients can access in one part of England compared with other parts [4]. It also shows that implementation rates may vary between disease types; for example, cancer patients may be well informed and request new treatments. Cost-effective technologies can only benefit patients and the health care service if they are used in practice.

Timely implementation of recommended interventions has the potential for benefits in terms of gains in Quality-Adjusted Life-Years (QALYs) to patients and/or cost savings to the NHS. It is possible to ensure that technologies are used in practice by investing in implementation activities. Early work aimed at improving implementation was primarily focused at medical staff [10]. However, this has been supplemented by a wider range of policy initiatives aimed at promoting the uptake of health technologies [11]. These include: mandatory inclusion in the hospital formularies; financial incentives to providers (such as the Quality and Outcomes Framework (QOF) and Commissioning for Quality and Innovation (CQUIN) scheme); regulatory measures (such as the NICE compliance regime, benchmarking and leading by example); initiatives by local NHS organizations; and the NICE implementation program [11]. Further investment in initiatives designed to promote implementation may lead to benefits to patients and the NHS. The key question is: How and how much should be invested in implementation initiatives given that those funds could also be used for other health-generating activities?

Methods to Evaluate Implementation Initiatives

There are methods to evaluate whether it may be worth investing in initiatives to speed up implementation. Mason and colleagues developed a simple deterministic framework to show how the cost-effectiveness of behavior change was a function of population size, together with the cost-effectiveness of the health technology and the cost-effectiveness of the behavior change intervention [12]. More recently, Fenwick et al. developed a unified framework that brought together value of information methods with the issue of implementation. This framework allowed a probabilistic evaluation of investments in implementation initiatives as expressed through the concepts of expected value of perfect and specific implementation [13]. This framework was extended further by Hoomans et al and Willan and Eckermann and then applied to an NHS policy initiative by Walker et al. [14], [15], [16].

Walker et al. developed further the previous work on value of implementation to include patient subgroups, multiple patient cohorts over time, and the impact of natural diffusion [17]. Walker et al. defined three concepts, based on those proposed by Fenwick et al., which are described here. The expected value of perfect implementation represents the maximum that can be gained from achieving full implementation and as such represents a maximum the NHS would be willing to pay. The expected value of actual implementation represents the maximum the NHS can invest in implementation activities for specific increases in utilization (i.e., for a specified % increase). All things equal, the expected value of actual implementation is larger for interventions with more favorable cost-effectiveness estimates or with larger patient populations. The value of the implementation activity is the difference between the expected value of actual implementation and the cost of the implementation activity. The value of the implementation activity is larger the smaller the costs and the larger the increase in utilization (effectiveness).

This article reports on the application of the value of the implementation framework to the case study of natriuretic peptide (NP) testing for the diagnosis of chronic heart failure (HF) [18]. NP testing was deemed to be cost-effective but has variable uptake; hence, we were interested in knowing the value of investing in implementation activities to increase the uptake of NP testing. We note that the uptake of NP testing is changing over time in the absence of implementation activities (natural diffusion). Also, we wanted to estimate the investment for both the current prevalent population and future cohorts presenting (given the future natural diffusion). This case study demonstrates how to estimate the value of implementation for multiple patient cohorts over time.

Methods

First, the data and assumptions used for the NP testing case study are described in the next five subsections and subsequently single-period and multi-period analyses undertaken are described in the last subsection.

NP Testing for Suspected HF

B-type natriuretic peptides (B-type natriuretic peptide [BNP] or N-terminal pro–B-type natriuretic peptide [NTproBNP]), referred to as NPs, are markers of HF. In 2010, NICE released clinical guideline 108 (CG108) on the diagnosis and management of HF. One of the recommendations in CG108 is testing for NPs in patients with suspected HF without previous myocardial infarction (MI) in order to accelerate diagnosis and avoid unnecessary echocardiography [18].

The standard of care for trusts that are not yet utilising NP testing is not dependent on MI history. Typically either a NP test, electrocardiogram (ECG), or both are used to rule out HF for all patients independent of MI history. CG108 recommended that patients with previous MI be referred to specialist assessment and ECG within 2 weeks. CG108 recommends NP testing for patients without previous MI:

If NP testing shows high levels (BNP > 400 pg/ml or NTproBNP > 2000 pg/ml), the patient is referred directly to specialist assessment and ECG within 2 weeks.

If NP testing shows raised levels (BNP 100–400 pg/ml or NTproBNP 400–2000 pg/ml), the patient is referred to specialist assessment and ECG within 6 weeks.

If NP testing shows normal levels (BNP < 100 pg/ml or NTproBNP < 400pg/ml), HF is unlikely and the patient is not referred further.

Cost and Effectiveness Data for NP Testing

The value of NP testing corresponds to the lifetime net benefit from using NP testing for the diagnosis of HF, as described in NICE CG108, for the average patient presenting with suspected HF. The economics of the diagnostic section of NICE CG108 was informed by the health technology assessment (HTA) report by Mant et al. [19]. However, the cost-effectiveness analysis did not match the decision problem faced by commissioners in two important ways: 1) use of diagnostic pathways was determined by MICE (Male, Infarction, Crepitations, Edema) score rather than by the history of MI as indicated in CG108 and in clinical practice and 2) in the comparison, current care is “do nothing” rather than ECG [20]. Consequently, the model and results from the Mant et al. HTA needed to be adapted to more closely represent the CG108. Whilst the lack of ECG as a comparator could not be resolved within this project, data obtained from the Mant et al. HTA on MICE score frequencies allowed cost-effectiveness by MI history to be estimated. The incremental values for CG108 versus “do nothing” for 1000 persons with suspected HF were calculated based on data from Mant et al. and are £3881 and +76.4 QALYs (See appendix for full details of calculations).

Population with Suspected HF

The population eligible for NP testing consists of persons presenting with suspected HF, and the scope of this study was to produce predictions for England and Wales. The size of the annual patient population eligible for NP testing in England and Wales varied by data source. Data from the General Practice Research Database suggests 22,542, Hospital Episode Statistics suggests 23,000 to 40,000, an HF study reported 59,000, Sheffield heart failure clinic data suggest 55,000, and a clinical expert contacted for this study estimated 70,000 [21], [22], [23]. Some of these estimates are suspected to be underestimates, such as how the Sheffield HF diagnostic clinic data did not include patients who were admitted to hospital with acute HF. Hence, the population was assumed to be 70,000 in the base-case, with 50,000 used in a scenario analysis. Based on the Cowie et al. [22] study, the prevalence of HF in persons presenting with suspected HF is 33%. Hence, the population of persons presenting with suspected HF is estimated to be 210,000 for the base-case analysis (3.7 per 1000 population) and 150,000 in the scenario analysis.

Utilization and Diffusion Rates

Data on the current utilization of NP testing were obtained from the NHS atlas of variation in diagnostic services [24]. The average rate of NP testing was 4.4 per 1000 practice population (minimum = 0; maximum = 14.4; 95% percentiles 0.1–11.4).

Optimal utilization is the average rate of NP testing if the guidance was fully implemented. This would require data on the number of persons presenting with symptoms of HF but who did not have a previous MI. Because no information was found on the optimal utilization rate, it was assumed to correspond to the NP testing rate in the 10% and 20% commissioning areas using NP most frequently, at 8.6 tests for the base-case and 7.1 tests per 1000 people (scenario), respectively. The maximum NP testing rate of 14.4 tests per 1000 people was not used because only one commissioning group achieved this rate. This gave a current optimal utilization of 51% for base-case (63% for scenario analysis). Furthermore, there may be some excess NP testing undertaken outside CG108, such as using NP as a screening test in the absence of symptoms [25]. Hence, optimal utilization will be lower than the observed maximum rate.

A review was undertaken prior to this project which found that diffusion of healthcare technologies were heterogeneous [11]. Future utilization rates were predicted from an S-shaped curve (of the form f(t) = 1/(1 – exp(–at + k))), fitted to two data points for 2010 and 2012. This shape of curve was chosen because it reflects the diffusion curve predicted by theory and in numerous observational studies [26]. Utilization in 2012 was based on estimates from the Diagnostics Atlas data as described previously. In the NICE costing template (released August 2010), expert clinical opinion estimated that without the NICE CG108 approximately 30% of patients currently receive NP testing (a BNP or NTproBNP test) and approximately 90% currently receive an ECG [27]. These estimates reflect the situation in 2010. In the base case we assume that for these 30%, utilization was 50% of optimal maximum utilization, that is, 15%. The diffusion curves generated using these data are presented in Figure 1.

Fig. 1

Predicted diffusion of utilization without an implementation intervention.

Initiatives to Increase Implementation of NP Testing

From 2011 to 2013, an NP testing implementation initiative led by healthcare scientists took place in NHS London. In the first stage of the initiative, the barriers to implementation were determined and a pipeline adoption scale was developed. In 2012, an audit of 25 London provider trusts was undertaken to establish their use of NP testing (using the NTproBNP test). The audit showed that trusts’ use of NP testing ranged from “no NP testing” to 13,000 to 14,000 tests annually. The trusts delivering the lowest numbers or no tests, “intervention trusts,” were supported a variety of ways including: peer support from London Scientific & Diagnostic Network, help with leadership, and commissioning problems. A 2-hour workshop was attended by five exemplar trusts and three intervention trusts in February 2013 and knowledge, business plans, and contact details were shared [28], [29]. Data on quantity of NP testing following the initiative were available for only 2 of the 20 trusts that responded to the initial audit; therefore, a robust estimate of the effectiveness of the initiative could not be calculated.

An estimate of the cost of the workshop was developed using estimates of staff time and consumables based on information provided by the workshop organizer [28]. Full details are reported elsewhere [30]. Unit costs for staff time were taken from the literature [31]. The total cost was estimated to be £4172 for the 25 trusts in London (£28,187 for the whole of England and Wales).

A systematic review was conducted to establish the effectiveness of implementation activities [11]. For the NP case study, it is suggested that “organizational, educational, opinion leader, educational outreach, facilitation” are the most relevant initiative types. Estimates of effect size from the O’Brien et al. study were used because these were considered the most relevant [32]. In line with those estimates, the base-case analysis an effect size of 5% was used and a scenario analysis applied an effect size of 9%.

Analysis

The following quantities were estimated for the NP case study: expected value of perfect implementation, expected value of actual implementation, and value of the implementation activity. The formulae used in these analyses are shown in Figure 2.

Fig. 2

Components involved in value of implementation analysis together with formulae.

A multi-period analysis that calculates how value of implementation varies over time was undertaken. The multiperiod analysis was started in 2012 because this was the time point for which utilization data were available and run for 10 years. Future costs and QALYs accrued were discounted at a rate of 3.5% in line with the NICE methods guide. The population presenting with suspected HF was assumed to increase at a rate of 4.2% per annum. This was based on national Hospital Episode Statistics data on the number of finished consultant episodes with a primary diagnosis of HF, with increases of 3.6% from 2010-2011 to 2011-2012 and 4.8% from 2011-2012 to 2012-2013 and assuming that an increase in NP testing does not reduce the population size [33]. The total value of the implementation activity over the 10-year period was calculated by summing the values from the 10 years.

Results

Base Case Analysis

The results of the base case analysis that considers a single cohort are presented in Table 1. The net monetary benefit (NMB) of NP testing in patients without previous MI was estimated at £1,524 for a threshold of £20,000 per QALY gained and £2,288 for a threshold of £30,000 per QALY gained. The current value to the NHS, given the patient population currently receiving testing, is approximately £164 million or 8,230 QALYs for a threshold of £20,000 per QALY gained.

Table 1

Static population analysis with base-case assumptions

Static population analysis	WTP = £20,000		WTP = £30,000
NMB	NHB	NMB	NHB
Net benefit to the NHS (per patient)	£1,518	0.076	£2,282	0.076
Current value of technology given current utilization and population size for England and Wales. Population = 210000, current utilization = 4.43 (51%)	£163,969,752	8198	£246,476,558	8216
Expected value of perfect implementation. Value of increasing utilization from current to desirable maximum. Current utilization = 4.43, desirable maximum = 8.62 (100%)	£154,895,869	7745	£232,836,850	7761
Expected value of actual implementation. Value of increasing utilization from current to achievable. Current utilization = 4.43, achievable utilization with intervention = 4.86 (56%)	£15,943,281	797	£23,965,670	799
Value of the implementation activity. Expected value of actual implementation minus cost of intervention (£28,187)	£15,915,094	796	£23,937,484	798

NHB, net health benefit; NHS, National Health Service; NMB, net monetary benefit; WTP, willingness to pay.

The expected value of perfect implementation represents the value of NP testing in the eligible population currently not receiving testing. The expected value of perfect implementation is approximately £155 million or 7,774 QALYs for the suspected HF population in England and Wales for a threshold of £20,000 per QALY gained.

The expected value of actual implementation is based on an implementation activity that is assumed to increase utilization by 5% (from 51% to 56%). The expected value of actual implementation is much smaller than the expected value of perfect implementation. For the overall population in England and Wales, the NHS could invest up to £16 million for an activity that increases utilization by 5% at a threshold of £20,000 (approximately £3 million for a 1% increase in utilization).

The value of an implementation activity costing an average of £4,172 per 25 providers (£28,187 for the whole of England and Wales) was calculated. Because the implementation activity has a relatively low cost, the value of the implementation activity is similar to the expected value of actual implementation, providing additional value to the NHS of £16 million or 799 QALYs for England and Wales at a threshold of £20,000 per QALY.

Multi-period Analysis

The multi-period analysis demonstrates how the value of implementation activity accrues over a 10-year period. The results are presented in Figure 3, which demonstrates that after 5 years the utilization rates with and without the implementation intervention are very similar (both close to 100%). Therefore, the value of implementation is very small in the second half of the 10-year period. The total value of implementation activity over 10 years is £76 milllion.

Fig. 3

Value of implementation: Multiperiod analysis. HF, heart failure; WTP, willingness to pay.

Scenario Analyses

Three scenario analyses were undertaken to explore the impact of key uncertainties on model results: 1) Population size: annual rate of suspected HF: 0.27% (population of 150,000); 2) Current utilization of BNP testing: 63%; and 3) Efficacy of the implementation intervention: 9% increase. The results indicated that population size and efficacy of the implementation intervention both had a significant impact on the value of the implementation activity (reduction and increase to £11 million and £28 million, respectively) (see Table 2). Additional sensitivity analyses which explore the impact of varying parameter values further are provided in the appendix.

Table 2

Value of implementation: Scenario analyses

Scenario	Net monetary benefit (willingness-to-pay threshold: £20,000 per QALY)
Base case: Annual rate of suspected HF: 0.37%; Current utilization: 51%; Efficacy of implementation intervention: 5%	£16.0 million
Population size: Annual rate of suspected HF 0.27% (population of 150,000)	£11.4 million
Current utilization of BNP testing: 63%	£16.0 million
Efficacy of the implementation intervention: 9% increase	£28.1 million

BNP, B-type natriuretic peptide; HF, heart failure; QALY, quality-adjusted life-year.

Discussion

Summary of Results

This study demonstrates the application of a framework for calculating the value of implementation activity to a case study: NP testing in diagnosing chronic HF. A multi-period analysis uses diffusion curves to predict change in utilization over time and estimating the value of implementation over a 10-year time period. The analysis demonstrates that the value of investing in implementation may decrease over time as utilization increases. Based on this analysis, the NP testing implementation activity modeled here is predicted to provide additional value to the NHS of £16 million for England and Wales (at a threshold of £20,000 per QALY).

The framework can be applied to any existing cost-effectiveness analysis, thus helping a decision maker to quantify the value of investing resources in increasing utilization in a manner consistent with the value assessment of new interventions conducted by NICE.

Evidence to Populate a Value of Implementation Model

This case study provides a useful demonstration of the practical challenges faced in populating such a model. The main limitations of the analysis were a lack of evidence on cost-effectiveness, effectiveness of implementation interventions, utilization, diffusion of utilization, and population size. In these areas in which key uncertainties exist in data used to inform the model, undertaking scenario analyses is essential. This case study suggests that the availability of evidence to inform a model using this framework could be an issue in many situations.

Estimates of utilization were made, but these required assumptions regarding the maximum optimal utilization rate. A better understanding of the utilization and diffusion of health technologies is required for accurate modeling, and consideration should be given to routinely collecting this after reimbursement decisions. Without this information, it is not possible to assess what effect the reimbursement process has actually had, nor the value of investing in implementation initiatives.

There was considerable uncertainty surrounding the evidence on population size. Accurate estimates of population size are also essential to estimate utilization rates. The differences in estimates of population size from different published data sources highlights the importance of validating data sources with clinical experts.

In addition, there were no estimates of what the optimal utilization should be for the indicated population. In other areas of medicine, information on optimal utilization will be clearer because licensing of pharmaceuticals ensures clearer definitions of patient populations. However, license-based estimates are less useful if there are competitor products because the maximum will be dependent on market share.

The comparator in the economic analysis was “do nothing” rather than ECG, so the cost-effectiveness is likely to be overestimated; hence, results will overestimate the value of implementation activities. The importance of publishing incremental costs and QALYs related to clinical guidelines compared with current care is thus highlighted. Data on the effectiveness of the London implementation initiative were not available. Robust data on the cost and effectiveness of implementation interventions should be collected where possible to allow for evaluation.

Comparison with Previous Studies

One of the first systematic reviews of implementation initiatives by Grimshaw et al. identified 11 cost-effectiveness analyses [10]. The studies were purely observational, with no attempt at modeling the longer-term effects on diffusion and patient outcomes. Furthermore, neither QALYs nor probabilistic sensitivity analyses were used in any of the studies. A more recent review found a similar situation, with poor methods and reporting [34].

Mason et al. present a method for calculating policy cost-effectiveness and illustrate it with examples from a trial of educational outreach by community pharmacists to influence physician prescribing in England [12]. The intervention to increase implementation was found to be cost-effective; however, this is based on a one-off increase in uptake, which fails to recognize the dynamics of diffusion and patient incidence.

In the United Kingdom, the QOF policy initiative to change the clinical behavior of general practitioners has been examined in considerable detail, including five economic evaluations [35]. Of the five studies cited, only Walker and colleagues produces ICER [16]. Due to lack of availability of data on baseline performance, the actual cost-effectiveness could not be determined by Walker et al., and as a result, the authors analyzed the potential cost-effectiveness. Despite using a probabilistic framework, the analyses did not examine the value of implementation.

Hoomans et al. examined the value of information and the value of implementation for treatments for metastatic hormone-refractory prostate cancer. The use of a probabilistic framework made it possible to examine whether it was worthwhile investing in implementation strategies and the maximum amount of investment that would be cost-effective. This was then extended in another case study, this time using empirical estimates of the effectiveness of alternative implementation strategies and examining variations between different service providers [36], [37].

The other case study within our program of work was used to explore the impact of patient subgroups in a more detailed way. This study of novel oral anticoagulants (NOACs) for the prevention of stroke and systemic embolism in patients with atrial fibrillation found that there is value in additional implementation activities, particularly in targeting patients with average or poor warfarin control. Additional investment in an educational activity that increases utilization by 5% in the entire population currently on warfarin generates additional 254 QALYs versus 973 QALYs in the subgroup with average to poor warfarin control [38].

This study adds to the results of previous research by highlighting the limitations of available data and exploring the impact of these uncertainties via sensitivity analyses. We conclude that data limitations have a considerable impact on estimates of value of implementation. No previous studies have undertaken a dynamic multi-period analysis and as such underestimate the value generated by the treatment of future cohorts and ignore the importance of diffusion dynamics. Our study demonstrates how it is possible to estimate diffusion curves on the basis of historic data to produce predictions of future utilization and how these can be incorporated into a multi-period analysis.

Further Work

A probabilistic approach to representing parameter uncertainty (in which parameters are sampled from distributions) could be utilized within this framework. Incorporation of this would allow analyses examining the value in reducing uncertainties related to the model parameters. As such, in situations in which the value of implementation of the implementation activity is low, assessments could be made as to research priorities aimed at reducing key uncertainties.

Further research on diffusion curves for health technologies is recommended because this is a key modeling component for estimating the value of implementation. Furthermore, it is known that the level of implementation can change the prices of some medical technologies. For example, evidence shows that price can be negatively related to the uptake for some technologies, with lower prices being generated through reductions in production costs and increased supplier competition [39]. Including such a relationship within analyses is potentially important; however, the empirical evidence on this relationship is sparse for health technologies.

A further relationship to consider is whether there may be a higher cost of implementation for a subgroup consisting of the slowest adopters. Lastly, consideration should be given as to whether the nature of the patient population changes with implementation. For example, Hoyle and Anderson investigate the impact of a technology being applied to changing patient populations [40]. Research funding bodies or reimbursement agencies should consider making key information required by this framework compulsory in submissions made by manufacturers. For example, past and current utilization (to allow a diffusion curve to be constructed), current and future population size, and the maximum implementation level. Likewise, reimbursement bodies should build in data collection to the postdecision period so that the level of implementation can be measured and the value of initiatives assessed.

Conclusions

The framework on the value of implementation can help health services such as the NHS quantify the value of investing resources in increasing utilization in a manner consistent with the value assessment of new interventions. The expected value of actual implementation is the maximum investment that the NHS should consider for a given increase in utilization. Implementation activities should be considered for investment by the NHS as long as their cost is not greater than their value of actual implementation (value of implementation ≥ 0). Data problems exist for analyses of this kind; however, they present the correct framework for making efficient investment decisions. Collecting the necessary data as part of the reimbursement process is essential if we are to assess what effect the reimbursement process has actually had and the value of investing in implementation initiatives.