A utility of model input uncertainty analysis in transferring tobacco control-related economic evidence to countries with scarce resources: results from the EQUIPT study.

AIMS: To inform the transferability of tobacco control-related economic evidence to resource-poor countries.
METHODS: We ran a univariate sensitivity analysis on a return on investment (ROI) model, the European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD), to identify key input values to which the ROI estimates were sensitive. The EQUIPTMOD used a Markov-based state transition model to estimate the ROI of several tobacco control interventions in five European countries (England, Germany, Spain, Hungary and the Netherlands). Base case ROI estimates were obtained through average values of model inputs (throughout the five countries), which were then replaced one at a time with country-specific values. Tornado diagrams were used to evaluate the significance of sensitivity, defined as a ≥ 10% difference in ROI estimates from the base case estimates.
RESULTS: The ROI estimates were sensitive to 18 (of 46) input values. Examples of model inputs to which ROI estimates were sensitive included: smoking rate, costs of smoking-related diseases (e.g. lung cancer) and general population attributes.
CONCLUSION: Countries that have limited research time and other resources can adapt EQUIPTMOD to their own settings by choosing to collect data on a small number of model inputs. EQUIPTMOD can therefore facilitate transfer of tobacco control related economic evidence to new jurisdictions.

Introduction

Transferability of evidence from one context to another has recently been the subject of growing academic and policy research 1, 2, 3. Using existing evidence—even though it was generated in another country—can save research resources and time 4, and economic evidence concerning tobacco control is not an exception to this. However, differences in local characteristics such as population attributes, prevalence data and the structure of the health‐care system may lead to significantly different outcomes, even in countries that are deemed to be similar in many respects to the one where the evidence was produced originally. Kaló et al. 5, for example, have explained why the recommendations of the English National Institute for Health and Care Excellence (NICE) are not transferable without sufficient adjustment in the analyses to reflect local data that can alter the conclusions on cost‐effectiveness. Drummond et al. 6 argued that it is legitimate to observe non‐systematic variations of cost‐effectiveness results for pharmaceuticals even within western Europe alone. In order to ensure that the outcomes would reflect the local setting more accurately on tobacco control‐related evidence transfer, it is necessary to assess whether the input values of an economic model need any adjustment. Although a growing body of literature 7, 8 suggests that they do, it is not clear to what extent this would apply to economic models evaluating behaviour change interventions such as tobacco control.

Research aimed at understanding intercountry variation in the cost‐effectiveness of tobacco control interventions is growing. A number of studies have analysed the factors influencing smoking cessation in different countries, e.g. western Europe, the United States, Greece and Iran 9, 10, 11, 12, 13. Vemer et al. 9 focused upon the identification of the main causes of significant differences in the incremental net monetary benefit of pharmacological smoking cessation therapies between six western European states. They later looked at the relationship between the willingness to pay threshold and the other factors 10. Although their findings are necessary in understanding the transferability of economic models to different settings they are not sufficient with regard to decisions relating to economic modelling, which require a substantive amount of data.

The European study on Quantifying Utility of Investment in Protection from Tobacco (EQUIPT) aimed to transfer an existing return‐on‐investment (ROI) Markov state transition model developed in England to other European Member States. Following an ‘inverted cone’ approach of transferability, the model was subjected to adaptation in two spaces: (i) working space, in which the existing English model was used to develop a new suite of models for use in four EU Member States (Germany, Spain, Hungary and the Netherlands); and (ii) transfer space, in which the model was assessed for transferability to wider European countries, in particular Central and eastern Europe (CEE), with a view to guiding pan‐European, comprehensive tobacco control policies.

This paper reports the analysis conducted as part of the latter with the objective of identifying key model input values to which the ROI estimates were significantly sensitive. Earlier work with stakeholders 14, which also assessed the intention to use an economic model in decision‐making 15, highlighted the need for such a model to be contextually relevant in order to be transferable. To that end, a within‐country analysis looking at the sensitivity of ROI estimates was first conducted on the original English ROI model 16. This was followed by a separate intercountry analysis, which is reported here.

Methods

The economic model

We used the European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD), a Markov‐based state transition model developed to evaluate several polices concerning tobacco control and smoking cessation interventions, with a view to informing resource allocation decisions at national or regional levels. EQUIPTMOD uses a wide range of model inputs. Details of this model are described elsewhere 16, but a top‐level summary is provided below.

The Markov model has three states—current smokers, former smokers and death. As interventions are received, those smokers who will make a quit attempt during the next 12 months may stop smoking and become former smokers. However, some of the recipients may not quit and some quitters may relapse. Interventions can also influence smokers’ motivation to quit. The cohort is subject to smoking‐attributable morbidity [coronary heart diseases (CHD), chronic obstructive pulmonary disease (COPD), stroke, and lung cancer] as well as mortality (smoking‐attributable and all‐cause), depending on the risks based on their age and sex. At the end of each cycle (1 year), quality‐adjusted life years (QALYs) and costs of treating smoking‐attributable diseases are calculated and then adjusted by applying population weights. The number of QALYs, smoking‐attributable treatment costs and costs of the interventions are then used to generate ROI estimates (see Table 1 for the definition of ROI estimates). A cost‐effectiveness threshold value of £20 000 per QALY recommended by the National Institute for Health and Care Excellence (NICE) 17 in the United Kingdom was used to estimate the incremental net benefit. The model takes a health‐care system perspective: various time horizons (2, 5 and 10 years and life‐time) and future costs and benefits are discounted. This is to comply with relevant guidance in countries, e.g. NICE in England 18.

Table 1

List of the seven outputs of the European study on Quantifying Utility of Investment in Protection from Tobacco (EQUIPT) model included in the analysis.

Name of output	Definition of output
Avoided burden of disease across all smokers	The number of QALYs averted across all smokers in the population
Benefit–cost analysis: health‐care savings	The sum of health‐care cost savings per recipient divided by the cost of the intervention per recipient
Benefit–cost analysis: health‐care savings and value of health gains	The sum of health‐care cost savings per recipient and value of health gains (monetary value of QALY multiplied by the number of QALYs gained), divided by the cost of the intervention per recipient
ICER per life year gained	The incremental cost of the package minus the sum of health‐care cost savings divided by the number of life years saved
ICER per QALY gained	The incremental cost of the package minus the sum of health‐care cost savings divided by the number of QALYs gained
Average health‐care cost savings	The sum of health‐care cost savings per recipient less implementation cost per recipient
Health‐care savings and value of health gains	The sum of health‐care cost savings and value of health gains (monetary value of QALY multiplied by the number of QALYs gained) per recipient less implementation cost per recipient

QALY = quality‐adjusted life years; ICER = incremental cost‐effectiveness ratio.

Population, intervention, comparator and outcomes (PICO)

The population of interest was the general adult population (16+ years, and in the case of Germany 18+ years) categorized by their smoking status. Interventions included in the model were classified as either increasing quit attempts (e.g. smoke‐free legislation) or increasing quit success (e.g. smoking cessation support). The quit rate was assumed to be a function of the implementation of these interventions. For the purpose of this analysis, ‘intervention’ referred to a ‘package’ comprised of 19 types of smoking cessation interventions compared with baseline (zero investment scenario) (Table 2). The baseline represents the theoretical gross cost of tobacco to society if all ongoing financial investment in interventions and policies were cut immediately, i.e. a scenario with no smoking cessation interventions, except a smoking ban and taxation at the current level. The latter were included in the baseline, as it was impossible to exclude those two interventions to create such a counterfactual. The health outcomes included were: lung cancer, COPD, CHD and stroke; costs (both the cost of implementing a package as well as cost of treating smoking‐attributable diseases); and QALYs.

Table 2

Composition of the intervention package and comparator.

Intervention name	Description	Included in the ‘Baseline’ packagea	Included in the ‘Intervention’ package
Interventions designed to increase quit attempt success
Pharmaceutical
OTC mono NRT	Pharmaceutical interventions with or without the requirement of a prescription to help individuals to quit smoking successfully A portion of smokers are assumed to receive pharmaceuticals in combination with non‐pharmaceutical interventions when attempting to quit		X
OTC combo NRT			X
Rx mono NRT			X
Rx combo NRT			X
Varenicline (standard duration)			X
Varenicline (extended duration)			X
Bupropion			X
Nortriptyline			X
Cytisine			X
Non‐pharmaceutical
Specialist behavioural support: one‐to‐one	Non‐pharmaceutical interventions to help individuals to quit smoking successfully A portion of smokers are assumed to receive non‐pharmaceutical interventions in combination with pharmaceuticals when attempting to quit		X
Specialist behavioural support: group‐based			X
Telephone support: proactive			X
SMS text messaging			X
Printed self‐help materials			X
Interventions designed to increase the number of smokers making quit attempts
Brief physician advice	Health‐care professionals advising people to improve their health by stopping smoking		X
Social marketing	Launching social marketing campaigns on the subject of quitting smoking		X
Cut down to quit	Reducing tobacco use in those current smokers who are currently unwilling to make quit attempts. The intention here is to reduce the harm from tobacco by cutting down, which may lead eventually to quitting		X
Taxation increase	Increasing the taxation of tobacco products	X	X
Indoor smoking ban	Banning smoking in all enclosed public places	X	X

X: the intervention is included in the package.

The baseline represents the theoretical gross cost of tobacco to society if all ongoing financial investment in interventions and policies were cut immediately (i.e. no smoking cessation interventions, except a smoking ban and taxation at the current level, as it was impossible to exclude those two interventions to create such a counterfactual). OCT = over‐the‐counter; NRT = nicotine replacement therapy; SMS = short messaging service.

ROI estimates

This analysis considered seven ROI estimates, listed in Table 1 19. These estimates—which included both economic and health indicators—fell under the umbrella of ROI estimates as suggested by NICE 20. Six of the seven estimates were calculated on a per‐capita basis and only one estimate (avoided burden of disease) was computed for the smoking population as a whole.

Univariate sensitivity analysis

In the univariate sensitivity analyses 21 we changed one input value in the model at a time and examined the impact of the change on the model's outputs. This method was used to identify key model inputs to which ROI estimates were sensitive.

Between the five EQUIPT countries (England, Germany, Spain, Hungary and the Netherlands), a total of 46 country‐specific input values were collected, although the model had many more inputs which were deemed relevant to all countries (e.g. relative risks). In some cases the value of an input was available from only one country, and this served as a default value for all other countries.

Some input values were changed together as a set, e.g. mortality tables stratified by age and smoking status as mortality is likely to vary between countries, but this variation is likely to be similar for different age and sex categories 22. Interval transformations were applied to those sets of inputs where they were linked to intervals, but the intervals were not uniform across countries. The arithmetic means of the values from all the relevant intervals were also calculated to assign values to the newly established uniform intervals.

A reference country was created in the model as a base case using the unweighted arithmetic means of all available values of any given input across the five EQUIPT countries. During the sensitivity analysis, the average estimate for each input was replaced with the available country‐specific input value, one at a time, while the values of other inputs remained unchanged. The outcomes were recorded after changing each input in the model.

Evaluation of sensitivity and identification of key inputs

Sensitivity was evaluated by assessing the difference in the resulting ROI estimates between the use of a particular country‐specific input value and the use of base case input value. The sensitivity range was calculated as the difference between the minimum and maximum values of the ROI estimates in response to changes in a particular input value. Sensitivity that is greater than 10% of the base case output values was considered large (significant). The choice of 10% as the threshold to define large/significant sensitivity was arbitrary, but in the absence of any empirical data supporting this we assumed that a 10% (or higher) sensitivity could lead potentially to changing or questioning a decision based on the cost‐effectiveness criteria. Depending on the size of sensitivity throughout different ROI estimates, several key model inputs were identified with the help of Tornado diagrams.

Construction of the shortlist

To present the final results of our analysis, a table of key model inputs was compiled. The list of high‐sensitivity inputs was based on at least four (out of seven) model outputs where large sensitivity was evident.

Results

Table 3 provides an example of how the changes were observed in the ROI estimates, when the values of seven inputs (of the 46 inputs included in the assessment) were replaced one at a time with country‐specific values. For example, the second row presents base case values of the avoided burden of disease across all smokers: 60 162 QALYs, compared to 59 398, 60 744, 59 918, 60 466 and 60 012 QALYs, respectively, for the English, Spanish, German, Dutch and Hungarian populations. These figures show that the selected ‘package’ of interventions has the potential to generate that number of QALYs throughout all smokers and they vary across countries. Similarly, other ROI estimates are shown in the rows below (Table 3).

Table 3

Results of the univariate sensitivity analysis: base case value versus country‐specific values of age‐ and sex‐specific population data.

ROI measure	Base case value	With the English value	With the Spanish value	With the German value	With the Dutch value	With the Hungarian value
Avoided Burden of Disease across all smokers (QALYs gained across all smokers)	60 161.80	59 398.48	60 743.96	59 918.23	60 466.37	60 012.18
Benefit–Cost Analysis: quasi‐societal savings (Return on every currency unit invested)	0.51	0.50	0.53	0.50	0.52	0.51
Benefit–Cost Analysis: quasi‐societal savings and value of health gains (return on every currency unit invested)	3.50	3.45	3.55	3.47	3.52	3.49
ICER per Life Year gained (currency unit per Life Year gained)	9017.64	9483.33	8559.63	9328.83	8903.13	9197.21
ICER per QALY gained (currency unit per QALY gained)	5865.14	6120.35	5602.73	6059.50	5789.62	5943.67
Average cost savings (currency unit per smoker)	−40.10	−41.31	−38.67	−41.25	−39.78	−40.53
Savings and value of health gains (currency unit per smoker)	205.19	200.86	208.98	203.03	206.74	204.14

In the analysis, euros were set as the currency for all cases. QALY = quality‐adjusted life years; ICER = incremental cost‐effectiveness ratio; ROI = return on investment.

The difference between the new and the base‐case values in this example was −1.27, 0.97, −0.40, 0.51 and −0.25% of the base case value, respectively. The difference between the largest (Spanish) and smallest (English) value was 1345 QALYs; as it was only 2.24% of the base case value, the variation was not considered to be large.

Table 4 presents several examples of sensitivity that were observed once a specific input value was changed from the base case value to country‐specific values. This is reported in the table for three of the seven ROI estimates considered in the analysis (avoided burden of disease across all smokers, and the incremental cost‐effectiveness ratios calculated as incremental cost‐effectiveness ratio (ICER) per life year gained and ICER per QALY gained). Table 4 also shows that avoidable burden of disease across all smokers was most sensitive to the population size (174.15%). The sensitivity depended upon the choice of ROI estimates, although all ROI estimates included in this table were significantly sensitive (≥ 10%) to model inputs such as discount rate of outcomes and the background quit rate.

Table 4

Selected ROI measures and their sensitivity to the changes in values of selected inputs.

	Difference between lowest and highest value of the particular output (percentage of base case output values)
Input	Avoided burden of disease across all smokers	ICER per life‐year gained	ICER per QALY gained
Actuarial life tables	8.44%	30.03%	3.43%
Background quit rate	15.43%	32.97%	32.40%
Coronary heart disease prevalence	6.50%	23.74%	30.44%
Chronic obstructive pulmonary disease prevalence	7.46%	55.51%	61.20%
Cost discount rate	0.00%	15.16%	15.16%
Cost of coronary heart disease	0.00%	21.45%	21.45%
Cost of chronic obstructive pulmonary disease	0.00%	53.52%	53.52%
Cost of lung cancer	0.00%	31.23%	31.23%
Cost of stroke	0.00%	24.85%	24.85%
Lung cancer prevalence	0.80%	13.14%	13.96%
Outcome discount rate	49.18%	51.06%	41.73%
Population aged 16 or older	174.15%	0.00%	0.00%
Population numbers by sex and age	2.24%	10.24%	8.83%
Smoking rate	52.45%	0.00%	0.00%
Smoking status by sex and age	3.88%	46.99%	48.86%
Stroke prevalence	0.50%	11.96%	11.92%

Bold type = difference greater than 10%. ICER = incremental cost‐effectiveness ratio; ROI = return on investment; QALY = quality‐adjusted life years.

Figures 1 and 2 are tornado diagrams showing sensitivity concerning model outputs (ROI estimates), as discussed above. As seen from Fig. 1, avoidable burden of disease across all smokers was most sensitive to the population size (hence the largest bar). Other input values resulted in smaller sensitivity; hence, shorter bars are displayed in the tornado diagram. In order to construct a shortlist of key model inputs, all relevant tornado diagrams (not shown here) for all seven ROI estimates were assessed in this manner.

Figure 1

Tornado diagram showing how sensitive the return on investment (ROI) measure [avoided burden of disease across all smokers, i.e. quality‐adjusted life years (QALYs) gained across all smokers] is to the change in values of inputs

Figure 2

Tornado diagram showing how sensitive the return on investment (ROI) measure [incremental cost per quality‐adjusted life years (QALYs) gained] is to the changes in values of inputs

Table 5 summarizes the process of identifying key model inputs to draw up the shortlist. As seen from Table 5, changes in four input values led to a large sensitivity (> 10%) in all ROI estimates. These inputs were: background quit rate, quit rate in those smoking 10+ cigarettes per day, proportion of those who smoke 10+ cigarettes per day and unassisted quit rate. An additional 11 inputs led to large sensitivity (> 10%) in at least four of the seven ROI estimates: COPD prevalence, CHD prevalence, cost of COPD, outcome of discount rate, smoking status by sex and age, cost of discount rate, cost of CHD, cost of lung cancer, cost of stroke, lung cancer prevalence and stroke prevalence. These 15 inputs were therefore included in the provisional shortlist.

Table 5

Simplified algorithm to identify key model inputs for the shortlist.

Input	Avoided burden of disease across all smokers	Benefit–cost analysis: quasi‐societal savings	Benefit–cost analysis: quasi‐societal savings and value of health gains	ICER per life‐year gained	ICER per QALY gained	Average quasi‐societal cost savings	Quasi‐societal savings and value of health gains	Number of Xs (greater than 10% variability in the outcomes)
Background quit rate	X	X	X	X	X	X	X	7
Quit rate in those smoking 10+ cigarettes per day	X	X	X	X	X	X	X	7
Proportion of those who smoke 10+ cigarettes per day	X	X	X	X	X	X	X	7
Unassisted quit rate	X	X	X	X	X	X	X	7
COPD prevalence		X	X	X	X	X	X	6
CHD prevalence		X		X	X	X	X	5
Cost of COPD		X		X	X	X	X	5
Outcome discount rate	X		X	X	X		X	5
Smoking status by sex and age		X		X	X	X	X	5
Cost discount rate		X		X	X	X		4
Cost of CHD		X		X	X	X		4
Cost of lung cancer		X		X	X	X		4
Cost of stroke		X		X	X	X		4
Lung cancer prevalence		X		X	X	X		4
Stroke prevalence		X		X	X	X		4
Actuarial life tables		X			X	X		3
Will make quit attempt				X	X		X	3
Threshold value			X				X	2
Population numbers by sex and age				X				1
Population aged 16 or older	1							1
Smoking rate	1							1
Days lost due to smoking								0
Ex‐smoking rate								0
Smokers’ employment rate								0

X = greater than 10% sensitivity in the return on investment (ROI) measures. CHD = coronary heart disease; COPD = chronic obstructive pulmonary disease; ICER = incremental cost‐effectiveness ratio; QALY = quality‐adjusted life years.

Bold type = input selected for the shortlist of key input.

Two additional inputs were added to the provisional shortlist based on sensitivity concerning the avoided burden of disease across all smokers. These two inputs were population aged 16 years or older and smoking prevalence. Another input, threshold value, was also added to the list, as it is essential for converting health gains into monetary values for the two ROI estimates: benefit–cost analysis: quasi‐societal savings and value of health gains and quasi‐societal savings and value of health gains. The addition of these three inputs resulted in the 18‐item shortlist.

Five inputs (i.e. intervention uptake, intervention unit cost, inflation adjustment, average hourly wage and passive smoking costs) were not always available for Germany, Spain, Hungary and the Netherlands. Therefore, for practical reasons it was decided to add those five inputs to the shortlist. The final shortlist therefore contained 23 key inputs.

Discussion

Transferability of evidence—especially backed up by an economic model—is often a debated issue, because transferring evidence requires context‐specific data to go into the model 23, but often such data are either not available or the collection of such data is severely constrained by limited resources 5. Not being able to transfer evidence because of the unavailability of context‐specific data has severe opportunity cost implications; if transferred, the evidence could have saved lives (health outcomes) as well as money (economic outcomes) in new countries 24. To this end, our study is innovative in filling this important gap by providing a shortlist of key inputs of an economic model (EQUIPTMOD) that is highly relevant to several European countries, including the Central and eastern European (CEE) countries where the health and economic impact of smoking‐attributable disease is tremendous 25, 26.

The shortlist is the product of a thorough transferability evaluation. The EQUIPT study adopted an ‘inverted cone’ approach to transferability, with three distinct phases: evidence base, working space and transfer space 27. In this framework, the evidence (i.e. the English model) first underwent a rigorous adaptation to five countries (working space). This analysis, a part of the transfer space, then examined in detail the uncertainty concerning model inputs and its impact on predicted ROI estimates throughout different jurisdictions included in the working space. The rigour thus adopted to identify the most uncertain inputs (18 of 46) is a testimony to the fact that without such an analysis, one would always tend to collect more data than necessary to enable the transfer of existing evidence to a new jurisdiction. Although previous studies 2 have indicated the need for such an analysis before country adaptation can begin, our study provides much‐awaited empirical evidence to that end. Therefore, while the five EQUIPT countries had to collect data on 46 inputs, any country outside the EQUIPT sample—including CEE countries—now can collect data on just 18 inputs and the five additional inputs mentioned in the final paragraph of the Results section to be able to obtain their own context‐specific outputs from the EQUIPT model.

The inputs in the shortlist can be assigned to three major categories: inputs describing the whole population (e.g. population size and smoking status by sex and age); inputs related to cost‐effectiveness calculation methods (e.g. discount rates); and inputs measuring prevalence or the costs of smoking‐attributable diseases. Although this categorization is generally consistent with previous studies 28, our empirical evidence‐based shortlist can provide potential modellers with even more specific direction as to what input to choose from each of the listed categories above. This is a major contribution to the empirical literature concerning transferability of economic evidence.

The process of getting to the shortlist was not free of challenge. First, of the 46 country‐specific inputs, 21 inputs caused large sensitivity in at least one of the seven ROI estimates. One ROI estimate (ICER per life year gained) was sensitive to a total of 17 inputs, while another ROI estimate (avoided burden of disease across all smokers) was least sensitive (to seven inputs only). One could attribute this difference to the complex calculations that go into the output figure and the wide range of inputs which are needed to achieve that calculation. However, a simplified decision algorithm was needed to draw up a shortlist, and we believe the one that was used here (Table 5) is fit for purpose.

Secondly, some mention must be made of those inputs that were associated with no or very little sensitivity in ROI estimates in this analysis. They were largely the unit costs of interventions (e.g. cost of telephone support or printed self‐help materials). This was caused most probably by two things: (a) lower variation in the input values across the five countries (i.e. greater certainty in available sample data), mainly because availability of such interventions were limited and countries opted to follow the English values (an important limitation of our study); and (b) the fact that these type of interventions are much cheaper, and thus the relative cost of the intervention to the cost of treating smoking‐attributable disease is negligible. Notwithstanding the decision algorithm, it was therefore deemed reasonable to add the additional five inputs to the shortlist for practical reasons.

Finally, it is important to note that by changing only one input value at a time, our method did not take the possible correlations between different inputs into account. This is a widely known limitation of univariate sensitivity analyses 29, 30 and can be the subject of a future research project. Welte et al. 28 also stated that a multivariate or probabilistic sensitivity analysis can be a promising way to quantify the uncertainty associated with the transfer of the results of economic evaluations. However, for a model as large and sophisticated as EQUIPTMOD, pragmatism must be weighed against aspiration. Future studies arising from EQUIPT may examine this important limitation. Further areas of research may also include the analysis of correlations between inputs themselves or the analysis of the effects of changing input settings (e.g. changing the health‐care perspective to societal perspective) on the ROI estimates.

Conclusion

Countries that have limited research time and other resources can adapt EQUIPTMOD to their own settings by choosing to collect data on a small number of model inputs. EQUIPTMOD can therefore facilitate transfer of tobacco control‐related economic evidence to new jurisdictions.

Ethics approval

Not applicable to this study, as this was based on secondary data. However, the main study EQUIPT has obtained ethics approval from the Brunel University London Research Ethics Committee and relevant authorities from participating countries.

Declaration of interests

None.