Cost effectiveness of pediatric pneumococcal conjugate vaccines: a comparative assessment of decision-making tools.

BACKGROUND: Several decision support tools have been developed to aid policymaking regarding the adoption of pneumococcal conjugate vaccine (PCV) into national pediatric immunization programs. The lack of critical appraisal of these tools makes it difficult for decision makers to understand and choose between them. With the aim to guide policymakers on their optimal use, we compared publicly available decision-making tools in relation to their methods, influential parameters and results.
METHODS: The World Health Organization (WHO) requested access to several publicly available cost-effectiveness (CE) tools for PCV from both public and private provenance. All tools were critically assessed according to the WHO's guide for economic evaluations of immunization programs. Key attributes and characteristics were compared and a series of sensitivity analyses was performed to determine the main drivers of the results. The results were compared based on a standardized set of input parameters and assumptions.
RESULTS: Three cost-effectiveness modeling tools were provided, including two cohort-based (Pan-American Health Organization (PAHO) ProVac Initiative TriVac, and PneumoADIP) and one population-based model (GlaxoSmithKline's SUPREMES). They all compared the introduction of PCV into national pediatric immunization program with no PCV use. The models were different in terms of model attributes, structure, and data requirement, but captured a similar range of diseases. Herd effects were estimated using different approaches in each model. The main driving parameters were vaccine efficacy against pneumococcal pneumonia, vaccine price, vaccine coverage, serotype coverage and disease burden. With a standardized set of input parameters developed for cohort modeling, TriVac and PneumoADIP produced similar incremental costs and health outcomes, and incremental cost-effectiveness ratios.
CONCLUSIONS: Vaccine cost (dose price and number of doses), vaccine efficacy and epidemiology of critical endpoint (for example, incidence of pneumonia, distribution of serotypes causing pneumonia) were influential parameters in the models we compared. Understanding the differences and similarities of such CE tools through regular comparisons could render decision-making processes in different countries more efficient, as well as providing guiding information for further clinical and epidemiological research. A tool comparison exercise using standardized data sets can help model developers to be more transparent about their model structure and assumptions and provide analysts and decision makers with a more in-depth view behind the disease dynamics. Adherence to the WHO guide of economic evaluations of immunization programs may also facilitate this process. Please see related article: http://www.biomedcentral.com/1741-7007/9/55.

Background

The availability of a seven-valent pneumococal conjugate vaccine (PCV7), which is efficacious in preventing invasive pneumococcal disease (IPD), pneumonia, and acute otitis media (AOM) [1-3], has prompted policymakers to ponder whether or not this vaccine should be included in the national immunization program. The more recent licensure of higher valent PCV vaccines (10-valent and 13-valent pneumococal conjugate vaccines) makes this a harder task still. Cost-effectiveness (CE) analyses can be useful to aid decision making. Numerous economic evaluation studies of PCV-7 have been published since 2000 [4-29]. Despite the availability of a comprehensive critical literature review [30], there is a lack of critical review of the computational reliability and convergent validity of accessible PCV models [31]. Indeed, there is a constant need to verify the actual model structures, model attributes, input parameters, key assumptions and whether different models will produce the same outputs if standardized input parameters are used. Moreover it is not clear what the strengths and possible pitfalls of the available PCV7 models are and the potential to use it for new PCV combinations. In order to guide policymakers in particular from low-income and middle-income countries (LMIC) on the optimal use of decision-making tools, in 2009 the World Health Organization (WHO) invited model developers from both the public and private sectors to submit their tools for a systematic comparative assessment. Rather than having the aim of recommending a single tool, the overall objective of the assessment was to provide decision makers with a menu of CE tools and their characteristics for their optimal use for vaccine adoption decisions.

This paper critically reviews a number of potentially influential and accessible cost-effectiveness tools. We assess the differences and similarities between models and evaluate to which extent the outputs differ given the use of standardized input parameters.

Methods

Comparison of cost-effectiveness models

Several publicly available cost-effectiveness modeling tools with both public and private provenance were identified through literature searches and key informants and contacted by WHO (RH, AMH). Three models were provided to our research team: Pan-American Health Organization (PAHO) ProVac Initiative's [32] TriVac model version 108.5 developed by researchers from London School of Hygiene and Tropical Medicine (LSHTM) [33], the PneumoADIP model (beta version) constructed by researchers from University of Medicine and Dentistry New Jersey (UMDNJ) [34], and the SUPREMES model (version 3.0, succinct Population Research & Economic Modeling Exercise for Synflorix™) developed by modelers from GlaxoSmithKline Biologicals (GSK) [35] (Table 1). The developers of the models provided access to their tools and explained the model framework via telephone conferences to our research team (RH, NC). These tools are intended by their developers to provide modeling team with predefined set of parameters and mathematical algorithms with which users can interact in order to develop their own cost-effectiveness analyses.

Table 1

Characteristics of cost-effectiveness models (PneumoADIP, Pan-American Health Organization (PAHO) TriVac, and SUPREMES)

	PneumoADIP	PAHO TriVac	SUPREMES
Types of study	CEA and CUA	CEA and CUA	CEA and CUA
Population	Number of live birth of 10 cohorts (secular trend analysis)	Number of births 5 years prior to start of intervention and number of live birth per year	Cross-sectional analysis of entire population
Perspective	Society	Society and government	Healthcare payera
Vaccination	Pneumococcal	Pneumococcal, Rotavirus, Hib	Pneumococcalb
Outcomes	Cost per DALY averted, cost per life year saved	Cost per DALY averted, cost per life saved, cost per life year gained, cost per case averted, cost per hospitalization averted	Cost per QALY gained, cost per life year gained
Time horizon	5 years	5 years	Cross sectional
Diseases captured	Pneumococcal pneumonia, pneumococcal meningitis, invasive NPNM, otitis media	Pneumococcal pneumonia, pneumococcal meningitis, invasive NPNM	Pneumococcal pneumonia, pneumococcal meningitis, pneumococcal bacteremia, otitis mediac
Herd immunity	Included only in vaccine-targeted cohorts; no switch 'off' and 'on'	Included only in vaccine-targeted cohorts; can be switched 'off' and 'on'	Included for vaccine targeted and non-targeted population; can be switched 'off' and 'on'
Serotype replacement	Does not allow an explicit consideration of serotype replacement, (allows for serotype adjustment)	Allows for an explicit direct adjustment for serotype replacement over time	No direct adjustment for serotype replacement over time (the individual component of replacement is included in the direct vaccine effectiveness estimate)
Sequelae	Pneumococcal pneumonia, pneumococcal meningitis, invasive NPNM	Pneumococcal pneumonia, pneumococcal meningitis, invasive NPNM, pneumococcal OM	Pneumococcal meningitis, pneumococcal OM (can be switch 'off' and 'on')
Discounting	3%	3%	No need for discounting
Sensitivity analysis	NR	One-way sensitivity analysisd	One-way sensitivity analysise

aSpecified that the model can add the societal perspective as well when including indirect cost estimates.

bSpecified PCV as Synflorix™ (10-valent pneumococcal Haemophilus influenzae protein D-conjugate vaccine (PHiD-CV)).

cSpecified that other sensitivity analyses including a probabilistic one were under development. Currently, we can only perform one-way sensitivity analysis of some parameters (for example, discount, age weighted, perspective, vaccine price, percentage decline in price).

dModel has an option to include the vaccine effect in preventing otitis media due to non-typeable Haemophilus influenzae (NTHi).

eModel can vary assumptions and determine its effects on results. These assumptions changes include AOM, AOM sequelae, NTHi in invasive pneumococcal disease (IPD), NTHi in community acquired pneumonia (CAP), herd effect, and varying vaccine coverage. The model set 20% range change under and above the baseline values for most of the variables in the model. The model specifies probabilistic sensitivity analysis to be performed with @Risk software initiating acceptability curve http://www.palisade.com/risk/.

AOM = acute OM; CEA = cumulative effects analysis; CUA = cost utility analysis; DALY = disability-adjusted life years; NPNM = non-pneumonia non-meningitis; OM = otitis media; QALY = quality-adjusted life years.

We adopted the framework on infectious disease mathematical model attributes by Kim and Goldie to describe all three models [36]. Models were compared in terms of structure, attributes, input parameters and key assumptions against the WHO Guide for standardization of economic evaluations of immunization programs [37].

As models are often updated and revised, the reader should note that for each model the August 2009 version was reviewed.

Standardized input parameters

Despite the differences in terms of model structures and data requirements, we attempt to use a common standardized set of input parameters regarding country demographics, epidemiology and vaccine characteristics in order to facilitate the comparison exercise (see Table 2). Meanwhile it should be clear that only a dataset for a cohort model approach has been developed and not one for a population-based model. A cohort model will focus on the accumulated benefit over time for individuals within the cohort, whereas for a population model the focus is on the immediate benefit across different age cohorts making up the population at the time of the assessment. Hence an analysis of the SUPREMES results was difficult to assess [38]. In order to determine the robustness of the similarities of outputs independent of country-specific data sets, we chose to compare standardized input parameters for three hypothetical but typical countries each representing different regions: sub-Saharan Africa (SSA), South-East Asia (SEA) and Latin America and the Caribbean (LAC).

Table 2

Input values for standardized analysis in each hypothetical country in the sub-Saharan African (SSA) region, Southeast Asian (SEA) region and Latin America and the Caribbean (LAC) region

Model parameters	Standardized values
	SSA region	SEA region	LAC region
Population:
Total number of birth per year	60,477	932,115	140,414
Life expectancy at birth (in years)	59.45	70.62	72.9
Burden of disease:
Incidence of pneumococcal pneumonia (per 100,000 per year)	1,793	1,793	1,771
Incidence of meningitis (per 100,000 per year)	10	10	18
Incidence of invasive NPNM (per 100,000 per year)	77	77	137
Incidence of otitis media (per 100,000 per year)	400	400	400
Case fatality rate of pneumococcal pneumonia	1	1	1.8
Case fatality rate of pneumococcal meningitis	20	20	36.8
Case fatality rate of invasive NPNM	10	10	20.46
Vaccine information:
Vaccine coverage*	89%	97%	83%
Serotype coverage of pneumococcal pneumonia	60%	60%	60%
Vaccine efficacy of pneumococcal pneumonia	97.40%	97.40%	97.40%
Vaccine efficacy of pneumococcal meningitis	97.40%	97.40%	97.40%
Vaccine efficacy of pneumococcal invasive NPNM	97.40%	97.40%	97.40%
Vaccine efficacy of otitis media	57%	57%	57%
Costing information:
Dose cost	$26.35	$26.35	$26.35

Standardized input of vaccine cost (US$26.35) = £16.096 (rate US$1 = £0.61) (from http://www.xe.com/ucc/) [1-3].

NPNM = non-pneumonia non-meningitis.

Sensitivity analysis

All three models have different ways of specifying values for uncertainty and sensitivity analysis. The SUPREMES model applies a predefined non-adjustable rule of 20% adjustment upwards and downwards from the baseline value. The PneumoADIP model does not allow for sensitivity and/or uncertainty analysis, however it can be performed by repeating the model runs for specific scenarios with different input parameters. The PAHO TriVac model provides the user with a default set of parameters, specifying the maximum and minimal values of each parameter if uncertainty information from the user is lacking. We used a series of one-way sensitivity analyses to determine the major drivers of the models by using 20% adjustment upward and downward for several parameters where possible and calculated the boundary values accordingly for the different tools assessed. We adopted the hypothetical country data from the SSA region as an example of the sensitivity analysis in our study.

We categorized the influences of parameters on outputs into four groups: minimally influential (percentage change < 5%), mildly influential (5% to 30%), moderately influential (30% to 50%), and highly influential (> 50%). Tornado plots, a graphical presentation of the results of a series of sensitivity analyses, were used to help visually determine the influential parameters. Regardless of models, parameters with at least mild influence were selected and listed on the Y axis of tornado plots in order to compare the influences of parameters across models. In order to allow for direct comparisons, we chose to present the sensitivity analysis results of PAHO TriVac using 20% adjustment rather than the default values in the model.

Outputs

The outputs chosen for investigation while varying input parameters were differences in outcomes and costs as well as an incremental cost-effectiveness ratios (ICERs). The differences in outcome were presented as disability-adjusted life years (DALYs) averted for the PAHO TriVac and PneumoADIP models and quality-adjusted life years (QALYs) gained for the SUPREMES model, while the differences in cost were expressed in US dollars (US$ 2009). Similarly, the ICERs were presented as incremental costs per DALY averted for the PAHO TriVac and PneumoADIP models and as incremental costs per QALY gained for the SUPREMES model (Table 3).

Table 3

Influencing parameters for outcome, incremental cost and incremental cost effectiveness ratio (ICER) in the models

Parameters	Outcomes				Incremental costs				ICER
	DALY averted			QALY gained					Per DALY averted			Per QALY gained
	TriVac (default)	TriVac (± 20%)	ADIP (± 20%)	SUPR (± 20%)	TriVac (default)	TriVac (± 20%)	ADIP (± 20%)	SUPR (± 20%)	TriVac (default)	TriVac (± 20%)	ADIP (± 20%)	SUPR (± 20%)
Population:
Total number of births per year	+	+	++	0	+	+	+++	++++	+	+		++
Life expectancy at birth	0	+	++	++	0	0	+	0	0	+		++
Under 5 years mortality per 1,000 live births	0	++	-	-	0	+	-	-	0	++		-
Burden of disease:
Incidence of pneumococcal pneumonia (per 100,000 per year)	++	++	++	-	+	+	++++	-	++	++		-
Incidence of meningitis (per 100,000 per year)	++	+	+	-	+	+	+	-	++	+		-
Incidence of invasive NPNM (per 100,000 per year)	++	++	+	-	+	+	+	-	++	++		-
Prevalence of bacteremia sp < 10 years (cases per year)	-	-	-	++	-	-	-	+	-	-		++
Case fatality rate of pneumococcal pneumonia	0	++	++	+	0	0	+	0	0	++		+
Case fatality rate of invasive NPNM	0	++	+	++	0	0	+	0	0	++		++
Vaccine information:
Vaccine efficacy of pneumococcal pneumonia	++	++	++	++	+	+	+	+	++	++		++
Vaccine efficacy of pneumococcal invasive NPNM	+	++	+	++	+	+	+	+	+	++		++
Vaccine efficacy adjusted NTHi	-	-	-	++	-	-	-	+	-	-		++
Vaccine coverage	+	+	++	++	+	++	++++	+++	+	+		++
Serotype coverage of pneumococcal pneumonia	+++	++	++	+	+	+	++++	+	+++	++		++
Number of doses	-	-	-	+	-	-	-	+++	-	-		+++
Dose cost	0	0	0	0	0	+++	++	+++	0	++		++
Other parameters:
Coverage 20 years after vaccine introduced (dose 1)	+	++	-	-	+	++	-	-	+	++		-
Coverage 20 years after vaccine introduced (dose 3)	+	++	-	-	+	++	-	-	+	++		-
GP visits < 10 years (AOM)	-	-	-	++	-	-	-	+	-	-		++
Proportion sp of AOM							-		-	-		++
Probability of death	-	-	-	++	-	-	-	0	-	-		++

0 = no change; - = not available; + = percentage change < 5%; ++ = percentage change 5% to 30%; +++ = percentage change 30% to 50%; ++++ = percentage change > 50%.

AOM = acute otitis media; NPNM = non-pneumonia non-meningitis; NTHi = non-typeable Haemophilus influenza.

ADIP = Pneumo ADIP; SUPR = Supremes

Results

Model structures

Both the PneumoADIP and PAHO TriVac models are similar as they are deterministic state-transition models. Both models follow more than one live birth cohort over time. The models allow individuals to enter and exit over time, and thus these can be termed 'open' cohort models. This is similar to how vaccination is likely to be implemented in cohorts over time rather than in a single cohort (although from a decisional point of view, this difference in approach should not matter, as this is relevant for the budget-impact calculations, but not for the cost effectiveness). The SUPREMES model has adopted a different approach from the previous two models. Instead of following cohorts over time, it assesses the impact of the vaccine in a cross-sectional population. The developers of the SUPREMES model view their model as a prevalence-based and user-friendly model (succinct model). It relies on a comprehensive, 1-year population-based, age-compartmental modeling exercise that essentially highlights the potential importance of herd immunity effects. According to the developers, the SUPREMES model and the comprehensive model have a close match of > 95% regarding end results. The model assumes the full effects of vaccination on the vaccinated and unvaccinated population at steady state. Therefore, this model assumes that the vaccination has been implemented in the population for a long time and that the complete vaccine effects have been achieved across the whole population, such that a new post-vaccination endemic equilibrium is reached. The evaluation is made over a 1-year period using a prevalence-based approach for assessing all costs and benefits across the whole population.

In terms of structure the three tools are static-deterministic models requiring calculations based on proportions similar to a decision tree approach (see Table 1). Both the PAHO TriVac and PneumoADIP models have a default for input values estimated based on international sources, for example, UN population database, WHO, WHO-CHOICE, UNICEF, Global Burden of Disease (GBD) and Domestic Household Surveys (DHS).

Model attributes

All three models allow for vaccine effectiveness adjustment. All models estimate the vaccine effectiveness based on vaccine coverage and percentage of circulating pneumococci that are caused by the serotypes included in the vaccine (individual or all serotypes). The PAHO TriVac model allows for an adjustment for serotype replacement over time, through an input parameter representing a percentage reduction in circulating pneumococci caused by vaccine serotypes. Despite no direct adjustment for serotype replacement over time, in the SUPREMES model the effects of serotype replacement are combined with those of herd protection into an input parameter expressing the overall indirect vaccine net herd effect. The PneumoADIP model allows for serotype adjustment for each disease separately (for example, pneumococcal pneumonia, meningitis, non-pneumococcal pneumonia).

All models claim to incorporate the herd immunity effects using different approaches. The SUPREMES model estimates that the incidence of vaccine type pneumococcal-related outcomes is lowered with a certain magnitude of reduction in all age groups. This assumes the benefits of herd immunity in both vaccinated (direct effects) and unvaccinated (indirect effects) people. The PAHO TriVac model assumes indirect effects only within the cohort that is targeted for vaccination. In the cost-effectiveness results of both the PAHO TriVac and SUPREMES models herd immunity effects can be included or excluded. In the PneumoADIP model herd effects can be included in the results section. It should be noted that in the base-case cost-effectiveness analysis the herd effects are not accounted for. According to the results presented on the herd effects in the Results section, the model estimates the herd effects among the entire population excluding those less than 1 year of age. The PneumoADIP model does not allow an explicit consideration of serotype replacement to produce net herd immunity effects.

Both the SUPREMES and the PneumoADIP models consider sequelae due to meningitis and acute otitis media. The PAHO TriVac model categorizes sequelae as single and multiple sequelae, and determines the weighted average for all single and multiple sequelae. It also presents, based on the input values, an overall rate and proportion of having sequelae in those with pneumococcal disease.

Input parameters

Some of the requirements for input parameters are quite similar in all three models, but nonetheless several differences should be noted.

The assessed version of PneumoADIP model does not include default values for disease burden estimates (incidence and case-fatality rate) like the other two models. The costing components in the PneumoADIP model include direct medical and non-medical costs and cost for caregivers. The PAHO TriVac model includes both direct medical and non-medical costs and has options for household/government expenditure by disease, by geographical area, various settings and by outpatient pediatric department and inpatient pediatric department. Only direct medical costs have been included in SUPREMES model. The PneumoADIP model takes the perspective of society, the TriVac model takes the perspective of the society and government while the SUPREMES model takes the perspective of healthcare payer with an option to add the societal perspective. Both of the PneumoADIP and SUPREMES models estimated costs from a combination of acute care costs and long-term sequelae costs.

Assessing differences in outputs using standardized input parameters

We found the outputs of the similarly structured PAHO TriVac and PneumoADIP models to be comparable in terms of outcomes, costs, and incremental cost-effectiveness ratio. For example, the incremental cost per DALYs averted for the hypothetical country in the SSA region was US$3,842 and US$3,878/DALY for the PAHO TriVac and PneumoADIP models, respectively. DALYs averted and incremental costs per person were of the same order of magnitude (947 and 1,254 DALYs averted; US$60 and US$80 for TriVac and PneumoADIP, respectively). All outputs for the three hypothetical countries in the three regions are shown in Table 4.

Table 4

Outputs of all three models using standardized set of input parameters for hypothetical countries in the subA-Saharan African (SSA) region, Southeast Asian (SEA) region and Latin America and the Caribbean (LAC) region

	Hypothetical country, SSA region					Hypothetical country, SEA region					Hypothetical country, LAC region
Results	ADIP		Pan-American Health Organization (PAHO) TriVac	SUPR		ADIP		PAHO TriVac	SUPR		ADIP		PAHO TriVac	SUPR
	D	S	D/S	D	S	D	S	D/S	D	S	D	S	D/S	D	S
Outcomes:
Life years gained	5,657	1,228	18,500a	1,511	1,781	109,405	22,584	295,344a	1,511	2,228	17,568	7,648	99,204a	1,511	3,847
Death averted	-	-	567a	18	30	-	-	8,786a	18	32	-	-	2,924a	18	53
QALYs gained	-	-		2,438	3,523	-	-	-	2,438	4,123	-	-	-	2,438	5,474
DALYs averted	5,681	1,254	18,950a	-	-	109,839	23,084	301,953a	-	-	17,640	7,761	100,736a	-	-
Costs (US$):
Incremental cost per live birth population	-74,020	4,863,640	72,796,429	61,454,589	-36,736,421	84,234,567	83,385,318	1,025,138,030	61,454,589	40,278,151	-540,048	10,300,512	160,611,146	61,454,589	-28,181,083
Incremental cost per person	-1.23	80	60	84	-607	90	89	55	84	43	-3.85	73	57	84	-201
Incremental cost-effectiveness ratio (US$):
Incremental cost per DALYs averted	-	3,878	3,842	-	-	767	3,612	3,395	-	-	-	1,327	1,594	-	-
Incremental cost per life year gained	-	3,961	3,935	40,683	-	770	3,692	3,471	40,683	18,081	-	1,347	1,619	40,683	-
Incremental cost per death averted	-	127,290	-	-	-	25,763	123,289	-	-	-	-	45,206	-	-	-
Incremental cost per case averted	-	-	1,348	-		-	-	1,311	-	-	-	-	1,324	-	-
Incremental cost per hospitalization averted	-	-	9,278	-	-	-	-	10,706	-	-	-	-	9,047	-	-
Incremental cost per death averted	-	-	128,397	3,336,028	-	-	-	116,673	3,336,028	1,276,894	-	-	54,935	3,336,028	-
Incremental cost per QALYs	-	-	-	25,210	-	-	-	-	25,210	9,770	-	-	-	25,210	-

Default means the outputs using the predefined model inputs. Standardized means the outputs using the standardized model input.

aPer 20 cohorts.

ADIP = Pneumo ADIP; SUPR = Supremes; D = default; S = standardized

Identifying influential parameters in the models

The TriVac and PneumoADIP models yield incremental costs per DALY averted while the SUPREMES model yields incremental cost per QALY gained. Furthermore incremental costs, DALYs averted, and QALYs gained are also reported separately (Table 3).

The PneumoADIP model for the hypothetical country example in the SSA region proposes that the use of PCV results both in cost savings and improved health outcomes, and hence in this model PCV vaccination is the dominant option versus doing nothing (resulting in a negative ICER, which we choose not to present as it could be misleading).

The robustness of the findings in the three models was evaluated with a series of sensitivity analyses using QALYs or DALYs, incremental costs and incremental cost-effectiveness ratio as outputs and presented as tornado plots (Figures 1,2,3).

Figure 1

Tornado plots displaying sensitivity analysis (20% change) for TriVac model, PneumoADIP model and SUPREMES model for a hypothetical country in the sub-Saharan Africa (SSA) region using outcomes (disability-adjusted life years (DALYs) averted for TriVac model and PneumoADIP model, quality-adjusted life years (QALYs) gained for SUPREMES model) as an output.

Figure 2

Tornado plots displaying sensitivity analysis (20% change) for TriVac model, PneumoADIP model and SUPREMES model for a hypothetical country in the sub-Saharan Africa (SSA) region using incremental costs as an output.

Figure 3

Tornado plots displaying sensitivity analysis (20% change) for TriVac model, PneumoADIP model and SUPREMES model for a hypothetical country in the sub-Saharan Africa (SSA) region using incremental cost-effectiveness ratio as an output. For PneumoADIP model, incremental cost-effectiveness ratio was not calculated because it was dominant strategy.

Serotype coverage of pneumonia was found to be the only parameter with moderate influence on the number of DALYs averted in the PAHO TriVac model (Figure 1).

Parameters influencing cost outputs were varying depending on the models (Figure 2). For the PAHO TriVac model, the vaccine dose cost was rated a moderately influential parameter. The total number of births per year was rated as moderately influential for the PneumoADIP model, while rated as highly influential for the SUPREMES model. Vaccine coverage was highly influential for the cost outputs in the PneumoADIP model and moderately influential in the SUPREMES model. Other parameters such as serotype coverage of pneumococcal pneumonia and incidence of pneumonia were shown to be highly influential in the PneumoADIP model, while moderate influence on cost output in the SUPREMES model included vaccine dose cost and the number of vaccine doses.

For the incremental cost-effectiveness ratio (Figure 3), serotype coverage of pneumonia and the number of vaccine doses were a moderately influential parameter in the PAHO TriVac model and SUPREMES models, respectively.

In line with cost-effectiveness literature of immunization programs, discounting was another influential parameter [39,40]. In the PAHO TriVac model, the changes of this parameter from 0% to 3% and 10% resulted in an important change of DALYs averted and incremental cost per DALY averted. Note that this parameter could not be varied in the PneumoADIP model. For the SUPREMES model, discounting was not needed as the population model evaluates the impact of interventions across all ages at one cross sectional time period of 1 year [38]. Therefore, discounting was not listed as the key influential parameter in this report as it can be varied in only one model.

Discussion

To the best of our knowledge, this research work is the first study conducted with an aim to compare accessible models developed by both the public and private sectors. This review provides insights into differences and similarities of model attributes and key assumptions to understand disease dynamics and key drivers of the results. Importantly, having a full understanding why different tools produce different results will help identify data needs and pitfalls important to the users of such models. Although the models we reviewed were developed to determine value for money of vaccination program, different vaccines were assessed: PneumoADIP assessed the CE of PCV, TriVac assessed the CE of PCV, Haemophilus influenzae B (Hib) and rotavirus vaccines and SUPREMES appraised pneumococcal Haemophilus influenzae protein D-conjugate vaccine (PHiD-CV).

The different model approaches translated in different data input requirements, which made it a challenge to use a standardized data set for the tool comparison exercise. The incidence-based cohort approaches adopted in the PneumoADIP and the PAHO TriVac models are more data demanding and may be more intuitively appealing than the SUPREMES model. The latter model evaluates under static conditions an entire population at specific time point using the total population size and it specific age distribution to estimate the impact of a new vaccine program on total population health during a fixed time period (usually 1 year) and therefore requires fewer data inputs. This feature makes the SUPREMES model useful if the vaccine causes intermediate and important changes to the population and if budget-impact estimates after vaccine introduction are of interest to the decision maker. In addition, the model is easier use and can be applied widely to those countries with limited data but may be more difficult to justify as users would need to agree with several implicit assumptions embedded in the modeling approach.

It is important to note that while all three models were at an advanced stage of development at the time of the comparison exercise, models of this kind are prone to continued extension and elaboration, and weaknesses or absent features listed in this article may be corrected or improved with time. This comparison process can serve a useful purpose in this regard, by giving modelers the opportunity to identify key omissions and improve the accuracy, transparency and usability of their models. For example, since the evaluation in August 2009 the TRIVAC model now includes otitis media outcomes, advanced options for probabilistic sensitivity analysis and scenario analysis, and key assumptions about waning efficacy and age-appropriate vaccination. The present version of the PneumoADIP model allows discount rates to be varied while GSK now has developed a cohort-based version of the SUPREMES model to deal with the different vaccine impact on disease events using cohort-based or population-based model methods [38]. In addition, while none of the three models had been through a validation process at the time of the comparison exercise, the TRIVAC model has since been evaluated by an external expert panel [41]. As models and cumulative effects analysis (CEA tools evolve continued appraisal is needed.

The differences between these models are not only due to the approach and data requirements, but are difficult to distinguish without fully transparent user manuals. The TriVac model has an advanced user-friendly and self-guided function enabling users to properly interpret the research findings.

All three models include the possibility to incorporate the herd immunity effects of the vaccination program, however there are important differences in how this is incorporated and the underlying assumptions in each model. Herd immunity effects have already been shown to be influential in various analyses published before 2006 [30]. However, the closely related effects of serotype replacement, which can partly nullify the beneficial impacts of herd immunity are not taken into account in any of these models. All three models have shown that they acknowledge an important component of outcome, the morbidity or health-related quality of life, because they present the results in terms of incremental costs per DALY averted (PneumoADIP and TriVac) or per QALY gained (SUPREMES).

From the user's perspective, it is very important to be able to identify influential parameters. A series of sensitivity analyses using tornado diagrams has shown that vaccine efficacy against pneumonia, serotype coverage and replacement, vaccine coverage, and disease burden parameters including incidence and case fatality of pneumococcal diseases are the key driver of the outcomes. Unfortunately, evidence to inform one or more of these parameters is often lacking in low-income and middle-income countries and frequently parameters are populated through generalizations made using data from industrialized countries. Vaccination costs was also a parameter of major importance (though not included in the standard sensitivity analyses).

The outputs for TriVac and PneumoADIP were found to be very similar, while the outputs of the SUPREME model were markedly different from the other two models. Given the differences in various aspects such as parameters included in the models, the baseline value used in the model, and model structure, it is not possible to distinguish the sources that could explain the differences in findings. Because discounting is very influencing in this long-term outcome modeling, the lack of discounting in the SUPREME model could be considered a crucial factor that might explain, to a certain extent, the differences in QALYs gained and cost differences. To be more specific, the future outcomes earned by cohort vaccinated previously are measured at the same time for cohort currently being vaccinated.

Conclusions

The main aim of our comparison was not to make the difference between models understandable to model developers but foremost to potential users of these models at the country level. Understanding the differences and similarities of these CE tools will enhance the intended use of these tools by decision makers in terms of feeling ownership of results. This structured tool assessment exercise has shown that the models we compared differed in terms of model structure and key assumptions. Using a standardized input data set for three PCV cost-effectiveness analysis tools revealed that vaccine cost, efficacy and assumptions for critical clinical endpoints are the most influential parameters. Comparison of CE tools should help local policymakers understand the different approaches used in these models, and their advantages and disadvantages in relation to the data they have at their disposal to conduct and economic analysis. For instance, due to the complex nature of the transmission and ecology of pneumococci, many simplifying assumptions were implicitly made about the transmission dynamics in each of these models. All assumptions and simplifications should be well understood, and be transparent to users of such models. Models should be as transparent as possible according to the criteria and requirements presented in the WHO CEA guidelines [37]. By identifying the most relevant key parameters influencing the results of these models, country and region-specific data collection could be focused as appropriately as possible. Priority should be given to improve the data to inform key parameters. In addition, efforts to improve and develop transmission dynamic models with realistic age structures that better describe the natural history of Streptococcus pneumoniae infection and progression to pneumococcal disease and capable of improved investigation of herd immunity and serotype replacement effects are also important. Lastly, the fact that the tools are evaluated and other tools are publicly available enables different stakeholders to learn from each other's experiences. Adherence to the WHO guide of economic evaluations of immunization programs may also facilitate this process [37].

Competing interests

JE's partner is currently employed for GlaxoSmithKline, manufacturer of a pediatric pneumococcal vaccine. All the other authors declare that they have no competing interests. AMH and RH are staff members of the World Health Organization. The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions, policy or views of the World Health Organization.

Authors' contributions

RH and AMH conceived the study. NC and RS integrated the results from different models and drafted the manuscript with input from the other authors. All authors provided advice on the methodology and the data analyses of tool comparison exercise. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1741-7015/9/53/prepub