Economic evaluation of vaccines in Canada: A systematic review.

BACKGROUND: Economic evaluations should form part of the basis for public health decision making on new vaccine programs. While Canada's national immunization advisory committee does not systematically include economic evaluations in immunization decision making, there is increasing interest in adopting them. We therefore sought to examine the extent and quality of economic evaluations of vaccines in Canada.
OBJECTIVE: We conducted a systematic review of economic evaluations of vaccines in Canada to determine and summarize: comprehensiveness across jurisdictions, studied vaccines, funding sources, study designs, research quality, and changes over time.
METHODS: Searches in multiple databases were conducted using the terms "vaccine," "economics" and "Canada." Descriptive data from eligible manuscripts was abstracted and three authors independently evaluated manuscript quality using a 7-point Likert-type scale scoring tool based on criteria from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR).
RESULTS: 42/175 articles met the search criteria. Of these, Canada-wide studies were most common (25/42), while provincial studies largely focused on the three populous provinces of Ontario, Quebec and British Columbia. The most common funding source was industry (17/42), followed by government (7/42). 38 studies used mathematical models estimating expected economic benefit while 4 studies examined post-hoc data on established programs. Studies covered 10 diseases, with 28/42 addressing pediatric vaccines. Many studies considered cost-utility (22/42) and the majority of these studies reported favorable economic results (16/22). The mean quality score was 5.9/7 and was consistent over publication date, funding sources, and disease areas.
CONCLUSIONS: We observed diverse approaches to evaluate vaccine economics in Canada. Given the increased complexity of economic studies evaluating vaccines and the impact of results on public health practice, Canada needs improved, transparent and consistent processes to review and assess the findings of the economic evaluations of vaccines.

Introduction

Canadian public health expenditures on vaccines amount to approximately $450 million annually and are expected to grow considerably in the decade to come. While the majority of vaccines have traditionally been cost-saving, the cost-effectiveness of new vaccines – such as those protecting against meningococcal and pneumococcal disease – have recently been at the center of considerable debate. As such, decision makers looking to adopt new vaccines into health care systems are faced with evaluating the economic value of these new vaccines relative to other alternative uses of health care budgets.

In Canada and abroad, wide scale adoption of some vaccines post-licensure has not occurred because their economic value has been questioned. The World Health Organization (WHO) recommends that countries establish technical advisory committees that serve as a resource to health authorities and as deliberative bodies to formulate guidance enabling evidence based decisions. These independent “National Immunization Technical Advisory Groups” (NITAGs) are to advise on vaccine adoption based on obvious factors such as vaccine efficacy and safety, but also on economic considerations, such as the cost-effectiveness of the immunization program and affordability. Federal countries with a tradition in evidence-based public health practice such as the US, UK, Australia and Canada have longstanding NITAGs that provide recommendations to their respective jurisdictions. All of these countries, with the exception of Canada, have coherent and transparent processes for the evaluation of economic evidence in immunization decision making. This clear gap in Canadian vaccine evaluation capacity is not mirrored on the drug evaluation process. Both the Common Drug Review (CDR) and the Pan-Canadian Oncology Drug Review (pCODR) have world class capabilities and processes for the review of economic evidence necessary for drug adoption decisions. The gap has been highlighted in a published report by multiple Canadian stakeholders including the Public Health Agency of Canada, the vaccine industry committee and academics.

In the absence of a technical adjudicating committee on the economics of vaccines, national and provincial immunization advisory bodies in Canada may be relying heavily on available published studies and on unpublished presentations made by industry or academic researchers. Given the potential unfiltered impact of the literature on decision making, we set out to conduct a systematic review of economic evaluations published on vaccines in Canada. Our goal was to critically assess the comprehensiveness across jurisdictions, studied vaccines, funding sources, study quality, changes over time, and to summarize their major findings.

Methods

Search strategy

To identify all published economic evaluation studies of vaccines and vaccination programs in Canada, we conducted a literature search in MEDLINE (National Library of Medicine), EconLit (ProQuest), Web of Science (Thomson Reuters), EMBASE (Elsevier), the Cochrane Collaboration, Scopus (Elsevier), and HealthSTAR (Ovid). MEDLINE searches were performed via PubMed using the combination of “Vaccines” and “Economics & Statistics and Numerical Data” MeSH terms. We limited our search results to publications in Canada. Other databases were searched using a combination of terms: economic evaluations and appraisals; immunization programs and vaccines; and Canada. The detailed search strategies are described in Table S1.

Inclusion and exclusion criteria

Studies were included if they were published in English language peer-reviewed journals, generated original economic evaluation estimates of specific vaccines or immunization programs, and were applied to the Canadian setting. Research articles were screened in two stages. Titles and abstracts of the all retrieved citations were first reviewed by two authors (MS and VHN) for relevance against inclusion and exclusion criteria. In the second stage, the full text articles of included citations were screened for relevance against inclusion and exclusion criteria by 4 authors (AC, PG, JW, and JML). See Fig. 1 for further details of the screening procedure.

Figure 1.

Flowchart of study selection for systematic review of vaccine economics in Canada.

Data collection and quality scoring

A standardized template was used in the abstraction of the research articles that satisfied our screening criteria. Three authors (MS, JL and AC) reviewed each article and performed the data abstraction that collected descriptive characteristics on the following: study objective, study population, intervention(s), time horizon, outcome measures, cost measurements, discount rates for outcomes and costs, geographical location, vaccine type, type of mathematical model used, intervention effectiveness measure, herd effect consideration, study perspective, sensitivity analyses used, prices and costs, and threshold values or conclusions on economic efficiency. We summarized the descriptive characteristics and the findings of the cost-utility studies.

A scoring tool was developed to assess the quality of the economic evaluation methods used in each article. Using the criteria from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) a Likert scale (1-7) was created to assign scores to three components: clinical and epidemiological evidence, mathematical model, and economic analysis. Three reviewers; JML, JW and PG, independently evaluated and scored the articles along these three dimensions. JML scored the articles for the quality of the epidemiological evidence, JW scored the articles for the quality of the mathematical model, and PG scored the articles for the quality of the economic analysis. Each article therefore had three Likert scores. These scores were then averaged to produce an overall quality score for each paper. Table SII contains the scoring tool used by the reviewers.

To study factors that may impact study quality, we examined the correlation of quality with publication date, analysis type (cost benefit, cost effectiveness or cost utility), funding source, industry author participation, and journal impact factor. In addition we used the Mann-Whitney non-parametric rank test to explore if industry funded cost-utility studies would report more favorable results compared to cost-utility studies not funded by industry. In assessing whether the vaccines evaluated in the cost-utility studies were cost effective, we elected to use the implicit Canadian cost-effectiveness threshold (between $20,000/QALY to $100,000/QALY), rather than the WHO recommended threshold. Since there is no explicit economic threshold in Canada, there is no mandate for the government to adopt or reject a technology because it crosses a pre-specified cost-effectiveness value. Further, we reviewed statements of Canada's National Advisory Committee on Immunization (NACI) to determine if any of the retrieved studies were cited in recommendation statements. NACI was chosen as they are Canada's national technical advisory group on vaccines and their recommendations have long been followed closely at the provincial level in Canada.

Results

The overall characteristics of the 42 studies that met our screening criteria, published between 1993 and 2013, are summarized in Table 1 (Table SIII provides detailed descriptions of each individual study we have identified). We observed an increase in the number of studies per year over time (Fig. 2 panel A). Vaccine manufacturers funded 40% of the studies and government funded 17%. Only 5% of studies were funded through academic research grants (Table 1), although there appears to be an increase in co-funded studies in recent years (Fig. 2 panel A). The majority of studies (60%) adopted a national perspective, 35% the perspective of the 3 most populous Canadian provinces (Ontario, Quebec and British Columbia) and only 4% of studies examined programs from the perspective of the remaining 7 provinces and 3 territories in Canada (Table 1). We observed studies against 10 diseases (Table 1), although there were no reported studies of older vaccines such as Smallpox, Tetanus, Diphtheria, Polio, Rubella, Mumps, and Haemophilus influenzae. Further, there were no studies of commonly recommended travel and specialty vaccines such as Typhoid fever, Yellow fever, Cholera, Japanese Encephalitis, Rabies and Tuberculosis.

Table 1.

Characteristics of canadian health economic studies on vaccines.

	Number	%
Study Design
Modeling research	38	90%
Experimental research	4	10%
Disease
Pneumococcal Disease	8	19%
Influenza	6	14%
Hepatitis B	5	12%
HPV/Cervical Cancer	5	12%
Pertussis	5	12%
Meningococcal Disease	4	10%
Zoster	2	5%
Rotavirus	2	5%
Varicella	2	5%
Measles	1	2%
Hepatitis A	1	2%
Hepatitis C	1	2%
Focus of Study
Canada	25	60%
Ontario	6	14%
Quebec	6	14%
British Columbia	3	7%
Alberta	1	2%
Manitoba	1	2%
Population
Pediatric	28	67%
All ages	8	19%
Adolescents	3	7%
Adults ≥ 65 y	3	7%
Funding Source
Industry only	17	40%
Government only	7	17%
Multiple Sources	5	12%
Research Grants only	2	5%
Non-Profit only	1	2%
Not Determined	10	24%
Analysis Type
Cost-utility	22	52%
Cost-effectiveness	39	93%
Cost-benefit	42	100%

Figure 2.

Evolution of Canadian economic evaluations of vaccines over time. Temporal trends in (A) funding source and (B) study design of 42 studies published from 1993-2012.

All economic analyses took the form of decision analysis, with the majority of studies reporting benefit in the form of quality adjusted life years (QALYs). The proportion of these cost-utility studies increased over time and as of 2007 they became the most widely used economic evaluation (Fig. 2 panel B). Table 2 summarizes the results of the 22 cost-utility studies. Only 2 of these studies reported results that were not cost-effective based on the implicit Canadian cost-effectiveness thresholds, with 7 studies reporting that the vaccine was dominant (the intervention costs less and is at least as effective as the comparator). 13 out of 22 studies did not report results from a societal perspective which underestimate the full benefit of the vaccines. Out of these studies, only one reported that the vaccine was not cost-effective (as per the WHO threshold), while 4 others reported dominance. In the Mann-Whitney rank test to explore if industry funding was associated with higher ICERs, the mean rank for industry funded studies was 11.38, while the mean rank of non-industry funded studies was 11.6. This would indicate that industry funding was not associated with more favorable results when compared to studies that relied on non-industry sources of funding.

Table 2.

Summary of Canadian cost-utility studies.

			Base Case Median Incremental Cost-Effectiveness Ratio ($/QALY*)
Publication Year	Reference	Vaccine Type	Societal Perspective	Payer Perspective	Study Conclusion
2002	25	Meningococcal	68,000	NR	Cost-effective‡
2004	26	Meningococcal	42,000	NR	Cost-effective
2005	27	Hepatitis C	D§	NR	Dominant
2007	28	Hepatitis A	NR†	D	Dominant
2007	29	HPV	NR	25,786	Probably Cost-effective‡
2007	30	Meningococcal	113,000	NR	Not Cost-effective
2008	31	Herpes Zoster	NR	33,000	Probably Cost-effective
2008	32	HPV	NR	1,249-3,291	Cost-effective
2009	33	Herpes Zoster	NR	41,709	Probably Cost-effective
2009	34	HPV	NR	27,398	Probably Cost-effective
2009	35	Pneumococcal	466	18,000	Cost-effective
2010	36	Influenza	NR	9,388	Cost-effective
2010	37	Influenza	NR	12,154	Cost-effective
2010	38	Pneumococcal	NR	D	Dominant
2011	39	Hepatitis B	NR	3,648,123	Not Cost-effective‡
2011	40	HPV	NR	1,839	Cost-effective
2011	41	Influenza	NR	1,612	Cost-effective
2011	42	Influenza	D	NR	Dominant
2011	43	Pertussis	NR	D	Dominant
2012	44	HPV	NR	D	Dominant
2012	45	Rotavirus	NR	2,400	Cost-effective
2012	46	Rotavirus	D	115,000	Dominant

QALY: Quality-adjusted Life Years

NR: Not reported

D: Dominant: The intervention costs less and is at least as effective as the comparator.

Cost-effectiveness: defined by Canada's implicit threshold of $20,000/QALY – 100,000/QALY

Categories of cost-effectiveness:

Cost-effective – less than $20,000 CAD/QALY;

Probably cost-effective – between $20,000 CAD/QALY –$100,000 CAD/QALY;

Not cost-effective – greater than $100,000 CAD/QALY.

The majority of published studies were mathematical modeling studies that simulate expected costs and benefits through decision models. There were only 4 evaluations that did not use a mathematical model but instead empirically measured actual costs and benefits of established immunization programs (Table SIII). We noted an increase in the complexity of the mathematical models employed over time through the introduction of several dynamic transmission models in the latter years (Table SIII).

On the 7 point scale, the mean quality score was 5.9 (95% CI: 5.6, 6.2). The mean 5-year impact factor of the journals in which the studies were published was 3.8 (95% CI: 3.1, 4.6). Both mean quality score and mean impact factor did not vary by publication year, disease type or funding source, or industry author participation (Fig. 3; Table SIV, Figs. SI and SII). Notably, 43% of the studies were published in the journal Vaccine, and the mean quality of these studies was 6.2, with a range from 5.3 to 7.0. We observed a slight positive correlation (Slope: y = 0.0977x + 5.5479, R2: 0.0841) between journal impact factor and study quality (Fig. SIII). While economic evaluations are not within the purview of NACI, 5 economic evaluations were cited in 8 NACI recommendations, on the topics of Meningococcal disease, influenza, varicella and Herpes Zoster. Compared to the broader group of articles, NACI cited articles have a similar mean quality score (6.3 vs. 6.0) and a slightly higher mean 5-year impact factor of (5.95 vs. 3.83). The details of the NACI statements and the economic studies that are cited by these statements are summarized in Table SV.

Figure 3.

Quality and impact factor of Canadian economic evaluations of vaccines. The temporal trends in (A) mean quality score and (B) 5-year impact factor of the 42 publications from 1993-2012 identified in this study. Also presented are the average (C) quality score and (D) 5-year impact factor of the studies grouped by funding source. Quality scores were determined by an expert panel evaluating the clinical and epidemiological evidence, mathematical model, and economic analyses performed in each publication. Note: error bars denote the range of scores for a given year or funding source.

Discussion

This is the first systematic review of the literature on economic evaluations of vaccines in Canada. We found that Industry funded the majority of the studies in the literature (17/42 studies), followed by government (7/42 studies), while academic research grants only funded a few (2/42) studies. 22/42 studies considered cost-utility analysis and 16/22 of these reported favorable economic results, i.e. below the Canadian implicit cost-effectiveness threshold of $20,000 - $100,000/QALY. However, had the WHO-recommended cost-effectiveness threshold (up to 3x the per-capita GDP/QALY) been used in our analysis, all but one cost-utility study would have been reported favorable results. 38/42 studies employed mathematical models and we noted an increase in model complexity over time. Canada-wide economic evaluations made up most of the literature (25/42 studies), while the few provincial level studies focused on populous provinces. Overall, the quality of the literature was good, with a mean quality score of 5.9 on a 7 point Likert scale. This was consistent over time, across funding sources, and disease areas.

In line with the mean quality score, the mean score for the modeling component was 5.9/7. We found that most Canadian vaccine studies were aligned to the guidance offered in the report of the ISPOR-SMDM modeling task force with regard to communicable diseases. Compared to the broader group of articles, NACI cited articles have a similar mean quality score (6.3 vs. 6.0) and come from journals with a slightly higher mean 5-year impact factor of (5.95 vs. 3.83). However, the higher 5-year impact factor is driven by the Sander et al. article which was published in a journal with 5-year impact factor of 16.43. Indeed, this article is an outlier when considering all articles in our study as the next highest 5-year impact factor score is 4.83 and the lowest value in our overall study range is 1.56.

There are no widely adopted standards to assign a quality score to health economic evaluations and authors attempting to asses literature quality have taken a variety of approaches. Our approach was to task three Canadian experts, one in each of the three major disciplines in health economic evaluations: 1) clinical epidemiology, 2) modeling and 3) economics; to assess the papers on these three critical dimensions. We believe that this approach is relatively robust as the experts each have over 20 years' experience in their relevant discipline and are all professors at Canadian academic institutions. Our approach could have been improved through blinding reviewers to the study authors and employing a larger number of reviewers to generate multiple scores per category. However, these improvements would be challenging to implement. Notably, the small nature of the scholarly community studying the economics of vaccines in Canada makes it challenging to find suitable reviewers and to blind the ones enlisted in the reviews.

Cost-utility studies, which account for more than half of the literature, point to a potential lost opportunity in Canada. These studies are unique in that their results can be compared to other interventions across the healthcare system. This is facilitated through the use of the QALY as the unit of benefit. A rational healthcare system is one that seeks to maximize QALY gains and minimize incremental cost. Health technologies, policies and interventions will all have a different incremental costs/QALY gained and health care resources should be diverted to areas demonstrating the lowest incremental cost/QALY. As a pragmatic course of action the WHO recommend the adoption of health technologies that fall below a specific cost/utility threshold. In this study we used the implicit Canadian threshold to make this assessment. We observed that all but 2 of the cost-utility studies reported results that would be considered higher than the upper limit of the Canadian threshold $100,000/QALY. Several of these programs, however, are not universally adopted across Canada. For instance, quadrivalent Meningococcal (42,000/QALY – $113,206/QALY), Herpes Zoster ($33,000/QALY), and Rotavirus (cost-saving) vaccine programs have seen limited to no adoption across Canadian jurisdictions. It should be noted that these vaccines are approved by Health Canada, and NACI statements are supportive of the use of these vaccines in Canada.

In our limited data set we did not observe that industry funded studies were of lower quality than non-industry funded ones. Nor did we observe a bias in industry funded studies toward reporting more favorable results. Further, an analysis of studies which included industry authors showed similar results. Industry authorship was not associated with lower quality or a lower journal impact factor. These findings are not surprising. A report by Barbieri and Drummond outlined that industry bias in economic evaluation is generally observed in the selection of technologies to study rather than in the economic evaluations themselves. Industry is less likely to invest resources in studying the economics of established vaccines and is less likely to report a vaccine as not being cost-effective given industry's ability to modify prices.

It was not evident from our literature review or from the NACI statements why vaccines reported to be cost-effective in the literature were not widely adopted in Canada. While there might be good reasons for this, the rationale for such decisions are not publically accessible. The absence of a formal and transparent national process for the review of economic evaluations of vaccines in Canada is therefore noteworthy, especially given the contrasts with the robust infrastructure for review of some other health technologies. For example, the pan Canadian Oncology Drug Review (pCODR) process, which is used by all provinces with the exception of Quebec, has established early engagement with industry to plan for new technology introduction; capacity to conduct technical reviews in a pre-specified time frame; the ability to complete full review and recommendations in parallel with the Health Canada market authorization process; and transparency on technical review details and recommendation deliberations. Many countries have established processes for including economic evaluations into immunization decision making. The United States Advisory Committee on Immunization Practices (ACIP), for example, has explicit standards and processes for economic evidence that can be considered in the recommendation process. As such, ACIP recommendations are based on thoroughly vetted economic data. Upon a positive ACIP recommendation, the vaccine is then provided by law. In the UK, the Joint Committee on Vaccines and Immunization (JCVI) sets an explicit monetary threshold for cost-effectiveness of vaccines for public health programs. In Australia the Pharmaceutical Benefits Scheme (PBS) reviews the economic evidence for both drugs and vaccines in Australia. These established methodologies can serve as templates for Canada, or other countries, that seek to incorporate economic evidence into their vaccine program planning.

Our review has two limitations. First, there may be a tendency to published studies with favorable or desired results. For instance, industry might not be motivated to publish results that indicate that a vaccine is not cost-effective. Similarly, government might not be motivated to publish on a vaccine that is cost-effective if they are unable to fund it. A second limitation is related to the overall small number of studies. This made quantitative analysis difficult and we could not make any robust conclusion on the statistical significance of any of our trend analysis.

In conclusion, we observed diverse approaches to evaluate vaccine economics in Canada. Since economic analyses may influence public health decision making about immunization programs, it is important to be aware of how varying methodologies can affect results, and interpretation of results. Canada does not have a systematic process to review and assess the findings of the economic evaluations of vaccines; as such, there is some risk of variation in understanding published economic analyses and their implications.