Benefit-Risk Assessment of Vaccines. Part I: A Systematic Review to Identify and Describe Studies About Quantitative Benefit-Risk Models Applied to Vaccines.

INTRODUCTION: Understanding the balance between the benefits and risks of vaccination is essential to ensure informed and adequate public health decision making. Quantitative benefit-risk models (qBRm) represent useful tools to help decision makers with supporting benefit-risk assessment throughout the lifecycle of a medical product. However, few initiatives have been launched to harmonise qBRm approaches, specifically for vaccines.
OBJECTIVES: The aim of this paper was to identify publications about qBRm applied to vaccines through a systematic literature review, and to describe their characteristics.
METHODS: Medline, Scopus and Institute for Scientific Information Web of Knowledge databases were searched to identify articles in English, published from database inceptions up to December 2019. The search strategy included the combination of three key concepts: 'benefit-risk', 'modelling' and 'vaccines'. Data extracted included the modelling context and the methodological approaches used.
RESULTS: Of 3172 publications screened, 48 original publications were included. Most of the selected studies were published over the past decade and focused on rotavirus (15), dengue (10) and influenza (6) vaccines. The majority (30) of studies reported analyses related to high-income countries. The methodology of the studies differed, particularly in modelling techniques, benefit-risk measures, and sensitivity analyses. The present work also pointed out a high level of variability in the quality of reporting across studies, with particular regard to input parameters and methodological approaches.
CONCLUSIONS: This review provides an extensive list of qBRm applied to vaccines. Discrepancies across studies were identified during our review. While the number of published qBRm studies is increasing, no reporting guidance for qBRm applied to vaccines is currently available. This may affect decision makers' confidence in the results and their benefit-risk assessment(s); therefore, the development of such reporting guidance is highly needed.

Key Points

Introduction

Vaccination is one of the most successful public health achievements for disease prevention [1]. Compared with most other medicines, vaccines have some unique features; (i) they can be administered to large populations of healthy subjects, including children [2], and (ii) they can be introduced by health authorities as mandatory [3]. These specificities require vaccines to have a highly favourable benefit–risk profile [4]. In practice, this means that the benefits of vaccination need to be compared with its risks, in order to ensure informed and adequate public health decision making. Benefit–risk assessment (BRA) is the basis of regulatory decisions in both pre- and post-marketing review processes [5]. Although qualitative judgments have been used in BRA of health interventions, quantitative benefit–risk models (qBRm) may provide better consistency, transparency and predictability of decision making [6-8].

qBRm integrate evidence from multiple sources to quantify and put into perspective the benefits and risks of a health intervention. They can be simulation models (applying simulation techniques of various complexity degrees to estimate final outcomes) or non-simulation models (using a simple calculation to obtain final outcomes) [8-10]. They have been increasingly considered by decision makers (regulatory authorities, pharmaceutical companies, payers, guideline developers, etc.) to support the BRA of drugs and vaccines throughout their lifecycle [11-13]. However, although the advantages of using qBRm to weight benefits against risks are recognised by some regulatory authorities, formal guidelines are currently not available [14-16].

During the last decade, several initiatives have been undertaken to harmonise qBRm approaches applied to drugs through the identification, appraisal and classification of qBRm methods and the development of recommendations or frameworks to perform qBRm; for example, the Benefit–Risk Methodology Project driven by the European Medicines Agency (EMA) [9, 17], the Pharmacoepidemiological Research on Outcomes of Therapeutics (PROTECT) consortium in Europe [8, 10, 18–21], the Unified Methodologies for Benefit–Risk Assessment (UMBRA) [22] and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Risk–Benefit Management Working Group [23].

Vaccine specificities may require specific modelling approaches, like those considering dynamic transmission and the occurrence of both individual and indirect protection through herd immunity. For this reason, similar initiatives aiming at streamlining qBRm approaches applied to vaccines have been launched, that is, the Accelerated Development of Vaccine benefit–risk Collaboration in Europe (ADVANCE) [24] and the Vaccine Monitoring Collaboration for Europe (VAC4EU) [25].

However, to date, a comprehensive review of the literature about qBRm applied to vaccines, comprising details on the modelling context and methodological approaches used, is lacking. To this end, we developed this first paper (Part I) aiming at systematically reviewing publications about qBRm applied to vaccines, and to describe their characteristics. Based on this mapping exercise and in order to ease the interpretation of future qBRm, we developed a second paper (Part II) that proposes standards in reporting qBRm applied to vaccines [26].

Materials and Methods

The definitions of technical terminologies (quantitative and qualitative BRA, qBRm) used in this article, and adapted from the glossary developed by PROTECT [8, 27], are provided in the Appendix Table 1 (see Electronic Supplementary Material [ESM]).

Literature Search and Study Selection

Medline, Scopus and the Institute for Scientific Information (ISI) Web of Knowledge databases were searched to identify articles published from database inceptions up to 31 December 2019. The search strategy combined three key concepts: (i) benefit–risk, (ii) modelling and (iii) vaccines. The search was conducted on 13 March 2020 and was limited to articles in English (see Appendix Table 2, ESM).

All citations were downloaded and imported in the reference management software EndNote version X7 (Thomson-Reuters Corp, New York, NY, USA). Duplicate citations were identified and excluded using the reference management software and manual title examination. Two reviewers (HA and NP) independently screened all titles and abstracts using predefined exclusion criteria (see Appendix Table 3, ESM). Subsequently, all publications retained after screening were assessed for eligibility by examining their full text. Disagreements between the two reviewers were resolved through discussion. A secondary manual search was performed through snowballing the reference lists of the eligible articles to identify additional articles. A grey literature search was performed through searching major public health organisation websites and using targeted search terms in Google.

Data Extraction

The following data were extracted by reviewers: (i) the general information including study publication date and the funding source; (ii) the modelling context including vaccine indication, the geographical location and the income level of the countries considered in the analyses according to the World Bank classification, the targeted population, the alternatives used for comparison with standard vaccination and the perspectives from which the intervention’s benefit and risk outcomes are evaluated; and (iii) the methodological approaches, that is, the modelling techniques, the model attributes, the benefit–risk measures, the sensitivity/scenario analyses and whether a standardised framework, a tabular representation summarising input parameters, discount rates and utility or preference information, were used or not. See Appendix Table 4 (ESM) for more details and definitions.

The data extraction template was developed based on similar initiatives about qBRm applied to drugs, particularly on the modelling context [9, 18, 28, 29] and the methodological approaches [9, 10, 17, 21, 23], and was adapted to consider vaccine specificities (i.e. dynamic modelling technique, herd immunity and waning effect attributes) [24, 30, 31].

Two reviewers (HA and KB) independently extracted data from the selected articles. Disagreements were resolved through discussion. The authors of the papers selected for data extraction were not contacted to provide additional information.

Tables 1 and 2 describe the general information, the modelling context and the methodological approaches of each study. Table 3 summarises the distribution of methodological approaches depending on the use of a simulation model or non-simulation model.

Table 1

Characteristics of the general information and the modelling context used in quantitative benefit–risk models applied to vaccines

Source	General information		Modelling context
	Publication date	Pharmaceutical company funding	Indication	World Bank income groupsa	Targeted population	Alternatives	Perspectives
Bollaerts et al. [32]	2018	Yes	Rotavirus	HIC (UK)	Paediatric	No vaccination	Population
Bollaerts et al. (1) [33]	2019	Yes	Pertussis	HIC (Europe)	Paediatric	No vaccination	Population
Bollaerts et al. (2) [34]	2019	Yes	Pertussis	HIC (Europe)	Paediatric	Several alternatives	Population
Bruijning-Verhagen et al. [35]	2018	No	Rotavirus	HIC (Netherlands)	Paediatric	Several alternatives	Both
Bruun et al. [36]	2019	No	Rotavirus	HIC (Norway)	Paediatric	Several alternatives	Population
Carlin et al. [37]	2013	No	Rotavirus	HIC (Australia)	Paediatric	No vaccination	Population
Cho et al. [38]	2010	No	Meningococcal	HIC (US)	Not specified	No vaccination	Population
Clark and Cameron [41]	2006	No	Tuberculosis	HIC (Canada)	Paediatric	No vaccination	Population
Clark et al. [39]	2014	No	Rotavirus	HIC (England)	Paediatric	Several alternatives	Population
Clark et al. [40]	2019	No	Rotavirus	LMIC (135 countries)	Paediatric	Several alternatives	Population
Coudeville et al. [43]	2016	Yes	Dengue virus	Worldwide	Paediatric	No vaccination	Population
Coudeville et al. [42]	2018	Yes	Dengue virus	LMIC (Philippines)	Paediatric	No vaccination	Both
Desai et al. [44]	2013	No	Rotavirus	HIC (US)	Paediatric	No vaccination	Population
Desai et al. [45]	2012	No	Rotavirus	LMIC (Latin American countries)	Paediatric	No vaccination	Population
Ferguson et al. [46]	2016	No	Dengue virus	Worldwide	Paediatric	No vaccination	Both
Flasche et al. [47]	2016	No	Dengue virus	Worldwide	Paediatric	No vaccination	Population
Hladish et al. [48]	2016	No	Dengue virus	LMIC (Mexico)	Paediatric	Several alternatives	Population
Kelly et al. [49]	2010	No	Influenza	HIC (Australia)	Paediatric	No vaccination	Population
Kent [50]	1992	No	Influenza	HIC (US)	Not specified	No vaccination	Individual
Koplan et al. [51]	1979	No	Pertussis	HIC (US/Western Europe)	Paediatric	Other vaccine(s)	Population
Lamrani et al. [52]	2017	No	Rotavirus	HIC (France)	Paediatric	No vaccination	Population
Ledent et al. [54]	2016	Yes	Rotavirus	HIC (Japan)	Paediatric	No vaccination	Population
Ledent et al. [53]	2018	Yes	Rotavirus	HIC (France)	Paediatric	No vaccination	Population
Ledogar et al. [55]	2009	No	Measles	LMIC (Pakistan)	Paediatric	No vaccination	Population
Lee et al. [56]	2005	No	Pertussis	HIC (US)	Several populations	Several alternatives	Population
Lourenco and Recker [57]	2016	No	Dengue virus	Worldwide	Paediatric	Several alternatives	Population
Maier et al. [58]	2017	No	Dengue virus	LMIC (Brazil)	Paediatric	No vaccination	Population
Manissero et al. [59]	2008	No	Tuberculosis	HIC (Europe)	Paediatric	Several alternatives	Population
Marcelon et al. [60]	2016	Yes	HPV	HIC (Europe)	Paediatric	No vaccination	Population
Maro et al. [61]	2014	No	Influenza	HIC (US)	Not specified	Several alternatives	Population
Meltzer[62]	2003	No	Smallpox	HIC (US)	Several populations	No vaccination	Individual
Monath [63]	2012	Yes	Yellow fever virus	Worldwide	Several populations	No vaccination	Population
Nokes and Anderson [64]	1991	No	Mumps	HIC (England/Wales)	Paediatric	Other vaccine(s)	Both
Oleksiewicz et al. [65]	2015	No	Smallpox	HIC (Denmark)	Whole population	Several alternatives	Population
Onorato et al. [66]	1989	No	Measles	HIC (US)	Paediatric	Several alternatives	Population
Oster et al. [67]	2010	Yes	Influenza	HIC (US)	Paediatric	Other vaccine(s)	Population
Patel et al. [68]	2009	No	Rotavirus	LMIC (117 countries)	Paediatric	Several alternatives	Population
Patel et al. [70]	2011	No	Rotavirus	LMIC (Brazil/Mexico)	Paediatric	No vaccination	Population
Patel et al. [69]	2012	No	Rotavirus	LMIC (158 countries)	Paediatric	Several alternatives	Population
Perkins et al. [71]	2016	No	Dengue virus	LMIC (Peru)	Paediatric	No vaccination	Population
Phillips et al. [72]	2013	No	Influenza	HIC (Europe)	Whole population	No vaccination	Population
Rodriguez-Barraquer et al. [73]	2014	No	Dengue virus	LMIC (Thailand)	Paediatric	No vaccination	Population
Rothberg et al. [74]	2002	No	Varicella	HIC (US)	Paediatric	Several alternatives	Both
Rouillon and Waaler [75]	1976	No	Tuberculosis	Worldwide	Paediatric	No vaccination	Population
Shim [76]	2019	No	Dengue virus	LMIC (Mexico)	Paediatric	No vaccination	Population
Tosti et al. [77]	1999	No	HBV	HIC (Italy/France)	Several populations	No vaccination	Population
Yung et al. [78]	2015	No	Rotavirus	HIC (Singapore)	Paediatric	No vaccination	Population
Zalkind and Shachtman [79]	1980	No	Influenza	HIC (US)	Whole population	No vaccination	Individual

HBV hepatitis B virus, HIC high-income countries, HPV human papillomavirus, LMIC low/middle-income countries, US United States

aThe income level of the countries in the analyses were based on World Bank income groups (LMIC ≤ US$12,235 < HIC) [89]

Table 2

Characteristics of the methodological approaches used in quantitative benefit–risk models applied to vaccines

Source	Methodological approaches
	Framework	Modelling techniques		Model attributes					Discount rates	Utility or preference information solicited	Benefit–risk measures	Sensitivity/scenario analyses
		Simulation vs Non-simulation model	Type of model	Herd immunity	Waning effect	Open/closed model	Deterministic/probabilistic model	Aggregate/individual model
Bollaerts et al. (2018) [32]	Yes	Simulation	Not specified	No	Yes	Closed	Probabilistic	Aggregate	No	No	Both	Sensitivity
Bollaerts et al. (2019 (1)) [33]	Yes	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	No
Bollaerts et al. (2019 (2)) [34]	Yes	Simulation	Microsimulation	No	No	Closed	Probabilistic	Individual	No	Yes	Both	Sensitivity
Bruijning-Verhagen et al. (2018) [35]	No	Simulation	Not specified	Yes	Yes	Closed	Probabilistic	Aggregate	Yes	No	Both	Both
Bruun et al. (2019) [36]	No	Non-simulation	Simple calculation	No	Yes	Closed	Deterministic	Aggregate	No	No	Both	Scenario
Carlin et al. (2013) [37]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	No
Cho et al. (2010) [38]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	Yes	No	Single	Sensitivity
Clark and Cameron (2006) [41]	No	Simulation	Markov model	No	No	Closed	Probabilistic	Aggregate	Yes	Yes	Both	Both
Clark et al. (2014) [39]	No	Simulation	Not specified	No	Yes	Closed	Probabilistic	Aggregate	No	No	Both	Both
Clark et al. (2019) [40]	No	Simulation	Markov model	No	Yes	Closed	Probabilistic	Aggregate	No	No	Both	Both
Coudeville et al. (2016) [43]	No	Simulation	Dynamic model	No	Yes	Closed	Deterministic	Aggregate	No	No	Trade-off	Both
Coudeville et al. (2018) [42]	No	Simulation	Dynamic model	Both	Yes	Closed	Deterministic	Aggregate	No	No	Trade-off	Both
Desai et al. (2013) [44]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	No	No	Both	Sensitivity
Desai et al. (2012) [45]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	No	No	Both	Sensitivity
Ferguson et al. (2016) [46]	No	Simulation	Dynamic model	Yes	Yes	Closed	Deterministic	Aggregate	No	No	Trade-off	Both
Flasche et al. (2016) [47]	No	Simulation	Dynamic model	No	No	Closed	Both	Aggregate	Yes	No	Trade-off	Both
Hladish et al. (2016) [48]	No	Simulation	Dynamic model	No	Yes	Closed	Probabilistic	Individual	No	No	Trade-off	Both
Kelly et al. (2010) [49]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Kent (1992) [50]	No	Simulation	Decision tree	No	No	Closed	Deterministic	Aggregate	No	No	Single	Both
Koplan et al. (1979) [51]	No	Simulation	Decision tree	No	Yes	Closed	Deterministic	Aggregate	No	No	Single	Both
Lamrani et al. (2017) [52]	No	Simulation	Not specified	No	Yes	Closed	Probabilistic	Aggregate	No	No	Both	Both
Ledent et al. (2016) [54]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	No	No	Both	Both
Ledent et al. (2018) [53]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	No	No	Both	Both
Ledogar et al. (2009) [55]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Both	Scenario
Lee et al. (2005) [56]	No	Simulation	Markov model	Yes	Yes	Closed	Deterministic	Aggregate	Yes	Yes	Both	Both
Lourenco and Recker *2016) [57]	No	Simulation	Dynamic model	No	No	Open	Probabilistic	Individual	No	No	Trade-off	Both
Maier et al. (2017) [58]	No	Simulation	Dynamic model	No	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Scenario
Manissero et al. (2008) [59]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Marcelon et al. (2016) [60]	Yes	Non-simulation	Simple calculation	Yes	No	Closed	Deterministic	Aggregate	No	Yes	Both	Both
Maro et al. (2014) [61]	Yes	Simulation	Dynamic model	Yes	No	Closed	Deterministic	Aggregate	No	Yes	Single	Scenario
Meltzer 2003) [62]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Scenario
Monath 2012) [63]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	No
Nokes and Anderson 1991) [64]	No	Simulation	Dynamic model	Yes	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Oleksiewicz et al. (2015) [65]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	Yes	Single	Scenario
Onorato et al. (1989) [66]	No	Simulation	Decision tree	No	No	Closed	Deterministic	Aggregate	No	Yes	Single	Scenario
Oster et al. (2010) [67]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Patel et al. (2009) [68]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Patel et al. (2011) [70]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Both	No
Patel et al. (2012) [69]	No	Simulation	Not specified	No	No	Closed	Probabilistic	Aggregate	No	No	Both	Both
Perkins et al. (2016) [71]	No	Simulation	Dynamic model	Yes	No	Closed	Probabilistic	Individual	No	No	Trade-off	Both
Phillips et al. (2013) [72]	No	Simulation	Decision tree	No	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Both
Rodriguez-Barraquer et al. (2014) [73]	No	Simulation	Dynamic model	No	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Both
Rothberg et al. (2002) [74]	No	Simulation	Markov model	Yes	Yes	Closed	Deterministic	Aggregate	No	Yes	Both	Both
Rouillon and Waaler 1976) [75]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Both	Scenario
Shim (2019) [76]	No	Simulation	Dynamic model	Yes	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Scenario
Tosti et al. (1999) [77]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Yung et al. (2015) [78]	No	Non-simulation	Simple calculation	No	No	Closed	Deterministic	Aggregate	No	No	Single	Scenario
Zalkind and Shachtman (1980) [79]	No	Simulation	Decision tree	No	No	Closed	Deterministic	Aggregate	No	No	Trade-off	Scenario

Table 3

Distribution of the methodological approaches stratified by use of a simulation or non-simulation model

Methodological approaches	Simulation model (N = 32)n (%)	Non-simulation model (N = 16)n (%)	Total (N = 48)n (%)
Standardised framework
Yes	3 (9%)	2 (13%)	5 (10%)
Model attributes
Static model as opposed to transmission dynamic model	20 (63%)	16 (100%)	36 (75%)
No herd immunity effect	21 (66%)	14 (88%)	35 (73%)
No waning effect	20 (63%)	15 (94%)	35 (73%)
Closed cohort as opposed to open cohort	31 (97%)	16 (100%)	47 (98%)
Deterministic model as opposed to stochastic model	15 (47%)	16 (100%)	31 (65%)
Aggregate model as opposed to individual-based model	28 (88%)	16 (100%)	44 (92%)
Table summarising input parameters
Yes	22 (69%)	4 (25%)	26 (54%)
Discount rates
Yes	5 (16%)	0 (0%)	5 (10%)
Utility and preference information
Yes	6 (19%)	2 (13%)	8 (17%)
Benefit–risk measuresa
Singles indices	20 (63%)	15 (94%)	35 (73%)
Impact number	20 (63%)	14 (88%)	34 (71%)
Number needed to vaccinate/harm	2 (6%)	6 (38%)	8 (17%)
Others	2 (6%)	0 (0%)	2 (4%)
Trade-off indices	26 (81%)	6 (33%)	32 (67%)
Benefit–risk ratio	10 (31%)	3 (19%)	13 (27%)
Benefit–risk difference	16 (50%)	2 (13%)	18 (38%)
Others	2 (6%)	1 (6%)	3 (6%)
Visual representation used	31 (97%)	16 (100%)	47 (98%)
Sensitivity/scenario analysesb
Deterministic sensitivity analyses	12 (38%)	1 (6%)	13 (27%)
One-way deterministic sensitivity analyses	12 (38%)	1 (6%)	13 (27%)
Multiple-way deterministic sensitivity analyses	4 (13%)	0 (0%)	4 (8%)
Probabilistic sensitivity analyses	19 (59%)	1 (6%)	20 (42%)
Scenario analyses	25 (78%)	12 (75%)	37 (77%)

aSome studies can use several measures (among single and trade-off indices)

bSome studies can perform several sensitivity/scenario analyses

Distribution of the number of publications according to the source of funding and the modelling techniques used over time is described in Fig. 2 and Appendix Fig. 1 (see ESM).

Fig. 2

Distribution of the number of publications according to the source of funding over time

Results

Selection of Studies

The literature search retrieved 4917 potentially relevant publications, of which 1745 were duplicates and hence excluded. Of the remaining 3172 publications screened, 3081 were excluded due to lack of relevance, as determined by evaluation of their title and abstract. The full texts of the remaining 91 publications were extensively evaluated and discussed, leading to the exclusion of 52 articles. Reasons for exclusion were repetitive data from original studies (2), non-relevant publication type (14), non-qBRm studies (35) and no access to the full-text article (1). Nine additional publications were included through snowballing the reference lists. A total of 48 publications [32-79] were retained for data extraction (Fig. 1). Tables listing the publications and reasons for exclusion and inclusion after assessment of the eligibility of the full texts are provided in the ESM (See Appendix Tables 5 and 6).

Fig. 1

PRISMA flow diagram

General Information

A temporal trend analysis of the selected articles according to their publication date showed that the number of publications focusing on vaccine qBRm was limited before the year 2000 and increased from 2008 onwards, with an average of three publications per year (Fig. 2). Ten studies disclosed funding by pharmaceutical companies.

Modelling Context of Selected Studies

Most of the vaccines included in the selected publications were targeting viruses (40). Two-third of the studies were performed for three viruses (rotavirus (15), dengue (10) and influenza (6)). By contrast, other indications were only supported by a single publication (e.g. Human Papillomavirus or Hepatitis B virus) (Table 1).

Most countries considered in the analyses were high-income countries (30) and the majority were located in the United States and in Europe (Table 1).

In most studies (45), the targeted population analysed in the selected studies was clearly specified and most referred to paediatric populations (38). Others focused on the general population without any further specification (3) or were conducted in several populations (e.g. paediatric and adult populations or general and high-risk populations) (4).

Most studies compared standard vaccination versus absence of vaccination (30), and only a few compared different vaccines of the same class (3). In some cases (15), the vaccine was compared with several alternatives (e.g. standard vaccination versus absence of vaccination and standard vaccination versus vaccination with extended indication).

A large proportion of studies described the perspectives at a population level only (40), compared with individual (3) or both levels (5).

Methodological Approaches Used in Selected Studies

Few studies (5) used a structured approach or tool that facilitates the development of the qBRm (Tables 2 and 3). Among them, the multi-criteria decision analysis (MCDA) was the main standardised framework used (3) [34, 60, 61].

Studies included in the review used non-simulation (16) or simulation models (32) to estimate final outcomes. Among the simulation models, the most common types were the dynamic model (12), followed by the decision tree (5), Markov model (4) and microsimulation (1). Of note, one-third (10) of the simulation models did not specify the modelling technique used. We did not observe a temporal trend of modelling techniques (non-simulation versus simulation) (See Appendix Fig. 1 in the ESM).

Most of the qBRm included were static (36), deterministic (31), did not consider herd immunity and waning effect (35), assumed a closed model (i.e. the model does not allow new individual entrances over time) (47) and stayed at an aggregate level (i.e. no tracking of individuals’ behaviour) (44) (Tables 2 and 3).

Almost two-thirds (29) of the studies used a tabular representation to list and summarise all input parameters used in the qBRm.

Few studies (5) used discount rates to reduce the value of benefit and risk outcomes over time.

Health state preference values (also called utilities) are used to represent the strength of individuals’ preferences for different health states. In some studies (8), preference elicitation techniques were performed to weight benefit and risk outcomes. In these studies, participants were interviewed to obtain preference scores by different methods: standard gamble (1), time trade-off (1) and the Index of Well-Being (1); the other studies (5) referred to an expert panel for eliciting or weighing data.

Regarding the benefit–risk measures, one-third of the studies (16) used measures to estimate benefits and risks separately (single indices), followed by composite measures to integrate benefits and risks (trade-off indices) (13) and the remaining ones (19) used both (Tables 2 and 3). The most commonly used single indices were the impact numbers (34) and the number needed to vaccinate/to harm (NNV/NNH) (8). Impact numbers measure the number of events prevented by and/or cases attributable to vaccination within a given population. NNV and NNH provide the number of patients who need to be vaccinated for preventing or inducing one case of the event of interest. The most commonly used trade-off indices were the benefit–risk difference (or net health benefit [NHB]) (18) and the benefit–risk ratio (BRR) (13). NHB estimates the difference between the total various benefits and the total various risks of a specified vaccination. BRR divides a single benefit by a single risk. BRR and NHB expressed all outcomes in the same metric.

Almost all included studies (47) used a visual representation of their quantitative benefit–risk results, notably tables (34), line graphics (23), bar charts (8), scatter plots (6), box plots (5) and area graphs (2).

Most studies (44) conducted sensitivity and/or scenario analyses including one-way deterministic sensitivity analyses (13), multiple-way deterministic sensitivity analyses (4), probabilistic sensitivity analyses (20) and scenario analyses (37) (Tables 2 and 3). Most of the studies using non-simulation models only reported scenario analyses (12). By contrast, most studies with simulation models reported sensitivity (deterministic and/or probabilistic) analyses (25).

Discussion

To our knowledge, the present study is the first that systematically reviewed published information about qBRm applied to vaccines. This study was not intended to appraise the qBRm quality but aimed to build a comprehensive repository of the available publications about qBRm applied to vaccines. The present work is the first of two companion papers. Based on this mapping exercise (i.e. Part I), we developed a second paper that proposes standards of reporting for qBRm applied to vaccines (Part II) [26].

Since the late 1990s, the development of qBRm has been progressively encouraged by regulatory authorities [14–16, 80, 81]. Although the first qBRm applied to vaccines identified in our systematic literature review was published in 1976 [75], most studies were published over the last decade. The number of those developed by pharmaceutical companies has increased over time. This trend may be explained by the recent launch of international consortiums [18, 24, 25] facilitating collaboration between public and private partners to develop tools for assessing benefit–risk, particularly for vaccines.

Modelling Context

Among the 48 original publications included, the majority targeted the rotavirus, dengue and influenza vaccines. The focus on rotavirus vaccines might be explained by their association with a transient increased risk of intussusception following administration [82]. The high number of publications about the recent dengue vaccine is likely to be caused by the increased risk of ‘secondary-like’ infection in populations [83]; in this context, qBRm were performed to simulate optimal vaccine strategies according to different transmission intensity settings and serological status impacts. For influenza vaccines, the variable nature of the influenza viruses requiring the undertaking of a new vaccine version each fall is a likely explanation [84]. By contrast, qBRm were not performed for some vaccines, notably older ones, such as those for diphtheria or poliomyelitis, probably because the recent adoption of qBRm and the benefit–risk balance of older vaccines is no longer questioned.

Most qBRm targeted high-income countries. Their results may thereby not be representative for countries with lower income. Estimation of benefits and risks of vaccination may indeed vary according to the epidemiology of the diseases and the healthcare system. Those geographical and socio-economical specificities should be considered when interpreting qBRm results. Furthermore, the development of qBRm applied to vaccines in different settings should be encouraged for improving and adapting the decision-making process.

Methodological Approaches

The present work pointed out the high level of variability in the quality of reporting across studies. First, it was not straightforward to identify input parameters used for each qBRm. One-third of the studies did not use a tabular representation to list and define all input parameters. Second, information on modelling techniques and benefit–risk measures were not sufficiently outlined, which added complexity in extracting the data summarised in Tables 2 and 3. Incomplete and inadequate qBRm reporting may affect their result interpretation, their reproducibility, and potentially, any related decision-making process (e.g. the approval and/or recommendation of a new vaccine or its indications). Developing reporting guidance is therefore the key to helping ensure transparency and reproducibility of qBRm applied to vaccines, but is currently lacking.

One of the main findings of this review was the heterogeneity observed across studies in terms of their methodology. First, two-thirds of the studies used simulation models to estimate final outcomes. Among the simulation models, the number of static and dynamic models was similar. Dynamic models are particularly useful when targeting infectious diseases because they allow taking into account the evolution of certain factors that modulate the risk of infection over time, such as immunity [85]. Of note, except for two publications, all dynamic transmission models were performed for the dengue vaccine. The low number of transmission dynamic models used for other vaccines may be related to their high level of complexity, requiring more advanced programming skills [31]. Second, a few studies incorporated health state preference values, offering the advantage that it represents the strength of individuals’ preferences for different health states, thereby moving beyond the narrow biomedical model for evaluative research [86]. Various techniques were used to obtain health state preference value. Third, benefit–risk measures considered also differed from one study to another. Single, trade-off and both indices were each used in one-third of the selected studies. Single indices are valued for their intuitive interpretation by clinicians and decision makers. Conversely, by using a commensurable score, trade-off indices confer the advantage of providing a straightforward comparison of benefit and risk outcomes, thereby facilitating the BRA. Nevertheless, trade-off indices may be less intuitive and can only be performed when both benefit and risk outcomes use the same metric. Fourth, almost all studies performed uncertainty analyses by varying one or more key parameter values, using various techniques such as scenario (18), sensitivity (7) and both (18). The questions on benefit–risk decision are heterogeneous, due to differences like targeted population, benefit and risk outcomes and importance of indirect effects. Thus, there is no one-size-fits-all approach, as proved by the high level of methodological heterogeneity among the selected publications. This observation highlights the richness of methodological approaches available to perform qBRm applied to vaccines; each of them with their own strengths and weaknesses. The choice of methodological approaches depends on several considerations like the analyst’s technical skills, the required model complexity, the question at hand and the nature of the decision matter, the natural history and features of the particular infectious disease and the data available to parameterise and calibrate the model [87, 88]. Consequently, it is crucial to develop methodological guidance to help researchers use the best methodological approach suited to specific situations. Furthermore, the variety of methodological approaches available reinforces the need for specific standards in reporting.

The present review has some limitations. Even though we aimed to be comprehensive, we cannot guarantee that all relevant studies were identified, mainly because there are no specific keywords for qBRm. Furthermore, grey literature is not identifiable through conventional methods of bibliographic search, and consequently, some relevant websites or other source(s) of information might be missing. However, various search strategies were tested before selecting the one that allowed us to retrieve the largest number of relevant publications, while keeping the number of articles to be reviewed within acceptable limits.

Conclusion

This study is the first that systematically reviewed published information about qBRm applied to vaccines. Our findings showed (i) large differences in terms of methodology used to perform qBRm applied to vaccines and (ii) a noticeable heterogeneity across studies in terms of quality of reporting. Since the number of published qBRm studies is increasing over time and given that paucity of formal guidance for qBRm applied to vaccines may affect the confidence in these models’ results, the development of specific standards of reporting should help to ensure the transparency and reproducibility. Part II of this series of companion papers proposes an operational checklist aimed to improve the reporting of qBRm applied to vaccines in scientific articles.

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 217 kb)

Although quantitative benefit–risk models (qBRm) represent useful tools to support decision makers in assessing the benefits and risks throughout the lifecycle of a medical product, few initiatives have been launched to harmonise qBRm approaches applied to vaccines.

The aim of the present review is to provide a complete list of available studies about qBRm applied to vaccines and to describe their characteristics according to the modelling context and the methodological approaches used.

Discrepancies across studies in terms of quality of reporting and methodological approaches used were observed in our review. Thus, we advocate for the development of an operational checklist for improving the reporting in scientific articles.