Benefit-Risk Assessment of Vaccines. Part II: Proposal Towards Consolidated Standards of Reporting Quantitative Benefit-Risk Models Applied to Vaccines (BRIVAC).

INTRODUCTION: Quantitative benefit-risk models (qBRm) applied to vaccines are increasingly used by public health authorities and pharmaceutical companies as an important tool to help decision makers with supporting benefit-risk assessment (BRA). However, many publications on vaccine qBRm provide insufficient details on the methodological approaches used. Incomplete and/or inadequate qBRm reporting may affect result interpretation and confidence in BRA, highlighting a need for the development of standard reporting guidance.
OBJECTIVES: Our objective was to provide an operational checklist for improved reporting of vaccine qBRm.
METHODS: The consolidated standards of reporting quantitative Benefit-RIsk models applied to VACcines (BRIVAC) were designed as a checklist of key information to report in qBRm scientific publications regarding the assessed vaccines, the methodological considerations and the results and their interpretation.
RESULTS: In total, 22 items and accompanying definitions, recommendations, explanations and examples were provided and divided into six main sections corresponding to the classic subdivisions of a scientific publication: title and abstract (items 1-2), introduction (items 3-4), methods (items 5-15), results (items 16-17), discussion (items 18-20) and other (items 21-22).
CONCLUSIONS: The BRIVAC checklist is the first initiative providing an operational checklist for improved reporting of qBRm applied to vaccines in scientific articles. It is intended to assist authors, peer-reviewers, editors and readers in their critical appraisal. Future initiatives are needed to provide methodological guidance to perform qBRm while taking into account the vaccine specificities.

Key Points

Introduction

Vaccination is among the most effective public health achievements for disease prevention and has led to a dramatic decline of several devastating diseases such as poliomyelitis, diphtheria and smallpox [1]. Nevertheless, though generally well-tolerated, no vaccine can be considered absolutely safe [2]. Vaccines differ from most of the other medicinal products in the sense that they can be administered to large populations, encompassing mostly healthy people, including children, and they can be introduced by health authorities as mandatory [3, 4]. In this context, understanding the balance between the benefits and risks of vaccination is essential to ensure informed decision making [5]. Regulatory and public health authorities and pharmaceutical companies are increasingly using quantitative benefit-risk models (qBRm) to support their benefit-risk assessment (BRA) [6-8]. qBRm integrate evidence from multiple sources to quantify and put into perspective the benefits and risks of a health intervention using mathematical formulae, statistical techniques or simulations [9]. A systematic review aiming at identifying and describing available qBRm applied to vaccines has been developed as the first of two companion papers [10]. This initial work highlighted the lack of formal structure used to conduct and report the results of qBRm, leading to a potential lack of transparency and reproducibility. In contrast, a growing number of regulators have provided guidance and references on structured frameworks for assessing the benefit–risk profile of drugs and devices [11-14]. Nevertheless, consensus about the reporting of qBRm, regardless of area, is yet to be achieved. This study proposes an operational checklist aiming at standardising the reporting of qBRm applied to vaccines and is the second of two companion papers.

Methods

The consolidated standards of reporting quantitative Benefit-RIsk models applied to VACcines (BRIVAC) were designed as a checklist and based on multiple sources of information identified through a targeted literature search. These sources included (1) published documents providing recommendations on the conduct of qBRm applied to drugs [9, 12, 15–20] and to vaccines [21], (2) findings from a systematic literature review of qBRm applied to vaccines described in the first of two companion papers [10] and (3) guidelines on standards of reporting health research with a focus on observational studies (STROBE [STrengthening the Reporting of OBservational studies in Epidemiology] and RECORD [REporting of studies Conducted using Observational Routinely-collected health Data]) [22-26] and economic evaluations (CHEERS [Consolidated Health Economic Evaluation Reporting Standards]) [27].

To design the BRIVAC checklist, a preliminary list of items was created by one author (HA) by using these sources on study reporting, qBRm guidelines and vaccine specificities. The research team reviewed the items and established inclusion/exclusion status for each.

The BRIVAC statement provides guidance on key information to report in scientific articles on vaccine qBRm: the assessed vaccines, the methodological choices made, the results and the interpretation. Each item is supported by (1) its definition, (2) a recommendation of what authors should report, (3) an explanation on the importance of its reporting and (4) an example to illustrate how they should be reported.

Results

The BRIVAC checklist consists of 22 items divided into six main sections corresponding to the classical subdivisions of a scientific publication: title and abstract (items 1–2), introduction (items 3–4), methods (items 5–15), results (items 16–17), discussion (items 18–20) and other (items 21–22). Both sections and items are listed in Table 1 and further described in the rest of this article.

Table 1

BRIVAC checklist: items to be included when reporting quantitative benefit–risk models applied to vaccines

Section/item	Item no.	Recommendation	Item has been reported
Title and abstract
Title	1	Identify the work as a ‘quantitative benefit–risk model’, identify the vaccines of interest and, if applicable, the targeted geographical areas	□
Abstract	2	Provide a structured summary of the background and objectives, methods (including the targeted population(s), geographical location and time frame of the qBRm, perspectives, alternatives, choice of benefit and risk outcomes, time horizon and choice of model), key results and conclusions (interpretation and generalisability)	□
Introduction
Background	3	Provide an explicit statement of the relevant literature stressing the magnitude of the infectious disease burden and the benefit and risk outcomes of all health interventions available, the rationale for conducting the analyses and the relevance of the study question. If the study was performed at the request of a specific stakeholder, this should be clearly stated	□
Objectives	4	Detail the specific study objectives in conjunction with items 5–9 and 11	□
Methods
Targeted populations	5	Define the targeted populations by describing their characteristics and the rationale for selection. If several populations or sub-populations are targeted in the qBRm study, all of them should be reported	□
Geographical location and time frame of the qBRm	6	Describe the geographical areas and the study period	□
Perspectives	7	Describe the perspectives involved in the analyses (analysis performed at individual or population level, or both)	□
Alternatives	8	Identify the alternatives compared with the vaccines of interest in the analyses and describe their relevance	□
Choice of benefit and risk outcomes	9	Describe what outcomes were considered to define the benefits and risks of the vaccines of interest and the reasons to select these criteria. Specify measures (or unit) chosen to express benefit and risk outcomes. Providing a visual representation that displays the benefit and risk outcomes is strongly recommended	□
Measurement and valuation of preference	10	If applicable, describe the preference-elicitation techniques used to weight benefit and risk outcomes as well as the size and characteristics of the population from which the preference values were obtained	□
Time horizons	11	Describe the relevant time horizons for the benefit and risk outcomes evaluated and state why they are appropriate	□
Discount rates	12	If applicable, report the use of discount rates for benefit and risk outcomes and mention why they are relevant	□
Choice of model	13a	Model type: Describe the type of model used (simulation or non-simulation) and provide the rationale for its structure	□
	13b	Modelling attributes: Identify the key characteristics of the selected model, such as:  Dynamic vs. static model  Open vs. closed model  Probabilistic vs. deterministic model  Model integrating aggregated vs. individual-based data  Waning effect vs. no waning effect  Herd immunity vs. no herd immunity	□
Analytical methods	14	Describe all analytical methods employed in the analyses, any data transformation conducted prior to the analyses and the analytical software used	□
Model input parameters	15	List all model input parameters feeding the model and provide their values, ranges, sources and criteria for selection. When probabilistic simulation models are used (see items 13a and 13b), the probability distribution used for each input parameter should also be described. Providing a tabular representation summarising this information is strongly recommended	□
Results
Benefit and risk outcomes	16	Report measures, values and ranges resulting from the analyses to quantify the benefits and risks. Providing a visual representation summarising this information is strongly recommended	□
Sensitivity/scenario analyses	17	If applicable, describe sensitivity and/or scenario analyses performed to characterise uncertainty	□
Discussion
Key results	18	Describe the key results in light of the study objectives	□
Limitations	19	Identify all the possible and relevant limitations of the model. Discuss the impact of these limitations on the findings	□
Interpretation and generalisability	20	Provide an overall interpretation of the results considering similar analyses and other relevant evidence. Discuss the generalisability of the results and potentially suggest recommendations regarding the use of the vaccines assessed	□
Other
Source of funding	21	Describe any sources of funding and the role of any funders in the study	□
Conflicts of interest	22	Describe all relevant financial and non-financial relationships and activities and conflicts of interest that could be perceived as potentially influencing the submitted work	□

BRIVAC the consolidated criteria for reporting quantitative Benefit-RIsk models applied to VACcines, qBRm quantitative benefit-risk models

Title and Abstract

Item 1: Title

Definition: The title captures the content and/or purpose of the research paper.

Recommendation: The authors should (1) clearly mention in the title that the publication focuses on the development of a ‘quantitative benefit-risk model’ and (2) identify the targeted vaccines. If applicable, the targeted geographical areas should be added.

Explanation: To date, studies on qBRm applied to vaccines have used a variety of non-harmonised terms to describe their quantitative benefit–risk approach (e.g. benefit risk, benefit harm, risk benefit). Knowing that electronic databases use words in the title and abstract to yield search results, vague or ambiguous titles and abstracts might be inappropriately indexed [28]. A clear and precise title mentioning the type of analysis conducted and its scope will increase the likelihood of the publication being appropriately indexed.

Currently, no recognised terminology exists for publications focusing on qBRm. In this context, we propose systematically using in the publication’s title the ‘quantitative benefit-risk model’ terminology derived from the IMI-PROTECT glossary [29]. First, the terminology ‘quantitative’ ensures a clear differentiation from the ‘qualitative’ approach, based on descriptive templates or guidelines and relying on expert judgement only. Furthermore, authors need to identify in the title that the analysis targets to quantify ‘benefit–risk’ balance. Finally, ‘model’ should be preferred as a generic term that encompasses any theoretical construct or analysis describing behaviours of a system. It is a term widely used across different disciplines [30].

Example: Quantitative benefit-risk model of the quadrivalent human papillomavirus vaccine for preventing anal cancer in males in Europe. (Hypothetical example).

Item 2: Abstract

Definition: The abstract is a short summary of the major aspects of the research paper.

Recommendation: The qBRm abstract should provide accurate and sufficiently detailed information on (1) the background and objectives, (2) the methods (including targeted population[s], geographical location and time frame of the qBRm, perspectives from which the vaccine’s benefit and risk outcomes are evaluated, alternatives compared with the vaccines of interest, choice of benefit and risk outcomes, time horizon and choice of model, (3) the key results and (4) the conclusions (interpretation and generalisability).

Explanation: the abstract is used to help the readers quickly ascertain the paper’s purpose. A complete, structured and transparent abstract is important because most readers assess the relevance of a report or publication only on the basis of information provided in the abstract [31].

Example:

Background and Objectives: Meningococcal disease is an acute, serious illness caused by the bacterium Neisseria meningitidis. During 1995–2004, an estimated 1400–2800 cases occurred in the USA annually. For the prevention of meningococcal disease, the Advisory Committee on Immunization Practices (ACIP) recommends the conjugate vaccine ([MCV4] Sanofi Pasteur). Since the introduction of MCV4 in 2005, a number of non-fatal Guillain-Barré syndrome (GBS) cases have been reported. The study evaluates the benefits of MCV4 vaccination against the risk of vaccine-associated GBS.

Method: A simulation model was built simulating health events within a US cohort of 11-year-olds followed for 8 years to assist decision makers in setting policy. Using a quantitative benefit–risk model comparing vaccination and no vaccination, we assess the trade-offs between the MCV4-induced risks (GBS) and benefits (prevented meningococcal disease). Incident meningococcal disease and GBS cases were modelled and quality-adjusted life-years (QALYs) calculated as health outcome measures. Health utility indices were used in QALY calculations.

Key Results: Applying a 3% discount rate, MCV4 vaccination would save 2397 QALYs, whereas vaccine-attributable GBS could result in 5 QALYs lost.

Conclusions: Based on the result, MCV4 vaccination in the USA is strongly favoured despite possible vaccine-associated GBS risk. (Hypothetical example).

Introduction

Item 3: Background

Definition: The background section is a summary of topics most relevant to the BRA, allowing readers to understand the rationale and context of the study [32].

Recommendation: In the background section of publications on qBRm applied to vaccines, (1) relevant literature should be summarised, preferably stressing the magnitude of the infectious disease burden and all health interventions available with an outline regarding their benefit and risk outcomes; (2) the gap in the current knowledge and the relevance of the study question should also be stated; (3) if the study was performed at the request of a specific stakeholder, this should be clearly stated. Potential stakeholders involved in the decision-making process encompass vaccine manufacturers, regulatory and public health authorities, healthcare providers and recipients of the candidate vaccine [33].

Explanation: The background section links the knowledge on the research topic and the reported qBRm. A well-written background section explains why the qBRm was performed and its added value to inform vaccination decision making [32].

Example:

The incidence of reported pertussis in the USA has been increasing steadily in the past 2 decades. This trend is occurring despite the fact that childhood vaccination rates are at an all-time high and vaccine efficacy remains good. (…) However, several studies have suggested that immunity after vaccination wanes over time and protection may last only 10–15 years, leading to a susceptible population around the time of mid-adolescence. (…) The morbidity associated with pertussis among adolescents and adults can be severe and its economic impact quite substantial, with significant time lost from school and work for these individuals.

Routine use of an effective vaccine among adolescents and adults might not only reduce morbidity rates in these age groups but also prevent infant pertussis infection through herd protection. However, the potential benefits of vaccination need to be weighed against the possible problems. Vaccine adverse events, waning immunity after adolescent or adult vaccination and costs may all decrease the desirability of routine pertussis vaccination in these age groups. We conducted this study to assist policy makers in decisions about whether and how pertussis vaccination of adolescents and/or adults should be adopted in the USA. (Example based on Lee et al. [34]).

Item 4: Objectives

Definition: The research objectives are a concise description of what the study is trying to achieve.

Recommendation: The final paragraph of the introduction should clearly list the study objectives in conjunction with BRIVAC checklist items 5–9 (i.e. the targeted population[s], the geographical location and the time frame of qBRm, the perspectives, the alternatives and the choice of benefit and risk outcomes) and 11 (i.e. time horizons), as described in the following.

Explanation: A precise formulation of the objectives allows justification of the appropriateness of the qBRm.

Example:

This study assessed risks (vaccine-associated GBS cases) and benefits (incident meningococcal disease cases) of MCV4 vaccination versus non-vaccination, on a US hypothetical 11-year-old cohort enrolled in 2006 and followed over an 8-year period, to provide context to the Advisory Committee on Immunization Practices in making recommendations to continue or discontinue the meningococcal vaccination programme. (Hypothetical example).

Methods

Item 5: Targeted population(s)

Definition: The targeted population is the eligible population for which the qBRm was carried out.

Recommendation: The specific characteristics and the rationale for selection of the targeted population(s) should be precisely defined. If several populations or sub-populations are targeted in the qBRm study (e.g. neonates, infants, pregnant women, high-risk groups, elderly), they should all be reported.

Explanation: The choice of the targeted population(s) depends on the objectives of the study, the relevant literature stressing the vaccine benefit and risk outcomes, the infectious disease epidemiology and the medical practices that could be driven by vaccine recommendations. One of the advantages of qBRm is that it enables the quantification of the benefit–risk balance for different targeted population(s) (e.g. universal vaccination or targeted risk groups only).

Since the quantitative benefit–risk balance of a vaccine might vary markedly depending on the population, it is crucial to define and justify the choice of the targeted population(s).

Example:

We analysed the hypothetical experience of a cohort of 1 million children from birth to 6 years of age because virtually all pertussis mortality and severe morbidity occur in this age group and because most immunisation programmes do not recommend pertussis vaccination after the age of 6 years. (Example based on Koplan et al. [35]).

Item 6: Geographical location and time frame of the qBRm

Definition: The geographical location defines where the study was performed (countries, states or regions), and the time frame defines the period when the study was performed.

Recommendation: Authors should describe the geographical areas and the study period.

Explanation: Infectious disease epidemiology (e.g. pathogen evolution, seasonal characteristics or disease severity), the population distribution and density, preventive and curative measures and healthcare systems may vary over time and across regions, and all these may directly affect the qBRm. It is therefore important to provide a clear description of the geographical location and time frame of the qBRm so that readers can assess the generalisability of the results.

Example:

To re-evaluate the risk–benefit profile of the Italian strategy of hepatitis B vaccination. (…) To estimate the incidence rates of hepatitis B, we used data of new acute infections notified in 1996 to the SEIEVA (the Italian surveillance of acute viral hepatitis). (Example based on Tosti et al. [36]).

Item 7: Perspectives

Definition: The study perspectives are the viewpoints from which the vaccine’s benefit and risk outcomes are evaluated.

Recommendation: The perspectives involved in the analyses should be described (analysis performed at individual or population level, or both).

Explanation: In public health and disease prevention, a distinction should be made between the individual (benefit and risk for the candidate vaccine recipient only) and the population (positive and negative impacts for the whole population) perspectives [37]. For vaccines, both perspectives are relevant when intending to conduct a qBRm, as the potential benefits and risks are not always borne by the same individual. For example, non-vaccinated individuals might benefit from vaccination while not being exposed to the risks induced by vaccination (i.e. indirect protection or herd immunity) [38, 39].

Furthermore, the benefit–risk decision could differ according the perspective used. For instance, when targeting disease eradication (e.g. measles, polio), the aim of vaccination is more to prevent disease re-emergence than to ensure individual protection [33].

Consequently, the results should be interpreted in light of the perspective chosen.

Example:

The model can be used to evaluate a person’s perspective of the risks and benefits of receiving a smallpox postexposure vaccination. I considered a person who has been exposed to somebody who may or may not have smallpox. (Example based on Meltzer [40]).

Item 8: Alternatives

Definition: The alternatives correspond to all options to be evaluated against the main intervention of interest.

Recommendation: The alternatives compared with the vaccines of interest in the analyses should be described, as well as their relevance.

Explanation: qBRm are never made on an absolute scale but are relative to a reference scale. The targeted vaccines need to be compared with alternatives such as the absence of vaccination, other vaccines, other preventive measures (e.g. human papillomavirus [HPV] screening vs. HPV vaccination), alternative indications (e.g. different vaccination schedules for the targeted vaccines) and alternative policies for the implementation of the vaccination programmes (e.g. recommended vs. mandatory implementation) [33]. The qBRm results and their interpretation are directly impacted by the comparator chosen.

Example:

The three strategies involved either (1) vaccinating all infants at 12 months of age, (2) delaying vaccination until 10 years of age and then vaccinating only if a child has no history of varicella or (3) not vaccinating at all. (Example based on Rothberg et al. [41]).

Item 9: Choice of benefit and risk outcomes

Definition: The choice of outcomes corresponds to criteria considered to define the benefits and risks of the vaccine of interest.

Recommendation: The authors should detail (1) the choice of criteria to estimate benefit and risk outcomes, (2) the reasons to select these criteria and (3) the specific measures (or unit) chosen to express benefit and risk outcomes. Furthermore, providing a visual representation that displays the benefit and risk outcomes is strongly recommended.

Explanation: The benefit outcomes represent protective effects of vaccination against an infectious disease and complications. There are different ways to express the vaccine benefits: efficacy (direct effect of vaccination measured in pre-licensure randomised controlled trials, where vaccination is allocated in optimal conditions), effectiveness (direct, indirect, total or global effects of vaccination measured in routine) or impact (the proportionate reduction in disease burden, comparing incidences between a pre-vaccine and a post-vaccine period in the same population) [42, 43]. Risk outcomes represent adverse events following immunisation (AEFI). Reported AEFI can either be true adverse reactions (i.e. resulting from the vaccine or immunisation process) or coincidental events that are not due to the vaccine or immunisation process but are temporally associated with immunisation. Consequently, the causal association between the event and vaccination should be defined.

Furthermore, a huge diversity of measures to express benefits and risks are available. Measures used in qBRm might include, but are not limited to, measures expressed in natural units (e.g. impact numbers, number needed to harm or vaccinate [NNH or NNV], life-years gained) or composite health measures based on preferences for health (e.g. quality-adjusted life-years [QALYs], disability-adjusted life-years [DALYs] or healthy life expectancy [HALE]). Measures used in qBRm are summarised elsewhere [9, 11, 16, 19, 20] and can be divided into single and trade-off indices. Single indices use measures to quantify benefit and risk outcomes separately, whereas trade-off indices integrate benefit and risk outcomes into a composite measure index allowing a direct interpretation of the benefit–risk balance [10].

Considering the existing classification of benefit and risk outcomes and the diversity of measures to express it, it is crucial to precisely describe all the outcomes considered in the qBRm and the measures used to leverage the relevance of qBRm findings. The visual representation is a helpful tool to clearly present all benefit and risk outcomes considered.

Example:

We conducted a risk–benefit analysis using published rotavirus and intussusception epidemiologic data to model the impact of a vaccination programme with either 20 or 90% coverage scenarios compared with no vaccination programme. Based on the methodology of Patel et al. 2011, we used a birth cohort in 2005 to estimate the number of hospitalisations attributable to rotavirus that could potentially be prevented and the number of excess intussusception hospitalisations that could be caused by the vaccination. We also calculated the number of infants who would need to be vaccinated to prevent one rotavirus hospitalisation or cause one excess intussusception hospitalisation. Because rotavirus and intussusception mortality is negligible in Singapore, we analysed hospitalisation as our outcome of interest. (Example based on Yung et al. [44]).

Item 10: Measurement and valuation of preference

Definition: The health state preference values (also called utilities) are used to represent the strength of individuals’ preferences for different health states. Several preference-elicitation techniques (e.g. time trade-off approach, standard gamble approach or discrete choice experiment) can be used to obtain health state preference values [45].

Recommendation: When applicable, authors should describe (1) techniques used to weigh benefit and risk outcomes and (2) the size and characteristics of the population from which the preference values were obtained.

Explanation: Composite health measures (see item 9) combine different health outcomes (morbidity and mortality) into a single commensurable score by using techniques to determine preference values.

It is critical to describe the population from which preference values were obtained because they may differ depending on respondents (e.g. general population, candidate vaccine recipients, parents, healthcare providers or experts) [46-51]. For some vaccines (e.g. travellers’ vaccines), both the potential benefits and risks are borne by the same individual. Hence, preference values from candidate vaccine recipients are informative. For vaccination programmes aiming at reducing disease transmission within the general population, one might argue that the general population and/or public health experts play an important role in generating these preference values. For vaccines administered to young children, the vaccination decision usually falls to the child’s parents and hence, parent’s preferences are informative.

Example:

To estimate a utility value for the state of permanent neurological sequelae following tuberculous meningitis, subjects were recruited to participate in an interview involving the standard gamble technique. During the interview, subjects were offered two alternatives: the first was the possibility of perfect health for the rest of the life with a probability of immediate death, and the second was living the rest of the life with permanent sequelae from tuberculous meningitis. (…) Volunteers for interviews were recruited from three groups: first-year medical students at the University of Ottawa (year of entrance 2002); employees at Health Canada; and staff at the Department of Social Development and Health, Mohawk Council of Akwesasne. (…) A total of 107 subjects were interviewed in the survey. (Example based on Clark and Cameron [52]).

Item 11: Time horizons

Definition: The time horizon refers to the duration over which benefit and risk outcomes are evaluated.

Recommendation: For all benefit and risk outcomes, it is important to mention and provide the rationale for the appropriateness of the selected time horizons.

Explanation: Lifetime horizon can be used as a first instance. Nevertheless, a shorter time horizon may be used when it is demonstrated that all relevant outcomes are captured. For vaccines, the time horizon is based on the nature of the illness (e.g. seasonal for influenza, lifetime for rubella), the vaccine characteristics (e.g. waning effect) and the policies by which the vaccine is administered (e.g. age of vaccination). Therefore, qBRm applied to vaccines will be particularly sensitive to the choice of the time horizons [53].

Example:

Although it has been recommended elsewhere that a lifetime horizon be considered in such studies, our model followed a theoretical cohort from birth to the age of 14 years. This approach was chosen for consistency with earlier modelling studies on BCG and because current evidence suggests BCG protection lasts perhaps 10–15 years. Although the prevention of childhood TB and disseminated BCG infection have long-term benefits in terms of life-years gained, BCG does not impact on any TB-related risk in adulthood and, as such, has no impact on outcomes beyond the age of 15 years, or as described above, the risk of morbidity and mortality in the population. Most importantly, the lifetime horizon approach would not have affected the most important outcome in the model, namely those threshold values for severe combined immunodeficiency (SCID) incidence, which alter the decision to use BCG for a given risk of tuberculous infection in the population. (Example based on Clark and Cameron [52]).

Item 12: Discount rates

Definition: The discount rates are used to reduce the value of benefit and risk outcomes over time.

Recommendation: If discount rates are used, they should be mentioned and their relevance justified.

Explanation: The benefit and risk outcomes may occur at different points in time, present or future. This is particularly true of vaccination benefits (protection against an infectious disease) that may appear in the far future while some adverse events could be identified early after vaccination and vice versa. Sometimes, vaccination benefits and risks may even impact on future generations. However, outcomes that are predicted to occur in the future might be valued less than those predicted to occur in the present by applying discount rates. Discount rates are not universal and will vary in different settings [54]. Discounting schemes applicable to vaccines are presented by Jit and Mibei [55]. Consequently, it is important to report discount rates because benefit and risk outcomes, specifically those in the far future, may be particularly sensitive to the choice of the discount rates [45].

Example:

To compare health outcomes occurring in different time periods, we discounted health outcomes to the present value. A discount rate of 3% was applied to health outcomes from GBS and those from meningococcal disease as suggested in the literature. (Example based on Cho et al. [56]).

Item 13: Choice of model

Item 13a: Model type

Definition: The model is an umbrella term that encompasses any theoretical construct describing behaviours of a system, and it is widely used across different disciplines [30].

Recommendation: The research paper should describe the type of the model used for the analyses (simulation or non-simulation) and provide the rationale for its structure.

Explanation: The choice of the model type depends on several considerations such as the analyst’s technical skills, the required model complexity, the question at hand and the nature of the decision problem, the natural history and features of the particular infectious disease of interest and the data available to parameterise and calibrate the model [57, 58]. qBRm can be categorised into simulation and non-simulation models. In the first, benefit and risk outcomes are derived from simulation techniques of various degrees of complexity including as many components and interaction as possible (e.g. Markov model, microsimulation or dynamic transmission models) [59]. Non-simulation models are based on a simple calculation, mathematical function or statistical model [30]. Transparent and clear communication of the model type is of importance for readers to assess and/or reproduce the model results.

Example:

We adapted an age-structured disease transmission model and added an influenza vaccination adoption function modelled as a Bass diffusion process while assuming a universal vaccination policy. Bass diffusion models are commonly used in the marketing literature to describe the diffusion of innovations. (…) a detailed description of the model is in S1 File. (Example based on Maro et al. [60]).

Item 13b: Modelling attributes

Definition: The modelling attributes are key dimensions in describing the approaches used in qBRm.

Recommendation: The modelling characteristics should be clearly reported according to the following dichotomous attributes based on the classification defined by Kim and Goldie [30]: (1) Dynamic versus static model: force of infection is assumed to change over time (dynamic) or not (static); (2) Open versus closed model: individuals are tracked in the model and allowed to enter and exit the cohort (open) or not (closed); (3) Probabilistic versus deterministic model: the uncertainty around the input parameters is taken into account (probabilistic) or not (deterministic); (4) Model integrating aggregated versus individual-based data: the population’s behaviour in the model is simulated using aggregate variables of which values are population averages (aggregated data) or the behaviours of individuals in the population are tracked (individual-based data); (5) Waning effect consideration versus no waning effect consideration: vaccine efficacy/effectiveness decreases with time or not; (6) Herd immunity consideration versus no herd immunity consideration.

Explanation: It is important to stress that, unlike other medicines, herd immunity (i.e. protection of individuals who are not immunised against an infectious disease while a large proportion of the whole population is immunised against that infectious disease) is a unique attribute of vaccines, and its incorporation or not in the model should be mentioned and discussed. An example of vaccination strategy that exploits the herd immunity is the cocooning (i.e. vaccination of parents and family members against infectious diseases to protect infants who are too young to get vaccinated themselves).

A clear description of modelling attributes facilitates the understanding and assessment of the model type and techniques used.

Example:

We simulated a closed static cohort of 4.3 million adopters of influenza vaccination using aggregated data from the PRISM system database. We did not take into consideration the induced herd protection, and a sigmoid (reverse S) shape curve was fitted to year 1 and year 2 efficacy to account for waning of protection with time post-vaccination. Credible intervals around the benefit–risk ratios and differences were calculated based on probabilistic uncertainty analyses and Monte Carlo simulations. (Hypothetical example).

Item 14: Analytical methods

Definition: The analytical methods are statistical techniques supporting the evaluation.

Recommendation: The general principle is to report (1) all the analytical methods employed, (2) any data transformation conducted prior to the analysis and (3) the analytical software used.

Explanation: The methods used depend on the qBRm design. This can include methods for dealing with censored, skewed or missing data; methods for pooling data; approaches to validate or make adjustments to a model; extrapolation methods; and methods for handling population heterogeneity and uncertainty [27]. To judge the reliability, appropriateness and validity of the methods and related qBRm results, the analytic strategy used should be fully reported. The guiding principle of reporting statistical analyses is to describe analytical methods in a detailed and clear enough manner to enable knowledgeable readers with access to the original data to reproduce the reported findings.

Example:

The analysis included only case patients (and controls, for the case–control analysis) for whom vaccination records were available on the Australian Childhood Immunisation Register (ACIR). The combined data set included all 282 cases from the state-based admissions data plus another 38 captured by Paediatric Active Enhanced Disease Surveillance (PAEDS). In the eight cases of patients with incomplete records, the ACIR documented receipt of a second (or third) dose of rotavirus vaccine but data were missing for the earlier dose(s) (…) Analysis was performed using Stata 11.2 software (StataCorp; 2009). (…) Results (obtained from a conditional Poisson regression model) are reported as relative incidence (RI), with 95% confidence intervals (CIs), for each exposure period compared with time outside this window. (…) Alternative approaches to the method of age adjustment, which varied the age categorisation and smoothed the age effect, were explored in sensitivity analyses. (…) The annual average number of ICD-coded intussusception cases was 240; adjustment by a factor of 0.6 has been made to estimate the proportion of Brighton level 1 cases. (Example based on Carlin et al. [61]).

Item 15: Model input parameters

Definition: The model input parameters are all evidence combined into the model for the benefit–risk estimates.

Recommendation: Authors should list (1) all input parameters used in the analyses, (2) their values, (3) their ranges and (4) their sources, with the appropriate references and (5) the criteria for selection. When probabilistic simulation models are used (see items 13a and 13b), (6) the probability distribution used for each input parameter should also be described. A tabular representation summarising these elements is strongly recommended.

Explanation: To conduct a qBRm, authors should identify and combine available data that allowed feeding the input parameters. Those data are rarely extractable from a single study and are usually collected from diverse sources, including clinical trials, observational studies, administrative databases, case-series, expert opinion, assumptions from authors and/or secondary analyses (e.g. meta-analysis) [62]. Unlike other medicines, vaccine model input parameters will more often rely on post-authorisation data to characterise the risks and potentially the effectiveness. Indeed, the occurrence of some AEFI is rare. Moreover, for most vaccines, long-term protection is expected, and that cannot be fully investigated using pre-authorisation studies following-up small cohorts, during a limited time period [2, 63]. Furthermore, qBRm applied to vaccines can require specific input parameters, including those necessary to consider transmission dynamics, heterogeneity in contact patterns and policy for the vaccination programme.

As with any model, the results obtained with qBRm depend on the quality of the data inputs, the methods used to derive these (see item 14) and the suitability of the model structure (see items 13a and 13b). Model input parameters should therefore be clearly documented and justified to ensure transparency, to allow readers to assess the validity of the model and to facilitate its reproducibility. A tabular representation is a helpful tool to clearly present all model input parameters considered.

Example:

All parameters included in the benefit–risk analysis and their random distributions are listed in Table 1. (Example based on Ledent et al. [64]).

Results

Item 16: Benefit and risk outcomes

Definition: The benefit and risk outcomes are the results obtained from the analyses.

Recommendation: For each alternative, the measures, values and ranges of benefit and risk outcomes should be reported. Providing a visual representation summarising this information is strongly recommended.

Explanation: The outcomes should be explicitly described in order not to bias reader interpretation. The use of appropriate visual representation (e.g. table, scatter plot, line graph, bar chart/graph) can support qBRm assessment and constitute a key support for communicating the qBRm findings [16].

It is even more important considering that results derived from qBRm are used by various stakeholders to assess and compare the benefit–risk balance for different strategies of interest and can support decision making.

Example:

In the absence of a rotavirus vaccine program in France, we estimated a median incidence of 15,059 (95% CI 12,100–18,476) rotavirus gastroenteritis (RVGE) hospitalisations and 10.13 (95% CI 4.64–19.46) RVGE deaths for an average French birth cohort of 791,183 children followed from birth to 5 years of age. Within the same cohort, we estimated an annual number of 323 (95% CI 257–400) intussusception hospitalisations and 0.45 (95% CI 0.19–0.88) intussusception deaths in infants below 1 year of age. (…) The results following vaccination with two doses of Rotarix are presented in Table 2. We estimated that vaccination would prevent approximately 75% of all RVGE hospitalisations and deaths in the total number of French children below 5 years of age, leading to a reduction of 11,132 (95% CI 7842–14,408) hospitalisations and 7.43 (95% CI 3.27–14.68) deaths. We also estimated that vaccination would cause 6.86 (95% CI 2.25–38.37) intussusception hospitalisations and 0.0099 (95% CI 0.0024–0.060) intussusception deaths in one French birth cohort of infants below 1 year of age. (…) The Rotarix benefit–risk ratio for hospitalisation is 1624 (95% CI 240–5243) for children below 5 years of age (Table 2) (…) Similarly, for each intussusception death caused, 743 (95% CI 93–3723) RVGE deaths would be prevented by vaccination. (Example based on Ledent et al. [65]).

Item 17: Sensitivity/scenario analyses

Definition: The sensitivity and scenario analyses are used to explore the effects of uncertainty on the results by varying the value(s) of one (or more) key parameter(s).

Recommendation: If applicable, the authors should describe the sensitivity and/or scenario analyses performed to characterise the degree of uncertainty in the analyses.

Explanation: Various sources of uncertainty must be considered in qBRm, such as statistical uncertainty, management and quality of the data sources, implications of missing data, etc. Furthermore, vaccines require consideration of additional uncertainties such as the disease transmission factor, as well as the uncertainties related to vaccine policy and acceptance by individuals.

However, one of the methodological strengths of modelling is the possibility to challenge the robustness of its result and to quantify the range of uncertainty through sensitivity/scenario analyses [66]. Sensitivity analyses evaluate how the uncertainty of model inputs affect the model outcomes. Sensitivity analyses can be categorised as deterministic (‘one-way’ or ‘multiple-way’) or probabilistic [45]. In scenario (or ‘what-if’) analyses, different epidemiological or healthcare scenarios of interest are investigated (such as different population coverages, different ages at vaccination or different background incidence rates). In addition, authors can also assess the evolution of the benefit–risk results in the future, based on plausible scenarios relying on retrospective data and other assumptions [55].

Example:

Deterministic central estimates (i.e. best estimates for each input parameter) and probabilistic 95% uncertainty intervals (UIs) were calculated for 11 age-restricted schedules and 18 age-unrestricted schedules. All input parameters and their distributions are shown in the appendix. Central estimates were also calculated for six what-if scenarios: relative risks (RRs) of intussusception varying with under-5 mortality (figure, appendix p 12); double the RR of intussusception for the first dose when given after 15 weeks of age; vaccine efficacy and waning equivalent to low mortality settings; less rapid waning efficacy (based on a power function described in detail elsewhere); less rapid waning efficacy for all primary doses administered as part of a neonatal schedule (appendix p 14); and pessimistic access to hospital for intussusception cases (based on the proportion of children who could reach a public hospital within 2 h). (Example based on Clark et al. [67]).

Discussion

Item 18: Key results

Definition: The key results section summarises the main findings derived from the analyses with reference to study objectives.

Recommendation: The discussion should include a succinct and objective summary of the key results and should refer to the objectives listed in the introduction.

Explanation: A clear summary of key results will help readers assess whether the authors’ interpretation (see item 20) is supported by the results of the model or not.

Example:

This study quantified and compared the risk of meningococcal conjugate vaccine-associated GBS with the risks and benefits of no versus full immunisation of a cohort of 11-year olds. We found that the benefits of MCV4 vaccination substantially exceeded the risks of MCV4-associated GBS when measured by cases, deaths, and QALYs. The occurrence of death or long-term disability in up to one-third of meningococcal disease cases as well as the burden of disease caused by serogroups covered by the vaccine contrast with the small and time-limited risk of GBS following vaccination. Further, the decision analysis continues to strongly favour the vaccination program at risks three times those observed in post-licensure surveillance. Also, even at lower meningococcal disease incidence rates, QALYs saved by vaccination are greater than QALYs lost due to vaccine-associated GBS. (Example based on Cho and Clark [56]).

Item 19: Limitations

Definition: The limitations section covers the weaknesses of the analyses and assesses their potential impact on the study results and the interpretation thereof.

Recommendation: The authors should identify and discuss all the relevant limitations of the model and their impacts on the findings.

Explanation: The discussion on limitations provides the readers with potential sources of bias and/or lack of precision. A clear presentation of limitations strengthens the study conclusions and the generalisability of the findings. qBRm applied to vaccines often implies several simplifying assumptions and methodological choices (e.g. time horizon, alternatives, modelling approach). For this reason, it is crucial to identify and discuss the related potential limitations of the model as well as their possible effects on the estimated benefit–risk balance.

Example:

The model was limited in scope, with its exclusive focus on death and life-debilitating effects caused by the pandemic influenza and by the vaccine, respectively. All considerations of non–life-threatening morbidity, hospitalisations and other outcomes were not considered, nor were other populations, such as children and pregnant women, explicitly taken into account. The model is further based on the assumption that vaccination after approval occurs in a timely and broad manner, which does not happen in reality because of differences in vaccine supply and geographical and seasonal differences between healthcare environments. However, even if this were to be taken into account, it is unlikely to change the resulting preference for the earlier decision. (Example based on Phillips et al. [68]).

Item 20: Interpretation and generalisability

Definition: The interpretation and generalisability section contextualises the main findings by comparing them with previous work and discussing the external validity of the study results.

Recommendation: The authors should provide an overall interpretation of the results, considering similar analyses and other relevant evidence. They should discuss the generalisability of the results and potentially suggest recommendations regarding the use of the assessed vaccines.

Explanation: This part of the discussion is crucial to objectively compare qBRm with other studies available in the literature and provide explanation in case of discrepancies. Importantly, it helps readers understand and appreciate the added value of the study to the existing literature [27]. The authors should also discuss the potential generalisability of their results to other populations and geographical settings and suggest recommendations regarding the use of the assessed vaccines based on results of the model.

Example (interpretation):

A study in the USA estimated the prevention of 1093 rotavirus admissions for each additional intussusception admission, closer to our estimate of 1360 averted rotavirus hospitalisations per excess intussusception case under the current age restrictions. In France, it was reported that for every intussusception hospitalisation and every intussusception death caused by vaccination, 1624 rotavirus hospitalisations and 743 deaths were prevented by vaccination, respectively. (Example based on Bruun et al. [69]).

Example (generalisability):

Through the model, we have assumed a homogeneous annual risk of infection throughout the population at a national level. However, it is clear from current notification data that cases are often concentrated in high-risk populations. It is therefore likely that the risk of infection in such sub-population well exceeds the calculated average. Given the above limits, this model nevertheless provides good estimates when compared with surveillance data from France and Italy and it could represent a valid tool to start a decision-making process on BCG vaccination in European countries. (Example based on Manissero et al. [70]).

Other

Item 21: Source of funding

Definition: The source of funding section reports resources received (directly or indirectly) to enable the completion of the work.

Recommendation: The authors should report all sources of funding and the role of each funder, if any, in the different steps of the research conduct (e.g. design, data collection, analysis, drafting of the manuscript, decision to publish). Depending on individual journal policies, source of funding and the contributions of each funder may be listed in an ‘Acknowledgements’ section.

Explanation: The funding can come from various sources, including research councils, charities, governing bodies and industry. Public trust in the scientific work and the credibility of published articles depends in part on how transparently an author’s relationships and activities, directly or topically related to a work, are handled during the scientific process. Consequently, to prevent any controversy and vaccine hesitancy, it is crucial that authors of publications of qBRm applied to vaccines report all sources of funding.

Example:

This work was carried out with the aid of a grant from the International Development Research Centre (IDRC), Ottawa, Canada, as part of the Canadian International Immunization Initiative Phase 2 (CIII2). This initiative is a project of the Global Health Research Initiative (GHRI). The views expressed are those of the authors and do not necessarily reflect the views of IDRC. (Example based on Ledogar et al. [71]).

Item 22: Conflicts of interest

Definition: The conflicts of interest section describes situations where the impartiality of research may be compromised because the researcher stands to profit in some way from the conclusions they draw [72].

Recommendation: The authors should report any financial and non-financial relationships and activities and conflicts of interest that could be perceived as potentially influencing the submitted work. In the absence of a journal policy, authors should refer to the International Committee of Medical Journal Editors [73].

Explanation: As for the source of funding, disclosing any financial and non-financial relationships and activities and conflicts of interest ensures compliance with rules and public trust.

Exmple:

KT received personal fees from the GSK group of companies, Pfizer, and Merck Sharp & Dohme unrelated to this work. The other authors have no conflicts of interest. (Example based on Yung et al. [44]).

Discussion

As qBRm is increasingly used in vaccinology, developing standards for reporting the methods and results is key to ensure a transparent disclosure of the analysis and its reproducibility, thereby facilitating interpretation of the study results and comparison with existing literature. To the best of our knowledge, the BRIVAC statement is the first operational checklist aiming at standardising the reporting of qBRm.

Adequate reporting should allow identification of the strengths and weaknesses of a modelling exercise and provide a valuable basis for decision and communication about vaccine benefit–risk profiles [74]. Consequently, BRIVAC is intended to (1) assist authors in adequately reporting methodologies and results of qBRm applied to vaccines, (2) support editors and peer reviewers when considering such articles for publication and (3) help readers in their critical appraisal.

A statement such as BRIVAC should not be interpreted as an attempt to confine the conduct of qBRm applied to vaccines in a rigid format. Its objective is to ensure that methodological choices are adequately disclosed. The BRIVAC checklist can be used to assess the quality of qBRm reporting but not of its conduct.

Detailed guidance on choice of methods in vaccine qBRm has not yet been achieved and is currently lacking [7]. Future initiatives should be launched to develop methodological guidance to perform qBRm taking vaccine specificities into account. The present series of papers, findings from the Accelerated Development of Vaccine Benefit-Risk Collaboration in Europe (ADVANCE) project [21, 75, 76] and health technology assessment methods should support the development of such guidelines [77].

There may be some limitations to the present approach. First, the BRIVAC checklist assumes that the information required for adequate reporting does not exceed the conventional space limits of some scientific journals. If this is not the case, we recommend using online appendices (when available) to provide an extended description of the methodological approaches used.

In addition, the BRIVAC’s contributors were not representative in terms of geography (over-representation of European researchers), research interests and disciplines. However, we believe that this checklist will serve as a starting point to improve the reporting of vaccine qBRm. The next steps in updating and validating the BRIVAC checklist will include using a large Delphi panel with a multidisciplinary team of editors and content experts in BRA, vaccines, infectious diseases and reporting and representatives from academia, clinical practice, industry, and government.

To enable the validation, dissemination and routine implementation of the BRIVAC checklist, the reporting guidance could be integrated in the EQUATOR (Enhancing the Quality and Transparency of Health Research) network [74], which is an international initiative set up to help improve the reliability and value of health literature by promoting responsible reporting of health research. The EQUATOR website offers an up-to-date centralised resource compiling most reporting guidance related to health research [78].

Conclusion

The BRIVAC statement is an operational checklist for improving the reporting of qBRm applied to vaccines in scientific articles. Its aim is to ensure that methodological choices and results are adequately disclosed to ensure transparent and reproducible analyses, thereby facilitating study result interpretation and comparability. The BRIVAC checklist is the first initiative aiming at structuring and harmonising vaccine qBRm reporting. Future initiatives are needed to validate BRIVAC on a large scale and to provide methodological guidance to perform qBRm taking vaccine specificities into account.

Although quantitative benefit-risk models (qBRm) applied to vaccines are increasingly used as an important tool to help decision makers with supporting benefit-risk assessment, many publications on vaccine qBRm provide insufficient details on the methodological approaches used.

The aim of the present paper is to provide an operational checklist for improved reporting of vaccine qBRm in scientific articles: the consolidated standards of reporting quantitative Benefit-RIsk models applied to VACcines (BRIVAC) checklist.

The BRIVAC checklist is intended to (1) assist authors in adequately reporting qBRm methodologies and results, (2) support editors and peer reviewers when considering such articles for publication and (3) help readers in their critical appraisal.