Benefit-Risk Monitoring of Vaccines Using an Interactive Dashboard: A Methodological Proposal from the ADVANCE Project.

INTRODUCTION: New vaccines are launched based on their benefit-risk (B/R) profile anticipated from clinical development. Proactive post-marketing surveillance is necessary to assess whether the vaccination uptake and the B/R profile are as expected and, ultimately, whether further public health or regulatory actions are needed. There are several, typically not integrated, facets of post-marketing vaccine surveillance: the surveillance of vaccination coverage, vaccine safety, effectiveness and impact.
OBJECTIVE: With this work, we aim to assess the feasibility and added value of using an interactive dashboard as a potential methodology for near real-time monitoring of vaccine coverage and pre-specified health benefits and risks of vaccines.
METHODS: We developed a web application with an interactive dashboard for B/R monitoring. The dashboard is demonstrated using simulated electronic healthcare record data mimicking the introduction of rotavirus vaccination in the UK. The interactive dashboard allows end users to select certain parameters, including expected vaccine effectiveness, age groups, and time periods and allows calculation of the incremental net health benefit (INHB) as well as the incremental benefit-risk ratio (IBRR) for different sets of preference weights. We assessed the potential added value of the dashboard by user testing amongst a range of stakeholders experienced in the post-marketing monitoring of vaccines.
RESULTS: The dashboard was successfully implemented and demonstrated. The feedback from the potential end users was generally positive, although reluctance to using composite B/R measures was expressed.
CONCLUSION: The use of interactive dashboards for B/R monitoring is promising and received support from various stakeholders. In future research, the use of such an interactive dashboard will be further tested with real-life data as opposed to simulated data.

Key Points

Introduction

New vaccines are launched based on their benefit–risk (B/R) profile anticipated from clinical development. Subsequently, proactive post-marketing surveillance is necessary to assess whether the vaccination uptake and the B/R profile are as expected and, ultimately, whether further public health or regulatory actions are needed. There are several, typically not integrated, aspects of post-licensure or post-marketing vaccine surveillance: the surveillance of vaccination uptake and compliance, safety, vaccine effectiveness (VE) and impact. Vaccination uptake refers to the fraction of the vaccination-eligible population that is vaccinated, whereas compliance refers to the fraction that is vaccinated following the recommendations. Both indicators are important to evaluate the success of vaccination programmes and are typically measured based on vaccine registries, routine administrative reports or household surveys. Vaccine safety monitoring is normally implemented upon introduction of a new vaccine, and may be further enhanced in the event of switching vaccine brand or expansion of the targeted population. Timely safety monitoring of pre-identified adverse events (AEs) of special interest (AESIs), which often include serious and rare AEs and/or those with a long latency period, is critical since many of these events may not have been detected in pre-licensure studies because of a lack of power and/or limited follow-up time [1]. A wide variety of methods are available to carry out such safety monitoring of vaccines during the post-licensure period [2], with an increased focus on near real-time surveillance using existing electronic health record (EHR) databases [3-5]. VE and impact are also considered following the launch of a vaccine. The initial emphasis here is placed on monitoring incidence rates of the vaccine-preventable disease (e.g. using laboratory-confirmed cases or hospital admissions), with VE and impact more robustly estimated through epidemiological analyses conducted at one point in time once sufficient data have been accumulated.

Although quantitative B/R assessments, by which the benefits of a medical intervention are offset against its risks at one point in time, are increasingly performed [6, 7], integrated post-marketing monitoring of coverage, benefits, risks, and B/R measures is—to our knowledge—not yet implemented in practice. Recently, Gagne et al. [8] were the first to explore the feasibility of near real-time monitoring of the comparative safety and benefits of drugs using EHR databases. For vaccines, several one-point-in-time B/R assessments have been carried out (e.g. [9-11]), but none of them considered near real-time monitoring.

With this work, we propose a framework and explore methodology for near real-time B/R monitoring of vaccines. We explored its technical feasibility and solicited feedback on its added value amongst a range of stakeholders experienced in the post-marketing monitoring of vaccines. Specifically, the methodology visualises key data for monitoring vaccine coverage, benefits, and safety, which are then combined into composite measures of the vaccine B/R profile as it evolves over time. To facilitate monitoring, we developed an interactive dashboard, which we illustrated using simulated data reflective of the introduction of rotavirus vaccination in the UK.

This work was carried out under the auspices of the Accelerated Development of Vaccine Benefit-Risk Collaboration in Europe (ADVANCE) project, launched in 2013, funded by the Innovative Medicines Initiative (IMI) (http://www.advance-vaccines.eu/). The aim of ADVANCE is to help health professionals, regulatory agencies, public health institutions, vaccine manufacturers, and the general public make well-informed and timely decisions on the benefits and risks of marketed vaccines by establishing a framework and toolbox to enable rapid delivery of reliable data on vaccine benefits and risks.

Methods

Benefit–Risk Monitoring Framework

At the core of the mission of ADVANCE and many of its stakeholders is the concept of vaccine B/R monitoring. Monitoring in this context should be understood as a periodic assessment of several key parameters, including coverage and the incidence of the vaccine-preventable disease and AEs, to trigger an alert if and when there is an indication that the B/R profile in the population is different from what is expected (based on clinical trials, observational studies or similar products). This alert would generate a subsequent and possibly more formal assessment and analysis of the vaccine. Monitoring should, in principle, start as soon as a new vaccine is introduced in a given country and continue throughout the vaccine’s lifecycle, and would also be applied to vaccines with established B/R profiles. B/R monitoring requires information that is available in a timely fashion. Hence, the goal is to have access to near real-time information, which is defined as either weekly or monthly refreshed data that are only a few days old. Within this context of B/R monitoring, post-licensure vaccine safety monitoring is restricted to the ongoing evaluation of AESI such as those safety signals identified during clinical development or from experience in previous vaccine campaigns [12].

Data Simulation

To illustrate the B/R monitoring, we simulated data reflective of the introduction of rotavirus vaccine in the UK. The national immunisation programme in the UK includes RV1, a vaccine for the prevention of severe gastroenteritis caused by rotavirus infection in young children. It was introduced into the schedule in July 2013. The vaccine is administered orally in primary care, in two doses at 2 and 3 months of age. We chose this test case because the benefits are expected to be immediate, there is at least one serious identified AE (which was a pre-identified AESI prior to vaccine introduction) and many publications on the safety and benefits of rotavirus vaccination in the UK exist [9, 13–15], including a B/R analysis [9], which we used to inform the key parameters of our data simulation model.

To illustrate the monitoring, we chose to include two benefit outcomes and one risk associated with RV1 vaccination. The benefits were defined as reductions in rotavirus gastroenteritis (RVGE)-related general practioner (GP) visits and hospital admissions. The risk was intussusception (IS), a rare but also naturally occurring serious condition where part of the intestine prolapses into itself. IS was shown to be temporarily associated with administration of a previous rotavirus vaccine that was withdrawn from the market [16] and, while not observed in large pre-licensing clinical trials, was later found to be also associated with the newer rotavirus vaccines in use today [17].

To simulate the data, we closely followed the model as detailed in Clark et al. [9]. All assumptions and parameters, except the coverage and age at vaccination, were obtained from that publication. Specifically, we simulated data on five consecutive birth cohorts of an arbitrary size of 300,000 children each. All children were followed from date of birth until 12 months of age. The first two birth cohorts were from prior to the introduction of the vaccination programme to allow estimation of baseline rates and detection of changes over time unrelated to the vaccine. We generated dates of birth, dates of vaccination with the first and second RV1 dose, dates of onset of RVGE resulting either in a GP visit or a hospital admission and dates of onset of IS (see Table 1 for parameters and assumptions). The coverage and age at vaccination for the first and second doses reflected the actual RV1 uptake in the UK, for which a two-dose coverage of 88% at 12 months of age was reported for the year of vaccine introduction [15]. For vaccinated subjects, in order to estimate the number of RVGE prevented events, the likelihood of prevention was simulated as a function of the dose- and outcome-specific VE and the time of the prevented event since last dose accounting for waning of protection [9]. For vaccinated subjects, the risk of IS was simulated for two risk windows (1–7 days and 8–21 days post-vaccination, where day 0 is the day of vaccination) following both the first and second dose. The statistical package R. 3.3.1 was used to simulate the data [18].

Table 1

Parameters and probability distributions used to generate the simulated data

Parameter	Value/distribution
RV1 vaccinationa
Coverage (at 12 months): dose 1	93%
Age at vaccination (in weeks): dose 1	~ Gamma (rate = 1.42, shape = 12.16)|(0, 52.14)
Coverage (at 12 months): dose 2	88%
Age at vaccination (in weeks): dose 2	~ Gamma (rate = 0.68, shape = 3.05, shift = 8)|(0, 52.14)
RVGEb
Annual baseline incidence per 1000 births (< 5 years)
RVGE GP visits	28.4
RVGE hospitalisations	4.5
Age at RVGE in weeks (mean = 70.7, SD = 36.6)	~ Gamma (rate = 0.053, shape = 3.73, shift = 8.26)|(0, 52.14)
VEc
RVGE GP visits: dose 1	87.4%, decay curve: VE(1-Φ_t(μ=3.2,σ=0.55)
RVGE GP visits: dose 2	95.2%, decay curve: VE(1-Φ_t(μ=3.11,σ=0.96)
RVGE hospitalisations: dose 1	96.04%, decay curve: VE(1-Φ_t(μ=3.17,σ=0.42)
RVGE hospitalisations: dose 2	99.4%, decay curve: VE(1-Φ_t(μ=3.43,σ=0.77)
IS**
Annual baseline incidence per 100,000 births (< 12 months)	28.1
Age at IS in weeks (mean = 30.8, SD = 14.2)	~ Gamma (rate = 0.15, shape = 4.7, shift = − 0.36)|(0, 52.14)
Relative risk of vaccine-related IS vs background rate
Risk period (1–7 days): dose 1	6.76
Risk period (8–21 days): dose 1	3.45
Risk period (1–7 days): dose 2	2.84
Risk period (8–21 days): dose 2	2.11

GP general practitioner, IS intussusception, RVGE rotavirus gastroenteritis, SD standard deviation, VE vaccine effectiveness

aFrom Public Health England [15]

bFrom Clark et al. [9]

cSigmoid lognormal decay curve: with VE at the time of vaccination and with time t in months

Near Real-Time Benefit–Risk Monitoring

The components of a composite B/R measure for vaccine B/R monitoring are measures of vaccination coverage, benefits and risks. We monitored the components separately to allow interpretation of potential changes in the composite B/R measures and to provide data that can be used to support early decision-making and communication. We built a web application with an interactive dashboard to facilitate monitoring. The dashboard is interactive as it allows users to define certain options. The architecture of the dashboard is described in Electronic Supplementary Material #1. All analyses were carried out using R 3.3.1 [18], and the web application was built using the Shiny package [19]. Details on the calculations are given in Electronic Supplementary Material #2.

Coverage

To provide exposure information for potential safety signal evaluation analysis, such as observed-to-expected analyses using individual case reports [20], we monitored the weekly number of RV1 doses by (user-defined) age groups for doses 1 and 2. The number of administered doses recorded in the database was then extrapolated to the whole UK population, accounting for the age structure of the active population captured in the database.

Compliance to the recommended vaccination schedule was assessed through monitoring the weekly vaccination coverage (%) by age. Specifically, the weekly coverage was calculated by birth cohort (defined by year and month of birth) and as the proportion of children who had been vaccinated among those who had reached a certain (user-defined) age within a given week.

Risk

The IS incidence rate (per 10,000 person years) and pointwise exact Poisson 95% confidence intervals (CIs) were estimated for two risk windows (1–7 and 8–21 days) after each dose. The rates were estimated cumulatively over time, i.e. using the accruing data. We opted to do so to maximise sample size and, hence, accuracy, as the expected absolute vaccine-associated AE rate is small and unlikely to change over time.

The baseline IS incidence rates were estimated for children of vaccination-eligible age from the two pre-vaccination birth cohorts. Specifically, the baseline rates were estimated for children aged 8–10, 9–12, 12–14 and 13–15 weeks for comparison with the incidence within 1–7 days post dose 1, 8–21 days post dose 1, 1–7 days post dose 2 and 8–21 days post dose 2, respectively. These age groups were chosen given the age-specific recommendations for each dose (8 weeks at dose 1 and 12 weeks at dose 2), with time after those ages included to account for the variability in the age at vaccination as well as the length of the risk windows.

Benefits

The incidence rate (per 10,000 person years) and pointwise exact Poisson 95% CIs of RVGE GP visits and of hospitalisations in the total population of infants aged 0–1 year were calculated with a ‘moving average’ method. This means that a ‘window’ of defined width was moved over the data, starting 2 years prior to the vaccine introduction until the most recent data. Then for every ‘window’ of data, the incidence rate was calculated. The width of the windows can be chosen by end users, to allow balancing freshness of data (i.e. only using the most recent data) versus accuracy of the rate estimates (i.e. the more data, the more accurate the estimates will be).

At the beginning of a vaccination campaign, the vaccine benefits are often not yet observable and the VE in this population is by definition unknown. We therefore use the ‘expected’ VE, which is a user-defined parameter. Ideally, the ‘expected’ VE is evidence based (e.g. informed by clinical development) and/or set to a conservative value. Then, starting from the ‘expected’ VE and the baseline incidence, we derived the ‘expected’ incidence while accounting for the observed age-specific vaccination coverage and age distribution of the database population. For simplicity, and because most of the vaccinated children received two doses, a conservative estimate of ‘expected’ incidence was obtained ignoring the protection from one dose only (i.e. children receiving one dose only were considered unvaccinated) and ignoring the potential indirect effects. The ‘expected’ incidence then serves as a benchmark for the observed incidence.

Benefit–Risk

Two composite B/R measures were used for visual B/R monitoring: the incremental net health benefit (INHB) and the incremental benefit–risk ratio (IBRR) [18, 19]. They are the only two trade-off indices recommended by IMI-PROTECT [21] and are valued for their simplicity, which makes them suited for monitoring.

The INHB is essentially a difference between a sum of weighted incremental benefits and a sum of weighted incremental risks or,where and refer to the number of benefit and risk outcomes, where the incremental benefits are the difference between the benefits in the absence of vaccination or baseline benefits ( and the benefits after vaccination (), and similarly for the incremental risks (). The weights are all positive and reflect the relative severity of the health outcomes. Note that, because and are subtracted from their baseline values, the incremental benefits are positive and the incremental risks are negative. The IBRR is the ratio of the incremental benefits to the incremental risks or,with positive terms for both numerator and denominator.

The INHB and IBRR were calculated for the total population of infants aged 0–1 years, with the incremental benefits being estimated by comparing the observed incidence rates (per 10,000 person years) of RVGE GP visits and of hospital admissions within ‘moving windows’ after vaccine introduction to the baseline incidences, calculated based on the pre-vaccination birth cohorts (see Sect. 2.3.3). The incremental (or excess) IS risk for each risk window after each dose was obtained by estimating the attributable number of cases (per 10,000 person years). First the attributable fractions (AF) were calculated from the relative incidences (RI) as AF = (RI − 1)/RI, with the RI estimated using the accrued data (see Sect. 2.3.2). Then the observed number of cases within each risk window was multiplied with the AF to obtain the attributable number of cases, which were then used to calculate the attributable incidence (per 10,000 person years). The pointwise 95% Wald CIs of the INHB and IBRR were obtained as well (see Electronic Supplementary Material #2 for the derivations of the CIs).

Immediately after vaccine launch, the benefits are often not yet observable, whereas the short-term risks are. To be able to contextualise the short-term risks when the benefits are not yet observable, we calculated the INHB and IBRR based on expected benefits for an assumed level of VE. Specifically, the ‘expected’ benefits were derived from baseline incidence as observed pre-vaccination and user-defined VE, while accounting for the observed age-specific vaccination coverage and age distribution of the database population. For simplicity, we assumed that all vaccinated children would be completely vaccinated. The incremental risks were calculated as before.

User-Feedback Solicitation

In order to assess the added value of the proposed methodology, user feedback was solicited from potential users from different stakeholders (pharmaceutical companies, public health institutes, and regulatory agencies). The dashboard was demonstrated, and user feedback was subsequently solicited through completion of a semi-structured questionnaire and follow-up discussions.

Results

Demonstration with Fictitious Data

The dashboard is available from http://apps.p-95.com/BRMonitor/. The weekly number of administered doses within user-defined age groups is depicted using stacked area charts (Fig. 1). Dose 1 was mostly given to 8- to 10-week-olds, whereas dose 2 was mostly given to 12- to 14-week-olds. The weekly vaccination coverage (%) by user-defined age groups and by month-year birth cohort showed a rapid uptake of the vaccine upon introduction (Figs. 2, 3). The IS incidence rates (per 10,000 person years) are displayed using line charts, with the shaded areas representing the 95% pointwise CIs (Fig. 4). Immediately after the introduction of the vaccine, the CIs of the IS incidences within the post-vaccination risk windows were wide. For accumulating data, the CIs narrowed, with the largest increased risk observed 1–8 days after dose 1. The observed incidence rates of RVGE GP visits and hospitalisations (per 10,000 person years) within ‘moving windows’ with a width of 26 weeks were lower compared with the ‘expected’ incidences when baseline incidences were as observed pre-vaccination and the user-defined VE was 60% (Figs. 5, 6, 7). As should be, the observed incidence was lower compared to the ‘expected’ incidence, with the former being based on simulated data for which VE > 90% was assumed (see Table 1). Note that there are only small differences between the observed and ‘expected’ incidence rates before week 20. This is because only limited disease is prevented during the first weeks after vaccine introduction because the vaccinated children (age at vaccination is 8–14 weeks) are still too young to develop the vaccine-preventable disease (RVGE occurs on average at 52 weeks [22]).

Fig. 1

Number of administered doses (left: dose 1, right: dose 2) in the UK population by user-defined age groups, by calendar time (in weeks). The vertical red line indicates the time at vaccine introduction

Fig. 2

Coverage (%) (left: dose 1, right: dose 2) in children who reached a certain user-defined age, by calendar time (in weeks). The vertical red line indicates the time at vaccine introduction

Fig. 3

Coverage (%) (left: dose 1, right: dose 2) by birth cohort defined by year and month. The vertical red line indicates the time at vaccine introduction

Fig. 4

Incidence rate per 10,000 person years (95% confidence interval) of intussusception estimated cumulatively over time, incidence prior to vaccination (7–12 weeks and 11–16 weeks) and in risk windows (1–7 and 8–21 days post vaccination) by dose. The vertical red line indicates the time at vaccine introduction

Fig. 5

Incidence rate per 10,000 person years (95% confidence interval) of AGE GP visits (left) and hospital admissions (right) in total population within ‘moving windows’ with user-defined width. The ‘expected’ incidence is calculated for user-defined levels of baseline incidence and vaccine effectiveness. AGE acute gastroenteritis, GP general practitioner. The vertical red line indicates the time at vaccine introduction

Fig. 6

INHB (left) and the IBRR (right) (95% confidence interval) with observed benefits for a population of 10,000 children with the vaccination coverage as observed followed from birth until 1 year of age. For the INHB, the weighted components are also displayed. AGE acute gastroenteritis, GP general practitioner, INHB incremental net health benefit. The vertical red line indicates the time at vaccine introduction, IBRR incremental benefit–risk ratio

Fig. 7

INHB (left) and the IBRR (right) (95% confidence interval) with ‘expected’ benefits per 10,000 fully vaccinated children followed from birth until 1 year of age. For the INHB, the weighted components are displayed as well. AGE acute gastroenteritis, GP general practitioner, IBRR incremental benefit–risk ratio, INHB incremental net health benefit. The vertical red line indicates the time at vaccine introduction

For defining the preference weights, we selected RVGE GP visits as the reference outcome for which a preference weight of 1 was assumed. Then the preference weights for RVGE hospitalisations and IS were assumed to be 20 and 50, respectively. As this work concerns methods development, the weights were chosen by the study authors and not robustly elicited from a relevant panel. Given the assumed preference weights, the composite B/R measure with observed benefits was initially negative and turned positive at ± 27 weeks after the vaccine's introduction. The INHB was 336.5 (95% CI 279.1–393.9) at week 40 and 997.9 (95% CI 974.9–1021.0) at week 80 for a population of 10,000 children with a vaccination coverage as observed followed from birth until 1 year of age. The INHB is to be interpreted as the net change in the rate of RVGE GP visits, which was selected as a reference outcome. The IBRR showed similar trends. The initial negative IBRR is explained by comparing immediate risks (excess risk of IS within the first 3 weeks after vaccination recommended at 8 and 12 weeks of age) with long-term benefits (with the peak age of RVGE at ± 52 weeks). The INHB and IBRR with ‘expected’ benefits derived from baseline incidences as observed pre-vaccination and a user-defined VE of 60% were always positive with an INHB of 693.2 (95% CI 689.8–696.5) at week 40 and 689.2 (95% CI 685.9–693.5) at week 80. Note that initially the INHB based on observed benefits was lower compared to the one based on ‘expected’ benefits. However, this reversed over time.

User Feedback

Eleven potential end users from seven different organisations provided solicited feedback. Of the organisations, two were regulatory authorities, two were public health institutes and three were pharmaceutical companies. Of the seven organisations, five indicated that the dashboard is of added value, although with some caveats: one was not sure and one indicated that the dashboard was of limited value. The dashboard was also presented to the ADVANCE independent Scientific Advisory Board.

The feedback has been summarised and can be broadly categorised into the following considerations:

Added value of such a dashboard:

Both the visualisation of individual components within this dashboard and the interactive interface were appreciated, and it was felt that it could add value for informing some evaluations.

Comments on the implementation of the dashboard

The dashboard is limited to pre-specified AESIs (by design); therefore this is not ‘complete’ B/R monitoring, but rather monitoring of expected benefits and risks. Not all vaccine/AE/benefit combinations lend themselves equally suitable for such monitoring.

In using a time window for the effectiveness data, the analyses may be seen as more like a quality assurance process than a standard B/R process, and so raises the following questions: What time window is appropriate, giving the best balance between obtaining sufficient data for a stable estimate, but also responding quickly to arising problems, and what signalling criteria will give good operating characteristics?

The B/R output is dependent on weighting, which can vary appreciably between users and stakeholders; at the same time, however, the dashboard can help to make the thinking behind the weighting and other parameters specification more explicit and allows experimentation with different weights, which could be seen as a way of assessing the sensitivity of the B/R measures to different value judgments. However, multiple sensitivity analyses regarding the weighting might be required, which is not easy to do using such a dashboard.

The quantitative B/R analyses may be easily misinterpreted, and a structured visual summary of the results such as an effect table may be preferred by regulatory decision-makers.

Potential limitations of using EHR for monitoring

Not all events are appropriate for monitoring using EHR databases given their rarity or a lack of or misclassification in the data, and for some events in particular, other sources might have to be considered.

A main issue that remains is the availability, or lack thereof, of timely data that are appropriate for monitoring.

Discussion

In this paper, we propose a framework and methodology for near real-time B/R monitoring of vaccines. Particularly, we visually monitor the composite B/R measures INHB [23] and IBRR [24] as well as their components: vaccination coverage, benefits and risks. The main purpose of this monitoring is to enable the use of EHR databases for prospective signal detection, not only with regard to a number of pre-specified AESIs, but also early signals of changes in vaccine uptake, changes in, or differences from the expected, effectiveness, and potential changes in the anticipated B/R measures. We are not seeking to generate robust data to fully demonstrate the safety or effectiveness of a vaccine, but are trying to prospectively gather data to build upon our knowledge base, which is particularly important when a new vaccine is first launched. This proactive approach also helps stakeholders to be prepared should there be a need to rapidly consider a new safety signal. To facilitate the monitoring, we built a web application with an interactive dashboard.

The INHB, which underlies many B/R assessment methods [25], and the IBRR are two of the most simple and intuitive trade off indices [21], making them best suited for B/R monitoring. For vaccines, risks often become apparent earlier than benefits so this needs to be considered when signalling or interpreting the composite B/R measures. Both measures are also commonly used in cost-effectiveness research [26]. In the context of immunisation, the INHB is an absolute measure indicating how much the total disease burden in the population changes due to vaccination, accounting for both increases in disease burden as a result of reductions in vaccine-preventable disease and decreases in disease burden as a result of induced AEs. The INHB is > 0 when the prevented disease burden outweighs the induced disease burden. The INHB requires preference weights on a scale with a clearly defined reference point to allow a meaningful interpretation of changes in INHB. In this work, we presented the INHB using RVGE GP visits as the reference outcome, for which a weight of 1 was assumed. Alternatively, absolute weights such as the disability adjusted life expectancy (DALY) weights with a scale from 0 (perfect health) to 1 (equivalent to death) could be considered as well [27]. The IBRR is a relative measure, indicating how much disease burden is prevented relative to the disease burden induced, accounting for the relative importance of the various benefit and risk outcomes. The IBRR is > 1 when the prevented disease burden is larger than the induced burden. The IBRR only requires weights on a relative scale. Many different techniques for obtaining preference weights exist, including multi-criteria decision analysis (MCDA) swing-weighting, by which preferences from medical experts can be obtained, or discrete choice experiments, which have become the most common technique to elicit preferences from patients. However, recommendations on how preferences should be assessed and inform decision-making are still lacking and initiatives on this topic are ongoing (e.g. IMI-PREFER project [28]). For vaccines, it is particularly challenging to elicit preferences from ‘patients’ because in this context, there are no patients, but candidate vaccine recipients (or their parents), who are often not familiar with the disease the vaccine prevents nor with the potential AEs.

To the best of our knowledge, this is the first time that a web application with an interactive dashboard has been developed in the context of post-marketing monitoring of vaccines. Dashboards are used in many fields and are well suited for monitoring, as changes over time can be visualised and the underlying data can be seamlessly updated. The dashboard was made to be interactive, allowing end users to select age groups or time windows, to calculate the B/R measures for different sets of preference weights and to conduct sensitivity analyses.

For developmental and illustrative purposes, we use simulated data reflective of the introduction of rotavirus vaccination in the UK. Several simplifications were made when simulating the data. We ignored herd immunity and seasonal trends in gastroenteritis. Although the dashboard was developed with the ultimate objective of near real-time B/R monitoring using EHR databases, our simulated data assume no disease misclassification (e.g. RVGE events are mostly not recorded as such in healthcare databases, but rather as unspecified gastroenteritis), no exposure misclassification, no confounding and no incomplete follow-up, which are all commonly present in healthcare databases.

Obviously, successful near real-time B/R monitoring depends on data being available in a timely fashion, i.e. both frequent refresh and small time-lag between the occurrence of the event and it being recorded in the databases. The EHR databases are potential sources for implementing near real-time B/R monitoring. Recently, it was explored whether the Clinical Practice Research Datalink (CPRD), an EHR database from the UK currently having data on 4.4 million active patients, could be used for near real-time vaccine safety monitoring. The researchers observed reasonable reporting delays, making near-real time safety monitoring possible [29], although concerns regarding power remain [30]. In future research, we will explore data availability and reporting delays for monitoring coverage, benefits and risks in several European EHR databases from Denmark, Italy, Spain and the UK.

Such a real-life scenario of near real-time B/R monitoring using one or several EHR databases should possibly account for confounding, misclassification, incomplete follow-up, and, when combining data, heterogeneity across databases. The visualisations and underlying calculations will probably have to be modified depending on the vaccine, health outcomes of interest and data sources. For instance, for rotavirus vaccine, the benefits of vaccination could be assessed based on population-level impact, while for other vaccines, such as traveller vaccines, the benefits can only be assessed by estimating VE. Also, depending on the AE of interest, the risk window after vaccination will be different. These and other study design considerations are product specific. While cost-effectiveness is beyond the scope of this work, some of the outputs from the B/R monitoring could be used to monitor key variables that impact on cost-effectiveness as well.

The feedback from a group of potential end users was generally positive, particularly with regard to the analyses of the individual components. There was reluctance to use composite B/R measures as they typically require weights, and there were also concerns about their misinterpretation. The inclusion of composite measures therefore needs to be carefully considered. While there may be interest in these measures internally within some organisations for signal detection purposes, this is dependent on users having sufficient understanding of the analyses and their limitations. It is currently not recommended that such composite measures are disseminated more widely as this increases the risk of misunderstanding. For wider dissemination, approaches by which coverage, benefits and risks are separately monitored might be preferred. While there is no uniform consensus from the surveyed potential end users on the value of the dashboard, the feedback does not preclude further exploration of near real-time B/R monitoring of vaccines using interactive dashboards. Such monitoring might complement, but not replace, other activities. For example, signal detection using spontaneous reports and other data sources will still run in parallel, and if a new safety concern is identified, then the event can be easily added to the monitoring tool.

Conclusion

This work showed the technical feasibility and investigated the potential added value of using an interactive dashboard as a potential methodology for near real-time B/R monitoring of vaccines. The solicited user feedback indicated that the proposed methodology is promising despite the reluctance to use composite B/R measures. The EHR databases are potential sources for implementing near real-time B/R monitoring. Next steps include testing the methodology with real-world data from various European EHR databases. Finally, we would like to stress that the results presented in this paper should not be used to support any conclusions with regard to the actual B/R profile of rotavirus vaccination in the UK, as this work used simulated data and several simplifying assumptions were made with the sole purpose of methodology development.

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 963 kb)

We explored the feasibility and added value of near real-time monitoring of the post-marketing benefits and risks of vaccines.

To this end, we developed an interactive dashboard for monitoring vaccination coverage, benefits, risks, and benefit–risk measures, populated with simulated data.

The solicited user feedback indicates that the proposed methodology is promising but requires testing with real-life data.