Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System.

PURPOSE: To develop the infrastructure to conduct timely active surveillance for safety of influenza vaccines and other medical countermeasures in the Sentinel System (formerly the Mini-Sentinel Pilot), a Food and Drug Administration-sponsored national surveillance system that typically relies on data that are mature, settled, and updated quarterly.
METHODS: Three Data Partners provided their earliest available ("fresh") cumulative claims data on influenza vaccination and health outcomes 3-4 times on a staggered basis during the 2013-2014 influenza season, collectively producing 10 data updates. We monitored anaphylaxis in the entire population using a cohort design and seizures in children ≤4 years of age using both a self-controlled risk interval design (primary) and a cohort design (secondary). After each data update, we conducted sequential analysis for inactivated (IIV) and live (LAIV) influenza vaccines using the Maximized Sequential Probability Ratio Test, adjusting for data-lag.
RESULTS: Most of the 10 sequential analyses were conducted within 6 weeks of the last care-date in the cumulative dataset. A total of 6 682 336 doses of IIV and 782 125 doses of LAIV were captured. The primary analyses did not identify any statistical signals following IIV or LAIV. In secondary analysis, the risk of seizures was higher following concomitant IIV and PCV13 than historically after IIV in 6- to 23-month-olds (relative risk = 2.7), which requires further investigation.
CONCLUSIONS: The Sentinel System can implement a sequential analysis system that uses fresh data for medical product safety surveillance. Active surveillance using sequential analysis of fresh data holds promise for detecting clinically significant health risks early. Limitations of employing fresh data for surveillance include cost and the need for careful scrutiny of signals.

Introduction

The H1N1 pandemic of 2009 created an urgent need to stand up multiple vaccine safety surveillance systems to support public health efforts to safely vaccinate the U.S. population.1 The Centers for Disease Control and Prevention‐sponsored Vaccine Safety Datalink 2 had already pioneered the development and application of sequential analysis methods using timely data from managed care organizations,3, 4, 5 and the Food and Drug Administration (FDA) and Centers for Medicare and Medicaid Services (CMS) had developed the technical and methodological capability to use weekly Medicare administrative data for safety monitoring. In conjunction with the existing VSD and CMS systems,6, 7 H1N1 vaccine safety surveillance by the FDA‐sponsored Post‐licensure Rapid Immunization Safety Monitoring (PRISM) system was launched in the fall of 2009, using data from large health insurers and state immunization registries.1, 8 A major challenge was the critical need to obtain and analyze recent data on a frequent basis in order to detect safety problems in time to intervene, as influenza vaccines are typically given within a short span of time.9

Subsequently, FDA wished to determine the feasibility of conducting timely sequential analysis for influenza vaccine safety as an integral part of the Sentinel System (formerly the Mini‐Sentinel Pilot10). Currently, Sentinel data are refreshed on a quarterly basis and contain relatively settled and complete data, the most recent of which are 6–9 months old. The time required for data to settle would limit the ability to inform regulatory decisions about the use of influenza vaccine in a timely manner. The goal was to access, use, and evaluate fresh data for timely influenza vaccine safety surveillance in the Sentinel population in order to develop a potentially sustainable infrastructure to apply to other FDA‐regulated medical products requiring faster access to safety information.

Methods

Study periods, populations, and data sources

The 2012–2013 influenza season was used to pilot and evaluate the system. We conducted actual surveillance for influenza vaccine safety in 2013–2014, incorporating data from 1 September 2013 to 30 April 2014. Aetna, HealthCore (with WellPoint/Anthem data), and Humana (“Data Partners”) provided claims data on vaccine exposures and health outcomes of interest. Additional immunization data for Data Partner members were obtained from eight participating immunization registries: Florida, Michigan, Minnesota, New York City, New York State, Pennsylvania, Virginia, and Wisconsin.

Data processing

The source files were internal member‐level files at each Data Partner that included only claims that were adjudicated or, if no reimbursement was expected, recorded. They were refreshed by Data Partners in the last half of each month and normally included data on healthcare events through the end of the prior calendar month. Approximately every 2 months, 3–4 times during 2013–2014, each Data Partner translated their source files to standard‐format patient‐level files including all medical and pharmacy claims with service/fill dates ≥ 1 September 2012. A distributed SAS program aggregated data into a vaccine file and a diagnosis file, each with a summary count of the cumulative number of members in each stratum defined by week of vaccination, age group, sex, and other variables.

Data Partners provided lists of enrolled members as of October 2013 to the eight participating registries once during 2013–2014. The registries returned available immunization data for members, which Data Partners converted into a standard format. Immunization registry data were incorporated into the last generation of each Data Partner's aggregate data for 2013–2014.

Data refreshes were staggered among the Data Partners, and sequential analysis was conducted after each, for a total of 10 analyses over the course of the season.

Vaccine exposures

Vaccination was ascertained by CPT, CVX, HCPCS, and NDC codes. We conducted separate sequential analyses for live attenuated influenza vaccine (LAIV) and for pooled inactivated influenza vaccines (IIV). NDC codes are highly specific, while other coding systems have been less so. All inactivated influenza vaccines were combined for sequential analysis, given that the counts of health outcomes of interest after specific vaccines were expected to be too low to produce interpretable results in separate statistical analyses.

Health outcomes of interest

We monitored the risk of two health outcomes, anaphylaxis and seizures. The case‐finding algorithms are shown in Table S1 in the Supporting Information. A previous Mini‐Sentinel study had found a positive predictive value of 69% for the anaphylaxis algorithm.11 The seizures algorithm had a positive predictive value of 70% for febrile seizures in a PRISM study of children 6–59 months of age.12 Because the increase in risk of febrile seizures following IIV was greater among children receiving concomitant pneumococcal conjugate vaccine (PCV13) in the risk vs. the control period in 2010–2011 in the VSD system,13 we stratified seizures in the 6‐ to 23‐month‐old age group by the presence/absence of concomitant PCV13. We also monitored seizures in the 24‐ to 59‐month‐old age group.

Sequential analysis designs and statistical methods

We used a cohort design (“current‐vs.‐historical comparison”) for anaphylaxis. The cumulative number of cases in a pre‐specified risk interval following vaccination was compared with the number expected based on historical rates after vaccination.6 This approach has often been used in sequential analysis for rare outcomes, because it has better power than comparisons with concurrent controls, including the self‐controlled risk interval (SCRI) approach described below.14 A limitation of the current‐vs.‐historical approach is that historical influenza vaccinees may not be an entirely appropriate comparison group for influenza vaccinees in the season of interest because of different population characteristics, secular trends in coded health outcomes, different concomitant vaccines, and/or differences in influenza vaccines over time.

For seizures, we designated the SCRI design5, 6, 13, 15, 16 as primary. The SCRI design is a special (and simpler) case of both the case‐crossover17 and the self‐controlled case series18 designs. The cumulative numbers of cases in pre‐specified risk and control intervals are compared, adjusting for unequal interval lengths. The analysis is conditioned on the individual, and only those with a seizure in either the risk or the control interval contribute to the analysis. This self‐controlled design is our preferred approach for influenza vaccine safety monitoring, because it controls for all fixed potential confounders, such as sex and co‐morbidities. A limitation of the method is that time‐varying confounders, such as age and seasonality, may bias the findings. However, such confounding was mitigated here by the less‐than‐3‐week‐long follow‐up period (Table 1). Another limitation is that for rare outcomes, power to detect signals in a timely fashion may be low, particularly if the effect size is modest. We used current‐vs.‐historical comparison as a secondary method for seizures to detect any increased risk earlier than would have been possible with the SCRI method alone.

Table 1

Sequential and pre‐specified end‐of‐season analyses

HOI	Influenza vaccine type	Age group	1° sequential analysis method	2° sequential analysis method	Risk interval	Control interval for SCRI	Historical data to be used for current vs. historical comparison
Anaphylaxis	IIV	≥6 months	Current vs. historical (CmaxSPRT)*	n.a.	0–1 days	n.a.	0–1 days post‐IIV
Anaphylaxis	LAIV	2–49 years	Current vs. historical (Poisson maxSPRT)*	n.a.	0–1 days	n.a.	0–1 days post‐IIV †
Seizures in youngest, concomitant PCV13	IIV	6–23 months	SCRI (binomial maxSPRT)	Current vs. historical (Poisson maxSPRT)*	0–1 days‡	14–20 days§	0–1 days post‐IIV¶
Seizures in youngest, no concomitant PCV13	IIV	6–23 months	SCRI (binomial maxSPRT)	Current vs. historical (CmaxSPRT)*	0–1 days‡	14–20 days§	0–1 days post‐IIV¶
Seizures	IIV	24–59 months	SCRI (binomial maxSPRT)	Current vs. historical (CmaxSPRT)*	0–1 days‡	14–20 days§	0–1 days post‐IIV¶
Seizures	LAIV	24–59 months	SCRI (binomial maxSPRT)	Current vs. historical (two) (CmaxSPRT)*	1–3 days‡	15–20 days§	1–3 days post‐LAIV and 0–1 days post‐IIV¶, with rate augmented by 50% to match 3‐day post‐LAIV risk interval

See References 4 and 19 for an explanation of the distinction between the maxSPRT and the CmaxSPRT.

Historical data on anaphylaxis after LAIV are typically very sparse, so post‐IIV historical rates were used instead of post‐LAIV historical rates.

Seizures risk windows after IIV and LAIV were based on Rowhani‐Rahbar, et al. 20

Control window starts a multiple of 7 days after start of risk window to minimize bias from day‐of‐week effects. Control window starts 2 weeks (instead of 1 week) after vaccination in order to exclude period of increased risk of seizures after MMR or MMRV vaccination. (This is less relevant for the 24‐ to 59‐month age group, but control windows were kept similar for consistency.)

The historical rates used were from prior to July 2010, which is also largely prior to any concomitant PCV13 usage. The purpose of this restriction was to exclude influenza seasons in which the risk of post‐IIV seizure was elevated and to exclude most concomitant PCV13.

Three different variants of the Maximized Sequential Probability Ratio Test (maxSPRT) were used, which adjusted for the repeated looks at the accumulating data entailed in sequential analysis: the maxSPRT for Poisson data,3 the maxSPRT for binomial data,3 and the conditional maxSPRT (CmaxSPRT) for Poisson data.4 The test statistic was the log‐likelihood ratio (LLR). One‐tailed tests were used, because we were looking only for elevated risks from vaccination rather than for protective effects; alpha was set at 0.05. These tests and their inputs are elaborated upon elsewhere.19 The details of the various sequential analyses and comparisons are summarized in Table 1.

Adjustment for incomplete data in sequential analysis

We conducted analyses using fresh data in order to obtain timely results. However, fresh data are typically incomplete because of delays in the submission and processing of medical claims. We used documented data lag adjustments.21 To characterize lag times, each Data Partner quantified claims data accrual in early 2012 by week after care date for each medical setting. For the current‐vs.‐historical (Poisson and CmaxSPRT) analysis, we multiplied the expected by the fraction of data projected to have arrived, according to these data‐lag characterizations, to adjust the expected. For the self‐controlled (binomial maxSPRT) analysis, we excluded from analysis events in the risk and control intervals associated with a vaccination week until data for both intervals were determined to be ≥85% complete, according to the data‐lag characterizations. Among the three Data Partners, the number of weeks needed to achieve ≥85% completeness was 7–13 for the emergency department setting and 10–18 for the inpatient setting.

Signal investigation

To investigate a signal, we conducted logistic regression analysis of IIV vaccinees in 2012–2013 and 2013–2014, adjusting for Data Partner, week of season, age, sex, dose, and season.

Evaluation

An evaluation of the fresh data was conducted, reported on separately.19

Results

The Data Partners provided cumulative refreshed data 3–4 times each, on a staggered schedule. One to two sequential analyses were conducted each month between December 2013 and May 2014, each analysis incorporating new data from one Data Partner. Figure S1 in the Supporting Information shows the sequence of tests conducted. Analyses were routinely conducted by approximately 6 weeks after the last care date in the respective batch of data.19

A total of 6 682 336 doses of IIV and 782 125 doses of LAIV had been captured by the end of surveillance (Figure S2 in the Supporting Information). The proportion contributed by immunization registries was 4.3%.

Current‐vs.‐historical design: A statistical signal appeared for seizures in 6‐ to 23‐month‐olds receiving IIV with concomitant PCV13 in Test #7, conducted in March 2014. There were nine cases observed among 86 329 concomitant vaccinees, a relative risk (RR) of 3.0, and a LLR of 3.978, surpassing the critical value. By Test #10, now with 12 cases observed among 116 133 concomitant vaccinees, the RR had decreased slightly to 2.7 (Table 2 and Figure 1). There were no other signals for IIV, nor any for LAIV. Fifteen cases of anaphylaxis after IIV had appeared by Test #10, compared with 23.7 expected, for a RR of 0.63; there were 0 cases after LAIV.

Table 2

Sequential analysis results

			Current vs. historical							Self‐controlled risk interval
Vaccine	Outcome	Risk interval (days)	Cum. doses	Cum. events observed in risk interval (current)	Cum. events expected in risk interval (historical)	RR (current vs. expected)	Log‐likelihood ratio (LLR)*, †	Critical value of LLR	Sequential signal?	Control interval (days)	Cum. events in risk interval	Cum. events in control interval	RR (risk interval vs. control interval)	Log‐likelihood ratio (LLR)*, ‡	Critical value of LLR	Sequential signal?
Analysis #1, mid‐December 2013
IIV	Anaphylaxis, ≥6 months	0–1	223 794	0	0.17	0			No
	Seizures, 6–23 months, with PCV13	0–1	3 877	0	0.02	0			No	14–20						No
	Seizures, 6–23 months, without PCV13	0–1	10 271	0	0.06	0			No	14–20						No
	Seizures, 24–59 months	0–1	10 375	1	0.06	17.66			No	14–20						No
LAIV	Anaphylaxis, 2–49 years	0–1	43 374	0	0.03	0			No
	Seizures, 24–59 months§	1–3	10 780	0	0.11	0			No
	Seizures, 24–59 months¶	1–3	10 780	0	0.14	0			No	15–20						No
Analysis #2, mid‐December 2013
IIV	Anaphylaxis, ≥6 months	0–1	420 459	0	0.19	0			No
	Seizures, 6–23 months, with PCV13	0–1	7 088	0	0.03	0			No	14–20						No
	Seizures, 6–23 months, without PCV13	0–1	18 965	0	0.08	0			No	14–20						No
	Seizures, 24–59 months	0–1	18 890	1	0.06	16.45			No	14–20						No
LAIV	Anaphylaxis, 2–49 years	0–1	82 488	0	0.04	0			No
	Seizures, 24–59 months§	1–3	20 229	0	0.12	0			No
	Seizures, 24–59 months¶	1–3	20 229	0	0.16	0			No	15–20						No
Analysis #3, mid‐December 2013
IIV	Anaphylaxis, ≥6 months	0–1	558 879	0	0.48	0			No
	Seizures, 6–23 months, with PCV13	0–1	7 089	0	0.03	0			No	14–20						No
	Seizures, 6–23 months, without PCV13	0–1	18 968	0	0.08	0			No	14–20						No
	Seizures, 24–59 months	0–1	18 897	1	0.06	16.42			No	14–20						No
LAIV	Anaphylaxis, 2–49 years	0–1	82 731	0	0.04	0			No
	Seizures, 24–59 months§	1–3	20 234	0	0.12	0			No
	Seizures, 24–59 months¶	1–3	20 234	0	0.16	0			No	15–20						No
Analysis #4, early January 2014
IIV	Anaphylaxis, ≥6 months	0–1	2 225 659	2	6.75	0.30			No
	Seizures, 6–23 months, with PCV13	0–1	33 064	1	0.90	1.11			No	14–20	0	3	0			No
	Seizures, 6–23 months, without PCV13	0–1	112 420	1	2.97	0.34			No	14–20	0	1	0			No
	Seizures, 24–59 months	0–1	117 299	5	2.46	2.03	0.798	2.829	No	14–20	1	1	3.50			No
LAIV	Anaphylaxis, 2–49 years	0–1	323 939	0	0.88	0			No
	Seizures, 24–59 months§	1–3	74 106	0	2.32	0			No
	Seizures, 24–59 months¶	1–3	74 106	0	3.14	0			No	15–20	0	1	0			No
Analysis #5, mid‐January 2014
IIV	Anaphylaxis, ≥6 months	0–1	2 555 486	4	7.48	0.53	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	33 072	1	0.90	1.11			No	14–20	0	3	0			No
	Seizures, 6–23 months, without PCV13	0–1	112 457	1	2.97	0.34			No	14–20	0	1	0			No
	Seizures, 24–59 months	0–1	117 356	5	2.46	2.03	0.797	2.829	No	14–20	1	1	3.50			No
LAIV	Anaphylaxis, 2–49 years	0–1	324 471	0	0.88	0			No
	Seizures, 24–59 months§	1–3	74 142	0	2.32	0			No
	Seizures, 24–59 months¶	1–3	74 142	0	3.15	0			No	15–20	0	1	0			No
Analysis #6, mid‐February 2014
IIV	Anaphylaxis, ≥6 months	0–1	4 952 572	10	14.27	0.70	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	67 832	6	2.24	2.68	2.158	2.874	No	14–20	0	3	0			No
	Seizures, 6–23 months, without PCV13	0–1	233 728	4	7.76	0.52	0	3.231	No	14–20	0	2	0			No
	Seizures, 24–59 months	0–1	231 934	9	3.71	2.42	1.879	2.829	No	14–20	1	2	1.75			No
LAIV	Anaphylaxis, 2–49 years	0–1	635 920	0	1.66	0.00			No
	Seizures, 24–59 months§	1–3	139 857	1	3.42	0.29			No
	Seizures, 24–59 months¶	1–3	139 857	1	5.04	0.20			No	15–20	0	1	0			No
Analysis #7, early March 2014
IIV	Anaphylaxis, ≥6 months	0–1	5 573 643	11	17.83	0.62	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	86 329	9	2.96	3.05	3.978	2.874	Yes	14–20	2	5	1.40	0.077	2.850	No
	Seizures, 6–23 months, without PCV13	0–1	282 686	6	9.95	0.60	0	3.231	No	14–20	2	19	0.37	0	3.247	No
	Seizures, 24–59 months	0–1	268 441	9	5.12	1.76	0.789	2.829	No	14–20	5	13	1.35	0.152	2.953	No
LAIV	Anaphylaxis, 2–49 years	0–1	704 460	0	2.07	0.00			No
	Seizures, 24–59 months§	1–3	153 893	2	4.40	0.45			No
	Seizures, 24–59 months¶	1–3	153 893	2	6.35	0.32			No	15–20	0	5	0	0	2.904	No
Analysis #8, late March 2014
IIV	Anaphylaxis, ≥6 months	0–1	5 757 300	13	18.55	0.70	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	86 502	9	2.96	3.04	3.964	2.874	Yes	14–20	2	5	1.40	0.077	2.850	No
	Seizures, 6–23 months, without PCV13	0–1	283 514	6	9.99	0.60	0	3.231	No	14–20	2	19	0.37	0	3.247	No
	Seizures, 24–59 months	0–1	270 072	9	5.14	1.75	0.775	2.829	No	14–20	5	13	1.35	0.152	2.953	No
LAIV	Anaphylaxis, 2–49 years	0–1	705 730	0	2.07	0.00			No
	Seizures, 24–59 months§	1–3	154 258	2	4.41	0.45			No
	Seizures, 24–59 months¶	1–3	154 258	2	6.36	0.31			No	15–20	0	5	0	0	2.904	No
Analysis #9, early April 2014
IIV	Anaphylaxis, ≥6 months	0–1	6 296 295	14	21.41	0.65	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	104 075	12	3.94	3.04	5.301	2.874	Yes	14–20	2	8	0.88	0	2.850	No
	Seizures, 6–23 months, without PCV13	0–1	322 972	7	12.68	0.55	0	3.231	No	14–20	4	35	0.40	0	3.247	No
	Seizures, 24–59 months	0–1	299 866	10	5.70	1.75	0.835	2.829	No	14–20	8	25	1.12	0.038	2.953	No
LAIV	Anaphylaxis, 2–49 years	0–1	755 894	0	2.37	0.00			No
	Seizures, 24–59 months§	1–3	164 263	3	4.79	0.63	0	2.952	No
	Seizures, 24–59 months¶	1–3	164 263	3	7.02	0.43	0	2.972	No	15–20	2	6	0.67	0	2.904	No
Analysis #10, late May 2014
IIV	Anaphylaxis, ≥6 months	0–1	6 682 336	15	23.71	0.63	0	3.371	No
	Seizures, 6–23 months, with PCV13	0–1	116 133	12	4.47	2.69	4.326	2.874	Yes	14–20	4	10	1.40	0.154	2.850	No
	Seizures, 6–23 months, without PCV13	0–1	349 628	8	13.91	0.57	0	3.231	No	14–20	5	39	0.45	0	3.247	No
	Seizures, 24–59 months	0–1	318 239	10	6.56	1.52	0.474	2.829	No	14–20	8	26	1.08	0.017	2.953	No
LAIV	Anaphylaxis, 2–49 years	0–1	782 125	0	2.55	0			No
	Seizures, 24–59 months§	1–3	169 089	3	5.22	0.57	0	2.952	No
	Seizures, 24–59 months¶	1–3	169 089	3	7.59	0.40	0	2.972	No	15–20	3	9	0.67	0	2.904	No

Log likelihood ratio set to 0 where RR < 1.

Number of cumulative events observed must be ≥3 for analysis to be conducted.

Number of cumulative events observed in both intervals must be ≥4 for analysis to be conducted.

Historical rates used are post‐IIV.

Historical rates used are post‐LAIV.

Figure 1

Trajectories of IIV doses, seizure relative risks (RRs), and log likelihood ratios (LLRs) for 6‐ to 23‐month‐olds with concomitant PCV13 over the course of sequential analysis. The LLR was set to 0 where RR < 1. CvsH refers to the current vs. historical analysis, SCRI to the self‐controlled risk interval analysis. The LLR critical values for the two analysis methods were too close to distinguish from each other; the horizontal purple line represents the LLR critical values (CV) of both. A statistical signal emerged from the current vs. historical analysis in Test #7, and the RR was 2.7 by Test #10. No statistical signal appeared with the SCRI analysis; the RR at Test #10 was 1.4

SCRI design: For each stratum, data in the control window had to be ≥85% complete before any cases could be analyzed. This, together with the pre‐specified minimum of four cases in risk plus control windows in order to do an analysis,19 meant that no SCRI analysis was possible until Test #7, conducted after most of the influenza vaccine for the season had been administered. No statistical signals emerged in SCRI analysis. Regarding seizures in 6‐ to 23‐month‐olds receiving concomitant IIV and PCV13, in the last SCRI analysis, there were four cases in the risk interval, 10 in the control interval, a RR of 1.4, and a LLR well below the signaling threshold (Table 2 and Figure 1).

In investigating the seizures signal, we found that 6‐ to 23‐month‐old children receiving concomitant IIV and PCV13 had a greater risk of seizures in the 0–1 days following vaccination compared with those receiving IIV without concomitant PCV13, with an adjusted OR of 3.1 (95% CI: 1.7, 5.9; p = 0.0004).

Discussion

This surveillance effort demonstrates the feasibility of conducting vaccine safety surveillance for important public health outcomes using healthcare data as recent as 6 weeks old. These fresh data, which could be updated on a monthly basis, allow more frequent statistical testing and faster signal detection. The freshness and updating frequency of these fresh data streams hold promise for monitoring the safety of products whose evaluation could substantially benefit from a 6‐ to 9‐month lead‐time over standard Sentinel data.

The statistical signal for seizures in 6‐ to 23‐month‐old children after IIV and concomitant PCV13 vaccination seen in the secondary analysis merits further investigation. The comparison group was IIV vaccinees (largely without concomitant PCV13) in prior seasons. No statistical signal was seen in the primary, SCRI analysis, which compared the risk between exposed and unexposed time from the same concomitant vaccinees. Although our signal investigation found that 6‐ to 23‐month‐old concomitant vaccinees had a greater risk of post‐vaccination seizures compared to those receiving IIV without PCV13, this was not a self‐controlled analysis and thus was subject to potential residual confounding. Moreover, the analysis was not designed to examine the effect of PCV13 vaccination by itself.

Indications have emerged of a possible increased risk of febrile seizures after IIV vaccination in young children in the U.S. in some prior seasons.13, 22, 23 Given the usual annual change in influenza vaccine antigenic composition, the relevance of results from earlier seasons to our finding of 2013–2014 is unclear. PRISM did not find a statistically significant elevated risk for febrile seizures in the 2010–2011 season.12 VSD found an increased risk of seizure after IIV in 2010–201113 and 2011–201223 (in which seasons the antigenic composition of the vaccine was the same) but not in 2012–201323 or 2013–2014.24 In 2013–2014, unlike PRISM, VSD did not stratify the exposure for the 6‐ to 23‐month‐olds into IIV with and IIV without concomitant PCV13. This difference may explain the apparently different findings between the two systems that season—indeed, if we pool our 6‐ to 23‐month‐old IIV vaccinees (i.e. without regard to concomitant PCV13), the number of observed cases is 20 vs. 18.38 expected, for a RR of 1.09 (derived from Table 2, Analysis #10). (In the future, when the safety of the same vaccine is to be monitored by more than one system, it would be worthwhile to harmonize certain features of the methods, such as exposure and health outcome definitions, so as to facilitate comparison and interpretation of the results. Although a case could be made for combining data to maximize statistical power, comparing results from different observational studies and designs can be informative, after which meta‐analysis can be employed to obtain a composite result.25)

The independent risks of IIV and PCV13 with respect to seizures will be examined in a separate study,26 using the SCRI design retrospectively on mature data and implementing adjustments similar to those used in the PRISM study of 2010–2011 IIV.12

Although the signal did not appear until Analysis 7, conducted on 10 March 2014,19 it is worth considering if and when the system would have signaled under circumstances of a true increased risk of the magnitude found for Fluvax and Fluvax Junior in Australia in 2010. The ratio of observed to expected in that instance was approximately 9.27 This is an important example, because the risk identified was sufficiently high to result in changes to Advisory Committee on Immunization Practices recommendations for the U.S.‐equivalent of the Fluvax vaccine (Afluria, CSL Limited)27 and to lead to label changes mentioning the seizures risk and restricting the FDA‐approved usage of Afluria to children aged 5 years or older.28 Through simulations, we found that, if the true RR of seizures among 6‐ to 23‐month‐olds receiving concomitant IIV and PCV13 vaccines had been 9, as it was for Fluvax, the probability of seeing a signal would have been 90% by Analysis 4 (Figure 2), which was conducted on 9 January 2014.19 By Analysis 6, conducted on 18 February 2014,19 the power to see a signal in the case of a true RR as low as 4 was almost 80%.

Figure 2

Probability of signaling with the Poisson maxSPRT for a given relative risk at a given look, using the actual expected counts for the 6‐ to 23‐month‐old IIV+PCV13 concomitant vaccinees in 2013–2014 and a required minimum‐number‐of‐observed‐cases‐to‐signal of 3. The midpoint of the color scale, yellow, was set to correspond to 0.50, such that values >0.50 are greenish and values <0.50 are reddish. * The expected count at each look was based on Data Partner‐specific background rates and the cumulative number of IIV doses administered concomitantly with PCV13 to 6‐ to 23‐month‐olds as of that look, with data lag adjustment applied.

Instead of restricting ourselves to the relatively small group of 6‐ to 23‐month‐olds receiving PCV13 concomitantly with IIV in which our signal occurred, we could consider all 6‐ to 23‐month‐olds, summing the expected counts for the children with IIV with and without concomitant PCV13 and repeating the simulations. As can be seen in the resulting Figure 3, the probability of detecting a signal for a given true RR was higher at earlier looks than in Figure 2. For example, by Look 4 the probability of detecting a signal in the case of a true RR as low as 4 was 90%.

Figure 3

Probability of signaling with the Poisson conditional maxSPRT for a given relative risk at a given look, using the expected counts for all 6‐ to 23‐month‐old IIV vaccinees in 2013–2014 and a required minimum‐number‐of‐observed‐cases‐to‐signal of 3. The midpoint of the color scale, yellow, was set to correspond to 0.50, such that values >0.50 are greenish and values <0.50 are reddish. * The expected count at each look was based on Data Partner‐specific background rates and the cumulative number of IIV doses administered to all 6‐ to 23‐month‐old IIV vaccinees as of that look, with data lag adjustment applied.

There are some significant limitations to conducting surveillance using fresh, frequently updated data, particularly for influenza vaccine safety monitoring. One is cost. Processing, quality‐checking, and analyzing fresh data were resource‐intensive for both Data Partners and coordinating center because of the non‐routine nature of the work and the frequency of data‐processing. To institutionalize such a system would require developing new infrastructure capabilities. Doing so would lower the incremental cost of monitoring additional products. Another problem is the unpredictability of events affecting data quality and timeliness. Although two serious data quality problems were found and resolved during the 2012–2013 pilot season, there were a number of system‐wide changes or other events at the Data Partners in the 2013–2014 surveillance season,19 which, if not noticed and addressed, would have affected data quality and which did lead to delays in data provision. Related to the dynamic, unsettled nature of fresh data, using such data requires particularly careful scrutiny of statistical signals. The comparison of fresh vs. mature data for the 2012–2013 season identified some differences in dose counts, case counts, and risk estimates between the two data types.19 Risk estimates were more divergent between fresh and mature data for SCRI analyses than for current‐vs.‐historical analyses. These differences between results from fresh and mature data were likely related to low case counts in the SCRI analysis and consequent instability of risk estimates. Inaccuracies in the lag adjustment might have been a factor, too, considering that the lag characterization was necessarily conducted on older data, and timeliness of claims data‐processing may have changed.

Finally, suboptimal statistical power and time to signal can be a concern, especially for influenza vaccine safety surveillance, where the period of vaccine administration is so compressed. Even with the more timely current‐vs.‐historical analysis, the statistical signal for seizures in 6‐ to 23‐month‐old concomitant IIV and PCV13 vaccinees was not discovered until March 2014, after the influenza season was essentially over. Statistical power and time‐to‐signal would be more favorable if the sample size were larger, achievable with a larger surveillance system (more data partners), longer surveillance, a wider age range, or a more common outcome. Longer surveillance seems a particularly feasible option where products other than influenza vaccine are concerned.

Notwithstanding concerns about statistical power, sequential surveillance using fresh data may be valuable to public health agencies, not necessarily because the system can identify the smallest risks early (which it may not be able to) but because it can function as a safety net to ensure that the largest and most clinically significant health risks be detected as early as possible. The potential for detecting clinically significant risks in a timely fashion, even in a quite small age group, is illustrated by Figures 2 and 3. The true advantage of fresh data is best viewed by comparing a surveillance system that can detect safety concerns at levels that might impact policy just months after initial product uptake with a system using mature data that must wait a year or more to assess the safety of a product.

Conflict of Interest

The authors declare no conflict of interest.

Although the FDA Sentinel System routinely uses mature data for its medical product safety surveillance, it can obtain and analyze “fresher” data as recent as 6 weeks after the last care date in the batch.

Sequential analysis of successive, cumulative batches of fresh data offers the potential for detecting clinically significant risks in a more timely fashion than with mature data, even in smaller demographic subgroups. This holds promise for monitoring the safety of influenza vaccines and other medical countermeasures.

Limitations of employing fresh data for near real‐time surveillance include the costs associated with frequently updating and checking the quality of the fresh data and the need for careful scrutiny of signals.

Ethics Statement

This project was conducted under FDA's public health authority and as such was exempt from IRB review.

Supporting info item

Click here for additional data file.