Pandemrix™ and narcolepsy: A critical appraisal of the observational studies.

A link between Pandemrix™ (AS03-adjuvanted H1N1 pandemic influenza vaccine, GSK Vaccines, Belgium) and narcolepsy was first suspected in 2010 in Sweden and Finland following a number of reports in children and adolescents. Initial scepticism about the reported association faded as additional countries reported similar findings, leading several regulatory authorities to restrict the use of Pandemrix™. The authors acknowledge that currently available data suggest an increased risk of narcolepsy following vaccination with Pandemrix™; however, from an epidemiologist's perspective, significant methodological limitations of the studies have not been fully addressed and raise questions about the reported risk estimates. We review the most important biases and confounders that potentially occurred in 12 European studies of the observed association between Pandemrix™ and narcolepsy, and call for further analyses and debate.

Introduction

In April 2009 the World Health Organization declared an influenza pandemic caused by a novel H1N1 strain and appealed for accelerated vaccine development. In Europe, the resulting H1N1-vaccine coverage ranged between 0.4%–59% for the entire population, and 0.2%–74% for children. Of the approximately 40 million persons vaccinated, over 30 million received Pandemrix™.

An increase in narcolepsy cases was observed in Finland and Sweden toward the end of the 2009 pandemic. Preliminary investigations suggested a temporal link to Pandemrix™, the only pandemic vaccine used in these 2 countries. This led to numerous observational studies at country level, and a large multi-country case-control study in Europe (). The relative risk estimates of the association between Pandemrix™ and narcolepsy ranged in children from 1.5–25.0 with confidence intervals (CIs) from 0.3–48.5, and in adults from 1.1–18.8, with CIs from 0.6–207.4 ().

Table 1.

Summary of the design of 12 publically available studies assessing an association between pandemic AS03-adjuvanted H1N1 vaccination and narcolepsy

		Population			Case ascertainment		Vaccine ascertainment
Study	Design	Geographic origin (period)	Size	Age	Source	Validation	Source	Coverage
MPA-registry cohort, Sweden8	RC	7 counties (2009–2011)	5.8 M	All	Contact with hospitals and sleep labs, spontaneous reports	No expert review	Regional vaccination registries	60%
MPA case-inventory, Sweden5	RC	Nationwide (2009–2010)		All	Registers on hospitalisation and specialist care	By 2 experts in neurology/sleep disorders	Regional vaccination registries	60%
Stockholm county cohort, Sweden16	RC	Stockholm county (1998–2010)	2 M	< 20 (for narcolepsy)	Hospital registers, child rehabilitation, neurophysiology centers	No expert review	Local vaccination registry (Vaccinera)	52.6%
Western Sweden cohort11	RC	Western Swedish health care region (2000–2010)	0.4 M	2–17 yrs	National and local hospital registers, register 3 specialized centers	No expert review	Unclear
Finnish childhood cohort24	RC	Nationwide (2009–2010)	0.9 M	4–19 yrs	National hospital registers	By 2 narcolepsy experts. Discrepancies adjudicated by a narcolepsy expert panel	Electronic primary health care databases	75%
Finnish adult cohort25	RC	Nationwide (2009–2011)	3.3 M	Adults	National hospital registers + direct contact pediatric neurologists	By 2 narcolepsy experts. Discrepancies adjudicated by a narcolepsy expert panel	Electronic primary health care databases	48%
Finnish case series10	Eco	Nationwide (2002–2010)		All	National care register + direct contact health care professionals	By 5 experts in neurology/sleep disorders	Vaccine certificates
Irish cohort26	RC	Nationwide (2009–2010)	4.2 M	4–19 yrs, ≥ 20 yrs	Direct contact sleep and pediatric neurology centers	By an adult and pediatric neurologist	Reimbursement database and mass vaccination database	22.5%
English case-coverage9	CCo	Nationwide (2008–2011)	9.1 M	4–18 yrs	Direct contact sleep centers	By 3 narcolepsy experts	GP questionnaires	1.9%*
French case-control13	CC	Nationwide (2009–2011)	65 M	All	Direct contact sleep centers	By 2 narcolepsy experts	Telephone interviews	6.3%**
VAESCO EU multi-country3 Denmark, France, Italy, the Netherlands, Norway & UK: non-signaling + Sweden & Finland	CC	Nationwide or regional, (April 2009-June 2010)	30 M	All	Varied by country - registers, direct contact with sleep centers	Country dependent	Variety of methods	Very low to high
Norwegian cohort27	RC	Nationwide (120 weeks from 2009 onwards)	1 M	<20 yrs	Medical institutions and practitioners	By a pediatrician and expert in sleep disorders	National vaccination register	50%

MPA = Medical products Agency, M = millions, RC = retrospective cohort. CS = case series, CC = case control, CCo = case coverage, Eco = ecological study,

37% in the 2–15 y old risk group and includes some use of unadjuvanted vaccines in pregnant women and young infants,

mostly 9 y of age and older.

Figure 1.

Risk estimates and 95% confidence intervals for Pandemrix™ vaccination and narcolepsy.

When faced with such a safety signal, vaccine manufacturers will typically rely upon internal and external expertise to critically assess any studies that may influence the benefit-risk profile of the marketed product. As epidemiologists employed or sub-contracted by the manufacturer, the authors have identified a number of potential pitfalls that we believe have not necessarily been highlighted or discussed in detail in the published studies describing risk estimates of narcolepsy following vaccination with Pandemrix™. Our intent is to flag those potential pitfalls with an eye to future research into similar vaccine safety signals for rare or complex outcomes such as neurological/immune-mediated diseases. The objective of this review is therefore not to endorse or refute the observed association.

What are the Limitations of These Studies?

Kleinbaum et al distinguish 3 major sources of error in epidemiological research: information bias (the main concerns here being ascertainment bias and recall bias), selection bias and confounding. In the studies presented here, all of these sources of errors might have occurred to varying degrees ().

Table 2.

Summary of main potential sources of error

Study	Weaknesses	Possible source of error21
MPA-registry cohort8	No validation of cases	Ascertainment bias
	Unclear models and adjustments	Confounding
	Clear degree of residual bias present	Confounding
	Role media attention not addressed	Ascertainment bias
MPA case-inventory5	Inclusion of spontaneous reports	Selection bias
	Blinding undefined	Ascertainment bias
	Extrapolation of regional vaccination coverage data	Confounding
	Unclear models and adjustments	Confounding
Stockholm county cohort16	Blinding undefined	Ascertainment bias
	No validation of cases	Ascertainment bias
	Low power	Confounding
	Role media attention not addressed	Ascertainment bias
Western Sweden cohort11	Unclear index date	Information bias
	Uncertain validation of cases	Ascertainment bias
	Historical comparator	Confounding
	Unclear source for vaccination history	Recall bias
Finnish childhood cohort24	Potential impact of medical/media attention	Ascertainment bias
	No control for potential confounders	Confounding
	Blinding undefined	Ascertainment bias
Finnish adult cohort25	Potential impact of medical/media attention	Information bias
	Uncertain validation of vaccination	Information bias
	No adjustment for confounders	Confounding
	Blinding undefined	Information bias
Finnish case series10	Ecological comparison of incidence rates	Confounding
	Unclear source of symptom onset	Recall bias
	Blinding undefined	Ascertainment bias
	Unclear role of testing as part of the study	Ascertainment bias
Irish cohort26	Case findings through direct contacts with potential bias toward inclusion vaccinated cases	Ascertainment bias
	Vaccination information potentially incomplete	Information bias
	Role media attention uncertain	Ascertainment bias
	No control for other confounders such as risk status	Confounding
English case-coverage9	Case findings through direct contacts with potential bias toward inclusion vaccinated cases	Ascertainment bias
	Low Vaccination coverage	Confounding
	Comparability source cases and controls uncertain	Selection bias
	Study period includes period high media attention	Ascertainment bias
French case-control13	Participation bias	Selection bias
	Potential bias toward inclusion vaccinated cases	Information bias
	High proportion of HCP among controls	Selection bias
	Vaccination status ascertained through interviews	Recall bias
	Blinding undefined	Ascertainment bias
VAESCO EU multi-country3	Heterogeneity in methods	Selection bias
	Low vaccination coverage	Confounding
	Blinding not defined for some countries	Ascertainment bias
	Recruitment via direct contact with sleep centers	Selection bias
	Vaccination status ascertained through interviews	Recall bias
	Limited adjustment for confounders	Confounding
Norwegian cohort27	Incomplete capture vaccine register	Information bias
	Potential bias toward inclusion vaccinated cases	Ascertainment bias
	Self-reported recall onset symptoms	Recall bias
	Blinding undefined	Ascertainment bias
	No control for potential confounders	Confounding

HCP = healthcare personnel.

Ascertainment bias

Ascertainment bias would have occurred if narcolepsy cases were more likely to be classified as cases if vaccinated. Such bias could have arisen at each step in the progression from symptoms to diagnosis (seeking care, being referred, undergoing sleep tests, and finally being diagnosed).

Vaccinated patients may have been more likely to seek care earlier if they were aware of the reported association between Pandemrix™ and narcolepsy, such as through media attention. Data from several studies suggest that biased healthcare seeking behavior occurred. In the Swedish MPA-registry study for example, a decrease in the risk estimate was reported when analyses included additional cases from a more recent registry release (RR of 4.2 versus 2.9 after an additional year of follow-up). This decrease was a likely consequence of more unvaccinated cases being diagnosed and captured in the updated registers. Likewise, in the English case-coverage study, a large increase in the number of unvaccinated cases was seen when the study period was extended, compared to a minimal change in the number of vaccinated cases. This reduced the risk estimate from 22.2 to 11.0.

The referral pattern of primary healthcare providers may have been influenced by heightened disease awareness and knowledge of the vaccination status of the presenting patients. This would result in a shorter time interval from symptom onset to diagnosis among the vaccinated compared to the unvaccinated. Such difference was observed in most studies, with the time-to-diagnosis up to 5–6 times shorter among the vaccinated in the Western Sweden cohort and Finnish case-series. The shortened time-to-diagnosis could also be explained by a more severe clinical presentation in vaccinated patients. However, the comparison of other disease characteristics, such as hypocretin levels or sleep latency test results, does not support the notion of a different clinical presentation among vaccinated patients.

At the referral center, patients may have been managed differentially based on vaccination status. Illustrative for this is the difference in rates of hypocretin testing between vaccinated (59%) and unvaccinated (only 17%) cases, as reported in the French study. Finally, the classification of a patient (at the referral center) as having narcolepsy could be differential based on vaccination status. Evidence of such differential misclassification can be assessed in studies where experts reviewed the reported cases. A differential misclassification would lead to more vaccinated patients being falsely labeled positive at the referral center and thus a relatively high vaccination rate among the cases classified as non-cases by expert review. In the MPA case-inventory study, the proportion of vaccinated among the rejected cases was 78% (14/18) compared to a national coverage of 63% in the same age group. Differential validation of vaccinated cases is avoidable by blinding the validating experts to vaccination status. Such blinding did not occur or was not explicitly reported in most studies ().

Recall bias

The onset of symptoms relied on patient recall in many studies and was thus prone to recall bias. Given the media attention that occurred before most studies took place, it is plausible that onset of symptoms was preferentially linked to the onset of the pandemic and the associated vaccination campaigns. While recall bias is difficult to prove including in the studies considered, its existence in other vaccine safety studies has been previously highlighted.

Selection bias

Selection bias resulting in falsely increased risk estimates would have occurred if vaccinated cases or unvaccinated controls were preferentially enrolled. In the MPA case-inventory study, cases reported to the spontaneous reporting system were included. These cases were by definition vaccinated, and their inclusion may have skewed the results toward falsely inflated risk estimates in this group. The French case-control study relied upon a selection of controls that differed from cases in some important aspects, such as the proportion of healthcare professionals, a group targeted for vaccination. In the English case-coverage study, the controls were drawn from an independent subset of the general population, with limited information allowing no matching and minimal adjustment for potential confounders.

Confounding

The most important confounders in the studies of Pandemrix™ and narcolepsy are confounding by indication and confounding by natural H1N1 infection. Confounding by indication would have occurred if the indication for which H1N1 vaccination was recommended also carried an increased risk to develop narcolepsy. Although influenza risk factors are not known to be linked with higher narcolepsy risk, an elevated (non-significant) odds ratio of 3.53 for H1N1 vaccination in the first 45 d of the campaign was found for subjects with prevalent narcolepsy in the Stockholm county-cohort study, suggesting confounding by indication. In the English case-coverage study, matching by risk group reduced the odds ratios nearly two-fold, a further illustration that would support potential confounding. Few other studies had the possibility to adjust for this confounder.

The timing of vaccination campaigns and epidemics had a near-perfect match in most European countries. Natural infection could have acted as a confounder if individuals infected by H1N1 virus were more prone to seek care and be targeted for vaccination. The observed association could thus incorrectly be attributed to the vaccine instead of the viral infection itself. The strong temporal correlation between the incidence of narcolepsy and the H1N1 pandemic wave observed in China suggests such a confounding effect is plausible. A recent attempt to test for past H1N1 infection among vaccinated narcolepsy cases did not find a higher exposure rate among narcolepsy cases. However, the approach used to establish evidence of past infection is not validated and debatable.

Additional potential sources of confounding are numerous and include healthcare seeking behavior, socio-economic status, ethnic background and frailty in general. The MPA-registry cohort study showed that the vaccinated cohort had a higher number of ambulatory care visits and hospitalisations prior to the study start, illustrating the potential confounding by healthcare seeking behavior. In the Stockholm county cohort study, adjustment for healthcare utilization decreased the risk estimates for nearly all outcomes, including narcolepsy. The MPA-registry study showed that vaccinees had a higher income level and were more likely to be born in Nordic countries. The link between these determinants and the risk of having/being diagnosed with narcolepsy is obvious for some parameters (Nordic origin is associated with higher levels of the HLA allele carriage) and cannot be excluded for the others.

What Could Have Been Done Differently (Or Can Still be Done)?

Most studies were pragmatic in nature, taking advantage of pre-existing datasets such as registries, and combining data thereof with vaccination data from different sources into a cohort or case-coverage design. As a result, there was no systematic collection of comparable data across the comparator groups and therefore minimal opportunity to control for confounding factors. While these studies may have been the most efficient and rapid means to analyze and report the available information, few of their limitations were thoroughly addressed. Beyond varying index dates and observation period, no systematic assessment of other potential biases such as those listed above was performed and certainly no integrated analyses of all these biases combined were performed. Performing such analyses is increasingly recognized as good practice in pharmaco-epidemiological research, particularly in studies with such far-reaching public health implications.

Alternative methods to analyze the data could also have been considered, such as the self-controlled case series (SCCS) or case-negative designs. The SCCS implicitly controls for fixed confounders such as healthcare seeking behavior and confounding by indication, but it cannot control for time-dependent covariates such as infection and is suboptimal for assessment of chronic onset disease. In the English study, the risk estimates from the SCCS analyses were about ten-fold lower compared to the analyses from the case-coverage study, and were not significant unless the study period was increased. Possibly the most appropriate design may be a test-negative case-control design in which vaccination rates would be compared between cases validated as narcoleptic to subjects suspected for narcolepsy but confirmed not to be narcoleptic after assessment by an expert. This approach would ensure that cases and controls are drawn from a population with comparable propensity to seek care, including vaccination, or be referred for diagnosis. This design has been extensively used in influenza vaccine effectiveness studies using similar arguments.

Summary

In summary, there are limitations to the observational studies of the association between Pandemrix™ and narcolepsy, putting into question whether the relative risks observed in them reflect the true risk associated with Pandemrix™ vaccination. No systematic assessment was done of the potential impact of all potential biases or confounders. The consistency of the findings, as well as the strength of the association have been repeatedly mentioned as arguments toward a true association. But consistency in bias and confounding may also lead to consistently false positive results. While we acknowledge that a single confounder or bias may not explain the risk estimates observed, the combined effect of several confounding factors should not be underestimated. We advocate that researchers engage in a collaborative effort involving all stakeholders (vaccine manufacturers, academia, public health and regulators) to examine the possibility of reanalysing the data using designs that may be less prone to bias, and perform more systematic sensitivity analyses to assess the potential role of these biases. Whether the observed strength of the association will still stand after the use of more appropriate designs and adjustment is an open question. As a minimum, better estimates of the attributable risk will allow for a more informed assessment of benefit-risk.

Key messages

Epidemiological studies suggest an association between Pandemrix™ and narcolepsy. Whether this temporal association can also be interpreted as a causal association is less clear, and should be considered with caution.

The important methodological concerns that apply to a certain extent to all available epidemiological studies are various ascertainment biases, recall bias, selection bias, confounding by indication, and the impossibility to distinguish between exposure to the vaccine and exposure to the virus due to their close temporal proximity.

For each of these potential errors there are indications that they may have affected the risk estimates. A systematic assessment of the potential combined impact of these biases and confounders is needed for informed benefit/risk decision making.

Alternative designs such as the test-negative case-control design can be expected to account for several of the biases and confounders observed.

Disclosure of Potential Conflicts of Interest

TV and KB received consulting fees from GSK for the work reported here. TV, GF and VS are formers employees of GSK group of companies. CC, VS, and VB are GSK employees and own stock options/restricted shares in the company. GDS is a full-time consultant (Business & Decision Life Sciences) on behalf of GSK.