Further evidence for bias in observational studies of influenza vaccine effectiveness: the 2009 influenza A(H1N1) pandemic.

Preinfluenza periods have been used to test for uncontrolled confounding in studies of influenza vaccine effectiveness, but some authors have claimed that confounding differs in preinfluenza and influenza periods. We tested this claim by comparing estimates of the vaccine-mortality association during the 2009/2010 influenza year, when there was essentially no circulation of seasonal influenza in the United States, and 2007/2008, a typical influenza year. We pooled data on seniors (adults aged ≥65 years) from 7 US managed care organizations that participated in the Vaccine Safety Datalink Project. We defined influenza vaccination, all-cause mortality, and potential confounders from administrative databases. We quantified the vaccine-mortality association using Cox regression. During 2007/2008, the adjusted hazard ratio was 0.44 prior to influenza season, 0.62 during influenza season, and 0.71 after influenza season. A similar pattern was observed during 2009/2010, when any effect of seasonal influenza vaccine observed during all time periods must have resulted from confounding: 0.65 during the autumn, 0.80 during the winter, and 0.84 during the summer. In a year with minimal seasonal influenza, we found no evidence that confounding in autumn preinfluenza periods is qualitatively different from confounding in winter. This supports the use of preinfluenza periods as control time periods in studies of influenza vaccine effectiveness.

Influenza vaccine effectiveness (VE) in seniors (adults aged ≥65 years) remains controversial (1–5). Numerous observational studies have estimated that influenza vaccine reduces the risk of all-cause mortality among seniors during the winter months by 40% or more and have concluded that influenza vaccine is highly effective (4, 6–8). In recent years, other researchers have noted that these estimates are implausibly large given the small proportion of winter deaths attributable to influenza (9) and that the apparent VE is perhaps entirely the result of uncontrolled confounding (1, 10).

Strong evidence for confounding comes from studies that have estimated the association between influenza vaccination and risk of death during time periods before, during, and after the seasonal circulation of influenza. Time periods before and after influenza circulates are designated control time periods, during which no true vaccine benefit is expected. Studies including these control time periods have consistently found that vaccinated seniors have a significantly lower risk of death than unvaccinated seniors during all time periods, with the strongest association being observed prior to circulation of influenza viruses (1, 11–14). A highly plausible explanation for this trend is that seniors at high risk of dying are less likely to receive influenza vaccine, leading to a strong but spurious association between vaccination and reduced mortality during the months immediately following vaccination. Over time, the seniors at high risk of mortality (who are predominantly unvaccinated) die. This reduces the underlying differences in mortality risk between vaccinated and unvaccinated seniors, and the spurious association between vaccination and mortality risk attenuates towards the null. Notably, the apparent VE during influenza seasons can be fully explained by the confounding observed in the preinfluenza control time period.

Although incorporation of the preinfluenza control periods is effective and innovative, their use has been criticized by some (3, 15). These authors have claimed that the confounding observed in autumn preinfluenza seasons is qualitatively different from confounding in winter influenza seasons. Specifically, they claim that persons who die prior to influenza season may be less likely to be offered vaccine, or may have fewer opportunities to receive vaccine, than persons who survive until influenza season. On this basis, they infer that confounding observed during preinfluenza seasons is not present when influenza circulates and that preinfluenza periods are not appropriate control time periods for studies of influenza VE. This claim could be tested by estimating the association between seasonal influenza vaccination and risk of mortality over time in a year with no seasonal influenza. If the same trends in apparent associations between vaccination and mortality were observed during a year when there could be no true vaccine effect, the claim that confounding differs between time periods would be refuted.

Circulation of the 2009 influenza A(H1N1) pandemic virus (hereafter called 2009 H1N1pdm) offered us a unique opportunity to examine confounding in studies of seasonal influenza VE. In the 2009/2010 influenza year (September 2009 through August 2010), 2009 H1N1pdm accounted for essentially all (>99%) circulating influenza virus isolates in the United States, and circulation of all influenza viruses was essentially over by late December 2009 (16). Thus, we hypothesized that during the entire 2009/2010 influenza year, no true association between seasonal influenza vaccine and disease outcomes should be found if confounding factors were appropriately adjusted for in multivariate analyses. We estimated the association between influenza vaccination and all-cause mortality among seniors during the 2009/2010 influenza year to see whether the confounding trend seen in prior years with typical influenza circulation remained in a year with no seasonal influenza.

MATERIALS AND METHODS

Study design and population

We conducted a retrospective cohort study among seniors enrolled in managed care organizations (MCOs) participating in the Vaccine Safety Datalink Project, which has been described elsewhere (17). Seven Vaccine Safety Datalink sites participated in the present study: Group Health Cooperative (Seattle, Washington); Kaiser Permanente Northwest (Portland, Oregon); Kaiser Permanente of Northern California (Oakland, California); Kaiser Permanente of Southern California (Los Angeles, California); Kaiser Permanente of Colorado (Denver, Colorado); Marshfield Clinic Research Foundation (Marshfield, Wisconsin); and Health Partners Research Foundation (Minneapolis, Minnesota). This study protocol was approved by institutional review boards at all study sites. Study subjects were enrollees in one of these 7 MCOs who were aged 65 years or older as of September 1, 2009, and who had been continuously enrolled since September 1, 2008. To compare VE estimates in this population with prior VE estimates from nonpandemic years, we also studied a cohort of enrollees who were aged 65 years or older as of September 1, 2007, and who had been continuously enrolled since September 1, 2006.

Exposure, covariates, and outcome

The primary exposure of interest was receipt of the seasonal influenza vaccine. The date of vaccine receipt for study subjects in each cohort was identified from vaccine registries at each of the participating MCOs. We also identified the date of receipt of 2009 H1N1pdm vaccine for the 2009/2010 cohort. We defined covariates on the basis of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes assigned to outpatient and inpatient medical encounters during the baseline period for each cohort. The baseline period for the 2007/2008 cohort was September 1, 2006, through August 31, 2007; for 2009/2010, it was September 1, 2008, through August 31, 2009. We defined 2 sets of covariates. The first set included covariates that have typically been used in observational influenza VE studies among seniors (4, 7, 18). This set consisted of binary indicators for the presence or absence of diagnosis codes for chronic conditions in broadly defined categories (Appendix Table 1).

Adjusting for this “traditional” set of variables has been shown to have a negligible impact on confounding in observational VE studies (1, 19), in part because the conditions defined within each binary variable are overly broad (10). We also explored whether confounding could be reduced by adjusting for a second “alternate” set of more precise covariates. In addition to covariates based on ICD-9-CM codes, the alternate comorbid conditions included indicators for filled prescriptions for bronchodilators, corticosteroids, and statins (Appendix Table 1), based on pharmacy data from each MCO, since adjusting for prescription data has been shown to attenuate confounding in studies of influenza VE against all-cause pneumonia (19).

Subjects were followed for 1 year from the date of study enrollment (September 1 of 2007 or 2009). The outcome of interest was all-cause mortality. Subjects were followed until death, disenrollment from the MCO, or August 31 of 2008 or 2010 (respectively), whichever was earliest.

Analysis

We used Cox proportional hazards regression to estimate the hazard ratio for the association between receipt of seasonal influenza vaccine and incidence of all-cause mortality. Influenza vaccination was treated as a time-varying covariate, with individuals being classified as vaccinated 14 days after receipt of vaccine. We fitted 3 separate models for each year. In the first model, only age (as a set of categorical variables), sex, and MCO were included as model covariates. In the second model, we adjusted for age, sex, MCO, and the traditional set of covariates. In the third model, we adjusted for age, sex, MCO, and the alternate set of covariates. Hazard ratio estimates were stratified by time by including an interaction term between vaccination and time period. In 2007/2008, we divided time into preinfluenza, influenza, and postinfluenza periods. These time periods were defined at each MCO based on regional influenza surveillance (20). In 2009/2010, we divided time into pandemic (September 1–December 15), winter (December 16–March 31), and summer (April 1–August 31) periods. To further explore the change in hazard ratios over time, we estimated the vaccine-mortality hazard ratio separately for each 2-week period during each study year, adjusting for age, sex, and MCO. For the 2009/2010 influenza year, we repeated all analyses while adjusting for receipt of monovalent 2009 H1N1pdm vaccine as a time-varying covariate.

RESULTS

Population characteristics

The 2007/2008 cohort consisted of 930,059 seniors, of whom 67.0% received the 2007/2008 seasonal influenza vaccine. The 2009/2010 cohort consisted of 984,053 seniors, of whom 63.8% received 2009/2010 seasonal influenza vaccine and 19.2% received monovalent 2009 H1N1pdm vaccine. The proportion of cohort members from each MCO was consistent between the two study years. Vaccinated seniors in both cohorts tended to be older than unvaccinated seniors and were more likely to have ICD-9-CM codes for heart disease, cancer, lung disease, and other chronic illnesses (Table 1). Compared with vaccinated seniors in 2007/2008, seniors receiving the 2009/2010 seasonal influenza vaccine were more likely to have ICD-9-CM codes for chronic diseases, including heart disease, renal disease, lipid disorders, and chronic renal failure. A total of 33,399 deaths (3.6% of the cohort) occurred during the 2007/2008 follow-up period, and 33,239 deaths (3.4% of the cohort) occurred during the 2009/2010 follow-up period. Mortality rates were similar across all 7 MCOs.

Table 1.

Distribution of Covariates (%) According to Proportion of Vaccinated Person-Years Contributed Versus Unvaccinated Person-Years Contributed Among Seniors From 7 US Managed Care Organizations That Participated in the Vaccine Safety Datalink Project During the 2007/2008 and 2009/2010 Influenza Seasons, by Season

Covariate	2007/2008 Influenza Season			2009/2010 Influenza Season
	Total (PY = 896,384)	No Vaccine (PY = 398,551)	Seasonal Vaccine (PY = 497,833)	Total (PY = 954,738)	No Vaccine (PY = 427,970)	Seasonal Vaccine Only (PY = 411,265)	H1N1 Vaccine Only (PY = 77)	Both Seasonal and H1N1 Vaccines (PY = 115,425)
Managed care organization
Kaiser Permanente of Northern California	40.0	39.4	40.5	39.8	38.7	41.3	28.7	38.6
Kaiser Permanente of Colorado	6.4	5.5	7.2	6.5	7.8	5.3	1.2	6.0
Health Partners Research Foundation	3.9	3.7	4.2	3.8	3.8	3.3	16.4	6.0
Marshfield Clinic Research Foundation	2.5	2.8	2.2	2.5	2.7	2.1	12.0	2.9
Kaiser Permanente Northwest	5.2	4.8	5.5	5.2	5.8	3.9	6.3	7.9
Group Health Cooperative	5.7	5.4	5.9	5.5	4.8	5.5	3.3	8.0
Kaiser Permanente of Southern California	36.2	38.4	34.5	36.6	36.4	38.6	32.0	30.5
Age group, years
65–74	56.0	59.5	53.1	56.5	60.0	52.1	65.2	58.8
75–84	33.4	29.9	36.1	32.3	29.2	35.3	25.2	33.2
≥85	10.7	10.5	10.8	11.2	10.8	12.6	9.7	8.1
Sex
Female	56.0	56.4	55.7	55.9	55.9	56.8	51.8	52.5
Male	44.0	43.6	44.3	44.1	44.1	43.2	48.2	47.5
Traditional comorbid conditions
Heart disease	27.1	24.1	29.6	27.8	24.5	30.3	33.3	31.0
Cancer	11.9	10.3	13.1	11.7	10.1	12.7	14.8	13.8
Diabetes	20.9	19.1	22.3	21.7	19.6	23.6	23.9	22.8
Renal disease	13.8	12.5	14.9	18.2	16.1	20.4	20.6	18.6
Lung disease	27.3	24.5	29.7	28.1	24.7	30.7	32.9	31.8
Rheumatological disease	2.3	2.0	2.6	2.5	2.1	2.7	2.6	2.9
Atrial fibrillation	8.0	6.8	8.9	8.5	7.1	9.5	9.3	9.9
Lipid disorders	46.0	41.3	49.8	52.9	47.7	57.1	56.1	57.3
Hypertension	60.3	55.6	64.1	62.1	57.0	66.8	63.8	64.5
Dementia	7.5	8.1	7.0	7.7	7.9	8.1	7.1	5.8
No. of outpatient visits
0	5.3	9.1	2.3	5.2	8.7	2.5	1.3	1.5
1–3	20.2	23.7	17.4	21.3	25.3	18.8	13.4	15.7
4–6	20.6	20.5	20.7	21.0	21.2	21.0	18.8	20.3
7–9	15.4	14.2	16.4	15.3	14.1	16.2	16.5	16.4
≥10	38.4	32.5	43.2	37.2	30.7	41.5	50.0	46.1
No. of pneumonia hospitalizations
0	98.6	98.6	98.6	98.6	98.7	98.4	97.1	98.6
≥1	1.4	1.4	1.4	1.4	1.3	1.5	2.9	1.4
Alternate comorbid conditions
Congestive heart failure	6.7	6.2	7.2	6.6	5.8	7.3	9.4	6.7
Serious/metastatic cancer	2.4	2.2	2.6	2.5	2.1	2.7	3.4	2.8
Diabetes complications	9.9	8.9	10.7	11.8	10.4	13.2	13.3	12.2
Chronic renal failure	11.3	10.1	12.2	15.9	13.9	17.9	18.5	16.2
Asthma	7.0	5.9	7.9	7.7	6.4	8.5	10.8	9.8
Pneumonia	3.6	3.3	3.8	3.5	3.1	3.8	5.9	3.8
Chronic lung disease	19.7	17.5	21.5	21.2	18.5	23.1	25.1	24.3
Inhaled corticosteroid use	2.2	1.7	2.5	2.8	2.2	3.0	5.4	4.0
Oral corticosteroid use	8.7	7.7	9.6	9.1	7.9	10.0	13.3	10.9
Bronchodilator use	9.3	8.2	10.2	11.5	9.8	12.5	15.9	13.9
Statin use	46.0	40.2	50.6	49.2	42.9	54.0	54.4	55.4

Abbreviation: PY, person-years.

Association between vaccine and all-cause mortality

During the 2007/2008 year, a year with typical influenza circulation, receipt of seasonal influenza vaccine was associated with a decreased risk of all-cause mortality after adjusting for age, sex, and MCO (Figure 1A). Consistent with prior studies (1, 11–14), the apparent association was strongest in the preinfluenza time period (hazard ratio = 0.44, 95% confidence interval (CI): 0.42, 0.46), decreasing to 0.62 (95% CI: 0.60, 0.65) during influenza season and to 0.71 (95% CI: 0.69, 0.74) in the summer time period. Also consistent with prior studies, adjusting for the traditional set of comorbidity indicators did not appreciably alter the estimated associations. Adjusting for the alternate comorbidity indicators also did not appreciably alter the estimated associations (Figure 1).

Figure 1.

Association between influenza vaccination and all-cause mortality, by year and season, among seniors from 7 US managed care organizations that participated in the Vaccine Safety Datalink Project during the 2007/2008 influenza year (A) and the 2009/2010 influenza year (B). Hazard ratios were adjusted for age, sex, managed care organization (base model), and different sets of comorbid conditions; horizontal bars represent 95% confidence intervals.

During the 2009/2010 pandemic year, receipt of seasonal influenza vaccine was also associated with a decreased risk of all-cause mortality after adjusting for age, sex, and MCO (Figure 1B). The apparent association was strongest during the pandemic (fall) period (hazard ratio = 0.65, 95% CI: 0.62, 0.68), decreasing to 0.80 (95% CI: 0.77, 0.83) during the winter and 0.84 (95% CI: 0.81, 0.87) during the summer. Adjusting for the traditional or alternate comorbid conditions did not appreciably change the vaccine-mortality association. Adjusting for monovalent 2009 H1N1pdm vaccine also did not appreciably affect the results.

When hazard ratios were estimated biweekly, both study years showed similar patterns over time (Figure 2, parts A and B). After initial instability during weeks when few doses of seasonal influenza vaccine had been given, the hazard ratio trended steadily toward the null over time.

Figure 2.

Biweekly estimates of the association between seasonal influenza vaccination and all-cause mortality among seniors from 7 US managed care organizations that participated in the Vaccine Safety Datalink Project during the 2007/2008 influenza year (A) and the 2009/2010 influenza year (B). Hazard ratios were adjusted for age, sex, and managed care organization, with biweekly vaccine coverage; vertical bars represent 95% confidence intervals. HR, hazard ratio.

DISCUSSION

Our findings refute the claim that confounding in preinfluenza seasons does not operate in influenza or summer seasons (3, 15). If this claim were true, we would have observed a non-null association between influenza vaccination and mortality during the fall of 2009, followed by a null association during the winter and summer of 2009/2010. Instead, we observed a pattern in the vaccine-mortality association over time during the 2009/2010 pandemic year that was similar to the pattern observed in typical influenza seasons (1, 11, 14), including the 2007/2008 season, which we examined in the current study. Specifically, the estimated association was highest during the preinfluenza season and trended steadily toward the null over time, but it remaining significantly protective in all time periods. Our findings are consistent with our hypothesis about confounding in observational VE studies among seniors: Differences between seniors who choose to receive vaccine and those who do not are greatest when vaccine is first made available, and the differences between vaccinated and unvaccinated groups decline slowly over time as seniors at high risk of mortality (who are predominantly unvaccinated) die, making vaccinated and unvaccinated groups more similar with respect to mortality risk over time. This study supports the use of preinfluenza periods for identifying uncontrolled confounding in observational influenza VE studies. Furthermore, this study affirms that the magnitude of uncontrolled confounding in observational VE studies is strong, as the observed confounding during preinfluenza periods is sufficient to entirely account for the apparent VE during influenza seasons.

Finally, this study affirms that adjustment for the traditional comorbidity indicators does not remove or reduce the confounding in observational studies of influenza VE against nonspecific outcomes in seniors (1, 11, 21). Prior studies have shown that the prevalence of comorbid chronic illnesses (based on ICD-9-CM codes) is higher in vaccinated seniors than in unvaccinated seniors (1, 4, 11). This could contribute to confounding by indication and underestimation of VE (i.e., bias toward the null) if these chronic illnesses are associated with an increased risk of death (22). However, we have previously shown that the use of ICD-9-CM codes to define chronic illnesses can lead to substantial misclassification of health status and frailty (10). Importantly, this misclassification is differential by influenza vaccination, and adjustment for the presence or absence of chronic illness (defined by ICD-9-CM codes) can actually increase bias in influenza VE estimates. In the present study, we found that this bias was not reduced even when we used the expanded set of ICD-9-CM comorbidity indicators. This finding suggests that increasing the specificity of potential confounders defined from ICD-9-CM codes does not capture relevant information on the difference between vaccinated and unvaccinated seniors. Available evidence suggests that detailed data on cognitive impairment, functional status, and severity of comorbid illnesses (rather than presence or absence) is necessary to substantially reduce or eliminate confounding of influenza VE estimates in seniors (19). This study provides further evidence that outcomes such as all-cause mortality that are highly nonspecific to influenza are generally not appropriate endpoints for influenza VE estimation in seniors.

We note that the hazard ratio estimates were closer to the null during the 2009/2010 pandemic year than during the 2007/2008 influenza year. We do not believe there was a true difference between associations in these two years, as the hazard ratio estimates were further from the null during the 2007/2008 control time periods (preinfluenza and summer, when any apparent association is the result of uncontrolled confounding) as well as during influenza season. Rather, our findings suggest that the availability of the pandemic vaccine and expected lack of benefit of seasonal influenza vaccine against the pandemic virus may have resulted in atypical use of seasonal influenza vaccine during 2009/2010 compared with previous recent influenza seasons. This possibility is supported by the slightly lower uptake of seasonal influenza vaccine in 2009/2010 (63.8%) compared with 2007/2008 (67.0%) and the differences in the proportion of seasonal vaccinees with measurable comorbidity between 2007/2008 and 2009/2010 (Table 1). Although the pharmacy- and ICD-9-CM-based comorbidity indicators do not capture information on the true confounders, they may indicate differences in the seniors receiving seasonal influenza vaccine between the two study years.

A limitation of this study is that, in contrast to typical years, influenza viruses were circulating intensely during the fall of 2009, which could make the 2009 fall atypical with regard to death rates as compared with other preinfluenza (fall) seasons. However, the influenza viruses circulating in the fall of 2009 were almost entirely 2009 H1N1pdm, which caused little morbidity and mortality among seniors (16), as many of these seniors had cross-protective antibodies from exposure to influenza A(H1N1) strains that circulated before 1957 (23). Thus, the impact of the fall pandemic wave on deaths in seniors was small and should not have affected the presence of confounding in our study. Another potential limitation is that seniors could have received influenza vaccine outside of the MCO in which they were enrolled; if vaccination records were not subsequently transferred to the MCO, these seniors might have been falsely classified as unvaccinated. While previous studies conducted with the Group Health Cooperative and Northern California Kaiser populations have found that vaccine misclassification makes a negligible contribution to the bias in VE estimates (1, 24), pandemic influenza vaccination may have been less likely to be captured in MCO electronic records than seasonal vaccination, as pandemic vaccines were more likely to be administered in non-MCO settings in some of the geographical areas in which the participating MCO provided care.

When evaluated in a year with essentially no seasonal influenza circulation, estimates of the association between influenza vaccination and all-cause mortality showed a pattern of steady trend towards the null from fall through summer. Our findings are consistent with strong confounding by unmeasured factors, the strength of which declines over time. They also illustrate the usefulness of the preinfluenza time period as a control period for identifying and removing confounding in studies of influenza VE. Investigators conducting observational studies of influenza VE against nonspecific outcomes should report adjusted VE estimates during preinfluenza periods, to demonstrate the presence and strength of residual confounding.

Appendix Table 1.

ICD-9-CM Diagnosis Codes for Comorbid Conditions, Medical-Care Utilization Indicators, and Prescription Medications Used as Covariates in a Model of the Association Between Seasonal Influenza Vaccination and All-Cause Mortality Among Seniors

Variable	ICD-9-CM Code(s), Procedure Code(s), or Prescription
Traditional covariates
Heart disease	093, 112.81, 130.3, 391, 393–398, 402, 404, 410–429, 745, 746, 747.1, 747.49, 759.82, 785.2, and 785.3
Cancer	140–198, 199.1, and 200–208
Diabetes	250 and 251
Renal disease	274.1, 408, 580–591, 593.71–593.73, and 593.9
Lung disease	011, 460, 462, 465, 466, 480–511, 512.8, 513–517, 518.3, 518.8, 519.9, and 714.81
Rheumatological disease	446, 710, 714.0–714.4, 714.8, 714.89, and 714.9
Atrial fibrillation	427.3
Lipid disorders	272
Hypertension	401
Dementia	290–294, 331, 340, 341, 348, and 438
Outpatient visits	Not applicable—count of all outpatient visits during baseline year
Pneumonia hospitalization	One or more hospitalizations with ICD-9-CM code 480–487.0 during the baseline period
Alternate covariates
Congestive heart failure	428
Serious or metastatic cancer	150, 151, 155, 157, 158, 159, 162, 163, 191, 196–202, and 204–208
Diabetes complications	250.4, 250.5, 250.6, and 250.7
Chronic renal failure	585 and 586
Asthma	493
Pneumonia	480–487.0 and 507.0
Chronic lung disease	011, 488–511, 512.8, 513–517, 518.3, 518.8, 519.9, and 714.81
Inhaled corticosteroids	Beclometasone, budesonide, flunisolide, betamethasone, fluticasone, triamcinolone, mometasone, and ciclenonide
Oral corticosteroids	Fludrocortisone, desoxycortone, betamethasone, dexamethasone, fluocortolone, methylprednisolone, paramethasone, prednisolone, prednisone, triamcinolone, hydrocortisone, cortisone, prednylidene, deflazacort, and cloprednol
Bronchodilators	α- and β-adrenoreceptor agonists, nonselective β-adrenoreceptor agonists, selective β2-adrenoreceptor agonists, anticholinergics, and xanthines
Statins	3-Hydroxy-3-methyl-glutaryl-coenzyme A reductase inhibitors

Abbreviation: ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.