Caution Required in the Use of Administrative Data and General Laboratory Submissions for Influenza Vaccine Effectiveness Estimation.

To the Editor—During the 1980s, observational study designs were used to estimate influenza vaccine effectiveness (IVE) against serious outcomes in older adults, through convenient, retrospective linkage of large, administrative data sets that were originally assembled for another clinical purpose [1, 2]. Unfortunately, these approaches led to biased IVE estimates, owing to unrecognized selection biases that were only later detected through the critical scrutiny of others [3, 4].

Since 2004, the test negative design (TND) has enabled IVE estimation against laboratory-confirmed influenza [5]. Despite improved outcome specificity, the TND remains an observational design that is susceptible to bias and beholden to core principles for valid vaccine effectiveness estimation: notably, accurate vaccine status ascertainment and consistent case finding [6, 7].

In estimating IVE against influenza-associated hospitalization in pregnant women, Thompson et al [8] retrospectively applied the TND to administrative data sets and general laboratory specimens, submitted at the clinician’s discretion and pooled across multiple countries (Australia/Canada/Israel/United States) and seasons (2010–2016). Australia, however, contributed only 7 vaccinated participants in total, leaving us to wonder how it could have meaningfully contributed to multivariate analyses.

The authors reassure readers that “all sites reported high data-capture rates for influenza vaccination.” However, the site contributing the most participants (Ontario, Canada) has reported substantial misclassification of influenza vaccine status in the physician billing claims used, with a sensitivity of just 32% among adults of childbearing ages [9]. A data set that misclassifies 6–7 of 10 vaccinated participants as unvaccinated raises serious validity concerns, generally leading to an underestimation of IVE. Since without an adjustment for this misclassification the authors considered their findings to be within expectation, with proper adjustments their findings would necessarily exceed expectation. Regardless, such intervention misclassification renders the quantification of intervention effects uncertain.

The authors started with 19 450 hospitalized, pregnant women who met an expansive list of administrative diagnostic codes that were labelled as acute respiratory or febrile illnesses, amongst whom just 6% were tested but 58% were influenza positive [8]. Such high test positivity within such a non-specific clinical entity and across seasons—spanning 5 months, on average—suggests a strong clinician bias in the selection of women to test. TND studies of IVE against hospitalization may be especially prone to selection biases since, in addition to patient health-care–seeking behaviors, physician inclinations and institution-specific algorithms influence who gets admitted, tested, and included. The impacts of these biases may vary in magnitude and direction, and their overall effects on IVE estimates are difficult to ascertain retrospectively.

To mitigate these concerns, it has become the standard of practice in prospective TND evaluations of IVE to require consistent clinical criteria for influenza testing [5, 10]. This is to help ensure that cases and controls emerge from the same source population, with comparable influenza exposure risks among vaccinated and unvaccinated participants. When investigators instead retrospectively apply the TND to general laboratory submissions, without standardization of the influenza testing indication, they inherit a greater burden of proof to show that their findings are valid. Such publications should document the criteria used by clinicians for influenza testing and display the proportion vaccinated among test-negative controls, with the data sufficiently stratified to demonstrate that controls represent the intended source population. Where privacy concerns preclude such a display, as cited by Thompson et al [8], then proper external scrutiny and validity assessment are also precluded.

The legacy of biased IVE estimation in older adults should serve as a caution whenever routine administrative or diagnostic data sets are used for secondary evaluation of intervention effects. Potential biases, though likely different between elderly adults and pregnant women, require the close scrutiny of all such convenience studies, regardless of the target group.