Variations in Seasonal Influenza Vaccine Effectiveness due to Study Characteristics: A Systematic Review and Meta-analysis of Test-Negative Design Studies.

BACKGROUND: Study characteristics influence vaccine effectiveness (VE) estimation. We examined the influence of some of these on seasonal influenza VE estimates from test-negative design (TND) studies.
METHODS: We systematically searched bibliographic databases and websites for full-text publications of TND studies on VE against laboratory-confirmed seasonal influenza in outpatients after the 2009 pandemic influenza. We followed the Cochrane Handbook for Systematic Reviews of Interventions guidelines. We examined influence of source of vaccination information, respiratory specimen swab time, and covariate adjustment on VE. We calculated pooled adjusted VE against H1N1 and H3N2 influenza subtypes, influenza B, and all influenza using an inverse-variance random-effects model.
RESULTS: We included 70 full-text articles. Pooled VE against H1N1 and H3N2 influenza subtypes, influenza B, and all influenza was higher for studies that used self-reported vaccination than for those that used medical records. Pooled VE was higher with respiratory specimen collection within ≤7 days vs ≤4 days of symptom onset, but the opposite was observed for H1N1. Pooled VE was higher for studies that adjusted for age but not for medical conditions compared with those that adjusted for both. There was, however, a lack of statistical significance in almost all differences in pooled VE between compared groups.
CONCLUSIONS: The available evidence is not strong enough to conclude that influenza VE from TND studies varies by source of vaccination information, respiratory specimen swab time, or adjustment for age/medical conditions. The evidence is, however, indicative that these factors ought to be considered while designing or evaluating TND studies of influenza VE.

Vaccination is the most effective prevention for seasonal influenza. Observational studies, rather than randomized controlled trials, are used to examine seasonal influenza vaccine effectiveness (VE) due to feasibility and ethical considerations. Continuous changes that occur in influenza viruses (antigenic drift) [1] mean that influenza vaccines have to be re-formulated every influenza season and that vaccine virus strains may be mismatched with circulating virus strains. Influenza VE studies are conducted each season in many jurisdictions worldwide to assess vaccine performance and to inform subsequent influenza season vaccine development.

Studies on influenza VE often have differences in their design. Studies approach participant recruitment differently, and influenza vaccination status may be determined by either self-report or medical record ascertained. Clinic presentation and timing of respiratory specimen swab collection differ across study participants. The characteristics of study participants, such as age and health status, also vary and may impact VE [2]. Adjustment in analysis of VE varies across studies, and adjustment for specific potential confounders such as age and medical conditions may lead to differences in VE estimations. Due to these variations and other factors, influenza VE estimates vary between jurisdictions.

The test-negative design (TND), an observational study design type, is an increasingly popular design for estimating influenza VE [3, 4]. In a TND study, patients presenting with influenza-like symptoms are tested for influenza. Those with a positive test result become the cases, and those with a negative test result become the controls. Influenza VE (represented as a percentage) is calculated as 1 minus the adjusted ratio of the odds of vaccination in those with positive test results to the odds of vaccination in those with negative test results, multiplied by 100. The TND has been credited with reducing biases due to differential health care–seeking behavior between vaccinated and unvaccinated individuals and differential misclassification of influenza infection status [3]. However, if stringent methods for study participants’ enrollment and influenza testing are not applied, the TND may fail to correct for differential health care–seeking behavior among vaccinated and unvaccinated individuals [5].

We systematically identified, critically appraised, and summarized the findings of published TND studies that examined seasonal influenza VE in primary care settings since the 2009 pandemic influenza. We conducted a systematic review and meta-analysis following the Cochrane Handbook for Systematic Reviews of Interventions guidelines [6], and we reported our findings following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [7].

METHODS

Search Strategy and Selection Criteria

We developed and registered a review protocol in the international prospective register of systematic reviews (PROSPERO) before commencement of this review (registration number CRD42017064595). We searched the MEDLINE (Ovid), Embase (Ovid), PubMed, Scopus (Elsevier), Web of Science, and Google Scholar bibliographic databases. Our literature search strategy (Supplementary Table 1) was reviewed by a knowledge synthesis librarian using the PRESS checklist [8]. The literature search was first conducted in April 2017 and updated in July 2018. Corresponding authors of regional influenza surveillance studies were contacted to check if our searches missed any relevant studies. Identified literature citations were imported and screened in a specially designed Microsoft Access 2016 database (Microsoft Corporation, Redmond, WA, USA).

We were interested in community-based TND studies conducted in primary care settings (outpatients) after the 2009 pandemic influenza (from influenza season 2010/2011 onwards). Only studies that reported multivariable-adjusted influenza VE estimates against laboratory-confirmed influenza of any type or subtype were considered for inclusion in the review. We included only studies with influenza confirmation based on reverse transcriptase polymerase chain reaction (RT-PCR) assay or viral culture of a respiratory specimen and only full-text study publications, irrespective of language of publication. We also included only studies in which patients deemed to have received influenza vaccination did so at least 14 days before their symptom onset, and their symptom onset must not have been >7 days before medical consultation, specimen collection, and study enrollment. Studies involving only hospitalized patients and studies that reported results from mixed hospitalized patients and outpatients without reporting separate results for the 2 patient groups were excluded. We also excluded studies based on retrospective analysis of respiratory samples obtained for clinical diagnostic testing. Furthermore, we excluded studies conducted in military barracks, prisons, care homes, schools, and in subgroups such as individuals with chronic diseases. The outcomes of our interest were adjusted influenza VE against the H1N1 and H3N2 influenza subtypes, influenza B, and all influenza. Two reviewers independently screened the identified citations against the eligibility criteria using a 2-stage sifting approach to review titles/abstracts and full-text articles. Disagreements during this process were resolved through discussion between the 2 reviewers or by involvement of a third reviewer. The number of ineligible citations at the title/abstract screening stage and both the number and reasons for ineligibility at the full-text article screening stage were documented.

Data Extraction

We extracted data in MS Excel 2016 (Microsoft Corporation, Redmond, WA, USA). One reviewer independently extracted data from the included articles, and a second reviewer independently checked the extracted data for errors. Disagreements during this process were resolved through discussion between the 2 reviewers or by involvement of a third reviewer. We extracted study details such as name of the first author, publication year, country, and funding source; study characteristics such as influenza season, participant recruitment strategy, number of participants, source of vaccination information, respiratory specimen swab time, influenza vaccine type, influenza diagnostic test, and the adjusted covariates in analysis; study outcome: influenza VE against the H1N1 and H3N2 influenza subtypes, influenza B, and all influenza; and study results: multivariable-adjusted influenza VE and associated 95% confidence interval (CI). Vaccine antigenic similarity with circulating virus strains was determined from articles, where reported. Where incidence of confirmed influenza was reported, we considered the season’s vaccine to be antigenically similar if the strain that caused a majority of the cases (at least 75%) was similar to that contained in the vaccine, antigenically partially similar if there was modest similarity with strains covered in the vaccine, and antigenically dissimilar if circulating strains were not similar to the strains covered in the vaccine.

Data Synthesis and Analysis

The main study characteristics were synthesized in tabular form. We pooled reported multivariable-adjusted influenza VE estimates and their associated 95% CIs using inverse-variance random-effects models implemented in STATA (version 13; StataCorp LP, Texas, USA). Heterogeneity between the pooled adjusted VE estimates was assessed and quantified statistically using the I2 statistic [9]. The chi-square statistic (χ 2) was used to assess the statistical significance (P value) of the difference between 2 groups of pooled adjusted results. We assessed publication bias (where appropriate) visually using funnel plots and, statistically, using the Egger’s regression test [10]. Subgroup analysis was conducted according to the source of participants’ influenza vaccination status, respiratory specimen swab time, and whether studies included age or age and medical conditions in their multivariable adjustment models. Subgroup analyses were conducted for all patients, and for each of the following age groups: <5 years, 5 to 17 years, 18 to 49 years, 50 to 64 years, and ≥65 years. We included only results for age groups that clearly fell within these predefined age groups without overlapping with another age group.

RESULTS

From a total of 10 041 identified citations, 70 full-text articles met our eligibility criteria (Figure 1) [11-80]. The main characteristics of these articles are summarized in Table 1. There were 11 articles each from the United States and Spain, 8 articles from Australia, 7 articles from the I-MOVE group (involving multiple European countries), and 6 articles each from the United Kingdom and Canada. There were 3 articles from China and 2 articles each from Germany, Israel, Netherlands, Romania, and South Africa. One article each was from Austria, Croatia, Italy, Japan, New Zealand, Portugal, Taiwan, and Turkey. The sample size from the studies in these articles ranged from 197 to 11 430 participants. All the studies were funded by nonindustry sources, and 1 study received funding from both industry and nonindustry sources.

Figure 1.

Modified Preferred Reporting Items for Systematic Reviews and Meta-Analysis flowchart.

Table 1.

Summary of Characteristics of Included Studies

Study	Country	Influenza Season (Study Period)	Respiratory Specimen (Diagnostic Test)	No. of Participants	Circulating Influenza Type(s)	Dominant Influenza Type	VE Outcomes Assessed
Kissling et al. (2011) [11]	Europe	2010/2011	Nasal or throat swab (PCR & culture)	3254	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 Influenza B
Jimenez-Jorge et al. (2012) [12]	Spain	2010/2011	Not reported (PCR)	1369	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 Influenza B
Fielding et al. (2012) [13]	Australia	2011	Nose and/or throat swab (PCR)	529	H1N1, H3N2, influenza B	H1N1 first half, H3N2 mid to later season, influenza B throughout	All influenza H1N1 H3N2 Influenza B
Treanor et al. (2012) [14]	USA	2010/2011	Nasal and throat swabs (children aged <2 years provided nasal swabs only; PCR)	4757	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 H3N2 Influenza A Influenza B
Skowronski et al. (2012) [15]	Canada	2010/2011	Nasal/nasopharyngeal specimen (PCR)	1718	H1N1, H3N2, influenza B	H3N2	All influenza H1N1 H3N2 Influenza A Influenza B
Pitigoi et al. (2012) [16]	Romania	2010/2011	Not reported (PCR)	255	H1N1, H3N2, influenza B	H1N1 and influenza B	All influenza H1N1 Influenza B
Castilla et al. (2013) [17]	Spain	2011/2012	Nasopharyngeal and pharyngeal swabs (PCR)	588	H3N2, influenza B	H3N2	All influenza
Kelly et al. (2013) [18]	Australia	2010 & 2011	Combined nose and throat swab specimens (nose swab specimens were only obtained from children aged <2 years; PCR)	309 (2010) 398 (2011)	H1N1, H3N2, influenza B	2010 H1N1, 2011 H3N2	All influenza H1N1 H3N2 Influenza B
Sullivan et al. (2013) [19]	Australia	2010, 2011, & 2012	Not reported (PCR)	420 (2010) 630 (2011) 678 (2012)	H1N1, H3N2, influenza B	2010 H1N1, 2011 influenza B, 2012 H3N2	All influenza
Martínez-Baz et al. (2013) [20]	Spain	2010/2011	Nasopharyngeal swabbing (PCR)	530	H1N1, H3N2, influenza B	H1N1	All influenza
Kissling et al. (2013) [21]	Europe	2011/2012	Nasopharyngeal swab (PCR & culture)	4362	H1N1, H3N2, influenza B	H3N2	H3N2
Jimenez-Jorge et al. (2013) [22]	Spain	2011/2012	Not reported (PCR & culture)	378	H1N1, H3N2, influenza B	H3N2	All influenza H3N2
Pebody et al. (2013) [23]	UK	2011/2012	Respiratory samples (PCR)	3560	H1N1, H3N2, influenza B	H3N2	H3N2
Bateman et al. (2013) [24]	USA	2010/2011	Nasal and oropharyngeal swab (PCR)	1549	H1N1, H3N2, influenza B	H3N2	H1N1 H3N2 Influenza A
Englund et al. (2013) [25]	Germany	2010/2011	Nasal or pharyngeal swabs or nasopharyngeal aspirates (PCR)	1866	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 Influenza B
Lo et al. (2013) [26]	Taiwan	2011/2012	Throat or nasal swabs (PCR & culture)	918	H1N1, H3N2, influenza B	Influenza B	All influenza Influenza A Influenza B
Pebody et al. (2013) [27]	UK	2010/2011	Mouth swab (PCR)	7121	H1N1, influenza B	HINI	H1N1 Influenza B
Sullivan et al. (2014) [28]	Australia	2012	Nasal and throat samples (PCR)	600	H1N1, H3N2, influenza B	H3N2	All influenza H3N2
Levy et al. (2014) [29]	Australia	2010 to 2012	Two nose swabs and 1 throat swab (PCR)	448 (2010) 351 (2011) 1361 (2012)	H1N1, H3N2, influenza B	H1N1 in 2010 and 2011, H3N2 in 2012	All influenza H1N1 H3N2 Influenza B
Ohmit et al. (2014) [30]	USA	2011/2012	Throat swab and nasal swab (or nasal swab only in patients aged <2 years; PCR)	4771	H1N1, H3N2, influenza B	H3N2	All influenza H1N1 H3N2 Influenza A Influenza B
Kissling et al. (2014) [31]	Europe	2012/2013	Nasopharyngeal swab (PCR & culture)	6609	H1N1, H3N2, influenza B	Influenza B	H1N1 H3N2 Influenza B
Suzuki et al. (2014) [32]	Japan	2011/2012	Nasopharyngeal swab (PCR)	309	H1N1, H3N2, influenza B	H3N2	All influenza Influenza A
Skowronski et al. (2014) [33]	Canada	2011/2012	Nasal/nasopharyngeal swabs (PCR)	1507	H1N1, H3N2, influenza B	Influenza B	All influenza H1N1 H3N2 Influenza A Influenza B
Savulescu et al. (2014) [34]	Spain	2010/2011	Not reported (PCR & culture)	5057	H1N1, H3N2, influenza B	H1N1 and influenza B	H1N1 Influenza B
Nunes et al. (2014) [35]	Portugal	2012/2013	Nasopharyngeal swab or a combined nasopharyngeal and oropharyngeal swab (PCR & culture)	335	H1N1, H3N2, influenza B	H1N1	All influenza
Skowronski et al. (2014) [36]	Canada	2012/2013	Nasal or nasopharyngeal swabs (PCR)	1501	H1N1, H3N2, influenza B	H3N2	All influenza H1N1 H3N2 Influenza A Influenza B
Yang et al. (2014) [37]	China	2012/2013	Pharyngeal swabs (culture)	1998	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 H3N2
Andrews et al. (2014) [38]	UK	2012/2013	Not reported (PCR)	3286	H1N1, H3N2, influenza B	Influenza B	H1N1 H3N2 Influenza A Influenza B
McAnerney et al. (2015) [39]	South Africa	2010 to 2013	Nasopharyngeal swab (PCR)	5344	H1N1, H3N2, influenza B	2010 influenza B, 2011 H1N1, 2012 H3N2, 2013 H1N1	All influenza
Pitigoi et al. (2015) [40]	Romania	2012/2013	Not reported (PCR)	197	H1N1, H3N2, influenza B	Influenza B	All influenza H1N1
Valenciano et al. (2015) [41]	Europe	2013/2014	Nasopharyngeal swab (PCR)	3020	H1N1, H3N2, influenza B	H3N2	H1N1
Helmeke et al. (2015) [42]	Germany	2012/2013	Throat or nasopharyngeal swab (PCR)	834	H1N1, H3N2, influenza B	Influenza B	All influenza H1N1 H3N2 Influenza B
Carville et al. (2015) [43]	Australia	2013	Nose or throat swab (PCR)	262	H1N1, H3N2, influenza B	Influenza A and B	All influenza H1N1 Influenza B
Chen et al. (2015) [44]	USA	2010/2011 & 2011/2012	One nasal and 1 throat swab (PCR)	927	H1N1, H3N2, influenza B	H1N1	All influenza
McLean et al. (2015) [45]	USA	2012/2013	Nasal and throat specimens (for children aged <2 years, only nasal specimens were obtained; PCR)	6452	H1N1, H3N2, influenza B	H3N2	All influenza H3N2
Jimenez-Jorge et al. (2015) [46]	Spain	2010/2011, 2011/2012, & 2012/2013	Nasal or nasopharyngeal (PCR & culture)	3180:SISS, 1369:cycEVA (2010/2011) 3484:SISS, 1446:cycEVA (2011/2012) 3357:SISS, 1432:cycEVA (2012/2013)	H1N1, H3N2, influenza B	2010/2011 H1N1, 2011/2012 H3N2, 2012/2013 influenza B	H1N1 H3N2 Influenza B
Jimenez-Jorge et al. (2015) [47]	Spain	2010/2011, 2011/2012, 2012/2013, & 2013/2014	Nasal or nasopharyngeal (PCR & culture)	(cycEVA)	H1N1, H3N2, influenza B	2010/2011 H1N1, 2011/2012 H3N2, 2012/2013 influenza B, 2013/2014 H3N2 and H1N1	All influenza H1N1 H3N2 Influenza B
Kurecic- Filipovic et al. (2015) [48]	Croatia	2010/2011	Not reported (PCR)	495	H1N1, influenza B	H1N1	All influenza H1N1
Martinez- Baz et al. (2015) [49]	Spain	2012/2013	Nasopharyngeal and pharyngeal swabs (PCR)	522	H1N1, H3N2, influenza B	Influenza B	All influenza Influenza B
Skowronski et al. (2015) [50]	Canada	2013/2014	Nasal/nasopharyngeal specimens (PCR)	1700	H1N1, H3N2, influenza B	H1N1	All influenza H1N1
Pebody et al. (2015) [51]	UK	2014/2015	Not reported (PCR)	2931	H1N1, H3N2, influenza B	H3N2	All influenza H3N2 Influenza A Influenza B
Gherasim et al. (2016) [52]	Spain	2014/2015	Not reported (PCR)	5044	H3N2, influenza B	H3N2	H3N2 Influenza B
Fielding et al. (2016) [53]	Australia	2015	Nose/throat swabs (PCR)	2443	H1N1, H3N2, influenza B	Influenza B	All influenza H1N1 H3N2 Influenza B
Pebody et al. (2016) [54]	UK	2015/2016	Respiratory samples (PCR)	3841	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 Influenza B
Rizzo et al. (2016) [55]	Italy	2014/2015	Nasal or throat swab (PCR)	1193	H1N1, H3N2, influenza B	H1N1 and H3N2	All influenza H1N1 H3N2 Influenza B
Castilla et al. (2016) [56]	Spain	2014/2015	Double swabs, nasopharyngeal and pharyngeal (PCR)	660	H1N1, H3N2, influenza B	H3N2 and influenza B	All influenza H3N2 Influenza B
Redlberger- Fritz et al. (2016) [57]	Austria	2014/2015	Nasopharyngeal swabs (PCR)	815	H1N1, H3N2, influenza B	H3N2	All influenza H1N1 H3N2 Influenza B
Thompson et al. (2016) [58]	USA	2011/2012 & 2012/2013	Nasal and throat specimens (or nasal specimens only for children aged <2 years; PCR)	1441 (2011/2012) 1327 (2012/2013)	H1N1, H3N2, influenza B	H3N2 in both seasons	All influenza H3N2 Influenza B
Pierse et al. (2016) [59]	New Zealand	2014	Nasopharyngeal or throat swab (PCR)	1154	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 H3N2 Influenza A Influenza B
Van Doorn et al. (2017) [60]	The Netherlands	2010/2011, 2011/2012, 2012/2013, & 2013/2014	Nose and throat swabs (PCR & culture)	Unclear	H1N1, H3N2, influenza B	2010/2011 H1N1; 2011/2012, 2012/2013, and 2013/2014 H3N2	All influenza
Kelly et al. (2016) [61]	Australia	2011, 2012, & 2013	Not reported (PCR)	642 (2012/2013) 684 (2012) 354 (2013)	H1N1, H3N2, influenza B	Not reported	All influenza
Wang et al. (2016) [62]	China	2011/2012	Nasopharyngeal specimen (PCR)	668	Not reported	Not reported	All influenza
Cowling et al. (2016) [63]	USA	2010/2011, 2011/2012, & 2012/2013	Nasopharyngeal, oropharyngeal or nasal swab (PCR)	4208 (2010/2011) 2164 (2011/2012) 4278 (2012/2013)	H1N1, H3N2, influenza B	H1N1, H3N2, and influenza B in 2010/2011; H3N2 in 2011/2012; H3N2 and influenza B in 2012/2013	All influenza H1N1 H3N2 Influenza B
Skowronski et al. (2016) [64]	Canada	2014/2015	Nasal/nasopharyngeal specimens	1930	H1N1, H3N2, influenza B	H3N2	All influenza H3N2 Influenza B
Zimmerman et al. (2016) [65]	USA	2014/2015	Nasal and throat swabs (children aged <2 years provided nasal swabs only; PCR)	9311	H3N2, influenza B	H3N2	All influenza H3N2
Gaglani et al. (2016) [66]	USA	2013/2014	Combined nose and throat swab specimens (nose swab specimens were only obtained from children aged <2 years; PCR)	5637	H1N1, H3N2, influenza B	H1N1	H1N1
Valenciano et al. (2016) [67]	Europe	2014/2015	Nasopharyngeal specimens (PCR)	6524	H1N1, H3N2, influenza B	H3N2	H1N1 H3N2 Influenza B
McAnerney et al. (2017) [68]	South Africa	2015	Throat and/or nasal swabs (PCR)	899	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 H3N2 Influenza B
Darvishian et al. (2017) [69]	The Netherlands	2010/2011, 2011/2012, & 2012/2013	Throat swab and nose swab (PCR)	Not reported	H1N1, H3N2, influenza B	H3N in 2011/2012, influenza B in 2012/2013, H3N2 in 2013/2014, influenza B in 2010/2011	All influenza H1N1 H3N2 Influenza B
Ma et al. (2017) [70]	China	2014/2015	Oral pharyngeal swab (PCR)	9297	H3N2, influenza B	H3N2	All influenza H3N2 Influenza B
Pebody et al. (2017) [71]	UK	2016/2017	Not reported (PCR)	2881	H1N1, H3N2, influenza B	H3N2	All influenza Influenza A H3N2 Influenza B
Skowronski et al. (2017) [72]	Canada	2015/2016	Nasal/nasopharyngeal swab (PCR)	2008	H1N1, H3N2, influenza B	H1N1	All influenza Influenza A H1N1 H3N2 Influenza B
Jackson et al. (2017) [73]	USA	2015/2016	Nasal/oropharyngeal swab (PCR)	6879	H1N1, H3N2, influenza B	H1N1	All influenza H1N1 H3N2 Influenza B
Gherasim et al. (2017) [74]	Spain	2015/2016	Not reported (PCR & culture)	661	H1N1, influenza B	Influenza B	H1N1 Influenza B
Stein et al. (2018) [75]	Israel	2016/2017	Nasal and throat swabs (PCR)	1088	H1N1, H3N2, influenza B	H3N2	H3N2
Yaron-Yakoby et al. (2018) [76]	Israel	2014/2015	Nose and throat swabs (PCR)	1005 (2014/2015) 1658 (2015/2016)	H1N1, H3N2, influenza B	H3N2 in 2014/2015, H1N1 & influenza B in 2015/2016	All influenza & H3N2 in 2014/2015 All influenza, H1N1, & influenza B in 2015/2016
Poehling et al. (2018) [77]	USA	2015/2016	Nasal swab (PCR)	1012	H1N1, influenza B	H1N1	All influenza H1N1 Influenza B
Valenciano et al. (2018) [78]	Europe	2011/2012 to 2016/2017	Nasopharyngeal swab (PCR)	Not clear	H1N1, influenza B (2015/16) H3N2 (2016/17)	H3N2	H1N1 H3N2 Influenza B
Hekimoglu et al. (2018) [79]	Turkey	2014/2015	Nasal, nasopharyngeal, throat, nasal plus throat, nasopharyngeal plus throat, nasal plus nasopharyngeal (PCR)	2561	H1N1, H3N2, influenza B	Influenza B	All influenza H1N1 H3N2 Influenza B
Kissling et al. (2018) [80]	Europe	2015/2016	Nasopharyngeal or combined naso- and oropharyngeal specimens (PCR)	11 430	H1N1, H3N2, influenza B	H1N1 Influenza B	H1N1 Influenza B

Abbreviations: PCR, polymerase chain reaction; pdm09, pandemic 2009.

Pooled Adjusted VE by Method of Confirmation of Vaccination Status

Although not statistically significant, we observed a 10% higher pooled VE against H1N1 (P = .191), 7% against H3N2 (P = .626), and 5% against both influenza B (P = .529) and all influenza (P = .554) (Figure 2) for self-reported vaccination compared with medical record vaccination confirmation (Table 2). Almost all of the studies with self-reported vaccination were, however, from 1 research group in Canada. More of the studies with self-reported vaccination compared with those with medical record vaccination confirmation adjusted for both age and medical conditions. Zero percent (for H1N1), 20% (for H3N2, and influenza B), and 14% (for all influenza) of the studies with self-reported vaccination were from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 8.3% (for H1N1), 30.8% (for H3N2), 23.1% (for influenza B), and 16% (for all influenza) of the studies with medical record vaccination confirmation were from seasons in which vaccine virus strains were antigenically dissimilar. Similar observations were made against H1N1 in 18- to 49-year-olds and against all influenza in ≥65-year-olds (Supplementary Table 2).

Figure 2.

Forest plot of vaccine effectiveness against all influenza by confirmation of vaccination status. Abbreviations: CI, confidence interval; VE, vaccine effectiveness.

Table 2.

Pooled Adjusted VE for All Patients (Irrespective of Age)

Influenza Types and Subtypes Analyzed Subgroups	No. of Studies	Pooled VE Across All Seasons (95% CI)	I 2 Statistic, %	Publication Bias, Egger’s Test P Value
H1N1
Vaccination status: medical records	24	52 (45–58)	32.7	.031
Vaccination status: self-reported	6	62 (46–73)	55.0	N/A
Respiratory specimen swab: ≤7 d	39	54 (49–58)	39.5	.022
Respiratory specimen swab: ≤4 d	7	59 (47–69)	0.0	N/A
Adjusted age	26	57 (51–63)	32.1	.034
Adjusted age & medical conditions	20	53 (46–59)	43.6	.148
H3N2
Vaccination status: medical records	26	25 (15–34)	55.0	.988
Vaccination status: self-reported	5	32 (-0–53)	76.9	N/A
Respiratory specimen swab: ≤7 d	35	28 (22–34)	57.5	.301
Respiratory specimen swab: ≤4 d	8	18 (-26–47)	63.3	N/A
Adjusted age	23	34 (28–40)	11.5	.794
Adjusted age & medical conditions	20	21 (10–30)	70.5	.997
Influenza B
Vaccination status: medical records	26	43 (31–52)	70.3	.701
Vaccination status: self-reported	5	48 (36–59)	28.2	N/A
Respiratory specimen swab: ≤7 d	33	48 (43–53)	28.2	.974
Respiratory specimen swab: ≤4 d	10	38 (4–60)	77.5	.070
Adjusted age	22	50 (44–56)	26.5	.893
Adjusted age & medical conditions	21	40 (27–51)	70.7	.252
All influenza
Vaccination status: medical records	39	43 (35–49)	75.3	.807
Vaccination status: self-reported	7	48 (31–61)	84.5	N/A
Respiratory specimen swab: ≤7 d	56	46 (41–51)	70.6	.152
Respiratory specimen swab: ≤4 d	12	38 (15–55)	77.3	.009
Adjusted age	32	47 (42–52)	56.5	.477
Adjusted age & medical conditions	37	43 (34–51)	79.8	.184

Abbreviations: CI, confidence interval; N/A, not applicable; VE, vaccine effectiveness.

Pooled Adjusted VE by Timing of Respiratory Specimen Swab Collection

Despite a lack of statistical significance, we observed a 10% higher pooled adjusted VE against H3N2 (P = .596) and influenza B (P = .491), and 8% against all influenza (P = .447) (Figure 3), for swab collection within ≤7 days compared with ≤4 days of symptom onset (Table 2). In contrast, a 5% higher pooled adjusted VE was observed against H1N1 (P = .410) for swab collection within ≤4 days compared with swab collection within ≤7 days of symptom onset. There was no meaningful difference between studies with swab collection within ≤7 days and ≤4 days with regards to adjustment for both age and medical conditions in their analyses. Fifteen percent (for influenza B) and 18.5% (for all influenza) of the studies with swab collection ≤7 days were, however, from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 22.2% (for influenza B) and 27.3% (for all influenza) of the studies with swab collection within ≤4 days were from seasons in which vaccine virus strains were antigenically dissimilar. Similarly, 5% (for H1N1) of the studies with swab collection within ≤7 days were from seasons in which vaccine strains were antigenically dissimilar, whereas 0% of the studies with swab collection within ≤4 days were from seasons in which vaccine strains were antigenically dissimilar. Evidence was conflicting across age groups (Supplementary Table 2).

Figure 3.

Forest plot of vaccine effectiveness against all influenza by timing of respiratory specimen swab collection. Abbreviations: CI, confidence interval; VE, vaccine effectiveness.

Pooled Adjusted VE by Covariate Adjustment

Notwithstanding a lack of statistical significance apart from for H3N2, we observed a 4% higher pooled adjusted VE against H1N1 (P = .375), 13% against H3N2 (P = .029), 10% against influenza B (P = .144), and 4% against all influenza (P = .427) (Figure 4) for studies that included age among the adjusted covariates compared with those that included both age and medical conditions (Table 2). Three point eight percent (for H1N1), 13% (for H3N2), 13.6% (for influenza B), and 6.7% (for all influenza) of the studies that included age but not medical conditions were, however, from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 5.3% (for H1N1), 36.8% (for H3N2), 20% (for influenza B), and 30.6% (for all influenza) of the studies that included age and medical conditions among the adjusted covariates were from seasons in which vaccine virus strains were antigenically dissimilar. Evidence was conflicting across age groups (Supplementary Table 2).

Figure 4.

Forest plot of vaccine effectiveness against all influenza by covariate adjustment. Abbreviations: CI, confidence interval; VE, vaccine effectiveness.

DISCUSSION

Despite a lack of statistical significance, we observed differences in pooled adjusted influenza VE between sources of influenza vaccination confirmation, respiratory specimen swab timing, and adjustments for 2 key confounders in study analysis. In our analysis of all study participants (irrespective of age), small differences were found between self-reported and medical record–confirmed influenza vaccinations, with higher pooled VE observed for self-reported vaccination, contrary to our expectations. However, almost all of the studies for self-reported vaccination were conducted in Canada and by the same group of researchers. We found substantial differences between respiratory specimen swab within ≤7 days and ≤4 days, with higher pooled VE observed for swab within ≤7 days. We also found substantial differences between studies that adjusted for age and those that adjusted for both age and medical conditions, with higher pooled VE observed for studies that adjusted for age. The above findings differed across age groups.

Studies have found that exposure misclassification can lead to significant bias in VE estimation [81, 82]. Self-reported vaccination is susceptible to recall and social desirability (individuals wanting to present a vaccine-compliant image) biases, with the potential for vaccination status misclassification. Smedt and colleagues showed in their simulation study that decreased exposure sensitivity and specificity underestimate true VE when misclassification of exposure (vaccination status) is nondifferential, but that when misclassification is differential, the bias could go in either direction, with the estimated VE deviating largely from the true VE. Compared with vaccination confirmation from medical records, self-reported vaccination usually has a higher sensitivity across various populations [83, 84] but a lower specificity in some population subgroups [85, 86]. Compared with whites, Hispanics were 2.7 times more likely to claim receipt of vaccination (self-report), and compared with younger individuals, self-reported influenza vaccination in the elderly had low specificity [84]. The observed higher pooled adjusted VE for self-reported compared with medical record–confirmed influenza vaccination status in this review, although not expected, may be due to differential misclassification of vaccination status, which Smedt and colleagues showed could either inflate or underestimate the true VE. This becomes more plausible considering that the studies with self-reported vaccination were almost all from Canada and from the same research group. Study center influence such as characteristics of the study participants, participant recruitment strategy, and influenza testing may also explain our findings.

Influenza incubation averages 2 days (range, 1–4 days) [87]. To maximize influenza virus detection from respiratory specimens, it is advocated that, ideally, swabs be collected within <4 days of influenza-like symptom onset. The longer swab collection is from symptom onset, the lower the likelihood of detecting influenza and the greater the potential for false-negative testing. Accurate reporting of symptom onset is therefore important, as a good TND study is predicated on patient symptom onset of ≤7 days. It will also help minimize outcome misclassification bias. False-negative testing among the vaccinated leads to VE overestimation, while false-negative testing among the unvaccinated leads to VE being underestimated. The observed higher pooled adjusted VE for swab collection of ≤7 days compared with ≤4 days in this review may therefore be due to a higher proportion of false negatives among the ≤7 days swab collection group, although this is not confirmable. Additionally, studies that included swab collection within ≤4 days possibly used more stringent swab collection criteria, resulting in reduced precision of VE estimation.

Seasonal influenza VE can vary from person to person. Various individual factors impact the VE [88], and 2 main factors (age and medical conditions) are known to play an important role in determining the likelihood that a vaccine will protect a person against influenza and to what extent. Age-dependent patterns in influenza vaccine protection have been reported from season to season, implicating the potential effect of age-related immune response in seasonal influenza VE [89]. For example, VE in the elderly population is reduced because of lower seroconversion rates that arise due to poorer immunological response to vaccination [90]. How well an individual responds to a vaccine may also be determined by underlying health conditions [91]. The observed higher pooled adjusted VE for studies that included age but not medical conditions compared with those that included both age and medical conditions among adjusted covariates in studies is in line with expectations, as adjusting for both age and medical conditions is likely to diminish VE compared with adjusting for age.

It is widely known that antigenic drift can markedly reduce seasonal influenza VE. For example, Flannery (2016) found that VE against H3N2 was almost 0 for an antigenically drifted genetic group of H3N2 viruses and 44% against a genetic group of H3N2 viruses that were antigenically similar to the seasonal vaccine strains [92]. This may explain the observed higher pooled adjusted VE in the subgroups with lower proportions of studies in which the seasonal influenza vaccine was antigenically dissimilar to the circulating virus strains. Variations in study design, sample size, vaccine type, and the demographic and temporal patterns underlying VE estimates from the included studies may also explain the variations observed in the pooled adjusted VE between compared groups. This, together with vaccine antigenic similarity with the circulating virus strains, may explain the high heterogeneity in many of the pooled adjusted VE. Where there were adequate numbers of studies for exploration of heterogeneity using metaregression, the available covariates tended to be highly collinear, thus limiting the usefulness of metaregression. Second, it was impossible to disentangle the effects of vaccine type and the underlying patient-level variations, as the analysis was conducted at the study level and these were not clearly reported in studies.

To our knowledge, our review is the first to evaluate differences in VE due to source of influenza vaccination status, respiratory specimen swab time, and confounder adjustments in statistical models for analysis. Irving et al. (2009) evaluated influenza vaccination status determined by self-report and by a real-time vaccination registry and found that the sensitivity and specificity of self-reported influenza vaccination compared with vaccination registry records were 95% and 90%, respectively, and that self-reported vaccination status was a sensitive and somewhat specific indicator of actual vaccine status, with misclassification being more common among young people [83]. However, the study did not compare influenza VE from these 2 sources of vaccination. No reviews seem to have compared seasonal influenza VE by respiratory specimen swab time and inclusion of main confounders in statistical models for analysis as we have done.

Our decision to include only influenza seasons after the 2009 pandemic influenza may have limited the number of potentially relevant TND studies for this review. However, it allowed us to focus on studies conducted from when public funding of influenza vaccination increased in most Western jurisdictions. It should be noted that some eligible studies conducted during this stated period may not have been published by the time we conducted our literature search, and therefore would not have been included in this review. Despite growing evidence to suggest that VE may be influenced by prior vaccinations [93, 94], the included studies did not report whether the study participants received the previous season’s influenza vaccination; hence, we could not assess the impact on VE estimates in our analyses. Furthermore, due to insufficient data, we could not examine VE against all outcomes for our subgroup analyses and for all age groups. We could also not separate individual study participants’ effects from study center effects (eg, effectiveness of vaccine policies and programs, participant recruitment strategy, and slight differences in symptom definitions), as the studies were conducted in different jurisdictions with potentially unique jurisdictional characteristics. Finally, we could not assess the reliability of reported estimates from the included studies because we could not ascertain if the studies met all of the assumptions that well-conducted TND studies are expected to meet to ensure that effect size estimates from the studies are not biased [5]. Although many of the studies adjusted for age or age and medical conditions, there were differences in the other covariates adjusted for in the studies. This may have contributed to the high heterogeneity observed in some of our pooled VE estimates.

Our review has many merits. We developed and registered a detailed protocol in PROSPERO before the execution of our search strategy, and we fully complied with the Cochrane Handbook for Systematic Reviews of Interventions guidelines throughout the review. We utilized the expertise of a methodologist trained in evidence synthesis literature searching to develop a comprehensive search strategy for the review, and this was subsequently reviewed by a professional knowledge synthesis librarian using the PRESS checklist. We searched appropriate bibliographic databases for literature and properly screened retrieved citations (against the eligibility) following the standards specified in the Cochrane Handbook for Systematic Reviews of Interventions. Where necessary, we requested additional data from the corresponding authors of the included studies to ensure completeness of the analyzed data. We included only studies in which influenza testing was conducted using the gold standard tests (PCR or viral culture). Furthermore, we examined variations in seasonal influenza VE across all clinically relevant age groups (<5 years, 5 to <18 years, 18 to 49 years, 50 to 64 years, and ≥65 years). We conducted the review to the highest expected standards and have reported in accordance with the PRISMA guidelines.

CONCLUSIONS

The available evidence from TND studies conducted after the 2009 pandemic influenza is not strong enough to conclude that influenza VE varies by source of vaccination status, respiratory specimen swab time, or adjustment for age/medical conditions. However, the evidence is indicative that these factors should be considered while designing or evaluating influenza VE from this study type. There is a need for researchers to ensure that age and medical conditions are both adjusted for in influenza VE estimations from TND studies, while uniformity in covariate adjustments across studies would help reduce heterogeneity and increase precision of pooled VE.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.