Effectiveness and harms of seasonal and pandemic influenza vaccines in children, adults and elderly: a critical review and re-analysis of 15 meta-analyses.

Fifteen meta-analyses have been published between 1995 and 2011 to evaluate the efficacy/effectiveness and harms of diverse influenza vaccines--seasonal, H5N1 and 2009 (H1N1)--in various age-classes (healthy children, adults or elderly). These meta-analyses have often adopted different analyses and study selection criteria. Because it is difficult to have a clear picture of vaccine benefits and harms examining single systematic reviews, we compiled the main findings and evaluated which could be the most reasonable explanations for some differences in findings (or their interpretation) across previously published meta-analyses. For each age group, we performed analyses that included all trials that had been included in at least one relevant meta-analysis, also exploring whether effect sizes changed over time. Although we identified several discrepancies among the meta-analyses on seasonal vaccines for children and elderly, overall most seasonal influenza vaccines showed statistically significant efficacy/effectiveness, which was acceptable or high for laboratory-confirmed cases and of modest magnitude for clinically-confirmed cases. The available evidence on parenteral inactivated vaccines for children aged < 2 y remains scarce. Pre-pandemic "avian" H5N1 and pandemic 2009 (H1N1) vaccines can achieve satisfactory immunogenicity, but no meta-analysis has addressed H1N1 vaccination impact on clinical outcomes. Data on harms are overall reassuring, but their value is diminished by inconsistent reporting.

Introduction

A large body of evidence has been generated on influenza vaccines for different types of virus strains and different populations and settings. As an effort to integrate this evidence, several meta-analyses have been published between 1995 and 2011 to evaluate the benefits and harms of influenza vaccines,- which are considered the most important tool to control influenza pandemics. Such meta-analyses have evaluated diverse influenza vaccines (seasonal, pre-pandemic H5N1, and H1N1) and in different age-classes (healthy children, adults or elderly). Even meta-analyses on the same vaccination and age-class have often used different stratified analyses and study selection criteria. It is therefore difficult to have a clear picture of vaccine benefits and harms examining single meta-analyses. An over-arching evaluation of all recent meta-analyses on this field, by means of an umbrella review, may offer some insights about the broader picture of the evidence on influenza vaccination. We performed here such an umbrella review and we also re-analyzed data from previously published meta-analyses. Within each age group and type of vaccine, we performed analyses that included all trials that had been selected by at least one previous meta-analysis. This allowed us to explore better whether the differences between previous meta-analyses were due to specific studies being included or excluded and to reach more solid inferences regarding the summary estimates.

Results

Meta-analyses on seasonal vaccination for healthy children

We found five meta-analyses providing an overall summary of the benefits and harms of seasonal influenza vaccines for healthy children, as compared with placebo or no intervention (Table 1).,-, The first was published by Cochrane researchers in 2005,, it has been updated in 2008 and the next update is expected in 2012. Negri et al. also published a meta-analysis in 2005; and Manzoli et al. in 2007. More recently, Rhorer et al. published a meta-analysis solely focused on RCTs on the live attenuated vaccine (LAV) that was approved for use in the USA (FluMist ©), using as outcome culture-confirmed cases of influenza (a subset of LCC, which typically also include cases assessed serologically). Finally, Osterholm et al. also evaluated only vaccines licensed in USA to prevent RT-PCR or culture-confirmed influenza infections in all ages, including both RCTs and observational studies. RCTs were included in all meta-analyses, two of which also included observational studies., The funding source was not declared in one meta-analysis; a manufacturing company funded the work by Rhorer et al. and the other three meta-analyses were either funded by not-for-profit institutions, of had received no funding.

Table 1. Meta-analyses on influenza vaccines for healthy children / adolescents

	Negri8	Manzoli7	Jefferson4	Rhorer *12	Osterholm9
End date of the search (mm/yy)	12/2003	05/2005	09/2007	Not reported	02/2011
Participant’s age-range (years)	≤ 18	≤ 18	< 16	≤ 17	All ages §
Included study designs	RCTs	RCTs	RCTs, Obs.	RCTs	RCTs, Obs.
Funding source	NR	None	Public institutions	MedImmune	Not-for-profit foundation
Laboratory-confirmed cases
- Overall ψ
N. data sets (sample)	11 (2711) α12 (5935) α, β	18 (8574)	RCTs: 10 (7629)Obs.: 7 (1956)	6 (10,717) *	RCTs: 10 (12,052) §Obs.: 4 (2067) §
Vaccine efficacy, % (95% CI)	59 (43; 71)74 (57–84) β	67 (51; 78)	RCTs: 72 (38; 88)Obs.: 51 (30; 65)	75 (71; 79) *	RCTs: 72 (16; 91) §Obs.: 54 (11; 76) §
- Live-Attenuated
N. data sets (sample)	5 (1748)6 (4406) β	7 (4325)	RCTs: 5 (6001)Obs.: 1 (83)	6 (10,717) *	RCTs: 8 (11,266) §Obs.: 0 (0) §
Vaccine efficacy, % (95% CI)	54 (20; 74)80 (53–91) β	72 (38; 87)	RCTs: 82 (71; 89)Obs.: 44 (9; 65)	75 (71; 79) *	RCTs: 83 (69; 91) §
- Parenteral Inactivated
N. data sets (sample)	6 (1833)6 (2262) β	11 (4249)	RCTs: 5 (1628)Obs.: 6 (1873)	0 (0) *	RCTs: 2 (786) §Obs.: 4 (2067) §
Vaccine efficacy, % (95% CI)	63 (43; 76)65 (45–77) β	62 (45; 75)	RCTs: 59 (41; 71)Obs.: 58 (27; 75)	–	RCTs: 46 (-63; 82) §Obs.: 54 (11; 76) §
Clinically-confirmed cases
- Overall ψ
N. data sets (sample)	17 (148,738) α	19 (247,517)	RCTs: 12 (207,806)Obs.: 13 (33,839)	NA	NA
Vaccine efficacy, % (95% CI)	33 (29; 36)	36 (31; 40)	RCTs: 33 (29; 38)Obs.: 38 (32; 43)	NA	NA
- Live-Attenuated
N. data sets (sample)	10 (141,532)	10 (231,911)	RCTs: 7 (188,418)Obs.: 2 (22,077)	NA	NA
Vaccine efficacy, % (95% CI)	34 (31; 38)	35 (30; 40)	RCTs: 33 (28; 38)Obs.: 37 (31; 43)	NA	NA
- Parenteral Inactivated
N. data sets (sample)	7 (19,849)	9 (15,606)	RCTs: 5 (19,388)Obs.: 11 (11,762)	NA	NA
Vaccine efficacy, % (95% CI)	33 (22; 42)	45 (33; 55)	RCTs: 36 (24; 46)Obs.: 45 (30; 58)	NA	NA
Acute otitis media
- Overall ψ
N. data sets (sample)	NA	11 (11,349)	RCTs: 7 (4508) **Obs.: 1 (119)	NA	NA
Vaccine efficacy, % (95% CI)	NA	51 (21; 70)	RCTs: 6 (-28; 31)Obs.: 52 (-3; 78)	NA	NA
- Live-Attenuated
N. data sets (sample)	NA	5 (9962)	RCTs: 3 (3280) **Obs.: 0 (0)	NA	NA
Vaccine efficacy, % (95% CI)	NA	73 (25; 90)	RCTs: 59 (4; 295)	NA	NA
- Parenteral Inactivated
N. data sets (sample)	NA	6 (1388)	RCTs: 4 (1228) **Obs.: 1 (119)	NA	NA
Vaccine efficacy, % (95% CI)	NA	32 (-16; 60)	RCTs: -14 (-39; 6)Obs.: 52 (-3; 78)	NA	NA

CI, Confidence Interval; NA, Not assessed; NR, Not reported; RCT, Randomized clinical trial; Obs, Observational studies; ψ, Some meta-analyses only reported separated estimates for PIV or LAV. In these cases, the overall estimate of efficacy was derived combining PIV and LAV summary estimates using a generic inverse variance approach, with a random-effect method; α, The total sample of the overall meta-analysis (which includes both LAV and PIV) has been recomputed because authors repeated placebo data for each sub-trial (see for more details); β, The values in the first line are referred to serologically-confirmed influenza cases; those in the second line to culture-confirmed influenza cases (the definition of which, however, differed from the meta-analyses by Rhorer et al. and Osterholm et al.: see ); *Authors included only studies on FluMist ® live-attenuated vaccine, assessing only culture-confirmed symptomatic influenza cases; §, Authors included only studies on vaccines licensed in USA, assessing RT-PCR or culture-confirmed influenza cases. Estimates on PIV from RCTs were re-elaborated from Table 2. Estimates on PIV from observational studies were re-elaborated (to compare results with other meta-analyses, we included only outpatient subjects). All estimates in the table are referred to children only; **To be comparable, analyses were re-elaborated from analyses 8.6 and 9.6 (Colombo 2001 study was added to the meta-analysis and Vesikari 2006 data were only included once; after two doses—see for details and references); It was not possible to report data on safety outcomes for children/adolescents because of the heterogeneity in their presentation in the included studies (see text for details).

Table 2. Meta-analyses on influenza vaccines for healthy adults

	Villari13	Jefferson2	Osterholm9
End date of the search (mm/yy)	12/2002	06/2010	02/2011
Participant’s age-range (years)	15–65	16–65	All ages §
Included study designs	RCTs	RCTs	RCTs (Obs.) §
Funding source	Public institutions	None	Not-for-profit foundation
Laboratory-confirmed cases
- Overall ψ
N. data sets (sample)	25 (18,920)	23 (37,748) γ	11 (35,215) §
Vaccine efficacy, % (95% CI)	63 (13; 71)	61 (52; 69)	49 (16; 69) §
- Live-Attenuated (LAV)
N. data sets (sample)	7 (6661) Ω	6 (8524)	3 (3054) §
Vaccine efficacy, % (95% CI)	53 (35; 66)	62 (45; 73)	32 (-2; 55) §
- Parenteral Inactivated (PIV)
N. data sets (sample)	18 (12,259) Ω	17 (31,265) γ	8 (32,161) §
Vaccine efficacy, % (95% CI)	67 (55; 76)	61 (48; 70)	59 (51; 67) §
- Aerosol Inactivated (AIV)
N. data sets (sample)	0 (0)	0 (0)	0 (0)
Vaccine efficacy, % (95% CI)	–	–	–
Clinically-confirmed cases
- Overall ψ
N. data sets (sample)	49 (46,022)	35 (34,898) γ	NA
Vaccine efficacy, % (95% CI)	22 (16; 28)	19 (6; 30)	NA
- Live-Attenuated
N. data sets (sample)	8 (13,964) Ω	6 (12,688)	NA
Vaccine efficacy, % (95% CI)	15 (8; 23)	10 (6; 16)	NA
- Parenteral Inactivated
N. data sets (sample)	35 (30,121) Ω	25 (25,065)	NA
Vaccine efficacy, % (95% CI)	23 (15; 30)	20 (11; 29)	NA
- Aerosol Inactivated
N. data sets (sample)	6 (1937) Ω	4 (1674)	NA
Vaccine efficacy, % (95% CI)	55 (27; 72)	42 (17; 60)	NA
Mild/moderate adverse events
- Local harm*
N. data sets (sample)	NA	LAV: 3 (4921); PIV: 14 (6833); AIV: 3 (565)	NA
Increase in Risk, % (95% CI)	NA	LAV: 56 (31; 87); PIV: 211 (108; 366); AIV: 15 (-12; 50)	NA
- Fever
N. data sets (sample)	NA	LAV: 3 (713); PIV: 8 (2775); AIV: 0 (0)	NA
Increase in Risk, % (95% CI)	NA	LAV: 28 (-57; 279); PIV: 17 (-20; 72); AIV:–	NA
-Systemic, any
N. data sets (sample)	NA	LAV: 5 (1018); PIV: 8 (2603);AIV: 3 (565)	NA
Increase in Risk, % (95% CI)	NA	LAV: 40 (-18; 138); PIV: 29 (1; 64); AIV: -17 (-46; 27)	NA
Serious adverse events
N. studies (sample)	NA	NR**	NA
Increase in Risk, % (95% CI)	NA	–	NA

RCT, Randomized clinical trial; Obs., Observational studies; CI,Confidence Interval; NA,Not assessed; ψ Some meta-analyses only reported separated estimates for PIV or LAV. In these cases, the overall estimate of efficacy was derived combining PIV and LAV summary estimates using a generic inverse variance approach, with a random-effect method. * The outcome “Local harm” includes: local soreness (for PIV); local - any or highest symptom (for LAV and AIV). Ω Villari et al. included both PIV and LAV into a single meta-analysis. Thus, to avoid placebo data replication, they had to split several placebo arms that were in common for both PIV and LAV arms into the same study. If PIV and LAV would have been separately meta-analyzed, as in Jefferson et al. and Osterholm et al. studies, splitting placebo data was unneeded, and the overall totals would have been the followings: LCC-LAV n = 8761; LCC-PIV n = 14,359; CCC-LAV n = 16,064; CCC-PIV n = 32,433; AIV-CCC n = 2149). γ The total sample for PIV was recomputed due to an error (60 subjects missed into a placebo arm) in PIV data extraction, and because several placebo arms had to split to avoid data replication (see the above point and the for more details). ** Not reported: narrative review. § Observational studies on adults were searched but not found. Authors included only studies on vaccines licensed in USA, assessing RT-PCR or culture-confirmed influenza cases. Estimates on LAV from RCTs were re-elaborated from Table 3. All estimates reported in the table are referred to adults only.

For LCC, despite some diversities in outcome definition, study inclusion criteria (and discrepancies in their application), and the dates of search end (which have been detailed in the ), the overall vaccine efficacy coming from RCTs (considering all vaccine types) was relatively high, ranging from 59% to 75% across all meta-analyses.

Stratifying by type of vaccine, small differences could be noted for parenteral inactivated vaccines (PIV), the efficacy of which ranged from 59% to 65%, with the exception of US vaccines (46%). The latter estimate, however, was based upon a relatively small sample (n = 786). Some more variability was observed for LAV, but with the exception of the serological-confirmed outcome of the oldest meta-analysis, all other results were consistent, with vaccine efficacy ranging from 72% to 83%. Except for PIV licensed in USA, both PIV and LAV were always able to confer substantial protection against LCC. Notably, when only observational studies were considered, vaccine efficacy was lower than that from RCTs in most cases, and it fell below 50% in one analysis, which was based on one small study on LAV.

The results derived from an overarching meta-analysis considering all studies that were included in at least one meta-analysis appear in and . PIV efficacy remained around 60%, with small or any change over time and adopting various inclusion criteria (data not shown). LAV efficacy did not substantially change, being around 68%, with small variation over time. Only the choice of more stringent criteria for LCC increased the efficacy to 78%.

Probably because of the larger sample size, and/or because the estimates are less-specific, which precludes higher point estimates, the agreement among meta-analyses was even higher when CCC were evaluated: the overall any-type vaccine or LAV efficacy ranged between 33% and 38%; for PIV, estimates ranged from 33% to 45%. The estimates from observational studies agreed with those from RCTs. In the overarching meta-analysis including all studies ( and ), the summary estimates of vaccine efficacy were 33% (95% Confidence Interval – CI: 29–38%) and 38% (95% CI: 28–47%) for LAV and PIV, respectively. As for LCC, no substantial differences were observed with varying inclusion criteria and over time.

In addition to the traditional outcomes (LCC and CCC) two meta-analyses also evaluated vaccine efficacy to prevent acute otitis media., For this outcome, the discrepancy between the two reviews was large, especially on the number of included studies (details into the ). Despite these differences, however, both analyses reported a significant efficacy of LAV (73% by Manzoli et al.; 59% by Jefferson et al.), and no statistically significant effect of PIV on acute otitis media (32% and -14%, respectively). A single observational study also showed a non-significant trend in favor of PIV, although the sample was probably too small to achieve enough power. Notably, one recent study (sponsored by MedImmune) did not perform a formal meta-analysis but simply pooled (summed up) the results of eight RCTs evaluating the efficacy of LAV in preventing AOM in children aged 6–83 mo. This pooled analysis included a total of 24,046 subjects and showed a significant reduction of AOM cases in LAV recipients ranging from 61.7% to 91.4% for children aged 24–83 mo, from 47.5% to 77.8% for children aged 6–24 mo.

The safety of influenza vaccination in healthy children was evaluated in one meta-analysis only. Another meta-analysis was not considered due to the inclusion of both healthy and subjects with underlying diseases and estimates for the two groups could not be separated. The authors of the included meta-analysis reported that “in spite of the large amount of data available (on PIV), particularly for temperature reactions, we could not carry out meta-analysis for any outcome because of the heterogeneity in the presentation of outcomes in the included studies.” As for other influenza vaccines or other medical interventions (i.e., antibiotics), evidence of reporting bias was reported for LAV, and authors highlighted the need for a complete safety outcomes disclosure, into a standardized format. Although no quantitative estimates are available, however, some brief comments on the safety of influenza immunization for children could be attempted. Vaccination seems to be associated with higher rates of mild or moderate adverse events. We also extracted all data on serious vaccine-related adverse events from both RCTs and observational studies (). No deaths were observed, and in the few studies in which some serious adverse events were reported, the number of events among vaccinated and unvaccinated participants were 23 (among 20,289 participants) vs. 7 (among 8451 participants), respectively, a difference that is not beyond chance. Importantly, one recent study (sponsored by MedImmune) used no meta-analytic technique but pooled the results of 20 RCTs to evaluate the safety of Ann Arbor strain LAV in children aged 2–17 y, finding no evidence of an increase in any potential vaccine-related serious adverse event in LAV recipients.

With regard to the interpretation of the results or specific stratified analyses or outcomes, besides AOM, no author outlined a substantial improvement or worsening of vaccine efficacy over time, and there was agreement across meta-analyses on a significant efficacy of influenza vaccination, on a higher efficacy of LAV vs PIV, and on the scarcity of data on children aged < 2 y. In particular, only two data sets from one study (including 786 subjects) were available on PIV efficacy to prevent LAV for children under 2 y (overall efficacy 45%), while six data sets from four studies evaluated LAV efficacy in young children (6–36 mo), with a summary efficacy of 74%, based upon 10,001 subjects (). However, LAV is not recommended for children aged < 2 y, while PIV is recommended in several countries., In addition, very limited data are available on the safety profile of both vaccines. Therefore, although there may be few reasons to believe that data from older children cannot be transferred to younger children, more evidence is strongly needed on children aged < 2 y.

As a final remark, Manzoli et al. reported that vaccination efficacy in preventing CCC substantially improved (from 36% to 61%) when former USSR studies were excluded. The authors suggested that the larger average sample size of USSR studies (20,470 vs. 478 of non-USSR trials) might be a potential explanation for the observed finding, because careful and standardized criteria are needed to diagnose CCC and diagnoses may have been more specific in the smaller non-USSR studies. Jefferson et al. also highlighted methodological flaws of the included Russian studies (), and performed several sensitivity analyses excluding Russian studies. When these were not included, both meta-analyses of RCTs and cohort studies showed substantial increases in the overall efficacy of PIV in preventing CCC (from 36% to 66% and from 45% to 74%, respectively). Such an issue was no more noticeable in the overarching meta-analysis for LAV, while it was still apparent for PIV (Figs. 3 and 4): when Russian studies were excluded, vaccine efficacy to prevent CCC rose to 58% (95% CI: 15–79%). The sample, however, was reduced to 2087 individuals (data not shown).

Meta-analyses on seasonal vaccination for healthy adults

Three published meta-analyses evaluated the efficacy of influenza vaccines for healthy adults,,, while only one of them also assessed harms (Table 2). As mentioned before, the most recent meta-analysis, by Osterholm et al., evaluated only vaccines licensed in USA to prevent RT-PCR or culture-confirmed influenza infections. All meta-analyses included RCTs only and compared vaccines vs. placebo or no intervention. All meta-analyses were either funded by not-for-profit institutions, or had received no funding. The detailed list of included studies and inclusion criteria for each meta-analysis is reported in the . As noted also by Osterholm et al., in the meta-analysis by Jefferson et al. we observed some major discrepancies between inclusion criteria and their application: besides minor “physiological” issues, five large data sets from four RCTs,- published from 2006 to March 2010 into highly-reputed journals were not included nor mentioned in the review. The potential impact of study inclusions/exclusions is discussed separately for each outcome.

Table 3. Meta-analyses on influenza vaccines for the elderly

	Gross1	Vu14	Jefferson3	Osterholm9
End date of the search (mm/yy)	Not reported(published in 1995)	12/2000	10/2009	02/2011
Participant’s age-range (years)	≥ 65	≥ 65	≥ 65	All ages §
Included study designs	Obs.	RCTs, Obs.	RCTs, Obs.	RCTs *, Obs.
Funding source	Public institution	NR	Public institutions	Not-for-profit foundation
Laboratory-confirmed cases
- Parenteral Inactivated
N. data sets (sample)	NA	NA	RCTs: 3 (2217)Obs.: 10 (20,190) φ	Obs.: 2 (395) §
Vaccine efficacy, % (95% CI)	NA	NA	RCTs: 58 (34; 73)Obs.: 41 (-15; 70) φ	Obs.: 63 (28; 81) §
Clinically-confirmed cases
- Parenteral Inactivated
N. data sets (sample)	23 (9043)	RCTs and Obs.: 3 (6271) Ω	RCTs: 4 (6894)Obs.: 37 (46,239) φ	NA
Vaccine efficacy, % (95% CI)	56 (39; 68)	RCTs and Obs: 35 (19; 47) Ω	RCTs: 41 (27; 53)Obs.: 26 (13; 38) φ	NA
Hospitalization for influenza or pneumonia
- Parenteral Inactivated
N. data sets (sample)	9 (24,324)	Obs.: 9 (> 446,336) Ω	Obs.: 8 (949,215) β	NA
Vaccine efficacy, % (95% CI)	48 (28; 65)	Obs.: 33 (27; 38) Ω	Obs.: 27 (21; 33) β	NA
Mortality for any cause
- Parenteral Inactivated
N. data sets (sample)	30 (30,028)	Obs.: 4 (163,087) Ω	RCTs: 1 (699)Obs.: 7 (742,575) β	NA
Vaccine efficacy, % (95% CI)	68 (56; 76)	Obs.: 50 (45; 56) Ω	RCTs: -2 (-872; 89)Obs.: 47 (39; 54) β	NA
Mild/moderate adverse events
- Local pain
N. data sets (sample)	NA	NA	4 (2560)	NA
Increase in Risk, % (95% CI)	NA	NA	256 (161; 387)	NA
- Fever
N. data sets (sample)	NA	NA	3 (2519)	NA
Increase in Risk, % (95% CI)	NA	NA	57 (-8; 171)	NA
-Systemic, any
N. data sets (sample)	NA	NA	1 (672)	NA
Increase in Risk, % (95% CI)	NA	NA	75 (-26; 312)	NA
Serious adverse events (Guillain-Barré syndrome)
N. data sets (sample)	NA	NA	4 (> 100 millions) **	NA
Increase in Risk, % (95% CI)	NA	NA	60 (-53; 444)	NA

RCT, Randomized clinical trial; Obs., Observational studies; CI, Confidence Interval; NA, Not assessed; NR, Not reported. * RCTs were searched but none was found including only elderly. Only two out of four studies reported outcome stratified by age, allowing data extraction for subjects aged 64 and over; the other two studies included subjects aged 18 and over, with no stratification. Ω Authors included solely the studies enrolling community-living elderly only; with samples larger than 30; in which the influenza vaccine strain matched the circulating strain. It was not possible to extract the total number of subjects enrolled in the studies evaluating hospitalizations. Cohort and case-control studies were pooled together. § Authors included only studies on vaccines licensed in USA, assessing RT-PCR or culture-confirmed influenza cases. Estimates on LAV from RCTs were re-elaborated from Table 3. All estimates reported in the table are referred to elderly only. φ Results have been re-elaborated combining studies on community-dwelling elderly (analysis 2.1) and elderly from nursing homes, with (analysis 1.1) or without (analysis 1.7) a clear definition of the outcome. Only meta-analyses on cohort studies have been used. β = Adjusted rates of community-dwellers only. ** Re-elaborated from Table 1; the samples were the entire USA population in different seasons plus 21 million subjects from another study.

For LCC, the summary estimates of efficacy for PIV were comparable among the three meta-analyses, ranging from 59% to 67%. In contrast, the efficacy of LAV differed between the two meta-analyses by Villari et al. (53%; 95% CI: 35% to 66%) and Jefferson et al. (62%; 95% CI: 45% to 73%), and that by Osterholm et al. (32%; 95% CI, -2% to 55%). In a comment on Osterholm et al. results, Kerry and Valenciano acknowledged that such a difference is most probably caused by the more restrictive selection criteria for study inclusion used by Osterholm et al. In fact, the latter authors emphasized the need for routine effectiveness studies of presently licensed influenza vaccines with virus-confirmed endpoints, especially RT-PCR diagnosed infections, because culture could miss cases and serology alone would overestimate vaccine efficacy and effectiveness. As shown in the , even some supposedly minor differences in study inclusion criteria resulted in large discrepancies among meta-analyses in the number of included studies. In example, eight data sets on PIV that used control groups receiving influenza B or other vaccines were included by Villari et al. and excluded by Jefferson et al. When we investigated the potential role of study inclusion/exclusion criteria/outcome definition, and time through an overarching meta-analysis (in which we included all studies that were considered in at least one meta-analysis – and), PIV and LAV efficacy did not substantially vary, remaining around 60% and 50%, respectively, with small or any change over time. For both vaccines, stratification by outcome showed that, as compared with cases with cultural and/or serological confirmation (LCC-S), the use of culture-confirmed cases only (LCC-C) lead to lower summary estimates of vaccine efficacy. However, the differences were not significant in all analyses, and when estimates from the same studies providing both LCC-C and LCC-S data were indirectly compared, the summary risk ratios did not substantially differ (). Finally, when we re-computed the results by Jefferson et al., adding the five large data sets that were apparently missed in the search,- both PIV and LAV overall efficacy did not substantially vary (60% and 53%, respectively – data not shown).

Less discrepancy was observed for CCC: both Villari et al. and Jefferson et al. reported a significant although low overall efficacy of LAV (15% and 10%, respectively), and PIV (23% and 20%), and a higher protection ability of aerosol inactivated vaccines (AIV) (55% and 42%, respectively). All authors noted that the sample size of AIV studies, however, was relatively small (n < 2000 for both meta-analyses), and highlighted the need for more research. Expectedly, the overarching meta-analysis did not show substantial variations in efficacy for both vaccines by study inclusion criteria ( and), but an interesting trend over time was observed for PIV: the efficacy tended to decrease with time, and the meta-analysis restricted to the five RCTs published in the last decade failed to show a significant protection of PIV (efficacy 9%; 95% CI: -9%; 23% –Fig. 8).

Concerning mild or moderate adverse events, the Cochrane meta-analysis showed that LAV and PIV were associated with a significantly higher likelihood of local harms (+56% and +211%, respectively) and systemic events (PIV only: +29%). In contrast, AIV did not significantly increase the risk of any of the selected harms. Sparse data were available on vaccine-related serious adverse events, and no quantitative analyses were made. The authors discussed the results of three observational studies on Guillain-Barré syndrome with contrasting results, but focused on the results of one study, which estimated the incidence of vaccine-related Guillain-Barré syndrome as 1.6 extra cases per million vaccinations.

Jefferson et al. expressed some concerns on publication bias (which was also reported for CCC by Villari et al.), and warned against a potential reporting bias of privately sponsored studies.

Meta-analyses on seasonal vaccination for the elderly

We found six meta-analyses which evaluated the efficacy/effectiveness of influenza vaccination in people older than 64 y,,,,,, and only one of them also assessed harms (Table 3). One meta-analysis focused on hospitalizations only and included a subset of studies (n = 8) already considered in the Gross et al. meta-analysis: it is therefore not discussed. Another meta-analysis was excluded because only subjects with underlying chronic diseases were included. All remaining reviews did not consider (or treated separately) studies which included only selected groups of elderly (i.e., affected by a specific disease such as diabetes etc.), as they were interested in the whole population of elderly.,,, One meta-analysis, however, included only community-living elderly. The most recent meta-analysis also adopted restrictive inclusion criteria, as authors evaluated only vaccines licensed in USA to prevent RT-PCR or culture-confirmed influenza infections. All reviews also considered RCTs in addition to observational studies, but only one provided summary estimates for RCTs-RCTs are uncommon because most ethics committees reject experimental study designs for interventions that are recommended, such as influenza vaccination for the elderly. Also, overall only one study was found on LAV (showing a significant 42% vaccine efficacy in preventing RT-PCR/culture confirmed influenza cases), and one study on AIV (which failed to show a significant protection by vaccination), thus all estimates and our discussion only refer to PIV. The funding source was not reported in one meta-analysis, while all others were funded by not-for-profit institutions.,,

Two meta-analyses evaluated vaccine efficacy/effectiveness against LCC., Combining the existing three RCTs (n = 2217), the vaccine was significantly better than placebo (efficacy = 58%; 95% CI: 34% to 73%). The summary estimates from cohort studies however varied: in the Jefferson et al. meta-analysis, which included 10 cohort studies, vaccination was not able to provide a significantly higher protection than no intervention (41%; 95% CI: -15% to 70%), whereas a significant protection (63%; 28% to 81%) was found when we combined the results of the only two studies included in Osterholm et al. meta-analysis that evaluated vaccine effectiveness through a more specific outcome (RT-PCR or culture-confirmed influenza infections only). These studies were both published after the end of the search by Jefferson et al., and thus no meaningful comparison is possible. Notably, the inclusion of LCC based on serology alone (as made in Jefferson et al. review), if any, should have lead to an over- rather than under-estimation of vaccine effectiveness. Therefore, no firm conclusions can be drawn and some uncertainty remains on this important issue. Given that vaccination was found to be significantly effective in preventing the other traditional outcome—CCC (as discussed below)—which is typically characterized by lower efficacy estimates, the LCC finding in Jefferson et al. is to some extent paradoxical.

Concerning CCC, all reviews showed a significant protection conferred by vaccination. The four RCTs showed a summary efficacy estimate of 41%, while the overall effectiveness from meta-analyses of cohort studies ranged from 56% to 24%. Eleven data sets (some of which with large samples) were published after the meta-analysis by Gross et al., and only three of these showed a significant effectiveness by vaccination. Thus, the Jefferson et al. results could simply be more updated, and no discrepancy really exists. As regards Vu et al., effectiveness was relatively low (35%), due to restrictive inclusion criteria (studies enrolling community-living elderly only; with samples larger than 30; in which the influenza vaccine strain matched the circulating strain), such an estimate was based on 3 studies only with different designs: one RCT, one non randomized clinical trial, and one cohort study. Moreover, when we performed a meta-analysis restricted to the seven data sets that were published after 2000 (the year of the search end by Vu et al.), the summary vaccine efficacy was similar to the overall one reported by Jefferson et al. (31%; 95% CI: -1%; 53%—). Therefore, overall, the summary estimate by Jefferson et al. could be considered the most reliable one. Although the effectiveness of vaccine in preventing CCC in the elderly is modest (24%), it matches quite well that of the adults (19–22%) and there are no reasons to believe that it should be relevantly higher.

Three meta-analyses evaluated also other outcomes than CCC and LCC.,, With regard to hospitalizations due to influenza or pneumonia, PIV was significantly better than placebo in all meta-analyses, however the summary estimates varied, ranging from 48% to 27%. Besides the more selective inclusion criteria discussed above, both case-control and cohort studies were included by Vu et al., thus their results could not be compared with those from other reviews. Gross et al. used unadjusted estimates, mostly included elderly from nursing homes and also included one RCT. In fact, when the Gross et al. results are compared with those of the stratified meta-analysis of cohort studies in nursing homes by Jefferson et al., using unadjusted estimates, the summary estimates are practically identical (respectively, 48% and 49%, with similar confidence limits). However, eight community cohort studies that were published after the Gross et al. meta-analysis provided adjusted rates of hospitalizations due to influenza and pneumonia, and showed a lower—and probably more reliable—overall effectiveness of vaccination: 27% (95% CI: 21–33%).

Mortality was evaluated by three meta-analyses.,, Gross et al. only considered all-cause mortality, while Jefferson et al. and Vu et al. also analyzed mortality due to influenza or pneumonia. Surprisingly, the estimates of vaccine effectiveness in preventing cause-specific mortality were similar to those of all-cause mortality, despite influenza-related mortality accounts for only a modest portion of total mortality. This inconsistency might be due to selection bias, as discussed in more detail below. In any case, given that the estimates of the two outcomes were similar in both reviews we therefore discuss only all-cause mortality. The only RCT that assessed this outcome failed to show a significant protection by vaccination. Notably, however, only four deaths occurred during the season and the sample was clearly underpowered to detect any effect of vaccination. When observational studies were combined, all meta-analyses found that vaccines were able to significantly reduce deaths for all causes, with summary estimates of effectiveness ranging from 68% to 47%. Gross et al. included most studies with unadjusted rates from nursing homes, while Jefferson et al. could also meta-analyze seven cohort studies – all published after Gross et al. – that used adjusted rates and included community-dwelling elderly. Also, Vu et al. results cannot easily be compared with the other meta-analyses, because of different inclusion criteria and the inclusion of both cohort and experimental studies into the analyses. In any case, when the five large data sets (total n = 585,633) that were published after the end of the search by Vu et al. were meta-analyzed separately, the summary vaccine efficacy was 46% (95% CI: 36–55%, Fig. 11). This finding suggests that, besides inclusion criteria, the observed differences across meta-analyses might simply be due to the difference in time among meta-analyses. As a matter of fact, currently the estimate for the overall effectiveness of vaccination in preventing deaths in the Cochrane review (47%; 95% CI: 39–54%), which incorporates the above recent studies and is thus based upon a much larger sample, may be the most reliable one. Even this estimate, however, is likely to be grossly inflated due to unaccounted confounding, sponsorship bias, selective reporting and other biases (see further discussion below).

Concerning mild or moderate adverse events, the Cochrane meta-analysis of RCTs showed that PIV was associated with a significantly higher likelihood of local pain than placebo, but failed to show significant differences in the rates of fever and any systemic adverse event. Concerning serious adverse events, Jefferson et al. reported the results of four large data sets from three surveillance studies on the association between vaccination and Guillain-Barré syndrome. When the results of these studies were combined, the overall estimate of risk was not nominally significant (odds ratio: 1.60; 95% CI: 0.47–5.44). The authors concluded that “safety does not appear to be a particular problem: the public health safety profile of the vaccines is acceptable.”

The impact of immunization is theoretically expected to be higher in the presence of a good antigenic match between the epidemic and the vaccine strain. Osterholm et al. did not quantitatively address this issue, and Vu et al. included only studies with a good matching. When Gross et al. stratified the analyses by matching, they found a significant effectiveness of vaccination even in seasons in which the circulating strain was a drift variant of the vaccine strain. Finally, few direct comparisons were possible between matching and non-matching seasons in Jefferson et al., mainly due to the scarce available data from seasons with poor matching. The available evidence, however, suggest substantial differences only for the outcomes “hospitalizations due to influenza or pneumonia” and “all-cause mortality”: unsurprisingly, vaccine effectiveness was substantially greater in seasons with good matching, either in nursing homes or in community, from unadjusted or adjusted estimates (only adjusted estimates in community studies for mortality). Notably, a cluster randomized trial published in 2010, in which data on frail elderly from five influenza outbreaks were combined, concluded that influenza vaccine can be effective against disease and severe outcomes despite incomplete vaccine match.

Concerning other factors potentially affecting reported vaccine effectiveness, no meta-analysis on the elderly discussed explicitly publication bias, and only Jefferson et al. investigated the potential impact of private sponsorship. They stated that “government funded studies were less likely to have conclusions favoring the vaccines (odds ratio: 0.45; 95% CI: 0.26 to 0.90),” but they did not clarify which particular outcome this conclusion referred to.

With regard to the interpretation of the results, both Gross et al. and Vu et al. concluded that influenza vaccines are effective in preventing influenza cases, hospitalizations and death in the elderly., On the contrary, Osterholm et al. concluded that “evidence for protection in adults aged 65 years or older is lacking,” and Jefferson et al. stated that “the available evidence is of poor quality and provides no guidance regarding the safety, efficacy or effectiveness of influenza vaccines for people aged 65 years or older.” The conclusions by Osterholm et al. are apparently due to their choice of restrictive inclusion criteria (and did not take into account one study reporting a significant efficacy on LAV), while those by Jefferson et al. may be influenced in particular by the evidence of potential biases identified in their meta-analysis. In summary, Jefferson et al. affirmed that (1) “evidence from RCTs is scant and badly reported”; (2) “evidence from non-RCTs was of low quality”; (3) “vaccine effectiveness shows an implausible sequence: the vaccines are apparently ineffective in the prevention of LCC, CCC, pneumonia, hospital admissions or deaths from any respiratory disease but are effective in the prevention of hospital admission for influenza and pneumonia and in the prevention of deaths from all causes.” According to Cochrane quality assessment, 18 studies were at low risk of bias; 31 at medium risk; and 14 at high or very high risk of bias. Subgroup analysis by quality showed that low-risk of bias studies still lead to nominally significant—although modest (22%; 95% CI: 6%; 35%)—vaccine effectiveness for CCC (Comparison 16). Jefferson et al. also claimed that “the 47% reduction in risk of all-cause mortality in elderly community-dwellers observed in this review, exceeds by far the estimated possible impact of influenza on winter-seasonal mortality of 5% in an average season.” It is very likely that observational studies on elderly are likely to be affected by unaccounted confounding and selection biases, and thus these studies probably over-estimate vaccine effectiveness. However, even if the actual reduction in risk, especially in seasons with low rates of infection and with poor matching, would be as low as 2–3% rather than 47%, this does not imply that vaccination is ineffective. Indeed, relative risk reductions of < 5% are extremely difficult to prove beyond doubt with observational studies and would require the conduct of very large pragmatic RCTs.

Meta-analyses on pre-pandemic vaccines (H5N1) and pandemic 2009 (H1N1) vaccines

One meta-analysis and one systematic review evaluated the immunogenicity and harms of “Avian” influenza H5N1 vaccines. Manzoli et al. included only RCTs evaluating all vaccines (including a total of 58 data sets with more than 10,000 subjects), while Prieto-Lara et al. considered also non-randomized studies, however evaluating only licensed vaccines (including a total of 17 data sets with 6476 subjects). Both stopped their search during 2009 and tried to identify the best formulation among several doses of vaccines containing either no adjuvant, adjuvants based on aluminum or oil-in-water emulsion-based adjuvants. In addition to traditional head-to-head comparisons, Manzoli et al. also synthesized the evidence using multiple treatments meta-analysis that can incorporate the evidence from all comparisons of different treatments within a single analysis, allowing a better appreciation of the relative merits of each treatment within a common analytical framework. Despite such differences, the conclusions were in agreement: the best available option in a pandemic is currently represented by oil-in-water adjuvanted vaccines, administered in two doses containing each 3.8–6 μg of hemagglutinin antigen. These formulations were more prone to cause adverse reactions, but they were the only preparations showing acceptable immunogenicity rates (≥ 70%), so the trade-off may be considered acceptable. Finally, both reviews found no serious vaccine-related adverse events, and concluded that all tested vaccines had an acceptable safety profile. In the absence of studies on clinical outcomes, however, the efficacy/effectiveness of the vaccine cannot be taken for granted.

Also the two meta-analyses on pandemic influenza 2009 (H1N1) vaccines come to substantially similar conclusions:, after two doses, all split/subunit inactivated vaccines were able to confer adequate seroprotection (≥ 70%); after one dose only, all split/subunit vaccines were highly immunogenic in adults and adolescents, while only high doses of non-adjuvanted vaccines or oil-in-water adjuvanted formulations (even at doses as low as 1.8 μg of hemagglutinin antigen) showed acceptable results in elderly and children., The latter preparations (oil-in-water emulsion-based) were also more immunogenic at any dose. As regards harms, the findings were similar to those on H5N1 vaccination: both meta-analyses found a higher (and high) frequency of mild or moderate adverse events by oil-in-water adjuvants, although they concluded that such a lower tolerability could be acceptable in a pandemic, and a low rate of serious adverse events (three, all solved in 10 d, out of 22,826 vaccinated subjects). Such conclusions were based upon a large set of meta-analyses including a total of 52 data sets from 17 clinical trials (17,921 subjects), or 76 data sets from 18 RCTs (enrolling 16,725 subjects) and 18 data sets from 14 clinical trials (2495 subjects). No formal meta-analysis has addressed clinical efficacy/effectiveness. However, scattered large observational studies evaluating clinical outcomes seem to confirm the favorable results on immunogenicity and tolerability.- Also, one recently published computer simulation model concluded that 2009(H1N1) vaccination for children and adults is cost-effective compared with other preventive health interventions under a wide range of scenarios. The lack of formal clinical efficacy/effectiveness meta-analyses, however, is a concern that cannot be dismissed.

For both H5N1 and 2009 pandemic (H1N1) vaccines, Manzoli et al. highlighted the need for more RCTs (especially if publicly sponsored, given than most trials were sponsored by manufacturing companies) comparing vaccines including different adjuvants, and reported a high potential for publication bias, in particular for the meta-analysis on H1N1 vaccination., In fact, after 2.5 y from the pandemic start, only 21 RCTs evaluating influenza 2009 (H1N1) vaccines were published out of 73 RCTs that were registered in trial registries (68 of them had also been completed by June 30, 2011).

Methods

Aims and search strategy

The main purpose of this umbrella review is to systematically compile the main findings, including estimates of effects for major outcomes for all age-classes and influenza vaccines from published meta-analyses. We aimed to juxtapose these results for an overall comparative evaluation of the data. Furthermore, we have tried to evaluate whether any substantial differences in meta-analyses findings (or their interpretation) exist, and, if so, which could be the most reasonable explanations, e.g., inclusion or exclusion of specific studies, or evolution of the effects over time with differences in earlier vs. more recent studies. We focused on healthy participants derived from the general population of different age-groups, excluding meta-analyses that focused on people with specific diseases or comorbidities.

Meta-analyses or systematic reviews evaluating influenza vaccine safety and/or efficacy/effectiveness in humans were retrieved through searches in MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews with no language restriction (last update December 1, 2011). Search terms were “influenza,” “vaccine* or vaccination,” and “meta-analysis or pooled analysis or systematic review” in all fields. The bibliographies of all relevant articles including reviews were reviewed for further eligible references. We included meta-analyses of either randomized controlled trial (RCTs) and observational studies, on any type of influenza vaccine, assessing protection vs. naturally occurring infection.

Eligible meta-analyses and outcomes

We focused on meta-analyses evaluating clinical outcomes and/or harms. When information on clinical outcomes was not available (as in the case of H5N1 and 2009 H1N1 vaccines), meta-analyses on immunogenicity outcomes were also examined. We considered consistently the two traditional clinical outcomes—laboratory confirmed cases (LCC) and clinically confirmed cases (CCC) —that have been sometimes also defined as “efficacy” and “effectiveness,” respectively, based upon their different specificity (much lower for CCC). One should be aware that this distinction has recently been challenged, because both outcomes are extracted from RCTs, while in classic epidemiology efficacy refers to the relative risk reduction attributed to vaccination as estimated from a RCT, and effectiveness refers to the same measure of effect from an observational study. Additional outcomes considered were: effect on acute otitis media (for children), hospitalizations (for adults and elderly), and mortality (for elderly). We excluded meta-analyses that focused on specific topics or hypotheses related to influenza vaccine efficacy/effectiveness or safety (e.g., gender differences, or other postulated effect modifiers) without providing overall estimates of vaccine impact on any eligible outcomes.-

Funding, potential biases and interpretation

For each meta-analysis, we also reported whether it stated sources of funding (in particular public/governmental and industry). We also recorded potential biases identified by the authors of each meta-analysis (including confounding, selection and information biases in the included studies, as well as sponsorship, publication, and selective reporting biases in the accumulated available evidence). Finally, we recorded the interpretation of the authors for the overall results and juxtaposed these final interpretations and conclusions across meta-analyses on the same age group.

Comparative evaluation of included/excluded studies and overarching meta-analyses

Different published meta-analyses on the same age group and type of vaccine may reach different conclusions, because they vary on which trials they include or exclude. This may be due to differences in eligibility criteria, non-sensitive literature searches, differences in timing of the meta-analyses (more recent papers would include more trials), or other reasons. In order to probe these possibilities for each major age group and type of vaccine, we juxtaposed the included studies in each published meta-analysis and recorded the apparent reasons for the non-inclusion/exclusion of each trial from each meta-analysis where it had not been used in the summary effect calculations. We also performed overarching meta-analyses: these are re-analyses for each age group and vaccine that included all the trials that had been included in at least one published meta-analysis of the same age group and vaccine type. Data were synthesized using the risk ratio metric and using a random effects model. Heterogeneity metrics are also provided (chi-square based Q test and I-squared metric), but should be interpreted cautiously in the presence of few studies per meta-analysis. All calculations were made in RevMan 5.0 (Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2008). Studies were ordered chronologically in the forest plots, so as to discern any strong evidence for changes in effect sizes over time. Formal cumulative meta-analyses are also available from the corresponding author. Any obvious data errors in the previous meta-analyses were also corrected in the re-analysis process.

Conclusions

Most influenza vaccines have been shown to confer some protection against naturally acquired infection and no evidence for major harms has emerged. In adults and children, the efficacy/effectiveness of current seasonal vaccines was generally high for laboratory-confirmed cases (especially for LAV in children aged 2–17 y), and modest for clinically-confirmed cases and for the elderly. For children aged < 2 y, while several studies support LAV efficacy, the evidence on PIV efficacy and safety data remains scarce. Some of the outcomes have results that seem incongruent when juxtaposed, e.g., the huge impact on all-cause mortality in the elderly as opposed to far more modest effects against CCC. Data on harms are reassuring, and there is no evidence that Guillain-Barré syndrome should be a concern. However, the overall quality of the harms data are suboptimal, and this information seems to suffer from lack of standardized definitions and data collection and inconsistent and potentially selective reporting. Pre-pandemic H5N1 and 2009 pandemic H1N1 vaccines in particular can achieve satisfactory immunogenicity, when given in proper doses and formulations, but no meta-analysis has addressed H1N1 vaccination impact on clinical outcomes.

Although we identified several discrepancies among meta-analyses on seasonal vaccines for children and elderly, it is possible to conclude that most seasonal influenza vaccines showed statistically significant efficacy/effectiveness, the magnitude of which, however, largely varied.

The use of influenza vaccines is recommended worldwide, and this makes the conduct of pragmatic RCTs with hard clinical outcomes difficult in some settings. Cost-effectiveness issues have to be properly re-assessed in times of economic recession. We certainly embrace the request by Osterholm et al. for a new generation of more highly effective seasonal vaccines. There is also still an unmet need for adequately powered publicly-funded RCTs on both young children and elderly. Ethics committees should acknowledge this need and allow the conduct of well-planned experimental studies in particular in children and in people aged 65 y and older. Finally, these RCTs should not only be registered in public trial registries, but also promptly published without selective analysis and reporting biases affecting the results or their interpretation.