Real-world effectiveness of BNT162b2 mRNA vaccine: a meta-analysis of large observational studies.

This paper aims to summarize through meta-analyses the overall vaccine effectiveness of the BNT162b2 mRNA vaccine from observational studies. A systematic literature search with no language restriction was performed in electronic databases to identify eligible observational studies which reported the adjusted effectiveness of the BNT162b2 mRNA vaccine to prevent RT-PCR confirmed COVID-19. Meta-analyses with the random-effects model were used to calculate the pooled hazard ratio (HR) and pooled incidence rate ratio (IRR) at 95% confidence intervals, and the vaccine effectiveness was indicated as (pooled HR - 1)/HR or (pooled IRR - 1)/IRR. Nineteen studies were included for this meta-analysis. The meta-analysis revealed significant protective effect against RT-PCR confirmed COVID-19 ≥ 14 days after the first dose, with vaccine effectiveness of 53% (95% confidence interval 32-68%), and ≥ 7 days after the second dose, with vaccine effectiveness of 95% (95% confidence interval: 96-97%). Despite its effectiveness, reporting vaccine safety data by relevant stakeholders should be encouraged as BNT162b2 mRNA is a new vaccine that has not gained full approval. There have been limited data about vaccine effectiveness among immunocompromised patients; thus, the vaccine should be used cautiously in this patient population.

Introduction

The global rollout of vaccines against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) offers a glimmer of hope toward ending the coronavirus disease 2019 (COVID-19) pandemic. As of the time of writing, there have been with more than 1.7 billion people worldwide received at least one dose of any COVID-19 vaccine, and over 790 million people worldwide are fully vaccinated (Our World in Data 2021).

The phase 3 randomized controlled trial of the BNT162b2 mRNA vaccine against SARS-CoV-2 demonstrated the efficacy of 95% in preventing symptomatic COVID-19, which has led to the emergency conditional approval of the vaccine in many countries (Polack et al. 2020). However, it should be noted that the clinical trial was performed in a highly controlled setting that may not simulate the real-world mass rollout of COVID-19 vaccines.

Therefore, it is imperative to determine the population-level vaccine effectiveness from the mass vaccination campaigns and to report data on the safety aspects of vaccines. This paper aims to summarize through meta-analyses the overall effectiveness of the BNT162b2 mRNA vaccine from large observational studies, which could be important to inform the development of the public health policy related to mass vaccination.

Methods

A systematic literature search with no language restriction was performed in electronic databases, including PubMed, Google Scholar, Scopus, and preprint servers (medRxiv, Research Square, SSRN), to identify eligible studies published up to June 05, 2021. The search strategy was built based on the following keywords and MeSH terms: “BNT162b2”, “Pfizer”, “BioNTech”, “mRNA vaccine”, “mRNA vaccination”, and “effectiveness”. The reference lists of relevant articles were also reviewed to retrieve additional studies. Two investigators (CSK and SSH) independently performed the literature screening to identify eligible studies.

Studies eligible for inclusion were observational studies of any design (case–control, case–cohort, and prospective cohort), which reported the effectiveness of the BNT162b2 mRNA vaccine to prevent reverse transcription–polymerase chain reaction (RT-PCR) confirmed COVID-19 (through comparison between vaccinated and unvaccinated individuals) and adjusted for covariates. For two or more studies which utilized the same data source for their investigations on vaccine effectiveness, we included the one that performed analysis on the latest record. We excluded randomized trials, studies that reported unadjusted effectiveness estimates, studies that reported only non-specific outcomes such as COVID-19-related mortality or COVID-19-related hospitalization, studies where RT-PCR did not use to confirm the diagnosis of COVID-19, and studies that reported vaccine effectiveness against a specific variant(s) of SARS-CoV-2.

Our outcome of interest, namely vaccine effectiveness, is defined as a relative reduction in RT-PCR risk confirmed COVID-19 in vaccinated individuals compared with unvaccinated individuals (Weinberg and Szilagyi 2010). Each included study was independently evaluated by two investigators (CSK and SSH), who also extracted the study characteristics. Study characteristics extracted had the first author’s surname, study design, country, sample population, number of participants, the incidence of COVID-19 in both vaccinated and unvaccinated individuals, and adjusted vaccine effectiveness estimates and covariates adjusted. Two investigators (CSK and SSH) assessed the quality of included observational studies using the Newcastle–Ottawa Scale, with a score of > 7 indicating high quality (Wells et al. 2013).

Meta-analyses with the random-effects model were used to calculate the pooled hazard ratio (HR), pooled incidence rate ratio (IRR), or pooled odds ratio (OR) at 95% confidence intervals, comparing the incidence of RT-PCR confirmed COVID-19 in vaccinated participants relative to unvaccinated participants, when there were three or more studies reporting the same type of effect measure (either HR, IRR, or OR). The vaccine effectiveness was indicated as (pooled HR − 1)/HR, (pooled IRR − 1)/IRR or (pooled OR − 1)/OR, together with a 95% confidence interval. We examined the heterogeneity between studies using the I2 statistics and the χ2 test, with significant heterogeneity set at > 50% and P < 0.10. All analyses were performed using Meta XL, version 5.3 (EpiGear International, Queensland, Australia).

Results

Our literature search yielded 712 abstracts. After deduplication and application of the eligibility criteria, 38 relevant articles were shortlisted for inclusion through full-text examination (Fig. 1). Of these, 19 studies were excluded since they either did not report vaccine effectiveness, reported non-specific outcomes such as COVID-19-related mortality and COVID-19-related hospitalization, or reported unadjusted effectiveness estimates. Therefore, 19 studies (Angel et al. 2021; Björk et al. 2021; Cabezas et al. 2021; Chung et al. 2021; Dagan et al. 2021; Emborg et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Gras-Valentí et al. 2021; Haas et al. 2021; Hall et al. 2021; Lopez Bernal et al. 2021; Mason et al. 2021; Monge et al. 2021; Pritchard et al. 2021; Regev-Yochay et al. 2021; Shrotri et al. 2021; Swift et al. 2021; Thompson et al. 2021) were included for this meta-analysis; 12 studies (Chung et al. 2021; Dagan et al. 2021; Emborg et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Gras-Valentí et al. 2021; Haas et al. 2021; Lopez Bernal et al. 2021; Mason et al. 2021; Monge et al. 2021; Regev-Yochay et al. 2021) were retrospective in design with seven database reviews (Dagan et al. 2021; Emborg et al. 2021; Glampson et al. 2021; Haas et al. 2021; Mason et al. 2021; Monge et al. 2021; Swift et al. 2021), three retrospective case–control studies (Chung et al. 2021; Gras-Valentí et al. 2021; Lopez Bernal et al. 2021), and two retrospective cohort studies (Fabiani et al. 2021; Regev-Yochay et al. 2021); the remaining seven studies (Björk et al. 2021; Cabezas et al. 2021; Hall et al. 2021; Menni et al. 2021; Shrotri et al. 2021; Thompson et al. 2021; Pritchard et al. 2021) were prospective cohort studies (n = 6) (Cabezas et al. 2021; Hall et al. 2021; Menni et al. 2021; Shrotri et al. 2021; Thompson et al. 2021; Pritchard et al. 2021) and prospective database review (n = 1) (Björk et al. 2021). The included studies (Björk et al. 2021; Dagan et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Haas et al. 2021; Hall et al. 2021; Mason et al. 2021; Menni et al. 2021; Monge et al. 2021; Thompson et al. 2021; Pritchard et al. 2021) were originated from 8 countries: the United Kingdom (n = 6) (Glampson et al. 2021; Hall et al. 2021; Lopez Bernal et al. 2021; Mason et al. 2021; Pritchard et al. 2021; Shrotri et al. 2021), the United States (n = 2) (Swift et al. 2021; Thompson et al. 2021), Canada (n = 1) (Chung et al. 2021) Sweden (n = 1) (Björk et al. 2021), Israel (n = 4) (Angel et al. 2021; Dagan et al. 2021; Haas et al. 2021; Regev-Yochay et al. 2021), Italy (n = 1) (Fabiani et al. 2021), Denmark (n = 1) (Emborg et al. 2021), and Spain (n = 3) (Cabezas et al. 2021; Gras-Valentí et al. 2021; Monge et al. 2021). Study characteristics are depicted in Table 1. The included studies (Angel et al. 2021; Björk et al. 2021; Cabezas et al. 2021; Chung et al. 2021; Dagan et al. 2021; Emborg et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Gras-Valentí et al. 2021; Haas et al. 2021; Hall et al. 2021; Lopez Bernal et al. 2021; Mason et al. 2021; Monge et al. 2021; Pritchard et al. 2021; Regev-Yochay et al. 2021; Shrotri et al. 2021; Swift et al. 2021; Thompson et al. 2021) are deemed moderate-to-good quality with Newcastle–Ottawa Scale ranging from 7 to 8.

Fig. 1

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) flow diagram of process of study selection

Table 1

Characteristics of included studies

Study, country	Design	Sample	Total number of participants	Incidence/frequency of COVID-19
				Unvaccinated	≥ 14 days after dose 1	Adjusted estimate	Unvaccinated	≥ 21 days after dose 1	Adjusted estimate
Hall et al., UK	Prospective multicenter	Adults (aged ≥ 18 years) working in publicly funded hospitals in the United Kingdom	23,324	137.5 per 100,000 person-days	98.6 per 100,000 person-days	HR = 0.44 (0.34–0.57)	137.5 per 100,000 person-days	79.6 per 100,000 person-days	HR = 0.44 (0.31–0.63)
Mason et al., UK	Retrospective database review	Vaccinated: Individuals aged 80–83 who were not residents of care homes and had no prior history of COVID-19 Unvaccinated: Individuals aged 76–79 who were not yet eligible for vaccination	301,462	34.0 per 100,000 persons-days	28.2 per 100,000 persons-days	IRR = 0.83 (0.63–0.91)	30.0 per 100,000 persons-days	13.4 per 100,000 persons-days	IRR = 0.45 (0.34–0.59)
Björk et al., Sweden	Prospective database review	Individuals aged 18–64 years residing in Skåne county, Sweden, on 27 December 2020 when vaccinations started	805,741	42.0 per 100,000 persons-days	24.3 per 100,000 persons-days	IRR = 0.58 (0.37–0.86)	42.0 per 100,000 persons-days	16.7 per 100,000 persons-days	IRR = 0.40 (0.19–0.73)
Dagan et al., Israel	Retrospective database review	Individuals insured in Clalit Health Services	1,760,152	–	–	IRR = 0.54 (0.41–0.60)	–	–	IRR = 0.40 (0.34–0.47)
Pritchard et al., UK	Prospective cohort study	Randomly selected individuals aged ≥ 16 years	373,402	–	–	–	–	–	OR = 0.33 (0.28–0.39)
Glampson et al., UK	Retrospective database review	Adults aged ≥ 16 years and registered with a general practitioner, or with a resident postcode, in the North West London catchment area	2,183,939	–	–	HR = 0.42 (0.36–0.50)	–	–	HR = 0.22 (0.18–0.27)
Monge et al., Spain	Retrospective database review	Residents aged ≥ 65 years and residing in elderly homes	296,093	188.5 per 100,000 persons-day	92.4 per 100,000 persons-day	HR = 0.49 (0.48–0.50)	155.8 per 100,000 persons-day	59.3 per 100,000 persons-day	HR = 0.38 (0.37–0.39)
Fabiani et al., Italy	Retrospective cohort study	Frontline health-care personnel employed at the local health unit that serves the entire province of Treviso in the Veneto region	9878	103.0 per 100,000 persons-day	16.0 per 100,000 persons-day	HR = 0.16 (0.04–0.60)	28.0 per 100,000 persons-day	27.0 per 100,000 persons-day	HR = 0.15 (0.02–1.35)
Haas et al., Israel	Retrospective database review	Residents of Israel (ie, the census population) aged 16 years and older	154,648	91.5 per 100,000 persons-day	34.1 per 100,000 persons-day	IRR = 0.42 (0.40–0.45)	–	–	–
Swift et al., US	Retrospective database review	Actively employed health-care personnel at the Mayo Clinic	71,152	–	–	IRR = 0.22 (0.18–0.27)	–	–	–
Gras–Valentí et al., Spain	Retrospective Case–control study	Healthcare personnel at the Department of Health of General University Hospital of Alicante	268	n = 31/91 (34.1%)	n = 39/177 (22.0%)	OR = 0.47 (0.23–0.99)	–	–	–
Lopez Bernal et al., UK	Retrospective test negative case–control study	Adults aged 70 years or older in England who reported having symptoms and tested for COVID-19	80,545	n = 37,320/126697 (29.5%)	n = 811/3285 (24.7)	OR = 0.84 (0.77–0.91)	n = 37,320/126697 (29.5%)	n = 367/2036 (18.0%)	OR = 0.61 (0.54–0.69)
Angel et al., Israel	Retrospective cohort study	Healthcare workers at Tel Aviv Sourasky Medical Center	6710	–	–	–	–	–	–
Chung et al., Canada	Retrospective test negative case–control study	Community-dwelling adults aged ≥ 16 years who were tested for SARS-CoV-2 and had COVID-19 symptoms	310,880	n = 51,220/302761 (16.9)	n = 636/8119 (7.8%)	OR = 0.41 (0.38–0.45)	–	–	–
Shrotri et al., UK	Prospective cohort study	Care home residents aged ≥ 65 years from 310 long-term care facilities	4274	213.9 per 100,000 persons-day	282.6 per 100,000 persons-day	HR = 0.77 (0.37–1.58)	213.9 per 100,000 persons-day	266.7 per 100,000 persons-day	HR = 0.94 (0.50–1.79)
Regev-Yochay et al., Israel	Retrospective cohort study	Healthcare workers at Sheba Medical Center	9650	–	–	–	–	–	–
Emborg et al., Denmark	Retrospective database review	5 priority groups: Individuals living in long-term care facilities; ≥ 65 years living at home requiring practical help and personal care; individuals aged 85 and older; frontline health-care workers; individuals with high risk of severe COVID-19	864,096	–	–	HR = 0.93 (0.85–1.01)	–	–	HR = 0.58 (0.50–0.67)
Thompson et al., US	Prospective cohort study	Healthcare personnel, first responders, and other essential and frontline workers in eight locations	5969	121.9 per 100,000 persons–day	16.2 per 100,000 persons-day	HR = 0.20 (0.10–0.40)	–	–	–
Cabezas et al., Spain	Prospective cohort study	Nursing home residents	28,191	266.2 per 100,000 persons-day	175.8 per 100,000 persons-day	HR = 0.77 (0.69–0.86)	–	–	–
		Nursing home staff	26,075	138.6 per 100,000 persons-day	121.1 per 100,000 persons-day	HR = 0.80 (0.68–0.93)	–	–	–
		Healthcare workers in nursing home	47,106	103.2 per 100,000 persons-day	98.9 per 100,000 persons-day	HR = 0.85 (0.77–0.95)	–	–	–

COVID-19 coronavirus disease 2019 HR hazard ratio IRR incidence rate ratio Newcastle–Ottawa Scale OR odds ratio

Age, sex, ethnicity, index of multiple deprivation, working in a care-home, having a patient-facing role in health or social care, presence of long-term health conditions, household size, multigenerational household, rural–urban classification, direct or

indirect contact with a hospital or care-home, smoking status, mode of travel to work, work location, visit frequency, geographic area

Symptomatic:

149.8 per 100,000 persons-day

Asymptomatic:

67.0 per 100,000 persons-day

Symptomatic:

4.7 per 100,000 persons-day

Asymptomatic:

11.3 per 100,000 persons-day

Symptomatic:

IRR = 0.03 (0.01–0.06)

Asymptomatic:

IRR = 0.14 (0.07–0.31)

Symptomatic:

146.3 per 100,000 persons-day

Asymptomatic:

69.9 per 100,000 persons-day

Symptomatic:

2.1 per 100,000 persons-day

Asymptomatic:

4.2 per 100,000 persons-day

Symptomatic:

IRR = 0.02 (0.01–0.06)

Asymptomatic:

IRR = 0.06 (0.02–0.22)

The meta-analysis of eight studies (Cabezas et al. 2021; Emborg et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Hall et al. 2021; Monge et al. 2021; Shrotri et al. 2021; Thompson et al. 2021)  which presented effect measure as HR revealed significant protective effect against RT-PCR confirmed COVID-19 14 days or more after the first dose of BNT162b2 mRNA vaccine (pooled HR = 0.58; 95% confidence interval: 0.45–0.75; Fig. 2), where pooled estimate indicates vaccine effectiveness of 42% (95% confidence interval 25%–55%). Similarly, the meta-analysis of five studies (Björk et al. 2021; Dagan et al. 2021; Haas et al. 2021; Mason et al. 2021; Swift et al. 2021) which presented effect measure as IRR revealed significant protective effect against RT-PCR confirmed COVID-19 14 days or more after the first dose of BNT162b2 mRNA vaccine (pooled IRR = 0.47; 95% confidence interval: 0.32–0.68; Fig. 3), where pooled estimate indicates vaccine effectiveness of 53% (95% confidence interval 32%–68%).

Fig. 2

Pooled hazard ratio (HR) of the incidence of COVID-19 14 as well as 21 days post first dose of vaccine (A) and 7 as well as 14 days post second dose of vaccine (B) relative to no vaccination

Fig. 3

Pooled incident rate ratio (IRR) of the incidence of COVID-19 14 as well as 21 days post first dose of vaccine (A) and 7 as well as 14 days post second dose of vaccine (B) relative to no vaccination

Even higher vaccine effectiveness was observed 21 days or more after the first dose of BNT162b2 mRNA vaccine, where the meta-analysis of six studies (Emborg et al. 2021; Fabiani et al. 2021; Glampson et al. 2021; Hall et al. 2021; Monge et al. 2021; Shrotri et al. 2021) which presented effect measure as HR reported pooled HR of 0.42 (95% confidence interval: 0.31–0.57; Fig. 2), and thus vaccine effectiveness of 58% (95% confidence interval: 43%–69%). Likewise, the meta-analysis of three studies (Björk et al. 2021; Dagan et al. 2021; Mason et al. 2021) which presented effect measure as IRR reported pooled IRR of 0.41 (95% confidence interval: 0.36–0.47; Fig. 3), and thus vaccine effectiveness of 59% (95% confidence interval: 53–64%).

The recipient of the second dose of the BNT162b2 mRNA vaccine further boosted the vaccine effectiveness. The meta-analysis of three studies (Emborg et al. 2021; Fabiani et al. 2021; Hall et al. 2021) which presented effect measure as HR reported pooled HR of 0.18 (95% confidence interval: 0.16–0.20; Fig. 2) 7 days or more after the second dose, and thus vaccine effectiveness of 82% (95% confidence interval: 80–84%). Similarly, the meta-analysis of five studies (Angel et al. 2021; Björk et al. 2021; Dagan et al. 2021; Haas et al. 2021; Regev-Yochay et al. 2021) which presented effect measure as IRR revealed significant protective effect against RT-PCR confirmed COVID-19 7 days or more after the second dose of BNT162b2 mRNA vaccine (pooled IRR = 0.09; 95% confidence interval: 0.05–0.19; Fig. 3), where pooled estimate indicates vaccine effectiveness of 91% (95% confidence interval 81%–95%). The findings from the meta-analysis of three studies (Chung et al. 2021; Lopez Bernal et al. 2021; Pritchard et al. 2021) which presented effect measure as OR are also consistent (pooled OR = 0.19; 95% confidence interval 0.09–0.40) and show vaccine effectiveness of 81% (95% confidence interval 60%-91%) 7 days or more after the second dose of BNT162b2 mRNA vaccine. The meta-analysis of three studies (Hall et al. 2021; Regev-Yochay et al. 2021; Thompson et al. 2021) which presented effect measure as HR reported pooled HR of 0.12 (95% confidence interval: 0.08–0.16; Fig. 2) 14 days or more after the second dose, and thus vaccine effectiveness of 88% (95% confidence interval: 84%–92%). Likewise, the meta-analysis of three studies (Angel et al. 2021; Haas et al. 2021; Swift et al. 2021) which presented effect measure as IRR revealed significant protective effect against RT-PCR confirmed COVID-19 14 days or more after the second dose of BNT162b2 mRNA vaccine (pooled IRR = 0.04; 95% confidence interval: 0.03–0.05; Fig. 3), where pooled estimate indicates vaccine effectiveness of 96% (95% confidence interval 95–97%).

Discussion

The findings of the meta-analyses align with the phase 3 randomized controlled trial (Polack et al. 2020) of BNT162b2 mRNA vaccine, though with a lower protective rate: 82% after the first dose (versus overall vaccine effectiveness of 48–55% [14–21 days or more] after the first dose in the current study; Fig. 2) and 95% (7 days or more) after the second dose (versus overall vaccine effectiveness of 86–94% [7–14 days or more] after the second dose in the current study; Fig. 3). Variability in the protective rate between clinical trial and real-world studies could stem from the difference in the definition of confirmed COVID-19; confirmed COVID-19 was defined in the clinical trial as the presence of symptoms and positive RT-PCR test for SARS-CoV-2; while the included studies of our meta-analyses, confirmed COVID-19 was defined as positive RT-PCR test for SARS-CoV-2 regardless of the presence of symptoms.

In addition, individuals with comorbidities (e.g., hypertension, diabetes, and obesity) who are predisposed to severe COVID-19 constituted only about one-fifth of the study population in phase 3 randomized controlled trial (Polack et al. 2020) of BNT162b2 mRNA vaccine. Individuals with comorbidities (e.g., hypertension, diabetes, and obesity), especially those with old age, are often prioritized in the real-world mass vaccination campaign. Therefore, this could explain the lack of reproducible vaccine efficacy reported from the highly controlled clinical research settings compared to the real-world settings since these individuals with comorbidities mainly constituted the real-world study population. Indeed, elderly individuals with comorbidities often have diminished immune responses to vaccines (Kwetkat and Heppner 2020).

Nevertheless, with up to 59% of real-world protective rate after the administration of the first dose of the BNT162b2 mRNA vaccine, it seems reasonable to delay the administration of the second dose in an attempt to allow vaccination in a higher proportion of individuals to reduce the risk of transmission of COVID-19 to an acceptable level. Our study was limited by the fact that included studies were originated in only a few countries. Therefore, the generalizability of our findings is unknown, especially to the countries where variants of concern of SARS-CoV-2 are circulating. Future studies should aim to investigate the vaccine effectiveness against different variants of concern of SARS-CoV-2 and with longer follow-ups to determine the duration of protection against COVID-19. Furthermore, the effectiveness of the BNT162b2 mRNA vaccine among immunocompromised individuals as well as individuals who receive treatment with immunosuppressive therapy should also be investigated since they had been excluded from the participation of phase 3 randomized controlled trial (Polack et al. 2020). Despite its effectiveness, reporting vaccine safety data by relevant stakeholders should be encouraged as BNT162b2 mRNA is a new vaccine that has not gained full approval.