Real-world effectiveness of COVID-19 vaccines: a literature review and meta-analysis.

OBJECTIVE: To estimate the coronavirus disease 2019 (COVID-19) vaccine effectiveness (VE) against concerned outcomes in real-world settings.
METHODS: Studies reporting COVID-19 VE from August 6, 2020 to October 6, 2021 were included. The summary VE (with 95% confidence intervals (95% CI)) against disease related to COVID-19 was estimated. The results were presented in forest plots. Predefined subgroup analyses and sensitivity analyses were also performed.
RESULTS: A total of 51 records were included in this meta-analysis. In fully vaccinated populations, the VE against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, COVID-19-related hospitalization, admission to the intensive care unit, and death was 89.1% (95% CI 85.6-92.6%), 97.2% (95% CI 96.1-98.3%), 97.4% (95% CI 96.0-98.8%), and 99.0% (95% CI 98.5-99.6%), respectively. The VE against infection in the general population aged ≥16 years, the elderly, and healthcare workers was 86.1% (95% CI 77.8-94.4%), 83.8% (95% CI 77.1-90.6%), and 95.3% (95% CI 92.0-98.6%), respectively. For those fully vaccinated against infection, the observed effectiveness of the Pfizer-BioNTech vaccine was 91.2% and of the Moderna vaccine was 98.1%, while the effectiveness of the CoronaVac vaccine was found to be 65.7%.
CONCLUSIONS: The COVID-19 vaccines are highly protective against SARS-CoV-2-related diseases in real-world settings.

Introduction

Globally, as of October 15, 2021, there had been more than 239.4 million confirmed cases of coronavirus disease 2019 (COVID-19), including over 4.8 million deaths (WHO, 2021b). Since the outbreak of COVID-19, several vaccines have been tested and granted emergency use authorization. Phase III trials reported high vaccine effectiveness (VE) against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection with these vaccines, such as 70.4% effectiveness of the ChAdOx1 nCoV-19 vaccine (AZD1222; Oxford-AstraZeneca) (Voysey et al., 2021), 95% effectiveness of the BNT162b2 mRNA COVID-19 vaccine (Pfizer-BioNTech) (Skowronski and De Serres, 2021), 94.1% effectiveness of the mRNA-1273 vaccine (Moderna) (Baden et al., 2021), and 50.7% effectiveness of an absorbed COVID-19 (inactivated) vaccine (CoronaVac) (Palacios et al., 2020). Given that the outcomes of clinical trials may be influenced by the various study settings, it is necessary to estimate the effectiveness of vaccines rolled out to the public in real-world settings.

Recently, a series of studies have reported real-world VE from all over the world. A nationwide mass vaccination setting in Israel showed 92% effectiveness for documented infections after the second dose of the BNT162b2 vaccine (Dagan et al., 2021). The UK government adopted a strategy of delaying the second dose to increase the vaccine coverage, and a study suggested VE of 51.4% against SARS-CoV-2 infection after one dose of the BNT162b2 vaccine (Chodick et al., 2021). In another study, 73% effectiveness against COVID-19 cases was observed among the elderly after vaccination with one dose of the ChAdOx1 vaccine in England (Lopez Bernal et al., 2021). In Chile, the Sinovac vaccine rolled out to the general population aged ≥16 years showed an effectiveness of 16.13% after the first dose and 66.96% after the second dose (Ministerio de Salud, 2021).

The World Health Organization (WHO) has stated that there is an urgent need to evaluate COVID-19 VE against several major outcomes, including symptomatic COVID-19, severe diseases, and death related to COVID-19 (Patel et al., 2021). Therefore, this review and meta-analysis was conducted to estimate the COVID-19 VE against concerned outcomes in real-world settings based on the latest evidence.

Methods

Search and inclusion criteria

For this literature review and meta-analysis, a systematic search of PubMed was performed using the terms “COVID-19” or “SARS-CoV-2” and “vacc*” and “eff*” to identify articles published between August 6, 2020 (Deplanque and Launay, 2021) and October 6, 2021, from any country, reporting VE in a vaccinated population. In addition, major news media platforms were searched to track reports from governments and health authorities around the world evaluating the effectiveness of the COVID-19 vaccine. The review process is described in detail in Figure 1 .

Figure 1

Results of the literature review according to the inclusion criteria.

Flow diagram of the study selection for meta-analysis. Fifty publications and one record reporting COVID-19 vaccine effectiveness in 14 countries from August 6, 2020 to October 6, 2021 were included in the meta-analysis.

Observational studies (cohort, case–control, test-negative case–control) were included. Studies reporting exclusively on the immunogenicity of COVID-19 vaccine, review articles, data only in abstract form, ecological studies, and mathematical modeling analysis studies were excluded. If two or more articles and reports presented results from the same dataset, all articles that included unique data points for the vaccine, study population, or vaccination status were included. In situations where the findings in one article were a subset of findings in another article (e.g., study sites and population), only the most comprehensive article was included in the overall analysis, but the subset was included in subgroup analyses. At least two reviewers examined articles to confirm that the inclusion criteria were satisfied and to reach a consensus when necessary.

Data were extracted by two independent authors in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA) (Page et al., 2021); the checklist is presented in Table S1. The following information was abstracted: the summary VE; the stratified VE estimates by COVID-19 vaccine against a range of SARS-CoV-2 outcomes, including confirmed SARS-CoV-2 infection by reverse transcription PCR (RT-PCR), and COVID-19-related hospitalization, admission to the intensive care unit (ICU)/severe or critical hospitalization, and death; vaccination status, either partially or fully vaccinated (as the partial and full vaccination status of the individuals varied according to the recommendations of the health authorities in different countries and regions, we relied on those reported in the literature and included these in the analysis of the results); vaccine brand; study population; and study characteristics, such as the study design, study population, and sample size. These were recorded in a Microsoft Excel database.

Data analysis

Descriptive statistics and percentages were calculated for the article attributes. The real-world effectiveness of COVID-19 vaccines against a range of SARS-CoV-2 outcomes and according to vaccination status (partial and full vaccination), vaccinated population, and vaccine brand were estimated. Estimates of VE expressed as a percentage (%) with their 95% confidence intervals (95% CI) were derived from the effect measures (odds ratio, relative risk, hazard ratio, and incidence rate ratio) using the following equation: VE = [1 − effect measure] × 100. A VE estimate >0% suggests a protective effect. The results were presented in forest plots.

The heterogeneity of outcomes across studies was assessed with the I 2 statistic and was quantified as low (≤25%), moderate (25–50%), or high (>50%). A random-effects model was used if the I 2 statistic for the data was >25%. Otherwise, a fixed-effects model was chosen. Stratified meta-analyses were conducted to explore potential sources of study heterogeneity.

The Newcastle–Ottawa scale (NOS) (range from 0 to 9 points) was used to assess the quality of the included observational studies; a higher total NOS score suggests better quality (Wells et al., 2011). The influence of the inclusion of a study on the results of the meta-analyses was assessed by sensitivity analysis. Additionally, Egger's test and a funnel plot of the standard error were used to evaluate publication bias in analyses with 10 or more included articles. Analyses were conducted in R software (version 4.1.0; R Foundation) using the metafor package for meta-analyses (‘metafor’, 2021).

Results

Characteristics of the studies

A total of 13 018 records were identified and screened in the literature review; 13 016 were identified in PubMed and two reports were from the official website of a government health department.

As shown in Figure 1, 13 018 records were identified. In the preliminary review, 12901 were excluded because 12 645 were not relevant to research question based on titles, 28 were mathematical modelling analysis, 56 were clinical trial, 21 were interim guidance by official organization, 53 were other enpoints and safety, 88 were systematic review, and 10 were costeffectiveness assessments. Therefore, 115 abstracts and 2 reports remained to be reviewed. Fifty-nine articles were then read in full, of which seven were excluded because they did not meet the methodological inclusion criteria. One report did not provide information on the time span of vaccination or the age and health status of the vaccinated persons, and was therefore also excluded. In total, 50 articles and one report were included from 14 countries, involving 38 821 141 individuals.

All of the included articles were published in 2021 and reported studies of high quality (NOS score ranging from 5 to 8) ( Table S2). Thirty-nine were cohort studies, eight were test-negative case–control studies, and four were case–control studies. The included studies investigated five brands of COVID-19 vaccine: Pfizer-BioNTech (46 articles), Moderna (19 articles), Oxford-AstraZeneca (10 articles), CoronaVac (5 articles), and Janssen (Johnson & Johnson) (1 article). Most articles presented VE estimation for fully vaccinated and partially vaccinated individuals (34 articles, 66.7%); 11 articles (21.6%) only presented effectiveness in partially vaccinated individuals.

Vaccine effectiveness for full vaccination

The effectiveness of COVID-19 vaccines against a range of SARS-CoV-2 outcomes was estimated. A total of 35 articles reported VE against SARS-CoV-2 infection among fully vaccinated people, and the summary VE was 89.1% (95% CI 85.6–92.6%) for the prevention of SARS-CoV-2 infection (Figure 2 ). In addition, 15 of the included studies estimated VE against COVID-19-related hospitalization, four studies estimated VE against COVID-19-related ICU admission or severe disease, and eight studies estimated VE against COVID-19-related death. The results showed 97.2% VE (95% CI 96.1–98.3%) for the prevention of hospitalization, 97.4% VE (95% CI 96.0–98.8%) for the prevention of ICU admission or severe disease, and 99.0% VE (95% CI 98.5–99.6%) for the prevention of COVID-19-related death (Figure 3 ). The Egger's test and funnel plots showed no publication bias for the VE against SARS-CoV-2 infection (t-value of Egger's test = −2.91, P = 0.0988) among fully vaccinated individuals, while there was publication bias of the VE against COVID-19-related hospitalization (t-value of Egger's test = −2.91, P = 0.006) ( Figure S1B). After correcting for publication bias with trim and fill methods, the summary VE against COVID-19-related hospitalization among fully vaccinated individuals was 97.2% (95% CI 94.4–100.0%) ( Figure S1C). The sensitivity analysis suggested lower VE against COVID-19-related hospitalization (93.0%, 95% CI 91.0–96.0%) and ICU admission and severe disease (89.0%, 95% CI 76.0–100.0%) when deleting the results of the study conducted by Haas et al. (Haas et al., 2021) ( Figures S2 and S3).

Figure 2

COVID-19 vaccine effectiveness (VE) for the prevention of SARS-CoV-2 infection in those with a fully vaccinated status.

Forest plot showing VE for the prevention of SARS-CoV-2 infection for fully vaccinated populations; a total of 35 studies contributed information on the effectiveness against SARS-CoV-2 infection.

Figure 3

COVID-19 vaccine effectiveness (VE) for the prevention of COVID-19-related hospitalization, admission to the ICU/severe or critical hospitalization, and death in those with a fully vaccinated status.

Forest plot showing VE for the prevention of COVID-19-related hospitalization, severe disease, and death for fully vaccinated populations; a total of 15 included studies contributed information on effectiveness against COVID-19-related hospitalization, four on effectiveness against COVID-19-related severe disease, and eight on effectiveness against COVID-19-related death.

Vaccine effectiveness for partial vaccination

Regarding individuals with a partial immunization status, 38 included studies reported VE against SARS-CoV-2 infection, 12 reported VE against COVID-19-related hospitalization, three reported VE against COVID-19-related ICU admission or severe disease, and eight reported VE against COVID-19-related death. The summary VE was 68.8% (95% CI 60.1–77.5%) for the prevention of SARS-CoV-2 infection and 67.8% (95% CI 51.6–83.9%) for the prevention of hospitalization, 66.4% (95% CI 25.9–100.0%) for the prevention of admission to the ICU and severe disease, and 58.4% (95% CI 28.0–88.7%) for the prevention of COVID-19-related death ( Figures S4 and S5). The Egger's test and funnel plots suggested no publication bias of the VE against SARS-CoV-2 infection (t-value of Egger's test = −1.31, P = 0.1956) or hospitalization (t-value of Egger's test = −0.28, P = 0.7839) for partially vaccinated individuals ( Figure S6). The sensitivity analysis suggested higher VE against SARS-CoV-2 infection (75.0%, 95% CI 71.0–80.0%) among the partially vaccinated individuals when deleting the results of the CoronaVac vaccine study from Chile reported by the Ministerio de Salud (Ministerio de Salud, 2021) ( Figure S7 and S8).

Subgroup analyses

The COVID-19 VE for the prevention of SARS-CoV-2 infection confirmed by RT-PCR was estimated in fully vaccinated individuals in different populations. An analysis in the predefined subgroups of the elderly (age ≥60 years), healthcare workers (HCWs), and the general population (adults ≥16 years) was conducted. There were 15, 17, and 11 articles including four brands of COVID-19 vaccine (Pfizer-BioNTech, Moderna, Oxford-AstraZeneca, and CoronaVac) presenting the VE against SARS-CoV-2 infection among the elderly, HCWs, and the general population, respectively. The summary VE was 83.8% (95% CI 77.1–90.6%) among the elderly, 95.3% (95% CI 92.0–98.6%) among HCWs, and 86.1% (95% CI 77.8–94.4%) among the general population (Figure 4 ).

Figure 4

COVID-19 vaccine effectiveness (VE) for the prevention of SARS-CoV-2 infection in different populations.

Forest plot showing VE for the prevention of SARS-CoV-2 infection for fully vaccinated populations based on different population groups; a total of 17 included studies contributed information on effectiveness against SARS-CoV-2 infection for healthcare workers (HCWs), 15 on effectiveness against SARS-CoV-2 infection for the elderly, and 11 on effectiveness against SARS-CoV-2 infection for the general population aged ≥16 years.

The VE of the different vaccine brands was also estimated among the fully vaccinated people. For the Pfizer-BioNTech vaccine, a total of 23 articles reported the VE for full vaccination. The summary VE was 91.2% (95% CI 87.9–94.5%) against SARS-CoV-2 infection (Figure 5 ), 97.6% (95% CI 96.5–98.7%) against COVID-19-related hospitalization, and 98.1% (95% CI 96.3–99.9%) against COVID-19-related death (Figure 6 ). Five articles estimated the effectiveness of the Moderna vaccine and three estimated the effectiveness of the CoronaVac vaccine, against SARS-CoV-2 infection. The Moderna vaccine presented the highest VE against infection, with a summary VE of 98.1 % (95% CI 96.0–100.0%). The VE against infection of the CoronaVac vaccine was 65.7% (95% CI 63.0–68.5%) (Figure 5). Only one article, from India, reported the effectiveness against infection of the Oxford-AstraZeneca vaccine in fully vaccinated individuals: VE was 88.6% (95% CI 81.6–92.4%) (Zacay et al., 2021). Furthermore, eight articles reported the VE of the Oxford-AstraZeneca vaccine among partially vaccinated people, and the summary VE was 81.8% (95% CI 67.1–96.6%) ( Figure S9). No significant publication bias was detected for the subgroup analysis (P > 0.05).

Figure 5

Summary vaccine effectiveness (VE) against SARS-CoV-2 infection for the different brands in those with a full vaccination status.

Forest plot showing VE for the prevention of SARS-CoV-2 infection for fully vaccinated populations with different vaccine brands; a total of 23 included studies contributed information on the effectiveness of the Pfizer-BioNTech vaccine, five on the effectiveness of the Moderna vaccine, and three on the effectiveness of the CoronaVac vaccine.

Figure 6

Summary vaccine effectiveness (VE) against COVID-19-related hospitalization and death of the Pfizer-BioNTech vaccine among fully vaccinated individuals.

Forest plot showing VE for the prevention of COVID-19-related hospitalization and death for the Pfizer-BioNTech vaccine among fully vaccinated individuals; a total of nine included studies contributed information on effectiveness against COVID-19-related hospitalization and three on effectiveness against COVID-19-related death for the Pfizer-BioNTech vaccine.

Discussion

This review, including 51 up-to-date studies from 14 countries, reporting on the effectiveness of COVID-19 vaccines, provides estimates of the VE against disease with laboratory-confirmed SARS-CoV-2 infection, and COVID-19-related hospitalization, admission to the ICU, and death. Estimates of VE against infection in subgroup analyses for vaccine brand, vaccinated population, and vaccination status are presented. The results suggest that the vaccines currently approved for use have a good protective effect against the major outcomes related to COVID-19, especially for critical outcomes.

It was noted that there was high heterogeneity for the summary VE against SARS-CoV-2 infection among fully vaccinated individuals. In addition to the actual effectiveness of the different vaccines, the evaluation of population effectiveness depends on a series of factors, such as the vaccinated population, the severity of the epidemic, the completeness and validity of the data sources, study design, and potential methodological biases (Patel et al., 2021). Therefore, subgroup and sensitivity analyses were performed to explore the potential heterogeneity. Consistent with the results of phase III clinical trials, the effectiveness of different vaccines against confirmed infection in real-world conditions varied (Baden et al., 2021; Palacios et al., 2020; Skowronski and De Serres, 2021; Voysey et al., 2021). Synthesized evidence from different study settings showed 91.2%, 98.1%, and 65.7% effectiveness of the Pfizer-BioNTech vaccine, Moderna vaccine, and CoronaVac vaccine, respectively. For full vaccination, lower VE against COVID-19-related hospitalization and severe disease were observed when deleting the results of the study conducted by Haas et al. (Haas et al., 2021). This study, with a high quality score, revealed that two doses of BNT162b2 were highly effective across all age groups in Israel, based on nationwide surveillance data. The sensitivity analysis showed higher VE after omitting the results of the CoronaVac vaccine (Hitchings et al., 2021; Ministerio de Salud, 2021). The VE of CoronaVac, an inactivated whole virus vaccine, may be influenced in the setting of high SARS-CoV-2 Gamma variant transmission, whether in Brazil or Chile (Jara et al., 2021; Palacios et al., 2020; Ranzani et al., 2021). Similarly, VE is closely related to vaccination status. Subgroup analysis by vaccination status revealed 66.8%, 67.8%, 66.4%, and 58.4% effectiveness of partial vaccination against disease with confirmed SARS-CoV-2 infection, COVID-19-related hospitalization, ICU admission/severe disease, and death, respectively, despite being less effective than full vaccination. Therefore, this finding supports the proposal across many countries of extending the dosing interval to optimize vaccine coverage with the increasing number of new infections and the spread of SARS-CoV-2 variants (Chodick et al., 2021; Krutikov et al., 2021; Shrotri et al., 2021). Moreover, several studies have shown higher VE for longer periods of time since vaccination, whether for partial or full vaccination (Jones et al., 2021; Rudolph et al., 2021; Zaqout et al., 2021).

Given the highest mortality observed in the elderly in long-term care facilities and higher exposure risk for HCWs ((CDC COVID-19 Response Team, 2020); Wu and McGoogan, 2020; Zhou et al., 2020), many countries have prioritized both of these high-risk groups for vaccination (Britton et al., 2021; Lopez Bernal et al., 2021). However, elderly patients have been less represented in clinical trials, which have mainly enrolled young populations (Prendki et al., 2020). In this study, we synthesized real-world evidence by vaccinated population. The most protective effect was seen in the HCWs (VE = 95.3%), while less VE was observed in the elderly (VE = 83.8%). Due to immunosenescence and comorbidities, the elderly are more susceptible to infections and have poorer responses to vaccination (Brosh-Nissimov et al., 2021; Ciabattini et al., 2018; Frasca et al., 2010; McElhaney et al., 2013). Therefore, beyond vaccination, more measures for the elderly need to be implemented to reduce the severe outcomes related to infections and control the transmission in care facilities.

There are some limitations to this meta-analysis. Firstly, it was not possible to estimate the long-term effectiveness of vaccines due to the limited length of follow-up. In a test-negative case–control study conducted in England, protection of full vaccination with the Oxford-AstraZeneca vaccine in the elderly was maintained from the second week (VE = 22.0%, 95% CI 11.0–32.0%) after vaccination to the end of the follow-up (more than 6 weeks) (VE = 73.0%, 95% CI 27.0–90.0%) (Lopez Bernal et al., 2021). In the Mayo Clinic health system, the effectiveness after the second dose of the Pfizer-BioNTech vaccine or the Moderna vaccine increased from 53.6% (95% CI 40.9–63.8%) in the first week to 92.5% (95% CI 70.2–99.1%) in the sixth week (Pawlowski et al., 2021). Based on available evidence, there is increased VE within 6 weeks after full vaccination, but it is difficult to reveal the peak effectiveness and actual duration of immunization protection.

Secondly, the VE against infectiousness to others was not estimated. A retrospective cohort study in the USA suggested 80.0% (95% CI 91.0–56.0%) effectiveness against infectiousness to others after the second dose of the Pfizer-BioNTech vaccine (Tande et al., 2021). In addition, a single dose of the Moderna vaccine was estimated to reduce the potential transmission to others by 61.0% (95% CI 31.0–79.0%) (Lipsitch and Kahn, 2021). The vaccine could reduce the risk of transmission, but further studies are needed to assess the actual VE for every vaccine.

Thirdly, the emergence of SARS-CoV-2 variants has resulted in an increase in severe infections (Gomez et al., 2021). Four dominant variants of concern (VOCs) are B.1.1.7 (Alpha, UK, Sep-2020), B.1.351 (Beta, South Africa, May-2020), P.1 (Gamma, Brazil, Nov-2020), and B.1.617.2 (Delta, India, Oct-2020) (WHO, 2021a). Several clinical trials have reported the vaccine effectiveness against variants (Haas et al., 2021). There was no effectiveness against mild-to-moderate COVID-19 due to the B.1.351 variant (VE = 21.9%, 95% CI −49.9% to 59.8%) after two doses of the ChAdOx1 nCoV-19 vaccine (Madhi et al., 2021). Nevertheless, two doses of the Pfizer-BioNTech vaccine showed 87.0% (95% CI 81.8–90.7%) effectiveness against the B.1.1.7 variant and 72.1% (95% CI 66.4–76.8%) against the B.1.351 variant (Abu-Raddad et al., 2021). Moreover, the VE for two doses of CoronaVac vaccine was 59.0% (95% CI 16.0–81.6%) against the B.1.617.2 variant (Li et al., 2021). A single dose of the Ad26.COV2.S also showed 68.1% (95% CI 48.8–80.7%) effectiveness against the P.1 variant to prevent moderate-to-severe COVID-19 (Sadoff et al., 2021). The estimates of VE against SARS-CoV-2 variants in real-world settings are scarce, so it was not possible to evaluate VE with the variants. This needs to examined in future studies.

Studies in the real-world setting around the world have shown that the approved vaccines are highly protective against SARS-CoV-2; therefore, the aim should be full vaccination according to the standard schedule to achieve maximum VE. It is worth noting that vaccination cannot eliminate the risk of infection (Brosh-Nissimov et al., 2021), and preventive and control measures should be taken seriously, especially for high-risk groups.

In conclusion, consistent with the results of phase III clinical trials, the authorized vaccines are highly protective against SARS-CoV-2 in real-world settings. Furthermore, the actual VE depends not only on the effectiveness of the vaccine itself but also on the vaccinated population and status. Preventive measures remain essential.