Analysis of the Effectiveness of the Ad26.COV2.S Adenoviral Vector Vaccine for Preventing COVID-19.

Importance: Continuous assessment of the effectiveness and safety of the US Food and Drug Administration-authorized SARS-CoV-2 vaccines is critical to amplify transparency, build public trust, and ultimately improve overall health outcomes. Objective: To evaluate the effectiveness of the Johnson & Johnson Ad26.COV2.S vaccine for preventing SARS-CoV-2 infection. Design, Setting, and Participants: This comparative effectiveness research study used large-scale longitudinal curation of electronic health records from the multistate Mayo Clinic Health System (Minnesota, Arizona, Florida, Wisconsin, and Iowa) to identify vaccinated and unvaccinated adults between February 27 and July 22, 2021. The unvaccinated cohort was matched on a propensity score derived from age, sex, zip code, race, ethnicity, and previous number of SARS-CoV-2 polymerase chain reaction tests. The final study cohort consisted of 8889 patients in the vaccinated group and 88 898 unvaccinated matched patients. Exposure: Single dose of the Ad26.COV2.S vaccine. Main Outcomes and Measures: The incidence rate ratio of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts, measured by SARS-CoV-2 polymerase chain reaction testing.
Results: The study was composed of 8889 vaccinated patients (4491 men [50.5%]; mean [SD] age, 52.4 [16.9] years) and 88 898 unvaccinated patients (44 748 men [50.3%]; mean [SD] age, 51.7 [16.7] years). The incidence rate ratio of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts was 0.26 (95% CI, 0.20-0.34) (60 of 8889 vaccinated patients vs 2236 of 88 898 unvaccinated individuals), which corresponds to an effectiveness of 73.6% (95% CI, 65.9%-79.9%) and a 3.73-fold reduction in SARS-CoV-2 infections. Conclusions and Relevance: This study's findings are consistent with the clinical trial-reported efficacy of Ad26.COV2.S and the first retrospective analysis, suggesting that the vaccine is effective at reducing SARS-CoV-2 infection, even with the spread of variants such as Alpha or Delta that were not present in the original studies, and reaffirm the urgent need to continue mass vaccination efforts globally.

Introduction

As of April 2021, there have been more than 195 million cases of COVID-19 worldwide, with more than 4 million associated deaths.[1] After the emergency use authorizations issued by the US Food and Drug Administration (FDA) on February 27, 2021, more than 21 million doses of the Ad26.COV2.S COVID-19 vaccine (from Johnson & Johnson and Janssen) have been administered in the United States.[2] This vaccine consists of a single-dose injection of a recombinant, replication-incompetent, human adenovirus type 26 vector encoding the SARS-CoV-2 spike protein. A recent phase 3 trial has demonstrated the effectiveness (66.9% [95% CI, 59.0%-73.4%]) and safety profile of this vaccine.[3] Self-resolving, mild to moderate adverse effects were common in participants who received the vaccine, and serious adverse effects occurred rarely, with a frequency comparable to that of placebo.

As the SARS-CoV-2 vaccines continue to be administered more broadly, it is critical to continuously assess the safety and effectiveness data for various reasons. For example, the interpretation of vaccine trial outcomes is inherently limited by how representative the studied population is of the broader population that will ultimately receive the vaccine. Furthermore, effectiveness is a dynamic process that may be affected by the evolution of the virus. Variations in the spike protein regularly occur and may have the potential to escape the immune response triggered by the vaccine. Finally, the fraction of the population that has been vaccinated may affect the observed effectiveness through herd immunity.

Previous studies have used recent advances in deep neural networks to perform high-throughput, machine-augmented curation of electronic health record (EHR) systems.[4,5] These advances have enabled the rapid assessment of the real-world effectiveness and safety of the messenger RNA (mRNA) vaccines mRNA-1273 (Moderna) and BNT162b2 (Pfizer/BioNTech) as well as a targeted investigation of the incidence of cerebral venous sinus thrombosis among patients receiving COVID-19 vaccines (including Ad26.COV2.S) within the Mayo Clinic Health System.[6,7,8] We believe such approaches are essential for the development and deployment of vaccines and, most importantly, to build trust and transparency for the public. Here, we expand on this effort to conduct a preliminary assessment of the effectiveness of the Ad26.COV2.S vaccine using EHR data from the Mayo Clinic and Mayo Clinic Health System between February 27 and July 22, 2021.

Methods

Study Design and Participants

This was a comparative effectiveness research study of individuals who underwent polymerase chain reaction (PCR) testing for suspected SARS-CoV-2 infection at the Mayo Clinic and hospitals affiliated with the Mayo Clinic Health System between February 27 and July 22, 2021. We limited ourselves to states where the Ad26.COV2.S vaccine was administered (ie, Arizona, Iowa, Minnesota, Florida, and Wisconsin). This study was reviewed by the Mayo Clinic institutional review board and determined to be exempt from the requirement for institutional review board approval (45 CFR 46.104d, category 4). Participants were excluded if they did not have a research authorization on file. This study followed the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guideline.

The participant-selection algorithm for this study mirrored that outlined in a previous real-world effectiveness analysis of mRNA COVID-19 vaccines.[6] Specifically, in this retrospective study, patients with the following criteria were included: (1) underwent at least 1 SARS-CoV-2 PCR test at the Mayo Clinic between February 27 and July 22, 2021; (2) aged at least 18 years; and (3) and resides in a local area (based on zip code) in which at least 10 patients have received the Ad26.COV2.S vaccine. Matching based on geography and the number of individuals undergoing PCR testing is used to control for any bias in infection rates due to geographical constraints and changes in infection rates.

Exclusion criteria were as follows: (1) individuals with a positive SARS-CoV-2 PCR test result before the date of vaccine administration or the beginning of the study period (February 27, 2021); (2) individuals with no follow-up days after vaccination (ie, those who received the vaccine dose on the last date of data collection); (3) individuals who received the mRNA-1273 (Moderna) or BNT162b2 (Pfizer/BioNTech) vaccines; and (4) individuals with no research authorization on file. After the application of these inclusion and exclusion criteria, the study population included 8889 vaccinated patients and 230 790 unvaccinated patients (Figure 1A). Propensity score matching of the unvaccinated cohort was performed to match the vaccinated population.

Figure 1.

Effectiveness of the Ad26.COV2.S Vaccine and Comparison of COVID-19 Severity Between Vaccinated and Unvaccinated Patients

A, Real-world evidence supporting the effectiveness of the Ad26.COV2.S (J&J) COVID-19 vaccine. B, Comparison of COVID-19 severity between vaccinated and unvaccinated patients. J&J indicates Johnson & Johnson; PCR, polymerase chain reaction.

Effectiveness Analysis

Propensity Score–Matching Procedure Defining the Unvaccinated Cohort

We used 1:10 propensity score matching[9] to construct an unvaccinated cohort similar to the vaccinated cohort with respect to key risk factors for SARS-CoV-2 infection, as was described previously[6]: (1) geography (zip code of the patient’s residence), (2) demographic characteristics (age, sex, race, and ethnicity), and (3) records of PCR testing (number of negative PCR test results before February 27, 2021). As described previously, by obtaining the number of negative PCR test results, we intended to capture a combination of factors, including the potential levels of exposure to COVID-19 as well as the willingness and ability of individuals to undergo SARS-CoV-2 testing. Propensity scores were obtained by training regularized logistic regression models for each zip code using the software package sklearn, version 0.20.3 in Python (Python Software Foundation).

Using these propensity scores, we matched each of the 8889 individuals in the previously defined vaccinated cohort with 10 of the 230 790 eligible unvaccinated individuals, using greedy nearest-neighbor matching without replacement.[10] If an unvaccinated individual tested positive for SARS-CoV-2 prior to the vaccination date for a potential matched vaccinated individual, this match was considered invalid; in such cases, the unvaccinated individual was recycled (ie, made available to potentially match to other vaccinated individuals), and a new unvaccinated individual was selected from the pool.

Evaluation of Vaccine Effectiveness

Vaccine effectiveness was evaluated as described previously by comparing the rates of positive SARS-CoV-2 PCR test results in the vaccinated and unvaccinated cohorts.[6] For each vaccinated individual, the date of study enrollment (day 0) was defined as the date of vaccine administration. For each unvaccinated individual, the date of study enrollment was defined as the date of the vaccine administration for their matched vaccinated individual.

The cumulative proportional incidence of SARS-CoV-2 infection was compared between vaccinated and unvaccinated patients by Kaplan-Meier analysis. The cumulative proportional incidence at time t is the estimated proportion of patients who experience the outcome on or before time t (ie, 1 minus the standard Kaplan-Meier survival estimate). We considered the cumulative incidence starting at day 1 and day 14 relative to the date of study enrollment (day 0). Significance was assumed to be indicated by P < .05. Statistical significance was assessed with the 2-sided log-rank test. Censored data include individuals who did not test positive for SARS-CoV-2 at the end of the study as reported in the EHR. They were considered in the analysis as negative for SARS-CoV-2 infection.

We calculated the incidence rate ratio (IRR) of positive SARS-CoV-2 tests between the vaccinated and unvaccinated cohorts. Effectiveness was defined as 100% × (1 − IRR). We considered the incidence rate starting on day 1 after vaccination, day 8 after vaccination, and day 15 after vaccination. Incidence rates were defined as the number of patients testing positive for SARS-CoV-2 in the given period divided by the total number of at-risk person-days contributed in that period. For each individual, at-risk person-days are defined as the number of days in the period in which the individual has not yet tested positive for SARS-CoV-2 or died. The IRR was calculated as the incidence rate of the vaccinated cohort divided by the incidence rate of the unvaccinated cohort, and its 95% CI was computed using an exact approach described previously.[11] For comparison, Minnesota, Florida, and Wisconsin had a mean of 840, 4442, and 451 new COVID-19 cases daily, respectively, during the study period.[12] These trends are shown in eFigure 1 in the Supplement.[2] We also computed the daily prevalence of SARS-CoV-2 variants of concern as a percentage of new daily observations for each state using the GISAID (Global Initiative on Sharing Avian Influenza Data) data set[13] by plotting a simple moving mean with a sliding window of 7 days as a function of time for the duration of the study, illustrating the dynamics of SARS-CoV-2 variants (eFigure 2 in the Supplement).[14]

COVID-19 Severity Analysis

Propensity Score Matching Procedure Defining the Unvaccinated Infected Cohort

We applied 1:10 propensity score matching to construct a SARS-CoV-2–positive unvaccinated cohort similar in baseline clinical covariates to the cohort of patients who were vaccinated and subsequently tested positive for SARS-CoV-2. Patients were matched based on demographic characteristics (age, sex, race, and ethnicity) and comorbid conditions (asthma, cancer, cardiomyopathy, chronic kidney disease, chronic obstructive pulmonary disease, coronary artery disease, heart failure, hypertension, obesity, pregnancy, severe obesity, sickle cell disease, solid organ transplant, stroke or cerebrovascular disease, and type 2 diabetes). This list of comorbid conditions was derived from the list of risk factors for severe COVID-19 illness provided by the Centers for Disease Control and Prevention.[15] We used deep neural networks to automatically identify comorbid conditions from the clinical notes. To obtain the propensity scores, we trained an L2-regularized logistic regression model with these features and stratified patients by zip code using the software package sklearn, version 0.20.3 in Python.

Based on these propensity scores, we matched each of the 60 individuals who tested positive for SARS-CoV-2 after vaccination with 600 of the 2236 individuals who tested positive for SARS-CoV-2 and who were not vaccinated, using greedy nearest-neighbor matching without replacement.

Disease Severity Analysis

For each SARS-CoV-2–positive patient in both the vaccinated and unvaccinated cohorts, the index date for the analysis (day 0) was taken to be the date of the first positive PCR test result. Analysis was performed with patients in each cohort with at least 14 days of follow-up after their first positive PCR test result (60 vaccinated and 600 unvaccinated). Severe COVID-19 cases included those with hospital admission, those with intensive care unit (ICU) admission, and those who died. The following parameters were evaluated: (1) 14-day hospital admission rate (ie, the number of patients admitted to the hospital in the 2 weeks after their positive PCR test result); (2) 14-day ICU admission rate (ie, the number of patients admitted to the ICU in the 2 weeks after their positive PCR test result); and (3) 14-day mortality rate (ie, the number of patients who died in the 2 weeks after their positive PCR test result). For each outcome, we report the relative rate (ie, the rate in the vaccinated cohort divided by the rate in the matched unvaccinated cohort), 95% CI for the relative risk, and the Fisher exact test P value. Hospital-free and ICU-free survival were also compared via Kaplan-Meier analysis, with statistical significance assessed using the log-rank test.

Automated Clinical Data Extraction From Clinical Notes

We used the previously described BERT (bidirectional encoder representations from transformers)–based neural network model to identify comorbid conditions from the EHR for each patient and classify the sentiment for the phenotypes that appeared in the clinical notes.[16] In brief, we applied a phenotype sentiment classification model that had been trained on 18 500 sentences that achieves an out-of-sample accuracy of 93.6% with precision and recall scores above 95%. This classification model identifed 4 classes: (1) “yes” (confirmed diagnosis); (2) “no” (diagnosis ruled out); (3) “maybe” (possibility of disease); and (4) “other” (alternate context [eg, family history of disease]). For each patient, we applied the sentiment model to the clinical notes in the Mayo Clinic EHR. For each comorbidity phenotype, if a patient had at least 1 mention of the phenotype during the period with a confidence score of 90% or greater, then the patient was identified as having the phenotype.

SARS-CoV-2 Variant Analysis

The number of cases per state over time was obtained from the Centers for Disease Control and Prevention website.[2] The information on variants was obtained from GISAID.[13]

Results

Prevention of SARS-CoV-2 Infection in a Population Vaccinated With Ad26.COV2.S

Between February 27 and July 22, 2021, in the US, the pandemic entered a new phase characterized by a flattening in the number of cases due to the massive campaign for vaccination and the expansion of new SARS-CoV-2 variants (eFigures 1 and 2 in the Supplement). The Alpha variant was the most common (66.1% in Wisconsin, 71.7% in Florida, and 80.4% in Minnesota until May 31, 2021) but was replaced by the recent Delta variant in June and July (45.1% in Wisconsin, 37.9% in Florida, and 38.2% in Minnesota). A total of 8889 individuals who met the study inclusion criteria received the Ad26.COV2.S COVID-19 vaccine across the Mayo Clinic network (see Methods; Figure 1A). To evaluate the effectiveness of this vaccine in preventing SARS-CoV-2 infection, we identified a cohort of 88 898 unvaccinated individuals using 1:10 propensity score matching (see Methods; Table 1). The vaccinated and unvaccinated cohorts were balanced for sex, race, and age with 4397 (49.5%) vs 44 140 (49.7%) women, 8202 (92.3%) vs 81 458 (91.6%) non-Hispanic or non-Latino individuals, and a mean (SD) age of 52.4 (16.9) vs 51.7 (16.7) years, respectively. The cohorts were matched by zip code, and the distribution of states of residence was correspondingly equivalent. Patients resided in 15 different states. The median follow-up period was 111 days (IQR, 102-131 days) in both groups (Table 1).

Table 1.

Clinical Characteristics of Vaccinated and 1:10 Propensity-Matched Unvaccinated Cohorts

Clinical covariate	Individuals, No. (%)		Standardized mean difference
	Vaccinated cohort (n = 8889)	1:10 Propensity-matched unvaccinated cohort (n = 88 898)
Age (overall), y	52.4 (16.9)	51.7 (16.7)	0.04a
18-24	697 (7.8)	7286 (8.2)	0.01a
25-34	932 (10.5)	9752 (11.0)	0.02a
35-44	1360 (15.3)	13 978 (15.7)	0.01a
45-54	1556 (17.5)	15 547 (17.5)	0.00a
55-64	2019 (22.7)	20 980 (23.6)	0.02a
65-74	1647 (18.5)	16 424 (18.5)	0.00a
≥75	678 (7.6)	4931 (5.5)	0.09a
Ethnicity
Hispanic or Latino	312 (3.5)	3060 (3.4)	0.00a
Not Hispanic or Latino	8202 (92.3)	81 458 (91.6)	0.02a
Unknown	375 (4.2)	4380 (4.9)	0.03a
Sex
Female	4397 (49.5)	44 140 (49.7)	0.00a
Male	4491 (50.5)	44 748 (50.3)	0.00a
Unknown	1 (0.01)	10 (0.01)	0.00a
Patients in long-term care	22 (0.2)	319 (0.4)	0.02a
Race
Asian	191 (2.1)	1605 (1.8)	0.03a
Black or African American	274 (3.1)	2281 (2.6)	0.03a
Native American	31 (0.3)	267 (0.3)	0.01a
Native Hawaiian or Pacific Islander	12 (0.1)	100 (0.1)	0.01a
White	7945 (89.4)	79 692 (89.6)	0.01a
Other	222 (2.5)	2413 (2.7)	0.01a
Unknown	214 (2.4)	2540 (2.9)	0.03a
State of residence
Minnesota	5503 (61.9)	55 213 (62.1)	0.00a
Wisconsin	2192 (24.7)	21 817 (24.5)	0.00a
Arizona	541 (6.1)	5410 (6.1)	0.00a
Florida	514 (5.8)	5140 (5.8)	0.00a
Iowa	78 (0.9)	767 (0.9)	0.00a
Nevada	9 (0.1)	90 (0.1)	0.00a
North Dakota	27 (0.3)	258 (0.3)	0.00a
Arkansas	2 (0.02)	8 (0.01)	0.01a
Nebraska	2 (0.02)	20 (0.02)	0.00a
Illinois	9 (0.10)	85 (0.10)	0.00a
Georgia	1 (0.01)	10 (0.01)	0.00a
South Dakota	8 (0.09)	59 (0.07)	0.01a
Washington	1 (0.01)	8 (0.01)	0.00a
Michigan	1 (0.01)	6 (0.01)	0.01a
Oregon	1 (0.01)	7 (0.01)	0.00a
Previous PCR tests (overall)	1.098 (1.327)	1.136 (0.963)	0.04a
0	3734 (42.0)	19 331 (21.8)	0.48
1	2789 (31.4)	49 252 (55.4)	0.49
2	1183 (13.3)	13 455 (15.1)	0.05a
3	521 (5.9)	4020 (4.5)	0.06a
4	265 (3.0)	1462 (1.6)	0.1a
≥5	397 (4.5)	1378 (1.6)	0.22a
Follow-up period, d
Mean (SD)	103 (246)	103 (246)
Median (IQR)	111 (102-131)	111 (102-131)

Abbreviation: PCR, polymerase chain reaction.

Highly balanced covariates with a standardized mean difference of less than 0.1. Covariates for balancing include (1) demographic characteristics (age, sex, race, and ethnicity), (2) number of prior PCR tests (number of PCR tests that the individual received before February 27, 2021), and (3) location (zip code). The zip code is matched exactly between the 2 cohorts, so the proportion of individuals in each state is identical.

During the study period, 60 of 8889 vaccinated individuals (0.7%) had a positive SARS-CoV-2 PCR test result compared with 2236 of 88 898 unvaccinated individuals (2.5%) (Table 2; Figure 2A). Polymerase chain reaction testing was performed for symptomatic and asymptomatic patients (eg, screening for travel or prior to an intervention). The incidence rates of positive SARS-CoV-2 test results in the vaccinated and unvaccinated cohorts were 0.065 and 0.25 cases per 1000 person-days, respectively, for an IRR of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts of 0.26 (95% CI, 0.20-0.34), indicating an overall vaccine effectiveness of 73.6% (95% CI, 65.9%-79.9%) and a 3.73-fold reduction in SARS-CoV-2 infections (Table 2; Figure 2A).

Table 2.

SARS-CoV-2 Incidence Rates in Vaccinated Cohort and 1:10 Propensity-Matched Unvaccinated Cohorts and Corresponding Vaccine Effectiveness

Time perioda	Vaccinated		Unvaccinated		Incidence rate ratio (exact 95% CI)	Vaccine effectiveness, % (95% CI)
	Cases/person-days (per 1000 person-days)	No. of patients contributing	Cases/person-days (per 1000 person-days)	No. of patients contributing
Day 1 onward	60/924 405 (0.065)	8889	2236/9 095 980 (0.25)	88 898	0.26 (0.20-0.34)	73.6 (65.9-79.9)
Day 8 onward	51/862 229 (0.059)	8834	1849/8 474 875 (0.22)	88 052	0.27 (0.20-0.36)	72.9 (64.2-79.9)
Day 15 onward	41/800 599 (0.051)	8698	1561/7 860 932 (0.2)	86 495	0.26 (0.18-0.35)	74.2 (64.9-81.6)

Relative to vaccine dose for vaccinated cohort or study enrollment day for unvaccinated cohort.

Figure 2.

Kaplan-Meier Analyses of Cumulative Incidence Proportion of SARS-CoV-2 Infections Between Vaccinated and Unvaccinated Individuals

A, SARS-CoV-2 infection at any time after vaccination. Cumulative incidence proportion at time t is the estimated proportion of patients who experience the outcome on or before time t (ie, 1 minus the standard Kaplan-Meier survival estimate with onset on any day after the date of vaccination): 60 of 8889 vaccinated and 2236 of 88 898 unvaccinated (log-rank P = .001). B, SARS-CoV-2 infection within 14 days after infection. Cumulative incidence proportion: 18 vaccinated and 473 unvaccinated (log-rank P = 6.93 × 10-4). PCR indicates polymerase chain reaction.

The full effectiveness of the Ad26.COV2.S vaccine is expected to be achieved after several weeks.[3] Thus, we next analyzed the number of new SARS-CoV-2 infections starting 15 days after the study enrollment date (Table 2; Figure 2B) until the end of the study period (July 22, 2021). Of the 8698 vaccinated individuals with at least 2 weeks of follow-up, 41 (0.5%) tested positive for SARS-CoV-2 15 days or more after vaccination compared with 1561 of 86 495 unvaccinated individuals (1.8%) (a 3.91-fold reduction). This corresponds to a vaccine effectiveness of 74.2% (95% CI, 64.9%-81.6%) in preventing SARS-CoV-2 infection, with onset at least 2 weeks after vaccination (Table 2; Figure 2B). This result is similar to the effectiveness from day 1 and suggests that an immune response occurs more quickly than anticipated (<14 days).

Infection can occur after vaccination if an individual is exposed to SARS-CoV-2 shortly after vaccination (ie, before building immunity) or if the vaccine fails to elicit protective immunity. To distinguish between these 2 scenarios, we analyzed the distribution of time between vaccination and the first positive PCR test result in these 2 cohorts (Figure 3A). The median times to a positive PCR test result after vaccination were similar in the vaccinated and unvaccinated cohorts: 27 days (IQR, 12-49 days) vs 28 days (IQR, 12-79 days), respectively, with 19 of 60 infections vs 641 of 2236 infections. These results suggest that most infections in the vaccinated cohort are likely not due to viral exposure before the recipient was able to develop immunity. In addition, in Figure 3B, the incidence of infections in vaccinated individuals appears to parallel the increased frequency of the Delta variant (eFigure 2 in the Supplement), raising the concern of decreasing vaccine protection against variants.

Figure 3.

Temporal Distribution of Positive Polymerase Chain Reaction (PCR) Test Results in Vaccinated and Unvaccinated Individuals

A, Distribution of the time to the first positive PCR test result. B, Distribution of the date of positive PCR test results.

Rate of Severe COVID-19 Disease After Ad26.COV2.S Vaccination

To understand the effectiveness of the Ad26.COV2.S vaccine in preventing severe COVID-19 (hospitalization, ICU admission, and death), we analyzed the number of hospitalizations, ICU admissions, and deaths among the 60 vaccinated individuals who tested positive for SARS-CoV-2 and the 2236 unvaccinated individuals who tested positive for SARS-CoV-2. A composite outcome showed decreased odds of severe events in vaccinated individuals (odds ratio [OR], 0.33 [95% CI, 0.19-0.65]; P < .001). In subgroup analyses, we observed decreased odds of hospitalization (OR, 0.32 [95% CI, 0.18-0.66]; P = 2.8 × 10−4) and ICU admission (OR, 0.00 [95% CI, 0.00-1.43]; P = .01) but not mortality (OR, 0.83 [95% CI, 0.26-5.20]; P > .99) among vaccinated individuals (eTable 1 in the Supplement). The number of fatal events was low in both groups (1 of 8888 in vaccinated participants vs 12 of 88 886 in unvaccinated participants).

Currently, vaccination is prioritized for older individuals and at-risk individuals, which biases the comparison with the unvaccinated cohort, who are younger and healthier. To control for this bias, we designed a cohort of unvaccinated patients matching the 60 vaccinated patients who were infected with SARS-CoV-2 after vaccination (Figure 1B; see Methods). Using 1:10 ratio matching, we matched the cohort based on a propensity score derived from demographic characteristics (age, sex, race, and ethnicity) and risk factors for severe COVID-19 illness provided by the Centers for Disease Control and Prevention[15] (see Methods). The matching results are summarized in eTable 2 in the Supplement. There were no significant differences between the 2 cohorts in any of the clinical covariates that were included in propensity score matching. We then compared the 14-day hospitalizations, ICU admissions, and deaths between these cohorts (eTable 3 in the Supplement). We did not observe any significant differences in the composite outcome or in any individual analyses. However, only 41 vaccinated patients with COVID-19 had sufficient follow-up (≥14 days) for inclusion in this analysis. In conclusion, our results suggest a reduction in the severity of COVID-19 with the Ad26.COV2.S vaccine, although further analysis with more patients and events will be required for more precise estimates of the protection against hospitalization, ICU admission, and mortality.

Discussion

In this study, we assessed the effectiveness of the Ad26.COV2.S COVID-19 vaccine (Johnson & Johnson and Janssen) by analyzing longitudinal health records of 8889 vaccinated individuals and a 1:10 propensity-matched unvaccinated cohort in the multistate Mayo Clinic Health System (Minnesota, Arizona, Florida, Wisconsin, and Iowa). We observed reductions in rates of newly diagnosed COVID-19 infection, with an overall effectiveness of 73.6% (95% CI, 65.9%-79.9%) and effectiveness of 74.2% (95% CI, 64.9%-81.6%) after 14 days. These findings are consistent with the results of the phase 3 trial of Ad26.COV2.S, which demonstrated 66.9% efficacy against moderate to severe COVID-19 with onset at least 14 days after vaccination.[3] Our results highlight the effectiveness of the vaccine in a cohort in which the SARS-Cov-2 Alpha and Delta variants were the predominant circulating viruses in comparison with the original clinical trials.

Infections in vaccinated individuals were observed but were rare (0.7% of vaccinated individuals). They do not appear to be concentrated close to the injection date, which argues against an incubation period needed to generate a protective immune response. A spike of COVID-19 cases seems to be associated with the emergence of the Delta variant in June and July 2021, suggesting that the vaccine fails to trigger a protective immune response in a subset of patients, in particular with the emergence of variants. However, the number of cases is low, and the emergence of the Delta variant is too recent to be definitive.

Clinical trials have demonstrated that multiple COVID-19 vaccines, including Ad26.COV2.S, are highly efficacious in reducing the risk of severe illness. Our analysis shows a reduction in severe cases, in particular for risks of hospitalization (and trends for reducing the risk of ICU admission and mortality). However, because only 60 individuals tested positive for SARS-CoV-2 after receiving the Ad26.COV2.S vaccine, our study was underpowered for definitive assessment of mortality. As the vaccine is administered to more patients, we will continue to assess the rates of hospitalization, ICU admission, and mortality among individuals who become infected after Ad26.COV2.S vaccination.

Limitations

Biases, such as the care-seeking bias, and differences in COVID-19 incidence rates and comorbid conditions were controlled for using propensity-score matching for the number of PCR tests, geographical location, and comorbid conditions. However, this study has several limitations. First, the study was conducted using data from a single health system, with most individuals residing in Minnesota or Wisconsin. The cohorts, which are more than 90% White and approximately 49.5% female, are not demographically representative of the broader US or world population that is now eligible for vaccination. Second, the cohort analyzed is relatively small compared with the population analyzed in the phase 3 trial of the Ad26.COV2.S vaccination and compared with prior real-world analyses of FDA-authorized mRNA vaccines.[3,6] Third, our study does not take into account receipt of care outside the Mayo Clinic system, and participants might have received a PCR test that we would not detect in the Mayo Clinic EHR. Finally, sequencing information at the patient level was not available, so the prevalence of the Alpha and Delta variants was approximated by the GISAID population-level prevalence. We have assumed that the rate of such tests is equal in both cohorts, limiting the bias of censored data.

Conclusions

Despite these caveats, to our knowledge, this study is the first using EHR data in a propensity score–matched retrospective analysis of the effectiveness of the Ad26.COV2.S vaccine performed after emergency use authorization. Implementation of this framework will allow us to track in real time how the effectiveness of this 1-shot vaccine continues to evolve over the coming weeks and months. This information is particularly important in the context of the emergence of variants that could potentially escape vaccine-induced immunity. The extraction of such data from holistic health records is critical in shaping the global race against the ongoing pandemic.