Serious adverse events of special interest following mRNA COVID-19 vaccination in randomized trials in adults.

INTRODUCTION: In 2020, prior to COVID-19 vaccine rollout, the Brighton Collaboration created a priority list, endorsed by the World Health Organization, of potential adverse events relevant to COVID-19 vaccines. We adapted the Brighton Collaboration list to evaluate serious adverse events of special interest observed in mRNA COVID-19 vaccine trials.
METHODS: Secondary analysis of serious adverse events reported in the placebo-controlled, phase III randomized clinical trials of Pfizer and Moderna mRNA COVID-19 vaccines in adults (NCT04368728 and NCT04470427), focusing analysis on Brighton Collaboration adverse events of special interest.
RESULTS: Pfizer and Moderna mRNA COVID-19 vaccines were associated with an excess risk of serious adverse events of special interest of 10.1 and 15.1 per 10,000 vaccinated over placebo baselines of 17.6 and 42.2 (95 % CI -0.4 to 20.6 and -3.6 to 33.8), respectively. Combined, the mRNA vaccines were associated with an excess risk of serious adverse events of special interest of 12.5 per 10,000 vaccinated (95 % CI 2.1 to 22.9); risk ratio 1.43 (95 % CI 1.07 to 1.92). The Pfizer trial exhibited a 36 % higher risk of serious adverse events in the vaccine group; risk difference 18.0 per 10,000 vaccinated (95 % CI 1.2 to 34.9); risk ratio 1.36 (95 % CI 1.02 to 1.83). The Moderna trial exhibited a 6 % higher risk of serious adverse events in the vaccine group: risk difference 7.1 per 10,000 (95 % CI -23.2 to 37.4); risk ratio 1.06 (95 % CI 0.84 to 1.33). Combined, there was a 16 % higher risk of serious adverse events in mRNA vaccine recipients: risk difference 13.2 (95 % CI -3.2 to 29.6); risk ratio 1.16 (95 % CI 0.97 to 1.39). DISCUSSION: The excess risk of serious adverse events found in our study points to the need for formal harm-benefit analyses, particularly those that are stratified according to risk of serious COVID-19 outcomes. These analyses will require public release of participant level datasets.

Introduction

In March 2020, the Brighton Collaboration and the Coalition for Epidemic Preparedness Innovations partnership, Safety Platform for Emergency vACcines (SPEAC), created and subsequently updated a “priority list of potential adverse events of special interest relevant to COVID-19 vaccine trials.” [1] The list comprises adverse events of special interest (AESIs) based on the specific vaccine platform, adverse events associated with prior vaccines in general, theoretical associations based on animal models, and COVID-19 specific immunopathogenesis. [1] The Brighton Collaboration is a global authority on the topic of vaccine safety and in May 2020, the World Health Organization’s Global Advisory Committee on Vaccine Safety endorsed and recommended the reporting of AESIs based on this priority list. To our knowledge, however, the list has not been applied to serious adverse events in randomized trial data.

We sought to investigate the association between FDA-authorized mRNA COVID-19 vaccines and serious adverse events identified by the Brighton Collaboration, using data from the phase III randomized, placebo-controlled clinical trials on which authorization was based. We consider these trial data against findings from post-authorization observational safety data. Our study was not designed to evaluate the overall harm-benefit of vaccination programs so far. To put our safety results in context, we conducted a simple comparison of harms with benefits to illustrate the need for formal harm-benefit analyses of the vaccines that are stratified according to risk of serious COVID-19 outcomes. Our analysis is restricted to the randomized trial data, and does not consider data on post-authorization vaccination program impact. It does however show the need for public release of participant level trial datasets.

Methods

Pfizer and Moderna each submitted the results of one phase III randomized trial in support of the FDA’s emergency use authorization of their vaccines in adults. Two reviewers (PD and RK) searched journal publications and trial data on the FDA’s and Health Canada’s websites to locate serious adverse event results tables for these trials. The Pfizer and Moderna trials are expected to follow participants for two years. Within weeks of the emergency authorization, however, the sponsors began a process of unblinding all participants who elected to be unblinded. In addition, those who received placebo were offered the vaccine. These self-selection processes may have introduced nonrandom differences between vaccinated and unvaccinated participants, thus rendering the post-authorization data less reliable. Therefore, to preserve randomization, we used the interim datasets that were the basis for emergency authorization in December 2020, approximately 4 months after trials commenced.

The definition of a serious adverse event (SAE) was provided in each trial’s study protocol and included in the supplemental material of the trial’s publication. [2], [3], [4] Pfizer and Moderna used nearly identical definitions, consistent with regulatory expectations. An SAE was defined as an adverse event that results in any of the following conditions: death; life-threatening at the time of the event; inpatient hospitalization or prolongation of existing hospitalization; persistent or significant disability/incapacity; a congenital anomaly/birth defect; medically important event, based on medical judgment.

In addition to journal publications, we searched the websites of the FDA (for advisory committee meeting materials) and Health Canada (for sections of the dossier submitted by sponsors to the regulator). [5] For the FDA website, we considered presentations by both the FDA and the sponsors. [6] Within each of these sources, we searched for SAE results tables that presented information by specific SAE type; we chose the most recent SAE table corresponding to the FDA’s requirement for a safety median follow-up time of at least 2 months after dose 2.

For each trial, we prepared blinded SAE tables (containing SAE types without results data). Using these blinded SAE tables, two clinician reviewers (JF and JE) independently judged whether each SAE type was an AESI. SAE types that matched an AESI term verbatim, or were an alternative diagnostic name for an AESI term, were included as an AESI. For all other SAE types, the reviewers independently judged whether that SAE type was likely to have been caused by a vaccine-induced AESI, based on a judgment considering the disease course, causative mechanism, and likelihood of the AESI to cause the SAE type. Disagreements were resolved through consensus; if consensus could not be reached, a third clinician reviewer (PW) was used to create a majority opinion. For each included SAE, we recorded the corresponding Brighton Collaboration AESI category and organ system. When multiple AESIs could potentially cause the same SAE, the reviewers selected the AESI that they judged to be the most likely cause based on classical clinical presentation of the AESI.

We used an AESI list derived from the work of Brighton Collaboration’s Safety Platform for Emergency vACcines (SPEAC) Project. This project created an AESI list which categorizes AESIs into three categories: those included because they are seen with COVID-19, those with a proven or theoretical association with vaccines in general, and those with proven or theoretical associations with specific vaccine platforms. The first version was produced in March 2020 based on experience from China. Following the second update (May 2020), the WHO Global Advisory Committee on Vaccine Safety (GACVS) adopted the list, and Brighton commenced a systematic review process “to ensure an ongoing understanding of the full spectrum of COVID-19 disease and modification of the AESI list accordingly.” [7] This resulted in three additional AESIs being added to the list in December 2020. The subsequent (and most recent fourth) update did not result in any additional AESIs being added to the list. [1].

We matched SAEs recorded in the trial against an expanded list of AESIs created by combining Brighton’s SPEAC COVID-19 AESI list with a list of 29 clinical diagnoses Brighton identified as “known to have been reported but not in sufficient numbers to merit inclusion on the AESI list.” [7] Sensitivity analysis was used to determine whether use of the original versus expanded list altered our results.

Risk ratios and risk differences between vaccine and placebo groups were calculated for the incidence of AESIs and SAEs. We excluded SAEs that were known efficacy outcomes (i.e. COVID-19), consistent with the approach Pfizer (but not Moderna) used in recording SAE data. The Pfizer study trial protocol states that COVID-19 illnesses and their sequelae consistent with the clinical endpoint definition were not to be reported as adverse events, “even though the event may meet the definition of an SAE.” [8] For unspecified reasons, Moderna included efficacy outcomes in their SAE tables, effectively reporting an all-cause SAE result. Because we did not have access to individual participant data, to account for the occasional multiple SAEs within single participants, we reduced the effective sample size by multiplying standard errors in the combined SAE analyses by the square root of the ratio of the number of SAEs to the number of patients with an SAE. This adjustment increased standard errors by 10 % (Pfizer) and 18 % (Moderna), thus expanding the interval estimates. We estimated combined risk ratios and risk differences for the two mRNA vaccines by averaging over the risks using logistic regression models which included indicators for trial and treatment group.

We used a simple harm-benefit framework to place our results in context, comparing risks of excess serious AESIs against reductions in COVID-19 hospitalization.

Results

Serious adverse event tables were located for each of the vaccine trials submitted for EUA in adults (age 16 + for Pfizer, 18 + for Moderna) in the United States: Pfizer-BioNTech COVID-19 vaccine BNT162b2 (NCT04368728) [2], [9], [10] and Moderna COVID-19 vaccine mRNA-1273 (NCT04470427). [3], [11], [12] ( Table 1 ).

Table 1

Data sources for phase III trials.

Trial	Data cutoff date	Journal articles	FDA sources	Health Canada sources
Pfizer trial in ages 16 and above (NCT04368728)	14 Nov 2020 (supported Dec 2020 EUA)	Aggregate data only	Table 23in sponsor briefing document	Table 55in sponsor document C4591001 Final Analysis Interim Report Body
Moderna trial in ages 18 and above (NCT04470427)	25 Nov 2020 (supported Dec 2020 EUA)	Table S11 in publication	Table 27 in sponsor briefing document	Table 14.3.1.13.3in sponsor document mRNA-1273-P301 Unblinded Safety Tables Batch 1 (DS2)

Note: bolded font indicates dataset chosen for analysis; EUA = Emergency Use Authorization.

Reporting windows and serious adverse events

Moderna reported SAEs from dose 1 whereas Pfizer limited reporting from dose 1 to 1 month after dose 2. Both studies reported all data at the time of data cutoff (14 Nov 2020 for Pfizer, 25 Nov 2020 for Moderna). 17 SAEs that were efficacy endpoints were removed from the Moderna trial (16 “COVID-19” SAEs and 1 “COVID-19 pneumonia” SAE). One such efficacy endpoint meeting the definition of a SAE was removed from the Pfizer trial (“SARS-CoV-2 test positive” SAE).

The Pfizer trial exhibited a 36 % higher risk of serious adverse events in vaccinated participants in comparison to placebo recipients: 67.5 per 10,000 versus 49.5 per 10,000; risk difference 18.0 per 10,000 vaccinated participants (95 % compatibility1 interval 1.2 to 34.9); risk ratio 1.36 (95 % CI 1.02 to 1.83). The Moderna trial exhibited a 6 % higher risk of SAEs in vaccinated individuals compared to those receiving placebo: 136 per 10,000 versus 129 per 10,000; risk difference 7.1 per 10,000 (95 % CI –23.2 to 37.4); risk ratio 1.06 (95 % CI 0.84 to 1.33). Combined, there was a 16 % higher risk of SAEs in mRNA vaccine recipients than placebo recipients: 98 per 10,000 versus 85 per 10,000; risk difference 13.2 (95 % CI −3.2 to 29.6); risk ratio 1.16 (95 % CI 0.97 to 1.39). ( Table 2 ).

Table 2

Serious adverse events.

	Total events (events per 10,000 participants)a		Risk differenceper 10,000 participants(95 % CI)e	Risk ratio(95 % CI)e
Trial	Vaccine	Placebo
Serious adverse events
Pfizerb	127 (67.5)	93 (49.5)	18.0 (1.2 to 34.9)	1.36 (1.02 to 1.83)
Modernac, d	206 (135.7)	195 (128.6)	7.1 (–23.2 to 37.4)	1.06 (0.84 to 1.33)
Combinedf	333 (98.0)	288 (84.8)	13.2 (-3.2 to 29.6)	1.16 (0.97 to 1.39)
Serious adverse events of special interest
Pfizer	52 (27.7)	33 (17.6)	10.1 (-0.4 to 20.6)	1.57 (0.98 to 2.54)
Moderna	87 (57.3)	64 (42.2)	15.1 (-3.6 to 33.8)	1.36 (0.93 to 1.99)
Combinedf	139 (40.9)	97 (28.6)	12.5 (2.1 to 22.9)	1.43 (1.07 to 1.92)

Denominators for Pfizer were 18,801 in the vaccine group and 18,785 in the placebo group, and for Moderna were 15,185 in the vaccine group and 15,166 in the placebo group.

Pfizer excluded efficacy outcomes from its SAE table (COVID-19 illnesses and their sequelae meeting the definition of an SAE). However, at least one SAE appears to have been inadvertently included, which we removed from our calculations (“SARS-CoV-2 test positive”: 0 vaccine group; 1 placebo group).

Moderna included efficacy outcomes in its SAE table (COVID-19 illnesses and their sequelae meeting the definition of an SAE). We removed efficacy SAEs outcomes that could be identified: “COVID-19” and “COVID-19 pneumonia.” Lacking access to participant level data, SAEs that were sequelae of serious COVID-19 could not be identified and therefore remain included in this analysis.

“All SAEs” for Moderna was calculated using the “Number of serious AEs” row in Moderna’s submission to FDA.11.

Standard errors used to estimate 95% CIs were inflated by the factor √[#SAE]/[#patients with SAE] to account for multiple SAE within patients.

The combined risk differences and risk ratios were computed from the fitted logistic regression models and so may not exactly equal comparisons computed from the first two columns.

Serious adverse events of special interest

Regarding whether each SAE type was included on the SPEAC derived AESI list, agreement between the two independent clinician reviewers was 86 % (281/325); 40 of the 44 disagreements were resolved through consensus, and only four disagreements necessitated a third clinician reviewer. Supplemental Table 1 includes a full list of included and excluded SAEs across both trials.

In the Pfizer trial, 52 serious AESI (27.7 per 10,000) were reported in the vaccine group and 33 (17.6 per 10,000) in the placebo group. This difference corresponds to a 57 % higher risk of serious AESI (RR 1.57 95 % CI 0.98 to 2.54) and a risk difference of 10.1 serious AESI per 10,000 vaccinated participants (95 % CI −0.4 to 20.6). In the Moderna trial, 87 serious AESI (57.3 per 10,000) were reported in the vaccine group and 64 (42.2 per 10,000) in the placebo group. This difference corresponds to a 36 % higher risk of serious AESI (RR 1.36 95 % CI 0.93 to 1.99) and a risk difference of 15.1 serious AESI per 10,000 vaccinated participants (95 % CI −3.6 to 33.8). Combining the trials, there was a 43 % higher risk of serious AESI (RR 1.43; 95 % CI 1.07 to 1.92) and a risk difference of 12.5 serious AESI per 10,000 vaccinated participants (95 % CI 2.1 to 22.9). ( Table 2 ).

Of the 236 serious AESIs occurring across the Pfizer and Moderna trials, 97 % (230/236) were adverse event types included as AESIs because they are seen with COVID-19. In both Pfizer and Moderna trials, the largest excess risk occurred amongst the Brighton category of coagulation disorders. Cardiac disorders have been of central concern for mRNA vaccines; in the Pfizer trial more cardiovascular AESIs occurred in the vaccine group than in the placebo group, but in the Moderna trial the groups differed by only 1 case. ( Table 3, Table 4 ).

Table 3

Serious AESIs, Pfizer trial.

Brighton category	Vaccine	Placebo	Vaccine events per 10,000	Placebo events per 10,000	Difference in events per 10,000	Risk ratio
Association with immunization in general
Anaphylaxis	1	1	0.5	0.5	0.0	1.00
Association with specific vaccine platform(s)
Encephalitis/encephalomyelitis	0	2	0.0	1.1	−1.1	0.00
Seen with COVID-19
Acute kidney injury	2	0	1.1	0.0	1.1	N/A
Acute liver injury	0	1	0.0	0.5	−0.5	0.00
Acute respiratory distress syndrome	2	1	1.1	0.5	0.5	2.00
Coagulation disorder	16	10	8.5	5.3	3.2	1.60
Myocarditis/pericarditis	2	1	1.1	0.5	0.5	2.00
Other forms of acute cardiac injury	16	12	8.5	6.4	2.1	1.33
Subtotal	39	28	20.7	14.9	5.8	1.39
Brighton list of 29 clinical diagnoses seen with COVID-19
Abscess	4	1	2.1	0.5	1.6	4.00
Cholecystitis	4	2	2.1	1.1	1.1	2.00
Colitis/Enteritis	1	1	0.5	0.5	0.0	1.00
Diarrhea	1	0	0.5	0.0	0.5	N/A
Hyperglycemia	1	1	0.5	0.5	0.0	1.00
Pancreatitis	1	0	0.5	0.0	0.5	N/A
Psychosis	1	0	0.5	0.0	0.5	N/A
Subtotal	13	5	6.9	2.7	4.3	2.60
Total	52	33	27.7	17.6	10.1	1.57

Table 4

Serious AESIs, Moderna trial.

Brighton category	Vaccine	Placebo	Vaccine events per 10,000	Placebo events per 10,000	Difference in events per 10,000	Risk ratio
Association with specific vaccine platform(s)
Bell's Palsy	1	0	0.7	0.0	0.7	N/A
Encephalitis/encephalomyelitis	1	0	0.7	0.0	0.7	N/A
Seen with COVID-19
Acute kidney injury	1	3	0.7	2.0	−1.3	0.33
Acute liver injury	1	0	0.7	0.0	0.7	N/A
Acute respiratory distress syndrome	7	4	4.6	2.6	2.0	1.75
Angioedema	0	2	0.0	1.3	−1.3	0.00
Coagulation disorder	20	13	13.2	8.6	4.6	1.54
Generalized Convulsions	2	0	1.3	0.0	1.3	N/A
Myelitis	0	1	0.0	0.7	−0.7	0.00
Myocarditis/pericarditis	4	5	2.6	3.3	−0.7	0.80
Other forms of acute cardiac injury	26	26	17.1	17.1	0.0	1.00
Other rash	1	1	0.7	0.7	0.0	1.00
Rhabdomyolysis	0	1	0.0	0.7	−0.7	0.00
Single Organ Cutaneous Vasculitis	1	0	0.7	0.0	0.7	N/A
Subtotal	65	56	42.8	36.9	5.9	1.16
Brighton list of 29 clinical diagnoses seen with COVID-19
Abscess	1	0	0.7	0.0	0.7	N/A
Arthritis	3	1	2.0	0.7	1.3	3.00
Cholecystitis	4	0	2.6	0.0	2.6	N/A
Colitis/Enteritis	6	3	4.0	2.0	2.0	2.00
Diarrhea	2	1	1.3	0.7	0.7	2.00
Hyperglycemia	1	0	0.7	0.0	0.7	N/A
Hyponatremia	1	1	0.7	0.7	0.0	1.00
Pancreatitis	2	0	1.3	0.0	1.3	N/A
Pneumothorax	0	1	0.0	0.7	−0.7	0.00
Psychosis	1	1	0.7	0.7	0.0	1.00
Thyroiditis	1	0	0.7	0.0	0.7	N/A
Subtotal	22	8	14.5	5.3	9.2	2.75
Total	87	64	57.3	42.2	15.1	1.36

Sensitivity analysis

As a sensitivity analysis, we restricted the serious AESI analysis to those AESIs listed in SPEAC’s COVID-19 AESI list (i.e. separating out Brighton’s list of 29 clinical diagnoses “known to have been reported but not in sufficient numbers to merit inclusion on the AESI list.”) This reduced the total number of AESIs across the two trials by 48 (35 vaccine group, 13 placebo group). There was still a higher risk of serious AESI when limited to the SPEAC COVID-19 AESI list, but the magnitude of the excess (in both relative and absolute terms) was smaller than when using the larger AESI list. (Supplemental Table 2 ).

Harm-benefit considerations

In the Moderna trial, the excess risk of serious AESIs (15.1 per 10,000 participants) was higher than the risk reduction for COVID-19 hospitalization relative to the placebo group (6.4 per 10,000 participants). [3] In the Pfizer trial, the excess risk of serious AESIs (10.1 per 10,000) was higher than the risk reduction for COVID-19 hospitalization relative to the placebo group (2.3 per 10,000 participants).

Comparison with FDA reviews

In their review of SAEs supporting the authorization of the Pfizer and Moderna vaccines, the FDA concluded that SAEs were, for Pfizer, “balanced between treatment groups,” [15] and for Moderna, were “without meaningful imbalances between study arms.” [16] In contrast to the FDA analysis, we found an excess risk of SAEs in the Pfizer trial. Our analysis of Moderna was compatible with FDA’s analysis, finding no meaningful SAE imbalance between groups.

The difference in findings for the Pfizer trial, between our SAE analysis and the FDA’s, may in part be explained by the fact that the FDA analyzed the total number of participants experiencing any SAE, whereas our analysis was based on the total number of SAE events. Given that approximately twice as many individuals in the vaccine group than in the placebo group experienced multiple SAEs (there were 24 more events than participants in the vaccine group, compared to 13 in the placebo group), FDA’s analysis of only the incidence of participants experiencing any SAE would not reflect the observed excess of multiple SAEs in the vaccine group.

A more important factor, however, may be that FDA’s review of non-fatal SAEs used a different analysis population with different follow-up windows. The FDA reported 126 of 21,621 (0.6 %) of vaccinated participants experienced at least one SAE at data cutoff compared to 111 of 21,631 (0.5 %) of placebo participants. In contrast, our analysis found 127 SAEs among 18,801 vaccine recipients versus 93 SAEs among 18,785 placebo recipients. [15] While summary results for the population we analyzed was provided in a table, FDA did not report an analysis of them. The substantially larger denominators in FDA’s analysis (5,666 more participants) reflect the fact that their analysis included all individuals receiving at least one dose (minus 196 HIV-positive participants), irrespective of the duration of post-injection follow-up time. In contrast, our analysis was based on the study population with median follow-up ≥ 2 months after dose 2 (minus 120 HIV-positive participants), of which 98.1 % had received both doses. [2], [17] The FDA’s analysis of SAEs thus included thousands of additional participants with very little follow-up, of which the large majority had only received 1 dose.

Comparison with post-authorization studies

Although the randomized trials offer high level evidence for evaluating causal effects, the sparsity of their data necessitates that harm-benefit analyses also consider observational studies. Since their emergency authorization in December 2020, hundreds of millions of doses of Pfizer and Moderna COVID-19 vaccines have been administered and post-authorization observational data offer a complementary opportunity to study AESIs. Post-authorization observational safety studies include cohort studies (which make use of medical claims or electronic health records) and disproportionality analyses (which use spontaneous adverse event reporting systems). In July 2021, the FDA reported detecting four potential adverse events of interest: pulmonary embolism, acute myocardial infarction, immune thrombocytopenia, and disseminated intravascular coagulation following Pfizer’s vaccine based on medical claims data in older Americans. [18] Three of these four serious adverse event types would be categorized as coagulation disorders, which is the Brighton AESI category that exhibited the largest excess risk in the vaccine group in both the Pfizer and Moderna trials. FDA stated it would further investigate the findings but at the time of our writing has not issued an update. Similarly, spontaneous-reporting systems have registered serious adverse reactions including anaphylaxis (all COVID-19 vaccines), thrombocytopenia syndrome among premenopausal females (Janssen vaccine), and myocarditis and pericarditis among younger males (Pfizer and Moderna vaccines). [19], [20].

Using data from three postmarketing safety databases for vaccines (VAERS, EudraVigilance, and VigiBase), disproportionality studies have reported excess risks for many of the same SAE types as in the present study. [21], [22], [23] For example, a study using VAERS and EudraVigilance comparing the disproportionality of adverse event reports between the influenza vaccine versus the mRNA COVID-19 vaccines reported excess risks for the following Brighton AESIs: cardiovascular events, coagulation events, hemorrhages, gastrointestinal events, and thromboses. [22] While CDC published a protocol[24] in early 2021 for using proportional reporting ratios for signal detection in the VAERS database, results from the study have not yet been reported. [25] Among self-controlled case series, one reported a rate ratio of 1.38 (95 % CI 1.12–1.71) for hemorrhagic stroke following Pfizer vaccine, [26] another reported 0.97 (95 % CI 0.81–1.15), [27] while a cohort study[28] reported 0.84 (95 % CI 0.54–1.27).

Discussion

Using a prespecified list of AESI identified by the Brighton Collaboration, higher risk of serious AESI was observed in the mRNA COVID-19 vaccine group relative to placebo in both the Pfizer and Moderna adult phase III trials, with 10.1 (Pfizer) and 15.1 (Moderna) additional events for every 10,000 individuals vaccinated. Combined, there was a risk difference of 12.5 serious AESIs per 10,000 individuals vaccinated (95 % CI 2.1 to 22.9). These results raise concerns that mRNA vaccines are associated with more harm than initially estimated at the time of emergency authorization. In addition, our analysis identified a 36 % higher risk of serious adverse events in vaccinated participants in the Pfizer trial: 18.0 additional SAEs per 10,000 vaccinated (95 % CI 1.2 to 34.9). Consistent with the FDA evaluation, our analysis found no clear difference in SAEs between groups in the Moderna trial.

Results between the Pfizer and Moderna trials were similar for the AESI analysis but exhibited substantial variation in the SAE analysis. Caution is needed in interpreting this variation as it may be substantially explained by differences in SAE recording practices in the trials rather than differences in actual vaccine harm profiles. For reasons that are not documented in the trial protocol, Moderna included efficacy outcomes in its SAE tabulations, while Pfizer excluded them. As a result, Moderna’s SAE table did not present a traditional SAE analysis but rather an all-cause SAE analysis. The FDA analysis of the Moderna trial presented an all-cause SAE analysis, which estimates total vaccine effects on SAEs, including effects transmitted via effects on COVID-19. It did not however present a traditional SAE analysis with efficacy endpoints removed, which attempts to estimate only the direct effects on SAEs. While our analysis attempted to perform a traditional SAE analysis by excluding efficacy SAEs (serious COVID-19 and its sequelae), our effort was hindered because we did not have access to patient level data. Easily recognizable efficacy SAEs (“COVID-19”, “COVID-19 pneumonia,” and “SARS-CoV-2 test positive”) could be removed, but many participants who experienced a COVID-19 SAE likely experienced multiple other SAEs (e.g. pneumonia, hypoxia, and thrombotic events) which could not be identified and therefore remain included in our analysis. Of 17 total efficacy SAEs (16 “COVID-19” and 1 “COVID-19 pneumonia”) removed from our analysis of the Moderna trial, 16 were in the placebo arm. As a consequence, the background SAE risk (risk in absence of COVID-19) would be overestimated by the Moderna placebo group, resulting in underestimation of the actual risk of SAEs and AESIs attributable to the vaccine in the Moderna comparisons as well as in the combined analysis. Access to patient-level data would allow adjustments for this problem.

Rational policy formation should consider potential harms alongside potential benefits. [29] To illustrate this need in the present context, we conducted a simple harm-benefit comparison using the trial data comparing excess risk of serious AESI against reductions in COVID-19 hospitalization. We found excess risk of serious AESIs to exceed the reduction in COVID-19 hospitalizations in both Pfizer and Moderna trials.

This analysis has the limitations inherent in most harm-benefit comparisons. First, benefits and harms are rarely exact equivalents, and there can be great variability in the degree of severity within both benefit and harm endpoints. For example, intubation and short hospital stay are not equivalent but both are counted in “hospitalization”; similarly, serious diarrhea and serious stroke are not equivalent but both are counted in “SAE.” Second, individuals value different endpoints differently. Third, without individual participant data, we could only compare the number of individuals hospitalized for COVID-19 against the number of serious AESI events, not the number of participants experiencing any serious AESI. Some individuals experienced multiple SAEs whereas hospitalized COVID-19 participants were likely only hospitalized once, biasing the analysis towards exhibiting net harm. To gauge the extent of this bias, we considered that there were 20 % (Pfizer) and 34 % (Moderna) more SAEs than participants experiencing any SAE. As a rough sensitivity calculation, if we divide the Pfizer excess serious AESI risk of 10.1 by 1.20 it becomes 8.4 compared to a COVID-19 hospitalization risk reduction of 2.3; if we divide the Moderna excess serious AESI risk of 15.1 by 1.34 it becomes 11.3 compared to a COVID-19 hospitalization risk reduction of 6.4.

Harm-benefit ratios will be different for populations at different risk for serious COVID-19 and observation periods that differ from those studied in the trials. Presumably, larger reductions in COVID-19 hospitalizations would have been recorded if trial follow-up were longer, more SARS-CoV-2 was circulating, or if participants had been at higher risk of serious COVID-19 outcomes, shifting harm-benefit ratios toward benefit. Conversely, harm-benefit ratios would presumably shift towards harm for those with lower risk of serious COVID-19 outcomes--such as those with natural immunity, younger age or no comorbidities. Similarly, waning vaccine effectiveness, decreased viral virulence, and increasing degree of immune escape from vaccines might further shift the harm-benefit ratio toward harm. Large, randomized trials in contemporary populations could robustly answer these questions. Absent definitive trials, however, synthesis of multiple lines of evidence will be essential. [[30], [48], [49]].

Adverse events detected in the post-marketing period have led to the withdrawal of several vaccines. An example is intussusception following one brand of rotavirus vaccine: around 1 million children were vaccinated before identification of intussusception, which occurred in around 1 per 10,000 vaccinees. [31] Despite the unprecedented scale of COVID-19 vaccine administration, the AESI types identified in our study may still be challenging to detect with observational methods. Most observational analyses are based on comparing the risks of adverse events “observed” against a background (or “expected”) risk, which inevitably display great variation, by database, age group, and sex. [32] If the actual risk ratio for the effect was 1.4 (the risk ratio of the combined AESI analysis), it could be quite difficult to unambiguously replicate it with observational data given concerns about systematic as well as random errors. [33], [34], [35].

In addition, disproportionality analyses following COVID-19 vaccination also have limitations, particularly with respect to the type of adverse events seen in our study. The majority of SAEs that contributed to our results are relatively common events, such as ischemic stroke, acute coronary syndrome, and brain hemorrhage. This complicates signal detection because clinical suspicion of an adverse vaccine reaction following an event commonly seen in clinical practice will be lower than for SAEs like myocarditis.[50] For this reason, clinical suspicion leading to the filing of an individual case safety report--may be far less common in the post-authorization setting than in the trials. At the same time, heightened awareness about COVID-19 vaccine SAEs can result in under and overreporting. Public health messages assuring vaccine safety may lower clinical suspicion of potential causal relationships, whereas messages about potential harms can conversely stimulate reports that otherwise may not have been made. These factors can lead to bias both directions, further complicating interpretation. In contrast to these problems, in the randomized trials used in this analysis, all SAEs were to be recorded, irrespective of clinical judgment regarding potential causality.

Although our analysis is secondary, reanalyses of clinical trial data have led to the detection of adverse events well after the market entry of major drugs such as rofecoxib and rosiglitazone. [36], [37] Our analysis has an advantage over postmarketing observational studies in that the data are from blinded, placebo-controlled randomized trials vetted by the FDA, which were matched against a list of adverse events created before the availability of the clinical-trial results and designed for use in COVID-19 vaccine trials.

Our study has several important limitations. First, Pfizer’s trial did not report SAEs occurring past 1 month after dose 2. This reporting threshold may have led to an undercounting of serious AESIs in the Pfizer trial. Second, for both studies, the limited follow up time prevented an analysis of harm-benefit over a longer period. Third, all SAEs in our analysis met the regulatory definition of a serious adverse event, but many adverse event types which a patient may themselves judge as serious may not meet this regulatory threshold. Fourth, decisions about which SAEs to include or exclude as AESIs requires subjective, clinical judgements in the absence of detailed clinical information about the actual SAEs. We encourage third party replication of our study, with access to complete SAE case narratives, to determine the degree to which these decisions affected our findings. For additional sensitivity analyses, such replication studies could also make use of other AESI lists, such as those prepared by FDA, [38], [39], [40], [41] CDC, [24], Pfizer, [42], or a de novo AESI list derived from a list of COVID-19 complications understood to be induced via SARS-CoV-2′s spike protein. [43], [44].

A fifth important limitation is our lack of access to individual participant data, which forced us to use a conservative adjustment to the standard errors. The 95 % CIs[13], [14] calculated are therefore only approximate because we do not know which patients had multiple events. Finally, as described above, in the Moderna analysis, the SAEs that were sequelae of serious COVID-19 could not be identified and therefore remain included in our calculations. Because the vaccines prevent SAEs from COVID-19 while adding SAE risks of their own, this inclusion makes it impossible to separately estimate SAEs due to the vaccine from SAEs due to COVID-19 in the available Moderna data, as must be done to extrapolate harm-benefit to other populations. These study limitations all stem from the fact that the raw data from COVID-19 vaccine clinical trials are not publicly available. [45], [46].

We emphasize that our investigation is preliminary, to point to the need for more involved analysis. The risks of serious AESIs in the trials represent only group averages. SAEs are unlikely to be distributed equally across the demographic subgroups enrolled in the trial, and the risks may be substantially less in some groups compared to others. Thus, knowing the actual demographics of those who experienced an increase in serious AESI in the vaccine group is necessary for a proper harm-benefit analysis. In addition, clinical studies are needed to see if particular SAEs can be linked to particular vaccine ingredients as opposed to unavoidable consequences of exposure to spike protein, as future vaccines could then be modified accordingly or sensitivities can be tested for in advance. In parallel, a systematic review and meta-analysis using individual participant data should be undertaken to address questions of harm-benefit in various demographic subgroups, particularly in those at low risk of serious complications from COVID-19. Finally, there is a pressing need for comparison of SAEs and harm-benefit for different vaccine types; some initial work has already begun in this direction. [47].

Full transparency of the COVID-19 vaccine clinical trial data is needed to properly evaluate these questions. Unfortunately, as we approach 2 years after release of COVID-19 vaccines, participant level data remain inaccessible. [45], [46].

Author contributions

All authors had full access to all of the data in the study (available at ), and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition of data: Doshi.

Analysis and interpretation: All authors.

Statistical analysis: Jones, Greenland.

Drafting of the manuscript: Fraiman, Doshi.

Critical revision of the manuscript for important intellectual content: All authors.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.