Controversies around the statistical presentation of data on mRNA-COVID 19 vaccine safety in pregnant women.

The work entitled "Preliminary Findings of mRNA Covid-19 Vaccine Safety in Pregnant Persons" published on April 21, 2021, in The New England Journal of Medicine, presented data collected from American surveillance systems and registries. However, problems with an unanimous interpretation of those results appeared in the public debate and citing articles. Some stated that the risk of miscarriage in vaccinated women was similar to historical values reported before the vaccines' approval. The others stated that risk was highly above-normative in women vaccinated during the first and second trimesters. We found several problems with the statistical treatment/interpretation of the originally presented values: a substantial percentage (up to 95.6%) of missing data, an incorrect denominator used for risk estimation, and too short follow-up that disabled the evaluation of the study's endpoint in numerous participants. Eventually, the Authors published a corrigendum on September 8, 2021, and pointed to updated data. Herein, we explain the statistical controversies raised by the original presentation and stress that analyzing the trade-off between knowledge and confusion brought by the release of incomplete results of such a high social interest, should aid in solving the dilemma of whether to publish preliminary data or none.

The development of anti-SARS-Cov2 mRNA vaccines and their subsequent approval by the national regulators worldwide was a milestone in a global race for COVID-19 prevention. Parallel to the enthusiasm and willingness to be vaccinated, the social perception also embraced the hesitancy due to insufficient confidence in the safety of these new and relatively quickly developed vaccines (Nguyen et al., 2021). Pregnant women are one of the groups at increased risk for a severe course of COVID-19 disease (Zambrano et al., 2020), that might have been widely looking for help in reaching their informed decision whether to vaccinate against a SARS-Cov-2 infection. Indeed, an increased number of internet search queries related to coronavirus vaccines and fertility followed the authorization of mRNA vaccines (Diaz et al., 2021).

A timely contribution was the study by Shimabukuro et al. (2021) who published their work entitled “Preliminary Findings of mRNA Covid-19 Vaccine Safety in Pregnant Persons” on April 21, 2021, in The New England Journal of Medicine (NEJM). However, there were two extreme interpretations of their results: one argued that the vaccine was safe as the number of miscarriages reported in vaccinated pregnant women is similar to the numbers published before the COVID-19 pandemic, and the other concluded that the pregnancy loss rate in women vaccinated during the first and second trimesters was highly above-normative. Notably, such discrepancies in conclusions appeared not only among non-professionals but also professionals from the medical/vaccine field, as indicated in the discussion between Stroobandt and Stroobandt (2021) and Stuckelberger et al. (2021) published in a form of commentaries in August 2021 in the Viruses journal. The first commentary argues that the correctly derived percentage of spontaneous abortions in individuals vaccinated in the first and second trimester should be 82% instead of the 12.6% presented in the original report (Stroobandt and Stroobandt, 2021). In response to this comment, Stuckelberger et al. (2021) discuss selection bias that could lead to an unfair selection of the population at risk and propose that c.a. 10% is a reasonable risk estimate. There were more discrepancies in data understanding, as can be found in other publications citing the original report: e.g. the paper by Tariq and Gupta (2021), who state “no glaring concern except spontaneous abortions in pregnant women who got the mRNA vaccine”, the paper by Chen et al. (2021) claiming the safety of vaccination in the third trimester and publication by Wu et al. (2021) stating “no evidence of unexpected serious adverse effects”. We decided to examine the originally presented values released on April 21, 2021 that raised contradictory claims.

Shimabukuro et al. (2021) analyzed data collected between December 14, 2020, and February 28, 2021, from the V-safe surveillance system, V-safe pregnancy registry, and Vaccine Adverse Event Reporting System (VAERS) that were developed to monitor the Covid-19 vaccination program in the US. Data were collected from 3958 voluntarily enrolled females who received either the BNT162b2 (Pfizer–BioNTech) vaccine or the mRNA-1273 (Moderna) vaccines during pregnancy or not earlier than 30 days from the last menstrual period. Importantly, the number of enrolled participants who received vaccination during their periconception period, 1st, 2nd, and 3rd trimester were 92 (2.3%), 1132 (28.6%), 1714 (43.3%), and 1019 (25.7%), respectively. Analysis of the pregnancy outcomes was limited to only 827 participants, for whom the “completed” status of pregnancy including live birth (712 cases), spontaneous abortion (104 cases), stillbirth after gestational week 20 (1 case), induced abortion or ectopic pregnancy (10 cases) was available at the time of analysis. In Table 4 the Authors presented a calculation showing the rate of spontaneous abortion before gestational week 20, which is 104 losses divided by 827 participants that equals the rate of 12.6% chances of pregnancy loss in vaccinated individuals. The Authors compared this value to the published incidence rate, which is 10–26% (Dugas and Slane, 2021, American College of O, 2018, Anon, 2012). However, according to the information inside the article, 700 out of 827 participants with “complete” pregnancies received their first dose of vaccination during the third trimester, which would mean that the rate of spontaneous abortion in participants vaccinated in the first and second trimester is 104 divided by 127 equaling 81.9%. Additionally, one could find it written that “A total of 96 of 104 spontaneous abortions (92.3%) occurred before 13 weeks of gestation.”

We identified several concerns with the statistical treatment of the original data. The first was a substantial percentage of missing data about pregnancy outcome: 95.6% (2811 out of 2938 for individuals vaccinated during periconception, 1st and 2nd trimester) and 31.3% (319 out of 1019) for those vaccinated during the 3rd trimester. The second problem was a relatively short follow-up time, as the data analysis was performed on March 30, 2021. This means that participants enrolled in the very last days of the data collection period could be followed for approximately 5–6 weeks after being identified as pregnant so that the possible spontaneous abortions after 30th of March but before gestational week 20 could not be included in the report. The third problem is the choice of the denominator 827, which, together with the missing data issue, is not a valid estimator of the proportions of miscarriages in the particular trimesters. If the pregnancy loss took place within the first 20 weeks, this row concerned only participants vaccinated during their first trimester and a part of the second trimester, so it is hard to find any rationale for including participants vaccinated during the 3rd trimester into the denominator. To sum up, when considering the dataset originally presented by Shimabukuro et al., neither 12.6% nor 81.9% rates are reliable. The latter can be obtained if one carelessly takes the numbers out of context, i.e. ignores the very high percentage of missing data concerning patients at their highest risk of spontaneous abortion. The readers of the article could not know the reason why information on pregnancy status for 2811 out of 2938 participants vaccinated up to the 2nd trimester was not available at the time of the analysis. There could be a plethora of reasons for this, but it should be mentioned that the Authors themselves underline that “limited follow-up calls had been made at the time of this analysis” (Shimabukuro et al., 2021). Even, if the reporting system might induce a bias towards women whose pregnancy ended with spontaneous abortion, i.e. thus the actual rate of pregnancy loss before gestational week 20 was lower in all vaccinated participants than in those who reported it at the time of analysis (as suggested in Stuckelberger et al. (2021)), this was a purely speculative assessment not supported by any data presented in the original article (Shimabukuro et al., 2021).

An epilogue of this story is that on September 8, 2021, the Authors published a corrigendum in NEJM, in which they admitted that no denominator was available to calculate a risk estimate for spontaneous abortions. They also admitted that the follow-up through 20 gestational weeks was not available at the time of analysis for 905 out of 1224 women vaccinated up to the end of the first trimester. In response to the comment by Sun (2021) the Authors wrote that the missing follow-up for 905 women was completed and additional pregnant women from the v-safe pregnancy registry, who received at least one dose of vaccine before 20 weeks of gestation, were enrolled. They point to Zauche et al. (2021) analysis of this updated data. Out of the 2491 participants vaccinated between preconception and up to the 20th week of gestation, 33 reported spontaneous abortion before gestational week 6, and 2 reported ectopic pregnancies. Out of the remaining 2456 participants who received at least one dose of mRNA Sars-Cov-2 vaccines before gestational week 20 and were still pregnant at 6 weeks onwards, 165 reported spontaneous abortion before the 20th week of gestation, 8 reported ectopic or molar pregnancies, 8 reported induced abortion and 253 could not be reached at or after their gestational week 20. This presentation is definitely more informative than the original presentation from April 2021. A detailed description of raw data and a much lower number of missing data (10.1%) support the credibility of this particular dataset. The cumulative week-specific risk of spontaneous abortion between 6 and 20 weeks of gestation seems in line with the values published for historical cohorts (Zauche et al., 2021). However, as most participants (77.3%) were over 30 years old, were non-Hispanic White (78.3%), and worked as health care personnel (88.8%), extra care needs to be taken when extrapolating to the general population (as the Authors also point out in their Fig. 1 and discuss the comparator populations in their appendix) (Zauche et al., 2021).

There is an urgent need for information on COVID-19 vaccines safety and the necessity of dissemination of data, even these of preliminary character. However, special care must be taken by editors of scientific journals to release informative datasets, especially when these are related to sensitive matters such as pregnancy loss. Shimabukuro et al. provided important and conclusive results on local and systemic adverse effects of mRNA vaccines (injection-site pain, nausea, vomiting, fatigue, headache, myalgia, fever) in pregnant women, but no data confirming the safety of mRNA COVID-19 vaccines in women at the highest risk of spontaneous abortion (i.e. inversely correlated with the gestational age) was provided in the original dataset. Notably, the original article was accompanied by an editorial, which pointed out its limitations: the relatively small number of completed pregnancies and the fact that live births were mostly reported after vaccination in the third semester, which impairs conclusions on congenital abnormalities and rare neonatal outcomes but did not mention the obvious problem concerning the calculation of the pregnancy loss rate (Riley, 2021). This problem was noticed neither by reviewers nor editors but also most of the 49 citing articles (as of September 7, 2021) accepted the safety of COVID-19 mRNA vaccines during pregnancy at face value, without going into the details. On the other hand, the 95.6% of data missing permitted the imputation of numbers based on readers’ assumptions that would subsequently result in values supporting desired recommendations concerning the vaccination of women in early pregnancy. Indeed, referencing articles coming from one of the most prestigious journals in medicine and acknowledged as a trusted source of information but based on either largely incomplete datasets or conclusions unfounded by the data may influence individuals’ informed decisions and official recommendations. Nonetheless, our attention to the article by Shimabukuro et al. was first drawn by the discussions in the media associated with movements skeptical of the vaccines against COVID-19 (DEPOPULATION ALERT, 2021). Bloggers studied the data as they literally stood in the article, derived a percentage of spontaneous abortions in women vaccinated in the 1st and 2nd trimester that did not consider the missing data issue, and speculated about the intentions behind reporting the overall rate of 12.6% for all completed pregnancies instead of the percentage calculated for early pregnancies. Indeed, estimation of the risk of spontaneous abortion is a matter of very high social impact that may also generate an emotional attitude. Therefore, a data misinterpretation or misrepresentation discovered by the readers, could, in some situations, create concerns of being manipulated about vaccine safety.

To conclude, we would like to stress that a dilemma whether to publish interesting but incomplete results or none ought to be judged through the trade-off between knowledge and confusion they will introduce. Regardless, correct estimators should be used, or it should be stated that the particular data does not allow for the estimation of certain quantities. In our opinion, an elongated follow-up and reporting complete instead of preliminary data with a correct denominator would exclude speculations and improve the credibility of this particular study. In general, such discrepancies are (nearly) always spotted at some stage and taken care of through an academic debate in journals. Indeed, there were other articles in prestigious journals (Trostle et al., 2021, Jacoby et al., 2021) that raised a discussion about biases in estimating the risk of spontaneous abortion following vaccination or Sars-Cov-2 infection at the early gestational age (Sun, 2022, Sun, 2022). In the case of the study by Shimabukuro et al., a corrigendum was eventually provided, and updated statistics do not seem to exhibit an increased risk due to mRNA vaccines. However, the original presentation of the data existed for almost five months in circulation resulting in more doubts amongst people who were already worried about mRNA vaccine safety. On the one hand, the contribution of the general public to the academic debate is the envisioned open citizen science. On the other hand, the scientific community needs to take into consideration how to efficiently correct erroneous or imprecise statements which can carry an extremely high price tag.

Declarations

Funding: no specific funding was obtained for this study. KB’s research is supported by the (Vetenskapsrådet) Grant no. 2017-04951 and an ELLIIT Call C grant.

Conflicts of interests

Authors declare no conflict of interest.