Changes in incidence rates of outcomes of interest in vaccine safety studies during the COVID-19 pandemic.

BACKGROUND: The COVID-19 pandemic caused an abrupt drop in in-person health care (inpatient, Emergency Department, outpatient) and an increase in telehealth care, which poses challenges in vaccine safety studies that identify outcomes from in-person encounters. We examined the changes in incidence rates of selected encounter-based outcomes during the COVID-19 pandemic.
METHODS: We assembled a cohort of members from 8 Vaccine Safety Datalink sites from January 1, 2017 through December 31, 2020. Using ICD-10 diagnosis codes or laboratory criteria, we identified 21 incident outcomes in traditional in-person settings and all settings. We defined 4 periods in 2020: January-February (pre-pandemic), April-June (early pandemic), July-September (middle pandemic), and October-December (late pandemic). We defined four corresponding periods in each year during 2017-2019. We calculated incidence rates, conducted difference in difference (DiD) analyses, and reported ratios of incidence rate ratios (RRR) to examine changes in rates from pre-pandemic to early, middle, and late pandemic in 2020, after adjusting for changes across similar periods in 2017-2019.
RESULTS: Among > 10 million members, regardless of setting and after adjusting for changes during 2017-2019, we found that incidence rates of acute disseminated encephalomyelitis, encephalitis/myelitis/encephalomyelitis/meningoencephalitis, and thrombotic thrombocytopenic purpura did not significantly change from the pre-pandemic to early, middle or late pandemic periods (p-values ≥ 0.05). Incidence rates decreased from the pre-pandemic to early pandemic period during 2020 for acute myocardial infarction, anaphylaxis, appendicitis, Bell's palsy, convulsions/seizures, Guillain-Barré syndrome, immune thrombocytopenia (ITP), narcolepsy/cataplexy, hemorrhagic stroke, ischemic stroke, and venous thromboembolism (p-values < 0.05). Incidence rates of Bell's palsy, ITP, and narcolepsy/cataplexy were higher in all settings than in traditional in-person settings during the three pandemic periods (p-values < 0.05).
CONCLUSION: Rates of some clinical outcomes during the pandemic changed and should not be used as historical background rates in vaccine safety studies. Inclusion of telehealth visits should be considered for vaccine studies involving Bell's palsy, ITP, and narcolepsy/cataplexy.

Introduction

COVID-19 caused an abrupt drop in the use of in-person health care, accompanied by a corresponding surge in telehealth (TH) services [1], [2], [3], [4], [5], [6]. In-person visits including inpatient (IP), Emergency Department (ED), and outpatient (OP) visits have traditionally been the main venues for patients to seek care. TH is a health care provider’s use of information communication technology in delivery of clinical and non-clinical care services [7]. While TH has been available for decades, its usage had been low in the United States until the COVID-19 pandemic was declared a national emergency in March 2020. A recent study in a large integrated health care system showed that in the early days of the pandemic, IP, ED, and OP utilization dropped by 30.2%, 37.0%, and 80.9%, respectively, while TH visits increased 4-fold [8].

The sudden drop in in-person health care since March 2020 raised concerns about potential changes in incidence rates of outcomes of interest during the pandemic that may be studied in the Vaccine Safety Datalink (VSD), a collaboration between the Centers for Disease Control and Prevention (CDC) and integrated health systems. The VSD conducts vaccine safety studies using large electronic health record (EHR) databases which capture demographics, vaccinations, and information on medically-attended conditions diagnosed during IP, ED, and OP visits [9], [10]. Since December 2020, the VSD has been conducting rapid cycle analyses using a concurrent comparator design to monitor the safety of the three COVID-19 vaccines currently used in the United States: BNT162b2 (Pfizer-BioNTech), mRNA-1273 (Moderna), and Ad26.COV2.S (Janssen) [11]. While the concurrent comparator design is less impacted by potential changes in the background incidence rates of outcomes of interest during the pandemic, a historical comparator design may be affected because it requires stable background incidence rates over time. However, no large observational studies have been conducted to evaluate the changes in incidence rates of outcomes of interest in vaccine safety studies during the COVID-19 pandemic.

The objective of this study was to assess the changes in incidence rates of selected outcomes of interest during the COVID-19 pandemic that are traditionally identified in in-person settings for use in vaccine safety studies.

Methods

Study population and study period

We retrospectively assembled a cohort consisting of active members during January 1, 2017 through December 31, 2020 from 8 geographically diverse health care organizations: HealthPartners, Kaiser Permanente (KP) Colorado, Denver Health, KP Washington, Marshfield Clinic, KP Northern California, KP Northwest, and KP Southern California.

Outcomes of interest

We evaluated changes in incidence rates of 21 outcomes of interest. These outcomes are included in the VSD COVID-19 vaccine rapid cycle analysis (RCA) project (Appendix A) [11], and most outcomes were included on the priority list of potential adverse events of special interest for COVID-19 vaccines developed by the Brighton Collaboration based on potential association with vaccination or enhanced disease following vaccination [12]. The RCA is a collaborative effort among 8 VSD sites and the CDC that conducts safety monitoring of COVID-19 vaccines and provides timely safety data to policy makers and the public. Weekly sequential testing has been conducted since December 2020 to monitor pre-specified, clinically serious outcomes after COVID-19 vaccines. In this current study, outcomes were identified from EHR data using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10) diagnosis codes in certain health care settings described below. Thrombocytopenia was identified using laboratory criteria.

Identifying incident outcomes of interest in health care settings

We considered two types of settings for identifying incident events: 1) RCA settings (i.e., IP and ED, with or without OP) and 2) all settings (i.e., IP, ED, OP, and TH). RCA settings were traditional settings that were used to identify incident outcomes in evaluations of COVID-19 vaccine safety [11]: IP and ED settings for all outcomes, along with OP for Bell’s palsy, immune thrombocytopenia (ITP), narcolepsy/cataplexy, and venous thromboembolism (VTE).

To identify incident outcomes of interest, we first included members enrolled at any point during the study period and their encounters with an ICD-10 diagnosis of an outcome of interest in either RCA settings or all settings. Second, we applied exclusion criteria in a pre-specified time window to exclude other causes (see Appendix A for details). Third, we identified the first incident outcome in 1 year for all outcomes, except for anaphylaxis (first in 7 days) and myocarditis/pericarditis (first in 60 days).

Difference-in-difference (DiD) analyses to assess changes in incidence rates of outcomes of interest in RCA settings and all settings during the pandemic

We used DiD analyses to test the significance of changes in incidence rates during the pandemic in 2020 after adjusting for changes across the same time periods in 2017–2019 [13], [14], [15], [16]. For these DiD analyses, we selected January and February in 2020 as the pre-pandemic period (comparison period). We excluded March 2020 from all DiD analyses because on March 13, 2020, a national emergency concerning the COVID-19 pandemic was declared. We defined three pandemic periods in 2020: April-June as the early pandemic period, July-September as the middle pandemic period, and October-December as the late pandemic period. We then created the four corresponding periods (i.e., January–February, April–June, July–September, and October–December) during each of the pre-pandemic years in 2017–2019. Use of the same four periods during each calendar year facilitated year-over-year comparisons. For each outcome, incidence rates per 100,000 person-years were calculated as (number of incident events/person-years) × 100,000 using both RCA settings and all settings.

In the analytic datasets, we created two categorical variables: one for the four periods (i.e., t = 0,1,2,3 for January–February, April–June, July–September, and October–December, respectively), and the other one for calendar year (i.e., y = 0,1,2,3 for 2017, 2018, 2019, and 2020, respectively). In Poisson models, the number of incident events was the dependent variable, and an interaction term between the two categorical variables was the independent variable. We also included the natural logarithm of person-years as an offset and adjusted for overdispersion of the count data. Each model estimated 16 coefficients for the combination of four periods and four years without an intercept. Each coefficient represented the logarithm (incidence rate per person-year) in a period of each year. After changes in these coefficients from January–February at t = 1, 2 and 3 were obtained, we calculated at time t by comparing the changes of coefficients during 2020 to the average changes in coefficients during 2017–2019. The changes in 2017–2019 were weighted the same (1/3) because the person-years across 2017–2019 remained largely unchanged. The exponentiation of represented the ratio of incidence rate ratios (RRR) at time t. RRR > 1, RRR = 1, and RRR less than 1 represented an increased, equal, and decreased change in incidence rate during 2020 after adjusting for changes during 2017–2019, respectively. We also reported the 95% confidence intervals (CI) and statistical significance (p-value) for RRRs.

Since some of the vaccine safety outcomes of interest were also potential complications of COVID-19, we also conducted sensitivity analyses excluding patients who tested positive for SARS-CoV-2 or had a COVID-19 diagnosis.

Results

Monthly enrollment was stable between 2017 and 2020, with the number of enrolled members ranging from 10.8 to 11.4 million across VSD sites. Demographic characteristics of members enrolled in VSD sites did not change significantly during 2017–2020 (Table 1 ).

Table 1

Yearly enrollment in person-years by demographic characteristics during 2017–2020.

	2017	2018	2019	2020
Age (in years)
0–5	685,840 (6.32%)	702,207 (6.27%)	701,830 (6.21%)	697,373 (6.11%)
6–17	1,568,847 (14.47%)	1,594,130 (14.23%)	1,594,853 (14.11%)	1,595,202 (13.97%)
18–44	3,998,666 (36.87%)	4,164,678 (37.19%)	4,218,105 (37.32%)	4,275,114 (37.45%)
45–64	2,971,673 (27.4%)	3,053,391 (27.26%)	3,047,835 (26.97%)	3,057,336 (26.78%)
65+	1,619,987 (14.94%)	1,685,478 (15.05%)	1,739,402 (15.39%)	1,790,147 (15.68%)
Sex
Female	5,627,735 (51.89%)	5,812,122 (51.89%)	5,865,447 (51.9%)	5,929,690 (51.95%)
Male	5,216,799 (48.1%)	5,387,167 (48.10%)	5,435,576 (48.09%)	5,483,816 (48.04%)
Unknown	479 (0.00%)	594 (0.01%)	1,002 (0.01%)	1,665 (0.01%)
Race/Ethnicity
Hispanic	2,743,812 (25.3%)	2,837,220 (25.33%)	2,886,299 (25.54%)	2,916,599 (25.55%)
Non-Hispanic Asian	1,288,164 (11.88%)	1,356,904 (12.12%)	1,394,415 (12.34%)	1,412,046 (12.37%)
Non-Hispanic Black	739,557 (6.82%)	756,083 (6.75%)	757,435 (6.70%)	758,942 (6.65%)
Non-Hispanic Multi/Other	4,70,625 (4.34%)	489,439 (4.37%)	496,625 (4.39%)	500,072 (4.38%)
Non-Hispanic White	4,717,310 (43.50%)	4,785,443 (42.73%)	4,717,429 (41.74%)	4,622,288 (40.49%)
Unknown	885,545 (8.17%)	974,793 (8.70%)	1,049,823 (9.29%)	1,205,225 (10.56%)

For each outcome of interest, we presented the results from analyses for RCA settings and for all settings, both among all enrolled members and after excluding COVID-19 patients. The results are summarized in Supplementary Tables S1-S4.

Acute disseminated encephalomyelitis (ADEM). ADEM incidence rates were less than 0.5 per 100,000 person-years during 2017–2019 in IP and ED settings. Although the incidence rates of ADEM changed from the pre-pandemic to pandemic periods during 2020, these changes were not statistically significantly different from analogous changes during 2017–2019 both for RCA (IP and ED) settings (Table S1) and for all settings (Table S2), as well as after excluding COVID-19 patients (Tables S3-4).

Acute myocardial infarction (AMI). AMI incidence rates ranged from 64.54 to 80.65 per 100,000 person-years during 2017–2019 in IP and ED settings (Fig. 1 ). After controlling for the changes in 2017–2019, the AMI incidence rate significantly decreased during the early pandemic period relative to the pre-pandemic period and then started to increase, reaching a RRR of 1.10 (95% CI, 1.02 to 1.18) during the late pandemic period of 2020 (Fig. 1; Table S1). Similar results were observed in all settings, although additional AMI cases identified in OP and TH settings resulted in higher overall incidence of AMI (Table S2). After excluding COVID-19 patients, the AMI incidence rate significantly decreased during the early and the late pandemic periods relative to the pre-pandemic period in RCA settings (Table S3), and significantly decreased during the three pandemic periods in all settings (Table S4).

Fig. 1

Incidence rates of acute myocardial infarction per 100,000 person-years over time.

Acute respiratory distress syndrome (ARDS). After adjusting for the changes in 2017–2019, the incidence rates of ARDS identified in RCA settings (IP and ED) and in all settings increased significantly during 2020 relative to the pre-pandemic period, with RRRs for early, middle, and late pandemic periods ranging from 3.68 to 10.78 (Table S1) and 4.13 to 8.68 (Table S2), respectively. Excluding COVID-19 patients significantly lowered the incidence rates of ARDS during the three pandemic periods of 2020 and reduced RRRs significantly. After adjusting for the changes in 2017–2019, regardless of setting, the 2020 incidence rates during the early and middle pandemic periods were similar to those in the pre-pandemic period (Table S3). For all settings among members without COVID-19, the incidence rate of ARDS in the late pandemic period was significantly higher than that of the pre-pandemic period with an RRR of 1.47 (95% CI, 1.08 to 1.99) (Table S4).

Anaphylaxis. Anaphylaxis incidence rates ranged between 15.74 and 26.96 per 100,000 person-years during 2017–2019 in IP and ED settings (Fig. 2 ). After adjusting for the changes in 2017–2019, the incidence rates of anaphylaxis decreased significantly during the three 2020 pandemic periods relative to the pre-pandemic period, with RRRs ranging from 0.62 to 0.81 (Fig. 2; Table S1). The incidence rates of anaphylaxis identified in all settings were double those identified in RCA settings (Table S2). After excluding COVID-19 patients, the incidence rates during the three pandemic periods were lower than those observed during the pre-pandemic period, both for RCA settings and all settings and after adjusting for the changes in 2017–2019 (Tables S3 and S4).

Fig. 2

Incidence rates of anaphylaxis per 100,000 person-years over time.

Appendicitis. Appendicitis incidence rates ranged between 113.70 and 133.46 per 100,000 person-years during 2017–2019 in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rates of appendicitis decreased significantly during the early pandemic period relative to the pre-pandemic period of 2020 (RRR of 0.88; 95% CI, 0.82 to 0.93) and then increased back to the pre-pandemic level during the middle and late 2020 pandemic periods (Table S1). A similar finding was observed for appendicitis cases identified in all settings, although incidence rates were higher (Table S2). However, when COVID-19 patients were excluded, the incidence rates of appendicitis were lower in all three pandemic periods compared with pre-pandemic rates in 2020 after adjusting for the changes in 2017–2019 (Tables S3 and S4).

Bell's palsy. Bell's palsy incidence rates ranged between 88.05 and 105.35 per 100,000 person-years during 2017–2019 in IP, ED, and OP settings (Fig. 3 ). After adjusting for the changes in 2017–2019, the incidence rates of Bell's palsy were significantly lower during the three pandemic periods relative to pre-pandemic rates in 2020, with RRRs ranging from 0.76 to 0.88 (Fig. 3; Table S1). During 2017–2019, the incidence rates of Bell’s palsy identified in all settings were similar to those identified in IP, ED, and OP settings, suggesting that few cases were identified in the TH setting before the pandemic. However, during the three pandemic periods of 2020 the incidence rates of Bell’s palsy identified in all settings were higher than those identified in IP, ED, and OP settings (p-values < 0.0001). Consequently, incidence rates during the three pandemic periods were not statistically different from the pre-pandemic period of 2020 after adjusting for the changes during 2017–2019 (Table S2). Exclusion of COVID-19 patients did not substantially change the findings in RCA settings and in all settings (Tables S3 and S4).

Fig. 3

Incidence rates of Bell’s palsy per 100,000 person-years over time.

Convulsion/seizure. Convulsion/seizure incidence rates ranged between 135.86 and 158.73 per 100,000 person-years during 2017–2019 in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rates of convulsion/seizure were significantly lower during the three pandemic periods relative to the pre-pandemic period of 2020, with RRRs ranging from 0.79 to 0.88 (Table S1). The incidence rate of convulsion/seizure nearly doubled when OP and TH settings were also included for identifying cases. However, the incidence rates of convulsion/seizure remained significantly lower during the three pandemic periods relative to the pre-pandemic period of 2020 (Table S2). Exclusion of COVID-19 patients did not substantially change the findings (Tables S3 and S4).

Disseminated intravascular coagulation (DIC). DIC incidence rates were less than 10 per 100,000 person-years when cases were identified in IP and ED settings during 2017–2019. After adjusting for the changes in 2017–2019, the incidence rates of DIC during the three pandemic periods were not significantly different from the pre-pandemic period of 2020, with RRRs ranging from 0.87 to 1.10 (Table S1). Including DIC cases from OP and TH settings did not change the incidence rates of DIC, indicating that very few cases were identified in OP and TH settings even during the pandemic. The changes in incidence rates of DIC remained the same (Table S2). However, when COVID-19 patients were excluded, the incidence rates of DIC during the early pandemic period were significantly lower than the pre-pandemic period of 2020 in RCA settings and all settings, with RRRs = 0.70 and 0.69, respectively (Tables S3 and S4).

Encephalitis/myelitis/encephalomyelitis/meningoencephalitis (EMEM). EMEM incidence rates were less than 4 per 100,000 person-years during 2017–2019 in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rates of EMEM during the three pandemic periods were not significantly different from the pre-pandemic period of 2020, with RRRs ranging from 0.93 to 1.11 (Table S1). While the incidence rates of EMEM almost doubled in all settings, exclusion of COVID-19 patients did not change the incidence rates. Furthermore, neither inclusion of OP and TH settings for identifying cases nor exclusion of COVID-19 patients changed the finding of no difference in EMEM incidence rates during the three pandemic periods compared with the pre-pandemic period of 2020 (Tables S2-S4).

Guillain-Barré syndrome (GBS). GBS in the IP and ED setting occurred with incidence rates less than 7 per 100,000 person-years during 2017–2019 (Fig. 4 ). After adjusting for the changes in 2017–2019, the incidence rates of GBS decreased significantly during the early pandemic period relative to the pre-pandemic period of 2020, with an RRR of 0.64 (95% CI, 0.46 to 0.88) and then increased during the middle and late pandemic periods to levels that were not significantly different from the pre-pandemic period (Fig. 4; Table S1). The incidence rates of GBS identified in all settings were higher than those identified in IP and ED settings and the changes in incidence rates in all settings was similar to that observed in RCA settings (Table S2). After excluding COVID-19 patients, in RCA settings and in all settings, the incidence rates of GBS during the three pandemic periods were significantly or marginally lower than the pre-pandemic period of 2020 after adjusting for the changes in 2017–2019 (Tables S3 and S4).

Fig. 4

Incidence rates of Guillain-Barré syndrome per 100,000 person-years over time.

Immune thrombocytopenia. ITP identified in IP, ED, and OP settings occurred with incidence rates ranging between 17.91 and 25.55 per 100,000 person-years during 2017–2019 (Fig. 5 ). After adjusting for the changes in 2017–2019, the incidence rates of ITP were significantly lower during the three pandemic periods than the rates during the pre-pandemic period of 2020, with RRRs ranging from 0.41 to 0.74 (Fig. 5; Table S1). While very few ITP cases were identified in the TH setting before the pandemic, some additional ITP cases were identified in the TH setting after the start of the pandemic. After adjusting for the changes in 2017–2019, the incidence rates of ITP identified in all settings was significantly lower only during the early pandemic period relative to the pre-pandemic period of 2020. The RRRs during the middle and late pandemic periods were 0.89 (95% CI, 0.78 to 1.02) and 0.93 (95% CI, 0.81 to 1.07), respectively, suggesting that the setting of ITP diagnosis shifted to some extent from in-person to virtual encounters (Table S2). The changes in incidence rates of ITP identified in RCA settings and in all settings did not differ when COVID-19 patients were excluded (Tables S3 and S4).

Fig. 5

Incidence rates of immune thrombocytopenia per 100,000 person-years over time.

Kawasaki disease. Kawasaki disease incidence rates were less than 4 per 100,000 person-years during 2017–2019 in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rates of Kawasaki disease during the early and middle pandemic periods were lower but not statistically different from the pre-pandemic period of 2020, with RRRs equal to 0.72 and 0.78, respectively. However, the incidence rate during the late pandemic period was significantly lower than the pre-pandemic period of 2020, with an RRR equal to 0.58 (95% CI, 0.35 to 0.94) (Table S1). Similar results were observed in all settings and after excluding COVID-19 patients (Tables S2-S4). However, more Kawasaki disease cases were identified in all settings than in IP and ED settings.

Myocarditis and pericarditis. The incidence rates of combined myocarditis and pericarditis in IP and ED settings were less than 11 per 100,000 person-years during 2017–2019. After adjusting for the changes in 2017–2019, the incidence rates of myocarditis and pericarditis during the three pandemic periods were not significantly different from the pre-pandemic period of 2020, with RRRs ranging from 0.95 to 1.12 (Table S1). The incidence rates of myocarditis and pericarditis identified in all settings were higher than those identified in IP and ED settings and the changes in incidence rates during 2020 was similar to that observed in RCA settings (Table S2). After excluding COVID-19 patients, the incidence rate of myocarditis and pericarditis during the late pandemic period was significantly lower than the pre-pandemic period of 2020 in RCA settings or all settings (Tables S3 and S4).

Narcolepsy and cataplexy. The incidence rates of combined narcolepsy and cataplexy ranged between 11.48 and 16.53 per 100,000 person-years during 2017–2019 in IP, ED, and OP settings (Fig. 6 ). After adjusting for the changes in 2017–2019, the incidence rates of narcolepsy and cataplexy were significantly lower during the three pandemic periods than the pre-pandemic period of 2020, with RRRs ranging from 0.29 to 0.62 (Fig. 6; Table S1). During 2017–2019, the incidence rates of narcolepsy and cataplexy identified in all settings were similar to those identified in IP, ED, and OP settings, suggesting that very few cases were identified in the TH setting before the pandemic. However, during the three pandemic periods, the incidence rates of narcolepsy and cataplexy identified in all settings were higher than those identified in IP, ED, and OP settings (p-values < 0.0001). Consequently, RRRs increased and the incidence rate during the late pandemic period was no longer statistically different from the pre-pandemic period of 2020 after adjusting for the changes during 2017–2019 (Table S2). Exclusion of COVID-19 patients did not substantially change the findings in RCA settings and in all settings (Tables S3 and S4).

Fig. 6

Incidence rates of narcolepsy and cataplexy per 100,000 person-years over time.

Pulmonary embolism. Pulmonary embolism incidence rates were 75.35–86.89 per 100,000 person-years during 2017–2019 in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rates of pulmonary embolism during the early and middle pandemic periods were similar to the level during the pre-pandemic period of 2020, with RRRs equal to 0.97 and 1.06, respectively. However, the incidence rate during the late pandemic period was significantly higher than the pre-pandemic period of 2020, with an RRR equal to 1.29 (95% CI, 1.20 to 1.39) (Table S1). The incidence rates of pulmonary embolism in all settings were higher than those in IP and ED settings during the study period. After adjusting for the changes in 2017–2019, the incidence rate of pulmonary embolism during the early pandemic period was lower than the pre-pandemic period of 2020, while the incidence rates during the middle and late pandemic periods were significantly higher with RRRs equal to 1.11 and 1.32, respectively (Table S2). While exclusion of COVID-19 patients did not change the incidence rate of pulmonary embolism before the pandemic, it resulted in lower incidence rates during the three pandemic periods. The incidence rates in the early pandemic period were significantly lower than the pre-pandemic period of 2020 (Tables S3 and S4) in both RCA settings and all settings, while the RRRs during the middle and later pandemic periods were approximately 1.

Hemorrhagic Stroke. During 2017–2019, hemorrhagic stroke incidence rates were 48.52–57.94 per 100,000 person-years in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rate of hemorrhagic stroke during the early pandemic period was lower than the pre-pandemic period of 2020 with an RRR = 0.86 (95% CI, 0.79 to 0.95). However, incidence rates returned to the pre-pandemic level during the middle and late pandemic periods with RRRs equal to 1.04 and 1.03, respectively (Table S1). Similar changes in the incidence rates of hemorrhagic stroke were observed in all settings either among all members or after excluding COVID-19 patients (Tables S2 and S3). However, the incidence rates of hemorrhagic stroke in all settings after excluding COVID-19 patients were lower during the three pandemic periods with RRRs equal to 0.80 (95% CI, 0.73 to 0.87), 0.91 (95% CI, 0.83 to 0.98), and 0.91 (95% CI, 0.84 to 0.99), respectively (Table S4).

Ischemic Stroke. During 2017–2019, ischemic stroke incidence rates were 190.84–206.96 per 100,000 person-years in IP and ED settings. After adjusting for the changes in 2017–2019, the incidence rate of ischemic stroke during the early pandemic period was lower than the level during the pre-pandemic period of 2020 with an RRR = 0.81 (95% CI, 0.77 to 0.85). However, it returned to the pre-pandemic level during middle and late pandemic periods with RRRs equal to 0.96 and 0.97, respectively (Table S1). Including OP and TH settings for identifying ischemic stroke increased the incidence rates; but similar changes in incidence rates were observed after adjusting for changes during 2017–2019 (Table S2). Among members without COVID-19, the incidence rates of ischemic stroke in IP and ED settings were lower during the three pandemic periods with RRRs equal to 0.78 (95% CI, 0.74 to 0.82), 0.91 (95% CI, 0.87 to 0.96), and 0.89 (95% CI, 0.84 to 0.93), respectively (Table S3); the incidence rates of ischemic stroke in all settings were significantly lower during the early and middle pandemic periods but not significantly different in the late pandemic period (Table S4).

Thrombocytopenia. Thrombocytopenia was identified based on platelet count irrespective of setting; thus, the incidence rates and RRRs in Tables S1 and S2 were identical, and the incidence rates and RRRs in Tables S3 and S4 were identical. Thrombocytopenia incidence rates were 14.32–18.73 per 100,000 person-years during 2017–2019. After adjusting for the changes in 2017–2019, the incidence rate of thrombocytopenia during the early pandemic period was marginally lower than the pre-pandemic period of 2020 with an RRR = 0.85 (95% CI, 0.71 to 1.01). However, incidence rates returned to the pre-pandemic level during the middle and late pandemic periods with RRRs equal to 0.98 and 1.03, respectively (Table S1). Exclusion of COVID-19 patients did not substantially change the findings (Table S3).

Thrombotic thrombocytopenic purpura (TTP). During 2017–2019, TTP incidence rates were less than 2 per 100,000 person-years in IP and ED settings (Fig. 7 ). After adjusting for the changes in 2017–2019, the incidence rate of TTP during the early pandemic period was lower than the pre-pandemic period of 2020 with an RRR = 0.58 (95% CI, 0.32 to 1.04) with marginal statistical significance (Fig. 7; Table S1). However, incidence rates did not statistically differ from the pre-pandemic level during the middle and late pandemic periods with RRRs equal to 0.82 (95% CI, 0.47 to 1.43) and 0.75 (95% CI, 0.43 to 1.32), respectively (Table S1). Although the incidence rates of TTP increased in all settings, similar changes in the incidence rates of TTP was observed after adjusting for changes during 2017–2019 (Table S2). After excluding COVID-19 patients, the incidence rate during the late pandemic period was lower but not statistically significant, with an RRR equal to 0.58 (95% CI, 0.31 to 1.09) for RCA settings (Table S3), and 0.65 (95% CI, 0.40 to 1.07) for all settings (Table S4).

Fig. 7

Incidence rates of thrombotic thrombocytopenic purpura per 100,000 person-years over time.

Transverse myelitis (TM). TM incidence rates were less than 3 per 100,000 person-years during 2017–2019 in IP and ED settings. Although the incidence rates of TM during the three pandemic periods decreased relative to the pre-pandemic period during 2020 after adjusting for the changes during 2017–2019, the DiD analysis showed that the changes were not statistically significant with RRRs equal to 0.78 (95% CI, 0.44 to 1.38), 0.73 (95% CI, 0.42 to 1.27), and 0.75 (95% CI, 0.43 to 1.30), respectively (Table S1). The incidence rates of TM increased in all settings (Table S2). Incidence rates in the three pandemic periods became significantly lower than the pre-pandemic period of 2020 with RRRs equal to 0.60 (95% CI, 0.43 to 0.83), 0.66 (95% CI, 0.48 to 0.92), and 0.69 (95% CI, 0.50 to 0.96), respectively. The findings for TM were similar after excluding COVID-19 patients (Tables S3 and S4).

Venous thromboembolism. During 2017–2019, VTE incidence rates ranged from 124.90 to 148.17 per 100,000 person-years in IP, ED, and OP settings. After adjusting for the changes in 2017–2019, the VTE incidence rate significantly decreased during the early pandemic period relative to the pre-pandemic period with an RRR = 0.70 (95% CI, 0.66 to 0.75), then it increased during the middle pandemic period with an RRR = 0.92 (95% CI, 0.86 to 0.97). The RRR reached 0.97 (95% CI, 0.92 to 1.03) during the late pandemic period (Table S1). Similar changes in incidence rate of VTE were observed in all settings. While VTE incidence rates did not increase in all settings before the pandemic, they increased during the three pandemic periods, with RRRs ranging from 0.85 to 1.06 (Table S2). Among members without COVID-19, the incidence rates during all pandemic periods were significantly lower than the pre-pandemic period of 2020 with RRRs ranging from 0.69 to 0.92 in RCA settings (Table S3). In all settings, the incidence rate during the early pandemic period was significantly lower, while the middle and late pandemic periods were not different from the pre-pandemic period of 2020 with RRR equal to 0.97 and 1.00, respectively (Table S4).

Discussion

We examined changes in incidence rates of 21 outcomes of interest in vaccine safety studies using a retrospective cohort of > 10 million members of 8 VSD sites during 2017–2020. We found that incidence rates of outcomes defined using diagnostic codes assigned to in-person visits including ADEM, EMEM, and TTP did not change during the pandemic (p-values for RRRs ≥ 0.05). In contrast, decreases in incidence rates were observed at least during the early versus the pre-pandemic period for AMI, anaphylaxis, appendicitis, Bell’s palsy, convulsion/seizure, GBS, ITP, narcolepsy/cataplexy, hemorrhagic stroke, ischemic stroke, and VTE. Incidence rates of these outcomes in traditional in-person settings during the pandemic period may not be appropriate to serve as background rates in vaccine safety studies where a historical comparison is used.

In prior vaccine safety studies, Bell’s palsy, ITP, and narcolepsy/cataplexy have been defined using IP, ED, and OP settings [17], [18], [19]. In this study, we found that few events were identified in the TH setting before the pandemic. However, we observed a significant increase in incidence rates of Bell’s palsy, ITP and narcolepsy/cataplexy during pandemic periods in all settings compared to RCA settings among all members (p-values < 0.0001). Current and future vaccine safety studies ought to consider TH visits for identifying these three outcomes to maximize potential capture of such events.

Several vaccine safety outcomes of interest (ARDS, appendicitis, myocarditis/pericarditis, PE, ischemic stroke, and VTE) have been associated with COVID-19 infection [20], [21], [22], [23], [24], [25]. This could lead to higher incidence rates of these outcomes during the pandemic period when COVID-19 patients were included. Correspondingly, when COVID-19 patients were excluded from analyses, incidence rates of these outcomes during the pandemic were lower and RRRs decreased compared with analyses that included COVID-19 patients. For future COVID-19 vaccine safety studies, we recommend that sensitivity analyses excluding COVID-19 patients be considered for outcomes that are associated with COVID-19 infection.

There are several limitations in this retrospective cohort study. First, chart review was not conducted to confirm presumptively-identified code-based incident events. This may be especially important for TH visits where there is little prior data available to understand the potential accuracy of diagnostic codes. Future chart reviews should consider if laboratory tests (e.g., blood tests or sleep studies) were performed to confirm cases of ITP and narcolepsy/cataplexy. Thus, the quality of diagnoses in TH visits is unknown. Second, the study period ended on December 31, 2020. The changes in incidence rates of these outcomes after the study period are unknown. Third, rates reported in this study may differ from other studies due to difference in defining incident events and settings.

This study also had some important strengths. First, this cohort study consisted of more than 10 million sociodemographically diverse members of 8 VSD sites. Second, DiD analyses were conducted to adjust for changes during 2017–2019; thus seasonality and temporal trends were taken into account. Third, sensitivity analyses excluding COVID-19 patients minimized the impact of COVID-19 infection in assessing changes in incidence rates of outcomes that might be associated with COVID-19 infection.

In summary, changes in incidence rates during the pandemic differed by outcome, settings used to identify incident cases, and inclusion of COVID-19 patients. The TH setting should be considered for identification of Bell’s palsy, ITP and narcolepsy/cataplexy in current and future vaccine safety studies.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: JGD reports research support from Janssen for an unrelated study. LSS reports research support from Moderna, GlaxoSmithKline, Dynavax, and Seqirus for unrelated studies.