The influence of selection bias on identifying an association between allergy medication use and SARS-CoV-2 infection.

BACKGROUND: Medications to prevent and treat SARS-CoV-2 infection are needed to complement emerging vaccinations. Recent in vitro and electronic health record (EHR) studies suggested that certain allergy medications could prevent SARS-CoV-2 infection. We sought to carefully examine the potential selection bias associated with utilizing EHRs in these settings.
METHODS: We analyzed associations of three allergy medications (cetirizine, diphenhydramine or hydroxyzine) with testing negative for SARS-CoV-2, measuring the potential effect of selection bias on these associations. We used a retrospective cohort of EHR data from 230,376 patients (18 years+) who visited outpatient clinicians in a single, large academic center at least once but were never hospitalized (10/1/2019-6/1/2020). Main exposures included EHR documentation of three allergy medications and allergy, with an intermediate outcome of receipt of a SARS-CoV-2 test, and the primary outcome as testing negative.
FINDINGS: SARS-CoV-2 testing rates varied by sex, age, race/ethnicity and insurance. Increasing age and public insurance were associated with a higher adjusted odds of test negativity, while being Black or Hispanic was significantly associated with test positivity. Allergy diagnosis and use of any of three allergy medications were each associated with a higher likelihood of receiving a test (e.g. diphenhydramine - Odds Ratio (OR) 2.99, 95% Confidence Interval (CI) 2.73, 3.28; cetirizine 1.75 (95% CI 1.60, 1.92)). Among those tested, only use of diphenhydramine was associated with a negative SARS-CoV-2 test (adjusted OR = 2.23, 95% CI 1.10, 4.55). However, analyses revealed that selection bias may be responsible for the apparent protective effect of diphenhydramine.
INTERPRETATION: Diphenhydramine use was associated with more SARS-CoV-2 testing and subsequent higher odds for negative tests. While EHR-based observational studies can inform a need for interventional trials, this study revealed limitations of EHR data. The finding that diphenhydramine documentation conferred a higher odds of testing negative for SARS-CoV-2 must be interpreted with caution due to probable selection bias.Abbreviations: SARS-CoV-2, ACE2, COVID-19, EHR.

Evidence before this study

Increasing numbers of published studies using observational clinical data are finding associations between existing medications and possible protection from SARS-CoV-2 infection. However, most of the observational studies do not appear to take selection bias into account.

Added value of this study

Initial analysis of electronic health records from a large cohort suggested diphenhydramine might be protective against SARS-CoV-2 infection; however, further investigation demonstrated selection bias due to more SARS-CoV-2 testing among patients with allergies, likely explaining the observed protective association.

Implications of all the available evidence

Patients listing diphenhydramine as a medication were more likely to be tested for SARS-CoV-2, leading to a lower rate of SARS-CoV-2 infection and an apparent but unlikely protective effect among those tested. These analyses document the need for caution in the interpretation of observational studies utilizing clinical data.

Alt-text: Unlabelled box

Introduction

Even with the recent emergency use authorization of COVID-19 vaccines, [1,2] effective medications for prophylaxis against SARS-CoV-2 infection are needed, given the months-long process of vaccine production, distribution and vaccine acceptance required to achieve herd immunity, as well as the possibility of infection despite vaccination. A growing number of in silico and in vitro studies have provided insights for identifying candidate medications for further study. Of these, identifying therapeutic candidates that would represent a “repurposing” of an existing drug, whose safety profile is well-established, would be preferred. [3] Although the mechanism is not well understood, some antihistamines have been shown to exhibit direct antiviral activity against SARS-CoV-2 isolates in vitro; thus understanding whether these medications confer in vivo protection has been of particular interest. [3], [4], [5]

While randomized controlled trials are the gold standard to determine medication efficacy, they are costly, time-prohibitive and logistically challenging. Retrospective evaluation of population-level health data such as offered by electronic health records (EHRs) can offer insights into medications that could possibly decrease risk of developing disease, although caution is required when interpreting results from these analyses, as recently articulated by Griffiths et al. [6] In particular, a specific kind of selection bias, called collider bias, may be responsible for observed associations when the exposures of interest may be associated with an increased likelihood of being tested. [6] With the possibility of a potential protective effect of cetirizine, diphenhydramine or hydroxyzine on risk of SARS-CoV-2 infection, including the utilization of EHR data to support this hypothesis, [5] we sought to carefully examine the potential for collider bias and its implications in testing this hypothesis using EHR data. This methodological study highlights potential sources of bias that are important to consider when interpreting observed associations using EHR data.

Methods

To understand the association between medications frequently used to control allergy symptoms (diphenhydramine, cetirizine and hydroxyzine) and testing negative for SARS-CoV-2, we performed a retrospective analysis with the target population of all patients in a single, large academic medical center, UF Health, in Gainesville, Florida. From the EHR, we obtained race, ethnicity, sex, age and insurance type at date of encounter, allergy diagnoses as documented on the EHR problem list, (associated ICD10 codes: J30, Z91.0x, L51, L50, T78.4x), SARS-CoV-2 testing and results, and documentation of cetirizine, hydroxyzine, or diphenhydramine use in patient medication lists. Insurance type was grouped as public for patients with any documented use of public insurance over the study period. The analytic sample was restricted to all health system encounters of adults 18 years of age or older who had exclusively outpatient encounters documented in their EHR from October 01, 2019-June 30, 2020. We used an extended study period including a time period before the onset of the SARS-Co-V2 pandemic in order to include non-COVID-19 disease-related outpatient visits where usual care, medications and problems and diagnoses may be documented. We limited to exclusively outpatient visits since testing for inpatient admissions or procedures were frequently on different dates, and could not be consistently associated. Additionally, with many tests ordered to allow asymptomatic patients to receive whatever hospital-based care was needed, we sought to eliminate that source of testing bias.

Our methodological examination focused on a single, primary outcome (odds of being SARS-CoV-2 negative) with an intermediate outcome (odds of receiving a SARS-CoV-2 test). To examine odds of being SARS-CoV-2 negative among those with a documented test in the EHR (in examining the association between SARS-CoV-2 infection and documented allergies or medication use), it is also important to examine odds of ever receiving a SARS-CoV-2 test in general. SARS-CoV-2 testing was operationalized to the participant level as having any SARS-CoV-2 test (all types), and negativity was operationalized as never receiving a positive result for all tests administered. To compare the odds of these two outcomes among documented users of cetirizine, diphenhydramine, or hydroxyzine with non-users of these medications in our database, we used logistic regression to report unadjusted odds ratios (ORs, with 95% confidence intervals (CIs)) and adjusted odds ratios (aORs), controlling for simultaneous use of these medications and other confounders such as age, race/ethnicity, gender, insurance payer type (public versus private) and documented allergy symptoms. Statistically significant (α = 0.05) associations were indicated when 95% CI's for odds ratios did not include 1.0. Finally, we draw from a causal diagram presented in Smith and VanderWeele, [7] where the exposure of interest (allergy medications) is likely associated with selection (those who were tested for SARS-CoV-2) into this type of EHR-based study (Fig. 1). We thus implemented methods proposed to estimate the magnitude of the selection/collider bias [7] and make appropriate final inferences. We received human subjects approval from the University of Florida Institutional Review Board and follow the Strengthening the Reporting of Observations Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies.

Fig. 1

Hypothesized causal diagram of possible impact of collider bias on the examination of allergy medication and COVID-19 disease risk.

Selection into the study is shown in the box, namely having a SARS-CoV-2 test. We are interested in estimating the association between allergy medications and COVID-19 disease risk, testing the hypothesis that allergy medications reduces the risk of COVID-19 disease (red). However, allergy medications are used to treat symptoms that may overlap with symptoms of COVID-19 disease, and thus could increase likelihood of obtaining a SARS-CoV-2 test. Adapted from Smith, et al. (2019).7

*ORs in bold indicate statistical significance (p<0.05).

Role of funding sources

This study received University of Florida Children's Miracle Network funding to support the costs of extracting and preparing a limited data set from the University of Florida Integrated Data repository. The Children's Miracle Network had no role in the study design, data collection, analysis, or interpretation, writing and publication decisions.

Results

EHR-based observational data

Between October 1, 2019 and June 30, 2020, 230,376 patients had eligible outpatient encounters and available data for this analysis (Table 1). More women (58.9%) accessed the health care system. The majority of patients were white (63.2%), with fewer Black (16.5%), Hispanic (6.6%), or multiple or other races (13.7%). Nearly one-half (45.0%) used public insurance to pay for at least one visit during the study period. Among those tested for SARS-CoV-2, most (88.3%) received only one test, few received two (9.6%), with only 2.1% of the population receiving three or more tests. One person received the maximum of twelve SARS-CoV-2 tests. With respect to likelihood of testing, women were statistically more likely to have a SARS-CoV-2 test recorded in their EHR (5.6% versus 5.3%, p < 0.01), and more Blacks and Hispanics were tested for SARS-CoV-2 than other racial/ethnic groups (both 5.9%, versus white (5.3%) or other/multiple races (5.7%), p < 0.01), but these small differences are not clinically meaningful. Those with EHR-documented allergies (8.0%) were significantly more likely to be tested for SARS-CoV-2 than those without (10.8% versus 5.1%, p < 0.0001). Subjects who had cetirizine, diphenhydramine, or hydroxyzine noted on their EHR ‘medication list’ were more likely to be tested for SARS-CoV-2; odds of testing increased with the number of these medications listed (p < 0.0001). Of those who received a SARS-CoV-2 test, test negativity varied by age, race/ethnicity and insurance type. The proportion of SARS-CoV-2 negative tests did not vary by sex, allergy diagnosis or medication use (all p > 0.05).

Table 1

Adults 18+ Years: Demographics & SARS-CoV-2 Testing (N = 230,376).

Demographics	Overall n (%)	Patients with a SARS-CoV-2 test n (%)*	Among those with 1+ SARS-CoV-2 test
			Patients with a negative test, n (%)	p-value*
Sex
Female	135,784 (58.9)	7650 (5.6)	7340 (96.5)	0.5742
Male	94,494 (41.0)	5029 (5.3)	4815 (95.7)
Unknown	98 (0.0)	36 (36.7)	34 (94.4)
Age Category
65+	61,645 (26.8)	3126 (5.1)	3080 (98.5)	<0.0001
45–64	72,269 (31.4)	4059 (5.6)	3945 (97.2)
18–44	96,462 (41.9)	5530 (5.7)	5164 (93.4)
Race/Ethnicity
Non-Hispanic Black	38,010 (16.5)	2257 (5.9)	2138 (94.7)	<0.0001
Hispanic	15,102 (6.6)	893 (5.9)	815 (91.3)
Other/Multiple	31,565 (13.7)	1796 (5.7)	1719 (95.7)
Non-Hispanic White	145,699 (63.2)	7769 (5.3)	7517 (96.8)
Insurance Type
Private	126,762 (55.0)	6714 (5.3)	6341 (94.4)	<0.0001
Public	103,614 (45.0)	6001 (5.8)	5848 (97.5)
Allergy Diagnosis
No	211,857 (92.0)	10,718 (5.1)	10,263 (95.8)	0.1552
Yes	18,519 (8.0)	1997 (10.8)	1926 (96.4)
Allergy Medications
Medications: 0	217,805 (94.5)	11,403 (5.2)	10,918 (95.7)	0.0937
Medications: 1	11,865 (5.2)	1201 (10.1)	1165 (97.0)
Medications: 2+	706 (0.3)	111 (15.7)	106 (95.5)

Rates of SARS-CoV-2 testing and positive tests were compared via chi-square tests for all variables except age category and number of allergy medications (Cochran-Mantel-Haenszel test for trend). Within sex, testing conducted among males and females only; unknown category had insufficient sample size. P-values are only reported for comparisons of positive test rates; all comparisons of testing rates were statistically significant (p < 0.05).

We then examined individual allergy medication use and univariate associations with both testing and being negative for SARS-CoV-2 (among those who were tested; Table 2). Having a documented allergy diagnosis, and each allergy medication, were associated with higher rates of SARS-CoV-2 testing, most notably diphenhydramine (OR = 2.99, 95% CI: (2.73, 3.28)). Among those with a SARS-CoV-2 test, however, while cetirizine and hydroxyzine usage were not associated with a negative test, diphenhydramine revealed an apparent protective effect with high rates of negativity (unadjusted OR = 2.95 95% CI: (1.48, 5.91)). This calculation reflects a negativity rate of 98.6% for those who took diphenhydramine compared to 95.7% for those who did not have this documentation in their EHR.

Table 2

Allergy Medication Usage Among Adults 18+ (N = 230,376) Tested and Receiving a Negative SARS-CoV-2 Test.

	Association with Odds of a SARS-CoV-2 Test		Association with Odds of a Negative SARS-CoV-2 Test Among those with a COVID Test
Documented Use of Individual Medications⁎⁎	n (%) withSARS-CoV-2 Test	Odds Ratio*(95% CI)	n (%) with SARS-CoV-2 Test Negative	Odds Ratio*(95% CI))
No Cetirizine (n = 224,817)	12,207 (5.4)	Ref	11,667 (95.9)	Ref
Cetirizine (n = 5559)	508 (9.1)	1.75 (1.60, 1.92)	488 (96.1)	1.05 (0.68, 1.63)
No Diphenhydramine (n = 226,546)	12,160 (5.4)	Ref	11,608 (95.7)	Ref
Diphenhydramine (n = 3830)	555 (14.5)	2.99 (2.73, 3.28)	547 (98.6)	2.95 (1.48, 5.91)
No Hydroxyzine (n = 226,467)	12,349 (5.5)	Ref	11,807 (95.9)	Ref
Hydroxyzine (n = 3909)	366 (9.4)	1.79 (1.61, 2.00)	348 (95.1)	0.84 (0.53, 1.32)

Odds ratios in bold indicate statistical significance (p-value < 0.05).

Note: Medication usage is within each type, and does not indicate single usage.

Table 3 highlights the multivariable logistic regression that adjusted for patient sex, age, race/ethnicity, insurance type, documented allergy diagnosis and co-documentation of other allergy medications. These analyses revealed similar results; namely, usage of any of the three medications was associated with an increased likelihood of receiving a SARS-CoV-2 test as was the documentation of an allergy diagnosis. However, only diphenhydramine, which had the highest level of testing, was associated with an increased likelihood of a negative test (adjusted OR = 2.23; 95% CI: (1.10, 4.55)). Additionally, senior citizens (65 years old or greater), were less likely to receive a SARS-CoV-2 test (aOR = 0.85 (95% CI's 0.81, 0.90). Yet they, along with people aged 45–64 years, were also more likely to test negative (for 65+ years: aOR = 3.23 (2.30, 4.53) and for 45–64 years, aOR = 2.30 (1.85, 2.85)). Hispanic patients were more likely to receive a SARS-CoV-2 test, (aOR = 1.08 (1.00, 1.16) and were less likely to test negative (aOR = 0.45 (0.35, 0.59)); Black patients were also less likely to have a negative test (aOR = 0.54 (0.43, 0.69)).

Table 3

Logistic Regression Results: Odds of SARS-CoV-2 Testing and Odds of Being SARS-CoV-2 Negative (among those tested).

	Adjusted Odds Ratios (95% Confidence Intervals)*
Variable	Model 1: Odds of a SARS-CoV-2 Test	Model 2: Odds of SARS-CoV-2 Negative (among those with a test)
Sex
Female	1.00 (0.96, 1.04)	1.11 (0.93, 1.33)
Male	Ref	Ref
Age Category
65+	0.85 (0.81, 0.90)	3.23 (2.30, 4.53)
45–64	0.97 (0.93, 1.01)	2.30 (1.85, 2.85)
18–44	Ref	Ref
Race/Ethnicity
Non-Hispanic Black	1.01 (0.96, 1.06)	0.54 (0.43, 0.69)
Hispanic	1.08 (1.00, 1.16)	0.45 (0.35, 0.59)
Other/Multiple	1.12 (1.06, 1.18)	0.78 (0.59, 1.01)
Non-Hispanic White	Ref	Ref
Public Insurance	1.11 (1.07, 1.16)	1.76 (1.42, 2.18)
Documentation of:
Allergy Diagnosis	2.06 (1.96, 2.18)	1.16 (0.89, 1.51)
Cetirizine	1.23 (1.12, 1.36)	0.97 (0.61, 1.56)
Diphenhydramine	2.53 (2.30, 2.78)	2.23 (1.10, 4.55)
Hydroxyzine	1.46 (1.30, 1.63)	0.76 (0.47, 1.25)

Odds ratios in bold indicate statistical significance (p-value < 0.05). Analyses adjust for sex, race/ethnicity, insurance, allergy diagnosis and co-use of allergy medications.

Examination for selection (Collider) bias

Of interest was our adjusted observation of an aOR for diphenhydramine = 2.23 (95% CI = (1.10, 4.55) that suggested a statistically significant protective effect with respect to SARS-CoV-2. To quantify the magnitude of this potential selection bias on this estimate, we used a bounding factor (BF) [7] that required assumptions of four different ratios for risk (in this case, odds ratios):

Our observed OR=2.23 (95% CI = (1.10, 4.55)), and the bounding factor can be used to calculate the smallest the true OR for diphenhydramine could be; i.e., . First, we needed to estimate the maximum relative risk of being negative for SARS-CoV-2 associated with allergy symptoms (), regardless of diphenhydramine use. Such an estimate from our results is impossible given that we only observed those with tests and considering our hypothesized selection bias. However, a separate study of health care workers,[8] a group less prone to this selection bias, revealed an OR of a negative test associated with “nasal symptoms (runny, sneezing, congestion, sinus)” equal to 2.5, so we used this value for two of the four required parameters of this bounding formula (). From our data, we estimated that the OR of having a SARS-CoV-2 test (selection into the study population) associated with having an allergy diagnosis was 1.55 among those on diphenhydramine; and the OR for those not on diphenhydramine was 2.20. Given these four parameter assumptions the BF=1.89,[7] and the “true” aOR associated with diphenhydramine could be as small as aOR = 2.23/1.89 = 1.18 (95% CI = (0.58, 2.41) by dividing by 1.89 for both bounds). Such a result would indicate no true association between diphenhydramine and SARS-CoV-2 infection, and our observed association may be due (in part) to selection bias.

Discussion

The COVID-19 pandemic has galvanized the global scientific community to identify medications for protection against SARS-CoV-2 infection while we await widespread vaccination and hopefully, resultant herd immunity. Repurposing medications could be useful, especially when there is in vitro or epidemiological evidence of possible effectiveness.[5] However, the results of our study aimed to inform this discussion, and could have rested on the suggestion that among individuals with a documented SARS-CoV-2 test, those with diphenhydramine documented on their medication list were more likely to test negative for SARS-CoV-2. However, further examination of the potential bias, specifically analysis of the likelihood of having a documented test, revealed this result may be due to collider bias that is well-documented in the epidemiologic literature, including specifically related to SARS-CoV-2. [9] Since allergy symptoms are associated with allergy medication use, and allergy symptoms overlap with some symptoms of COVID-19 disease, [10] we demonstrate that the apparent protective effect of diphenhydramine may have been the result of a higher rate of SARS-CoV-2 testing. With the exposure of interest (allergy medications) highly associated with selection (those who were tested for SARS-CoV-2; Fig. 1), and as noted by Griffith, et al. specifically in regards to the COVID-19 pandemic, collider bias can lead to misinterpretation of evidence in observational studies. [6] To date, several studies have examined the relationship between existing medications and COVID-19 disease, [3,[11], [12], [13], [14], [15], [16], [17]] but few have focused on prevention. [18] Yet before large trials are funded, [19] the many biases that limit observational designs, such as selection, immortal time, and measurement biases, need to be considered. While recent in vitro research has shown evidence of a potential effect of diphenhydramine, this study highlights the perils of solely depending on EHR-based studies.

The selection bias in testing for SARS-CoV-2 can be seen beyond just the medications by examining the effect of age, and race and ethnicity. As is well publicized, COVID-19 disease is consistently the most severe in the oldest populations, yet as a group, they are tested less often than younger age groups. [20,21] With the public health messages of ‘stay at home’ most directed towards and well-received by this age group, [22] and testing messages centered around accessing a test when unable to social distance, when you have symptoms, or when you are at higher risk of getting severe disease, it is not surprising that seniors received fewer tests yet more likely to have a negative test result. The magnitude of this age-based finding in this study, a three-fold increase in SARS-CoV-2 test negativity, may be explainable by selection biases, even exceeding in magnitude the possible protective effect of diphenhydramine, which without correction was two-fold. In contrast, populations such as those who are Black or Hispanic showed a decreased likelihood of being tested with a lower likelihood of testing negative, possibly reflecting important and unresolved structural societal biases. [23], [24], [25]

Our study has several limitations, including documentation errors or omissions of medications, especially over-the-counter medications, and possible unmeasured confounding. While medication documentation errors are commonplace in any EHR, [26] over-the-counter medications may be even more likely to be poorly documented in the medical record. [27] Also, while we attempted to adjust for allergy diagnosis, EHR documentation of allergy diagnoses could be inaccurate or represent a range from mild to severe allergies. It is further possible that other indications for diphenhydramine (e.g., sleep disorders, undocumented allergies) may contribute to this bias. Additionally, it is possible that not all SARS-CoV-2 tests are documented in the EHR. While many testing sites are associated with health systems, departments of health, pharmacy chains that offer rapid tests, and some lab companies do not link back to the health system where a patient seeks care. This academic health care system has a robust internal testing mechanism, mitigating this risk, yet the extent that external testing persists without linkage is unknown. Nonetheless, certain groups or populations have different levels of testing for myriad reasons. [28] Finally, it is possible that other medications not analyzed here may yet provide protection from SARS-CoV-2 infection that could confound our results with co-administration. Famotidine, although reported to have possible therapeutic effects and potentially used in higher rates with allergy medications, [29] was not analyzed here given our inability to separate outpatient use (which would be potentially preventive for SARS-CoV-2) from inpatient (therapeutic) use.

In contrast, a strength of this analysis is that it complements other methods that reveal the strong testing selection bias that confounds associations, such as recently articulated by Mody, et al. where low area rates of testing black individuals is contrasted by very high hospitalization rates for COVID-19 disease. [25] Thus, in conclusion, while observational studies are important to inform and bridge basic science and human interventional trials, we demonstrate how results from EHR-based studies require attention for potential biases. Another potential explanation for an association between diphenhydramine and an increased likelihood of testing negative for SARS-CoV-2 is that allergies themselves, rather than the medication used to treat them, might be protective against SARS-CoV-2 infection. [30,31] However, in our study, allergies as documented in the EHR were not significantly associated with an increased likelihood of testing negative. Future studies of populations where testing is routine, systematic, and obligatory, such as certain geographically-isolated college campuses with universal, frequent testing, [32] might reduce the impact of the selection biases highlighted in this study and provide a better opportunity to measure possible protective effects of medications on the development of SARS-CoV-2 infection.

Declaration of Competing Interest

The following authors have interests not directly related to this study but worth noting:

Dr. Thompson is an editor for the patient pages of JAMA Pediatrics and as such receives an annual stipend. Dr. Ostrov has two patents: T18131 for methods to prevent and treat COVID-19 and T18371 for Diphenhydramine and Lactoferrin for prevention and treatment of COVID-19. Dr. Rasmussen has served on advisory committees for the Teva Pregnancy Registry and Solriamfetol Pregnancy Registry and has consulted for F. Hoffmann-La Roche AG as a litigation expert. All other authors have nothing to declare.