Potential for bias in assessing risk and protective factors for COVID-19: Commentary on Conlon et al.'s "Impact of the influenza vaccination on COVID-19 infection rates and severity".

In the manuscript titled, “Impact of the influenza vaccine on COVID-19 infection rates and severity,” Conlon et al. explored the relationship between influenza vaccination and positive COVID-19 test results among patients tested for COVID-19 within the Michigan Medicine health care system between February 27, 2020 and July 15, 2020. The authors found an association between vaccination for influenza in the past or current season (August 1, 2019 to July 15, 2020) and lower odds of testing positive for COVID-19 (aOR = 0.76) and, among those who tested positive, previous influenza vaccination was associated with lower odds of hospitalization (aOR = 0.58), receiving intensive care (aOR = 0.64), and mechanical ventilation (aOR = 0.45). According to the authors, these results add to a growing literature showing that vaccination against non-COVID-19 infections may be protective against COVID-19 incidence and severity. These are important questions since COVID-19 vaccine rollout has lagged globally; however, several methodological concerns limit the extent to which causal inference can be drawn from the results described in the paper.

Since only individuals who got tested for COVID-19, and not all individuals at risk for COVID-19, were included in the study, the study may be subject to selection bias. This concern is of particular relevance given the high proportion of mild (not requiring medical assistance) or asymptomatic infections and that access to testing was very limited during the study period.4, 5, 6 In addition to COVID-19 infection and related symptoms, other characteristics that put people at increased perceived risk for infection or complications (eg, comorbidities, health consciousness, access to healthcare) may lead to COVID-19 testing, so that inclusion in the analytic sample may be a common effect of the outcome of interest (having COVID-19) and the exposure (influenza vaccination) - or an effect of a potential cause of vaccination (eg, health-seeking behavior, health consciousness, comorbidities). If both exposure and outcome are positively associated with the common effect being conditioned on (ie, COVID-19 testing), a spurious, biased negative association would appear even if the null association is true (ie, biased OR<1). This issue, termed collider bias, has been identified as a major limitation of existing observational studies on risk and protective factors for COVID-19 infection rates and severity. The authors report that all patients with symptoms or recent high-risk exposure were referred to testing as long as they called the COVID hotline or health care provider, resources which were not uniformly accessible to all and are more likely to be sought after by patients at higher perceived risks of clinical complications or who are more health conscious (ie, the healthy user effect). Selection bias is likely greater before April 30, when routine testing of all patients being admitted was implemented in that hospital system. Potential approaches to limit selection bias would therefore consider stratification by calendar date (ie, later dates less subject to selection bias since COVID-19 testing was more readily available), which unfortunately was not done.

Moreover, being an observational study, Conlon et al. is subject to confounding (non-causal associations between influenza vaccination and COVID-19 infection). Of potential confounders, the authors do attempt to adjust for baseline health status (eg, hypertension, diabetes, BMI, smoking), but only imperfectly: >30% of BMI and >50% of smoking data are missing and a composite comorbidity index was created (any vs no comorbidity), leading to confounder measurement error and residual confounding. Other potential confounders, such as healthcare access and individual characteristics that could be associated with vaccination and COVID-19 infection (eg, health consciousness) were presumably not measured and, therefore, not adjusted for. Two methods were used to adjust for confounding: multivariable regression for COVID-19 infection and inverse probability of treatment weighting (IPW) for COVID-19 severity. These two approaches are generally similar if the same set of covariates is used (as in Conlon et al.), though there might be changes in accuracy and robustness depending on statistical assumptions of the outcome and exposure models, respectively. These modeling assumptions, in addition to varying levels of selection bias across outcomes, may contribute to explaining the unexpected (and unacknowledged) findings that the measure of association between vaccination and COVID-19 infection shifted downwards (OR = 0.82 to aOR = 0.76) with adjustment in multivariable regression whereas associations between vaccination and COVID-19 severity outcomes (hospitalization, intensive care, mechanical ventilation, and death) changed upward in IP-weighted models. Additionally, that never or former smokers were at considerably higher COVID-19 risk compared to current smokers in multivariable regression models of COVID-19 cases (aOR = 2.52 and aOR = 1.88, respectively) raises concerns of confounding or other forms of bias despite adjustment.

Additionally, in drawing causal inferences, looking for evidence beyond statistical associations is crucial. In this case, the mechanism for cross-protection after vaccination (ie, “trained immunity”) is not well-understood and has not been shown to prevent other viral diseases to a significant magnitude. This hypothesis also lacks specificity. Similar “protective” effects of other vaccines with regard to COVID-19 (eg, BCG, MMR) in studies that were also subject to selection bias and confounding (see, eg, Pawlowski et al, which showed strong protective effects for pneumococcal and hepatitis vaccines) and/or other obvious methodological shortcomings (eg, the ecological study by Klingen et al) suggests that the common thread across this body of research may be methodological shortcomings and not a plausible causative mechanism. Indeed, the strength of the associations reported in Conlon et al. (ranging from 24% to 55% reduction in the odds of COVID-19-related outcomes) suggest that there might be extraneous factors biasing the observed relationships.

Therefore, given the potential for misinterpretation of scientific findings and substantial vaccine hesitancy, it is prudent to be extremely cautious in interpreting findings such as Conlon et al's. For the reasons outlined above, while this study does warrant further research on the effects of non-COVID-19 vaccination on COVID-19 outcomes, it does not provide enough grounds for suggesting a causal association between influenza vaccination and protection against COVID-19 infection.