The case test-negative design for studies of the effectiveness of influenza vaccine in inpatient settings.

Background: The test-negative design (TND) to evaluate influenza vaccine effectiveness is based on patients seeking care for acute respiratory infection, with those who test positive for influenza as cases and the test-negatives serving as controls. This design has not been validated for the inpatient setting where selection bias might be different from an outpatient setting.
Methods: We derived mathematical expressions for vaccine effectiveness (VE) against laboratory-confirmed influenza hospitalizations and used numerical simulations to verify theoretical results exploring expected biases under various scenarios. We explored meaningful interpretations of VE estimates from inpatient TND studies.
Results: VE estimates from inpatient TND studies capture the vaccine-mediated protection of the source population against laboratory-confirmed influenza hospitalizations. If vaccination does not modify disease severity, these estimates are equivalent to VE against influenza virus infection. If chronic cardiopulmonary individuals are enrolled because of non-infectious exacerbation, biased VE estimates (too high) will result. If chronic cardiopulmonary disease status is adjusted for accurately, the VE estimates will be unbiased. If chronic cardiopulmonary illness cannot be adequately be characterized, excluding these individuals may provide unbiased VE estimates. Conclusions: The inpatient TND offers logistic advantages and can provide valid estimates of influenza VE. If highly vaccinated patients with respiratory exacerbation of chronic cardiopulmonary conditions are eligible for study inclusion, biased VE estimates will result unless this group is well characterized and the analysis can adequately adjust for it. Otherwise, such groups of subjects should be excluded from the analysis.

Introduction

The antigenic variability of seasonal influenza viruses requires frequent reassessments of the effectiveness of vaccines designed to prevent influenza infection and morbidity. However, randomized controlled trials are no longer ethical in settings where seasonal influenza vaccination is widely recommended, such as in the USA. The so-called ‘test-negative’ design (TND) has become a popular choice for post-licensure observational studies of the effectiveness of vaccines for influenza. Both cases and controls are identified in a clinical setting among patients meeting certain clinical criteria, e.g. for acute respiratory infection (ARI) or ‘influenza-like illness’ (ILI) and who consent to participate in the study. Those testing positive for influenza with a sensitive and specific assay, usually by reverse-transcription polymerase chain reaction (RT-PCR), are designated cases and those who test negative for influenza are used as controls. Vaccine effectiveness (VE) is calculated as VE = (1- ORV) * 100%, where ORV is the ratio of the odds of being vaccinated in cases vs the odds being vaccinated in controls or, equivalently, the odds of being a case in vaccinated vs the odds of being a case in unvaccinated study subjects. The popularity of this design in ambulatory settings can be explained not only by its ease of implementation, but also by the implicit conditioning on healthcare-seeking practices, which eliminates an important source of selection bias. The general validity of VE estimates obtained from TND studies conducted in ambulatory settings (ambulatory TND) has recently been confirmed for a wide range of conditions.

Recent investigations have used a TND approach to examine the effectiveness of influenza vaccination against influenza-related hospitalizations. VE estimates from both inpatient and outpatient TND studies are used as measures of VE for specific influenza seasons and inform public health responses. The validity of the TND in the inpatient setting (inpatient TND), however, has yet to be examined. In fact, some VE estimates from inpatient TND studies appear to exceed VEs usually encountered in ambulatory settings. For example, Belongia et al. estimated influenza vaccine effectiveness for all ages against influenza-associated hospitalizations for the seasons 2006-7 in the USA to be 88% (95% confidence interval (CI) 13%, 100%), whereas North American ambulatory TND studies reported adjusted VE estimates of 52% (CI 22%, 70%) and 46% (CI 17%, 65%) for the same season. Similarly, Gefenaite et al. reported for the season 2012-13 an adjusted adult influenza VE of 86% (CI 19%, 97%) from an inpatient TND study in Lithuania. A European ambulatory TND study from the same year reported a VE of 49% (95% CI 32%, 62%). This discrepancy could reflect an effect of vaccination on disease severity, i.e. vaccinated individuals might tend to develop less severe influenza disease if infected and thus be less likely to require hospitalization, or it might simply reflect the lack of precision in the inpatient TND VE estimates. However, it could also be the result of selection bias if influenza-negative controls are substantially different from the source population in their uptake of influenza vaccination. For instance, subjects suffering from chronic conditions such as congestive heart failure or chronic obstructive pulmonary disease (COPD) (CP individuals) are more likely to be hospitalized with non-infectious respiratory disease (e.g., decompensation of their heart condition or COPD exacerbation). If they are also more likely to be vaccinated than the source population from which the cases are drawn, selection bias would result.

Here, we first examine the interpretation of VE estimates obtained from inpatient TND studies: does VE represent the level of protection against influenza hospitalizations? We then examine the effect on VE estimates of enrolling CP individuals for respiratory non-ARI exacerbation. We also investigate the effect of accurate and inaccurate adjustment for CP status on VE estimates. We theoretically derive the mathematical quantities of interest and use a simulator to verify the theoretical results.

Methods

Assumptions

We make use of notation previously described. Parameters with their symbols and baseline values are shown in Table 1. We assume that the incidence of influenza ARI and ARI of other aetiologies (‘non-influenza’) is driven by incidence rates , respectively, where represents time in days. We assume that influenza viruses represent one antigenic entity such that infection with the virus results in full immunity to influenza viruses for the remainder of the study. ARI of non-influenza aetiology does not change the future risk of acquiring ARI of any aetiology (influenza, non-influenza). Study inclusion criteria are broad enough to allow for the enrolment of subjects admitted to inpatient care for respiratory exacerbation of underlying chronic medical conditions, such as COPD, asthma or congestive heart failure, hence referred to as CP conditions. To differentiate such events from ARI-related events we will use the term non-ARI events. Individuals suffering from CP conditions are also assumed to have a higher vaccination uptake than the remainder of the population. All subjects are susceptible to influenza infection before influenza vaccination or natural infection with influenza virus. Influenza vaccination is completed before the beginning of the study period. VE is the same for CP and non-CP subjects and an ‘all-or-none’ model of the vaccination effect is assumed according to which a proportion (=VE) of those susceptible to influenza who were vaccinated become fully immune to influenza infection (vaccination-mediated); accordingly, influenza vaccination fails to protect a proportion against influenza virus infection (vaccine failure). Despite vaccine failure, an individual’s probability of becoming hospitalized with influenza may be reduced by the factor (Greek letter iota). This represents a mitigating effect of influenza vaccination on influenza disease severity. Influenza vaccination does not directly modify the probability of non-influenza outcomes. CP subjects are at higher risk for non-ARI events than non-CP subjects ( and , respectively). Given an ARI or non-ARI event, subjects are hospitalized with a given probability which depends on the type of event and on their CP status: For example, is the probability of a CP subject being hospitalized with non-influenza ARI; see Table 1. All subjects hospitalized with an ARI or non-ARI event are tested for influenza and included in the study with probability . The laboratory test used to assess influenza infection has perfect accuracy. Similarly, both vaccination and CP status (0 or 1) are assessed accurately. We assume that CP status is adjusted for in the analysis by the use of a binary covariate in the logistic regression analysis. For the sake of simplicity we ignored confounders of practical importance such as age and calendar time.

Table 1.

Parameter descriptions and symbols with default values. The index 0/1 denotes parameter values for individuals without chronic cardiopulmonary (CP) conditions and CP individuals, respectively

Parameter	Symbol	Baseline value
Total population size	N_pop	1E + 06
Target no.of cases	–	500
Target no.of controls	–	1000
Duration of study (days)	τ	150
Prevalence of CP status	η	0.2
Vaccination uptake (non-CP)	ν_0	0.4
Vaccination uptake (CP)	ν_1	0.8
Vaccine efficacy against infection	ϕ	0.6
Proportion of influenza hospitalizations prevented by vaccine, given infection	ι	0
Incidence constant (maximum daily influenza incidence rate per 1000 per day)	λ_0	4.0
Incidence rate of influenza infection at time t	λ_I(t)	λ_osint/τπ
Incidence rate of non-influenza infection at time t	λ_NI(t)	λ_ocost/τπ+absmin(cost/τπ, t∈(0,τ)
Incidence rate of non-acute respiratory infection (ARI) events (non-CP)	λ_CP0	Mean(λ_NI(t))/10
Incidence rate of non-ARI events (CP)	λ_CP1	Mean(λ_NI(t))
Testing probability of inpatients	σ	0.5
Test sensitivity	α	1.0
Test sensitivity reduction by vaccination	ξ	0.0
Influenza test specificity	ε	1.0
Vaccine status assessment sensitivity	ψ	1.0
Vaccine status assessment specificity	ω	1.0
CP status assessment sensitivity	ζ	1.0
CP status assessment specificity	θ	1.0
Probability of hosp.resulting from influenza (non-CP)	γ_I0	0.01
Probability of hospitalization (hosp) resulting from influenza (CP)	γ_I1	0.05
Probability of hosp.with non-influenza ARI (non-CP)	γ_NI0	0.01
Probability of hosp.with non-influenza ARI (CP)	γ_NI1	0.05
Probability of hosp.with non-ARI events (non-CP)	γ_CP0	0.02
Probability of hosp.with non-ARI events (CP)	γ_CP1	0.1

Simulation study

We simulated the daily incidence of the events of interest (influenza and non-influenza ARI, non-ARI events), the occurrence of relevant downstream events (hospitalization, influenza testing, study participation etc.) which resulted in simulated data sets that were analysed using logistic regression analysis. This allowed us to investigate the effect of certain parameters on resulting VE estimates (Supplement 2, available as Supplementary data at IJE online). We did not model transmission, but based the daily incidence on given incidence rate functions for all events (see below). Briefly, the population is subdivided into non-CP (‘normal’), and CP subjects, who can be either vaccinated or unvaccinated. Infections with influenza virus, with non-influenza ARI agents and non-ARI events were generated for each day of the simulation by applying the respective incidence rates to the respective population groups. Driven by values of the parameters that represent conditional probabilities, such as the probability of non-CP subjects being hospitalized given influenza infection (e.g. ), data generation from an inpatient TND study is simulated. To assess the ‘meaning’ of VE estimates from inpatient TND studies, hypothetical cohort studies, representing the whole population, were simulated in which inpatient TND studies were nested. We then varied to produce a specific VE against hospitalization:

(see Supplement 1 S9 available as Supplementary data at IJE online) and estimated as where is the relative risk of influenza hospitalization in those vaccinated compared with those not vaccinated, estimated by CP-adjusted binomial regression and exponentiating the coefficient estimate associated with vaccination. For each simulated TND study we estimated VE using logistic regression analysis, adjusting for CP status. We then compared the inpatient TND VE estimates with the cohort values in 10 000 simulations for each .

To investigate the effect of CP status on VE estimates from inpatient TND studies, we simulated 10 000 studies, for each calculating the crude VE, VE by CP status (separate analysis for CP and non-CP subjects) and adjusted VE using logistic regression analysis (Supplement 1). We also simulated the situation where CP subjects were not homogeneous with respect to their vaccination uptake and risk of non-ARI events, such that their marginal vaccination uptake and non-ARI risk remained the same (80% and 4 per 1000 per day, respectively), but two-thirds of them suffered 5-fold higher rates of non-ARI events compared with non-CP subjects and one-third suffered 20-fold higher rates of non-ARI events than non-CP subjects. Of the former group 75% were vaccinated, whereas 90% were vaccinated of the latter group.

The simulation model was implemented using R 3.1.1 and can be downloaded (Supplement 2). All simulations were based on fixed sets of parameters (Table 1) unless explicitly stated. The bias was calculated as the median difference between the estimated VE and , along with the 2.5th and 97.5th percentiles of that difference as empirical 95% confidence intervals (CI).

Results

The interpretation of VE estimates from inpatient TND studies

Theoretically, if the controls are representative of the source population with regard to vaccine receipt, if both outcome and vaccination status are accurately measured and if the vaccine provides ‘all-or-none’ protection, VE estimates from inpatient TND studies should represent unbiased estimates of the true VE against laboratory-confirmed influenza hospitalization for the general population (Supplement 1, equation S9). This was confirmed by simulation where the VE estimates based in inpatient studies were highly consistent with the actual protection from influenza hospitalization in the population cohort (Table 2, columns 2-4) even though they are derived from hospitalized subjects only. If, in addition, vaccination does not modify influenza disease severity (), VE against hospitalization equals VE against influenza virus infection (Supplement 1, S10, Table 2, first row). Given VE against infection (), VE against laboratory-confirmed influenza hospitalization, , increases linearly with the ‘attenuation factor’ because . This was confirmed by simulation (Table 2). The difference between VE against infection () and the VE against laboratory-confirmed influenza hospitalization (, given a certain ‘attenuation factor’ , is larger for smaller values of than it is for larger values (Supplement 1, S9). As only outcomes that test positive for influenza, usually by molecular methods such as RT-PCR, define case status, hospitalizations due to late complications of influenza will not be captured and the VE estimated in inpatient TND studies relates only to hospitalization with laboratory-confirmed influenza and not necessarily to all influenza-associated hospitalizations.

Table 2.

The comparison of vaccine effectiveness (VE) estimates from simulated inpatient test-negative design (TND) studies with the actual vaccine protection from influenza hospitalization (hosp.) in simulated cohort studies (see text) for different values of the proportion of influenza hospitalization that is prevented by vaccination (l)in ‘vaccine failures’. VE against infections is 60% for all scenarios. For each value l, 1 000 simulations were performed

ιa	VE against hosp.b	Cohort VE against hosp. (%)	Inpatient TND VE (%)	VE difference (% points)
0.0	60	60 (54.4, 65)	60.1 (48.5, 69)d	−0.1 (-9.1, 10.8)
0.1	64	64 (58.9, 68.6)	64.1 (53.8, 72.1)	−0.1 (-8.1, 9.7)
0.2	68	68 (63.1, 72.2)	68 (59, 75.1)	−0.1 (-7.2, 8.5)
0.3	72	72 (67.7, 75.8)	72.2 (64.3, 78.4)	−0.2 (-6.4, 7.3)
0.4	76	76 (72.2, 79.5)	76.1 (69, 81.5)	−0.1 (-5.6, 6.5)
0.5	80	80 (76.6, 83)	80.1 (74.3, 84.8)	−0.1 (-4.7, 5.4)
0.6	84	84 (81, 86.6)	84.1 (79.2, 87.9)	−0.1 (-4, 4.4)
0.7	88	88 (85.5, 90.1)	88 (84.1, 91.2)	0 (-3.1, 3.5)
0.8	92	92 (90.1, 93.6)	92.1 (89.2, 94.3)	−0.1 (-2.2, 2.5)
0.9	96	96 (94.8, 97)	96 (94.2, 97.5)	0 (-1.3, 1.5)

Type equation here

Influenza hosp. prevented by vaccination, given infection.

Calculated as .

Median (2.5th, 97.5th percentile).

Adjusted for chronic cardiopulmonary (CP) status.

Bias in VE estimate from an inpatient TND

If CP status is associated both with high vaccine uptake and with the risk of non-ARI events, and if such events are eligible for study inclusion, VE estimates that are not adjusted for the CP status (‘crude’) will be biased. This bias can be classified as selection bias because highly vaccinated controls are selectively included in the study, which leads to a misrepresentation of the source population in terms of vaccination prevalence, biasing VE estimates towards falsely high values (Table 3 and Supplement 1). The magnitude of the bias depends on the assumptions regarding the incidence of non-ARI events in both non-CP and CP individuals, as well as the vaccination coverage in both groups. If vaccination coverage does not depend on CP status, then unbiased estimates result without CP-adjustment (Supplement 1; simulation results not shown). CP-adjusted VE estimates were sensitive to parameters that drive the accuracy of the influenza test, as well as assessment of vaccination and CP status (Supplement 1). If CP status is either adjusted for accurately or VE is estimated separately for CP and non-CP individuals, unbiased estimates will result (Table 3 and Supplement 1).

Table 3.

Empirical bias distribution of crude vaccine effectiveness (VE) estimates and VE estimates adjusted for chronic cardiopulmonary (CP) status. These estimates were obtained from simulated inpatient test-negative design (TND) studies, using default parameter values in 10 000 simulations

Remarks	Analysis	Bias (% points)
No heterogeneity within CP categories	Crude	10.66 (3.24, 16.68)a
	Restricted to subjects without CP	0.02 (−23.45, 15)
	Restricted to subjects with CP	−0.06 (−13.27, 10.05)
	CP-adjusted	−0.08 (−11.09, 8.81)
Heterogeneity within CP = 1 (see text)	Crude	13.23 (6.28, 18.89)
	Restricted to subjects without CP	0.17 (−32.77, 18.48)
	Restricted to subjects with CP	8 (−1.87, 15.93)
	CP-adjusted, imperfectly	6.78 (-2.55, 14.32)

Median (2.5th, 97.5th percentile).

Bias in VE estimates from inpatient TND with heterogeneity in CP status

If CP status is not a binary characteristic, but CP individuals rather fall into two or more categories defined by vaccination uptake and study inclusion probabilities (controls), and if that heterogeneity is not fully characterized, adjustment for CP status will not result in full removal of the bias (Table 3). This scenario mimics a situation in which CP status cannot be fully characterized. Similarly, stratification on the imperfect binary CP indicator will result in a biased VE estimated among CP individuals (restricted analysis) (Table 3). However, as long as non-CP subjects (CP = 0) are homogeneous with respect to vaccination uptake and probability of being included in the study as controls, the analysis restricted to non CP subjects will give rise to unbiased VE estimates (Table 3).

Discussion

We demonstrated that, under certain general assumptions, VE estimates from inpatient TND studies can be interpreted as the level of protection bestowed by influenza vaccination against hospitalization with laboratory-confirmed influenza for the source population, despite the fact that the study is conducted only among inpatients. If vaccination does not mitigate disease severity in break-through influenza infections, then VE estimates from inpatient TND studies also quantify protection from influenza virus infection. Because proof of active infection with influenza virus is required for the case definition, late complications of influenza leading to hospitalization are not captured by inpatient TND study-derived VE estimates. Such test-negative VE estimates may thus underestimate the level of protection from complications of influenza infection.

The TND offers logistical advantages over other study designs since all enrolled patients are utilized either as cases or controls. Even though the methodological issues faced by inpatient TND studies do not fundamentally differ from the issues encountered by TND studies conducted in ambulatory patients, there are important practical differences: In studies of ambulatory patients seeking care for non-life-threatening ARI, the choice by individuals to visit a healthcare provider is a sine qua non for study inclusion. The TND implicitly corrects for the selection bias associated with healthcare-seeking behaviour which is, besides its practical advantages, the main benefit of the ambulatory TND. In the inpatient setting, access to care may introduce selection bias that could be avoided by conditioning on hospital admission.

We did, however, identify a potential source of selection bias that could affect VE estimates from inpatient TND studies. Certain chronic conditions, here referred to as CP conditions, that may be associated with high vaccination coverage may also cause inpatient admission of subjects for non-ARI events, e.g. respiratory exacerbation of their underlying condition that are not associated with influenza infection. If these events meet the study inclusion criteria, these subjects would be enrolled as controls. There, in fact, is indirect evidence for the over-representation of CP subjects among controls in some inpatient TND. For example, cases and controls may differ in indicators of chronic illness that are predictive both of vaccination status and of the probability to be hospitalized with respiratory non-ARI events. In several inpatient VE studies,,,,, controls were older than cases and much more likely to suffer from heart disease and pulmonary disease, as well as other conditions which, conceivably, might have increased their vaccination coverage and their probability of being admitted for inpatient care for non-ARI events and of thus of being enrolled in the study as controls.

We have shown, both by theoretical considerations and by simulation that, if these conditions (CP conditions) are adequately adjusted for or if the analysis is restricted to non-CP conditions and if both influenza infection and vaccination status are assessed with a high level of accuracy, unbiased VE estimates can be obtained from inpatient TND studies even if subjects are enrolled in the study because of non-ARI events. In reality, however, adjustment for CP status may not be straightforward. If, for example, “CP subjects’’ are heterogeneous with respect to their risk for non-ARI events that may lead to study inclusion and with respect to their vaccination uptake, inaccurate adjustment by CP status will result in biased VE estimates. This problem of inaccurate adjustment of selection bias is well known, for example, in educational research. A sensitivity analysis comparing full VE estimates with VE estimates obtained from the data restricted to not chronically ill patients might indicate a problem with selection bias if the two estimates are substantially different. In that case, the restricted estimate should be reported. It is important to note that we have focused our analysis on a scenario which is more representative of inpatient TND studies in adults rather paediatric populations which may offer quite different challenges.

Our analysis of the inpatient TND has some limitations. First, we assumed perfect accuracy in the assessment of both influenza and vaccination status. Jackson et al. recently showed that, although misclassification of influenza status tended to result in a slightly greater bias of VE estimates in TND studies compared with other designs, the magnitude of the bias was trivial under realistic assumptions regarding VE, the accuracy of RT-PCR and influenza attack rates. The difficulty in detecting late complications of influenza infection can also be construed as a problem of sensitivity. Even though the impact of inaccurate assessment of vaccination status is less understood, the potential effect of inaccurate characterization of vaccination status on VE estimates is concerning, although not specific for inpatient TND studies. It has been shown that vaccination self-reports may be unreliable, often leading to under-reporting of influenza vaccination, but occasionally to over-reporting. On the other hand, neither medical records nor vaccine registries are likely perfect sources for vaccination status.Misclassification of vaccination status could have unpredictable consequences for the resulting VE estimates. Our sensitivity analysis (Supplement 1) confirms that misclassification of case-control, vaccination or CP status is a source for concern. Second, we assumed an ‘all-or-none’ vaccination effect model, according to which vaccination either results in full immunity or full susceptibility to infection. As we have shown previously, odds ratio-derived VE estimates are biased toward 0 if vaccination reduces the instantaneous risk by a given fraction (VE) instead, a mechanism referred to as ‘leaky vaccine’ model. Third, our quantitative evaluation of biases in VE estimates is based on assumptions about the incidence of influenza infection, non-influenza ARI and non-ARI events, as well as assumptions about hospitalization probabilities and, importantly, the prevalence of underlying medical conditions that are associated with both vaccination coverage and the likelihood of study inclusion as controls (CP status). These parameters are highly context-dependent and can be chosen to produce both trivial and massive biases of VE estimates not adjusted for CP. Finally, we assumed influenza virus to represent a single antigenic entity. This assumption clearly does not capture the antigenic and immunological complexity of questions regarding vaccine effectiveness that arise from the interaction between sequential natural exposure and vaccination responses.,. More refined models of influenza circulation and of immunological mechanisms involved in the effect of influenza vaccination may reveal different sources of biases.

Funding

MJH’s research was supported by the National Institute of Allergies and Infectious Diseases of the National Institutes of Health (NIH) under Award R01AI110474, and by Intergovernmental Personnel Act (IPA) 1110376-05 from the Centers for Disease Controls and Prevention (CDC). The content of the manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the CDC.

Key Messages

Test-negative design (TND) studies of influenza vaccine effectiveness (VE) are increasingly used with inpatients

In the absence of bias, VE estimates from inpatient TND studies represent the protection of the source population from hospitalization with influenza

Late complications of influenza are not captured by this study design

Selection bias due to the inclusion of subjects for exacerbation of chronic pulmonary conditions that are not caused by acute respiratory infection is a potential problem of inpatient TND studies which requires careful consideration.

Click here for additional data file.