Comparison of local influenza vaccine effectiveness using two methods.

BACKGROUND: In some settings, research methods to determine influenza vaccine effectiveness (VE) may not be appropriate because of cost, time constraints, or other factors. Administrative database analysis of viral testing results and vaccination history may be a viable alternative. This study compared VE estimates from outpatient research and administrative databases.
METHODS: Using the test-negative, case-control design, data for 2017-2018 and 2018-2019 influenza seasons were collected using: 1) consent, specimen collection, RT-PCR testing and vaccine verification using multiple methods; and 2) an administrative database of outpatients with a clinical respiratory viral panel combined with electronic immunization records. Odds ratios for likelihood of influenza infection by vaccination status were calculated using multivariable logistic regression. VE = (1 - aOR) × 100.
RESULTS: Research participants were significantly younger (P < 0.001), more often white (69% vs. 59%; P < 0.001) than non-white and less frequently enrolled through the emergency department (35% vs. 72%; P < 0.001) than administrative database participants. VE was significant against all influenza and influenza A in each season and both seasons combined (37-49%). Point estimate differences between methods were evident, with higher VE in the research database, but insignificant due to low sample sizes. When enrollment sites were separately analyzed, there were significant differences in VE estimates for all influenza (66% research vs. 46% administrative P < 0.001) and influenza A (67% research vs. 49% administrative; P < 0.001) in the emergency department.
CONCLUSIONS: The selection of the appropriate method for determining influenza vaccine effectiveness depends on many factors, including sample size, subgroups of interest, etc., suggesting that research estimates may be more generalizable. Other advantages of research databases for VE estimates include lack of clinician-related selection bias for testing and less misclassification of vaccination status. The advantages of the administrative databases are potentially shorter time to VE results and lower cost.

Introduction

Since 2011, the US Flu VE Network has been estimating influenza vaccine effectiveness (VE) using a test negative design that requires specimen collection from patients seeking medical care for an acute respiratory illness. Recruitment takes place in outpatient settings such as urgent care centers, primary care offices and emergency departments. Consenting, enrolling and swabbing for PCR testing for presence of influenza is frequently performed by research personnel who are not members of the clinical staff.

With the onset of the COVID-19 pandemic, the ability to enroll patients with acute respiratory illness was abruptly curtailed. Many institutions temporarily ceased all but essential research, personal protective equipment (PPE) was in short supply and was being reserved for the protection of health care workers, while vast amounts of resources, human and otherwise, were diverted to the containment, treatment and prevention of SARS-CoV-2 infection.

In addition, the ability to test for influenza and other respiratory viruses locally, was hampered by the health system’s decision to prioritize SARS-CoV-2 testing over respiratory viral panels (RVPs) (Graham Snyder, MD, personal communication, 2020). The timeline for adding SARS-CoV-2 to the currently used multi viral testing platforms, and the duration and severity of the novel coronavirus pandemic remain unknown. It is unlikely that SARS-CoV-2 will displace influenza in the coming influenza season and the viruses will most likely co-circulate. Given that the need to determine influenza vaccine effectiveness continues, other methods of determining influenza vaccine effectiveness should be explored. Previous studies have used administrative databases to estimate influenza outpatient visits [1], track influenza outbreaks [2] and calculate influenza VE in specific population subgroups such as pregnant women [3] and older adults [4].

In this study, VE estimates using methodology from the US Flu VE Network (research database) are compared with VE estimates using data from a clinical surveillance software system (administrative database). The advantages and disadvantages of both methods of estimating influenza VE are discussed.

Methods

Data

The study took place in emergency departments, hospital-based clinics and outpatient primary and urgent care sites that are part of a large (40 hospital, 700 doctor’s offices and outpatient sites) integrated health system located in western and central Pennsylvania. Data used for this analysis were collected from three sources: 1) the health system’s clinical surveillance software system which extracts virology test results from the EMR (Theradoc); 2) the Pennsylvania Statewide Immunization Information System (PA-SIIS); and 3) research data from local outpatient facilities and emergency departments participating in the US Flu VE Network.

An IRB-approved honest broker extracted data from Theradoc on a cohort of Allegheny County residents who received an outpatient RVP test at a hospital-based clinic or emergency department of one of the general acute care hospitals in the health system during the study period that included the 2017–2018 and 2018–2019 influenza seasons. The RVPs were performed on a nasopharyngeal swab using the Genmark Luminex platform. RVP tests may be conducted year-round, but are more routinely conducted in October through April on patients presenting with acute respiratory symptoms. For patients with more than one visit with RVP tests ≤14 days apart, only data from the first visit were included. If visits occurred greater than 14 days apart, then data from both visits were included.

Analysts created annual Boolean indicators for influenza from the RVP results. If no positive influenza result was observed, the final specimen collection date was saved to confirm there was no infection as of that date. This list also contained basic demographic data of race, sex and age. This list of patients was combined with immunization records from PA-SIIS. In cases where an individual had more than one vaccination in a given influenza season, the immunization records were reduced to a single record per patient per influenza season by selecting the first vaccination date. This dataset is henceforth called the “administrative” database.

The “research” database was derived from participants who were recruited from ambulatory, urgent care clinics and emergency departments during the 2017–2018 and 2018–2019 influenza seasons for the US Flu VE Network study. Detailed study methods on the US Flu VE Network have been described elsewhere [5], [6], [7], [8], [9]. Briefly, patients aged ≥6 months presenting with an acute respiratory infection (ARI) including cough within 7 days of symptom onset were enrolled at participating outpatient healthcare facilities, including community physician offices, urgent care centers and emergency departments. Patients who had received antiviral medication in the 7 days before enrollment or had been enrolled in the prior 14 days were ineligible. Following informed consent, study staff collected respiratory specimens (nasal and throat swabs from patients aged ≥2 years or nasal swabs only from patients aged <2 years) for influenza virus testing (including virus type and subtype) by reverse-transcription polymerase chain reaction (RT-PCR). Demographic data were obtained from interview. Vaccination status was based on documented receipt of each year’s influenza vaccine from PA-SIIS.

The two databases were considered to be independent because they primarily included patients from different clinical sites and because enrollees who had clinical RVP testing and were enrolled in the US Flu VE network study accounted for <4% of the total administrative database.

Study periods

The influenza circulation period, defined as the dates between the first and last influenza positive research enrollment during each season, was determined for each year in both the administrative and research databases. Subjects with influenza testing performed outside the influenza circulation periods were excluded from analyses. The enrollment period details are shown in Table 1 . (See Fig. 1a, Fig. 1b ) for the flow diagrams of participants in each database.

Table 1

Description of analyzable sample from two sources.

Season	Enrollment datesa	N	Flu Circulation Period	Outside flu circulation period (n)	Within flu circulation period (n)	Excludedb (n)	Analyzable sample (n)
Administrative database
2017–2018	12/01/2017–03/29/2018	1003	12/08/2017–03/29/2018	5	998	35	963
2018–2019	12/05/2018–05/03/2019	1065	12/12/2018–04/15/2019	81	984	26	958
Total	–	2068	–	86	1982	61	1921
Research database
2017–2018	12/01/2017–03/29/2018	1376	12/04/2017–03/29/2018	6	1370	131	1239
2018–2019	12/05/2018–05/03/2019	1759	12/07/2018–04/29/2019	29	1730	124	1606
Total	–	3135	–	35	3100	255	2845

Enrollment dates for administrative database were set to those used for the research database.

Vaccinations <14 days and age <6 months.

Fig. 1a

Flow chart for administrative database.

Fig. 1b

Flow chart for research database.

Statistical analyses

Summary statistics of the demographic and clinical characteristics were determined for the administrative and research databases. Baseline characteristics were compared between vaccinated and unvaccinated patients using Chi-square tests for categorical variables.

A test-negative study estimates VE by comparing the odds of vaccination among RT-PCR confirmed influenza cases to the odds of vaccination among controls. Using odds ratios obtained from multivariable logistic regression models, VE estimates were calculated as (1-aOR) × 100.

A series of logistic regression models was conducted with RT-PCR confirmed influenza A and B as the dependent variables and vaccination status as the independent variable. The primary analyses determined VE for all influenza; subgroup analyses determined VE for influenza A/H1N1, influenza A/H3N2, and influenza B (both lineages combined due to small numbers of cases). The logistic regression models were adjusted a priori for age group (6 months-17 years, 18–49 years, 50–64 years and 65+ years), sex, race (white, non-white), influenza season (2017–2018, 2018–2019), prior vaccination status for the immediately preceding year and whether the visit took place in the emergency department. The VE and its 95% CI reported for the two databases were also stratified by age group and by season. Thus, age group was not adjusted for in the age-stratified model and season was not adjusted for in models which stratified seasons.

The significance of the difference between administrative and research database VE was identified through the effect of interactions in the logistic regression model. An indicator variable was created for database and the interaction of this binary indicator and the vaccination status was included in the model. Data were analyzed using SAS version 9.4 (SAS Institute, Cary, NC, USA). Statistical significance was defined as a two-sided P value < 0.05. The University of Pittsburgh IRB approved the study.

Results

There were significant differences in demographic characteristics between the administrative and research databases as shown in Table 2 . For example, compared with those in the administrative database, research participants were more likely to be younger, white and less likely to have been seen in the emergency department. Thus, the first difference between the methodologies is reflected in the demographics of the two populations.

Table 2

Characteristics of persons in administrative and research databases 2017–2019, overall, by vaccination status and by influenza status.

Characteristic	Administrative database N = 1910 (%)	Research database N = 2845 (%)	Administrative database vaccination status		Research database vaccination status		Administrative database influenza status		Research database influenza status
			No n = 1115 (%)	Yes n = 795 (%)	No n = 1666 (%)	Yes n = 1179 (%)	No n = 1245 (%)	Yes n = 665 (%)	No n = 1872 (%)	Yes n = 973 (%)
Age Group
6 months–17 years	31.4	34.7‡	29.9	33.5‡	35.3	33.7‡	23.2	46.6‡	34.7	34.5
18–49 years	35.4	39.1	41.0	27.7	45.4	30.4	37.4	31.7	37.8	41.7
50–64 years	15.9	15.9	15.2	16.8	13.6	19.1	18.4	11.1	16.5	14.7
≥65 years	17.3	10.3	13.9	22.0	5.7	16.8	21.0	10.6	11.0	9.1
White race, ref. = non-white	58.6	69.3‡	51.4	68.8‡	65.4	74.7‡	63.1	50.4‡	70.6	66.8*
Female sex, ref. = male	58.0	56.3	58.2	57.7	52.9	61.1‡	58.2	57.7	58.2	52.7†
Prior vaccination, ref. = not vaccinated prior year	38.1	39.5	19.3	64.5‡	18.8	68.7‡	40.2	34.3†	42.3	34.0‡
Emergency department testing/enrollment	72.1	34.7‡	76.9	65.5‡	37.3	31.0‡	66.4	83.0‡	35.7	32.8
Vaccinated, ref. = unvaccinated	41.6	41.4	–	–	–	–	45.8	33.8‡	46.9	31.0‡
Season
2017–2018	–	–	–	–	–	–	53.0	44.1‡	41.2	48.0‡
2018–2019	–	–	–	–	–	–	47.0	55.9	58.8	52.0

P < 0.05.

P < 0.01.

P < 0.001.

Table 2 also compares vaccinated with non-vaccinated participants and influenza cases and non-cases within each database. In the administrative database, the vaccinated compared with the unvaccinated differed by age, race, (69% vs. 51% white; P < 0.001), previous vaccination status (65% vs. 19%; P < 0.001) and number of emergency department enrollments (66% vs. 77%; P < 0.001). Among the individuals in the research database, compared with the unvaccinated, the vaccinated were older, more often white than non-white, female, previously vaccinated, and less frequently enrolled through the emergency department (all P < 0.001). In the administrative database, those infected with influenza were younger, non-white, not vaccinated in the previous season or the current season, enrolled in the 2018–2019 season and enrolled in the emergency department (all P < 0.01). Those in the research database with influenza were more likely to be non-white, male, not vaccinated in the previous season or the current season, and enrolled in the 2017–2018 season (all P < 0.05).

Influenza circulation differed in the two seasons; influenza A/H3N2 was the predominant strain in 2017–2018, although there was circulation of A/H1N1 and influenza B, whereas in 2018–2019, A/H3N2 and A/H1N1 circulated in nearly equal proportions with a small proportion of influenza B. Table 3 shows the unadjusted and adjusted VE estimates from each database for any influenza, influenza A, A/H1N1, A/H3N2, and influenza B for 2017–2018, 2018–2019, and for both seasons combined. Using the administrative database, significant VE estimates were observed for any influenza, and influenza A for both seasons individually and combined, and for both influenza A/H1N1 and A/H3N2 for 2017–2018 and both seasons combined. VE for influenza B was not significant for either season singly or combined. Using the research database, VE was significant for all strains and substrains measured in each season and overall, with the exception of influenza B in 2018–2019. Significant VE estimates ranged from 39% (95%CI = 15, 57) for A/H3N2 in 2017–2018 to 69% (95% CI = 35, 85) for A/H1N1 for the 2017–2018 season. The last column in Table 3 indicates the P value for the comparison of VE estimates between both data sources. The research VE estimates were not significantly different from administrative VE estimates with the exception of influenza B in 2017–2018. Although not statistically significant due to overlapping confidence intervals and limited sample size, the VE for any influenza for 2017–2018 was 12 percentage points higher for the research database (49%; 95%CI = 31, and 62) than for the administrative database (37%; 95%CI = 13, 54). The adjusted VE estimates for influenza B differed by >40 percentage points in 2017–2018 and the 2017–2019 combined seasons between the administrative and research VE estimates, but differences were not statistically significant.

Table 3

Comparison of vaccine effectiveness (VE) across all age-groups for 2017–2019 derived from administrative and research databases.

Strain	Season	Administrative database		Research database		Adjusted VE (95% CI)
		Vaccinated among cases	Vaccinated among controls	Vaccinated among cases	Vaccinated among controls	Administrative database	Research database	P valued
Any Influenza	2017–2018a	104/189	319/341	142/325	353/419	37 (13, 54)	49 (31, 62)	0.437
	2018–2019a	121/251	251/334	160/346	524/576	44 (19, 61)	46 (30, 58)	0.332
	2017–2019b	225/440	570/675	302/671	877/995	39 (22, 51)	47 (36, 56)	0.310
Influenza A	2017–2018a	80/162	319/341	119/258	353/419	43 (19, 59)	45 (24, 60)	0.960
	2018–2019a	121/249	251/334	155/342	524/576	43 (18, 60)	47 (31, 59)	0.260
	2017–2019b	201/411	570/675	274/600	877/995	41 (25, 53)	46 (34, 56)	0.515
A/H1N1	2017–2018a	7/28	319/341	11/54	353/419	70 (26, 82)	69 (35, 85)	0.997
	2018–2019a	20/55	251/334	88/218	524/576	46 (−1, 71)	48 (29, 62)	0.775
	2017–2019b	27/83	570/675	99/272	877/995	55 (26, 73)	52 (36, 64)	0.915
A/H3N2	2017–2018a	45/91	319/341	107/202	353/419	40 (7, 61)	39 (15, 57)	0.525
	2018–2019a	13/31	251/334	65/124	524/576	52 (−6, 78)	45 (21, 62)	0.797
	2017–2019b	58/122	570/675	172/326	877/995	42 (15, 60)	43 (27, 55)	0.572
Influenza B	2017–2018a	24/28	319/341	23/65	353/419	12 (−65, 53)	63 (32, 80)	0.034
	2018–2019a	1/2	251/334	3/4	524/576	48 (−656, 96)c	−4 (−441, 81)	0.758
	2017–2019b	25/30	570/675	26/69	877/995	14 (−57, 53)	58 (27, 76)	0.057

Bold indicates significance.

Adjusted for age group, race, sex, prior vaccination, and emergency department visit.

Adjusted for age group, race, sex, prior vaccination, season, and emergency department visit.

The validity of the model fit is questionable due to zero cell frequencies between race, and sex when using as classification variable and excluded from the model. Age group was used without classification specification in the model.

P value for comparison of VE from administrative and research databases.

Data from both 2017–2018 and 2018–2019 were combined for VE analyses by age group shown in Supplemental Table 1. Significant VE estimates were observed consistently in both databases in the youngest age group (6 months-17 years) for any influenza (55% and 64%), influenza A (60% and 64%), influenza A/H1N1 (78% and 72%) and influenza A/H3N2 (78% and 56%). VE point estimates for 18–49-year-olds in the research database were significant for any influenza (41%), influenza A (35%), and A/H3N2 (36%).

Table 4 shows the demographic characteristics and VE estimates of patients in each of the databases split into emergency departments and outpatient clinics. In the emergency departments, patients in the administrative databases were significantly older (P < 0.001), more often white (50.5% vs 43.9%; P = 0.002) and more often female (58.5% vs. 49.1%; P < 0.001) than those in the research database. Furthermore, when interaction terms were used, VE estimates from the administrative database were significantly lower than from the research database against all influenza (46%, 95% CI = 29%, 59% vs. 66%, 95% CI = 52%, 76%; P < 0.001) and any influenza A (49%, 95% CI = 33%, 62% vs. 67%, 95% CI = 53%, 77%; P = 0.002).

Table 4

Characteristics and vaccine effectiveness (VE) estimates of persons in administrative and research databases 2017–2019.

Outcome	Administrative database			Research database			P valueb
	Vaccinated among cases	Vaccinated among controls	Adjusted VEa (95% CI)	Vaccinated among cases	Vaccinated among controls	Adjusted VEa (95% CI)
Tested/enrolled in emergency departments only
Any Influenza	175/377	346/480	46 (29, 59)	67/252	299/369	66 (52, 76)	<0.001
Influenza A	155/354	346/480	49 (33, 62)	62/240	299/369	67 (53, 77)	0.002
A/H1N1	9/60	346/480	80 (56, 91)	32/141	299/369	68 (50, 80)	0.594
A/H3N2	32/89	346/480	52 (22, 70)	30/98	299/369	66 (45, 79)	0.408
Tested/enrolled in outpatient clinics only
Any Influenza	50/63	224/195	13 (−41, 46)	235/419	578/626	35 (18, 49)	0.762
Influenza A	46/57	224/195	11 (−46, 46)	212/360	578/626	32 (12, 47)	0.832
A/H1N1	18/23	224/195	−8 (−127, 49)	67/131	578/626	36 (6, 56)	0.487
A/H3N2	26/33	224/195	24 (−44, 59)	142/228	578/626	31 (7, 48)	0.774

Bold indicates significance.

Adjusted for age group, race, sex, prior vaccination, season.

P value for comparison of adjusted VE from administrative and research databases

Three differences between the databases for those enrolled in outpatient clinics were noted. Administrative database enrollees were significantly older (P < 0.001), and more often vaccinated both in the enrollment season (51.5% vs. 43.8%; P = 0.002) and in the prior season (47.7% vs. 40.3%; P = 0.002).

Discussion

In this comparison of influenza vaccine effectiveness estimates from a test-negative case-control research study and administrative data sources, subjects differed by demographic characteristics of age, race and site of enrollment. The administrative database included older patients, more non-whites and more patients enrolled in the emergency department. With this limited sample size, the majority of comparisons found no significant differences in VE estimates between the two databases; however, differences were found for influenza B in 2017–2018 and with a larger sample size may have been found for others, given the percentage point differences between estimates. When the databases were further divided into emergency department and outpatient clinics, new patterns emerged. In emergency departments, research enrollees were younger, more often male and non-white, and VE against any influenza and all influenza A was significantly higher than was found in the administrative database. Whereas, in the outpatient clinics, age distribution was more evenly distributed in the administrative database and vaccination rates appeared to be higher than in the research database, but there were no significant differences in VE estimates. These differences may be attributed to the fact that recruitment for this research study was limited to two emergency departments (one a pediatric hospital) among several included in the administrative database and to a greater emphasis on research enrollment at urgent and primary care centers. However, given that influenza VE is frequently lower among older persons [10], it is not unexpected that VE estimates would be lower in a population that was significantly older.

The results of this study and examination of some of the literature suggest advantages and disadvantages of each type of data for determining influenza VE as shown in Box 1 . Administrative database analysis for influenza vaccine effectiveness has some clear advantages that lie primarily in the potential sample sizes [4], [10], [11], [12] and relative cost per patient. Administrative databases can include large single health system data [2] or can combine data across large geographic areas [1], thus may more accurately estimate VE for a country or region. These large datasets would be expected to be sufficiently powered to produce narrow confidence intervals and thus instill confidence in their VE estimates. Large sample sizes, with rich EMR data, would also allow for inclusion of instrumental variables (those not collected at enrollment) in the analysis [10]. The per-patient cost of acquiring data on large numbers of patients would be lower because the cost of influenza testing would be part of clinical care and would not be included as part of the study. Furthermore, there would be no need to hire research assistants to screen, consent and enroll participants. Administrative data collection does not carry risk of infection to the research staff that in-person enrollment does. Finally, the results of clinical influenza testing are likely to be rapidly available because of their importance to clinical care and infection control.

Conversely, use of administrative databases for VE analyses has some disadvantages. For example, administrative databases may represent the subgroup of individuals who seek care at the specific types of facilities included in the database, such as the hospital-based clinics in this study. This situation may result in certain demographic, health or healthcare-seeking-behavior characteristics and may limit general applicability of VE estimates. There may be limited information available about patients who are included without consent [4]; the quality of the data received may vary; there may be delays in completing administrative databases [4], especially if data are being combined from several health systems; or there may not be indicators for factors that may affect VE in some groups, such as frailty among older adults [13].

Regarding influenza testing, there may be several types of tests used with varying sensitivity and specificity, influenza subtyping may not be available, and there may be policies that favor or disfavor influenza testing in certain settings that may introduce bias [14]. A study of inpatient testing in our locale did not find any association between influenza vaccination and clinical testing. However, clinical testing was significantly higher during the peak and post peak influenza periods than earlier in the season and higher among younger hospitalized patients with an ARI [15].

In administrative databases, vaccine verification is limited to electronic sources. It is not known how many health systems are linked electronically to their state’s immunization registries, what the lag time is for making those data transfers or the completeness of the registries’ data. Lack of an automatic feed to the EMR, or incomplete registry data could introduce bias by vaccination status misclassification [16].

Using research databases to determine influenza VE also has advantages and disadvantages. The primary advantage of research databases is control. For example, the researcher can determine the type or types of influenza testing to be used based on test characteristics, the clinical setting for recruitment, whether or not to oversample certain subpopulations. The researcher can set standards for data quality and ensure that those standards are met with training and monitoring of research staff and monitoring data completeness and quality as they are being collected. The data may be more informative and complete because subtyping of influenza virus can be conducted, and manual, as well as electronic, vaccine verification can be employed. Because misclassification of vaccination status can produce substantial biases in VE estimates, manual verification may be worth the extra effort and expense.

The primary disadvantages of research databases are cost and scale. Significant human and other resources are needed to train research staff; identify, approach, and enroll participants while risking infection; transport research specimens; and analyze them. These specimens may be batch analyzed, delaying return of results. Distance of sites from the research offices or the testing labs may limit geographical reach of the study and the number of sites may be limited by costs.

No matter which method is used, speed to release of VE estimates depends on the relative timeliness of all parts of the process from data collection to analyses to compiling the report. If administrative data were readily available, they might be the best source for rapid influenza VE estimates, with the caveat that their generalizability may be limited. Currently the US Flu VE Network provides mid-season VE estimates using self-reported vaccination status that are published in the Morbidity and Mortality Weekly Report. Earlier VE estimates based on administrative databases may be useful in a severe influenza season, or in the present era of potential co-circulation of SARS-CoV-2, when treatment and infection control measures might differ. A hybrid approach that uses clinical specimen collection and EMR data from a specified group, such as the Royal College of General Practitioners Research and Surveillance Centre [17]may be possible in areas where PCR testing for influenza is routinely conducted in outpatient settings.

Strengths and limitations

Strengths of this study include the test-negative design that utilizes the highly specific molecular methods to diagnose influenza, limiting misclassification of the outcome status. The sample size was limited, particularly for substrain analyses, as we found that point estimate differences of 40 were not significant, due to overlapping confidence intervals. A post hoc power calculation using our data and a hypothetical value for the overlap between databases indicated that an N of 5490 would be needed to detect differences in VE between databases with 80% power at P < 0.05. In general, post hoc power calculations are not advisable [18], [19]. The number of potential confounding factors included in regression modeling was limited due to the limited data available using administrative testing and vaccination databases. We checked for confounding for month of the influenza season for influenza status and for vaccination status. We found that month and season were both significant for influenza status in both databases. But season was significant for vaccination status only in the administrative database. Thus, we included season in the regression for both databases for uniformity for comparing VE between the two models. Whether residual confounding would differ between the administrative and research databases for other variables is unknown. Without additional sources of vaccination data, the differences in vaccination coverage between databases may be attributed to underreporting to the state registry, especially among adults. Incomplete vaccination data could result in inaccurate VE estimates. However, we attempted to minimize differences between databases by only using state registry-verified vaccinations. Those who self-reported vaccination in the research database that was not verified in PA-SIIS were not included in the analysis. Finally, this study was conducted in one locale and should be repeated in other networks.

Conclusions

The selection of the appropriate method for determining influenza vaccine effectiveness depends on a multitude of factors. Among those factors is sample size, which likely obscured significant differences between administrative and research database VE estimates in this study except when limiting the analyses to the emergency departments.In this case, the research estimates were better. Differences in the types of persons enrolled in the databases, suggest that research estimates may be more generalizable. Other advantages of research databases for VE estimates include lack of clinician-related selection bias for testing and less misclassification of vaccination status because multiple sources are used. The advantages of the administrative databases are potentially shorter time to VE results and lower cost.

Funding

This work was supported by the Centers for Disease Control and Prevention (CDC) [5U01IP001035-02] and by National Institutes of Health (NIH) [UL1TR001857]. This work represents the views of the authors and not the CDC or NIH. It is subject to CDC’s public access policy.

CRediT authorship contribution statement

G.K. Balasubramani: Conceptualization, Formal analysis, Writing - review & editing. Richard K Zimmerman: Conceptualization, Writing - review & editing, Funding acquisition, Supervision. Heather Eng: Data curation, Writing - review & editing. Jason Lyons: Data curation, Writing - review & editing. Lloyd Clarke: Data curation, Writing - review & editing. Mary Patricia Nowalk: Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: [Drs. Nowalk and Balasubramani, Ms. Eng and Mr. Lyons have grant funding from Merck & Co., Inc. for an unrelated project. Dr. Zimmerman has grant funding from Sanofi Pasteur and Merck & Co., Inc. Mr. Clarke has no conflicts to report].

Advantages	Disadvantages
Administrative database analysis
• Lower cost; no need to screen or enroll participants, RVP performed and charged as part of clinical care	• Population distribution may be dependent upon type of patients seen at clinics included in database, no oversampling possible (selection bias)
• Influenza testing results potentially reported rapidly	• Data quality is not guaranteed.
• No risk of exposure of research staff to infectious diseases	• Sensitivity and specificity of the PCR test cannot be guaranteed because the time between symptom onset and testing is not known and may lead to underestimation of VE.
• Potentially large sample sizes [4], [10], [11], [12]	• Limited information about patients is available without consent or is missing from the administrative database (Information bias, unidentified confounders) [4]
• Can use instrumental variable analysis to improve estimates and remove bias [10]	• Subtyping for influenza B may not be available and may not always be reported for influenza A
	• In some health systems, outpatients may not be routinely tested, physician testing bias may skew results
	• Different types of testing to identify cases, with varying accuracy may be used
	• Vaccination verification limited to electronic databases
	• Delays in completing administrative databases [4]
Research database analyses
• Oversampling of specific population subgroups possible	• Cost and other factors such as distance, could limit sample size
• More control over data quality and completeness	• Research results typically batched, resulting in reporting delays
• Influenza can be completely subtyped with lineages	• Research staff at risk of infection
• All enrollees are tested with the same, potentially high-quality test(s)	• Potentially eligible enrollees may decline to participate, could introduce bias
• Both electronic and manual confirmation of vaccination status is possible	• Non-electronic vaccine verification methods if used, can be time consuming
• Rapid reporting of results early in the season	• Substantial resources, both human and otherwise, are required to identify, screen, enroll smaller number of participants
• Less selection bias
• Control over timing of PCR testing improves sensitivity and specificity of test and VE estimates