Successive Wave Analysis to Assess Nonresponse Bias in a Statewide Random Sample Testing Study for SARS-CoV-2.

INTRODUCTION: Nonresponse bias occurs when participants in a study differ from eligible nonparticipants in ways that can distort study conclusions. The current study uses successive wave analysis, an established but underutilized approach, to assess nonresponse bias in a large-scale SARS-CoV-2 prevalence study. Such an approach makes use of reminders to induce participation among individuals. Based on the response continuum theory, those requiring several reminders to participate are more like nonrespondents than those who participate in a study upon first invitation, thus allowing for an examination of factors affecting participation.
METHODS: Study participants from the Indiana Population Prevalence SARS-CoV-2 Study were divided into 3 groups (eg, waves) based upon the number of reminders that were needed to induce participation. Independent variables were then used to determine whether key demographic characteristics as well as other variables hypothesized to influence study participation differed by wave using chi-square analyses. Specifically, we examined whether race, age, gender, education level, health status, tobacco behaviors, COVID-19-related symptoms, reasons for participating in the study, and SARS-CoV-2 positivity rates differed by wave.
RESULTS: Respondents included 3658 individuals, including 1495 in wave 1 (40.9%), 1246 in wave 2 (34.1%), and 917 in wave 3 (25%), for an overall participation rate of 23.6%. No significant differences in any examined variables were observed across waves, suggesting similar characteristics among those needing additional reminders compared with early participants.
CONCLUSIONS: Using established techniques, we found no evidence of nonresponse bias in a random sample with a relatively low response rate. A hypothetical additional wave of participants would be unlikely to change original study conclusions. Successive wave analysis is an effective and easy tool that can allow public health researchers to assess, and possibly adjust for, nonresponse in any epidemiological survey that uses reminders to encourage participation.

Nonresponse bias occurs when participants in a study are different from those eligible to participate in ways that can affect study conclusions.1,2 In June 2020, the National Academies of Science, Engineering, and Medicine described the challenges of estimating the population prevalence of SARS-CoV-2 using available data such as case counts, emergency visits, and deaths due to bias toward more severe cases in these data sources.3 The report concluded that representative samples, especially those that rely on random selection, are best suited for determining the extent to which a given population is affected by the pandemic. Nevertheless, even with random sampling, the selective decision of individuals to participate in testing could result in nonresponse bias, which can affect study conclusions, subsequent policy decisions, and the public health response.

As in all studies that rely on selective decisions of individuals to participate, response rates may be a function of a multitude of factors and high response rates do not guard against bias. To address nonresponse bias, researchers have several options including statistical adjustment such as inverse probability weighting4 or iterative proportional fitting5 so that respondents are weighted to represent the known characteristics of the population. These commonly used techniques are limited by the assumption that demographic characteristics are associated with the otherwise unknown selective decisions to participate. Alternatively, researchers can attempt to contact nonrespondents and ascertain their reason for not participating. However, doing so is time-sensitive and resource-intensive, making this solution suboptimal unless planned for a priori.6

An additional strategy to assess nonresponse bias is to utilize successive wave analysis to test for potential reasons that may influence selective participation.2,7 Invitations to participate in surveys are typically sent out initially and then through subsequent reminders (eg, survey waves) to solicit participation from those not yet responding. Successive wave analysis relies on the response continuum theory,8,9 which states that respondents to subsequent waves (eg, those who respond after one or more reminders) are increasingly more similar to nonrespondents than those who responded to the original survey invitation. Successive wave analysis uses existing survey items to compare responses of individuals based on how many reminders were needed to induce their participation. Researchers can utilize existing survey questions, beyond demographics, to determine whether certain characteristics are associated with the timing of participation—and if so, to what degree nonresponse bias from these attributes could change the conclusions of the study.

In the context of conducting a large statewide random sample SARS-CoV-2 prevalence study, we utilize successive wave analysis to accomplish 2 aims. First, we examine to what extent nonresponse bias was present in the Indiana statewide sample that achieved a 23.6% response rate.5 We hypothesize that several attributes may have influenced the likelihood to participate including the presence of symptoms, previous household infection, health status, or education. We assess how these characteristics differ by wave. Second, we show how successive wave analysis can be used in any epidemiological study that relies on selective participation among subjects. When nonresponse bias is detected with the use of successive wave analysis, several options are available to researchers, which will be described. Successive wave analysis is an underutilized method that has been limited to select health surveys and public opinion polls.

Methods

Data for the current study come from a statewide testing effort where selected participants were tested for active viral infection and the presence of antibodies for SARS-CoV-2 in Indiana. The data in the current analysis, part of the Indiana Prevalence Study,5,10–14 were collected from April 25 to 29, 2020. All participants were randomly selected from Indiana tax records to create a de-duplicated list of state residents that included those who filed or co-filed taxes and all of their dependents with an Indiana address. Updated contact information was then merged from the Indiana Bureau of Motor Vehicles and other state databases. The random sample was stratified by each of the state's 10 public health preparedness districts representing children and adults 12 years or older. In all cases, excluded individuals were those who were institutionalized, had an out-of-state address, or did not meet age eligibility. Randomly selected participants were contacted through postcards, text messages, e-mail, and/or telephone and given information to either voluntarily register online or speak to a live operator for more information and possible scheduling. All outreach to participants occurred between April 20 and 28, with the first reminders sent to those who had not participated after 3 days and the second reminder was sent after 2 additional days. Upon the decision to participate, participants were able to complete a survey intake form and select a testing location at one of the 68 statewide testing facilities. Participants who registered without completing the intake questions completed the survey when arriving for testing. The institutional review board at Indiana University deemed the study as not human subjects research under the public health surveillance exemption.

The main dependent variable in the current analysis is wave of participation. Wave of participation was determined on the basis of how many reminders an individual received before consenting to participate and thus scheduling a time to be tested. Individuals who registered upon first receipt of an invitation were in wave 1. Those who consented to participate, by registering for the study, after receipt of a single reminder notification (e-mail, text message, or phone call), were in wave 2. And those who participated after a second reminder were in wave 3.

The independent variables in the current study include questions from the registration survey that allow us to examine how key demographic and other characteristics differ by wave of response. The aforementioned intake form included questions about demographic characteristics (sex, age, education level, race, ethnicity), self-reported health status, use of tobacco products, and current COVID-19 symptoms. Symptoms, consistent with Centers for Disease Control and Prevention and World Health Organization guidance, were asked if they were present during the previous 2 weeks and included fever, cough, shortness of breath, chest pain, muscle ache, chills, fatigue, sore throat, runny nose, headache, diarrhea, vomiting, and loss of taste or smell. In addition, self-reported health status was collected using the following categories: excellent, very good, good, fair, or poor. Tobacco use was assessed by asking participants if they currently (every day or on some days) smoke cigarettes, use chewing tobacco, or use vaping products.

The intake form also included questions regarding reasons for participating in the current study. These questions were measured on a 4-point Likert scale ranging from “not important” to “very important” and included items assessing potential personal, clinical, or societal benefits. Several of the reasons for participation were included in the current study because they allowed us to examine whether motivation for participation differed by wave. Items measuring personal benefits included “feel good contributing to COVID-19 research,” “gaining knowledge about own COVID-19 status,” and “testing is free of charge.” Clinical benefits included “less risk of transmitting COVID-19 to family and friends,” and societal benefits included “helping inform public health officials about COVID-19,” “contributing to scientific knowledge,” and “receiving support from family and friends.”

Finally, because the overall purpose of the Indiana Prevalence Study was to determine SARS-CoV-2 positivity rates, we analyzed both active infections based on polymerized chain reaction (PCR) testing and evidence of previous infections based on antibody presences in the current study. To assess the extent of possible nonresponse bias, we conducted successive wave analysis. Specifically, we examined whether respondent demographics, health status, tobacco behaviors, COVID-19–related symptoms, reasons for participating in the study, and SARS-CoV-2 positivity rates differed by wave. Because respondents to later waves, consistent with the response continuum theory, are expected to be more similar to nonrespondents than those to the first wave, our analysis provides estimates of how these characteristics may have biased our sample.

Descriptive statistics were calculated for the sample using frequencies and percentages, and age, race, and educational attainment categories were created on the basis of risk and the distribution of responses in each variable to ensure sufficient sample sizes in each category. Consistent with previously published articles using successive wave analysis,15,16 the analyses consisted of bivariate chi-square tests for each variable of interest by wave. Age groups included (1) less than 40 years, (2) 40 to 59 years, and (3) 60+ years; race was dichotomized as White or other; and education levels were grouped as (1) high school or less, (2) some college, or (3) college degree or higher. All analyses were performed using SAS v.9.4, and P values less than .05 were considered statistically significant. Although not applicable to the current study (see nonsignificant results later), researchers can use Bonferroni corrections to prevent type II errors, given the number of bivariate analyses conducted.

Results

The study sample included 3658 individuals, including 1495 respondents in wave 1 (40.9%), 1246 respondents in wave 2 (34.1%), and 917 respondents (25%) in wave 3, for an overall participation rate of 23.6%. Table 1 presents the demographic characteristics of respondents, which uniformly did not differ by wave. Females made up 53.5%, 55.7%, and 54.6% in each of the 3 waves, which was not significantly different (χ2 = 2.5, P = .65). Similarly, White persons made up between 91.6% and 92.9%, depending on wave, which was not statistical different (χ2 = 1.58, P = .46). Finally, neither age groups (χ2 = 3.04, P = .55) nor education levels (χ2 = 11.3, P = .08) differed by wave.

TABLE 1

Demographic Characteristics of the Sample by Wave of Registration to a Population Prevalence Survey (N = 3658)

Characteristics	Wave 1 (n = 1495; 40.9%)	Wave 2 (n = 1246; 34.1%)	Wave 3 (n = 917; 25%)	P
Female	800 (53.5%)	694 (55.7%)	501 (54.6%)	.65
Male	691 (46.2%)	551 (44.3%)	414 (45.2%)
Unknown	4 (0.27%)	1 (0.08%)	2 (0.22%)
White persons	1369 (91.6%)	1152 (92.5%)	852 (92.9%)	.45
Non-White persons	126 (8.5%)	94 (7.5%)	65 (7.0%)
Age: <40 y	452 (30.3%)	359 (28.8%)	250 (27.3%)	.47
Age: 40-59 y	563 (37.6%)	475 (38.2%)	348 (37.9%)
Age: 60+ y	480 (32.1%)	412 (33.1%)	319 (34.8%)
Education levela
High school or less	373 (24.9%)	344 (27.6%)	218 (23.8%)	.08
1-3 y of college	426 (28.5%)	374 (30%)	246 (26.8%)
4 y+ of college	677 (45.3%)	513 (41.2%)	441 (48.1%)

aDifferential response to some items may result in sums that do not equal the total number of respondents in each wave.

Table 2 presents health status, reported number of COVID-19 symptoms, and tobacco use by wave; no differences were observed in any instance. The proportion of respondents in poor health ranged from 7.5% to 7.8% by wave, which was not statistically different (χ2 = 1.33, P = .86). The proportion of individuals reporting 0, 1, 2, or 3 or more symptoms did not differ by wave (χ2 = 3.6, P = .73). Finally, use of cigarettes (χ2 = 2.5, P = .29), chewing tobacco (χ2 = 0.85, P = .66), or vaping (χ2 = 2.97, P = .23) did not vary significantly by wave.

TABLE 2

Response Frequencies and Percentages by Wave and Characteristics of Participants in a Population Prevalence Study (N = 3658)a

Characteristics	Wave 1 (n = 1495; 40.9%)	Wave 2 (n = 1246; 34.1%)	Wave 3 (n = 917; 25%)	P
Health status
Excellent or very good	830 (55.6%)	668 (53.6%)	506 (55.2%)	.86
Good	522 (34.9%)	456 (36.6%)	319 (34.8%)
Fair or poor	111 (7.5%)	96 (7.71%)	72 (7.86%)
Symptoms
Participants reported having 0 symptoms	867 (57.9%)	702 (56.4%)	521 (56.8%)	.73
Participant reported having 1 symptom	211 (14.2%)	192 (15.4%)	150 (16.4%)
Participant reported having 2 symptoms	168 (11.3%)	152 (12.2%)	106 (11.6%)
Participant reported having ≥3 symptoms	249 (16.7%)	200 (16.1%)	140 (15.3%)
Tobacco use
Cigarettes	144 (9.6%)	106 (8.5%)	72 (7.8%)	.29
Chewing	31 (2.1%)	20 (1.6%)	18 (1.9%)	.66
Vaping	19 (1.3%)	20 (1.6%)	20 (2.2%)	.23

aDifferential response to some items may result in sums that do not equal the total number of respondents in each wave.

Table 3 focused on examining whether possible motivators for participation in the overall study differed by wave. Overall, the proportion of respondents who participated because they considered it somewhat or very important to contribute to research (χ2 = 4.2, P = .13) or gain knowledge of COVID-19 status (χ2 = 0.129, P = .94) or because testing was free (χ2 = 1.4, P = .50) did not differ by wave. Ultimately, receiving a positive test result did not differ by wave. PCR positivity (χ2 =1.12, P = .57), antibody positivity (χ2 = 4.17, P = .65), or positivity on either test (χ2 = 1.93, P = .38) was not statistically different by wave. Finally, living in a home where a household member was previously told by a provider that they were positive for COVID-19 did not differ by wave (χ2 = 1.4, P = .50).

TABLE 3

Possible Motivators for Participating by Wave Among Participants in a Population Prevalence Study (N = 3658)

Motivators	Wave 1(n = 1495; 40.9%)	Wave 2(n = 1246; 34.1%)	Wave 3(n = 917; 25%)	P
Reasons for participating
Feel good contributing to research
Not important/barely important	60 (4.1%)	48 (3.6%)	23 (2.5%)	.13
Somewhat/very important	1435 (96%)	1198 (96.2%)	894 (97.5%)
Gaining knowledge of own COVID-19 status
Not important/barely important	97 (2.7%)	77 (2.1%)	57 (1.6%)	.94
Somewhat/very important	1398 (93.5%)	1169 (93.8%)	860 (93.8%)
Testing is free of charge
Not important/barely important	320 (21.4%)	255 (20.5%)	207 (22.6%)	.50
Somewhat/very important	1175 (78.6%)	991 (79.6%)	710 (77.4%)
Positivity
PCR positive	16 (1.0%)	19 (1.5%)	12 (1.3%)	.57
Antibody positive	21 (1.4%)	17 (1.4%)	16 (1.7%)	.65
Any positivity (PCR or antibody)	39 (2.6%)	23 (1.8%)	23 (2.5%)	.38
Someone in household was previously positive for COVID-19	20 (1.34%)	17 (1.36%)	15 (1.64%)	.82

Abbreviation: PCR, polymerized chain reaction.

Discussion

The use of successive wave analysis, based upon the number of reminders needed to induce participation, failed to detect nonresponse bias from select individual demographic characteristics, health status and symptomatology, or other hypothesized motivators for participation. While such an approach does not guarantee the absence of nonresponse bias, it is a simple method to estimate the extent to which selective participation among subjects may have been a function of key characteristics that could bias the sample. Moreover, successive wave analysis is a complementary approach that is easier to implement than the more complex statistical methods typically used to adjust for nonresponse.14 The main goal of the Indiana Prevalence Study was to determine the positivity rate of participants. In the current analysis, we found no difference in either PCR or antibody positivity rates across waves. This suggests that hypothetical additional waves of respondents may not have changed the conclusions of the original study.5 Overall, we believe that successive wave analysis is easy to implement in any study that relies on reminders to recruit participants.

Had we detected any pattern, by wave, for demographic, health status, or other characteristics, we could have adjusted the main study's conclusions to account for this nonresponse. For example, if male gender was observed at higher rates from wave 1 to wave 2 to wave 3, the researcher would be concerned that males are underrepresented in the sample, which may bias the findings. The researcher could then compare the positivity rate (eg, the study's main outcome) among males and females to determine whether, and by how much, male underrepresentation in the sample was affecting the overall main findings of disease prevalence. If males were more likely to be positive, and it was determined on the basis of successive wave analysis that males were underrepresented, the researcher could weigh the overall prevalence by this difference. In addition to adjusting the main findings, the researcher can caveat their main findings as potentially biased by selective participation based on the specific attributes identified as different by wave (eg, male underrepresentation).

Researchers wanting to implement such an approach can utilize demographic characteristics already being collected in their study and/or other variables that allow them to test whether nonresponse is possibly due to other attributes. For example, in the Indiana Prevalence Study, there was concern that because of low statewide testing capacity at the time of the study (early in the pandemic) coupled with widespread concern about viral spread, health status and/or current symptomatology among participants would be a source of bias. Moreover, because some individuals may be more prone to participate in any research, we utilized answers to questions on the survey that allowed us to assess whether latter participants differed from early participants with respect to these motivators. Researchers should consider whether adding key questions, such as why a person was motivated to participate, to their survey will enable meaningful insights when analyzed in the context of successive wave analysis.

There are many reasons that can influence individual participation in a study that can lead to biased results. In the current study, we identified more than a dozen variables that allowed us to assess differences across successive waves of the study. The availability of additional variables enables the researcher to hypothesize and test a wider range of reasons that may influence participation. Given the number of contrasts that are available, researchers should also consider using Bonferroni or similar adjustment, given that the likelihood of spurious differences by wave increases as the number of analyses conducted increases. In our study, we did not use such adjustments because none of our identified relationships, by wave, were statistically significant at the traditional P < .05 level. Depending on the subject matter of a given study, researchers should be aware that the sensitivity of the topic (eg, illicit drug use, past sexual activities) can influence participation.17,18 In such cases, successive wave analysis can help identify the extent to which certain characteristics of participants differ by wave, which can help assess the extent of nonresponse bias.

Importantly, our study is limited in that we had a finite number of participant characteristics to examine. For example, we did not have information about occupation including essential worker status.18,19 Thus, we were unable to test whether such characteristics influenced participation in our sample. Likewise, we recognize the limitation of any omitted reasons (eg, trust of government, political affiliation)20,21 that we were unable to examine in the current study. Similarly, given how race was operationalized in the parent study,5 we recognize that a binary variable measuring White versus non-White persons may be a limiting factor. Furthermore, the frequency of the main dependent variable in the Indiana Prevalence Study was overall positivity rates, which occurred in less than 3% of participants. Rare outcomes make successive wave analysis more challenging if the sample sizes by wave provide insufficient statistical power to detect true differences even when the overall study sample size has enough power for its main objective. Researchers can mitigate the effects of this by ensuring that variables including demographics and other motivators for participation occur at higher frequency counts overall. Finally, we recognize that the collection of data occurred in a very narrow time frame early in the pandemic (April 2020), which limited the ability to gather more information on the population. Nevertheless, we believe successive wave analysis has broad applications in epidemiological and other public health research studies.

Conclusion

Successive wave analysis is an effective and easy tool to use based on reminder messages to encourage participation in studies that rely on selective participation by individuals. The approach described herein allows for analysis on individual and group variables and employs a simple statistical analysis that is complementary to traditional weighting schemes used to address nonresponse bias among participants. Understanding the extent of nonresponse bias continues to be important in epidemiological studies that make inferences about a population and/or are used to influence policy or resource allocation decisions. As such, we believe public health researchers should more commonly utilize successive wave analysis when appropriate. We note that based on our results, having had a hypothetical fourth or fifth wave may not have yielded different results but would have added unjustified costs and efforts to the overall project. Thus, successive wave analysis can also be used in real time by researchers needing to determine whether to continue recruiting for research a targeted group of participants or end data collection in the field.

The main purpose of the current study was to demonstrate how an underutilized method can more broadly be used in public health research and practice.

Overall, we showed how successive wave analysis is simple and quick and can be done without advanced statistical knowledge and software, allowing researchers to modify sampling if needed to account for nonresponse, including while data are being collected.

We believe this approach can either complement or substitute for more complicated and time-consuming methods of weighting variables or constructing propensity models that require access to population characteristics from which the sample was drawn.22,23

To help uptake of successive wave analysis among public health researchers, journal editors and reviewers can ask that researchers provide evidence of assessing nonresponse when applicable to their study methods and context. Doing so will strengthen the evidence base produced by epidemiological studies in the field.