Comparison of a web-push vs. mailed survey protocol in the Monitoring the Future panel study among adults ages 35 to 60.

Introduction: Updating the mode of data collection may affect response rates or survey results. The ongoing, national Monitoring the Future (MTF) panel study has traditionally used mailed paper surveys. In 2018, MTF experimented with a web-push data collection design for young adults ages 19-30, concluding that the web-push design improved response rates and did not change substance use estimates after controlling for sociodemographic characteristics (Patrick et al., 2021). The current study sought to replicate the web-push experiment with MTF adults ages 35 to 60 in 2020.
Methods: In 2020, the MTF panel study included an experiment to test a web-push protocol for respondents ages 35 to 60 (N = 14,379). Participants were randomized to the web-push (i.e., a web survey invitation, with paper surveys available for non-respondents) or traditional MTF (i.e., mailed paper surveys) data collection condition.
Results: Results indicated no significant difference in overall response rate for the web-push vs. standard MTF conditions in this age group. Differences in reported estimates of past 30-day substance use prevalence by condition were not significant after adjusting for sociodemographic characteristics. In multivariable models, participants in the web-push condition were less likely to respond via web (than paper) if they were Black, smoked cigarettes in the past 30 days, were unmarried, or did not have a college degree. Conclusions: Overall, the move to the web-push design had minimal impact on response rates and substance use prevalence estimates for this age group. However, in the web-push condition, sociodemographic differences were associated with mode of response.

Introduction

Web-push survey designs are intended to push survey respondents to respond via the web but offer alternative response modes (e.g., paper, telephone, in-person interviews) to those who are not willing or able to respond via the web (Dillman, 2007; Dillman, 2017; Bretschi, Schaurer, & Dillman, 2021). Prior research has examined effects of transitioning to web surveys on survey response rates, respondent characteristics, and outcome prevalence among adolescent, general adult, and older adult samples (e.g., Eaton et al., 2010; Kelfve et al., 2020; Lynn, 2020; Sun et al., 2020; Tassiopoulos et al., 2020). Results indicate web-only surveys historically reported lower response rates than other survey modes (Daikeler et al., 2020), but recent survey efforts utilizing web-push approaches have resulted in higher response rates than paper-only approaches among younger (McMaster et al., 2017; Patrick et al., 2021) and older (Kelfve et al., 2020) samples. Studies have indicated data quality appears to be comparable across web and paper surveys (Tassiopoulos et al., 2020), but respondent characteristics may be differentially associated with response mode preference (Kelfve et al., 2020; Tassiopoulos et al., 2020). Mixed results have been found in regards to outcome prevalence (Eaton et al., 2010; Kelfve et al., 2020).

The possibilities of higher response rates and lower costs than paper-only strategies make web-push and web-based data collection strategies particularly attractive. Additionally, web-based surveys can tailor questions to be relevant to different populations and respondents. Web-based surveys also have advantages in regards to programming flexibility and speed of data access (Tourangeau et al., 2013). These factors have led to growing web-push efforts among a range of large-scale data collection efforts, including the Monitoring the Future (MTF) study.

MTF is an ongoing study of substance use that began in 1975. MTF has two main components: (1) annual nationally representative cross-sectional surveys of U.S. 8th, 10th, and 12th grade students, and (2) longitudinal follow-up of a sub-sample of each annual 12th grade sample (Schulenberg et al., 2021). Longitudinal follow-up begins with a sub-sample of approximately 2,450 students selected from each 12th grade class, with those who report drug use during 12th grade oversampled. Each cohort of 2,450 students is split in half: one half is randomly assigned to begin follow-up one year after high school (at modal age 19) and the other half begins follow-up two years after high school (at modal age 20). Each respondent selected is surveyed every two years until they reach age 29 or 30. Then, follow-up surveys are conducted at five-year intervals, with all respondents surveyed at modal ages 35, 40, 45, 50, 55, and 60. The standard MTF protocol has been to mail paper questionnaires.

Prior research on MTF young adult web-push protocol and response rates

A total of four experimental studies among young adults investigated web-push survey administration. The first study in 2014 focused on data collection at ages 19–20. Three experimental mixed-mode conditions were compared: standard MTF, web-push (mailed survey invitation with survey login information, no emails), and web-push + email (identical procedures to web-push with the addition of emailed invitations and reminders). Results showed that web-push + email was the most promising mode based on response rates and lower costs relative to the other two conditions (Patrick et al., 2018). The second study used the sample to examine retention rates two years later at ages 21–22 in 2016. Participants who responded via web in 2014 were found to have higher rates of participation in 2016, suggesting that the web-push strategy could be a promising route for maintaining respondent engagement while reducing cost (Patrick et al., 2019). The third study extended previous web-push procedures by introducing text messaging and quick response (QR) codes in addition to email and optimizing the web-based survey for mobile response. The enhanced web-push condition further increased response rates compared to standard MTF condition (Patrick et al., 2020).

The fourth study expanded the enhanced web-push data collection design to include ages 19–30 in the 2018 data collection. In this study, a random half of all young adult participants aged 19–30 years were assigned to the standard MTF mail/paper survey condition, and the other half were assigned to the web-push condition. Overall, the response rate for the web-push condition was higher than the standard MTF condition, particularly for younger ages (modal age 19–20 and 23–24). After controlling for respondents’ socio-demographic characteristics, the web-push condition was associated with a 19% increase in the odds of responding compared to the standard MTF condition. Observed differences in substance use prevalence estimates became negligible when using attrition weights and controlling for socio-demographics assessed at baseline (Patrick et al., 2021). The young adult web-push experimental studies suggested that the web-push procedures produced higher response rates than standard mailed paper surveys without affecting substance use estimates once attrition weights and socio-demographic variables were included in models. Web-push procedures became the standard MTF panel data collection protocol from 2020 onward in follow-up data collection for all respondents ages 19–30.

Current study: MTF web-push protocol at ages 35–60

The utility of web-push procedures among adults in midlife and older adulthood is not yet fully understood. Compared with young adults, those aged 35 and older have had different levels of lifetime digital literacy, with such experience being particularly lower for those approaching older adulthood. Thus, it is critical to examine the nature of response to web-push survey efforts across the lifespan. In 2020, the MTF web-push experimental design used in the fourth young adult experiment was also implemented for respondents participating at ages 35–60. The current study’s purpose was to evaluate the effects of the web-push protocol compared with the standard MTF protocol among those surveyed in 2020. Five research questions (RQs) guided analyses:

RQ1: Did response rates differ between the web-push and standard MTF conditions?

RQ2: Did the effect of experimental condition on response rates differ by age and/or other socio-demographic characteristics?

RQ3: Did past 30-day substance use prevalence estimates differ based on experimental condition?

RQ4: Among those randomized to the web-push condition, did the likelihood of responding by web differ by baseline and concurrent sociodemographic characteristics and substance use?

RQ5: Among those randomized to the web-push condition, did past 30-day substance prevalence estimates differ by response mode (web versus paper)?

Methods

Data and procedures

The current study included adults who participated in the MTF longitudinal study in 2020 at ages 35–60 from the 12th grade classes of 1978, 1983, 1988, 1993, 1998, and 2003; total unweighted n = 14,379. As noted above, those who reported drug use at 12th grade (baseline) were oversampled, and weights were used to adjust for this sampling procedure.[1]

Experimental design

Participants who were eligible for a follow-up survey at ages 35 to 60 in 2020 were randomly selected on an equal basis to one of the two experimental conditions: (1) standard MTF or (2) web-push.[2] Respondent contact procedures for the two conditions are summarized in Table 1. In brief, participants in the standard MTF condition were sent up to three paper questionnaires with no mention of the potential to complete the survey via web. Participants in the web-push condition were sent login credentials to complete the survey online in all reminder mailings, and non-responders in the web-push condition were also sent up to two paper questionnaires and a text message. Of all age 35–60 participants, 32.8% had an email address on file just prior to the first day of the study; half of those with email addresses (52%) were in the web-push condition. Participants with emails in the web-push condition received up to 13 reminder emails (as soon as participants submitted a survey, no further mail/email/text contact was made). Regarding the text message, on the day of text message delivery, a telephone number with permission to text was available for 154 web-push participants who were sent the relevant communication. It is important to note that across conditions, minimal changes were made to survey layout, text of communications, and survey content in order to not confound differences in communication with the survey and invitation modes. Paper and web survey items had identical question wording by age; the total number of questions per survey ranged from 121 (at age 60) to 131 (at age 50). Median completion time for web surveys ranged from 37.2 minutes (at age 40) to 44.1 minutes (at age 60).

Table 1

Participant contact procedures for MTF standard and web-push conditions

Date	Form of contact
December 2019	StandardconditionIntroductionmailing Introduction letter announcing mailed survey in April and future $20 incentive Newsletter containing selected summary results from the MTF study in an informational format Change of address card for the respondent to update contact information
March 31, 2020 (Day 1) [a]	First survey mail contact Mailed paper questionnaire (Invitation Letter printed on questionnaire) Pencil and pre-paid return envelope $20 unconditional incentive check in respondent’s name
Day 7	Reminder postcard requesting completion of paper survey
Day 51	Begin non-response calling [b]
Day 56	Reminder mailing including second paper questionnaire
Day 140	Final mailing including third paper questionnaire
December 2019	Web-pushconditionIntroductionmailing Introduction letter announcing web-based survey in April and future $20 incentive Newsletter containing selected summary results from the MTF study in an informational format Change of address card for the respondent to update contact information
March 31, 2020 (Day 1) [a]	First survey mail contact Invitation Letter including web survey login information: survey uniform resource locator (URL), personal identification number (PIN), as well as QR code $20 unconditional incentive check in respondent’s name
Day 7	Reminder mailed letter with participant’s unique URL and PIN (those without email address)
Day 10	Email version of letter mailed Day 1 (those who had provided an email address [c])
Day 17	Email reminder 1 with URL and PIN
Day 36	Reminder postcard requesting completion of web survey
Day 42	SMS (short text message) with login information (those with mobile number who gave consent)
Day 51	Begin non-response calling [b]
Day 52	Email reminder 2 with URL and PIN
Day 56	Mailed reminder letter with URL and PIN plus paper questionnaire
Day 73	Email reminder 3 with URL and PIN
Day 94	Email reminder 4 with URL and PIN
Day 108	Email reminder 5 with URL and PIN
Day 122	Email reminder 6 with URL and PIN
Day 136	Email reminder 7 with URL and PIN
Day 140	Final mailed reminder letter with URL and PIN plus paper questionnaire
Day 150	Email reminder 8 with URL and PIN
Day 164	Email reminder 9 with URL and PIN
Day 178	Email reminder 10 with URL and PIN
Day 192	Email reminder 11 with URL and PIN
Day 199	Email reminder 12 with URL and PIN
Day 225	Email reminder 13 with URL and PIN

Due to COVID-19 and the imminent closure of our facility and U.S. Postal delivery problems, paper questionnaires and the first web-push survey contact were mailed on March 31, 2020, about ten days prior to the expected April study mailing date.

During non-response calling, available contact phone numbers were called up to four times.

All contact information (including email address) was provided by participants either during previous surveys or in response to mailings, including a change of address card.

Table 2 describes the characteristics of the sample by experimental condition (standard MTF or web-push). Only race/ethnicity showed significant differences by experimental condition. Additional analyses revealed that a higher percentage of White respondents were assigned to web-push than standard MTF (59.2% vs. 55.3%; p < .001), and a higher percentage of respondents who identified as Asian, multiracial, or another racial/ethnic group other than White, Black, or Hispanic were assigned to standard MTF than web-push (9.1% vs. 7.4%; p < .001). No significant differences were observed among those identifying as Black or Hispanic.

Table 2

Baseline (age 18) sample characteristics, by condition

	Standard MTF		Web-Push Condition
	%	(SE)	%	(SE)	p a
Follow-up age group					0.203
35 (n = 2427)	19.8	(1.91)	18.9	(1.84)
40 (n = 2429)	20.6	(1.91)	20.1	(1.88)
45 (n = 2418)	21.3	(2.05)	20.2	(1.86)
50 (n = 2418)	14.9	(1.51)	15.9	(1.54)
55 (n = 2366)	12.8	(1.26)	13.4	(1.34)
60 (n = 2321)	10.7	(1.12)	11.5	(1.16)
Sex (n = 14,379)					0.272
Female	43.6	(0.69)	44.7	(0.71)
Male	56.4	(0.69)	55.3	(0.71)
Race/ethnicity (n = 14,214)					0.002
White	55.3	(1.71)	59.2	(1.66)
Black	22.2	(1.47)	21.8	(1.59)
Hispanic	13.4	(1.41)	11.7	(1.04)
Other	9.1	(0.55)	7.4	(0.46)
Parent education (n = 13,953)			0.173
Some college or less	40.2	(1.09)	41.6	(1.05)
College/more	59.9	(1.09)	58.4	(1.05)
Will graduate from 4-year college (n=13,579)			0.807
Not definitely	58.3	(0.93)	58.5	(0.93)
Definitely	41.7	(0.93)	41.5	(0.93)
Any lifetime substance use
Alcohol (n=13,767)	86.6	(0.67)	85.8	(0.61)	0.315
Cigarettes (n=14,195)	66.5	(0.85)	66.4	(0.92)	0.908
Marijuana (n=14,062)	52.7	(0.99)	52.2	(0.96)	0.653
Other illicit drugs (n=13,967)	34.8	(0.82)	34.5	(0.82)	0.731

Note. Total unweighted n=14,379. Bold font indicates significance with p < 0.05.

SE = standard error

p-values from Rao-Scott chi-square tests.

Measures

Overall response and response mode (at 2020 follow-up)

The primary outcomes were (1) whether participants responded (yes, no) at the 2020 follow-up data collection, and (2) participants’ response mode (paper, web) for those randomized to the web-push condition. In regards to the overall response rate, the numerator included all respondents who submitted a paper or web survey (including partially completed surveys) with the exception of (a) respondents who provided inconsistent responses to four or more of the drug triplet (lifetime, 12-month, and 30-day use) measures, or (b) respondents who had someone other than themselves (i.e., the original 12th grade respondent) fill out their follow-up survey (this is usually discovered by data collection during correspondence with respondents). The denominator of the response rate included all base year respondents selected for longitudinal follow-up from the relevant 12th grade cohorts, excluding individuals reported as deceased or respondents identified later as having been foreign exchange students when surveyed in the 12th grade.

Baseline characteristics (12th grade, age 18)

Characteristics measured at baseline included gender (male, female), race/ethnicity (White, Black, Hispanic, or Other), parent education (coded as at least one parent had some college education or more vs. high school education or less), and four-year college plans (would “definitely” graduate from a four-year college program vs. probably will, probably will not, or definitely will not). Finally, four dichotomous any/none lifetime substance use measures indicated whether participants had ever used alcohol, cigarettes, marijuana, or other illicit drugs.

Concurrent characteristics (at 2020 follow-up)

Age at follow-up was coded as 35, 40, 45, 50, 55, or 60. Highest education achieved indicated whether the participant reported having a college degree versus some college or less. The question regarding highest education achieved was asked at ages 35 to 50, but not at ages 55 or 60; we took the maximum of the highest education achieved reported at all available previous waves. Employment indicated having a job (full-time, part-time, or two or more different jobs) versus not having a job (no outside jobs or paid employment, laid-off or waiting to start a job, or retired) during the 2020 data collection. Marital status indicated married versus not married (engaged, separated, divorced, widowed, never married/single). Lastly, three measures of substance use in the past 30 days indicated any use of alcohol, cigarettes, or marijuana.

Analytical approach

Analyses consisted of both cross-tabulations and logistic regressions using the complex survey design function to account for the oversampling of drug users and the complex survey design using SAS 9.4 PROC SURVEYFREQ and PROC SURVEYLOGISTIC procedures. Descriptive analyses compared follow-up age and baseline characteristics across conditions. For all analyses noted below, either pairwise or listwise deletion was used to address missing data.

Analyses for RQ1, RQ2, and RQ3 involved all cases assigned to an experimental condition. For RQ1, bivariate cross-tabulations compared overall response rates by experimental condition (standard MTF vs. web-push). For RQ2, initial bivariate cross-tabulations compared overall response rates by age at follow-up and baseline characteristics. Then, multivariable logistic regression was used to regress overall response on experimental condition, age at follow-up, and baseline characteristics (Model 1). Then, the regression was repeated, including the previously noted variables plus interactions terms between experimental condition and covariates (Model 2). Interactions were multiplicative terms, dummy variables were compared to the reference group, and PROC MULTITEST Benjamini-Hochberg test with a false discovery rate of 5% was used to adjust for multiple testing on interaction terms.

For RQ3, the prevalence of each current substance use measure by experimental condition was estimated. Then, two logistic regression models were fit for each substance use outcome. The first model only included experimental condition, whereas the second model included experimental condition, race/ethnicity, sex, parents’ education, college plans, highest degree at follow-up, and current employment status at follow-up.

Analyses for RQ4 and RQ5 were limited to respondents randomized to the web-push condition. For RQ4, the likelihood of responding via web (vs. paper) was examined using bivariate and multivariable logistic regression. Additional descriptive analyses were conducted to examine the different ways respondents accessed the web survey (e.g., typing in the URL from mailed communications, clicking on URL link in email communications, scanning QR code from invitation letter). For RQ5, the prevalence of each substance use measure by web response (vs. paper) was estimated and followed by bivariate and multivariable logistic regression models.

Descriptive analyses comparing follow-up age and baseline characteristics across experimental conditions, as well as analyses for RQ1, incorporated the MTF weight accounting for oversampling of drug users into the follow-up sample. Analyses for RQ2, RQ3, RQ4, and RQ5 incorporated age-specific MTF attrition weights that accounted for sample loss since the 12th grade baseline as well as oversampling of drug users.

Results

RQ1: overall response rates

Results for overall response rates by condition showed that the response rate for the web-push condition (42.1% [95% confidence interval (CI) = 40.7, 43.5]) was slightly higher than the standard MTF condition (40.4% [38.9, 41.9]). However, this difference was not statistically significant, with design-based F = 3.52; p = 0.061. Fig. 1 shows cumulative response by Study Day for the standard and web-push conditions. Overall, 87.6% of responses were submitted by Study Day 90; 93.9% of submitted responses were received by Study Day 120.

Fig. 1.

Cumulative response for MTF standard and web-push conditions

Notes: “x” Indicates contact (mailing, emailing, or SMS) on noted study day number (see Table 1). Solid grey line indicates non-response calling days.

RQ2: associations between sociodemographic characteristics, experimental condition, and response

Supplemental Table 1 presents bivariate associations between covariates and experimental condition (standard MTF vs. web-push) response rates. Some significant differences were observed, including that response likelihood was higher for those age 35 assigned to the web-push versus standard MTF condition (Fig. 2). In multivariable models examining the likelihood of overall response in 2020 controlling for covariates (but not including interactions), the web-push condition did not significantly change the odds of responding compared to the standard MTF condition (Model 1, Table 3). Compared to the age 60 group, all other follow-up ages had lower odds of responding. Males had lower odds than females of responding. Black, Hispanic, and other racial/ethnic participants had lower odds of responding relative to white participants. Having at least one parent with some college education and definite plans to graduate from a 4-year college in 12th grade were both positively associated with responding. Participants with any lifetime cigarette, marijuana, or other illicit drug use at baseline were less likely to respond (the relationship between lifetime alcohol use and response was non-significant). Results for Model 2 in Table 3 show associations with the inclusion of experimental condition by covariate interaction terms. No significant interactions were observed, indicating that the effect of experimental condition on response rates did not differ significantly by age or other socio-demographic characteristics.

Fig. 2.

Response rates in 2020 by experimental condition for each age group

Note: ** = difference between Standard MTF and web-push response rates was statistically significant (p < .01) within noted age group.

Table 3

Multiple logistic regression predicting any response (1) versus no response (0) based on experimental condition and baseline characteristics.

	Model 1 AOR (95% CI) p	Model 2 AOR (95% CI) p
Experimental condition (vs. standard MTF)
Web-push	1.05 (0.97, 1.14) 0.225	0.93 (0.65, 1.34) 0.370
Follow-up age (vs. 60)
35	0.43 (0.37, 0.50) <.001	0.36 (0.22, 0.60) <.001
40	0.42 (0.36, 0.50) <.001	0.52 (0.31, 0.87) 0.014
45	0.45 (0.38, 0.53) <.001	0.48 (0.29, 0.80) 0.005
50	0.63 (0.53, 0.74) <.001	0.75 (0.45, 1.25) 0.266
55	0.76 (0.65, 0.90) 0.001	1.03 (0.63, 1.70) 0.900
Sex (vs. Female)
Male	0.66 (0.61, 0.72) <.001	0.53 (0.41, 0.69) <.001
Race/ethnicity (vs. White)
Black	0.32 (0.28, 0.38) <.001	0.35 (0.21, 0.56) <.001
Hispanic	0.56 (0.47, 0.67) <.001	0.67 (0.39, 1.16) 0.151
Other	0.58 (0.49, 0.69) <.001	0.72 (0.41, 1.25) 0.248
Parent education (vs. less than college)
College or more	1.16 (1.07, 1.26) <.001	1.24 (0.94, 1.64) 0.131
Will graduate from 4-year college (vs. not definitely)
Definitely	1.39 (1.27, 1.51) <.001	1.13 (0.86, 1.47) 0.381
Any lifetime substance use (age 18)
Any alcohol (vs. none)	0.92 (0.81, 1.05) 0.225	0.65 (0.42, 1.01) 0.057
Any cigarettes (vs. none)	0.85 (0.77, 0.95) 0.004	1.04 (0.75, 1.44) 0.825
Any marijuana (vs. none)	0.72 (0.65, 0.80) <.001	0.69 (0.50, 0.96) 0.028
Any other illicit drugs (vs. none)	0.86 (0.78, 0.95) 0.003	0.98 (0.72, 1.33) 0.903
Web-push × follow-up age
× Age 35		1.13 (0.82, 1.51) 0.659
× Age 40		0.87 (0.64, 1.19) 0.659
× Age 45		0.96 (0.71, 1.30) 0.795
× Age 50		0.89 (0.66, 1.21) 0.659
× Age 55		0.82 (0.61, 1.10) 0.656
Web-push × male		1.16 (0.98, 1.36) 0.514
Web-push × race/ethnicity
× Black		0.95 (0.70, 1.30) 0.795
× Hispanic		0.88 (0.62, 1.25) 0.659
× Other		0.86 (0.60, 1.24) 0.659
Web-push × parental education		0.96 (0.80, 1.14) 0.772
Web-push × college plans		1.15 (0.97, 1.36) 0.514
Web-push × alcohol use		1.25 (0.96, 1.64) 0.514
Web-push × cigarette use		0.88 (0.72, 1.08) 0.656
Web-push × marijuana use		1.03 (0.84, 1.26) 0.795
Web-push × illicit drug use		0.92 (0.76, 1.11) 0.659

Note. Unweighted n = 12,242; Model 1 does not include interaction terms. p-values from logistic regression models for interaction terms account for multiple testing. MTF = Monitoring the Future; CI = confidence interval; AOR = adjusted odds ratio. Bold font indicates significance with p < 0.05.

RQ3: associations between experimental condition and substance use at follow-up

Current substance use prevalence estimates at follow-up by experimental condition are presented in Table 4, as well as bivariate and multivariable logistic regression associations. In bivariate models, past 30-day alcohol prevalence differed significantly across conditions (web-push = 68.3% vs. standard MTF = 65.3%; p = .047), but the difference became non-significant after adjusting for socio-demographic characteristics. There were no significant differences in past 30-day cigarette or marijuana use across experimental conditions.

Table 4

Associations between past 30-day substance use prevalence at 2020 follow-up by experimental condition

	%(SE)	p [a]	Model 1(bivariate) OR (95% CI) p [c]	Model 2(multivariable [b]) AOR (95% CI) p [c]
Alcohol use		0.047
Standard MTF	65.3 (1.26)		(ref)	(ref)
Web-push	68.3 (1.04)		1.15 (1.00, 1.32) 0.048	1.07 (0.93, 1.23) 0.333
Cigarettes		0.116
Standard MTF	10.5 (0.70)		(ref)	(ref)
Web-push	12.0 (0.67)		1.17 (0.96, 1.42) 0.116	1.06 (0.86, 1.32) 0.567
Marijuana		0.431
Standard MTF	12.2 (0.69)		(ref)	(ref)
Web-push	13.0 (0.72)		1.07 (0.90, 1.28) 0.432	1.11 (0.92, 1.34) 0.278

Notes. Unweighted ns: alcohol use 5594 (Model 1), 5033 (Model 2); cigarette use 5621 (Model 1), 5073 (Model 2); marijuana use 5621 (Model 1), 5058 (Model 2). Bold font indicates significance with p < 0.05. Sociodemographic variables included race/ethnicity, sex, parents’ education, college plans, highest degree at follow-up, marital status at follow-up, and current employment status at follow-up. SE = standard error; OR = odds ratio; AOR = adjusted odds ratio; CI = confidence interval.

p-value from design-based F tests.

Multivariable models for each substance adjusted for sociodemographic characteristics.

p-value from logistic regression models.

RQ4: mode of response among web-push participants

Among participants randomized to the web-push condition, significantly more participants responded via web (83.0%) than paper (17.0%). Supplemental Table 2 presents bivariate associations between covariates and web versus paper response mode among those randomized to the web-push condition. Significant differences by age group, race/ethnicity, parental education, college plans, and 12th grade substance use were observed. When both baseline and concurrent characteristics were included in a multiple logistic regression model (see Table 5), race/ethnicity, marital status, highest education achieved, and past 30-day cigarette use were significantly associated with response mode. Specifically, Black participants (compared to White participants) and those reporting past 30-day cigarette use were less likely to respond via web. In contrast, being married and having a college degree were associated with being more likely to respond via web.

Table 5

Multiple logistic regression predicting web response (1) versus paper response (0) based on baseline and concurrent characteristics: Participants randomized to web-push condition

	AOR (95% CI) p
Follow-up age (vs. 60)
Age 35	1.08 (0.67, 1.74) 0.772
Age 40	1.09 (0.69, 1.73) 0.738
Age 45	1.02 (0.64, 1.62) 0.923
Age 50	1.11 (0.71, 1.72) 0.650
Age 55	0.92 (0.60, 1.40) 0.409
Male (vs. Female)	1.16 (0.87, 1.54) 0.306
Race/ethnicity (vs. White)
Black	0.49 (0.30, 0.79) 0.004
Hispanic	0.96 (0.51, 1.79) 0.893
Other	1.27 (0.63, 2.56) 0.503
Parent some college education (vs. less)	1.26 (0.92, 1.71) 0.149
Will definitely graduate from 4-year college (vs. not definitely)	1.23 (0.91, 1.66) 0.173
Any lifetime substance use at age 18
Any alcohol use (vs. none)	1.05 (0.63, 1.75) 0.863
Any cigarette use (vs. none)	1.15 (0.78, 1.68) 0.481
Any marijuana use (vs. none)	0.87 (0.57, 1.33) 0.527
Any other illicit drug use (vs. none)	0.71 (0.50, 1.02) 0.065
Characteristics at 2020 follow-up
Married (vs. not married)	1.35 (1.01, 1.80) 0.044
Some college or more (vs. <=high school degree)	1.84 (1.39, 2.44) <.001
Employed (vs. not employed)	0.81 (0.58, 1.14) 0.225
Any past 30-day alcohol use (vs. none)	1.11 (0.83, 1.48) 0.493
Any past 30-day cigarette use (vs. none)	0.44 (0.30, 0.64) <.001
Any past 30-day marijuana use (vs. none)	0.92 (0.63, 1.34) 0.660

Note. Unweighted n=2,506. Bold font indicates significance with p < 0.05.

Simple unweighted analyses also were conducted to examine web survey access mode. Results (see Table 6) showed that, across age groups, the most common mode of accessing the web survey was via typing in the URL included on mailed communications (ranging from 51.6% to 67.7% of access occasions by age), followed by clicking on the study URL in email communications (ranging from 22.8% to 38.3% of access occasions by age). Email access prevalence was higher among respondents ages 35–40 versus 45–60 (p < .001), while mail access prevalence was higher among respondents ages 45–60 (p < .001). Access via scanning a QR code from mailed correspondence ranged from 6.2% to 10.6% of access occasions by age, and SMS access occasions ranged from 0.4% to 3.3% by age; there were no significant differences by age for QR or SMS survey access. Only one access occasion occurred as a result of obtaining a URL directly from study staff during a reminder phone call; no survey data were collected by telephone.

Table 6

Mode of access for web survey among respondents ages 35–60 during 2020 data collection

Follow-up age	Mode of access [a]	n of access occasions	% of access occasions	Total access occasions
35	Mail	333	56.1%
	Email	188	31.6%
	QR	53	8.9%
	SMS	19	3.2%
	Phone call	1	0.2%
				594
40	Mail	244	51.6%
	Email	181	38.3%
	QR	41	8.7%
	SMS	7	1.5%
				473
45	Mail	356	65.3%
	Email	134	24.6%
	QR	53	9.7%
	SMS	2	0.4%
				545
50	Mail	444	66.2%
	Email	162	24.1%
	QR	61	9.1%
	SMS	4	0.6%
				671
55	Mail	339	58.8%
	Email	170	29.5%
	QR	61	10.6%
	SMS	7	1.2%
				577
60	Mail	478	67.7%
	Email	161	22.8%
	QR	44	6.2%
	SMS	23	3.3%
				706

Notes: Data above represent a total of 3,566 web survey access occasions for 2,447 individuals.

Indicates the source of the link used to access the web survey. Mail = typed link on any form of hard copy communication. E-mail = link in any email communication. QR = quick response code from hard copy mailed communications. SMS = text message with direct survey link. Phone call = link provided verbally during reminder call.

RQ5: substance use differences at follow-up among web-push participants

Among those randomized to the web-push condition, estimates of past 30-day substance use at follow-up by response mode are shown in Table 7. We found no evidence that response mode was associated with differences in past 30-day alcohol or marijuana prevalence. Response mode was associated with past 30-day cigarette prevalence: 24.6% among those responding via paper versus 9.6% among those responding via web (p < .001). The observed association remained significant after adjusting for baseline and concurrent covariates.

Table 7

Associations between past 30-day substance use prevalence at 2020 follow-up by mode of response in web-push condition

	%(SE)	p [a]	Model 1(bivariate)OR (95% CI) p [c]	Model 2(multivariable[b])AOR (95% CI) p [c]
Alcohol use		0.185
Paper	65.2 (2.74)		(ref)	(ref)
Web	68.9 (2.44)		1.18 (0.92, 1.52) 0.186	0.98 (0.74, 1.29) 0.868
Cigarettes		<.001
Paper	24.6 (2.44)		(ref)	(ref)
Web	9.6 (0.65)		0.33 (0.24, 0.44) <.001	0.44 (0.31, 0.67) <.001 [d]
Marijuana		0.089
Paper	15.7 (1.93)		(ref)	(ref)
Web	12.4 (0.76)		0.76 (0.56, 1.04) 0.089	0.81 (0.58, 1.14) 0.224

Notes. Unweighted ns: alcohol use 2905 (Model 1), 2622 (Model 2); cigarette use 2905 (Model 1), 2679 (Model 2); marijuana use 2886 (Model 1), 2265 (Model 2). Bold font indicates significance with p < 0.05. Sociodemographic variables included race/ethnicity, sex, parents’ education, college plans, marital status, highest degree at follow-up, and current employment status at follow-up. SE = standard error; OR = odds ratio; AOR = adjusted odds ratio; CI = confidence interval.

p-value from design-based F tests.

Multivariable models for each substance adjusted for sociodemographic characteristics.

p-value from logistic regression models.

For cigarette use, Model 2 also included follow up age, concurrent substance use past 30 days, region, population density, and participant’s political beliefs.

Discussion

Results indicated no significant difference in overall response rate for the web-push vs. standard MTF conditions at ages 35 to 60, and sociodemographic characteristics did not significantly interact with experimental condition on the odds of survey response in multivariate models. Differences in past 30-day substance use prevalence estimates at follow-up by condition were not significant after adjusting for sociodemographic characteristics and attrition (via the use of attrition weights). Participants in the web-push condition were less likely to respond via web (vs. paper) if they were Black, had smoked cigarettes in the past 30 days, were unmarried, or did not have a college degree. The web-push approach appears to provide potential increases in response for specific sub-groups, while also ensuring that subgroups with a higher likelihood of response using paper have the opportunity to choose the mode that best suits their needs. Of primary importance for the MTF survey, estimates of substance use appear to be robust across modes after adjusting for sociodemographic characteristics and attrition.

Our results support the use of web-push data collection efforts among mid- and older adult general population subgroups. Prior analyses using web-only data collection among older adults have shown lower response rates than paper-only surveys (e.g., Bech & Kristensen, 2009; Daikeler et al., 2020). Studies exploring the utility of mixed-mode web-push approaches versus mailed paper surveys have found equal (Delnevo & Singh, 2021) or possibly higher response rates for web-push (Millar et al., 2018; Kelfve et al., 2020). The likelihood of responding via web has been found to decrease with respondent age when compared with responding via paper (de Bernardo & Curtis, 2012; Kelfve et al., 2020; Tassiopoulos et al., 2020; Delnevo & Singh, 2021) as well as via other modes such as personal interviews (Lynn, 2020). Our results showed that a web-push approach provided comparative response rates across ages 35 to 60 in a general U.S. population survey, with no significant interactions between age and experimental condition. In the current study, the only age differences in response rates between conditions were observed in bivariate analyses at age 35, where response was higher for those assigned to the web-push versus standard MTF condition.

The current study was able to take advantage of the fact that, due to the structure of the MTF panel study methodology, all longitudinal participants had participated in at least the 12th grade MTF survey and likely one or more longitudinal follow-up data collections. Thus, there was familiarity with the study. Further, the study was able to utilize email contact for respondents who had previously provided this information. By using email augmentation of mailed survey invitations, respondents are able to more easily access the survey due to electronic links to the survey (Dillman, 2017).

Research has shown that trying to identify a single mode for survey implementation is likely to result in lowered coverage and response, as well as higher nonresponse error (Dillman, 2017). Our research shows that providing a variety of ways to access the web survey also may be important for surveys targeting both mid- and older adults. While Dillman (2017) found that email augmentation resulted in easier survey access, our results indicated significant age differences in such associations. Email survey access was more prevalent than mail access for those ages 35–40, while the opposite was true for those ages 45–60.

The current study found that participants in the web-push condition who chose to respond using paper surveys were a small but distinct and important group. Specifically, these individuals were more likely to be Black participants (versus White), unmarried, without a college degree, and to have used cigarettes in the past 30 days. Prior studies have found similar results in regards to education status and living situation (Kelfve et al., 2020; Messer & Dillman, 2011; Sterrett et al., 2017). The implication of the current results is that solely relying on a web-based survey may result in underrepresenting certain segments of the population. Offering a paper version for respondents who cannot or prefer not to respond via the web appears to be beneficial for representativeness of the sample.

The results of the current study raise several important methodological questions for future research on web-push methodology. The current study was not able to clearly determine how many or which types of communications were associated most strongly with response in the web-push condition. Mailings sent via the U.S. Postal Service take different amounts of time to arrive, and we were simultaneously sending digital communications. We cannot be sure which form of contact, or combination of contacts, actually prompted a respondent to participate, nor can we rigorously evaluate the optimal timing of such communications. Future research should also consider whether sending the paper survey earlier would increase response rates, whether a single paper mailing would be as effective as the two used in the current study, and whether the total data collection window could be shortened without decrement to the overall response rate. As shown in Fig. 1, the vast majority of respondents responded during the first half of the data collection window. Prior research has indicated that use of time-limited incentives (which provide an added incentive if response is obtained within a certain time frame) may help improve early response in web-push studies (Peycheva et al., 2022). Further, the current study’s results should not be used to compare the speed of response for the web-push versus standard conditions. The initial mailing for the web-push condition utilized a regular business envelope. In contrast, the initial mailing for the standard condition was processed as a parcel (in a 9”×12” envelope that included a pencil and an envelope to return the completed paper survey using Business Reply Mail). Future studies designed to compare the overall timeline of web-push versus paper studies are needed.

The findings in this study are subject to limitations. The MTF longitudinal panel study is based on the 12th grade sample; individuals who dropped out of school prior to 12th grade were not represented. In addition to lower educational level, school dropout is associated with higher substance use (Tice et al., 2017), lower socioeconomic status, and Black or Hispanic identity (Dunham & Wilson, 2007). The current study found that a preference for responding via paper (vs. web) was associated with some of these characteristics; thus, the results presented here may underestimate the likelihood of responding via paper for the population overall. The implementation of the web-push experiment among respondents ages 35–60 was planned in 2019, but data collection occurred from March through November 2020—during the early months of the COVID-19 epidemic. The degree to which results may have been impacted by the pandemic cannot be assessed. However, results and conclusions are in line with those observed from the earlier web-push experiments among young adults (Patrick et al, 2021).

Conclusions

The current study was able to compare overall response rates between randomized paper and web-push survey modes, as well as response format (web vs. paper) within web-push respondents among national samples of adults from ages 35–60, representing 12th grade cohorts from 1978, 1983, 1988, 1993, 1998, and 2003. Results indicate that, overall, the move to the web-push design had minimal impact on response rates and substance use prevalence estimates for this age group. Web-push methodology with an option to respond via paper appears to be a strong and viable mode for future longitudinal data collection efforts among general population adult samples.