The effects of electronic alert letters for internet surveys of academic scientists.

Survey alert letters improve response rates and assure potential respondents that the research is legitimate and of high quality. Pre-notification by mail increases response rates for web surveys because it represents a second mode of communication and contributes to increases in respondent trust and study legitimacy. Due to work-from-home orders in response to COVID-19, postal alert letters are unlikely to reach research participants at their place of employment. We conducted three experiments testing the effects of sending academic scientists a pre-notification email message on web survey response rates as compared to no alert email message and variation in the timing of the pre-notification. The data comes from three random national samples of university-based scientists that were conducted during the early months of the COVID-19 pandemic. Two of three experiments suggest that email alert pre-notifications can have a minor effect on improving response rates to web surveys of academic scientists. The timing of those pre-notification messages, though, had no effect on survey response. These findings indicate pre-notification messages remain useful when studying academic scientists. Future research should compare the effects of electronic as compared to postal pre-notification on survey response among scientists, as postal pre-notification requires extensive resources. © Akadémiai Kiadó, Budapest, Hungary 2021.

Introduction

Survey research is a common tool used to collect data about public and expert opinion across social science disciplines. Well-designed surveys enable researchers to test theories about population perceptions and behavior. Surveys are often used to study academic scientists and their collaboration patterns and publication behaviors (Lopex-Navarro et al., 2015; Tsai et al., 2016; Youtie et al., 2014). As with all research methods, there are weaknesses and limitations with survey design and data collection. Survey administration varies for paper-and-pencil, face-to-face, telephone, and web formats, but many common practices hold across survey modes including formatting and labeling to signal the reputation of those conducting the survey; pre-notification of survey invitations; clear human subjects statements and specification of confidentiality and data use; mixed method contacts; and follow up communications to improve response rates and reduce nonresponse bias. The effectiveness of these various efforts is always under study (c.f., Dillman et al., 2014; Marsden & Wright, 2010; Wolf et al., 2016) .

Surveys of academic scientists provide a unique set of opportunities and challenges, as compared to surveys of the general public. First, academic scientists are much more likely to recognize the affiliation of the researchers and funding sources. Second, they are more likely to quickly understand and assess their rights as human research subjects and understand the low risk associated with completing a survey. Third, in the case of web surveys, academic researchers are more likely to have access to and literacy for digital technologies. And in our experience over the past 20 years, surveying academic scientists about program implementation, research funding, productivity, research networks, and university administration, they are more likely to participate in surveys—likely because of a sense of obligation to the funding agency, feelings of reciprocity toward other researchers, or a desire to share their work experiences. Given academic scientists’ higher than average response rates to surveys and fewer digital divide issues, they are an ideal population for testing web survey implementation methods.

Best practice in survey administration aims to reduce common sources of survey error including sampling, coverage, nonresponse, measurement, and processing error (Groves, 1989). A pre-notification letter to alert sampled individuals that they will be asked to participate in a study is an important mechanism used to improve response rates, increase legitimacy of research, and assure potential respondents that the research is high quality. Research consistently shows that alert letters improve response rates (Dillman et al., 2014; Tourangeau et al., 2013). In the case of web surveys, mailing alert letters on official letterhead has the added bonus of offering a mixed mode contact, which is also associated with increased response rates (Sue & Ritter, 2012), and may be particularly valuable when conducting research with professional groups such as scientists.

The ASU Center for Science, Technology, and Environmental Policy Studies (CSTEPS) regularly implements web surveys with mailed alert letters (Kim et al., 2017; Siciliano et al., 2018) and thus was interested in understanding if there would be an effect on survey response rates if we shifted to email alert letters or discontinued alert letters altogether. In spring 2020, with the work-from-home orders in response to COVID-19, postal alert letters were unlikely to reach research participants. While research indicates that mailed alert letters produce higher response rates than no alert letter (Bandilla et al., 2012; Kaplowitz et al., 2004), there is little research to indicate whether an alert letter sent by email would have the same intended effect as an alert letter mailed by traditional post. The research team decided to collect data on the effects of using an alert email or not. We report results from three random experiments designed to investigate the effects of receiving an alert email on web survey response rates in a sample of university-based scientists. We test the effects of receiving an alert email as compared to no alert email and variation based on the timing of the alert email.

Literature and hypotheses

The use of surveys that are completely electronic is the fastest growing form of survey research across the globe and related research continues to emerge as technology evolves (Dillman et al., 2014; Sue & Ritter, 2012). There are many elements of best practice for web surveys that are meant to improve the quality of web surveys, reduce sources of error (e.g. coverage, nonresponse, measurement) and increase response rates. Software programs ensure that the templates are flexible and easy-to-use, that only the investigators have control over the data and that costs are minimized (Dillman et al., 2014). Considerable research shows that making multiple contacts improves the chances persons who are sampled will agree to participate (Dillman et al., 2014).

Alert letters are pre-notice informing sampled individuals of the intent and date of the upcoming survey. Mailing hard copies of an alert letter prior to a web survey establishes the legitimacy of the project, primes the recipients for the survey’s arrival and enables an additional, mixed mode contact for respondents (Sue & Ritter, 2012). Kaplowitz et al. (2004) demonstrated that using postcard pre-notifications in a survey of college students produced a significantly higher response rate than no prenotification. Sending an alert letter via mail produced a significantly higher response rate compared to sending the letter through email (Crawford et al., 2004). The mailed pre-notification letter offers not only the opportunity to alert sampled individuals that the email invitation is coming but can serve as a secondary connection with the respondent should the initial email invitation get sent to a spam folder or blocked by a server.

While alert letters have many benefits, they are costly. First, there is the expense of paying postage. Producing the alert letter additionally includes the cost of letterhead, envelopes, printing, and the labor to print, fold, and stuff envelopes. Another, often underestimated cost to mailing alert letters is the labor required to acquire complete mailing addresses, which is much easier with small, known samples as compared to large random samples. For example, surveys within organizations where letters can be distributed through payroll or office mailboxes, surveys of students with addresses as part of their records, or surveys of homeowners where addresses are public records are more easily processed. If one of the primary reasons for the increase in web surveys is convenience and lower costs, mailing alert letters adds substantial inconvenience and cost.

The use of online pre-notice or email alert letters is less common and the research on the mode of pre-notification is limited (Dillman, et al., 2014; Tourangeau et al., 2013). Bosnjak et al. (2008) compared short mobile-messaging service (SMS), email and no pre-notice in a college survey. The SMS pre-notice produced a higher response rate than the email and no pre-notice. One reason for the lower response rate associated with an email alert letter could be the failure to receive or read the messages due to the widespread use of spam filters (Tourangeau, 2013). Bandilla et al. (2012) found that while any mailed letter is more effective than email alone, the pre-notification only had an effect for the email invitation group—implying a mixed mode effect. Web survey guides advise pre-notifying potential respondents by internet before sending out the invitation in order to maximize response rates (Sue & Ritter, 2012).

Following survey best practice, in previous web surveys of academic scientists, we have sent a mail alert letter printed on official university letterhead including handwritten signatures and postal stamps (rather than scanned postal codes). These personalized efforts are shown to help surveyors establish a connection with respondents to increase response rates (Dillman et al., 2014). The mixed mode approach adds a postal mail letter that is not subject to spam filters, typos in email addresses, and other technical errors that can occur with web surveys. Alert letters might result in the potential respondent contacting the research team or going directly to the survey website and signals that the survey is run by experienced professionals (Callegaro et al., 2015). The mailed letter helps establish the legitimacy and authority of the research team, and ultimately increases the response rate when contacting first-time research participants (Vehovar et al., 2002). Due to COVID19 stay-at-home orders, this mode of pre-notification was temporarily unavailable as scientists were not likely to be at their university office addresses. This created an opportunity to test the effects of using an electronic alert letter as compared to no pre-notification contact.

Hypothesis 1

Receiving an email alert letter will positively affect response rates to a web survey, as compared to no pre-notification.

We also tested the effects of the timing of email pre-notifications. The literature suggests that the timing of the survey implementation process can affect the response rate, although an exact schedule is rarely given. According to Dillman et al. (2014), researchers should allow adequate time for the recipients to read the content, but not so much time that the requests are forgotten. Thus, the shorter amount of time may make it easier for respondents as they have enough time to learn about the survey, but not too much time to forget about it. We thus expect that response rates will differ based on the timing of the alert letter:

Hypothesis 2

There will be a significant difference in response rates based on the timing of the pre-notification email, with response greater for shorter intervals between pre-notification and survey invitation.

Material and methods

The experiment was conducted as part of a project that regularly implements online surveys of academic scientists. The sampling frame for the three surveys, which covered the topics of (1) the effects of COVID-19 on academic research, (2) US visa and immigration issues affecting the scientific community, and (3) general science policy questions, included all full-time faculty with PhDs in four fields of science—biology and genetics, civil and environmental engineering, biochemistry, and geography—at 81 randomly selected Carnegie designated Research Extensive and Intensive (R1) universities in the United States (US). For each survey, universities were randomly sampled from within eight stratified geographic regions in the US. For each sampled university, we visited the department websites and collected the name and contact information of tenured and tenure track faculty (assistant, associate, and full professors), research professors and instructors with PhDs in each field. The final samples included contact information for 1968, 2443, and 2436 scientists, respectively.

The experiment was designed to test the effects of receiving an electronic alert letter and the timing of the alert letter on response rates. There were three conditions:

Condition 1: No electronic pre-notification letter

Condition 2: Week notice electronic pre-notification letter

Condition 3: Days-notice electronic pre-notification letter

Individuals in the sampling frame for each survey were randomly assigned across the three conditions. Individuals within each field, for each survey, were assigned a random number using MS Excel. The randomly assigned numbers were then sorted lowest to highest. The first 1/3 were assigned to condition 1, the second 1/3 to condition 2, the third 1/3 to condition 3, within each field of science. Table 1 notes the random assignment of the sample by condition. The experiments were pre-registered at AsPredicted.1

Table 1

Random assignment of sample by field of science and condition

	Condition 1	Condition 2	Condition 3	Total
	No email alert	One week email alert	Two days email alert
Survey 1. COVID-19
Biology (and genetics)	400	401	401	1202
Engineering (civil & environmental)	174	174	175	523
Biochemistry	81	81	81	243
Survey 2. Visa & Immigration issues
Biology	589	589	589	1767
Engineering (civil & environmental)	155	155	154	464
Geography	71	71	70	212
Survey 3. Science Policy Questions
Biology	587	587	586	1760
Engineering (civil & environmental)	149	150	149	448
Geography	76	76	76	228

We verified that participants were randomly assigned to each condition ("Appendix 1"). An analysis of the condition allocation on the covariates (gender and geographic region) shows statistically significant differences in some of the regions. We control for this possible bias by including these covariates in our model specifications.

Electronic pre-notification letters were sent using the Yet Another Mail Merge (YAMM), a Gmail add-on for mail merges. The text of the alert letter is presented in "Appendix 2". The web survey was administered using Sawtooth® software. The electronic pre-notification letters were sent five days apart for all three surveys. The survey administration calendar is reported in Table 2.

Table 2

Survey administration calendar

	One week notification	Two days notification	Start date	End date	Days survey open
Survey 1. COVID-19	Apr 23 2020	Apr 28 2020	Apr 30 2020	Jun 8 2020	39
Survey 2. Visa & Immigration issues	Oct 15 2020	Oct 20 2020	Oct 22 & 23 2020	Nov 25 2020	35
Survey 3. Science Policy Questions	Oct 13 2020	Oct 18 2020	Oct 21 & 23 2020	Nov 23 2020	34

All study variables are binary. For our analyses, the dependent variable was the individual response to each survey (1 = yes, 0 = no). Using AAPOR (2016) response rate formula RR2, response rates were calculated as the percentage of complete responses to the survey compared to eligible respondents who did not complete the survey (e.g. excluding uncontactable and ineligible cases).

The three experimental conditions resulted in 67% of each sample receiving an email alert for all three surveys; with one-third of each sample receiving prenotification two days prior to the actual invitation, and one-third receiving prenotification one week prior. The control variables were gender, academic field and region in the US. Table 3 reports descriptive information for all study variables.

Table 3

Descriptive Statistics: study variables, by survey

	Variable	Survey 1: COVID19			Survey 2: Immigration			Survey 3: Science Policy
		N	%	Std.Dev	N	%	Std.Dev	N	%	Std.Dev
	Response to survey	1925	19	39%	2435	15	36%	2417	17	37%
Treatment Group	Two day alert	1925	35	40%	2435	35	47%	2417	34	47%
	One week alert	1925	35	39%	2435	34	50%	2417	38	50%
	No alert	1925	30	46%	2435	31	48%	2417	28	45%
Gender	Female	1968	31	46%	2443	32	47%	2436	32	47%
Field	Biology	1968	61	49%	2443	72	45%	2436	72	45%
	Geography	n/a	n/a	n/a	2443	9	28%	2436	9	29%
	Engineering	1968	27	44%	2443	19	39%	2436	18	39%
	Biochemistry	1968	12	33%	n/a	n/a	n/a	n/a	n/a	n/a
Region	Region 1	1968	9	28%	2443	9	28%	2436	8	27%
	Region 2	1968	16	36%	2443	16	37%	2436	16	36%
	Region 3	1968	5	21%	2443	10	31%	2436	11	32%
	Region 4	1968	9	29%	2443	9	29%	2436	11	31%
	Region 5	1968	30	46%	2443	32	47%	2436	32	47%
	Region 6	1968	8	27%	2443	8	27%	2436	7	25%
	Region 7	1968	3	17%	2443	4	20%	2436	4	19%
	Region 8	1968	20	40%	2443	12	33%	2436	12	33%
Position	Assistant	1968	24	43%	2443	21	41%	2436	19	39%
	Associate	1968	22	41%	2443	22	42%	2436	22	42%
	Full	1968	42	49%	2443	43	49%	2436	45	50%
	Non-tenure	1968	11	32%	2443	14	34%	2436	13	34%

Differences in survey response by experimental condition were examined for each survey using crosstabulations and chi-square tests. Analyses were performed separately to investigate each hypothesis. We then estimated logistic regression models for each survey to examine our hypotheses after adjusting for the potential effects of gender, academic field, and geographic region. The response variable = 1 if the participant responded to the survey and is = 0 otherwise.

Results

Response rates for the three surveys were as follows: COVID19 survey 19%; Visa and Immigration survey 15%; and Science Policy Questions survey 17%.

To examine hypothesis 1, we compared survey participation across persons assigned to receive any pre-notification vs. those assigned to the no pre-notification condition. Results of this analysis are presented in Table 4. Just under 20 percent of persons who received an alert email participated in survey one (19.6%) while 16.6 percent of those who did not receive pre-notification participated. For survey two, 15 percent of persons who received an alert email participated compared to 15.7 percent of those who did not receive the pre-notification. Eighteen percent of persons who received a pre-notification for survey three participated, while 14.1 percent of those who did not get an alert email participated in the survey.

Table 4

Tabulation comparing responses for those who received an alert email or not

	Received an alert letter or not
	Survey 1: COVID19			Survey 2: Immigration			Survey 3: Science Policy
Response	No	Yes	Total	No	Yes	Total	No	Yes	Total
No	534	1035	1569	686	1378	2064	688	1326	2014
	83%	80%		84%	85%		86%	82%
Yes	106	252	358	128	243	371	113	290	403
	17%	20%		16%	15%		14%	18%
Total	640	1287	1927	814	1621	2435	801	1616	2417
Pearson chi2(1)	2.574, Pr = 0.11			0.226, Pr = 0.63			2.679, Pr = 0.02
Yule’s Q	0.10			−0.03			0.14

Yule’s Q pooled for all three surveys = 0.07

Although there is an apparent trend, chi-square tests revealed no significant effect of pre-notification on survey participation for surveys one and two. However, survey three had a significant chi-square value indicating a difference in response rates for those who received a pre-notification compared to those who did not (chi2 = 2.679, Pr = 0.017). This lends support to hypothesis 1, an association between pre-notification and increased survey response rate. We also obtained the appropriate effect sizes, Yule’s Q, for each survey and the pooled results. The results indicate the effects, where there are any, are relatively modest.

We next investigated whether response rates varied based on the timeliness of receiving an email alert letter. To test hypothesis 2, we compared the responses of the subsample randomly assigned to receive an electronic prenotification a week in advance with responses of the subsample assigned to receive an electronic prenotification with a few days’ notice. The results comparing prenotification timing and survey response are presented in Table 5. We found no significant difference in response rates between those receiving a week early alert email (response rate 19.8%, 14.3%, 18.8%, respectively, for surveys 1–3) and those receiving a few days’ prenotification (response rate 19.4%, 15.7%, 18.8%, respectively).

Table 5

Tabulation comparing responses by prenotification timing

Response	Prenotification: Week notice or a few days’ notice
	Survey 1: COVID19			Survey 2: Immigration			Survey 3: Science Policy
	One Week	Two Days	Total	One Week	Two Days	Total	One Week	Two Days	Total
No	521	514	103	684	694	1378	670	656	1326
	81%	80%		84%	85%		83%	81%
Yes	125	127	252	126	116	243	138	152	290
	19%	20%		16%	14%		17%	19%
Total	641	646	1987	810	811	1621	808	808	1616
Pearson chi2(1)	0.044, Pr = 0.83			0.57, Pr = 0.45			0.824, Pr = 0.36
Yule’s Q	0.02			−0.05			0.06

Yule’s Q pooled for all three surveys = 0.01

We then estimated logit regression models predicting survey response by experimental condition while controlling for gender, academic field, rank and region of the country. The results of these models are presented in Tables 6 and 7. The first set of models, Table 6, predict likelihood of sampled persons responding to the survey when receiving any prenotification alert email. The second set of models, Table 7, predict the likelihood of responding to the survey depending on the timing of the prenotification.

Table 6

Logit Regression predicting response to hypothesis 1: Alert email (odds ratios)

	Survey 1	Survey 2	Survey 3	Pooled
Alert email (Hypothesis 1)	1.24*	0.94	1.31**	1.16**
	(0.96–1.59)	(0.74–1.19)	(1.03–1.66)	1.01–1.33)
Female	1.27*	1.15	1.32**	1.25***
	(0.99–1.63)	(0.90–1.47)	(1.05–1.67)	(1.08–1.43)
Biology	1.44**	1.32*	1	1.23**
	(1.08- 1.93)	(0.96- 1.81)	(0.75- 1.33)	(1.03–1.45)
Biochemistry	0.95			0.88
	(0.61–1.49)			0.59–1.33)
Geography		2.57***	1.18	1.73***
		(1.66- 3.97)	(0.77–1.80)	1.31–2.30)
Region 1	1.25	0.77	0.78	0.91
	(0.79–1.98)	(0.47–1.28)	(0.48–1.27)	(0.69–1.21)
Region 3	0.94	1.24	0.69	0.94
	(0.49–1.79)	(0.81–1.90)	(0.44–1.08)	(0.71–1.23)
Region 4	1.04	1.07	1.01	1.04
	(0.63–1.74)	(0.68–1.70)	(0.67–1.54)	(0.80–1.35)
Region 5	1.28	1.1	1	1.12
	(0.88–1.87)	(0.78–1.55)	(0.73- 1.39)	(0.91–1.36)
Region 6	1.46	1.60**	1.02	1.33**
	(0.88–2.41)	(1.02–2.50)	(0.64–1.64)	(1.01–1.75)
Region 7	0.63	0.37**	1.1	0.70*
	(0.25–1.56)	(0.16–0.85)	(0.60–1.99)	(0.46–1.06)
Region 8	1	0.72	0.73	0.84
	(0.66–1.52)	(0.45–1.14)	(0.47–1.11)	(0..66–1.07)
Assistant professor	1.31*	1.22	1.02	1.17*
	(0.98–1.75)	(0.91–1.63)	(0.77–1.36)	(0.99–1.38)
Associate professor	0.84	0.91	0.59***	0.76***
	(0.61–1.15)	(0.67–1.22)	(0.44–0.80)	(0.64–0.91)
Non-Tenure	1.02	0.75	0.74*	0.81*
	(0.69–1.51)	(0.51–1.10)	(0.52–1.05)	(0.65–1.00)
Survey				0.90**
				(0.83–0.98)
Constant	0.12***	0.13***	0.19***	0.18***
	(0.08–0.19)	(0.09–0.20)	(0.12–0.28)	(0.13–0.24)
Obs	1927	2,435	2,417	6777
Pseudo R2(McFadden’s)	0.02	0.02	0.02	0.01

Confidence intervals are in parentheses below the odds ratios

Reference Categories: Field: Engineer; Region: Region 2; Rank: Full professor

p < 0.01, **p < 0.05, *p < 0.1

Given the disproportionate effects of COVID on women, we ran logistic regressions for the interactions between gender and the experimental conditions. The results were insignificant

Table 7

Logit Regression predicting response to hypothesis 2: Alert timing (odds ratios)

	Survey 1	Survey 2	Survey 3	Pooled
Alert timing (Hypothesis 2)	0.97	1.11	0.91	0.98
	(0.73–1.29)	(0.84–1.47)	(0.70–1.18)	(0.84–1.14)
Female	1.30*	1.04	1.16	1.16*
	(0.96–1.75)	(0.77–1.41)	(0.88–1.53)	(0.98–1.38)
Biology	1.40*	1.42*	0.89	1.17
	(1.00–1.97)	(0.94–2.15)	(0.64–1.24)	(0.96–1.44)
Biochemistry	0.73			0.68
	(0.41–1.28)			(040–1.14)
Geography		4.21***	0.87	1.83***
		(2.47–7.16)	(0.52–1.45)	(1.32–2.55)
Region 1	1.43	0.99	0.65	0.96
	(0.81–2.55)	(0.52–1.88)	(0.36—1.17)	(0.68–1.36)
Region 3	1.23	1.39	0.65	0.97
	(0.58–2.61)	(0.82–2.36)	(0.38–1.10)	(0.70–1.34)
Region 4	1.31	1.11	0.98	1.10
	(0.72–2.38)	(0.63–1.96)	(0.59–1.64)	(0.80–1.52)
Region 5	1.31	1.15	1	1.12
	(0.82–2.08)	(0.75–1.77)	(0.68–1.46)	(0.88–1.42)
Region 6	1.66*	1.61	1.14	1.43**
	(0.91–3.03)	(0.91–2.84)	(0.66–1.97)	1.04–1.98)
Region 7	0.83	0.40*	1.08	0.77
	(0.30–2.34)	(0.14—1.08)	(0.54–2.16)	(0.47–1.26)
Region 8	0.98	0.76	0.74	0.82
	(0.59–1.62)	(0.43–1.35)	(0.44–1.24)	(0.61–1.10)
Assistant professor	1.43**	0.96	1.1	1.14
	(1.01–2.03)	(0.66- 1.38)	(0.78–1.55)	(0.93–1.40)
Associate professor	0.83	0.72*	0.67**	0.73***
	(0.57–1.23)	(0.50–1.04)	(0.47–0.97)	(0.59–0.91)
Non-Tenure	0.92	0.76	0.91	0.85
	(0.57–1.48)	(0.48–1.21)	(0.61–1.35)	(0.66–1.10)
Survey				0.90**
				(0.81–0.99)
Constant	0.15***	0.11***	0.28***	0.22***
	(0.09–0.24)	(0.07–0.19)	(0.18–0.44)	(0.16–0.32)
Obs	1287	1,621	1,616	4522
Pseudo R2(McFadden’s)	0.02	0.03	0.01	0.01

Alert timing is the second experimental condition testing the effect of week v. days’ notice alert timing

Reference Categories: Field: Engineer; Region: Region 2; Rank: Full professor

p < 0.01, **p < 0.05, *p < 0.1

We found partial support for hypothesis one that receiving an electronic prenotification increases survey participation. For survey one, persons who received an electronic prenotification had 1.24 higher odds of participating in the survey than did their counterparts who did not receive a prenotification (p-value = 0.097). For survey three, those who received an email alert prenotification had 1.31 higher odds of participating in the survey than their counterparts who did not receive an email alert (p-value = 0.027). Pooling all three surveys, persons who received an electronic prenotification had 1.16 higher odds of participating than their counterparts (p-value = 0.037). The findings as a whole indicate that electronic pre-notification produces a modest increase in survey participation.

The second experimental comparison was also in the expected direction, with fewer days of advance notice improving the likelihood of responding by about one-third of a percentage point. However, the findings were not statistically significant. Three covariates were found to be significantly associated with survey participation for some of the surveys. Assistant professors, women and biologists were all more likely to participate in survey one and for the pooled regression. For all three surveys and pooled together, only a small amount of variance in response was explained (Pseudo R2 < 0.03). Thus, while there are other factors that may better explain why people respond to surveys, alert letters do play a small role.

Discussion

The goal of the study was to analyze how electronic alert pre-notifications and the timing of those notices would affect response rates to web surveys. We conducted this experiment to determine if receiving an email alert and the timing of those notices impacts response rates. Receiving an email alert results in a statistically significant improvement in response rates. For surveys one and three and the pooled regression, the significant findings were in the expected directions—receiving an email alert improved the likelihood of responding to the survey. Additionally, other factors including gender, field of science, and rank significantly influence response rates.

There are several limitations of the survey design and models. First, other omitted factors may also explain response or nonresponse to the survey, including level of involvement and engagement with the survey topic and timing of dissemination (Sue & Ritter, 2012). It is possible response rates varied due to interest in a specific topic (e.g. COVID19, immigration and visa policy). Second, COVID-19 has changed many academics’ schedules, workload, and stress levels, which could differently affect their likelihood of responding especially by rank and gender. Third, it is possible that because academic scientists primarily work online and have the ability to assess the legitimacy of the survey (e.g. academic institution, Institutional Review Board protocol), the effect of an alert letter on confirming the legitimacy or reputation of a survey request and driving response rates may be lower than with other populations.

A fourth limitation is that reliance on email to communicate makes it more difficult to ensure that every sampled individual received and opened the email pre-notification message and invitation to participate in the survey. Because this sample of academic scientists regularly use email for work and have full access to digital services, an email alert may be more successful with this sample than it would be with a general population survey. There is the possibility that some in the sample did not receive either the pre-notification or invitation due to spam filters or other digital barriers (Bandilla et al., 2012; Tourangeau et al., 2013). Our survey-monitoring program indicates that 694 (10%) of the individuals sampled across all three surveys did not open any of the emails about the surveys (pre-notification or otherwise). It is possible academics were not checking their email during the study period due to travel, fieldwork, or work from home pressures. We conducted analyses with and without the non-contacted 694 included in the denominator and found non-response due to no email contact did not vary across experimental conditions.

Previous research shows that postal mail pre-notifications positively influence response rates to web surveys (Bandilla et al., 2012; Kaplowitz et al., 2004). Our findings suggest email alert pre-notifications can also be effective for improving response rates. The question as to the relative effectiveness of email pre-notifications, compared to those that use another mode, however, remains unresolved. Further investigation is needed to compare the effects of receiving a paper mail pre-notification letter versus an email alert letter and no alert message.

It will also be important to compare the effects of email pre-notification on response rates with larger and more diverse samples. We note that this sample of academic scientists represents frequent email users with regular access to the Internet. The effect of email alert letters would potentially be lower for those who are not regular Internet users and the general public.

This study was motivated by COVID19 restrictions and concern that our inability to send out mailed pre-notification letters would negatively affect response rates from our sample of academic scientists, as office mailing addresses were temporarily inaccessible due to work from home orders. These and other similar circumstances can constrain survey researchers and continue to limit academic progress by restricting contact options and potentially adding to response biases. Ongoing experimentation is necessary as we continue to adapt research methods to the new social and cultural realities.

Table 8

Balance test of experiment conditions

	Svy 1	Svy 2	Svy 3
Female	1.16 (0.56)	0.33 (0.85)	0.55 (0.76)
Region 1 (New England)	7.99* (0.02)	18.65* (0.00)	7.76* (0.02)
Region 2 (Mid East)	0.05 (0.97)	5.10 (0.08)	0.59 (0.74)
Region 3 (Great Lakes)	0.69 (0.71)	2.75 (0.25)	2.47 (0.29)
Region 4 (Plains)	3.35 (0.19)	12.37* (0.00)	10.00* (0.01)
Region 5 (Southeast)	1.87 (0.39)	0.20 (0.91)	10.51* (0.01)
Region 6 (Southwest)	1.03 (0.60)	11.46* (0.00)	0.87 (0.65)
Region 7 (Rocky Mountains)	0.03 (0.98)	3.49 (0.18)	7.42* (0.02)
Region 8 (Far West)	1.52 (0.47)	5.84 (0.06)	10.26* (0.01)

P-values are in parentheses below the Pearson chi2 values. *p < 0.05,

Table 9

Alert letter text