Demographic Stability on Mechanical Turk Despite COVID-19.

Behavioral scientists often think in abstract terms about the people who participate in research. University students form a ‘subject pool’. People recruited outside the university are a ‘community sample’. And, people who complete studies online are part of an ‘online panel’ or a ‘crowdsourcing platform’. This language obscures something important: participants in human subjects research are people, shaped by their unique experiences and shared cultural events. Since early 2020, coronavirus disease 2019 (COVID-19) has been one of these events.

How might research participants change in response to COVID-19? One possibility raised by Lourenco and Tasimi [1] is that online participants may become less diverse. Because tens of millions of people have recently become unemployed [2], it is reasonable to wonder whether people who are struggling to pay the bills will be able to maintain an internet connection–let alone find the time and energy to participate in research. If people are forced to choose between essential things like food and nonessential things like the internet, the diversity of people in online platforms like Amazon Mechanical Turk (MTurk) may decrease, leaving researchers with a narrower group from which to sample. As Lourenco and Tasimi note, decreasing diversity should concern all researchers at a time when more research is moving online, because less diversity may affect the replicability of studies.

Fortunately, on MTurk, we do not see evidence of this so far. Demographic data showing who used MTurk before COVID-19 [3,4] compares favorably with data from today. Table 1 shows the race, income, and gender of people on MTurk from January 2019 through mid-May 2020. The data–which are from CloudResearch’s Metrics tool and are publicly available at https://metrics.cloudresearch.com/–are largely consistent. Despite small changes here and there, the race, income, and gender of people on MTurk have remained remarkably constant.

Table 1

Selected Participant Demographics on Mechanical Turk across 18 Monthsa

	2019												2020
Variable	January	February	March	April	May	June	July	August	September	October	November	December	January	February	March	April	May
Race
White	69	70	69	69	69	69	69	69	71	69	70	68	70	70	69	70	70
Hispanic	7	7	8	8	8	8	8	7	7	8	7	7	7	7	7	7	7
Black	10	9	9	10	10	10	10	10	10	10	10	10	11	11	10	10	10
Asian	11	11	11	10	11	11	10	11	10	11	10	12	10	10	11	11	11
Other	3	3	3	3	3	3	3	3	2	3	2	3	2	3	3	2	2
Gender
Men	45	45	46	44	45	44	43	44	43	43	43	45	44	44	44	44	45
Women	55	55	54	56	55	56	57	56	57	57	57	55	56	56	56	56	55
Income
<$10 000	5	5	5	5	6	6	6	6	6	6	6	6	5	6	6	6	6
$10 000–$19 999	8	8	8	8	8	8	8	7	7	8	8	8	8	8	8	8	7
$20 000–$29 999	12	12	12	12	12	12	12	12	12	12	12	12	12	11	11	11	11
$30 000–$39 999	13	12	13	13	13	13	13	13	13	13	13	13	13	13	13	12	12
$40 000–$49 999	12	11	11	11	11	11	12	12	11	11	12	11	11	11	11	10	11
$50 000–$59 999	11	11	11	11	11	11	11	12	12	12	11	11	12	12	11	11	11
$60 000–$69 999	9	9	9	9	8	8	8	8	9	8	9	8	9	8	9	8	8
$70 000–$79 999	8	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9
$80 000–$89 999	5	5	5	5	5	5	5	5	5	5	5	5	5	6	6	5	5
$90 000–$99 999	6	6	6	6	6	6	5	6	6	6	6	6	6	6	6	6	6
$100 000–$124 999	8	8	8	8	8	8	8	8	8	8	8	8	8	8	8	9	9
$125 000–$149 999	4	4	4	4	4	4	4	4	4	4	4	4	4	5	5	5	5
$150 000–$174 999	2	2	2	2	2	2	2	2	2	2	2	2	2	2	2	2	3
$175 000–$199 999	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
$200 000–$224 999	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
$225 000–$249 999	0.4	0.4	0.4	0.4	0	0.4	0.4	0.4	0.4	0.4	0.3	0.3	0.4	0.4	0.4	0.5	0.5
>$250 000	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1

Numbers are percentages of unique participants within each month from the CloudResearch database. A description of this database and how well it captures the Mechanical Turk population appears in Robinson et al. [7] and Moss et al. [8]. To examine the percentage of studies completed by people within each demographic group each month, visit https://metrics.cloudresearch.com/

Of course, demographic consistency thus far is no guarantee changes will not occur. For a few reasons, however, we are cautiously optimistic about the stability of participant demographics on platforms like MTurk. First, most people who use MTurk do so from home. According to one survey, less than 2% of people connect to MTurk in public spaces (https://www.cloudresearch.com/resources/blog/trends-of-mturk-workers/), meaning an inability to access the internet in places like a coffee shop or library is unlikely to change who is on MTurk.

Next, while a staggering number of people are currently unemployed, people who lost their job and previously used MTurk differ from those who lost their job and do not use MTurk in at least one important way: users of MTurk have access to an online platform that enables them to earn money from home. Although MTurk was not intended as a source of full-time employment, it provides an opportunity for people to supplement their income. This opportunity may be all the more important to people who have recently become unemployed and may rely on MTurk to temporarily make ends meet or pay for relatively small bills like an internet connection, something most people agree is currently important [5].

Finally, we are optimistic about the demographic consistency of people on platforms like MTurk because, ultimately, what matters more than a change in the overall demographics of participants, which have historically been stable [6], is the demographic composition of participants within samples. Although within sample variability is overlooked by many researchers, the composition of participants within samples can vary substantially [6]. For example, while platform level data on MTurk indicate a near even gender split [4], the percentage of men can vary from 25% to as high as 75% across samples. Similar variation occurs for other demographics [6]. If participant demographics are important for generalizability and estimating the strength of effect sizes, then this issue deserves more attention. Fortunately, because online platforms are bigger, more diverse, and easier to manage than many traditional sources of participant recruitment, researchers can control the demographics of participants within studies using quotas. Most participant recruitment platforms make setting such quotas easy.

To close, even though we are relatively sanguine about the prospect of demographic consistency within online research platforms, we would like to echo a point made by Lourenco and Tasimi [1] about the new normal of conducting behavioral research. COVID-19 has changed how social scientists approach their work. As the pandemic progresses, more changes may follow. To conduct rigorous research, behavioral scientists need to be creative. Finding ways to reach people where they are, to push for a close of the digital divide, and to connect with people who are not currently part of online research platforms will strengthen the quality of behavioral research.

Disclaimer Statement

All authors are employed at Prime Research Solutions, the company that owns CloudResearch (formerly TurkPrime). One CloudResearch offering is its MTurk Toolkit which helps researchers run flexible social science studies on Mechanical Turk.