No Participant Left Behind: Conducting Science During COVID-19.

Cognitive scientists have ramped up online testing in response to the COVID-19 pandemic. Although research conducted online solves the problem of data collection, the paucity of internet access among low-income and minority communities may reduce the diversity of study samples, and thus have an impact on the generalizability of scientific findings.

When her children’s school district closed its doors to curb the spread of COVID-19, Tamara Solis needed to make a difficult decision [1]. She could pay for internet access so that her kids could complete their virtual coursework, but doing so would limit her ability to afford food and rent.

The COVID-19 pandemic has upended life in more ways than one. People like Tamara Solis, for example, are being confronted with the harsh reality of how to pay for food and housing at a time when millions of people have lost their jobs [2]. Students graduating from college will not experience the ceremonial milestone they cherish. And cognitive scientists who study human populations have been forced to close their laboratories to uphold social distancing measures.

Among the challenges facing scientists today is how to collect the data needed for publications, grant submissions, dissertations, promotion materials, and so on. Online testing offers a potential solution to this problem, even for fields such as developmental psychology where in-person testing is paramount. Indeed, many child development laboratories are going online for the first time, a shift that is being facilitated by resources such as the Parent and Researcher Collaborative (https://childrenhelpingscience.com), a single, crowd-sourced platform where researchers from different laboratories can post their studies for families to participate in. Yet as more and more cognitive scientists conduct their research online, we find ourselves wondering who is (and who is not) participating in this research.

Online Research: Then and Now

Online testing offers numerous benefits – it is fast, efficient, and inexpensive [3]. One might even argue that platforms such as Amazon Mechanical Turk (MTurk) and TurkPrime have revolutionized behavioral data collection. As excited as we are about the promises of online testing (e.g., in fields such as developmental psychology where data collection is typically slow and expensive), we are also concerned about how the demographics of online participants may shift during the COVID-19 pandemic. In particular, we worry that online testing may reduce the diversity of participants – especially those from low-income and minority households – whose participation in scientific research has been essential in understanding diverse phenomena from language proficiency [4] and spatial reasoning [5] to academic achievement [6] and brain development [7].

At first glance, such a concern may appear to be unwarranted. After all, studies conducted online seem to be more diverse than studies conducted in laboratories [3,8,9]. In fact, with respect to household income, some estimates suggest that a greater number of lower-income workers are accessing MTurk than would be expected given the general population of workers [10]. Following this point, the monetary compensation associated with these studies could very well increase the diversity of participants, especially given that 'Turkers' participate in numerous online experiments to supplement their income [10,11]. Why then are we worried about less diverse online samples during this pandemic?

Participation in online studies requires internet access, a staple among many, although not all, households. In the USA there is a well-known ‘digital divide’ in which Americans in rural and poor communities are less likely to have internet access in their homes than are urban and more affluent communities [12]. The Federal Communications Commission estimates that >21 million Americans do not have a broadband connection with download speeds of at least 25 Mb/s, and nearly half of all households whose income is less than US$30 000 per year have no connection whatsoever. Moreover, Black and Hispanic Americans lag behind their White counterparts in internet adoption even after controlling for income. Nevertheless, in normal times, individuals without home internet access might be able to access the internet in public places such as schools, libraries, coffee shops, fast food restaurants, and even parking lots.

But these are not normal times. Amenities such as the internet may become a luxury as more and more people continue to lose their jobs and struggle to pay their bills. Moreover, people who would normally gain internet access in public places may now be unable to do so because of stay-at-home orders and/or fear of contracting COVID-19. Accordingly, it is possible that participants in online studies may now comprise those who can afford to weather the economic storm and/or have the time to participate in research studies given new demands on family care while working from home. Thus, the push to online testing at this time may exclude a large segment of the population, thereby limiting the generalizability of findings and posing challenges to reproducibility.

To be clear, we are not suggesting that online testing should stop. If anything, we commend our colleagues for the dedication and generosity that they have shown in promoting resources for online testing (especially within our own field of developmental psychology where such resources were not widely available). However, we think the time is now ripe for a discussion about how science can progress in a way that is mindful of access and inequality, a problem that strikes us as particularly acute these days.

Looking Ahead: Recommendations for Ensuring Diversity

We can take several steps to ensure and promote the diversity of study samples. We sketch out below some recommendations aimed at advancing demographic diversity during and after the COVID-19 pandemic. We recognize that this list is not exhaustive, and we also recognize that some of these recommendations will be difficult and costly to implement. Nevertheless, we hope that what follows may serve as a launching pad for further consideration.

First, we recommend that studies conducted online should collect and report detailed demographic information about study participants. One potentially efficient way to mandate demographic data collection is to set it as the default for all studies posted online. In this vein, we also encourage journals to require authors to report the characteristics of their study samples. For example, the Society for Research in Child Development (SRCD) enacted this policy across all its journals earlier this year, in January 2020. We recommend that other journals should follow suit. Although the simple reporting of demographics will not itself increase diversity within samples, it nevertheless ensures transparency and promotes future reproducibility.

Second, we urge scientists to make efforts to provide temporary internet access to low-income participants. Mobile hotspots are available for purchase, and these could be mailed to participants or dropped off at their homes to allow them to participate in online studies. We recognize that the financial burden here is likely to be disproportionate across researchers, some of whom will have funding to support such a move and others who may not.

Relatedly, as a longer-term goal that transcends the issue of online testing, scientists and grant-funding agencies might consider lobbying government for subsidies related to internet costs, and perhaps even advocate for universal availability of internet access, which is essential for living and operating in contemporary times. Some internet companies have already reduced prices for qualifying customers and have provided free access to online educational resources [13]. This is a promising start, but much more will be necessary to ensure availability to those who need, but cannot afford, internet access.

Finally, we recommend that scientists who have the resources to implement off-site testing should consider using such an approach to reach participants in low-income and minority communities. Some universities (e.g., University of Illinois at Chicago, University of Iowa, and Indiana University) have developed mobile laboratories with portable technology. These laboratories may be especially useful in the years to come, particularly if the necessary precautions are taken to ensure the safety of experimenters and participants, including the use of personal protective equipment (PPE) and regular disinfection of study materials.

Toward a New Normal

Even if we succeed in flattening the curve, resurgences in infection are likely until (or if) a vaccine for COVID-19 can be developed. Thus, we may be on the precipice of a new normal in which threats of disease may require long-term social distancing practices and may differentially impact on those in low-income and minority communities. At a time when access to a large segment of the population may be hampered, we must not forget the value and necessity of diverse populations in uncovering the human experience in terms of its universality and variability [14,15]. After all, this is what cognitive science is all about.