Representativeness of autistic samples in studies recruiting through social media.

Survey-based research with recruitment through online channels is a convenient way to obtain large samples and has recently been increasingly used in autism research. However, sampling from online channels may be associated with a high risk of sampling bias causing findings not to be generalizable to the autism population. Here we examined autism studies that have sampled on social media for markers of sampling bias. Most samples showed one or more indicators of sampling bias, in the form of reversed sex ratio, higher employment rates, higher education level, lower fraction of individuals with intellectual disability, and later age of diagnosis than would be expected when comparing with for example population study results from published research. Findings from many of the included studies are therefore difficult to generalize to the broader autism population. Suggestions for how research strategies may be adapted to address some of the problems are discussed. LAY
SUMMARY: Online surveys offer a convenient way to recruit large numbers of participants for autism research. However, the resulting samples may not fully reflect the autism population. Here we investigated the samples of 36 autism studies that recruited participants online and found that the demographic composition tended to deviate from what has been reported about the autism population in previous research. The results may thus not be generalizable to autism in general.

INTRODUCTION

There is an increasing recruitment of participants for research studies through online channels, particularly for studies centered around surveys, as these can be conducted fully online and at a low cost (Andrade, 2020). Online surveys allow broad geographical diversity and may be able to reach parts of a population that might not otherwise participate in scientific studies, for example, individuals with busy lives and/or with a preference to participate more anonymously (Evans & Mathur, 2005). Surveys advertised on social media may therefore be a useful way to obtain large sample sizes. However, the representativity of the cohorts obtained this way may be compromised.

Sample size correlates negatively with the variance of the sampling distribution of a statistic of interest (e.g., the sample mean). Larger samples minimize sample‐to‐sample variability and thus produce more precise estimates than smaller samples. However, a precise estimate is not necessarily accurate. The benefits of large samples can only be obtained if the sample is unbiased and representative, so that observations may be generalized from the sample to the entire population under study.

Unrepresentative samples are usually a result of a sampling bias, that is, a sampling process that over‐samples some parts of the population and under‐samples other parts. In the absence of sampling biases, larger samples will more closely resemble the population and produce smaller standard errors as given by basic statistical calculations. However, a large sample cannot make up for sampling biases. Indeed, in the presence of sampling bias, even very large samples can deviate considerably from the population (a notable example being the Literary Digest poll of the 1936 US presidential election [Squire, 1988]). In a representative sample, the distributions of all characteristics should be approximately equal to those of the population of interest. Conversely, unrepresentative samples may differ from the population on one or more parameters, in which case observations on the sample should not automatically be assumed to be generalizable to the target population.

While many characteristics of the population will be unknown a priori, previous research can provide a reference point for how some characteristics, for example, demographic variables are expected to be distributed in the target population. Therefore, demographic information such as sex ratio or educational level can inform on whether a sample diverges from what has generally been reported within research and can thus act as markers of sampling bias.

When sampling online, there may be biases related to response/nonresponse, meaning that those who choose to participate may systematically differ from those who choose not to (Andrade, 2020; Rubenstein & Furnier, 2021). Some demographic groups may be more likely to respond than others (Cheung et al., 2017), and individuals may choose whether to participate based on their interest in or position on the topic of the survey (Groves et al., 2004). There may also be biases related to exposure to the survey. Online surveys on autism may be circulated in online communities for autistic individuals. These communities may attract certain subgroups of the autism population, such as autism self‐advocates, more than other subgroups. Though the perspectives of these groups are relevant, they may not be representative of the autism population as a whole. To assess whether the theoretical risks of sampling bias are likely to have a practical implication, we have investigated the sample representativeness of a selection of studies that have recruited through social media.

METHODS

We investigated markers of sampling bias in studies that aimed to characterize the autism population using online surveys with participants recruited, entirely or in part, through social media. We used the following search string on Google scholar: ‘autism AND “online survey” AND “social media” AND (recruitment OR advert*) AND (“non‐autism” OR “non‐autistic”)’. Since studies that aim to make general statements about the autism population usually compare to a control group, we included the search term (“non‐autism” OR “non‐autistic”). Google scholar was used as it can search article full texts rather than just the abstract, and some of the words in our search query (e.g., social media) may be mentioned in methods sections rather than in abstracts. The full text of each hit was examined, and studies were included if they obtained part of their autism sample by recruitment on social media, including for example, “online autism communities.” Studies were excluded if they were not peer‐reviewed or not written in English. Additionally, studies were excluded if their aim was to investigate a subpopulation, such as autistics who are mothers, or university students with autism, since such samples should not be expected to be representative of the whole autism population. Studies focusing on individuals within a specific age bracket were included. For each included study, we extracted (if reported) the following markers of sampling bias: Sex ratio, average age at autism diagnosis, percentage with a college or university education, percentage of unemployment, and percentage with intellectual disability. To further examine the characteristics of the samples we also extracted information about the racial/ethnic composition and identification with another gender. Additionally, we examined whether the studies discussed potential sampling biases and whether they included enough information in the methods section to replicate the sampling process. The search was conducted on October 5, 2021.

RESULTS

One hundred seventy search hits were identified. Thirty‐six studies met inclusion criteria, all except two were published in 2019 or later. The included studies are summarized in Table 1. Many of the included studies reported very limited demographic and descriptive information about their samples. Figure 1 shows the percentage of the identified studies that reported each of the selected variables. A few additional studies reported similar variables that could not be compared, such as the percentage diagnosed as adults instead of the mean age of diagnosis. Sample characteristics from the studies that did report each of the examined demographic variables are shown in Figure 2. Most of the studies did not purely sample from social media, as the surveys were advertised through several channels. Five studies collected data from proxy‐informants and reported this separately from the data from autistic individuals themselves. For these studies, the weighted average of sample characteristics for the two groups were used, weighted by the group sizes.

TABLE 1

Overview of sample characteristics in the included studies

Study	Subject	Autism sample size	Mean age (years)	Female/male ratio	Mean age of diagnosis (years)	College/university(%)	Unemployed (%)	ID (%)	Non‐cis gender identity	Race/ethnicity	Other recruitment source(s)
Arnold et al. (2019)	Comorbidity	263 (221 self report, 42 proxy report)	40.3	1.10	31.3	55		16.3	4.2% “other”	88.2% Caucasian	Autism and disability organizations and groups, educational institutions, health care practices
Belcher et al. (2021)	Camouflaging	40	25.65	1.00	18.08				5% transgender		Local Universities and community centers
Benevides et al. (2020)	Mental health priorities	182	38.7	3.25					21.3% nonbinary	82.4% white
Bowri et al. (2021)	Alcohol use and misuse in autistic adults	237	41.92	1.67	35.98	79.3	12.7	4.6	5.5% “other”		Internet forums; The Cambridge Autism Research Database; charities
Camm‐Crosbie et al. (2019)	Experiences of support for mental health difficulties	200	38.9	1.58	34.1						The Cambridge Autism Research Database; charities; support groups; educational institutions
Caron et al. (2021)	Quality of life	427	35.4	2.44	31.9	62	21.5		19.9% “other”	6.3% “ethnic or cultural minority group”	Autism associations
Cassidy et al. (2021)	Validation of suicidal behaviors questionnaire	308	39.71	2.70	34.58	66.88	36.04	0.97	16.56% non‐cis	93.18% white or Caucasian	The Cambridge Autism Research Database; Autistica network
Cooper et al. (2021)	Identity and collective self ‐esteem	105	34	1.39	29.19					88% white	Online forums; autism association
Crane et al. (2019)	Mental health	109	20.57	1.70	13.34	*			15.6% nonbinary	87.1% white	Existing databases; personal contacts of the research team
Crane et al. (2022)	Autistic young people's experience of transitioning into adulthood	68	20.6	1.46	12.2				5.6% trans or nonbinary		Charities; Schools/colleges; support groups; professional contacts of the research team
Digard et al. (2020)	Bilingualism	297	32.4	2.58	26.4	53.5			16.8% “other” gender		Universities; autism networks
Griffiths et al. (2019)	Mental health, vulnerability and life satisfaction	426	44	1.18		62	33	3	12% trans/nonbinary/other		Cambridge Autism Research Database; charity websites
Halstead et al. (2021)	Impact of COVID‐19 on sleep in autistic adults	95	36.86	2.70		65.3	24.3		13.7% nonbinary	88.4% white	Support groups
Huang et al. (2021)	Factors associated with age at autism diagnosis	657	36.52	1.33	29.84			11.1	5.6% “other”	85.8% Caucasian	Autism and disability‐related associations; service providers; websites
Hull et al. (2021)	Camouflaging	305	41.9	1.74			11.8		7.2% trans or nonbinary		The Cambridge Autism Research Database; autism charities
Hwang et al. (2020)	Psychometrics properties of the Connor‐Davidson Resilience Scale	95	44	1.70				1			Autism and disability organizations; autism self‐advocacy groups; higher education institutions
Jose et al. (2021)	Need priorities	164 (77 self report, 87 proxy report)	29.3	0.62			53.7	20.7		92% Caucasian	Interviews in traditional media outlets;
Kenny et al. (2016)	Terminology	502		1.62					5% “other”	89.6% white	National Autistic Society; parent advocacy groups; practitioners; researcher networks; online fora
Maitland et al. (2021)	Social identities and wellbeing	184	41	1.98		67.4	18		6% “other”	92.9% white	Universities; local businesses; disability services; autism association
Micai et al. (2021a)	Psychiatric and medical co‐occurring conditions and health behaviors	964 (522 self report, 442 proxy report)		1.00							National/local organizations or service providers including residential facilities, job training, and education programs; e‐newsletters, websites, and social media
Micai et al. (2021b)	Interventions for autistic adults	565 (263 self report, 302 proxy report)		0.92		27	57				National/local organizations or service providers including residential facilities, job training, and education programs; e‐newsletters, websites; social media, and professional networks
Moseley et al. (2021)	Levels of self‐representation	120	40.7		35.6						Previous studies
Muniandy et al. (2021)	Trait resilience, coping strategies and mental health outcomes	78	46.6	1.40					7.7% “other”		State based organizations; research participant databases; educational institutions; advocacy and support groups; health practices
Murphy et al. (2020)	Autism and transgender identity	62	29.1	2.65					40.3% transgender		Databases; local student populations
Oomen et al. (2021)	Psychological impact of the COVID‐19 pandemic	613	38.36	2.16			34		0.5% intersex		Autism organizations; databases
Pellicano et al. (2014)	Views on researcher‐community engagement in autism research	122	39.4	0.93							Autism organizations; parent advocacy groups; practitioner and researcher networks
Perry et al. (2022)	Camouflaging and autism related stigma	223	34.19	2.45	28.67	52.4			17.5% nonbinary	92.8% white	Autism community groups; charities; word of mouth via personal contacts
Robinson et al. (2020)	Camouflaging	278									Database
Rogge (2021)	Sunk‐cost bias	187	38.8	1.37		61.5					Autism organization
Romualdez et al. (2021)	Experiences of disclosing an autism diagnosis	238		1.76		61	10.9		9.6% nonbinary or “other”	95% white	Online research network platform; employees of corporate partners
Stein Duker et al. (2019)	Health care	65 (34 self‐report, 31 proxy report)	28.1	0.18		37				67.7% white	Local support groups; clinics and hospitals; supported employment locations
Thompson et al. (2020)	Wellbeing	30	19.93	1.00		*			20% “other”		Local autism services and groups' websites and newsletters
Weir, Allison, Warrier, and Baron‐Cohen (2021)	Physical health	1156	40.98	1.77		59.43				88.3% white	The Cambridge Autism Research Database; autism support groups and charities
Weir, Allison, Ong, and Baron‐Cohen (2021)	Health outcomes	1183	41.04	1.71		59		1.78		88.3% white	The Cambridge Autism Research Database; autism support groups and charities
Weir, Allison, and Baron‐Cohen (2021)	Sexual health	1183	41.04	1.71		59		1.78	12.6% transgender	88.3% white	The Cambridge Autism Research Database; autism support groups and charities
Wilson et al. (2021)	Community mobility	295	40.8	1.10			45.8	12.9	3.7% “other”		Autism, disability, education, or employment‐related networks and services

Abbreviation: ID, Intellectual disability. An asterisk (*) indicates that the study did report the number of individuals who had completed college/university, but this information was omitted as the study only included individuals up to the age of 25.

FIGURE 1

Overview of the percentages of studies that reported each of the selected markers of sampling bias. This does not include studies that reported demographics in other ways, such as total years of education or percentage diagnosed as adults.

FIGURE 2

Distributions of sample characteristics for the five markers of sampling bias among the included studies that had reported each of them. Each point represents a single study. The boxes indicate the quartiles of the distributions. The thick blue lines indicate estimates of where the mean would be expected in the autism population in general, based on the literature (see main text for details). For college/university, a conservative estimate based on the general population is used. The expected mean age of diagnosis in the whole population is difficult to estimate from the literature. Five studies included proxy responses, for example, by a relative. Mean sample characteristics for these studies were calculated as the weighted average between the self‐reports and proxy‐reports and are highlighted as red squares

Sex ratio

A meta‐analysis of published studies (Loomes et al., 2017) estimated the male to female ratio in autism to be around 3:1. A recent register‐based study has also reported a high male preponderance in autism (2.57:1 in adults, 3.67:1 in children) (Posserud et al., 2021). Here, we generally observed a reversed sex ratio where most participants in most of the investigated studies were female, with the median male‐to‐female ratio being 1:1.6. A general tendency for a higher female respondence in survey research has been described (Aerny‐Perreten et al., 2015; Cheung et al., 2017), but the shift in sex ratio observed here is substantially larger (around 3:1 in the population and 1:1.6 in the samples), and likely indicates additional sampling bias.

Age of diagnosis

Autism is currently defined as a developmental condition where symptoms should present across contexts during childhood. Therefore, one would expect a substantial proportion of the autism population to have received their diagnosis in childhood. However, we are not aware of a good estimate of the exact expected age of diagnosis, which will likely depend strongly on the age of the sample being studied. In the investigated studies we found a tendency for the mean age of autism diagnosis to be high and often well into adulthood (median = 29.8 years). The two studies where mean ages of diagnosis were below 18 years were focusing specifically on younger adults up to 25 years. Although it is very possible that a part of the autism population has not been diagnosed as children despite showing symptoms, such a large percentage of participants having been diagnosed late could indicate the presence of a sampling bias.

College/university degree

Based on a recent US census, around 45% of the general population aged 25 or older had obtained a college or university degree, including 2‐year college degrees, of which 35% of the population had obtained at least a bachelor's degree (U.S. Census Bureau, 2018). Similarly, according to OECD data for the UK, 49% of all adults aged 25–64 had obtained post‐secondary education, of which 25% had a bachelor's degree (OECD, 2021). In our investigation we found the median percentage of autistic individuals that had obtained a college or university degree to be 60.2%. This does not correspond well with previous studies reporting a tendency for lower educational and occupational outcomes among individuals with autism compared to individuals without (Eaves & Ho, 2008; Howlin et al., 2013; Shattuck et al., 2012; Taylor & Seltzer, 2011).

Unemployment

Unemployment rates for individuals with autism have generally been found to be high, with more than 50% not having a job (Hedley et al., 2017). In the investigated studies we observed lower unemployment rates, with a median percentage of unemployment of approximately 28.7%.

Intellectual disability

A record linkage study (Idring et al., 2015) estimated the prevalence rate of co‐occurring intellectual disability at around 25.6% among 0‐ to 27‐year‐old individuals with an autism diagnosis, while a recent register‐based study (Rødgaard et al., 2021) observed a co‐occurrence rate of intellectual disability in 16‐year‐olds of approximately 17%. The prevalence of co‐occurring intellectual disability in the investigated studies tended to be much lower with a median percentage of 3.8%.

DISCUSSION

Distributions of reported sample statistics in the investigated studies departed markedly from what would be expected of the autism population based on previous research. This suggests sampling bias among many of the investigated studies. In these studies, the results should not be interpreted as characterizing the autism population as a whole. The sampling bias could potentially reflect that substantial groups within the autism population are under‐sampled because they may be less likely to be part of the autism communities on social media where online surveys may be circulated. Conversely, it is possible that those with a late autism diagnosis are more likely users of autism communities or networks on social media and generally more likely to respond to surveys about autism because the recency of their diagnosis leads to a greater interest in autism and a greater need for connecting with other individuals with an autism diagnosis. However, the present investigation cannot determine why recruiting autistic individuals on social media might result in a bias nor is it the scope of this study to estimate the precise risk of sampling biases associated with studies using online recruitment in general.

The descriptions of the recruitment processes were generally quite limited. It would be beneficial for studies to report more detailed methodological information concerning recruitment, for instance, the full advert text, the specific social media platforms and/or groups that were targeted, and how many participants were obtained from each recruitment source (e.g., different social media platforms or a database). Such information is relevant in order to replicate the studies (Rubenstein & Furnier, 2021) and may be useful in assessing the risk of a bias. For example, advert texts that ask participants to share the advert with their network (“snowball sampling”) may exacerbate sampling bias. Snowball sampling favors individuals with larger social networks as they will be more likely to receive the advert from someone they know (Baltar & Brunet, 2012; Magnani et al., 2005). Additionally, the individuals initially included will influence the composition of the rest of the sample (Magnani et al., 2005), which violates the assumption of independent observations made in most common statistical tests. Additional details about recruitment would have allowed for more quantitative meta‐analytic studies of the effects of online recruitment, for example, how the fraction of the sample that was recruited online correlates with markers of sampling bias, or whether a snowball sampling strategy is associated with specific sample characteristics. To help ensure transparency and reproducibility, journals could require authors to report essential descriptive statistics about their samples, that is, sex, age, race/ethnicity, diagnostic status (clinical or self‐diagnosed), age of diagnosis, education, employment, IQ, and intellectual disability, as well as include detailed descriptions of the recruitment and sampling procedures. It may even be beneficial to require that the most central elements of this information be mentioned in the abstract.

Some studies combined their recruitment through social media with recruitment from existing databases of participants. However, such databases may also contain data from individuals who were recruited to the database through online channels and may have similar biases. Recruitment through established databases may be a way to reduce sampling bias, provided that the databases themselves are representative of the autism population. It would thus be beneficial to continuously monitor relevant descriptive statistics of the individuals in each database. In studies that recruit individuals from a database, the possibility of response/nonresponse bias should still be considered.

Most of the investigated studies did include some discussion of the potential for sampling bias. Specifically, most studies mentioned that individuals with intellectual disability were likely underrepresented. Underrepresentation of intellectual disability is widespread in the autism literature (Russell et al., 2019), and this may thus not be a bias that is specific to autism research using online recruitment. Underrepresentation of intellectual disability could contribute to some of the deviations in employment and education level described here. However, although intellectual disability is overrepresented in the autism population, it is far from a majority (Idring et al., 2015). It is thus unlikely that excluding this group alone would bias demographic variables as much as we observed here (Figure 2). For example, the median proportion of samples with a college/university education was found to be markedly higher than what has been reported in the general population, which would also not be expected in the population of autistics without intellectual disability. Indeed, even in the absence of intellectual disability, specific symptoms associated with autism, for example, social or communicative difficulties would likely lead to poorer outcomes. A recent study found that individuals diagnosed with Asperger's syndrome, the criteria of which require that no cognitive delay be present, were reported to have significantly lower levels of both education and employment than matched individuals without any autism diagnoses (Toft et al., 2021). Furthermore, low outcomes for autistic individuals without intellectual disability has previously been reported for education and employment (Howlin et al., 2013). Underrepresentation of those with intellectual disability is also not likely to explain the large proportions of females observed in the present study.

Some studies included more elaborate considerations of representativeness, for example, suggesting that individuals receiving their diagnosis early may have been under‐sampled. However, even though individual studies may contain nuanced discussions of biases, there is still a risk that important limitations regarding interpretation are not reflected in titles and abstracts (Lord & Bishop, 2021). Furthermore, when numerous studies exploring the same topic use the same methodologies with the same risks of sampling biases, they may confirm each other's findings and give the impression that the findings are robust characteristics of autism, even though they may not generalize to the entire autism population. Discussion of limitations due to non‐representativeness can therefore likely not eliminate the problems of sampling bias.

It is widely agreed that the current autism spectrum population is very heterogenous (Lord et al., 2020; Masi et al., 2017) with immense differences between individuals with an autism spectrum diagnosis. Therefore, trying to derive general truths about the entire autism spectrum as currently defined may not be the most fruitful approach. Two studies with the same aim might obtain different results if their autism samples are different, for example, due to different sampling strategies. A study that manages to create a representative sample of the autism population and compares it to a representative control sample would obtain a result describing the autism population “on average.” For any given individual in the population, such an average result will not necessarily meaningfully describe the condition of that individual, due to the large heterogeneity.

Instead, it may be worthwhile to conduct research on more narrowly defined and more homogeneous subgroups of the autism population. To follow this strategy, researchers would define a subgroup of interest, by narrowing the range of variation of some variables (e.g., language or comorbidity) selected by the researcher. Results obtained by separately studying such a subgroup cannot be generalized to all individuals with autism. However, a sample that is narrowly selected on certain parameters may be more likely to share additional characteristics that would be identifiable even in moderately‐sized samples. Subsequent comparisons of findings from different subgroups could then allow us to learn under which conditions various subgroups may or may not be considered together. In reporting results, we should be explicit about the subgroup under study, in order to specify which part of the current spectrum the results may be generalized to. The chosen subgroup may or may not represent a categorical biologically distinct group, as long as it circumscribes a more homogeneous population. It has previously been suggested to study “prototypical autism” in a similar manner (Mottron, 2021), but the strategy can be similarly employed to study other subgroups of autism.

An alternative strategy could be to sample broadly among the entire autism population with the aim to investigate differences within such samples and drawing conclusions with these potential differences in mind. For example, after observing a small or moderate overall effect in the whole sample, it may be possible to identify subgroups that are driving this effect and thus draw more informative conclusions than making a statement about the entire autism population, which may not be relevant for a substantial part of it.

Finally, in cases where demographic variables indicate a bias, it would be beneficial to attempt replication in a sample with different characteristics, corresponding to those who may initially have been under‐sampled, for example, individuals diagnosed with autism as children, individuals with co‐occurring intellectual disability and individuals with lower education levels. One approach to collecting such a complementary sample could be to analyze responses where another person assists the autistic individual or answers on their behalf. In the present study, samples that also included proxy responses tended to have lower education level, a lower female–male ratio and more individuals with co‐occurring intellectual disability (Figure 2). Crucially, as a research community we should strive to develop a language to clearly and succinctly describe various autism subpopulations and compartments within the autism population. This may make it easier for researchers to define their target populations and make statements that do not inadvertently over‐generalize to the whole population before we have sufficient evidence for doing so.

Limitations

Identifying studies meeting the inclusion criteria for the present investigation is challenging for several reasons. Authors may use many different words and phrases to describe the online recruitment procedure making it difficult to construct inclusive search strings. Furthermore, the methodological details about recruitment are often not included in the abstract and will only be found if the search engine has indexed the full text. The search strategy used here therefore cannot be assumed to identify all studies of autism that have recruited online and should not be taken as a characterization of online recruitment studies as a whole, nor should the results of our search be seen as an estimate of the extent to which online recruitment is used in autism research. Rather, our investigation serves to demonstrate that sampling bias issues are not merely a theoretical concern but may exist in a substantial number of studies that recruit samples online. Studies with unrepresentative samples may still have scientific value. Generalizability issues aside, well‐designed online surveys may provide valuable qualitative insight into a phenomenon and may generate hypotheses that can be tested using complementary approaches. Many studies used mixed recruitment channels, and our results cannot elucidate whether online recruitment is the determining factor for the risk of sampling bias, or whether the included studies may share other characteristics associated with sampling bias. Higher education and unemployment may have been operationalized slightly differently across the investigated studies, for example, due to national differences in educational systems.

ETHICS STATEMENT

The study re‐analyzed data from previously published studies and did not involve any additional testing on human subjects.

CONFLICT OF INTEREST

Kamilla Woznica Miskowiak declares having received consultancy fees from Lundbeck A/S and Janssen‐Cilag A/S in the past 3 years. Eya‐Mist Rødgaard, Kristian Jensen, and Laurent Mottron declare no conflicts of interest.