Comparison of study samples recruited with virtual versus traditional recruitment methods.

BACKGROUND: Traditionally, clinical studies rely on brick-and-mortar sites to recruit participants. Newer technology-based studies have utilized non-traditional virtual methods that can potentially recruit more diverse populations and shorten recruitment timelines. This manuscript aims to quantify how sample metrics across three virtual studies compare to traditionally recruited samples, as a first step in building an empirical evidence base for the experience of participant recruitment in virtual studies.
METHODS: We conducted a systematic search of the literature using PubMed to identify relevant studies conducted in the United States in cognitive health, diabetes, and hypertension (which we called comparator studies) to compare to three virtual studies. For each included study, we extracted participant demographic characteristics and information on recruitment methods and timing. Two investigators independently extracted this data, compared results for consistency, and contacted comparator study authors for clarifications. Characteristics for measurement included age, sex, race/ethnicity, states represented, recruitment time, and recruitment rate.
RESULTS: We identified 19 comparator studies. Virtually recruited samples were slightly younger, had more female participants, and were split on enrollment of racial minorities as compared to comparator studies. Virtually recruited samples were more diverse geographically and recruited faster.
CONCLUSIONS: Virtual recruitment may enhance efficiency and enable more individuals to participate in clinical research. To our knowledge, this is the first rigorous and replicable study comparing participant demographic characteristics and recruitment metrics between virtual and traditional recruitment methodologies. Future research should compare a wider range of studies on other metrics such as overall cost of recruitment and quality of participants.

Introduction

It is impractical to enroll all members of a target population into a clinical research study [1]. Instead, the goal is to recruit a representative sample of the target population to enhance the external validity or generalizability of study findings to the population of interest [2,3]. A thoughtfully selected sample is important; this need, combined with other recruitment and enrollment hurdles, impacts research timelines. Participant recruitment and enrollment is often the most time-consuming aspect of the clinical research process [4]. An analysis of clinical studies found that recruitment can take up to 30% of the research timeline, and is the leading cause of missed clinical trial deadlines [5]. Another estimate suggests that up to 80% of clinical trials do not meet participant enrollment timelines [6,7].

Today, participant recruitment into clinical research occurs at a limited set of physical sites, and relies on in-person screening and consent. Studies struggle to enroll the target sample within the specified timeframe [8,9]. A recent study found that nearly 40% of sites for clinical studies under-enroll, and 11% fail to enroll one participant [10]. On-site recruitment requires time to identify enough eligible participants, and additional time to consent and enroll participants; this time yields higher costs and delays answering pressing research questions. A 2014 study estimated that patient recruitment and retention for clinical trials cost over $2.3 billion each year [11].

In recent years, however, virtual studies have begun to transform feasibility for participant recruitment into clinical research [12]. Virtual studies recruit, consent, and enroll participants online, and often administer interventions and track responses remotely via mobile or other online devices [[13], [14], [15], [16]]. The clearest potential advantage of virtual studies is that they maximize the number of individuals able to participate in each study [13]. By removing barriers to participation, virtual studies allow researchers to identify a larger pool of eligible participants more quickly, and enable individuals previously unable to participate (due to disability, geography, access to transportation, etc.) to now participate. Further, by reducing recruitment time and removing physical presence of staff at study sites, these studies can potentially reduce recruitment costs. Many virtual study platforms are also increasingly linking participation to online communities of individuals with shared health conditions, which may promote participant engagement and retention [13].

Yet, not all virtual studies have been successful with recruitment. Notably, the first virtual clinical trial conducted by Pfizer in 2011 failed to reach target enrollment numbers [4]. Virtual trials may face setbacks in participant recruitment; many individuals may enroll in studies because their physician recommends participation, they appreciate meaningful relationships that are built by visiting a site in-person, or they value the support system that is developed with study staff over time [9,12]. Further, many participants may have concerns about data privacy and protection issues unique to virtual studies [17].

The benefits of virtual recruitment seem substantial, yet concerns about hurdles are valid. With this paper, we begin building an empirical evidence base for the experience of participant recruitment. We compare the sample metrics of three virtually recruited samples to relevant traditionally recruited comparison samples.

Materials and methods

Study identification

We conducted a systematic search of the literature to identify relevant comparator studies for three virtual studies (conducted by Evidation Health, Inc., San Mateo, CA) in cognitive health, diabetes, and hypertension in the United States, and to extract selected information on participant and recruitment features. The included virtual studies were selected because they were the first three conducted by Evidation Health that relied exclusively on virtual recruitment across a broad eligible population (e.g., online recruitment through social media, relevant mobile apps, and targeted advertisements), and to have concluded recruitment at the time this review was initiated, thus providing a full study sample for comparison purposes. The virtual study on cognitive health was a 12-month long, prospective, intent-to-treat, single arm study that aimed to evaluate the impact of a virtual cognitive health coaching program (via telephone and email/text messaging) on cognitive function and mental health for older individuals who showed signs of subjective cognitive decline [16]. The virtual study on diabetes was a 12-week long, prospective, intent-to-treat, single-arm study that aimed to evaluate the impact of a mobile diabetes app and experts coaching program on HbA1c1 levels for individuals with Type 2 diabetes and HbA1c ≥ 7.5% [14]. The virtual study on hypertension was a 12-week long, prospective, intent-to-treat, 2-arm randomized clinical trial that aimed to evaluate the impact of a smartphone application on blood pressure control and self-reported medication adherence for patients with poorly controlled blood pressure [15]. All three studies were IRB-approved and conducted via an online study platform, where potential participants answered a set of screener questions to assess eligibility and signed an electronic informed consent form [[14], [15], [16]].

Comparator studies were eligible for inclusion if they were published in English, presented original research from the United States, involved a minimal risk intervention, and were conducted among a non-restricted population of individuals (i.e., not targeted to a specific racial/ethnic group) with Type 2 diabetes, hypertension, or cognitive decline recruited through non-virtual means. The PubMed database was searched using specific search terms for all relevant studies published in English in the three years preceding the search date (between 2014 and 2017) separately for each of three disease areas (cognitive decline, diabetes, and hypertension). Search terms included (1) ((cognitive decline[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/07/13"[Date - Publication]: “2017/07/13"[Date - Publication])); (2) (((type a diabetes mellitus[MeSH Terms]) AND diabetes[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/04/01"[Date - Publication]: “2017/04/27"[Date - Publication]); and [18] (((hypertension[MeSH Terms]) AND hypertension[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/06/01"[Date - Publication]: “2017/06/22"[Date - Publication]). The date of the search, the number of results, and reasons for exclusion were tracked for each study population. The search was conducted on July 13, 2017.

Study selection

One investigator reviewed all titles returned by the PubMed searches for relevance and eligibility with regard to the study inclusion and exclusion criteria. Studies not excluded by title review progressed to abstract review. One investigator read all remaining abstracts in each of the three study areas. In this second stage of screening, we excluded articles if the abstract indicated misalignment with the inclusion criteria. Following abstract review, two investigators reviewed the full text of remaining studies. As in prior steps, we excluded full text articles if the research was conducted in another country, within a narrowly targeted or overly broad population, focused on an unrelated outcome, included an invasive intervention, or presented results from a systematic review or meta-analysis (as opposed to an original research article).

Data extraction

For each included study, we extracted published information on participant demographic characteristics, as well as information on recruitment methods and timing. Demographic characteristics extracted included study population age (mean and standard deviation), sex (female/male/other), race (white versus another racial identity), and distribution of states of residence within the United States. Recruitment data included the number of participants recruited, the number of months over which that recruitment took place, and where and how participants were recruited. Two investigators independently extracted this data from final included studies and compared results for consistency. Where data on the above variables were missing in a manuscript, one investigator contacted the corresponding author via email to request the data. Of eight authors contacted, six replied with the requested information.

Analysis

We utilized a fixed-effects mini meta-analysis described by Goh et al. [19] to compare mean age weighted by sample size, and separately compared the proportion of the study population that identified as female, proportion of the study population that identified as white, number of states in which study participants resided, number of months to complete participant recruitment, and recruitment rate (average number of participants enrolled per month) across studies.

For continuous outcomes (age, number of states in which participants resided, number of participants recruited, recruitment rate), for each therapeutic area, we pooled data from the comparator studies to generate a group mean, weighted by sample size, and standard deviation. We then compared the pooled mean and standard deviation for each outcome to the individual mean and standard deviation from the virtually recruited study in that therapeutic area (cognitive health, diabetes, or hypertension) via unpaired two-sample t-tests that did not assume equal variance. Despite the robustness of the t-test to violations of the normality assumption in samples of these sizes, we also tested continuous outcomes for a difference of medians using the nonparametric unpaired Wilcoxon-Mann-Whitney test (all excluding age, which was reported as a mean). We assessed equality of proportions of white participants pair-wise between each included study and the comparator virtually recruited study with a two-sample test of proportions. Similarly, we assessed the equality of distributions of participant sex pair-wise between each included study and the comparator virtually recruited study using Fisher's exact test. As the data for age, race, and sex came from the same group of participants, separate comparisons of those metrics were not fully independent; however, this does not violate the basic rule of independence within a meta-analysis [19]. All statistical analyses were conducted in Stata version 15.

Results

Selected studies

We identified 19 comparator studies across the three disease areas (cognitive health, diabetes, and hypertension) that fit our eligibility criteria (Fig. 1a, Fig. 1b, Fig. 1ca–c, Table 1) (one study was relevant for both diabetes and hypertension comparisons). The four comparator cognitive health studies assessed the association between cognitive function in older adults and water-based exercise [20], yoga and meditation [21], meditation and music [22], and yoga [23]. The nine comparator diabetes studies assessed the association between a low-level behavioral weight loss intervention [24,25], telehealth intervention [26,27], cloud-based diabetes management program [28], physical activity and weight loss intervention [29], treatment adherence intervention [30], mobile coaching system [31], and diet and exercise counseling [32] and change in HbA1c levels (among other outcomes). Finally, the seven comparator hypertension studies assessed the association between a personalized physician learning intervention [33], provider communication skills intervention [34], telehealth intervention [26,35], electronic health game [36], dietary approaches [37], and a home based digital medicine program [38] and reduction in blood pressure.

Fig. 1a

Flow diagram for the cognitive health study selection process for the systematic search.

Fig. 1b

Flow diagram for the diabetes study selection process for the systematic search.

Fig. 1c

Flow diagram for the hypertension study selection process for the systematic search.

Table 1

Studies included in systematic search.

Study ID	Population	Total, N	Study Design	Review Outcomes reported
Courcoulas et al., 2015	Diabetes	61	RCT	Age, sex, geography, recruitment method, recruitment time
de vries McClintock et al., 2015	Diabetes	180	RCT	Age, sex, race, geography, recruitment method, recruitment time
Delahanty et al., 2015	Diabetes	57	RCT	Age, sex, race, geography, recruitment method, recruitment time
Edelman et al., 2015	Diabetes & Hypertension	377	RCT	Age, sex, race, geography, recruitment method, recruitment time
Eyre et al., 2016	Cognitive Decline	25	RCT	Age, sex, race, geography, recruitment method, recruitment time
Eyre et al., 2017	Cognitive Decline	79	RCT	Age, sex, race, geography, recruitment method, recruitment time
Fedor et al., 2015	Cognitive Decline	60	Observational	Age, sex, race, geography, recruitment method, recruitment time
Greenwood et al., 2015	Diabetes	90	RCT	Age, sex, race, geography, recruitment method, recruitment time
Hickman et al., 2015	Hypertension	144	RCT	Age, sex, race, geography, recruitment method, recruitment time
Hsu et al., 2016	Diabetes	40	RCT	Geography, recruitment method
Innes et al., 2016	Cognitive Decline	60	RCT	Age, sex, race, geography, recruitment method, recruitment time
Liss et al., 2016	Diabetes	331	RCT	Age, sex, race, geography, recruitment method, recruitment time
Manze et al., 2014	Hypertension	203	RCT	Sex, race, geography, recruitment method, recruitment time
Margolis et al., 2015	Hypertension	403	RCT	Age, sex, race, geography, recruitment method, recruitment time
Milani et al., 2017	Hypertension	556	Observational	Age, sex, race, geography, recruitment method, recruitment time
O'Connor et al., 2014	Hypertension	4568	RCT	Age, sex, race, geography, recruitment method
Quinn et al., 2016	Diabetes	118	RCT	Age, sex, race, geography
Rock et al., 2014	Diabetes	227	RCT	Age, sex, race, geography, recruitment method, recruitment time
Sayer et al., 2015	Hypertension	19	Random crossover	Age, sex, geography, recruitment time

Syntheses of results

Table 2a, Table 2b, Table 2ca–c present the data points extracted from each of the 19 included studies. Regarding sample demographic characteristics, 17 of the 19 included studies reported a mean and standard deviation for age, 18 of 19 reported sex, 16 of 19 reported race (two of which were obtained by emailing study authors), and all 19 reported the distribution of state of residence for the study population. Regarding recruitment, 17 of 19 included studies reported the method of recruitment used, and 15 of 19 included studies provided data on the duration of recruitment in months (seven of which were obtained via email to study authors).

Table 2a

Extracted data points from cognitive health studies.a

Study Name	Virtual study on cognitive health16	Eyre et al.,J of Alzheimer's Disease 2016		Eyre et al.,Int Psychogeriatr 2017		Fedor et al.,Arch Clin Neuropsych 2015		Innes et al.,Comp Ther Med 2016
Study Arm		Intervention	Control	Intervention	Control	Intervention	Control
Sample Size	n = 82	n = 14	n = 11	n = 38	n = 41	n = 27	n = 33	n = 60
Age, Mean±SD	64± 4	67±10	68±10	68±9	68±8	63±8	66±7	61±1
Sex, n (%)
Female	61 (74)	6 (43)b	6 (55)b	25 (66)	27 (66)	21 (78)	25 (76)	51 (85)
Male	20 (24)	8 (57)	5 (45)	13 (34)	14 (34)	6 (22)	8 (24)	9 (15)
Not listed	1 (1)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Race/Ethnicity, n (%)
White	72 (88)	12 (86)	8 (73)	24 (63)c	30 (73)c	26 (96)d	33 (100)d	56 (93)
Another race	10 (12)	2 (14)	3 (27)	14 (37)	11 (27)	1 (4)d	0 (0)d	4 (7)
Geographic distribution
Number of states represented	29	1		1		1		1
Participant Recruitment
Where recruited	Online study platform (Achievement)	Outpatient clinics, longevity center, community		Outpatient clinics, longevity center, community		Community recreation and wellness centers		Community, health care, and workplace settings
Recruitment method	Online	Advertisements		Advertisements		Flyers		Flyers and emails
Recruitment time	4mo	24mod		24mo		12mod		5mod
Avg number of participants recruited per month	21	1		3		5		12

Because of rounding, n's may not be exact.

p < 0.05 as compared to virtual study.

p < 0.01 as compared to virtual study.

Not specified in publication, but confirmed via email with corresponding author.

Table 2b

Extracted data points from diabetes studies.a

Study Name		Virtual study on diabetes management14		de vries McClintock et al.,J Behav Med 2015		Courcoulas et al., JAMA 2015	Delahanty et al.,Obesity 2015			Edelman et al., J Gen Int Med 2015
Study Arm				Control	Intervention		Control	Intervention		Control		Intervention
Sample Size		n = 146		n = 88	n = 92	n = 61	n = 29	n = 28		n = 184		n = 193
Age, Mean±SD		52±9		57±10	58±9	47±7	61±11	62±10		60±11		58±11
Sex, n (%)
Female		103 (71)		58 (66)	64 (70)	50 (82)	12 (41)b	11 (39)b		~101 (55)b		~104 (54)b
Male		43 (30)		30 (34)	28 (30)	11 (18)	17 (59)	17 (61)		~83 (45)		~89 (46)
Not listed		0 (0)		0 (0)	0 (0)	0 (0)	0 (0)	0 (0)		0 (0)		0 (0)
Race/Ethnicity, n (%)
White		107 (73)		36 (41)b	29 (32)b	48 (79)	18 (64)	24 (86)		~93 (50)b		~95 (49)b
Another race		39 (27)		52 (59)	63 (68)	13 (21)	11 (36)	4 (14)		~91 (50)		~98 (51)
Geographic distribution
Number of states represented		45		1		1	1			1
Participant recruitment
Where recruited		Online study platform (Achievement)		Primary care practices		Academic medical center	Primary care practices			Primary care practices
Recruitment method		Online		Medical records		Advertisements (TV, print, internet)	Medical records			Medical records
Recruitment time		3mo		12mo		18moc	10mo			25mo
Avg number of participants recruited per month		49		15		2	6			15
Table 2b(continued). Extracted data points from diabetes studiesa
Study Name		Greenwood et al.,J of Med Int Res 2015		Hsu et al.,Diabetes Tech & Ther 2016		Liss et al.,Contemp Clin Trials 2016	Quinn et al.,J of App Ger 2016d			Rock et al.,Diabetes Care 2014
Study Arm		Control	Intervention	Control	Intervention		Age <55 years		Age ≥55 years	Low-Fat Diet	Low-Carb Diet		Control Diet
Sample Size		n = 45	n = 45	n = 20	n = 20	n = 331	n = 37		n = 25	n = 74	n = 77		n = 76
Age, Mean±SD	58±11		54±10	54	53	57±11	47±7		59±3	56±9	57±9		57±9
Sex, n (%)
Female	19 (42)b		23 (51)b	–	–	166 (50)b	23 (62)e		8 (32)e	35 (47)b	37 (48)b		44 (58)b
Male	26 (58)		22 (49)	–	–	165 (50)	14 (38)		17 (68)	39 (53)	40 (52)		32 (42)
Not listed	0 (0)		0 (0)	–	–	0 (0)	0 (0)		0 (0)	0 (0)	0 (0)		0 (0)
Race/Ethnicity, n (%)
White	27 (60)		29 (64)	–	–	114 (34)b	20 (54)		19 (76)	59 (80)	63 (82)		59 (78)
Another race	16 (36)f		16 (36)f	–	–	217 (66)	17 (46)		6 (24)	15 (20)	14 (18)		17 (22)
Geographic distribution
Number of states represented	1			1		1	1			2
Participant recruitment
Where recruited	Large health care system			Tertiary diabetes care center		Primary care practices	Community physician practice groups			Universities
Recruitment method	Medical records			Health care providers		Primary care providers and medical records	–			Word of mouth, direct marketing letters mailed to large cohorts, radio ads, local e-mail subscription services, ClinicalTrials.gov, social media, and flyers
Recruitment time	–			–		19mo	–			6mo
Avg number of participants recruited per month	–			–		17	–			38

Because of rounding, n's may not be exact

p ≤ 0.001 as compared to virtual study

Not specified in publication, but confirmed via email with corresponding author

Reporting on intervention group only due to reporting limitations in original publication

p ≤ 0.01 as compared to virtual stud

Missing data for some participants.

Table 2c

Extracted data points from hypertension studies.a

Study Name	Virtual study on hypertension15		Edelman et al., J Gen Int Med 2015		Hickman et al., 2015				Manze et al., 2014
Study Arm			Control	Intervention	Intervention		Control
Sample Size	n = 411		n = 184	n = 193	n = 97		n = 47		n = 203
Age, Mean±SD	52±10		60±11	58±11	49±13		47±14		61 (range: 29–88)
Sex, n (%)
Female	247 (60)		~101 (55)	~104 (54)	57 (59)		31 (66)		147 (72)b
Male	162 (39)		~83 (45)	~89 (46)	40 (41)		16 (34)		56 (28)
Not listed	2 (1)		0 (0)	0 (0)	–		–		–
Race/Ethnicity, n (%)
White	268 (65)c		~93 (50)c	~95 (49)c	34 (24)c				68 (34)c
Another race	143 (35)		~91 (50)	~98 (51)	110 (76)				135 (66)
Geographic distribution
Number of states represented	46		1		1				1
Participant Recruitment
Where recruited	Online study platform (Achievement)		Primary care practices		Community				Primary care clinics
Recruitment method	Online		Medical records		Newspaper, business and local bus advertisements				Medical records
Recruitment time	5mo		25mo		10 mo				19mo
Avg number of participants recruited per month	82		15		16				11
Table 2c (continued). Extracted data points from hypertension studiesa
Study Name	Margolis et al., 2015		Milani et al., 2017		O'Connor et al., 2014				Sayer et al., 2015
Study Arm	Intervention	Control	Intervention	Control	PPL-EMR	PPL-Assess		Control
Sample Size	n = 206	n = 197	n = 156	n = 400	n = 1594	n = 1501		n = 1473	19
Age, Mean±SD	62±11	60±12	68 ± 10	68 ± 10	61±15	60±15		60±16	61±2
Sex, n (%)
Female	88 (43)c	86 (44)c	84 (54)d	216 (54)d	886 (56)c	758 (51)c		795 (54)c	13 (68)
Male	118 (57)	111 (56)	72 (46)	184 (46)	708 (44)	743 (49)		678 (46)	6 (32)
Not listed	–	–	–	–	–	–		–	0 (0)
Race/Ethnicity, n (%)
White	335 (83)c		120 (77)c,e	308 (76)c,e	1321 (83)c	1202 (80)c		1209 (82)c	–
Another race	68 (17)		36 (23)e	92 (24)e	273 (17)	299 (20)		264 (18)	–
Geographic distribution
Number of states represented	1		1		2				1
Participant Recruitment
Where recruited	Primary care clinics		Community health system		Primary care clinics				Community
Recruitment method	Medical records		Medical records		Medical records				–
Recruitment time	25moe		13moe		–				16moe
Avg number of participants recruited per month	16		43		–				1

Because of rounding, n's may not be exact.

p ≤ 0.01 as compared to virtual study.

p ≤ 0.001 as compared to virtual study.

p < 0.05 as compared to virtual study.

Not specified in publication, but confirmed via email with corresponding author

Comparisons based on demographic characteristics

The mean age of study participants in the virtually recruited cognitive health study did not differ from the pooled mean age across the four comparator studies (64 versus 65, p = 0.12). As compared to the eight studies related to diabetes that reported mean and standard deviation for age, the virtually recruited diabetes study was five years younger, on average (52 versus 57 years, p < 0.001). Similarly, as compared to the six hypertension studies with the requisite data on age, the virtually recruited hypertension sample was nine years younger on average (52 versus 61 years, p < 0.001).

In over half (n = 11) of the 18 studies that reported sex, the proportion of participants reported as female statistically significantly differed between the traditionally versus the virtually recruited samples. In 10 of the comparator studies, the percentage of female participants was lower compared to that in the virtually recruited studies. In one hypertension study, the virtually recruited sample had a lower percentage of female participants.

Of the 17 comparator studies that reported data on the racial identity (white versus another race) of study participants, the virtually recruited samples statistically significantly differed from the comparator samples in 10 instances. In six cases, the virtually recruited sample was less diverse (fewer participants of another racial identity) than the traditionally recruited sample, and in four cases, the virtually recruited sample was more diverse.

Study samples were also compared based on the distribution of state of residence of included participants. As compared to the 19 comparator studies that enrolled participants from an average of 1.1 states (range: 1–2), the three virtual studies included participants from an average of 40 states (range: 29–46) (p-value for difference = 0.02). A non-parametric Wilcoxon-Mann-Whitney (or rank sum) test of distributions confirmed the difference in number of included states (p < 0.01).

Comparisons based on speed of recruitment

The three virtual studies took an average of 4.0 months (SD 1) to recruit and enroll the target sample size, in contrast to an average of 15.9 months (SD 7) for the comparator studies (p-value for difference<0.01). A test of median recruitment time between virtual and comparator studies confirms a similar difference (virtual median = 4 months, traditional median = 16 months; p-value for difference<0.01).

Similarly, virtual studies enrolled a higher mean and median number of participants per month as compared to comparator studies. The virtual studies enrolled an average of 51 participants per month (SD 31), as compared to 13 participants per month (SD 12) among the 15 comparator studies (p-value for difference<0.01) (Fig. 2). The virtual studies enrolled a median of 49 participants per month as compared to a median of 12 participants per month for the comparator studies (p-value for difference = 0.02).

Fig. 2

Number of participants enrolled per month by study type.

Discussion

In this systematic search of the literature, we identified 19 studies in cognitive health, diabetes, and hypertension that relied on traditional recruitment methods and compared the resulting study samples to three virtually recruited samples on several demographic and timing metrics. Our goal was to report observational differences between traditionally and virtually recruited samples. We found that where traditionally recruited samples differed from virtually recruited samples, the virtually recruited samples were slightly younger, enrolled a higher percentage of female participants, and were split on enrollment of minorities (the virtual samples were less diverse in 32% of comparisons, but more diverse in 21% of comparisons). The virtually recruited samples outperformed the traditionally recruited samples in terms of geographic spread of participants, overall recruitment time, and average number of participants recruited per month. While some of the observed differences could be explained by confounding factors, our comparison approach nevertheless provides a view into correlational differences. The findings regarding geographic spread and recruitment volume per month, in particular, support the hypothesis that virtual recruitment methodologies may help facilitate opportunities for more individuals to participate in research studies.

A main goal of most studies is to recruit a representative sample so that the results can be applied to the population that the study is intended to help. Given the increasing costs of clinical research in recent years coupled with difficulty in achieving the required number of research participants, attention has focused on ways to increase research participation and improve participant recruitment [9]. Investigators have called for the development of novel and diverse strategies to expand the pool of eligible research participants to address this issue. Findings presented in this study provide encouraging evidence that virtual recruitment may be one such strategy.

Historically, clinical research has suffered from underrepresentation of women in study samples [39]. While gender identity is not always consistent with sex, the higher proportion of female participants in the virtually recruited studies assessed here could represent an important opportunity to address the gaps in our knowledge about women and other people assigned female sex at birth's unique response to treatments and disease trajectories. Further, as these results suggest that virtual studies may in some instances be less diverse in terms of the racial composition of samples, investigators can use this knowledge to carefully consider strategies for increasing outreach and recruitment of individuals from underrepresented populations that could address this potential limitation. Both researchers and designers of healthcare interventions alike should consider the advantages that virtual recruitment may offer for increasing the efficiency and impact of health research with regard to sample recruitment.

As with all research, this study has limitations. The three virtual studies assessed were all designed and run by one digital health company (Evidation Health, Inc., San Mateo, CA). Recruitment metrics from these study samples may therefore not be representative of other virtually recruited studies conducted by other entities. Further, the literature review was limited to peer-reviewed publications included in the PubMed database. While PubMed is considered the largest database for peer-reviewed articles in health research and is updated more frequently and with a broader scope than comparator databases such as MEDLINE [40], it is still possible that we may have missed relevant comparator studies by relying solely on PubMed. This review also only focused on studies run in the United States; there may be additional efforts taking place in other countries that could serve as beneficial comparison points. Some of the measures compared across studies could be influenced on a study-by-study basis by specific inclusion and exclusion criteria or recruitment targeting strategies (e.g., age restrictions). Along the same lines, characteristics of individuals who enroll into research studies vary widely by the therapeutic area and population of interest, and there can be differences in results among individuals with varying demographic characteristics. Therefore, findings from these three therapeutic areas may not be generalizable to other therapeutic areas. Lastly, the three virtual studies presented here not only utilized virtual recruitment methods, but were completely virtual in nature (i.e., allowed participants to participate in the study remotely, had no in-person or site-based study procedures). Part of the improved efficiency in recruitment and geographic diversity in the three studies that utilized virtual recruitment methods may be attributed to participants' increased willingness to participate in a study remotely. These study design characteristics would influence the values that then were compared across studies.

Conclusions

This is the first study, to our knowledge, to compare participant recruitment metrics between virtual and traditional recruitment methodologies. We adopted a clear and replicable study design so that similar studies can be conducted to compare metrics for other virtual studies as they are published, adding to the evidence base for the strengths and limitations of virtual recruitment for health research.

Although this study is by no means an exhaustive analysis of all virtual samples, it serves as a useful first contribution to improving our understanding of the opportunities that virtual studies may offer for enhancing efficiency and empowering a wider number of individuals to participate in health research. We hope that the findings presented here provide a useful quantification of the comparison of virtual and traditional recruitment, and will inspire future research to compare a wider range of studies on these metrics, as well as additional metrics that we were not able to assess, such as the overall cost of recruitment. While much work remains to be done, we are optimistic that virtual recruitment may help to accelerate the adoption of urgently needed health solutions for a wider range of people.

Declaration of competing interest

The authors declare that there is no conflict of interest.