Feasibility of recruitment and retention in a remote trial of gatekeeper training for close supports of military veterans: Mixed methods study.

Background: VA S.A.V.E. (Signs; Ask; Validate; Encourage/Expedite) is a gatekeeper training, designed to teach individuals how to identify and assist military veterans at risk for suicide. The aim of this pilot was to determine feasibility, barriers, and facilitators of recruitment and retention in a remote trial of VA S.A.V.E.
Methods: We recruited close supports (family and friends) of veterans through Facebook sponsored ads, automatically randomized them to VA S.A.V.E. or an unrelated video training, and followed them for six months. A subgroup completed interviews, and we used a mixed methods framework to integrate quantitative and qualitative findings.
Results: Of 214 participants, 61% were spouses or partners of veterans and 48% knew at least one veteran who had died by suicide. Of the three a priori feasibility benchmarks, two were achieved (enrollment, on average, of twenty participants per week and less than 50% loss to follow-up at the 6-month study endpoint) and one was not (enrollment of at least 50% of eligible individuals). There were three barriers (generic ads, ad text referring to "research," and Facebook as an ad platform) and five facilitators (audience segmentation focused on veterans' family members and friends, an urgent call to action to help a veteran, prior exposure to suicide, emphasizing the benefit of receiving training, and using a university as the campaign messenger) to study participation.
Conclusion: A fully remote trial of VA S.A.V.E. gatekeeper training was feasible in a population of close supports of veterans. Several strategies may further enhance study participation. Published by Elsevier Inc.

Clinical trial registration atclinicaltrials.gov: Trial number NCT04565951.

Background

National data suggest that Americans “overwhelmingly agree” they have an important role to play in preventing suicide and most (78%) are also interested in learning how they might be able to help someone who may be suicidal [1]. Gatekeeper training is a widely used public health intervention strategy designed to help individuals fulfill such a role, and typically does so by teaching them (“gatekeepers”) how to identify a person at risk of suicide and facilitate seeking professional mental health evaluation and treatment.

Despite widespread availability and implementation of numerous versions of gatekeeper training [[2], [3], [4]] and prioritization of help-seeking as a focus of research in suicide prevention [5], there is a lack of randomized controlled trials (RCT's) of gatekeeper training for prevention of adult suicidal behavior [6]. Of the few available RCT's identified in a recent systematic review, all were given low or unclear quality scores [7]. Furthermore, research on veteran-specific gatekeeper training is indicated since approximately 14% of the over 40,000 annual suicides in the United States each year occur among military veterans [8]. The observed age-adjusted suicide rates among Veterans consistently exceed those of the general population in the United States [[9], [10], [11]]. According to work by Bullman and Schneiderman (2021) male and female Veterans were at 40% and 85% greater risk of suicide, respectively, compared to their civilian counterparts [12].

Clinical settings are frequently used to recruit patient populations, but reaching individuals who have not sought care requires a different approach. This is particularly true for individuals experiencing psychiatric problems, including suicidal ideation, as they are often reluctant to seek care [13]. Friends and family of those at risk for suicide represent a logical target for gatekeeper training because individuals having frequent social contact with at-risk individuals may have the greatest potential to impact suicide prevention [14]. To reach friends and family of veterans, we turned to social media. Existing studies on gatekeeper training generally relied on a mixture of difficult-to-scale strategies for recruitment, such as community outreach events, newsletters, and mailings [[15], [16], [17]]. However, few have utilized social media as a primary recruitment strategy to reach national samples [7,18].

VA S.A.V.E. (Signs; Ask; Validate; Encourage and Expedite) is a gatekeeper training specifically tailored to the military veteran community and used extensively by the Veterans Health Administration (VHA). VA S.A.V.E. is available in a brief, online video format developed in partnership with the nonprofit PsychArmor Institute [19] and offering potential for widespread dissemination in veteran communities. Prior, more resource-intensive iterations of VA S.A.V.E., such as “Operation S.A.V.E.,” have been examined in VA employees and community leaders and shown to increase these gatekeeper's knowledge about suicide and confidence in their ability to assist at-risk Veterans [20,21]. However, the impact of the brief, online video format VA S.A.V.E. in a lay population is unknown.

The primary objective of this study was to determine the feasibility of using a paid social media campaign to recruit and retain participants in a remote trial [22] of VA S.A.V.E. We also sought to identify key barriers and facilitators to recruitment and study participation. We established three feasibility benchmarks [23], to inform whether to subsequently proceed with a full-scale RCT of VA S.A.V.E.: 1) enroll twenty participants per week, on average; 2) enroll at least 50% of eligible individuals; and 3) less than 50% will be lost to follow-up at the 6-month study endpoint.

Methods

Setting and participants

This study was fully remote, meaning recruitment procedures, intervention delivery, and outcome measurement all occurred without in-person interaction. To be eligible, participants had to: 1) be a family member or friend of a veteran, and 2) have social contact with a veteran at least once a week. Individuals were excluded if they: 1) did not provide a U.S. phone number and email address, 2) lacked computer access, 3) were not fluent in English, 4) had previously taken or intended to take VA S.A.V.E. training, or 5) provided indication of duplicate study entry or misrepresentation of their status as a veteran. The study was approved by our local institutional review board and registered at ClinicalTrials.gov (Trial number NCT04565951).

Current study

The current analysis was part of a randomized controlled pilot trial of VA S.A.V.E. In this study, participants were randomized to one of two study arms, both of which consisted of an approximately 24-min online training video, either VA S.A.V.E. or an attention control [24] unrelated to suicide prevention. The VA S.A.V.E. video included content addressing myths about suicide, the S.A.V.E. model, and suicide prevention and mental health resources. The attention control video, also created by PsychArmor Institute, included content on helping veterans prepare their finances, network, search for a job, and use educational benefits.

Campaign development

Working with the Strategic Communications Social Media team at our university affiliate, we designed a paid social media campaign, largely following procedures developed in our previous research [25]. We selected Facebook based on our prior experience with it leading to higher average click-through-rates and lower cost-per-click than other platforms. The campaign ran from March 13, 2021 to May 24, 2021. We designed a series of sponsored ads that varied in composition of text and image in order to target a national sample of individuals likely to meet our main eligibility criteria based on information listed in their public profile such as location, age, gender, interests, and employers. Two distinct ad sets and audience segments were created to reach: 1) military veterans and 2) close supports. See Fig. 1 for an example of two ads used in this campaign. These audiences were refined as the campaign progressed to adjust for age, sex and interest in order to reach more veterans.

Fig. 1

Examples of sponsored ads in the close support (LEFT) and veteran (RIGHT) audience segments.

Data collection

After interested individuals clicked on a sponsored ad, they entered the study website, managed through REDCap [26]. After anonymously responding to screening questions, individuals completed an online informed consent process, were automatically randomized into one of the two study arms, and were asked to provide contact information for follow-up. Next, participants completed a baseline survey and received exposure to the video of their assigned study arm. The video was presented on a separate webpage from other survey content and we used the amount of time participants spent on this page as an indicator of how long each participant watched the video. A limitation of this measure is that a participant could have loaded the page but then walked away or otherwise not actually watched the video. We believe this type of behavior was rare because there were few outliers (only 3 of the 102 participants assigned to the 24-min VA S.A.V.E. video spent more than an hour on the page). Follow-up study measures were administered as surveys immediately after training (post-training), as well as monthly for six months. Survey length was intentionally kept short to minimize burden on participants. Participants received up to two automated reminders via email or text message, with an additional phone reminder to non-responders. Participants from the intervention group who watched at least 5 min of the VA S.A.V.E. video were invited to participate in a qualitative interview, which included questions about VA S.A.V.E. and about sponsored ads used in the campaign. We sought a variety of perspectives based on differing gatekeeper behaviors, veteran status, service branch, gender, and education. Interviews took place approximately one month after exposure, and fifteen participants completed interviews. Participants received a $20 incentive for participation in the surveys and $25 for completing an interview.

Measures

We assessed feasibility by measuring engagment with our sponsored ads, recruitment, and retention throughout the study. A measure of engagement with sponsored ads was provided by Facebook, including impressions (number of times an ad is presented to Facebook users), clicks, click-through-rate (clicks divided by impressions), and costs. Recruitment measures included the number of individuals screened, eligible, and enrolled, as well as the proportion eligible who enrolled. Retention was measured four ways: 1) percent of participants who completed follow-up at Month 3 or later; 2) percent of participants who completed follow-up at Month 6 (study endpoint); 3) percent of follow-up surveys completed; and 4) cumulative number of follow-up surveys completed. Sociodemographic (age, gender, race, ethnicity, education, employment status, veteran status) and other characteristics (relationships with veterans and exposure to suicide) of participants were self-reported. Semi-structured qualitative interviews, which lasted approximately 1 h, were conducted by a single research team member experienced in qualitative methods and followed an interview guide that probed reactions/responses to the social media campaign, specific sponsored posts used in the campaign, and VA S.A.V.E. training.

Data analysis

We used an intervention mixed-methods framework [27] to assess two research questions: 1) Can we recruit and retain appropriate participants? and 2) What are barriers and facilitators to recruitment and study participation? Research question 1 was assessed using data on recruitment, participant characteristics, and retention, while research question 2 was primarily assessed using data from the qualitative interviews. We incorporated a convergent design (quantitative and qualitative data collected and analyzed during a similar timeframe) in the context of delivering our intervention. Mixed-methods integration was achieved through narrative using a contiguous approach, in which we present quantitative findings (recruitment, retention, characteristics of participants), followed by qualitative findings (barriers and facilitators to study participation).

As a check on randomization, we examined differences in baseline characteristics by group, using chi-square tests for categorical variables and t-tests for continuous variables. Using the same methods, we also compared characteristics between individuals who enrolled and those who did not, among eligible individuals. Analyses were performed in Stata Version 16 (StataCorp). For qualitative data, we transcribed audio recordings of interviews, then conducted qualitative content analysis with a directed grounded theory approach [28]. This approach allowed us to identify tacit and implicit content in transcripts [29]. Each transcript was double-coded in Atlas.ti Version 9 using a codebook developed and iterated by the investigative team. After coding was complete, the research team reviewed code reports and wrote conceptual memos to identify and track developing themes. An initial review of these themes led the research team to develop an analytic framework focused on barriers and facilitators to study participation. We used this framework to refine key themes and identify illustrative quotations.

Results

Research question 1: can we recruit and retain appropriate participants?

Recruitment

Fig. 2 shows recruitment results over the 10-week, paid social media campaign, resulting in a total enrollment of 281 participants. We thus met our first feasibility benchmark to enroll, on average, twenty participants per week. The number of eligible individuals fluctuated markedly during this time. Iterations to the campaign were made as campaign engagement and recruitment data were reviewed on a weekly basis; at one point the campaign was paused to allow time for more detailed review. Despite these fluctuations, the enrollment percentage remained in a relatively narrow band; the overall proportion of eligible individuals who enrolled in the study was 21.8% (281/1278). We thus failed to meet feasibility benchmark 2) enroll at least 50% of eligible individuals.

Fig. 2

Recruitment data and key campaign milestones (A through G) during the 10-week paid media campaign (March 13, 2021 to May 24, 2021). Results are shown as n. A: Recruitment launched targeting Veteran and Close Support audiences, each with two different ads; B: Campaign paused, C: Recruitment re-launched for Veteran audience only with four new ads; D: Recruitment re-launched for close support audience using original ad; E: Veteran audience ads discontinued; F: Budget adjusted to be split more evenly between Veteran and Close Support ads – both targeting the Close Support audience; G: Recruitment ended.

Participant flow through the study is shown in the CONSORT diagram in Fig. 3. In response to the paid social media campaign, 1540 completed eligibility screening (6.1% of clicks) and 434 consented (34.0% of eligible). Participants were automatically randomized immediately after completing the online consent process but before providing information necessary for enrollment (e.g., contact information). Consequently, 281 (64.7% of consented) were deemed “enrolled,” and of these, 214 responded to the baseline survey. This group of 214 participants (n = 102 in the VA S.A.V.E. arm, and n = 112 in the control arm) constituted the primary analytic sample.

Fig. 3

Participant flow through the study.

Characteristics of study participants

Among all eligible individuals (n = 1278) we compared those who enrolled to those who did not. Among the measured characteristics (number of veterans one is close to, frequency of contact with veterans, and veteran status), the only significant difference was veteran status, such that veterans were more likely to enroll than non-veterans or current active-duty service members (p = 0.004).

Baseline characteristics of participants including demographics, veteran characteristics, and suicide exposure are shown in Table 1. No significant differences (p > 0.05) were found between the VA S.A.V.E. and control arms on these characteristics. The majority of participants were women (189/214, or 88.3%), white (203/214, or 94.9%), and non-Hispanic (204/214, or 95.3%). The majority were not veterans themselves (146/214, or 68.2%) but had contact with at least one veteran several times a day (143/214, or 66.8%). More than half of the participants identified as the spouse or romantic partner of a veteran (130/214, or 60.8%). History of suicide exposure was substantial: 96.2% (203/214) of participants had ever worried that someone they knew was thinking about suicide, 76.2% (163/214) knew at least one person who died by suicide, and 47.7% (102/214) knew at least one veteran who died by suicide.

Table 1

Participant characteristics at baseline. Results presented as n (%) or mean (±SD).

	Both arms (n = 214)		Control arm (n = 112)	VA S.A.V.E. arm (n = 102)	p-value
Demographic characteristics
Age (years)	54.3 (±13.3)		53.1 (±13.8)	55.6 (±12.7)	0.19
Gender
Male/man	22 (10.3)		11 (9.8)	11 (10.8)	0.86
Female/woman	189 (88.3)		99 (88.4)	90 (88.2)
Genderqueer/non-binary	3 (1.4)		2 (1.8)	1 (1.0)
Race					0.63
American Indian or Alaska Native	7 (3.3)		3 (2.7)	4 (3.9)
Asian	4 (1.9)		1 (0.9)	3 (2.9)
Black or African American	4 (1.9)		3 (2.7)	1 (1.0)
White or Caucasian	192 (89.7)		102 (91.1)	90 (88.2)
Missing/other	7 (3.3)		3 (3.7)	4 (3.9)
Ethnicity					0.99
Non-Hispanic	204 (95.3)		107 (95.5)	97 (95.1)
Hispanic	8 (3.7)		4 (3.6)	4 (3.9)
Missing	2 (0.9)		1 (0.9)	1 (1.0)
Education
High school diploma or less	15 (7.0)		7 (6.3)	8 (7.8)	0.60
Post high school or some college	95 (44.4)		53 (47.3)	42 (41.2)
Bachelor's degree or higher	103 (48.1)		51 (45.5)	52 (51.0)
Missing	1 (0.5)		1 (0.9)	0
Employment status					0.47
Working	102 (47.7)		54 (48.2)	48 (47.1)
Not working but looking for work	10 (4.7)		3 (2.7)	7 (6.9)
Not working and not looking for work	96 (44.9)		51 (45.5)	45 (44.1)
Missing	6 (2.8)		4 (3.6)	2 (2.0)
Veteran status					0.79
No, never served	146 (68.2)		77 (68.8)	69 (67.7)
Yes, now on active duty	8 (3.7)		5 (4.5)	3 (2.9)
Yes, on active duty in the past	60 (28.0)		30 (26.8)	30 (29.4)
Relationship to closest Veteran					0.49
Spouse or romantic partner	130 (60.8)		70 (62.5)	60 (58.8)
Parent	24 (11.2)		10 (8.9)	14 (13.7)
Child	17 (7.9)		7 (6.3)	10 (9.8)
Other family member	18 (8.4)		10 (8.9)	8 (7.8)
Friend	19 (8.9)		10 (8.9)	9 (8.8)
Other	6 (2.8)		5 (4.5)	1 (1.0)
Suicide exposure
Number of people you worried were thinking about suicide, ever					0.78
0	7 (3.3)		4 (3.6)	3 (2.9)
1	16 (7.5)		7 (6.3)	9 (8.8)
2	41 (19.2)		22 (19.6)	19 (18.6)
3	39 (18.2)		18 (16.1)	21 (20.6)
4	22 (10.3)		11 (9.8)	11 (10.8)
5	36 (16.8)		19 (17.0)	17 (16.7)
6-100	49 (22.9)		30 (26.8)	19 (18.6)
Missing	4 (1.9)		1 (0.9)	3 (2.9)
Known a person who died by suicide, ever					0.60
No	39 (18.2)		18 (16.1)	21 (20.6)
Yes	163 (76.2)		86 (76.8)	77 (75.5)
Not sure	10 (4.7)		7 (6.3)	3 (2.9)
Missing	2 (0.9)		1 (0.9)	1 (1.0)
Number of people you know who died by suicide, ever					0.46
0	39 (18.2)		18 (16.1)	21 (20.6)
1	47 (22.0)		23 (20.5)	24 (23.5)
2	46 (21.5)		22 (19.6)	24 (23.5)
3	34 (15.9)		17 (15.2)	17 (16.7)
4	11 (5.1)		6 (5.4)	5 (4.9)
5	11 (5.1)		7 (6.3)	4 (3.9)
6-22	14 (6.5)		11 (9.8)	3 (2.9)
Not sure or missing	12 (5.6)		8 (7.1)	4 (3.9)
Of those, number who were Veterans or were in the military					0.60
0	57 (26.6)		31 (27.7)	26 (25.5)
1	61 (28.5)		29 (25.9)	32 (31.4)
2	19 (8.9)		9 (8.0)	10 (9.8)
3	10 (4.7)		7 (6.3)	3 (2.9)
4	6 (2.8)		4 (3.6)	2 (2.0)
5	4 (1.9)		3 (2.7)	1 (1.0)
7-10	2 (0.9)		2 (1.8)	0
N/A	55 (25.7)		27 (24.1)	28 (27.5)

Retention

Across both study arms, 81.3% (174/214) of participants completed at least one follow-up survey at Month 3 or later, and 72.4% (155/214) completed the Month 6 follow-up. We thus met our third feasibility benchmark (less than 50% will be lost to follow-up at the 6-month study endpoint). Overall, participants responded to 69.4% of follow-up surveys, with a significantly higher rate in the VA S.A.V.E. arm compared to the control arm (79.2 vs 60.4%, p < 0.001). The cumulative number of follow-up surveys completed from Months 1 through 6 is shown in Table 2. Across both study arms, 86.0% responded to at least one follow-up survey, 73.8% responded to half (at least three), and 46.3% (99/214) responded to all six follow-up surveys.

Table 2

Cumulative number (%) of surveys responses during six months of monthly follow-up.

	Both arms (n = 214)		Control arm (n = 112)	VA S.A.V.E. arm (n = 102)
None	30 (14.0)		22 (19.6)	8 (7.8)
At least 1	184 (86.0)		90 (80.4)	94 (92.2)
At least 2	170 (79.4)		81 (72.3)	89 (87.3)
At least 3	158 (73.8)		72 (64.3)	86 (84.3)
At least 4	147 (68.7)		64 (57.1)	83 (81.4)
At least 5	133 (62.2)		56 (50.0)	77 (75.5)
All 6	99 (46.3)		43 (38.4)	56 (54.9)

Research question 2: what are barriers and facilitators to recruitment and study participation?

Interviews with participants revealed that barriers and facilitators had direct influences on the degree to which the campaign captured their attention, garnered their trust in the study, and motivated them to follow-through with participation in the study. We identified three barriers and five facilitators.

Barrier 1: generic-looking sponsored ads

The first barrier was that sponsored ads for this study were at times difficult to distinguish from other sponsored ads targeting service members and veterans. Participants commented on being frequently exposed to such ads and that ours “blends in with everything else” (848). Some participants noted similarities to commercial advertisements from financial service institutions that target Veterans, such as USAA or Navy Federal:

•(1441) “I don't know if you've noticed, but a lot of your ads look like home loan ads, like the VA home loans. That family one specifically, really looks like that, so I have to look really close to see it's not a home loan.”

Barrier 2: copy referring to “research”

The second barrier was the use of copy (text contained in the ad) mentioning participation in research, particularly suicide-related research. While some participants reported being attracted to the idea of participating in research, there was concern about being “asked a lot of intrusive questions” (2360). Many participants suggested that “the word ‘research’ turns people off” (35) or simply has negative connotations, as illustrated by this participant:

•(551) “‘Research study’ just slapped me on the face …. You need to show them what they might get out of it.… You wouldn't hide the fact that it’s a research study obviously, but I'm not sure it's as clear to me that you're going to possibly gain some skill or some way to help a fellow military member from that one.”

Barrier 3: facebook as a platform

The last barrier we identified was the set of limitations inherent in using Facebook as a platform for recruitment. “The platform itself isn't designed for long-term focused engagement” (848). Participants reported seeing our sponsored ads in the middle of other activities, or while “taking a break” (1547) and participation involved more time than some anticipated:

•(1841) “I was actually outside smoking a cigarette getting ready to go to bed, and I started this thing up. I didn't realize that there was going to be a 25-min video. I was about 45 min past when I wanted to be in bed because of this.”

There was also concern about the legitimacy of the study because it was posted on Facebook. Participants described a general wariness about trusting what they see on Facebook and potential for scams that makes them hesitant to click on ads:

•(137) “It's really easy to go on Facebook, or Instagram, or whatever social media of your choice, and you can find things like this, but they are really just trying to sell you something or trying to get you to join something.”

The fully online nature of this study compounded this concern, and participants noted caution around providing any personal information online. Many participants recalled taking time to “Google the study” (551) to verify that it was legitimate before participating or completing a follow-up survey.

Finally, use of Facebook for recruitment limited our reach to people who actively use Facebook. While our ad set used features to target younger veterans, use of Facebook may have limited ability to reach younger veterans because “they're less likely to be on Facebook” (86).

Facilitator 1: audience segmentation focused on veterans’ family members and friends

The first facilitator of recruitment and study participation involved targeting close supports of veterans. Sponsored ads containing an image and copy referencing friends and family of veterans did well in capturing the attention and interest of close supports of veterans, as noted by these participants:

(1692) “Because I see my family in there! I see my son, I see his wife …. You see them as human beings and as people. They're not just a number …. I look at that and I see people I know. It's like a more human approach I guess is what I'm trying to say.”

(1841) “My wife is 100% disabled due to PTSD, so this was something I would be interested in and something to take care of my wife.”

While click-through-rates (2.35% for close support ads compared to 2.02% for veteran ads) and average cost-per-click ($0.34 for close support ads compared to $0.33 for veteran ads) were similar across ads targeting these two audiences, study enrollment of close supports was much higher throughout the campaign.

Facilitator 2: an urgent call to action to help a veteran

The second facilitator was presenting participation as an urgent call to action to help another veteran. Participants reported that helping veterans “gives a sense of mission” (2360) and is more motivating “than participating in a research study” (86). Asking to help “a veteran you care about” was seen as a reminder of your “brotherhood or sisterhood” and “helps you realize that you still have each other, and you can still make a difference” (1441). Interview participants reported that they would be even more likely to click on posts that conveyed a sense of urgency by “using key words like depression, anxiety, crisis” (280), or adding action-oriented language such as “save a life tonight” (35).

Facilitator 3: prior exposure to suicide

The third facilitator was prior suicide exposure or knowing someone who had experienced suicidal ideation. This was a strong motivator for participation among veterans themselves and non-veterans who were close supports of a veteran:

•(1441) “When I separated from the military, I already had lost a couple of my close friends in the Marines to suicide … and then I lost a few more when I first got home, so … I would definitely want to participate in something that would help other veterans adjust to their outside life.”

(35) “What motivated me was the VA, and suicide, and the man that I live with has been suicidal. And just the thought of that terrifies me quite frankly. Those two were probably the biggest motivators for me.”

Facilitator 4: emphasizing the benefit of receiving training

The fourth facilitator centered around the benefits of receiving training in suicide prevention. Participants reported wanting to “feel a little bit more in control of what might happen” (1547) if a loved one is thinking of suicide and felt training “can give you some skills and information so that you can make a difference” (86). Participants preferred framing involvement in the study as having a role “to evaluate training,” as opposed to participating in research (2360).

Facilitator 5: using a university as the campaign messenger

The final facilitator was using a university account as the messenger in the campaign. Having the university as the campaign sponsor helped establish legitimacy, trust, and credibility to the study, and this in turn exerted a strong positive influence on engagement with the campaign. Not surprisingly, this effect was most noticeable in the regional community surrounding the university.

In interviews, some participants added that trust could be further increased if the sponsored ads contained the word “hospital” and if they were shared on Facebook by another trusted source or organization, such as the Wounded Warriors Project. While our sponsored ads (intentionally) did not reference the VA, the study landing page and survey pages did so, and this elicited more mixed reactions with regards to trust. Fig. 4 uses a joint display to juxtapose representative quantitative and qualitative information, summarizing these findings in a single figure [30].

Fig. 4

Joint display of: quotations and themes from interviews (TOP); and a heat map showing the number of participants recruited in each state (BOTTOM). Together these data illustrate the influence of geography and institutional awareness on credibility and trust of the campaign.

Discussion

Key findings

In this pilot RCT of VA S.A.V.E. gatekeeper training for suicide prevention, we employed a fully remote trial design, which allowed us to conduct the trial essentially unperturbed by the COVID-19 pandemic. A social media campaign facilitated recruitment, and study procedures—including randomization, intervention delivery, and follow-up assessment—were conducted through a participant-facing, online interface. The scalability of this design allowed us to include many more participants than is common in a pilot of a behavioral intervention [31]. While there was a regional preference linked to the location of our study site, our study sample showed geographical distribution across the United States. Most importantly, we met two of our key a priori feasibility benchmarks; we enrolled and obtained baseline measures from over 21 participants per week of recruitment, and we retained 72% of our baseline sample at the six-month study endpoint. Taken together, these are meaningful accomplishments, particularly when considering the focus of the study was suicide—a highly sensitive topic, albeit a critical public health issue.

While we met our two key a priori feasibility benchmarks for recruitment and retention, we fell short of our goal of enrolling at least 50% of eligible individuals. In hindsight, this was likely too ambitious, given the nature of our study design and online privacy concerns highlighted in our qualitative findings (individuals could complete the brief 11-item screener anonymously but were required to provide identifying information in order to proceed with enrollment).

Guidance for future recruitment campaigns

Our campaign was most successful reaching family members of veterans, especially spouses, the vast majority of whom had prior exposure to suicide. This represents a key target population for gatekeeper training as these individuals have direct contact and connection with veterans at risk of suicide in ways that clinicians and healthcare systems like the VA are unlikely to. It is unclear why participation among veterans was relatively low. Campaign ads targeting veterans appeared to perform similarly to those targeting close supports of veterans, having similar click-through-rates and cost-per-click. Yet, this engagement with the campaign translated into relatively little enrollments among veterans. This phenomenon was also observed in a prior study recruiting partners of heavy drinking military service members [32]. Our findings also highlight the value of robust engagement of stakeholders in the design of recruitment campaign ads. Of note, we did involve veterans in developing recruitment campaign ads. Specifically, we presented draft recruitment materials to a veteran engagement board, and the study coordinator (AC) is a Navy veteran. However, even more robust stakeholder engagement—and partnering with participant-centric organizations where possible—is recommended for others conducting similar studies to bolster trust and credibility among the target audience [33]. For instance, stakeholder engagement might identify language such as “mission” and “brotherhood/sisterhood” that can be effectively deployed in veteran-centric studies. This sense of unity might also help address participant concerns regarding negative connotations associated with “research.” Stakeholders may also offer recommendations of national, veteran organizations with high name recognition to disseminate recruitment materials, establish study legitimacy, and provide broad appeal to veterans across the United States.

The gap between clicks and enrollment illustrates the critical concept of “conversion rate,” wherein a goal is to convert a website visit into a desired action. Possible explanations for a low conversion rate among veterans include less interest and/or perceived need, “fatigue” from frequent suicide prevention trainings during active duty, or more reluctance to participate in research. In future campaigns, one way to more closely examine the relationship between sponsored ad engagement and study enrollment would be to construct separate links to the study landing page for each sponsored ad.

Facebook appeared to be a useful social media platform for reaching middle-age adults, a result corroborated in another recent study targeting veterans with unmet mental health needs [34]. To reach other demographic groups, other platforms should be considered when planning a recruitment campaign. For instance, Instagram and Snapchat are likely to perform better with youth and young adults [35]. Other recommendations informed by our findings include stressing the urgency of helping veterans in the campaign's call to action and using a verified account from a trusted institution as the campaign messenger, rather than the common practice of creating a study-related social media account for recruitment [36].

Limitations

There are a number of limitations. Our sample consisted almost exclusively of non-Hispanic white women. While this reflects, in part, the underlying demographics of our study population, vulnerable populations with less experience or access to digital communications were likely not represented in this study. We feel it is essential for future work to address inclusiveness of racial/ethnic and other minority populations, so as not to exacerbate disparities as research moves toward more use of remote trial methods. Second, feedback on barriers and facilitators to study participation was necessarily limited to participants who engaged in the study and watched at least 5 min of the VA S.A.V.E. If we had interviewed participants who did not watch any of the VA S.A.V.E. video, we may have achieved a less biased sample, as well as obtained potentially useful information about barriers to study participation. Third, because we ran multiple sponsored ads simultaneously and used identical links, we were unable to directly connect enrollments to particular ads. By using tagged links, insight into individual ad performance is possible.

We chose an attention control over a waitlist control for two reasons. First and foremost, we believed it offered more internal validity due to its ability to test the specific effect of content contained in VA S.A.V.E. Also, a waitlist control would have lengthened the overall study length, a significant concern due to the grant period being limited to 18 months in this project. Unfortunately, this resulted in a retention rate in the control arm that was significantly lower than in the intervention arm, largely due to drop-out immediately after baseline. Fortunately, participant characteristics were on average no different between study arms. In hindsight, we realize lower desirability of an attention control leading to an elevated drop-out rate is a known methodologic concern in studies using an attention control design [37]; in a future study of VA S.A.V.E. a waitlist control design or an attention control with a more convincing therapeutic rationale that better meets close supports’ expectation of receiving suicide prevention training would be preferable.

Conclusions

The principal takeaway from this study is that a paid social media campaign was a feasible approach to recruit and retain participants in fully remote trial of VA S.A.V.E. gatekeeper training. The stakes of training in suicide prevention skills are high, and it remains for future work to determine the impact of VA S.A.V.E. training itself. However, we believe the promising preliminary results of this study, together with the potential for intervention scalability, make a compelling argument for conducting a randomized trial on the efficacy of VA S.A.V.E. gatekeeper training.

Funding

This study was supported by a grant (PPO 20-166) from the , Veterans Health Administration, , (HSR&D). This funding source had no involvement in study design; the collection, analysis and interpretation of data; or in the writing of this article.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Theme	Illustrative Quote
Familiarity with university partner facilitated trust	(1841) “For me it was almost crystal clear when I saw the Oregon Health and Science University and the Department of Veterans Affairs' logo. Being a vet here in Oregon, I knew both of those were connected together. So that cemented in my mind right away. I don't know how it would relate to somebody who is on the East Coast who doesn't have that connection to make.”
Familiarity with university partner facilitated trust	(137) “I look at who its from. So, ‘Oregon Health and Science University,’ okay, so, in my mind, I immediately think, okay it's a legitimate study. It's not a scam or someone trying to make money …. You have your URL there. You have a ‘.edu’ at the end of your web address, which again lends more credibility to the idea.”
VA not known as a research institution may be a barrier to trust	(86) “With the VA being an actual provider of care, people don't necessarily think about the VA as also doing a lot of research … and the sense is, if the VA is doing a study, then what am I, some kind of guinea pig in their medical treatment? And I don't want to be a guinea pig. I want treatment.”