Pilot feasibility trial of the MyPEEPS mobile app to reduce sexual risk among young men in 4 cities.

OBJECTIVES: Our study team adapted the MyPEEPS (Male Youth Pursuing Empowerment, Education, and Prevention around Sexuality) curriculum, an evidence-based human immunodeficiency virus (HIV) prevention intervention, from a face-to-face, group-based intervention to an individual-level mobile responsive web-based intervention to improve HIV risk behaviors in very young men, aged 13-18 years.
MATERIALS AND METHODS: In adapting the MyPEEPS intervention to mobile app, we used a series of methodologies, including expert panel reviews, weekly team meetings with the software development company, and conducted in-depth interviews with very young men. Following the iterative process, we conducted a 6-week pre-post feasibility pilot trial with 40 young men in Birmingham, AL; Chicago, IL; New York City, NY; and Seattle, WA. Primary outcomes of interest were uptake of the app, accessibility and satisfaction.
RESULTS: Across all 4 sites, 62.5% (25/40) of participants completed all modules in the app in an average of 28.85 (SD 21.69) days. Participants who did not attend to the follow-up visit did not complete any of the app modules. Overall participants reported that the app was easy to use, useful and has the potential to improve their sexual health knowledge and behavior and awareness in risky contexts. Participants also highly rated the app, information and interface quality of the app. DISCUSSION: Lessons learned from the pilot included the need for reminder systems and providing anticipatory guidance about Internet connectivity when using the app. These changes will be incorporated into study procedures for our multisite trial.
CONCLUSION: Overall, participants found the app to be highly usable and have the potential to positively improve their sexual risk behavior.

BACKGROUND

Young men who have sex with men (YMSM) are at a very high risk for acquiring human immunodeficiency virus (HIV). In 2016, 92.7% of HIV diagnoses among very young men, (13–19 years old) were the results of male-to-male sexual contact. In 2016, 6916 new cases of HIV were diagnosed among YMSM ages 13–24 in the United States.2 Among these new cases, 54% identified as Black/African-American, 25% identified as Latino, 16% identified as White, 2% identified as Asian, and <1% identified as Pacific Islander or American Indian and Alaskan Native. Thus, while the majority of new infections among YMSM occur among Black youth, just under half of incident cases occur in other racial/ethnic groups.

Engagement in high-risk sexual behavior is the main mode of transmission among YMSM. Relatedly, YMSM who have sex with older MSM face increased risk of HIV infection, as older MSM are more likely to be exposed to and living with HIV. Other social factors that make YMSM more vulnerable to becoming infected with HIV include stigma, homophobia, and racism, which cause many YMSM to feel rejected and isolated and, as a result, do not disclose their sexual orientation or seek HIV prevention services. Moreover, access to youth-centered HIV prevention services is limited and often inadequate., Importantly, many current HIV prevention interventions and education programs have little effect on reducing HIV risk in YMSM as they lack age-appropriate, culturally sensitive materials that address the needs of YMSM.,

Despite biomedical advances in HIV prevention, there remains a dearth of evidence-based HIV prevention interventions for racially and ethnically diverse YMSM. The current Center for Disease Control and Prevention (CDC) compendium of evidence-based and best practices for HIV prevention has no interventions that have shown efficacy among YMSM under the age of 18 years. The CDC compendium has identified 85 effective behavioral interventions (EBIs). Only 18 EBIs were developed for HIV-negative (or unknown status) youth. Of these, 5 were developed and tested with young women (age range: 13–25), 7 were developed for both young men and women (age range: 12–30), and 3 were developed for young men in juvenile justice facilities or targeted for African-Americans only (age range: 11–23). Additionally, 2 were developed for young gay and bisexual men (age range: 18–29), and one was developed for heterosexual males 18–29 years of age. We sought to test the feasibility and efficacy of a mobile HIV prevention intervention for diverse racial/ethnic populations of YMSM called MyPEEPS (Male Youth Pursuing Empowerment, Education, and Prevention around Sexuality).

The MyPEEPS curriculum was initially developed as a manualized, in-person, group-based intervention comprised of 6 modules focusing on key intermediate social and personal factors related to sexual risk-taking among YMSM, including sexual health knowledge (eg, correct way to use a condom), self-efficacy for safer sex, interpersonal communication skills, and behavioral skills. MyPEEPS was based on the Social-Personal Framework, which builds on Social Learning Theory by adding important psychosocial (eg, affect dysregulation) and contextual risk factors (eg, family, peer and partner relationships) related to youth risk-taking. It was tested with 101 diverse (23% white, 39% black, 27% Latino, 12% other) YMSM, ages 16–20 years and demonstrated evidence of feasibility, acceptability, and preliminary efficacy in reducing sexual risk behaviors. The in-person version of MyPEEPS is not included in the CDCs compendium of EBIs. Importantly, key difficulties with the group-based intervention included coordination of youth willing to participate in a group-based intervention and difficulty with the travel to access the group-based intervention. In response, we have adapted the MyPEEPS curriculum from a face-to-face, group-based intervention to an individual-level mobile responsive-driven web-based intervention to: (1) reach high-risk YMSM at a relatively low cost, (2) engage YMSM where they meet sex partners (eg, on the Internet), and (3) enable YMSM to receive a behavioral intervention on a computer, tablet, or smartphone on their own schedule and in a private setting.

As previously described, we adapted the MyPEEPS intervention into a mobile app and used a series of methodologies, including expert panel review, weekly team meetings with the software development company, and in-depth interviews with targeted end-users. We included information on pre-exposure prophylaxis (PrEP), post-exposure prophylaxis (PEP), and home-based HIV testing, which was not included in the original intervention. The research and software development team worked to ensure the adaptation stayed true to the original content, while increasing the level of engagement through activities and games (5–10 min per activity) and shortening the amount of material overall for mobile delivery. In-depth interviews were audio-recorded, transcribed verbatim, and analyzed using a directed content analysis approach. Findings from interview data analysis were used to adapt the MyPEEPS content into a mobile app intervention. Lastly, we conducted 2 usability evaluations: a heuristic evaluation with informatics experts to identify any violations of usability principles and end-user testing with 20 young males (15–18 years of age) to identify any obstacles with use of the app. The final version of the app includes 21 activities divided into 4 modules. Four characters, representing composite profiles of racially and ethnically diverse YMSM or “Peeps” (featured in the original version of the intervention) were integrated into app activities.

This article describes the results of the pilot study testing the MyPEEPS Mobile intervention, a web application, accessible by smartphone or other web-enabled devices for racially and ethnically diverse very young MSM ages 13–18 years. The goals of the pilot study were to: (1) understand app use and usability, (2) assess the timeframe needed for very young MSM to use the MyPEEPS Mobile App, and (3) gain direct feedback from participants about whether and to what degree the MyPEEPS Mobile system worked as intended.

METHODS

Recruitment, screening, and enrollment

We conducted a 6-week, pretest–posttest pilot study of the MyPEEPS Mobile intervention from March to June 2018. Study team members from Birmingham, AL; Chicago, IL; New York City (NYC), NY; and Seattle, WA, used both convenience and participant referral to recruit 10 participants at each site into the pilot study, for a total of 40 participants. We recruited individuals who were assigned male at birth, identified as male (or gender non-binary, genderqueer, or gender non-conforming) and were attracted to other people assigned male at birth. Other eligibility criteria for enrollment included: (1) 13–18 years of age, (2) HIV-negative or unknown status (self-report), (3) English-speaking, (4) living in either NYC, Chicago, Birmingham, or Seattle metro area, and (5) access to a smart phone, tablet, or computer. English proficiency was assessed via self-report during the screening process. We specifically asked participants “

Are you comfortable speaking and reading English?” Respondents needed to indicate yes to be eligible to participate. The study protocol was approved by the central IRB at Columbia University Medical Center. We obtained written informed assent (under 18 years)/consent (18 years) for all study participants with a waiver of parental permission.

Secure web-based baseline assessment

All baseline and follow-up visits took place in-person. A study team member collected data at baseline and at post-intervention (6 weeks) during in-person visits using both self-report and interviewer-administered electronic formats. We used Qualtrics software to collect survey data, which included demographic measures as well as assessment of usability and satisfaction. Participants were compensated for time and travel: $35 at the baseline visit, and $45 for 6-week visit, +$25 per module completed. The visit compensation is graduated and consistent with payments in prior studies.

Study measures

App use

To measure intervention exposure, we recorded the total number of participants who completed all 4 modules in the MyPEEPS app and the number of days to completion. Dosage of the intervention was measured through reports pulled directly from the MyPEEPS mobile app administrative dashboard.

Usability

We measured participants’ perceived usability of the app using 2 validated usability measures. The first scale, the Health Information Technology Usability Evaluation Scale (Health-ITUES), is a customizable questionnaire with a 4-factor structure and consists of 20-items rated on a 5-point Likert scale from strongly disagree (1) to strongly agree (5). The 20-item scale is comprised of 4 subscales: (1) impact, (2) perceived usefulness, (3) perceived ease of use, and (4) user control. Impact represents the system impact on daily life, perceived usefulness evaluates task accomplishment through system use, whereas perceived ease of use and user control capture user–system interaction. The overall Health-ITUES score was the mean of all the items with each item weighted equally, whereas a higher scale value indicates higher perceived usability of the app.

The second measure we used to assess app usability was the third version of the Post-Study System Usability Questionnaire (PSSUQ). The PSSUQ is an instrument for assessing user satisfaction with system usability, developed as a usability assessment tool specifically for use in the context of scenario-based usability testing. The PSSUQ consists of a 16-item survey instrument to assess system usability on a scale ranging from 1 (strongly agree) to 7 (strongly disagree), with a neutral midpoint. A lower score indicates higher perceived usability of the app.

Procedures

Once consent was obtained and completion of baseline assessments occurred, participants accessed the web application at: https://app.mypeepsmobile.org/login. Participants created a username and created a secure password. Participants were given 6 weeks to complete 21 mobile app activities divided into 4 sequential modules or “PEEPScapades”: (1) Intro: introduced participants to the program; (2) #realtalk: participants explored sexual risk scenarios, drug and alcohol use, social vulnerability, and HIV knowledge; (3) P Woke Up Like This: participants learn about HIV testing, HIV/STI risk behaviors, and steps for effective condom use; (4) Making Tough Situations LITuations: illustrated how intense emotions influence behavior and described strategies to manage stigma. Study participants logged in at their convenience; however, they were not able to access the subsequent module until the previous module had been completed. Participants were not allowed to complete more than 2 modules per week to promote absorption of the intervention material. Participants completed the first PEEPScapade (activities 1–4) at the baseline visit.

Follow-Up assessment

After 6 weeks, participants scheduled a follow-up visit with a study team member to complete any unfinished modules and the post-intervention assessment, including the Health-ITUES, PSSUQ, and a debriefing interview. During the interview, staff reviewed each MyPEEPS mobile module with the participant and asked follow-up questions pertaining to relevance, self-efficacy, comprehension, and technical difficulties in understanding the topic areas. The debrief interview was audio-recorded and answers were also typed into a data standardized collection form. The goal was to collect critiques of the material, content, delivery methods, and to identify subject matter that should be included to enhance relevance and efficacy for the full randomized trial of MyPEEPS Mobile intervention.

Analysis

We used descriptive statistics to analyze the study sample demographics and app use. Health-ITUES scores were calculated as the mean of the 20-item scale as well as the mean scores of each of the 4 subscales: (1) impact on daily life, (2) perceived usefulness, (3) perceived ease of use, and (4) user control. We reverse coded the Health-ITUES scores for ease of interpretation and comparison with the PSSUQ scores. The PSSUQ scores were calculated as the mean of the overall score as well as the mean of each of the subscales: system usefulness, information quality, and interface quality. Qualitative analysis of the interview data was completed through the use of open coding by segmenting data into meaningful expressions and organizing them by themes.

RESULTS

Study sample demographics

Our study sample comprised 40 YMSM, 15–18 years of age, with a mean age of 17.15 (SD = 0.88). Study participants self-identified their race/ethnicity as American Indian (N = 1), Asian (N = 5), Black (N = 12), Hispanic/Latinx (N = 10), White (N = 10), and multiracial (N = 2). Sexual orientation was reported as 67.5% (27/40) “only gay/homosexual,” 12.5% (5/40) “mostly gay/homosexual,” 17.5% (7/40) identified as “bisexual,” and 2.5% (1/40) “something else.” Only 2 study participants (5%) had ever dropped out of school. Additional study sample demographics are presented in Table 1.

Table 1.

Education, living status, primary language, country of origin among 40 MyPEEPS pilot participants by study site

	Total (n = 40)	Birmingham (n = 10)	Chicago (n = 10)	New York (n = 10)	Seattle (n = 1)0
Education
Eighth grade	3 (7.5)	0 (0)	1 (10)	1 (10)	1 (10)
Some high school	28 (70)	8 (80)	6 (60)	7 (70)	7 (70)
High school diploma/General Education Development (GED)	5 (12.5)	1 (10)	3 (30)	0 (0)	1 (10)
Some college	4 (10)	1 (10)	0 (0)	2 (20)	1 (10)
Live with in past 30 days
Parents and/or step-parents	26 (65)	7 (70)	4 (40)	8 (80)	7 (70)
Relatives—aunt, uncle, etc. but not parents	3 (7.5)	1 (10)	1 (10)	0 (0)	1 (10)
Foster care parents	1 (2.5)	0 (0)	1 (10)	0 (0)	0 (0)
An adult friend(s) of family	2 (5)	0 (0)	2 (20)	0 (0)	0 (0)
Friends of yours w/no adults present	5 (12.5)	1 (10)	2 (20)	2 (20)	0(0)
On your own	2 (5)	0 (0)	0 (0)	0 (0)	2 (20)
Someone else	1 (2.5)	1 (10)	0 (0)	0 (0)	0 (0)
Primary language
English	32 (80)	10 (100)	8 (80)	7 (70)	7 (70)
Spanish	5 (12.5)	0 (0)	1 (10)	3 (30)	1 (10)
Vietnamese	1 (2.5)	0 (0)	0 (0)	0 (0)	1 (10)
Something else	2 (5)	0 (0)	1 (10)	0 (0)	1 (10)
Born in United States
Yes	37 (92.5)	10 (100)	10 (100)	9 (90)	8 (80)
No	3 (7.5)	0 (0)	0 (0)	1 (10)	2 (20)

Abbreviation: MyPEEPS: Male Youth Pursuing Empowerment, Education, and Prevention around Sexuality.

Follow-Up visits

Participants were allowed to complete their follow-up visit as early as 30 days after their baseline visit and given up to 90 days from their baseline visit, to complete their follow-up visit, and be included in our final analysis. Across sites, participants completed their follow-up visits between 33 and 83 days from their baseline visit. Two study participants at the Birmingham site arrived at day 105 and 136 after the baseline visit. We allowed them to complete the follow-up survey and compensated them for their time, per the recommendation of our data safety monitoring board, but did not include them in our final analysis.

Participants were given app use and retention

Across all 4 sites, 25 of 40 (62.5%) completed all of the modules prior to the follow-up visit. Of those who completed the app prior to the follow-up visit, it took participants an average of 28.85 days to complete use of the app. There was a significant difference in the participants who completed the app modules by site (P = .033) with Birmingham having the smallest percentage of completers across sites. There was no significant difference (P = .693) among those who completed the app by racial/ethnic characteristics. Figure 1 presents the number of participants by site who completed all 4 modules prior to their follow-up visit. The participants who did attend their follow-up visit all completed only the first module.

Figure 1.

Number of participants by site who completed all 4 modules.

App use results are presented in Table 2 with a breakdown by each study site. During the pilot study, there was not a reminder system embedded in the app to remind participants to complete the app modules.

Table 2.

MyPEEPS app use prior to the follow-up visit among 40 MyPEEPS pilot participants by study site

	All sites (n = 40)	Birmingham (n = 10)	Chicago (n = 10)	New York (n = 10)	Seattle (n = 10)
Completed app, n (%)	25 (62.5)	4 (40)	6 (60)	8 (80)	7(70)
Days to complete, mean (SD)	28.85 (21.69)	43.75 (44.07)	29.43 (12.39)	24.63 (14.87)	24.57 (19.59)

Abbreviation: MyPEEPS: Male Youth Pursuing Empowerment, Education, and Prevention around Sexuality.

Usability of the App is reported in Table 3. Across all 4 sites, the Health-ITUES scores for overall usability and each of the subscales ranged from 1.07 to 2.41 with a score of 1 being the highest score, indicating a high perception of usability of the MyPEEPS mobile app and 5 being the lowest score. Similarly, the scores on the PSSUQ indicated high usability of the app with scores ranging from 1.80 to 2.08. The highest score on the PSSUQ is a score of 7 indicating a usability disaster and a score of 1 reflects a perfectly usable system. The Health-ITUES and PSSUQ are inversely proportional to each other, meaning higher scores on the Health-ITUES and lower scores on the PSSUQ indicate high usability standards. Results from both scales show similar findings which suggests the app is highly usable.

Table 3.

Usability, mean (SD) assessment among MyPEEPS pilot participants at follow-up visit by study site

	All sites (n = 35)	Birmingham (n = 8)	Chicago (n = 9)	New York (n = 10)	Seattle (n = 8)
Health-ITUES
Overall	1.41 (0.52)	1.37 (0.44)	1.28 (0.26)	1.40 (0.37)	1.92 (0.68)
Impact (daily life)	1.47 (0.58)	1.43 (0.74)	1.22 (0.33)	1.33 (0.27)	1.96 (0.70)
Perceived usefulness	1.49 (0.57)	1.48 (0.67)	1.25 (0.35)	1.37 (0.40)	1.92 (0.68)
Perceived ease of use	1.28 (0.43)	1.14 (0.22)	1.07 (0.14)	1.32 (0.42)	1.60 (0.62)
User control	1.83 (0.85)	1.38 (0.49)	1.81 (0.71)	1.70 (0.73)	2.41 (1.16)
PSSUQ
Overall	1.93 (0.81)	1.65 (0.72)	1.58 (0.45)	1.91 (0.76)	2.63 (0.94)
System quality	1.80 (0.78)	1.50 (0.61)	1.39 (0.37)	1.90 (0.80)	2.42 (0.90)
Information quality	2.03 (0.87)	1.73 (0.65)	1.74 (0.57)	1.95 (0.90)	2.75 (1.00)
Interface quality	2.08 (1.16)	1.88 (1.45)	1.70 (0.72)	2.00 (0.94)	2.79 (1.37)

Note: Health-ITUES scores range from 1 to 5 with higher values indicating better usability; PSSUQ scores range from 1 to 7 with lower values indicating better usability.

Abbreviations: MyPEEPS: Male Youth Pursuing Empowerment, Education, and Prevention around Sexuality; Health-ITUES: Health Information Technology Usability Evaluation Scale; PSSUQ: Post-Study System Usability Questionnaire.

Follow-up interview data

Interview data reflect several themes related to accessibility, usability, and engagement, as well as impact on sexual health knowledge and behavior and awareness in risky contexts. Interview data also reflects technical issues and bugs that were identified during the use of the app.

Accessibility and ease of use and engagement

A NYC participant comment was representative of many comments regarding the accessibility and concreteness of language in the app, “Everything is in vivid detail…everything was very easy to understand. I felt like the—it was very simple, like the language was very simple, but it was really specific at the same time, which was good. And a lot of times they use slang, which is good also, because a lot of people, even people who, let’s say aren’t into school, they could still use this app because it’s understandable what it’s trying to tell you” (NYC 10). Another participant described a high level of engagement with the educational content app, “Going through the whole game and then looking back on the progress that I made, I think that I actually learned a lot” (Chicago 04). One NYC participant explained, “I just want my friends to use the app, I loved it! I was trying to get them to come, but they just turned 19…I told all my friends and like my brother’s friends who are gay…I tell them, use the app whenever it comes out, it’s pretty cool” (NYC 03).

Sexual health knowledge and risk reduction

Participants reported an impact of the intervention on knowledge of: (1) biomedical HIV risk reduction approaches (eg. PEP and PrEP) and (2) sexual risk behavior, both of which are key components of the app.

PrEP and PEP. A participant from NYC explained learning about PrEP for the first time, “I didn’t even know there was a medication to prevent, to lower the risk of HIV and like now that I do, like I always tell people…there is this medication out there, it’s just that people don’t know it. They don’t teach this stuff in health class” (NYC 04).

Another YMSM from Seattle described specific prevention information he retained: “Knowledge that I did gain from this is about post-exposure prophylaxis. Because I didn’t know that. I didn’t know exactly how it worked. I just like if that you were raped or you went through a dangerous situation…and you want to make sure that there’s the lowest chance of you getting HIV that you talk to a doctor, you get post-exposure prophylaxis within 3 days of the incident and then you go through that, I believe a month, is what the app said” (Seattle 01). A participant in Chicago also commented on the PrEP resources provided, “I thought the activities were pretty easy and straightforward to complete…most of them gave a lot of useful information and at the end, they would give, some were phone numbers or websites were you could locate HIV testing, where you could get PrEP…and I thought that was really useful” (Chicago 09).

Sexual risk behavior. Another participant described how he believed that the app may change his sexual risk behavior. He said, “Coming from someone who typically did not use protection when engaged in sexual activities, I started using protection during sexual activities…because of the app. I learned a lot of different consequences, as well as other things you can get from not using protection…I guess I just learned more about myself” (Birmingham 01). Another participant described adopting safer-sex practices: “Well I can show more restraint now and for example, just last night, I was hanging out with someone that I’m really into and we were close to having sex, but I told him no because we didn’t have a condom, I don’t know his status and I don’t know my own either and I don’t even know if he knows his. I’ve actually been able to share some of the information with another friend of mine” (Birmingham 08). A 15-year-old youth from Seattle described how he is going to change his risk behaviors and detailed, “I’m definitely not going to do it with people who are drunk or stoned…and also make sure that I definitely have a condom on me or like, make sure that I definitely know…they are tested and make sure I don’t get any HIV or anything” (Seattle 10).

Contexts of sexual risk

Substance use. A Chicago participant described the value of learning about the harmful effects of substance use. He stated, “I learned a lot. I had no knowledge at all of poppers. I think that was really the one that stood out to me…and it’s funny because I had done this activity…learned about them and then a friend of mine mentioned them. It’s just something that I’m so glad I learned about on the app beforehand because had I learned about them from my friend as a source, then I’m sure I would have gotten a very one-sided explanation and description of them. And so I was really thankful to learn about the dangers” (Chicago 03). Another participant described the value of learning the signs of intoxication: “I knew about drugs but didn’t know how to identify when someone is under the influence…or what this or that could potentially do if I really used the wrong way…so I think that was very good” (Birmingham 05). Another participant described the value of learning about high-risk situations: “We get a lot of that (drug and alcohol information), and I felt it was good, because you don’t think about it in a sexual setting. They don’t really do a lot of relation between drugs and sex…none, actually, in school, so it’s good to actually have that somewhere laid out for you. Because I hadn’t thought about it” (Birmingham 06).

Stigma related to minority stress. One participant appreciated the content describing the realities of coping with stigma. He stated, “I personally really enjoyed the ways to manage stigma…it put into words what people, in this community face and like what they actually do to kind of cope with those things… that are like, not necessarily talked about” (NYC 01). Another participant described the immediate lessons learned from the activity: “I think there’s something reassuring about seeing a multitude of ways to manage stigma…it was comforting because I went from having sort of zero concrete ideas…on how to handle stigma to having 4 that I can access at any time if I need” (Seattle 05). Lastly, one participant described how the content reflected stressful situations he experienced in high school. He stated, “I related to a lot of these activities in my personal life and I was thinking when I was in a locker room in high school and you know a lot of the guys would mess around, like ‘oh he’s gay,’ not me, they were talking about someone else…like behind the scenes. The gay one. ‘Don’t let him see you.’ You know and I was like bro, ‘(that’s) ignorant’” (Chicago 02).

Technical challenges

Bugs

There were a number of technical challenges with the app. Specifically, the app is a sequential app which requires participants to complete the preceding activity before advancing to the next activity. During the usability testing, the lock between activities was not operational and so once it was implemented during our pilot study, there were numerous technical glitches. For example, a participant could not access activity #4 even after completing activity #3. It took numerous attempts at completing activity #3 until activity #4 would unlock. A similar issue happened to a number of study participants who had to repeat activity #4 to access activity #5. There was another bug on the screen which presented the 4 profiles of the MyPEEPS characters and forced participants to log back into the app again.

One Birmingham participant described issues with the slowness of the app. He stated, “When I got to the app, there were no real problems. It was just slow. It’s very laggy… and the more and more you advanced to the game, or the app, it just became slower and slower and I was just like, uh…the app worked pretty much how it worked on the computer, besides the speed” (Birmingham 05). Another participant commented on challenges with viewing the video content on both the app and on a computer. He stated, “I also was not able to view the videos on my computer, as well. So I stopped. But that was like the biggest technical issue I had overall with the app” (Birmingham 03). Another participant commented on their inability to complete the app modules. He stated, “I think it was 14…after passing that level, I tried to go back to the app and like do some work because I was actually into it and I wanted to like continue it, but every time I tried to log in… I cannot log in. And then I did the ‘forgot password’ and it still doesn’t allow me to log in… I think it’s a good app though. I really want to finish it. If I could get into it” (Birmingham 01). Participants across the remaining 3 sites had similar technical challenges. One participant described challenges with not being able to advance through the activities. He stated, “Like I said, no real technical issues other than for the week and a half or whatever, I couldn’t pass activity 4. That was like annoying. And I was like, is this like on purpose that you all can’t fix it or is it like a real glitch?” (Chicago 05). Another participant described challenges with viewing the video content. He stated, “The video wouldn’t load and it wouldn’t pop up at all at one point. And then just… it crashed my entire app. So like now the app doesn't work on my phone. So we had to switch over to a tablet. So I mean, that did suck. Because I know it wasn't my phone because I kept going on other websites, like Instagram, Facebook and everything was actually really going really fast… so I think maybe if the video got fixed, it would be better. Because some people just like give up. Some people just don’t have the patience for waiting for a video to load” (NYC 01). Lastly, another participant described the app crashing on their phone. He stated, “Activity eight. This is the one that broke, all the time. Like, I could not get through this one without—I had to go online to Safari and run it through there, rather than from the app. Because every time you got to the question about like your religion in the household, no matter what you answered for it, it would, the whole app would crash” (Seattle 05).

DISCUSSION

Our pilot study provided very useful information on the ease of use, usefulness, and potential impact of the MyPEEPS app on the proposed target population of very young YMSM. In the context of system development and technology acceptance, our end-users found the app to be useful, easy to navigate and reported it would likely have an impact on sexual health knowledge and behavior and awareness in risky contexts. Participants provided salient information on the knowledge gains achieved by the app and their plans to potentially change their sexual behaviors which is the ultimate goal of the MyPEEPS intervention. At the same time, there were 5 participants were lost to follow-up entirely, which means there is no data to support the usability or acceptability of the app in over 12% of the sample. This is important and may help contextualize the very high usability scores in those who completed the pilot study, which may reflect a response bias.

The goal of this pilot study was to inform the future use of this app in a multisite randomized control trial, and we learned a number of important lessons which will be incorporated into our future trial. First, in order to maintain engagement with the app, participants will require reminders to use the app during the intervention period of the trial. Only 25 participants completed all of the modules prior to the follow-up visit; however, 35 participants did attend their follow-up study visit. Therefore, study retention is not equivalent to app use and so both of these areas need to be monitored during our trial.

Participants did have technical difficulties using the app, specifically advancing between some activities, which were resolved by the end of the pilot study. As a result, we allowed the participants to complete the modules at the follow-up study visit. This allowed us to collect more robust data during our follow-up interviews with participants. Interestingly, participants at the Birmingham site reported some of the most technical difficulties which may have contributed to their lower app use, with only 4 participants completing use of the app prior to the follow-up visit.

In addition to implementing reminder texts, phone calls, or emails during the trial to remind participants to use the app, we also gained important information on explaining to study participants about the functionality of a web-app and its limitations. More specifically, during the pilot study many of the study participants tried to use the MyPEEPS app when moving into and out of Internet connectivity, consequently, their information did not save to the app and they would become frustrated when they could not connect. Initially, we did not explain this to participants at enrollment but quickly learned that this was essential information during the onboarding process to overcome the perception of technical difficulties.

The technical challenges were not limited to the Internet, such as inability to advance between activities, and we have fixed the code in response to this problem, which we believe this will enhance the overall ease of use of the system. For future large-scale implementation, migrating this app from a web application to a native app will help overcome many of the Internet connectivity issues; however, we may face the challenge that participants will not be willing to download an app which takes up too much space on their smartphone.

An additional goal of our pilot study was to inform future trial procedures and to consider study retention. We had 5 participants who didn’t complete a follow-up assessment within our specified follow-up period. This is 12.5% of the total sample; not an insignificant percentage. In contrast to research with adults or even older adolescents, our study participants are as young as 13 years old, making recruitment, enrollment, and follow-up challenging especially in the context of a study about sexual health. Parental consent is not a requisite for study participation and so study participants’ parents may not be aware of their participation, which can also make follow-up visits without parental knowledge and transportation a challenge.

Further, we had 2 participants who arrived at our Birmingham site and completed their follow-up visits after the 90-day window. These participants reported being away for their summer and so we were unable to conduct their follow-up visit prior to the end of the pilot study. Future trial procedures will allow us to send follow-up surveys to study participants via e-mail through a secure survey link. This will address some of the challenges related to parental disclosure, transportation hurdles and extended periods of travel.

Finally, the primary outcome of our intervention is behavioral change and so we had originally intended to report on the behavioral change as a result of the pilot. Given the short time frame (only 6 weeks) and the difficulties that many participants had using the app, we did not think that we would be able to detect meaningful behavior change in such a small sample over such a short period of time. We did not include a knowledge measure as one of our final study measures but agree that this would have been a useful outcome to study.

CONCLUSIONS

This pilot study supports the use of the MyPEEPS app in a future trial. Overall, participants found the app to be useful with the potential to positively impact daily life. Lessons learned from the pilot related to reminder systems and anticipatory guidance about Internet connectivity when using the app will be incorporated into study procedures for our multisite trial.

CONTRIBUTORS

All authors made substantial contributions to the conception or design of the study; or the acquisition, analysis, or interpretation of data for the work; M.I and R.S. drafted the manuscript. All other authors critically revised it for important intellectual content. All authors provided final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

FUNDING

Research reported in this publication was supported by the National Institute of Minority and Health Disparities of the National Institutes of Health under award numbers: U01MD011279 and U01MD011279-02S1. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflict of interest statement. None declared.