Usability and effectiveness of adherence monitoring of a mobile app designed to monitor and improve adherence to event-driven and daily HIV pre-exposure prophylaxis among men who have sex with men in Taiwan.

Objective: The UPrEPU mobile app is a self-monitoring system to enable men who have sex with men to optimize their pre-exposure prophylaxis adherence for HIV prevention. The app was designed to accommodate a rather complicated event-driven dosing schedule. We aim to evaluate the usability of the UPrEPU app and its effectiveness in improving adherence monitoring.
Methods: From May to October 2020, 35 participants were enrolled for the usability study and followed up for 4 months. Blood samples for the drug concentration in the dried blood spots were obtained once during the second to fourth follow-up visits. The effectiveness of adherence monitoring was analyzed using Cohen's kappa statistic to calculate the concordance between the average number of pills taken and drug concentration in the dried blood spots.
Results: Overall retention was 91.4% (32 participants) at the end of the study. Participants used the app for a mean of 29 days and made 2565 data entries in total, with an average of 76 data entries. The average systematic usability scale score for the app was 71.5, indicating acceptable usability. Slight agreement was reached between the dried blood spots measurement and the number of pills taken and recorded in the app (weighted kappa: 0.21). Conclusions: Our user-centered UPrEPU app demonstrated that it could accommodate both daily and event-driven dosing schedules for men who have sex with men clients with acceptable usability scores. We confirmed that complex behaviors such as different drug-dosing regimens that are contingent on sexual behaviors could be incorporated into the design of a mobile app.

Introduction

Oral pre-exposure prophylaxis (PrEP) has been regarded as a highly effective and comprehensive HIV prevention strategy. The efficacy of PrEP could reach over 90% when high adherence was kept among men who have sex with men (MSM).[1-3] The event-driven (ED) PrEP dosing regimen—which involves taking two pills of tenofovir disoproxil fumarate/emtricitabine (TDF/FTC) between 2 and 24 h before sex, followed by two pills of TDF/FTC taken 24 and 48 h after the first pill intake—was an alternative to the daily dosing regimen recommended to MSM. The ED PrEP dosing regimen among MSM has been gradually adopted in several countries, such as France, the Netherlands, Belgium, and Taiwan. A pre-coital dose of PrEP should be taken at least two hours before each sexual intercourse for an ED regimen, but users may not be able to correctly estimate the exact timing of a sexual event and thus miss the pills. The post-coital dose can be easily forgotten as well. Studies have shown that 81.9% of ED PrEP users miss their pills before sex, and 4.3% miss pills after sex.

The effectiveness of PrEP correlates with the level of adherence. Current studies use two major types to estimate PrEP adherence: self-reports and objective measurement of drug concentration. Self-reporting measures vary and often include indirect measurements such as pill counts in different time frames using different methods. A short message service is an approach that facilitates self-reporting because the participants receive messages to prompt them to report frequently, thus reducing recall bias.[9,10] Drug concentrations are often measured as a more objective way to estimate adherence and for comparison. One study that used a sample of adolescent and young MSM from the United States compared four types of adherence measurement: self-reported adherence, smart pill cap openings, TFV concentrations in hair, and TFV-DP concentrations in dried blood spots (DBS). Compared to the drug concentration methods, self-reported measures overestimated the adherence; smart bottle openings underestimated adherence because the bottles were not being used consistently. For evaluating adherence, however, none of these measurements is perfect for ED regimens in which the timing of the sexual event should be considered. MSM PrEP users might adopt PrEP dosing regimens based on their preferences and lifestyles.[5-7,13] Switching between two dosing regimens is not uncommon in the real world.[5-7] While ED PrEP already has a complex dosing schedule, it would be more difficult to measure the real-world ED PrEP adherence with dosing regimen switching involved. Approaches to ensure and improve PrEP adherence are crucial to maximize the benefits of PrEP.

Mobile health interventions have shown promising feasibility and acceptability in improving medication adherence and preventive behavior in various medical conditions.[14,15] Moreover, mobile phone applications have been adopted as one of the mobile health technologies to improve access to HIV prevention services and adherence for antiretroviral therapy among people who live with HIV. Given the ubiquity of smartphones, mobile apps for MSM offer distinctive opportunities for public health interventions targeting this population and empowering them to self-care and to handle their healthcare. Several studies have been evaluating smartphone and Internet-based interventions for HIV prevention and PrEP adherence. Most of the studies focus on MSM and aim to facilitate HIV testing or PrEP uptake by providing home delivery of prescriptions. Only a few other studies have been focusing on self-monitoring of PrEP adherence among MSM; however, these studies have had limitations of daily PrEP use only and have lacked a user-centered practices design.[18-20]

UPrEPU is a novel mobile application designed with a user-centered approach and was developed following the multiphase optimization strategy framework (comprising a preparation phase, an optimization phase, and an evaluation phase).[21-23] The UPrEPU app incorporates concepts of a participatory design approach, recognizes the needs and the direct input of the users, and took their needs and expectations into consideration in the design process. This app was designed to assist PrEP clients, including those who used the ED dosing regimen, to better manage PrEP use by themselves. Our app allows flexibility for MSM's dynamic dosing choices and aims to improve PrEP adherence by sending reminders based on the current dosing regimen and users’ sexual practices.

In this paper, we aim to describe the development of UPrEPU and evaluate the usability of this app. We further evaluated the effectiveness of adherence monitoring of the UPrEPU by comparing the pills taken and sexual intercourse recorded in the app with the drug concentration in DBS. We hypothesized that the UPrEPU app developed through a user-centered approach would be acceptable and easy to use. We expected moderate to high agreement between DBS measurement and the number of pills taken in the app among MSM PrEP users in Taiwan.

Methods

Development of the UPrEPU app

The design and development process of the UPrEPU app used a multidisciplinary and user-centered design approach.[25-27] Formative conceptualization qualitative interviews were conducted to understand the MSM PrEP clients’ needs. PrEP clients’ personas and journey maps were developed from the data collected in the interviews. One session of each participatory workshop was conducted with the personas and journey maps used to generate concept ideas and a low-fidelity prototype of the UPrEP app.

The participants for the participatory workshop included PrEP users, PrEP managers, researchers with expertise in human–computer interaction design or public health, and app developers. They were randomly assigned to two developing groups. A detailed description of the development process of the UPrEPU app was described in a previous report. A low-fidelity paper prototype of UPrEPU was generated under an agreement of all participants in the workshop (Table 1).

Table 1.

Features generated from the workshops (low-fidelity paper prototype).

	Group 1	Group 2
Planned features	Chatbota to answer PrEP-related questions.	Tailored reminder for PrEP uptake for daily/ED PrEP users
	Information for accessing HIV testing and PrEP.	Reward system to encourage adherence to PrEP.
	Reminders	Reminders and tailored feedback messages
	Visualization of drug concentration.	PrEP diary to record their daily PrEP use
	Social support through online platforms.	Communication and social support through the private message function.

Chatbot: A chatbot or chatterbot is a software app that simulates human-like conversations with users via text messages on chat.

PrEP: pre-exposure prophylaxis.

The research team refined a high-fidelity prototype afterward. The internal testers (research team members) accessed all features of the UPrEPU app to identify any remaining glitches, bugs, or usability concerns with the heuristic evaluation method. Then, the official UPrEPU app was developed regarding the feedback from internal testers. The UPrEPU app was developed for IOS and Android operation systems. A list of final app features and descriptions is shown in Table 2.

Table 2.

Features of UPrEPU app.

Features	Function
Customizable avatars	• Visual representations of users and their sex partners within the app to facilitate the development of an online identity.
Dashboard	The home screen where users can choose and switch their dosing regimens. Provides a visualization of the current adherence pattern. Shows the time and date of the last dose. Reminders on how many pills and when to take the next dose.
Calendar	A calendar allows users to create, edit, and delete the PrEP and sexual event diary and HIV testing records. Detailed information recorded for each sexual event, including the avatars, alias, sex position (bottom, top, or both), prevention strategies, and the mood at the moment of the sexual event. Signal on the calendar to show whether users have taken PrEP correctly regarding the users’ dosing regimens.
Data visualization of PrEP adherence and mood for sexual events	Provides a 14-day summary of PrEP adherence and mood for each sexual event with sex partners.
Information modules	Educational materials on HIV, PrEP, and other sexual health issues. Provides the nearest geographic location of HIV testing in Taiwan based on the users’ location.
Notification	Push notification text to remind app users to take PrEP, backfill their sex and PrEP record.

PrEP: pre-exposure prophylaxis.

Screenshots of the app are available in Multimedia Appendix 1. The highlighted features include visualized drug concentrations, a sex diary that allows users to create and record the details of their sexual events (including time for sex, prevention tools used in the sex, sex roles, and mood tracker), and a calendar. Participants also log their PrEP intake and choice of the dosing regimen. Based on the dosing regimen and the participants’ scheduled dates with their sexual partners—a feature that is particularly important for an ED dosing regimen—the app shows reminders for pills needed to be taken in the following days on the calendar. The resources provided on the app include HIV and PrEP-related educational multimedia materials and a global positioning system (GPS)-enabled map of HIV testing locations. Data visualization in the app shows a graph integrating the mood for each sexual event and PrEP use. Users can see the pattern of their mood with different sex partners with or without PrEP protection, which is also one of the highlighted features.

Usability, feasibility, and effectiveness of adherence monitoring of the UPrEPU app

Recruitment and enrollment

Participants were referred by physicians; PrEP navigators were recruited in the sexual health clinic from May to June 2020 and assessed for eligibility by an online screening tool. Eligibility requirements included the following: (1) Participants were men who were 20 years of age or older; (2) resided in Taiwan and able to understand, read, and speak Mandarin Chinese; (3) tested negative for HIV 3 months prior to enrollment for current PrEP users; (4) had laboratory results that indicated eligibility to initiate PrEP based on Taiwan's PrEP guidelines; (5) currently took PrEP or were willing to initiate PrEP after enrollment; (6) reported having at least four episodes of anal intercourse with men in the previous 1 month; (7) owned an Android or Apple operating system (iOS) smartphone and were willing to download the study app; and (8) were willing to wear the Garmin wearable fitness tracker that the research team provided during the study period.

Eligible participants were asked to complete informed consent forms upon entering the study. Participants were then asked to download the UPrEPU app, and watch a short clip embedded within the app for feature demonstration. The participants were also required to answer the baseline questionnaire to finish their enrollment in the study.

Participants were followed up monthly for 4 months. Each monthly visit included rapid testing for HIV antigen and antibodies and participants completed a follow-up questionnaire on their mental health scales, sexual behaviors, and PrEP use. Depression and anxiety were assessed using the Patient Health Questionnaire-9 (PHQ-9) and General Anxiety Disorder-7 scale (GAD-7), respectively.

The System Usability Scale (SUS) for the UPrEPU app was assessed during the first follow-up visit. Blood samples for the TFV-DP and FTC-TP concentration in the DBS were obtained once during the second to fourth follow-up visits.

Outcome measurement: Usability, feasibility, and effectiveness of adherence monitoring usability

We used the SUS, a reliable and low-cost usability scale, as the primary outcome for usability. It was a 10-item, 5-point Likert scale yielding a global view of subjective assessment of usability. In the SUS scale, even and odd questions were scored differently. Odd questions were scored 0–4 based on the 1–5 selection. Even questions were scored 4-0 on the 1–5 selection. Scores for all 10 questions were added up for a total score between 0 and 40 points. This total was multiplied by 2.5 to generate a SUS score between 0 and 100 points. A product with SUS scores below 50 indicated certainty that it had usability difficulties in the field; scores in the 70 s and 80 s indicated that a product was acceptable. We evaluated SUS of UPrEPU app in the month 1 visit. Participants were further asked to clarify their reasons for providing a neutral or negative response to the following question in the SUS scale: “I would not like to use this app frequently” and “app features integration had room for improvement.”

Feasibility

The primary feasibility outcomes were app engagement including the frequency of app logins, the use of app components such as PrEP-taking, and sexual behavior reports. The frequency of app logins and the use of app components such as anal intercourse and pill-taking records were logged and extracted from the app. For each sexual event, participants also recorded the mood at the moment of sex on the Smiley Face Likert scale (awful, not very good, okay, really good, and fantastic).

Secondary feasibility outcomes of this app included a focus on and potential improvement of the app in the future. All participants were asked an open-ended question: “What features would you like to add to the app?”

Effectiveness of adherence

Adherence was estimated by the proportion of anal sex acts that used PrEP correctly. It was calculated as the proportion of “sex-days” (i.e. the number of days that anal intercourses occurred as the denominator) for which PrEP was taken correctly (numerator). Whether PrEP was taken correctly in each sexual event was determined by the algorithm in the app. An algorithm was developed regarding the ED and daily PrEP instruments as follows:

Correct taking of ED PrEP was calculated as taking 2 pills 2–24 h prior to each sexual event, followed by 1 pill 24 h and 1 pill 48 h after the latest pill taken before sexual event. However, if at least one pill was taken 25–168 h (1–7 days) before each sexual event, taking only one pill 2–24 h before each sexual event was allowed. Correct use of a daily dosing regimen was defined as at least 4 pills taken 168 h (7 days) before each sexual event.

The effectiveness of adherence monitoring of this app was estimated as the consistency of PrEP use between drug concentrations in the DBS sample and the pill-taking reports in the calendar component in the UPrEPU app. The DBS sample was obtained once after the first month follow-up. The procedure to measure tenofovir-diphosphate (TFV-DP) and emtricitabine-triphosphate (FTC-TP) in DBS has been described previously. The lower limit of quantification was 25 fmol/sample for TFV-DP and 100 fmol/sample for FTC-TP. We used validated dosing cut-offs for TFV-DP in DBS to categorize participants’ weekly pill-taking on average: 7 doses/week (>1450 TFV-DP fmol/punch), 4–6 doses/week (800–1449 fmol/punch), 2–3 doses/week (400–799 fmol/punch), and < 2 doses/week (below the limit quantification 350 fmol/punch).[40,41] The pill-taking record of app users in the 6 weeks before DBS collection was estimated on a weekly average and classified in line with the categories of TFV-DP. The DBS samples were stored at − 20°C within 24 h of collection and shipped on dry ice to the Skaggs School of Pharmacy and Pharmaceutical Sciences (University of Colorado Anschutz Medical Campus, Aurora, CO, USA) for analysis.

Analysis

The demographics of participants were described in numbers with percentages. The number of pills taken, the record of sexual events, mood, and PrEP protection in the app were calculated by each individual participant. App usage was described as the total of the average of data entry and days. Descriptive statistics were used to examine the SUS results with an aggregate score. Weighted Cohen's kappa coefficient analysis was used to examine the consistency between the drug concentration in the DBS samples and the self-reported PrEP diary in the app. The difference in drug concentration in DBS was accessed by Wilcoxon rank-sum test. All analyses and visualizations were performed using R 4.0.4.

The study was approved by the Institutional Review Board of the National Cheng Kung University in Tainan City, Taiwan (A-ER-107-337).

Results

Study populations demographics

Thirty-five participants enrolled in this study from May to June 2020. The baseline characteristics of the participants are shown in Table 3. Most participants were 26–30 (16, 48.6%) years of age; 14 (39%) earned less than USD $1000 per month; most participants used iPhones (24, 70.6%); and a substantial number of participants (9, 25.7%) reported they had been switching between daily and ED PrEP prior to study participation. No participants were diagnosed with HIV or syphilis during the follow-ups.

Table 3.

Baseline characteristics of participants (n = 35).

		N (%)
Age (years)
	21–25	7 (17.1)
	26–30	16 (48.6)
	31–35	8 (22.9)
	> = 36	4 (11.4)
Monthly income range (USD) a
	≧500	4 (10.5)
	501–1000	10 (28.5)
	1001–1500	12 (34.3)
	>1500	9 (25.7)
	Relationship status
	Single	16 (45.7)
	Fixed relationship	6 (17.2)
	Open relationship	13 (37.1)
Sexually transmitted diseases diagnosed in the past year
	Gonorrhea	4 (11.4)
	Syphilis	9 (25.7)
	Genital herpes (HSV-1, HSV-2)	1 (1)
	Genital warts on the anus	5 (14.3)
	Genital warts on the penis	1 (2.9)
	Drug use during sex in the past month
	Rush (Poppers)	4 (11.4)
	Viagra, Cialis, and other aphrodisiac drugs	4 (11.4)
	Amphetamine, methamphetamine	2 (5.7)
	Alcohol	1 (2.9)
	Cannabis	1 (2.9)
	GHB/GBL	1 (2.9)
	Post-exposure prophylaxis (PEP) use in the past 6 months	13 (37.1)
Pre-exposure prophylaxis (PrEP) dosing regimen
	Daily	13 (37.1)
	On-demand	12 (34.3)
	Switch between daily and on-demand	9 (25.7)
	Smartphone b
	iPhone	24 (70.6)
	Android	10 (29.4)

Exchange rate: 30 New Taiwan Dollars equals 1 US dollar.

One missing.

Thirty-two participants completed the 4-month follow-ups (91.4%). Among them, 3 (8.6%) discontinued the study because of losing a device and refusing to wear the devices. Although not completing our 4-month follow-up, one person had DBS performed and examined, resulting in a sample size of 33 for DBS analysis (Figure 1).

Figure 1.

Enrolled and follow-up of study participants (n = 35).

Usability of the UPrEPU app

The results of the UPrEPU app's usability are shown in Table 4. Overall, participants found the app to be convenient to use and easy to use on their own. The average SUS score for the app was 71.5, which indicated that the UPrEPU app is acceptable and above average, with 18 participants (51.4%) who had scores of more than 70 in the SUS.

Table 4.

System Usability Scale (SUS) scores of the UPrEPU app by men who have sex with men (MSM) participants during the 4-month pilot study. Taiwan, 2020 (n = 35).

Statement	Mean a (SD)	Absolute b
I would like to use this app frequently.	2.6 (1.1)	2.6
The app was unnecessarily complex.	1.5 (1.0)	3.5
The app was easy to use.	3.2 (0.8)	3.2
I would need support from a technical person to be able to use this app.	0.8 (0.8)	4.2
Various functions in the app were well integrated.	2.6 (1.0)	2.6
There was too much inconsistency in this app.	1.6 (1.1)	3.4
Most people would learn to use this app very quickly.	3.3 (0.7)	3.3
The app was very cumbersome to use.	1.3 (1.1)	4.7
I felt very confident using the app.	3.3 (0.8)	3.3
I had to learn many things before I could get going with this app.	1.3 (1.1)	3.7
Calculated score		71.5 (12.4)

Scoring based on a scale from 1 = totally disagree to 5 = totally agree.

Adjust scores of negative statements, so larger numbers are associated with positive statements.

The main reasons for participants not using the app frequently included the following: infrequent sexual activities and ED PrEP-taking (n = 5, 35.7%), lack of motivations (n = 4, 28.6%), and few notifications from the app to remind users of opening the app (n = 4, 28.6%). Our participants also mentioned how the app could be improved: the data visualization for the relation between PrEP use, sex, and mood were difficult to understand (n = 6, 37.5%), and function in the calendar feature can be better integrated with other apps (n = 3, 18.8%), such as the Google calendar.

More than 95% (32) of the participants kept using the UPrEPU app during the study period and contributed an average of 641 data entries to the app each month (Figure 2). Participants used the app for a mean of 29 days (median: 24 days, interquartile range (IQR): 11–37 days) and made 2565 data entries in total, with an average of 76 data entries per day (median: 70, IQR: 47–106 data entries). Among the 32 participating users, data entry remained high for each follow-up (96.9%–100%).

Figure 2.

UPrEPU app engagement during the study period (n = 32).

The most used function was the PrEP intake record, with 1806 data of PrEP intake (1984 pills) and an average of 53 pills per person during the 4-month follow-up (median: 53, IQR: 34–74). PrEP was taken correctly in 378 (68.6%) sexual events with anal intercourse (ED PrEP: 254, 62.4%; Daily PrEP: 124, 86.1%). The median of correct PrEP use per person by the study protocol definition was 12 sexual events with an IQR from 4 to 20.

The second frequently used function was the scheduled dates. Participants reported 551 sexual events with anal intercourses, with an average of 17 events per person (median: 14, IQR: 8–21). Condoms were used in 123 (22.3%) events. A total of 207 (37.7%) events occurred with partners with unknown HIV status, 118 (21.5%) with partners who were HIV negative and on PrEP, 165 (30.1%) with partners who were HIV negative and not on PrEP, and 59 (10.7%) with HIV positive partners who had an undetectable viral load.

Participants recorded their mood as “really good” (264, 47.9%) and “fantastic” (181, 32.8%) in most of the sexual events (awful: 3, 0.5%; not very good: 8, 1.5%; okay: 95, 17.2%). There were 81.6% of sexual events with correct PrEP use rated at “really good” or above, and 75.3% of sexual events without correct PrEP use rated at “really good” or above.

The most wanted features for the UPrEPU app in the future was a calendar feature in the app to be integrated with their other calendar apps and keep the schedule consistent without repetitively typing content (22.9%). Some participants (8.6%) would like to have an additional note section to record their sexual events comprehensively. Given that the UPrEPU app has a lot of contexts related to sex and PrEP use, 11.4% of participants suggested that the UPrEPU app develop a social networking function to enable users to interact with each other or as an add-on function to integrate with common gay social apps, with a geosocial networking app for sexual encounters.

Effectiveness of adherence monitoring

The 33 DBS measurements from 33 participants were collected at the follow-up visits.

Overall, the median TFV-DP concentration was 821.8 (IQR: 411.9–984.8) fmol/punch, equivalent to an average of 4 doses taken per week. Twenty-three (66.7%) participants reported ED PrEP use and 10 (27.3%) reported daily PrEP use prior to the visit of DBS collection in the app. There was no significant difference in TFV-DP concentration between the ED and daily PrEP use; the medians of TFV-DP concentration were 821.8 (IQR: 439.1–961.2) and 984.6 (IQR: 763.9–1578.8), respectively (p-value: 0.145). One (3.0%) TFV-DP measurement was below the lower limit quantification (BLQ). FTC-TP concentrations were BLQ in 10 (30.3%) DBS measurements, meaning no dose intake within 48 h. Among these 10 DBS samples, 8 were from participants who reported ED PrEP use and 2 were from participants who reported daily PrEP use.

The median PrEP adherence in the app was 90%, with an IQR from 68.9% to 98.1%. Overall, there was a slight agreement between DBS measurement and the number of pills taken in the app (weighted Kappa: 0.21). According to the app data, during the 6 weeks, before the DBS measurements were collected, a median of 25 (IQR: 15–35) pills were taken. There were 22 participants (66.7%) who had inconsistency between the app data and the DBS measurements; 11 (33.3%) had higher DBS measurements than what the app data suggested; and 11 (33.3%) had higher pill numbers in the app data than DBS measurements (Table 5). Among 23 ED PrEP users, 14 (60.9%) had inconsistency between app data and DBS measurement: 8 (34.8%) had higher DBS measurement than app data and 6 (26.1%) had higher pill numbers in the app data than DBS measurement. Out of 10 daily PrEP users, 8 (80.0%) had inconsistency between app data and DBS measurement: 3 (30.0%) had higher DBS measurements than what the app data suggested and 5 (50.0%) had higher pill numbers in the app data than DBS measurements.

Table 5.

Comparing the self-report pills taken in the app and the estimated weekly average pills taken from dried blood spots (DBS).

	Self-reported pills taken in the app
Estimated weekly average pills taken from DBS		<2	2–3	4–6	7	Total
	<2	4	0	4	0	8
	2–3	3	0	5	0	8
	4–6	2	3	6	2	13
	7	0	0	3	1	4
	Total	9	3	18	3	33

Discussion

Principal findings

We presented a novel technology to assist MSM PrEP users to self-monitor their PrEP adherence and allow flexibility for the user's choice of an ED dosing regimen in real-world settings, with a high proportion of ED PrEP use and regimen switching. We assessed the usability and effectiveness of this user-centered innovative app with features of sexual event and pill diary and adherence visualization to assist MSM PrEP users. First, users reported that the UPrEPU app had an acceptable and above average usability level and was easy to use on their own. Second, regarding feasibility, participants showed high engagement and provided high numbers of sex, PrEP, and mood records in the app. Third, when compared to objective measurement of drug concentration, only slight agreement was observed between the DBS measurements and the number of pills taken and recorded in the app. Our study suggested that it is feasible for MSM to use a mobile phone app to manage their PrEP use and that the UPrEPU app is an acceptable tool in terms of technology usability; however, the potential for reporting bias using the app still exists. The app, like other technology developed to improve healthcare delivery, might be most helpful when accompanying other support, such as communicating with healthcare providers and users’ social network.

SUS being one of the most popular indexes for usability provides us with some information regarding how our app was perceived by the users. Other studies that applied an automated directly observed therapy approach in their app to improve PrEP adherence for daily MSM users reported having a median SUS score of 85 at week 4 and 80 at week 8, versus ours of 71.5 at month 1, a lower score but still in the acceptable and above average range. Other apps focusing on HIV prevention targeting different age ranges of MSM in the United States also have an above average SUS score,[42,43] showing an overall high acceptability across geographical regions and languages in applying technology in HIV prevention interventions, and have the potential to serve as a platform to apply artificial intelligence in the next step.

The app engagement remained high during the 4-month study period of our study. It is a key challenge for the majority of health-related apps since the level of engagement remains low or declines over time.[45-47] Several characteristics have been shown to enhance user engagement, including provision of personally tailored content, interactive components, social networking, and reminders.[48-51] Several common features were designed to improve adherence to PrEP, such as diary features with both a sexual activity log, medication log, and notifications. The application designed for PrEP adherence improved from other studies with those common features experienced reducing user retention over time. Apart from the shared common features, our app was designed to engage the users to self-monitor PrEP use, which was one characteristic to enhance retention. We added a mood log to show the correlation and trend in mental health of the user, with data visualization of the patterns of sexual activities, PrEP use, and mood. In the UPrPEU app, the avatar features provided the users with a virtual identity that enabled users to record their sexual events in a relatively safe environment and served as a personally tailored feature to enhance engagement. A data visualization of the PrEP protection level in the app also assisted users to understand their current adherence patterns and reduce their anxiety toward sexual activities.

Our study found a low agreement between the self-reported pill number collected by the app and the weekly pill number estimated by the DBS measurement. There was no significant difference between the proportions of overreported and underreported groups. Both overreporting and underreporting pill numbers compared to DBS measurement happened in our sample, showing that a bias in using self-report data in an app to measure adherence can go either way. The low consistency between DBS and self-report in the app may be owing to both the incorrect self-report use in the app and noise in the DBS measurement. The underreported group may be individuals with low motivation for using the app to self-monitor their PrEP use. The main reasons for participants not using the app frequently included infrequent sexual activities and ED PrEP taking, low motivation, and not enough notifications from the app to remind the users to open the app. These reasons all led to not using our app and resulted in an underreported PrEP use compared to DBS measurement. On the other hand, overreporting individuals might be those influenced by social-desirability bias, which gives socially desirable responses instead of choosing responses that are reflective of their true state.

Additionally, our study sample was recruited in Taiwan and is different from the majority of study samples in studies that have assessed TFV-DP in DBS in relation to adherence to tenofovir alafenamide; such studies usually have limited race or ethnic representativeness.[40,41] Cautions are needed in terms of race/ethnicity and weight, which not only affect TFV-DP variation,[40,41,56] but also interact with the association between age and TFV-DP. For example, African Americans had lower TFV-DP concentrations in DBS than White Americans which may be due to different genetic, diet, or environmental factors across races and ethnicities. There were limited Asian MSM data for the estimated average of pills used by TFV-DP concentration obtained from DBS. Future studies will be needed to establish the relationship between TFV-DP concentrations in DBS and adherence for Asian MSM besides samples from Thailand.[57,58]

Objective measures, such as TFV-DP concentration in DBS, are generally assumed to be a more reliable measure of adherence. As demonstrated by the AMPrEP study, the concept of adherence measurement for ED PrEP use is much more complex than daily PrEP use since that study involved the timing of sexual events and the number of pills taken. Moreover, objective measurement usually requires specialized equipment and trained technicians, which would be an obstacle for monitoring PrEP in the long term. Accurate self-report or technology-assist self-monitoring of PrEP adherence should be developed and used parallel with other objective measurements.

A high percentage of sexual events were protected by PrEP in our study; however, a wide IQR suggested that there was high diversity between the participants. The protection during sexual events from ED PrEP varied in previous studies. A high proportion of correct PrEP use for high-risk sexual activities has been observed in the studies using an online diary as a data collection tool in the Netherlands and Belgium, and in the IPERGAY study in France. Conversely, low adherence to ED PrEP was found among MSM and transgender women in Thailand and the United States in the HPTN 067/ADAPT study, and among MSM who sought sex partners via geosocial networking apps in China.[60-62] A high level of PrEP adherence in our study may be attributable to the flexibility of choosing PrEP dosing regimens and using an app. The possible explanations for the low adherence from the other two studies were that participants were assigned to one PrEP dosing regimen group without a cross-over in the HPTN 067/ADAPT study, and the participants had more frequent substance use in the Chinese study.

Conceptualization of PrEP adherence is shifting toward a prevention-effectiveness adherence (PEA) paradigm, which emphasizes the consideration of other preventive measures such as condom use along with PrEP use when determining the level of protection from HIV infection. To assess PEA, comprehensive information on the timing of sexual activities, PrEP use, and the adoption of other HIV prevention strategies (e.g. condoms, serosorting, and undetectable viral load) is required. Studies have been unable to measure PEA thoroughly since such information has been hard to access without innovative electronic tools.[6,64] The UPrEPU app allows users to record their PrEP adherence patterns in the long-term and provides real-time PrEP protection levels with indications for use of PrEP for each sexual event, which enables users to better manage their risk of HIV.

Limitations

The present study included several limitations. First, we may have limited generalizability due to our strict inclusion and exclusion criteria. MSM recruited in this study had a relatively higher frequency of sexual behaviors compared to general MSM PrEP users in Taiwan. Highly educated people were overrepresented in this study because these people could adopt innovative health technology more easily. The feasibility and usability data may not apply to those MSM who have much less frequent sex or to those who are less tech-savvy. Second, in terms of the small sample size in our study, the only inferential statistics in this paper is weighted kappa and the kappa can be calculated for fairly small sample sizes (e.g. five participants). The results in our study still need to be interpreted with caution given the relatively small sample size. Third, one major strength of the app is that it was designed to integrate ED PrEP users’ perspectives; however, routine daily PrEP users may be less motivated than ED PrEP users. Repeatedly logging daily pills may result in data entry fatigue. Fourth, we only measured DBS once in the study period. Multiple measurements may be needed to better examine the relationship between data of the number of pills used gathered from the app and the objective measurement of drug concentration.

Conclusion

This study demonstrated a user-centered designed app that accommodates both daily and ED PrEP use for MSM. We confirmed that complex behaviors such as different drug-dosing regimens that are contingent on sexual behaviors could be incorporated into the design of a mobile app. Even though adherence monitoring in the app was less effective, the app engagement remained high during the 4-month study period, showing the potential for such an app to support MSM to improve PrEP adherence regardless of the dosing regimen. While COVID-19 impeded access to PrEP services and changed the MSM's sexual lifestyles,[13,66] the UPrEPU app can not only assist PrEP users to self-manage their PrEP adherence regardless of their dosing regimens but also reduce the burden for healthcare providers by offering users personalized reminders to take PrEP based on their sexual lives. This can prevent PrEP case managers from answering repetitive questions and messages to some degree. Even though the mood at sexual events was rated positive in our app, this may not reflect the overall mental health among participants.

Future work for the UPrEPU app can incorporate more focus on addressing potential syndemic problems that include mental health, sexual health, and substance use. For example, a just-in-time adaptive intervention on the mobile app that aimed to provide interventions to improve mental health and to monitor substance use for MSM may further include the PrEP component to be a more comprehensive harm reduction tool. Only a few apps have been attempted, and even fewer have evaluated the feasibility and usability of the app. Future work should also include strategies to increase interest in retention and adoption of the app. Economic evaluation of mHealth interventions are crucial to explore the sustainability and cost-effectiveness of mHealth interventions.[69,70] Future studies on economic evaluation of mHealth interventions are needed to optimize such technology tools for HIV prevention.

Click here for additional data file.

Supplemental material, sj-docx-1-dhj-10.1177_20552076221102770 for Usability and effectiveness of adherence monitoring of a mobile app designed to monitor and improve adherence to event-driven and daily HIV pre-exposure prophylaxis among men who have sex with men in Taiwan by Huei-Jiuan Wu, Yi-Fang Yu, Stephane Wen-Wei Ku, Yuan-Chi Tseng, Chien-Wen Yuan, Chia-Wen Li, Po-Hsien Huang, Nai-Ying Ko, Peter L. Anderson and Carol Strong in Digital Health