Lessons Learned From Conducting Internet-Based Randomized Clinical Trials During a Global Pandemic.

As the severe acute respiratory syndrome coronavirus 2 pandemic evolved, it was apparent that well designed and rapidly conducted randomized clinical trials were urgently needed. However, traditional clinical trial design presented several challenges. Notably, disease prevalence initially varied by time and region, and the pockets of outbreaks evolved geographically over time. Coupled with an occupational hazard from in-person study visits, timely recruitment would prove difficult in a traditional in-person clinical trial. Thus, our team opted to launch nationwide internet-based clinical trials using patient-reported outcome measures. In total, 2795 participants were recruited using traditional and social media, with screening and enrollment performed via an online data capture system. Follow-up surveys and survey reminders were similarly managed through this online system with manual participant outreach in the event of missing data. In this report, we present a narrative of our experience running internet-based clinical trials and provide recommendations for the design of future clinical trials during a world pandemic.

The global severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic presents novel challenges to conducting high-quality randomized clinical trials. Internet-based clinical trials are a possible avenue for surmounting some of these challenges. In this report, we offer some of the lessons learned from conducting such a trial.

In late 2019, as SARS-CoV-2 was spreading throughout China, there was significant interest in rapidly repurposing existing antiviral medications (eg, chloroquine, hydroxychloroquine, and lopinavir/ritonavir) for treatment. A primary challenge was to create robust study protocols to capture high-quality data while remaining agile enough to adapt to the rapidly changing scientific landscape of a global pandemic. Initially, no coronavirus disease 2019 (COVID-19) clinical trials focused on outpatients, despite being a vital population to target to reduce the number of hospitalizations from SARS-CoV-2 infection. This was likely due to several pandemic-related factors that made traditional clinical trial models incredibly impractical.

On March 9, 2020, our group at the University of Minnesota began 2 randomized clinical trials to address SARS-CoV-2 postexposure prophylaxis and early treatment, enrolling our first trial’s initial participant on March 17, 2020, just 8 days after our initial meeting [1, 2]. Within another 8 days, the trial had gone international with Canadian partners joining these 2 trials. We would later launch a third trial, focused on pre-exposure prophylaxis (PrEP), on April 6, 2020. All 3 studies used an internet-based trial format, recruiting 2795 participants in total [1-3]. Internet-based clinical trial formats are increasingly gaining attention due to their ability to reach large, geographically distant audiences without expanding to additional physical sites [4-6]. This trial design can overcome many of the barriers that exist in performing traditional clinical trials in a pandemic setting. In this report, we present the unique challenges our group faced using this trial format and how they were addressed. Although our experience can be applied to future pandemic trial designs, we also hope that many of the unique advantages of this study methodology could be applied to pragmatic studies in areas where we have need of knowledge.

STUDY DESIGN DURING A PANDEMIC

As COVID-19 cases grew exponentially in the United States in March 2020, the need for a rapid exploration of possible therapeutics became apparent. Small, observational trials had been conducted in China and France, but randomized, controlled trials were lacking. To recruit a large number of participants quickly to get an answer as soon as possible, we would need to be able to reach outside of our local region. Further complicating the rapid development of clinical trials was the closure of in-person clinics due to the pandemic, as well as a shortage of personal protective equipment that would be required for in-person screening of outpatient-based participants. Given this, we opted to instead conduct our trials virtually, using an internet-based format to recruit participants and direct them to an online survey portal [7]. After enrollment, eligible participants were randomized to either hydroxychloroquine or placebo, and study medication was sent to them by overnight courier. Not only did this avoid in-person interactions with participants (and a risk of occupational exposure to COVID-19), it permitted rapid nationwide recruitment. Trial endpoints were designed as patient reported outcomes.

PATIENT CONTENT STATEMENT

All participants signed an electronic consent form before enrolling in the studies described above. The designs of the studies mentioned in this manuscript were approved by the University of Minnesota Institutional Review Board.

DATA COLLECTION

The Research Electronic Data Capture (REDCap) digital survey tool was used to capture self-reported participant survey data. We designed the surveys to guide the participants to provide accurate responses. Branching logic and field tags provided in REDCap were used to dynamically hide or show fields related to input from the user, establish upper and lower bounds to limit data input for numeric fields, and capture the date of survey completion without the participant manually entering it. Every attempt was made to ensure the participant had a clear understanding of what each question was asking in concise, plain English (or French in Quebec). Study-specific terms were defined at the beginning of the screening survey, and the wording of questions in follow-up surveys was chosen to maximize participants’ understanding even if a prior follow-up survey was missed. Utilizing an online survey format allowed us to make minor revisions as needed, often prompted by participant inquiries. All our surveys and automated features were designed and tested in 5 days. Further testing with nonmedical persons and patient partners likely would have identified clarity issues earlier.

As the global understanding of SARS-CoV-2 increased, adjustments needed to be made to our inclusion criteria and directed questioning. These changes were most often related to what constituted a high-risk exposure or which symptoms were considered consistent with infection after an exposure. Examples of this were anosmia and gastrointestinal symptoms, both of which were sporadically reported during the early outbreak in China but became clearly associated with COVID-19 several months later. The US clinical case definition was not standardized until April 6, 2020 [8], and Centers for Disease Control and Prevention guidance regarding healthcare worker risk evolved multiple times during the trial. Figure 1 provides a timeline of milestones during our trials’ development and progress, from project launches to publication.

Figure 1.

Progress timeline for all 3 studies (postexposure prophylaxis [PEP], preemptive treatment [PET], and pre-exposure prophylaxis [PrEP]). DMSB, Data and Safety Monitoring Board; FDA, US Food and Drug Administration; IND, investigational new drug; IRB, institutional review board.

SCREENING AND ENROLLMENT

To facilitate screening participants, we created a survey that prompted users to provide data relevant to our inclusion and exclusion criteria. A calculated field, hidden from participants, automatically determined eligibility based on their responses. Once the screening survey was completed, eligible participants were automatically sent an email containing a link to the enrollment survey; those who were ineligible were also notified via email. Emails were sent via the REDCap Automated Invitation system. This method of sending the enrollment survey only to eligible participants provided 2 benefits: (1) it required participants to complete and submit the screening survey before finding out if they qualified, preventing them from changing answers to alter their eligibility before submitting the survey; and (2) it verified that enrolled participants had a valid email address that could receive automated emails from the REDCap system, vital for complete collection of follow-up data. Participants need to be warned to check their spam/junk folder because some spam filters erroneously blocked REDCap emails even if they were sent from university email accounts.

Although the screening and enrollment were automated, we did institute a manual quality control process to prevent randomizing ineligible participants. The most common reason for not randomizing a screened participant was an incomplete enrollment form, which in most cases was an incomplete mailing address for the study medication shipment. If contacting the participant by email did not resolve the issue, they were not randomized. Similarly, participants who completed the screening survey more than once (identified by using the same email or mailing address) were contacted. If no adequate explanation was given for the duplicate, they were not randomized. Even though many of our exclusion criteria were not publicly available to potential enrollees, we needed to take precautions against participants who screened multiple times to “fish” for variables they could change on subsequent screening surveys to alter their eligibility. In 1 extreme case, an individual completed the screening survey 16 times over 1 hour, changing only 1 or 2 variables each time. Another example of record duplication involved a for-profit trial recruitment company completing our screening survey multiple times to probe our internal survey logic. Having a standard operating procedure (SOP) for carefully reviewing each new enrollment is key to minimizing these issues (Table 1).

Table 1.

Enrollments Not Randomized in Internet-Based Randomized Clinical Trial

Rationale	Total Records (n)	Recommended Response
Deemed ineligible by internal survey logic	6924	Pilot survey logic before launching trial
Incomplete enrollment form	44	Carefully review each enrollment for “valid” completion of key fields
Duplicate screening forms	14	Screen all new enrollees against full database, looking for duplication in identifying fields
Participant did not meet criteria	6	Continual review of internal survey logic, especially after updates
Participant located outside United States or Canada	1	Screen all new enrollee addresses; require valid US zip code for enrollment form submission
Potential fraud	1	Monitor enrollee emails for domains potentially linked to groups with interest in private trial data

NOTE: Table 1 presents the major reasons for not randomizing persons having completed both screening and enrollment surveys. The trial used a 2-part survey for screening then a follow-up email invite for an enrollment survey, if eligible. This 2-part process validated the email address. The vast majority of 6924 participants fully completing a screening survey were ineligible and screened out via automated internal survey logic and not offered the enrollment survey. The above issues were identified during manual review of completed materials before randomization. Total number of enrollment forms submitted for randomization = 1388 in postexposure prophylaxis and preemptive therapy (PEP and PET) trials [1, 2].

As the COVID-19 pandemic was evolving rapidly, the inclusion and exclusion criteria were updated as necessary between March 17 and April 24, 2020 from additional US Food and Drug Administration (FDA)-mandated exclusion criteria or from recognition of risk. An example was expanding the definition of healthcare workers to include first responders in late March 2020 and eventually a final change in late April 2020 to “occupational exposure.” As updates were made to our algorithm, we made new calculated REDCap variables to determine eligibility and updated the automated invitation logic. In some situations, the data collected may not have been as clear cut as it could have been, due largely to evolving case definitions, availability of testing, and public perception of the trials themselves affecting participation. Agility is required to make rapid adjustments because the circumstances of the pandemic can change.

RECRUITMENT

Our recruitment strategy was focused on spreading awareness of the trial using traditional and social media. Initially, an email address was the path for recruitment with incoming emails receiving an automated email response with a participant information sheet description of the trial and a URL link to the REDCap screening survey. We later created a University of Minnesota-supported website for our project (https://covidpep.umn.edu) to provide information about the trial for potential participants as well as linking to our screening survey. Team members promoted this website in social media posts and mentioned it in most interviews in which they participated. We also maintained a “call schedule” for monitoring study-related email addresses, aiming to quickly provide answers to participant questions or feedback. In several instances, this allowed us to clarify whether someone was indeed eligible for our trials.

The use of social media to spread knowledge of the trial created unique challenges. We attempted to advertise on a prominent social media platform and also a search engine to increase recruitment using targeted geographic advertisements based on emerging COVID-19 hotspots. However, their algorithms rejected our ads as promoting a commercial product and attempts to reach representatives at these companies were fruitless.

Social media was highly effective for word-of-mouth dissemination with mixed results. Social media posts in state-specific physician groups and broader healthcare worker groups were useful, particularly for our PrEP trial, because it focused more closely on persons in these professions. Less effective was social media in nonhealthcare groups. For example, 2 weeks into our study, approximately 2600 screening surveys had been completed. Unbeknownst to the study team, an inaccurate graphic (claiming the trial was giving out free COVID-19 medication) had spread via WeChat, a popular social media platform among the Chinese American community. As a result, over 6000 screening surveys were entered over the following 48 hours, creating a large amount of data that needed to be parsed for accuracy and legitimacy. Fortunately, most screening surveys were not fully completed, or automated algorithms screened out ~99%. This type of social media “enroll to get free hydroxychloroquine (or placebo!)” was overall unhelpful. Figure 2a and b provide a timeline of significant external events in relation to daily screening and enrollment numbers.

Figure 2.

Comparison of the number of completed screening surveys versus completed enrollment by date of completion in 2020 for the postexposure prophylaxis/early treatment trial (A) and pre-exposure prophylaxis trial (B). Bars represent the total number of completed screening surveys each day, whereas the line represents the number of enrollments each day. Lettered markers have been placed at the date of external events that may have influenced recruitment. (A) Interview on ABC Good Morning America; (B) incorrect infographic regarding study protocol spread over WeChat social media platform; (C) interview on Dr. Oz television program; (D) Brazilian study demonstrating risk with high-dose hydroxychloroquine [12]; (E) VA study demonstrating no benefit from hydroxychloroquine in hospitalized coronavirus disease 2019 patients [13], (F) US Food and Drug Administration (FDA) warning issued regarding safety of hydroxychloroquine.

One component of recruitment we did not focus on, largely due to time, was creating recruitment materials in non-English languages or that targeted minority communities. In Canada, where all study documents were translated to French, the bilingual materials greatly increased recruitment in the French-language speaking province of Québec. Efforts focused on translating important study documents to other languages can both help recruitment as well as create a more generalizable study population. Overall, our efforts led to 7139 completed screening surveys by the end of the postexposure prophylaxis/preemptive therapy (PEP/PET) trials. There were an additional 7081 incomplete screening surveys created by people who started but did not finish the survey. Of the completed screening surveys, 18.4% (1312 of 7139) resulted in a participant being enrolled and randomized in our trials.

STUDY MEDICATION ASSIGNMENT AND DELIVERY

Having an existing relationship with an academic pharmacy at our institution allowed us to quickly arrange for the assignment and distribution of study medications Monday through Saturday. It was a substantial effort on the part of the pharmacy, so much so that when we launched our PrEP trial, we recruited a commercial pharmacy to assist in medication management. The study team would transmit new prescriptions for blinded study medicine to the pharmacy daily. Randomization was performed at the pharmacy, with a log recording randomization sequence. Assigned prescriptions were filled by the pharmacy team, placed into sealed opaque brown bags for further blinding, and picked up by our study coordinator for packaging and shipment. In the early weeks of our trial, we did not have a dedicated team member who handled these tasks, instead rotating duties among team members. Efficiently managing medication shipments requires dedicated personnel as recruitment increases.

Only one third of participants completed enrollment surveys weekdays between 8:00 am and 4:00 pm. Participants who enrolled outside of daytime work hours (44.4%, 583 of 1312) or during the weekend (22.3%, 292 of 1312) typically experienced longer delivery times. The median time from initial enrollment survey completion to delivery was 36 hours (interquartile range [IQR], 23.7 to 42.4 hours). Figure 3 provides the frequencies of estimated shipping times, with variations being due to the time or day of the week in which the participant completed their initial enrollment survey. Although waiting until the next working day to enroll someone would have artificially decreased the shipping time, this would have simply delayed the time to study entry. In the PEP trial [1], the total time from reported high-risk exposure to study medicine receipt was a median of 3 days (IQR, 2 to 4), very similar to the timing of the Barcelona trial’s median of 4 days (IQR, 3 to 6 days) [9]. The ability for persons to enroll in a research study during nontraditional hours likely increases the numbers of persons able to participate. This is especially true for busy healthcare workers, who comprised two thirds of participants in our studies. Planning study workflow to account for after-hours screening and/or enrollment is essential.

Figure 3.

Histogram presenting the distribution of times from participant enrollment to delivery of medication in hours for participants. Total US packages sent was 1255, with a median delivery time of 36 hours (interquartile range [IQR], 23.7 to 42.4 hours). Two thirds of participants enrolled outside of weekday daytime hours (875 of 1312). The first peak is those enrolling during weekday daytime hours. The second peak is those enrolling in the evening or night, being shipped the next day. The final peak is those enrolling after 3:00 pm on a Saturday or on a Sunday, being shipped on Monday, and receiving delivery on Tuesday morning. For the postexposure prophylaxis trial, the median time from highest risk exposure to starting study drug was 3 days (IQR, 2 to 4) days.

Delivering medication via courier created a few challenges. In the United States, shipment was handled via FedEx and their Ship Manager software, which enabled batch processing of shipping labels based on exported data from REDCap. In Canada, shipping was donated by Purolator Inc. and used data exported from REDCap to their software. Study medicines were shipped overnight, generally. We asked where participants would be at 10:30 am the next day, which may have been at home, work, or other address (eg, hotel while quarantined). Our Day 1 survey was sent at ~11:00 am, shortly after the anticipated delivery time to verify delivery and if the participant had taken their first dose. When deliveries went astray (~2%), troubleshooting with the courier usually allowed us to arrange redelivery. Although rare, a few packages went missing (0.2%, 3 of 1312) or were undeliverable and returned.

FOLLOW-UP

We used REDCap to send out automated emails on a predefined schedule with a link to follow-up surveys. Follow-up surveys were concise, consistent with the pragmatic design, and focused on collecting information only on current symptoms, symptom severity via a 10-point visual analog scale, hospitalization, and side effects. In addition, medication adherence was asked at completion of study medicine (day 5) and the participant’s guess as to their randomization assignment to assess for adequacy of blinding (day 14). These concise follow-up surveys took ~2 minutes to complete. Approximately, 75% of follow-up surveys were completed without additional prompting.

Tracking down those missing surveys to minimize lost to follow up proved more difficult and where the majority of labor was expended. We created reports in REDCap that listed contact information for participants who were missing surveys, allowing us to follow-up with them specifically. Assume there will be incomplete surveys and develop an SOP for all participant follow ups utilizing a predetermined number of voice calls, short message service (SMS) messaging, and email. We primarily used texting and email formats because we believed these to be the least bothersome to participants. However, phone calls resulted in a high rate of data completion, especially on weekends. We developed prewritten scripts to ensure consistent information for calls including voicemail messages. Calls also provided participants a more direct means of asking questions, which often allowed us to clarify issues that could have resulted in loss to follow up. The SMS messaging was usually limited to last attempts to obtain any outcome data from participants who had proven difficult to contact. These messages were sent using a study-associated Google Voice number, which was linked to one of the study’s email accounts for responses. A branded caller ID name for phone calls would have likely been better because it would clearly identify who was attempting to get in touch with them. With using multiple strategies for participant outreach, 12.6% (165 of 1312) of our participants did not complete follow-up surveys, 7.2% (95 of 1312) of whom had unknown vital status information, less than the 20% assumed when designing the study [1, 2]. In the 12-week PrEP trial, 84% (1247 of 1483) of participants completed the final study termination survey [3]. By diminishing the lost to follow-up rate, this ultimately decreased the overall sample size requirement.

Accurately confirming all hospitalization and deaths among our study population was also challenging. Despite being asked, not all participants provided accurate emergency contact information. Of those who did, not all emergency contacts responded to follow-up calls. When a participant ceased all follow up, it was often unclear whether this was the result of a hospitalization or death or simply lack of interest in continuing participation. Developing a system for tracking down vital status is key. For instance, we used online searches for obituaries as one way of determining whether participants had died. Having social media contact information may have been a better way to assess vital status for younger participants.

In retrospect, we could have incorporated calls at an earlier stage in follow-up outreach, using them to ensure data completeness and as an open channel of communication for participants to ask questions. During the follow-up calls, we discovered common trends in participants’ misunderstanding of the trial, so adjustments were made to our email scripts to address the common questions directly. Common questions included whether they could still participate if they skipped a medication dose or delayed taking the first dose after receiving the package, how many surveys they needed to complete, and how often surveys were sent. Despite a downloadable consent form and comprehension assessment during enrollment, it became clear that it would have helped to have a more dynamic means of explaining participant expectations. Methods we retrospectively considered were an introductory video in the enrollment survey with a team member giving a verbal description of the study, consent processing over a brief Zoom meeting with a team member, or a “question line” staffed by a team member during business hours for study-specific questions in addition to the email line.

One significant advantage to our follow-up strategy was the ability to recruit current participants into new substudies. Two substudies conducted were serology and a hydroxychloroquine pharmacokinetic study. Utilizing our online database of enrolled participants, we sent an email asking for those interested in further research to complete a separate consent form via REDCap. Once consented, we created packets containing Whatman 903 Proteinsaver cards and lancets for the serology study, and Neoteryx volumetric absorbed microsampling kits (Neoteryx, Torrance, CA) for the hydroxychloroquine pharmacokinetic study, and sent them via courier to participants. These packets also contained written instructions for self-collection of either blood spots (for Proteinsaver cards) or whole blood (for microsampling kits). Although we did have significant success in having participants collect and return samples, we found that microsampling was a more useful method of collecting usable blood samples for a variety of possible downstream purposes, such as pharmacokinetics, serology, and immunology. Table 2 summarizes the challenges and lessons learned.

Table 2.

Challenges and Lessons Learned for Internet-Based Trials

Challenges	Lessons Learned
Case Identification	• Lack of laboratory testing in United States during March–April 2020 or PCR results were delayed for outpatients
	• Use of epidemiologic linkage allowed for rapid enrollment of symptomatic cases for early treatment (eg, symptomatic household contacts of PCR+ persons or exposed healthcare workers); yet, also defining this as an a priori subgroup for analysis.
	o Persons enrolling via epidemiologic linkage enrolled a mean of 1.2 days faster than those with PCR confirmation [2].
	o Over 4 times as many persons were excluded as enrolled due to inability to access PCR testing or receive results within 4 days of symptom onset for the early treatment trial [2].
	• Using a clinical case definition with independent adjudication in addition to PCR confirmation
	o False-negative rate of PCR is ~38% on day 1 of symptoms, creating challenges for early diagnosis [11].  o Cannot ignore “probable cases” clinical case definition with PCR+ epidemiologic linkage [8].
Recruitment	• Utilizing internet-based, automated recruitment enabled self-enrollment 24 hours per day; useful for persons working full time or those doing shift work, as well as expediting enrollment.
	• Advertise through both traditional (print, radio, TV) and nontraditional media (Twitter, Facebook, Instagram) accessed a wider audience.
	• More formal collaboration with coinvestigators to recruit participants instead of setting up competing trials.
	• Email hotline answered questions with an on-call schedule.
	• Transparency in recruitment posted to social media.
Recruitment of Minorities	• Although these trials over enrolled persons of Asian descent, due to circulation in Asian-American social media community, other minority populations were significantly underrepresented.
	• Targeted outreach to other minority communities is required; Involving local community groups and leaders to advertise the study.
	• Translation of study materials in multiple languages.
	• Videos explaining the research should include diverse speakers.
Obtaining Complete and Accurate Follow Up	• Continuous quality improvement of survey instructions and questions in response to feedback, common questions, or evolution of knowledge of COVID-19; particularly in the first 2 weeks of implementation.
	• Two part screen and then enrollment to verify email address and prevent altering answers regarding eligibility.
	• Not all criteria were publicly revealed to eliminate persons gaming their answers to meet enrollment eligibility.
	• Enrollment criteria via RedCAP survey logic were easily modified. Survey logic for prophylaxis trials were more restrictive to target a moderate/high-risk population than the broader protocol criteria. This allowed for modifying risk criteria as CDC guidance on exposure risk evolved over time.
	• The online nature of the trial led some participants to have less investment in the trial (or less clear understanding) for survey completion. Using video calls with participants at the beginning of the trial would have perhaps created more personalized interactions.
Shipping Study Drug	• FedEx overnight shipping was utilized. Sunday delivery does not occur. Consider either not enrolling on Saturday night and Sunday daytime or incorporating the shipping lag into enrollment criteria.
	• We asked for an address where a person could receive shipments at 10:00 am the next day. Further clarity on shipping timeline, based on evening/night/weekend enrollment, would have been helpful.
	• Verify addresses to minimize errors and send tracking numbers to participants so that they are aware when the package should arrive.
	• Quarantine resulted in change of addresses or use of temporary addresses by participants with COVID-19.
Study Design	• Create concise video explaining trial design—as a 21st century participant information sheet.
	• Needed a better explanation regarding study withdrawal.
	o Participants could stop the study medicine but remain in the study as outcomes were still wanted.
	o Standard templated consent language insufficient.
Follow Up	• Figure in the consent form displayed follow-up timeline. Yet, some participants appeared unclear as to the follow-up timeline. Sending this timeline in reminders may have been helpful.
	• Collect the type of phone number, eg, mobile vs landline
	• Ask preferred manner of contact for follow up; email vs text messages. Preference varies by age and by person.
	• REDCap has an integrated system to send out surveys via text messages with URLs.
	• Many people do not answer telephone calls from unknown numbers; having a branded caller ID may improve answer rate.
	• Obtain a back-up contact for people who become hospitalized or are too ill to respond to follow-up questionnaires.
	• Collect social media usernames as an alternative method to determine vital status.
External Events	• Ability to rapidly communicate updates to participants regarding external events; but with IRB-approved messages slowed the process.

Abbreviations: CDC, Centers for Disease Control and Prevention; COVID-19, coronavirus disease 2019; ID, identification; IRB, institutional review board; PCR, polymerase chain reaction; REDCap, Research Electronic Data Capture.

TRADE-OFFS

Medication adherence was mixed. In the PEP/PET trials, approximately 15% of persons never started the blinded study medicine (and/or were immediately lost to follow up and did not report taking the study medicine). Negative media attention, input from personal physicians or family, and FDA warnings on the dangers of hydroxychloroquine all influenced participants deciding not to take the study medicine. Whether an in-person trial would have altered this is unclear. We would recommend having a day-1 contact to enhance a personal connection and answer questions or concerns to minimize lost to follow up or trial withdrawal. This could have been via automated personalized text message or video call. In retrospect, we should have used a modified intent-to-treat analysis, excluding those who never initiated the study medicine. This did not alter our results, because a per-protocol analysis of those 100% adherent to the study medicine was included as a sensitivity analysis.

These trials used an existing FDA-approved medicine with a known safety profile that did not require laboratory monitoring [10]. Utilizing patient reported outcomes created relatively concise surveys focused on relevant topics of interest, not as comprehensive as one would need for an FDA registrational trial. This trade-off was made to keep survey questionnaires short, requiring <2 minutes to complete follow-up surveys. More comprehensive surveys may have resulted in higher lost to follow up.

COSTS

Although there were trade-offs, the budget was approximately $350 000 in total to conduct the 3 trials, funded by generous, unsolicited private donors. United States federal funding was requested but unsuccessful (n = 4). Numerous investigator salaries were covered from other sources including the following: provincial governments of Quebec and Manitoba, institutional sources, National Institutes of Health (NIH) career development awards (n = 3), T32 fellowship, Fogarty International Center fellowship, Doris Duke Foundation, and NIH R01 research project awards (n = 4) while research was paused from COVID-19. In comparison, the budget for traditional in-person trials using NIH networks was $10 million for an early treatment trial and $50 million for a PrEP trial. Although in-person trials can collect much more data, the cost is exponentially greater.

CONCLUSIONS

Internet-based trials are a promising means of designing agile, wide-reaching clinical trials, particularly for rare diseases or neglected (unfunded) diseases, such as most of infectious diseases clinical research. However, internet-based trials do come with unique challenges and limitations. In our trials, these challenges were compounded by conducting a clinical trial during a global pandemic using a study medication that became politicized. Having team members dedicated to participant outreach and follow up and a dynamic flow of information between participants and the team may help reduce loss to follow up or inadvertent deviations from study protocols. In addition, development of a more nuanced relationship with major social media platforms would provide more accurate and consistent messaging about clinical trials, although the onus for improving this relationship may primarily be on the platforms themselves. The challenges presented here are not insurmountable, although they are essential to consider when launching a primarily internet-based trial. We have recommended possible solutions, although new technological innovations in telemedicine and data collection could provide even more efficient means of communicating with participants in internet-based trials and ensure high-quality data capture using patient-reported outcomes.