Developing a virtual assessment protocol for the AMPLIFI Randomized Controlled Trial due to COVID-19: From assessing participants' preference to preparing the team.

Background During the COVID-19 pandemic, in-person research assessments needed to be adapted to ensure safety of participants and staff. Participants' willingness to participate in research activities, how to prepare assessors to ensure data integrity, and the feasibility of modified protocols, were unknown. Within the AMPLIFI randomized clinical trial (RCT) for cancer survivors, we elicited participants' preferences and willingness to participate in Clinic, Home, or Virtual assessments, prepared assessors for, and implemented virtual assessments. Methods 1) We conducted phone surveys of potential AMPLIFI participants; 2) Based on survey results, we modified assessments from in-person to virtual visits (VV) by videoconference. Assessors were trained and certified, i.e., assessors recorded 3 assessments that were reviewed and scored by 2 investigators. The modified protocol was proposed to 62 participants: we report numbers of those who agreed to attend VV. Results 1) Survey results: Among 74 survey respondents, 44.6% preferred, 75.7% were willing to attend Clinic Visits; 32.4% preferred, 83.8% were willing to do VV; 23% preferred, 77% were willing to do Home Visits. Survivors 70+ were less likely than 50-69 years old to be willing to do VV: no other differences were noted by gender, race, rural status or education. 2) Assessment uptake: 66.1% agreed to attend VV, and of them 75.6% completed them. Conclusion Diverse research participants adapted to protocols that prioritize their safety, although older participants may be reluctant to do virtual assessments. Virtual assessments are feasible and research teams can rigorously prepare to collect quality data through them.

Introduction

The coronavirus pandemic has significantly changed the research enterprise across the globe. With little notice, researchers had to find alternative ways to conduct human subject research while ensuring study rigor and the safety of both participants and staff. There were no guidelines or best practices on how best to change research protocols that required in-person assessment visits to measure study outcomes, and proposed solutions were often accompanied by new challenges to overcome. Further, there was no information from potential study participants regarding the acceptability of in-person assessment visits under the auspices of social distancing and personal protective equipment (PPE), or the acceptability of alternative virtual visits by videoconference. Addressing this knowledge gap was key to developing assessment protocols that would be safe, acceptable, and respectful to study participants.

In this paper, we describe the approach to change the assessment protocol in response to the COVID-19 pandemic for the Adapting MultiPLe behavior Interventions that eFfectively Improve Cancer Survivor Health (AMPLIFI) study. AMPLIFI is a National Cancer Institute (NCI) supported (P01 CA29777) randomized controlled trial (RCT) to test various web-based interventions to improve diet and physical activity and reduce weight. The RCT will enroll 652 survivors of obesity-related cancers with oversampling of minority, older, and rural participants. Outcomes include anthropometric, physical performance assessments and the collection of blood to assess inflammatory/metabolic biomarkers. According to the original AMPLIFI protocol, these were to be collected during in-person assessment visits conducted every 6 months in either the survivor's home, university research clinic, or in a community setting. The planned launch of the AMPLIFI trial coincided with the start of the pandemic in our region in March 2020. Given the increased risk of severe COVID-19 illness among individuals with a history of cancer, in-person assessments were not conducted and alternatives were evaluated to continue study assessments safely.

To address the lack of knowledge about study participants' acceptability of different assessment methods (e.g., in-person vs. virtual), we first describe our effort to elicit this information: we asked potential AMPLIFI RCT participants their preferences for, and willingness to, attend in person assessment visits with COVID-19 safety protocols approved by our university (at research clinics or at home), or virtual visits by videoconference. We obtained this information using a brief phone survey developed specifically for this study. Mindful of the diversity goal for AMPLIFI recruitment, we assessed acceptability by age, race, rural status, gender, and education. Given the results of the survey, AMPLIFI adopted a virtual protocol to supplement in-person assessments. In this paper, we then describe how we adapted the in-person assessment protocol to a virtual one, and the results of the initial uptake of the virtual protocol and challenges encountered.

Methods

Approval for this AMPLIFI RCT sub-study was obtained from the Institutional Review Board (IRB) of the University of Alabama at Birmingham (UAB). The UAB IRB serves as single IRB for this multisite study which includes the University of Tennessee Health Science Center as a partner.

Part 1: survey on acceptability of assessment methods

Survey participants

Survey responders were recruited among 92 AMPLIFI potential research participants, i.e., survivors who expressed their intent to participate in the AMPLIFI RCT before research activities stopped due to COVID-19. The AMPLIFI study targets cancer survivors who meet these eligibility criteria: (1) age 50 years old or older; (2) 1 year post from a diagnosis of multiple myeloma, or localized and regional cancers of the breast, colorectum, endometrium, prostate, or localized cancers of the ovary or kidney; (3) community dwelling and from areas with wireless coverage; (4) English-speaking/reading; (5) education of 8th grade education level or higher; (6) without physical limitations that preclude unsupervised physical activity, such as recent heart attack, dementia, blindness; (7) with some evidence of functional limitation (as assessed with the Medical Outcomes Study Short Form 36 [SF-36]); (8) BMI of at least 25 kg/m2 but less than 50 kg/m2; (9) <150 min of self-reported weekly moderate-to-vigorous level physical activity); and (10) intake of <2.5 daily servings of fruits and vegetables. Recruitment, screening, and enrollment for AMPLIFI was conducted by the Recruitment and Retention Shared Facility of the O'Neal Comprehensive Cancer Center at UAB. While the larger study will recruit nationally, for this current sub-study survivors were identified from state and hospital-based cancer registries of Alabama, North Carolina, and Mississippi.

Survey process

Surveys were conducted over the telephone by the AMPLIFI staff from July to August 2020, and responses recorded using the Research Electronic Data Capture (REDCap) platform. A detailed explanation of COVID safety for research clinic or home visits was provided and followed by a questionnaire which solicited preferences for clinic, home, or virtual (by videoconference) visits. The proposed videoconference platform was Zoom. Survey participants selected their first, second, and third preferred option. Moreover, they were asked if they were willing to attend assessments in clinic, at home or participate in virtual assessments.

Main outcomes

We examined preferences for, and willingness to attend, clinic, home, and virtual (by videoconference) research assessment visits.

Statistical analysis

Descriptive statistics were obtained to characterize the study sample, the proportion of respondents who chose clinic, home and virtual assessment visits as their first preference and the proportion willing to attend each assessment visit type, in total. In addition, we examined differences in the main outcomes by participants' gender, age, race, rural/urban status, and education level. These were tested for statistical significance (alpha = 0.05) using Chi-square tests (or Fisher's exact test if the assumptions for the chi-square test were not satisfied). Analyses were conducted using SAS (version 9.4; SAS Institute, Cary, NC).

Part 2: adaptation of assessments from in-person to virtual and uptake

The AMPLIFI assessments included weight and waist circumference measurements, balance tests, and functional performance tests, many of which were from the Senior Fitness Test battery [1], such as sit and reach, back scratch, chair stand, 8′ up and go, 8′ walk, and 2-min step test. In addition, we measured blood pressure, collected information on comorbidities, medications and supplements, and obtained blood and urine samples. Virtual assessments were done using Zoom. To be able to do the virtual assessment, participants had to have i) an internet-connected device with a webcam and microphone; ii) a partner to help during the assessment; iii) a scale to measure weight; iv) a standard height (non-upholstered) chair; and v) a 12′ space to perform the functional performance tests.

To prepare and conduct virtual assessments, we did the following:

Produced instructional videos to demonstrate what a virtual assessment would be like, with specific instructions for each test and measurement;

Added a call before the assessment visit to review instructions to set-up and use the videoconferencing software (i.e., Zoom);

Sent necessary supplies to participants: each box included a manual of instructions, two ribbons to measure waist circumference (in duplicate), a vinyl tape measure, a cone and stickers to mark the ground for the 8′ walk, and stickers to mark the wall for the 2-min step test. We also sent a wrist sphygmomanometer to measure blood pressure; and

Changed the method to obtain blood samples from venipuncture to finger stick using lancets and Dried Blood Spot collection cards (DBS) that participants could easily do on their own. Supplies were sent with the assessment supply box.

All supplies were sent back by participants to the AMPLIFI team. Moreover, because of the differential in sight and sound Zoom transmission rates, the assessment visits were recorded to obtain accurate readings of timed tests.

Four assessors who received training to conduct the in-person assessments were re-trained and certified for virtual assessments. The certification process for each assessor involved conducting assessments with three volunteer pairs who were each given standardized scenarios to “perform” as cancer survivors and partners. Each scenario included challenges such as equipment or participant-related problems (e.g., non-informative camera angles, participant imbalance or dizziness, functional tests not set up correctly), to test the assessors' knowledge of the assessment and safety protocols. Assessors recorded the sessions and submitted the recording as well as the data entry forms to the evaluation team. Evaluators (MP, WDW, LR, KH, TH) reviewed the videos and completed a certification form with a 62-point checklist on two domains: 1) communication skills, e.g., ability to establish rapport and provide appropriate instructions to participants, appropriately respond to questions; and 2) assessment and data entry accuracy, e.g., ability to identify appropriate and safe space for conducting assessment tests, perform assessment tests, capture appropriate measures and images on Zoom, and correctly record assessment data. Each assessor had to earn 80% of the points from two evaluators on each recording to pass the certification. If the assessor did not pass on a recording, the assessor repeated that recording which was then reviewed and scored by another evaluator. If evaluators disagreed on a recording, a third evaluator would review. At the end of the certification process, evaluators provided feedback to the assessors.

In Fall 2020, we proposed the virtual assessment protocol to 105 participants who had agreed to participate in AMPLIFI. Participants were contacted to consent or re-consent to the new protocol and schedule for a virtual baseline assessment. We report here the number of those who met requirements for virtual assessments, who agreed to do virtual assessments, and completed virtual assessments. We also report the challenges faced and the lessons learned.

Results

Survey participants were 74 survivors (response rate 80.4%) with the following characteristics (Table 1 ): 78.4% female; 69% were age 50–69 and 31% age ≥ 70 years old, 66.2% White and from urban areas (determined based on ZIP code), and almost 80% with at least some college education. The majority had a diagnosis of breast cancer (58.1%) and were 3–6 years post diagnosis (85.1%).

Table 1

Characteristics of survey participants (N = 74).

Characteristic	N (%)
Gender
Female	58 (78.4%)
Male	16 (21.6%)
Age categories
50–59	23 (31.1%)
60–69	28 (37.9%)
70–79	20 (27%)
80+	3 (4.0%)
Race/ethnicity
White	49 (66.2%)
Black or African American	24 (32.4%)
Other minority	1 (1.4%)
Rural/urban
Urban	49 (66.2%)
Rural	24 (32.4%)
Missing	1 (1.3%)
Education
Grade 12 or less	15 (20.3%)
College 1–3 years	30 (40.5%)
College 4 year or more	29 (39.2%)
Cancer type
Breast	43 (58.1%)
Prostate	12 (16.2%)
Colorectum	6 (8.1%)
Endometrium	5 (6.7%)
Kidney	3 (4.1%)
Ovary	3 (4.1%)
Multiple myeloma	2 (2.7%)
Years post cancer diagnosis
3–6	61 (85.1%)
7–10	13 (14.9%)

Clinic visit was chosen as the first preference for conducting assessments by 33 respondents (44.6%), virtual visit by 24 (32.4%), and home visit by 17 (23%) (Fig. 1 ). In addition, half of the respondents chose the home visit as their second choice. Respondents who preferred the clinic visit as their first choice did not differ significantly in gender, race, age, rural/urban status, or education from respondents who did not choose this option as their first choice (Fig. 2 ). Respondents who chose virtual visits as their first choice were more likely to be <70 years old (p = 0.04). Those who chose home visits as first preference were more likely to be 70 years old or older (52.9% vs. 25.0%, p = 0.02), with a trend toward a lower educational level (grade 12 or less 41.2% vs. 14.3%% p = 0.06) than other respondents (Fig. 2).

Fig. 1

Preference ranking for clinic, virtual and home assessment visits (N = 74).

Fig. 2

Proportion of survey respondents who ranked Clinic, Virtual, or Home visit as their first preference (N = 74). Respondents age 70+ were more likely to prefer Home visit (p = 0.02) and less likely to prefer Virtual (p = 0.04) than younger respondents. Other differences were not statistically significant.

Fifty-seven respondents (75.7%) stated they were willing to attend assessments in clinic. When asked what made them comfortable about coming to the clinic, 64.3% indicated that it was the cleaning and disinfecting procedures, 57.1% that it was the mask requirement and provision, 46.4% that it was the COVID screening before entering the clinic, 42.9% the COVID screening before the visit and the space available in the clinic. Of those not willing to come to the clinic, 25% indicated it was because of concerns about COVID in the region at the time.

When asked about their willingness to participate in the home assessment visit, 57 (77.0%) indicated they would. Measures that made them comfortable about this option included the mask requirement for more than 31%, not driving to the clinic for 28.1%, the COVID screening procedures for 21.0%, the option to do the visit in an outdoor space for 15.8%, the hand sanitizer and gloves provided and the space in their house for 12.3%, and the cleaning and disinfecting procedures for 10.5%.

Of all 74 survey participants, 66 (89.2%) were able to watch videos online, 73 (98.6%) owned a smart phone, computer, or tablet, and 66 (89.2%) had cameras on their devices. Overall, 62 (83.8%) were willing to participate in a virtual assessment visit: of them, 56 (90.3%) indicated they had a partner who could help. Respondents who were willing to participate in virtual assessments did not differ significantly in gender, race, rural/urban status and education, but those who were 50–69 years old (94.1%) were more likely to be willing to participate in the virtual assessment than those 70 or older (60.9%) (p = 0.004) (Fig. 3 ). Thus, while the virtual assessment visit was not the preferred method of completing assessments, it was explored further given that the clear majority of participants were willing to pursue this option and had the means to do so.

Fig. 3

Proportion of survey participants who were willing to have a virtual visit for AMPLIFI assessments by participant characteristic (N = 74). Respondents age 70+ less likely to indicate they were willing to participate in Virtual assessments than younger respondents (p = 0.004). Other differences were not statistically significant.

Part 2: adaptation of assessments from in-person to virtual, and uptake

Among 105 participants who were eligible to participate or had already given their consent to participate in AMPLIFI, we were able to reach 70, and of them, 62 (88.6%) were still eligible to participate in AMPLIFI. Among these eligible potential participants, 41 (66.1%) consented to the revised protocol for virtual assessments and blood collection, 9 (14.5%) did not want to use videoconference but were willing to be on a waiting list for in-person assessments, 5 (8.0%) did not give consent because they did not have a partner or because they did not want to do virtual assessments, and 7 (11.3%) refused participation for other reasons. Of the 41 who gave consent, 31 (75.6%) completed the baseline assessment: of them 28 (90.3%) were enrolled and randomized to the trial and three withdrew. The 9 participants on the waiting list were subsequently offered to attend a clinic assessment visit in December 2020, but none agreed to come citing pandemic concerns.

Several challenges are noted for virtual assessments. First, participants were now responsible for setting up and preparing for the assessments. Some felt overwhelmed when receiving the supply package and the instructions. It was also challenging to guide participants who never used the Zoom videoconference platform to set up and use the app, since this guidance was given over the telephone. Participants did not always review the instruction video ahead of time making the assessment session more challenging. Among those who reviewed the video ahead of time, some realized that their partners were not be able to assist. Second, the internet signal posed some problems. At times there were difficulties with the internet connection during the assessment that caused the signal to be lost or the videoconference video to freeze, thus disrupting the flow of the assessment. Moreover, signal delays made timing of some tests challenging: recording the sessions was imperative to get more accurate timing. Thus, the collection of objective and accurate assessment measures required considerable vigilance and commitment from the assessment staff.

Discussion

In a sample of cancer survivors who were interested in participating in the AMPLIFI RCT, most were willing to participate in virtual research assessments and a third preferred this option to the proposed in-person alternatives. Survivors ages 50–69 were more likely to embrace this option than those age 70 and older. No significant differences in willingness to participate in virtual assessments were found across gender, race, education and rural or urban status. When the virtual assessment protocol was proposed to participants, only a few refused or were unable to participate due to not having a partner or the equipment. Such virtual assessments are feasible although some challenges were noted.

Despite having different preferences for the proposed assessment visits, at least three out of four research participants reported being willing to participate in virtual or in-person assessment visits with appropriate safety protocols. However, concerns regarding the escalating COVID-19 situation may have impacted the actual uptake of in-person options. Alabama averaged 1700–1800 new COVID-19 cases per day when surveys were conducted, but spiked to 2600–2700 per day by the time clinic visits were offered to participants who were not willing or able to do a virtual visit (December 2020): this new context surely diminished interest. Therefore, depending on the risk involved, a segment of the participant population may be lost, especially older participants who were less willing to do the virtual visit. The willingness to participate in remote data collection may also be influenced by additional factors beyond demographic considerations. Our results are in line with others. In a longitudinal brain aging study of older adults in the United Kingdom, approximately 90% were willing to complete their study visit via videocall, and within the first few weeks of this new data collection protocol, 70% of the study visits were successfully conducted [2]. The authors note however, that because their participants were not assessment naive (i.e., prior to COVID-19, participants had completed the assessment battery multiple times, in person), their study results may not inform future research that would require remote assessments from the beginning of the study [2]. Our study shows that virtual assessments may be acceptable to research assessment naïve participants as well.

The virtual setting may discourage some participants due to a higher burden placed on them compared to a clinic or home visit. Assessors have to rely on the participant preparing and setting-up all tests correctly. In fact, in the same study described previously from the United Kingdom, 10% of the sample queried declined virtual study visits for reasons including an unwillingness to use the technology needed for the study visit, not having a home environment that would be conducive to a remote home assessment, and health impediments [2]. Another subset was willing to participate in virtual assessments, but did not have access to the needed equipment [2]. This is an important consideration for researchers as this extra burden may hinder individuals from enrolling in clinical trials, or contribute to withdrawing from them. In our sample, most participants had the required equipment: however, we continue to monitor whether this is an obstacle to virtual assessment visits as we continue recruitment. Perceptions of burden may be amplified in the context of a pandemic which is inherently stressful. In particular, we continue to monitor the enrollment of older survivors who in our survey were less likely to indicate a willingness to participate in virtual assessments. For this group, this perception of burden added to our other requirements, e.g., having a partner, may be considerable obstacles to participation in virtual assessments, and thus in the AMPLIFI research study.

The COVID-19 pandemic has presented scientific investigators with a myriad of unprecedented challenges. Importantly, COVID-19 has attenuated participants' desire and availability to engage in research and adhere to research-related assessment protocols. Recently, one weight loss study for a non-cancer population found that 63% of targeted participants perceived COVID-19 to be a serious enough threat to forego study participation which they may have considered prior to the pandemic [3]. Given that behavioral research in particular often requires regular in-person assessments, conducting assessments in a safe way is critical to recruitment, retention and timely and accurate data collection. Our findings show that virtual assessments are acceptable to participants. Moreover, given their acceptability across diverse groups of participants in rural and urban settings, these methods may allow outreach to populations that researchers fail to reach and engage in research activities given their location or unwillingness to travel to a research clinic or other assessment venue. Importantly, given that even prior to the COVID-19 pandemic about 19% of trials closed without meeting target accrual rates, the need to investigate these new and more flexible methods that may enhance research engagement is very relevant [4]. However, it is worth noting that virtual assessments require adequate staff training and time, as well as equipment for both research staff and participants, that may not be readily available in low resource settings. Future studies should examine the full costs of conducting virtual assessments and in person assessments, and contrast those to the safety and comfort of these options for participants to determine the most cost-effective option in different settings.

At the same time, this pandemic has presented opportunities for innovation. Our study found that utilizing trained research assessment staff for anthropometric and physical performance testing, along with the Zoom application, camera and internet connection, made these assessments feasible and safe. This is an important contribution to the literature and presents a different approach from other researchers who adapted in person research during COVID-19 in cancer survivors and other populations [2,[5], [6], [7]]. In one study, investigators chose to forgo collecting objective outcomes (e.g., body composition, 6-min walk test) in individuals with cancer and focused only on outcomes that they could collect remotely (e.g., questionnaire data and data from electronic medical records) [7]. Importantly, that study was conducted among cancer patients prior to their cancer surgery: investigators may have deemed that it was safest to omit in-person functional assessments because participants may have been most vulnerable at this point of their cancer journey. Future studies should continue to determine the contexts in which remote assessments of physical functioning are safe, feasible, and a reliable approach to data collection in survivors. The work by Moon and colleagues can offer guidance to researchers [8]. In their study of lung cancer survivors discharged home after lung transplant surgery (4 weeks or more post-surgery), the authors provide detailed considerations for assessing physical activity and physical functioning using remote assessments, outline the implementation process of the remote assessment protocol, and provide a plan for data analysis given the different data collection methods (remote vs. in person) and the use of surrogate measures [8]. Moreover, further research is needed to improve rapport building and communication. Researchers should consider bringing in experts on online teaching to improve how virtual assessments are conducted and how they could be adapted to ensure the engagement on different participant populations, for example older participants.

This paper has several strengths worth highlighting which include reporting the perspective of diverse participants, being able to compare these perspectives across different groups, and developing a rigorous certification process to increase the quality of the data that were collected. However, this study has several limitations for consideration. First, we surveyed only cancer survivors eligible for AMPLIFI and who had expressed a willingness to participate in the study. Moreover, participants were individuals 50 years old or older. Therefore, results may not generalize to the larger population of cancer survivors. Second, our study was limited to survivors of select obesity-related cancers who lived in Alabama, Mississippi, or North Carolina. Third, we do not have a pre-COVID comparison for preferences for different assessment visits. Fourth, a considerable amount of time elapsed from when survivors expressed an interest in participating in AMPLIFI to when we contacted them again to offer the virtual assessment protocol. By this time, several were not eligible or had lost interest in the study. Therefore, results on uptake may have been different if we had offered the protocol earlier.

In summary, under the COVID-19 pandemic, research participants reported a willingness to adapt to different research protocols that prioritize their safety. Importantly, similar preferences and willingness to participate were not significantly different across groups of participants. However, caution needs to be used when implementing a virtual assessment visit protocol as some groups such as older adults may be less willing to accept participation. Although some challenges to virtual assessment visits remain, these assessments were feasible and may represent a viable option to engage research participants during and beyond the COVID-19 pandemic. Research should continue to develop and adapt virtual assessments to better address barriers to participation in clinical trials, for example to engage older participants or those who are more difficult to reach either because they face time and travel barriers, or just a feeling of discomfort with the clinical setting.

Funding

This research is supported by the (P01 CA229997, R01 CA246695, R01 CA242737, P30CA013148), and the (134169-CRP-19-175-06-COUN).

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.