Acceptability, usefulness, and satisfaction with a web-based video-tailored physical activity intervention: The TaylorActive randomized controlled trial.

PURPOSE: This study aimed to examine the usage, acceptability, usability, perceived usefulness, and satisfaction of a web-based video-tailored physical activity (PA) intervention (TaylorActive) in adults.
METHODS: In 2013-2014, 501 Australian adults aged 18+ years were randomized into a video-tailored intervention, text-tailored intervention, or control group. Over 3 months, the intervention groups received access to 8 sessions of personally tailored PA advice delivered via the TaylorActive website. Only the delivery method differed between the intervention groups: video-tailored vs. text-tailored. Google Analytics and telephone surveys conducted at post intervention (3 months) were used to assess intervention usage, acceptability, usability, perceived usefulness, and satisfaction. Quantitative and qualitative process data were analyzed using descriptive statistics and thematic content analysis.
RESULTS: Of 501 recruited adults, 259 completed the 3-month post-intervention survey (52% retention). Overall, usage of the TaylorActive website with respect to number of website visits, intervention sessions, and action plans completed was modest in both the video-tailored (7.6 ± 7.2 visits, mean ± SD) and text-tailored (7.3 ± 5.4 visits) groups with no significant between-group differences. The majority of participants in all groups used the TaylorActive website less than once in 2 weeks (66.7% video-tailored, 62.7% text-tailored, 87.5% control; p < 0.001). Acceptability was rated mostly high in all groups and, in some instances, significantly higher in the intervention groups compared to the control group (p < 0.010). Usability was also rated high; mean Systems Usability Scores were 77.3 (video-tailored), 75.7 (text-tailored), and 74.1 (control) with no significant between-group differences. Perceived usefulness of the TaylorActive intervention was low, though mostly rated higher in the intervention groups compared to the control group (p < 0.010). Satisfaction with the TaylorActive website was mixed. Participants in both intervention groups liked its ease of use, personalized feedback, and tracking of progress, but also found completing action plans and survey questions for each session repetitive and tedious.
CONCLUSION: Providing personally tailored PA advice on its own (through either video or text) is likely insufficient to ensure good retention, usage, perceived usefulness, and satisfaction with a web-based PA intervention. Strategies to address this may include the incorporation of additional intervention components such as activity trackers, social interactions, gamification, as well as the use of advanced artificial intelligence and machine learning technologies to allow more personalized dialogue with participants.

1. Introduction

In adults, 150 min of moderate intensity physical activity (PA) per week can significantly reduce the risks of developing cardiovascular disease, type 2 diabetes, cancer, depression, or dementia, and increase life expectancy.‒ Despite these significant health benefits, over half of the Australian adult population (55%) is not meeting national PA guidelines., To reduce the burden of chronic disease and health care costs, effective and scalable population-based interventions that can reach large numbers of people are needed.

Web-based interventions that provide personally tailored PA advice (also known as computer-tailored interventions) have demonstrated effectiveness for increasing PA levels compared to interventions offering generic or targeted information. The personally relevant PA advice is generated through the use of brief online surveys, a message database containing all possible response options, and IF–THEN algorithms that determine what advice is provided based on the participants’ answers to online surveys. Compared with generic messages, tailored messages are more likely to be read, remembered, saved, and discussed with others. Content in web-based, computer-tailored interventions is typically delivered as text-based information. Yet, eye-tracking studies have shown that Internet-based reading predominantly happens by means of browsing, scanning, keyword spotting, and non-linear reading, with less time spent on in-depth and concentrated reading., This means that text-based information may not be effectively disseminated using the Internet., Consequently, computer-tailored intervention content may not be as effective as it could be when the personalized advice is not read, processed, and actioned as intended.

This obstacle may be overcome by delivering web-based PA advice through personally tailored videos. Watching web-based video content is increasingly popular, and videos may be more effective as they reduce the cognitive effort needed to process information, which can lead to better attention and comprehension. Furthermore, computer-tailored interventions using personally tailored videos in other fields (i.e., diet, weight gain prevention, smoking cessation) have demonstrated effectiveness.15, 16‒ This was further supported by our own research in the development of personalized videos, which demonstrated that they are feasible to use and implement, that they lead to more attention and recall (using eye tracking), and that they are effective in increasing PA.

To examine the effectiveness of a web-based video-tailored PA intervention in adults, we also conducted a large, fully powered, 3-group randomized controlled trial (i.e., the TaylorActive trial). The trial examined the long-term efficacy of a web-based PA intervention that provided personally tailored videos and compared it to a traditional personally tailored text condition and a control condition receiving generic information. 20 However, the findings, assessed using ActiGraph accelerometers (Version 3.2.1; ActiGraph GT3X+, Pensacola, FL, USA), showed the web-based, video-tailored TaylorActive intervention was ineffective at increasing PA behavior. Given the encouraging findings from the formative research, and pilot study, this outcome was surprising.

An important next step is to conduct an in-depth process evaluation to better understand why the TaylorActive intervention was ineffective at increasing PA and inform the development of future video-based computer-tailored interventions. To advance the field, the publication of research examining interventions that are ineffective is equally as important as the publication of research examining interventions that are effective. Associated with this, and equally important, is finding out why some interventions work or and others do not. Therefore, the aim of this study was to examine the usage, acceptability, usability, perceived usefulness, and satisfaction with a web-based video-tailored PA intervention for Australian adults (i.e., the TaylorActive intervention).

2. Methods

2.1. Study design

The TaylorActive intervention was a 3-group randomized controlled trial conducted at Central Queensland University in Rockhampton, Australia. Participants were randomized into 3 groups: video-tailored intervention, text-tailored intervention, and control group. Ethical approval for the study was received from the Central Queensland University Human Ethics Committee in August 2014 (H14/07-163). Written informed participant consent was obtained online. Trial assessments were conducted at baseline, 3 months, and 9 months. However, data collected in relation to the process evaluation presented in this paper were collected at the 3-month time point.

2.2. Participants

Eligible participants were aged 18+ years, had broadband Internet access, could speak and read English, were living in Australia, were insufficiently physically active (i.e., ≤150 min of moderate-to-vigorous PA (MVPA) per week), answered no to all questions on the Physical Activity Readiness Questionnaire or obtained medical clearance, were not pregnant, had a body mass index over 17.5, and reported no impairments that could prevent them from becoming more active. Participants were recruited through social media advertisements (i.e., Facebook), traditional media (e.g., radio, The Conversation), email (e.g., Central Queensland University staff), and third-party databases (i.e., www.trialfacts.com). Interested individuals were directed to a recruitment webpage that contained detailed study information and asked them to complete a screening survey to determine their study eligibility.

The TaylorActive intervention

The TaylorActive intervention was described in detail elsewhere., Briefly, TaylorActive was a web-based, personally tailored PA intervention. The intervention aimed to increase various domains of PA (e.g., leisure, active travel, house/garden work, occupation). A text-based library, available to all groups, contained 19 articles about different aspects of PA (e.g., “Why be active”, “Get started walking”, “Make time to be active”). The text- and video-tailored intervention groups received 8 sessions of tailored PA advice, which were generated in response to questions about participants’ current PA, as well as questions relating to evidence-based individual, social, and environmental determinants of PA. IF–THEN algorithms were applied to select personally relevant advice from a comprehensive message database. The intervention content was informed by established health behavior theories including Self-Determination Theory, Social Cognitive Theory, and Theory of Planned Behavior. The main focus was on behavior change through enhancement of self-efficacy, intentions, social support, knowledge, outcome expectancies, attitudes, facilitators, risk perception, intrinsic and extrinsic motivation, need for relatedness, peripheral and central cues, and habits. Furthermore, the following evidence-based behavior change techniques were applied to change theoretical constructs: providing feedback, self-monitoring, goal setting, habit formation, instruction, problem solving, and action planning. PA advice and goals were tailored to participants’ main motivation to increase activity levels as expressed in the baseline survey: (1) improve health, (2) increase fitness, (3) increase strength, (4) lose weight, and (5) reduce stress. The 8 sessions with tailored PA advice were delivered in a set order at a set time. New sessions could only be accessed when previous sessions were completed. Regarding the intervention sessions, it should be noted that the TaylorActive website was designed to be used infrequently (i.e., only when a new intervention session was available, which was weekly for the first 4 weeks and then every 2 weeks for the next 2 months). Up to 3 email reminders were sent when participants did not access new sessions. A website feature for creating PA action plans was also available for both intervention groups. At the end of each session (except for the first and last) participants were asked to set long-, medium-, and short-term PA goals and to create an action plan for how they would meet their PA goals. For example, participants were asked very specific questions about how they planned to meet their PA goals: What PA they will do, where they will do it, when they will do it, how often they will do it, how long will each activity session be, and with whom they will do it. When participants had completed all the questions, the TaylorActive website provided an action plan that could be printed on a single page.

Video-tailored vs. text-tailored interventions

The difference between the 2 intervention groups was how the personally tailored feedback was delivered. In the text-tailored intervention group, feedback was displayed as plain text on the TaylorActive website and supplemented with graphs indicating progress where relevant. In the video-tailored intervention group, feedback was presented as a video for each session with a male or female actor (generically named “Taylor” in either case) that could be selected by the participants. The content of text- and video-tailoring was identical, addressed participants by their first name and was kept brief, as the formative research for this study showed that the videos should be short (range: 4−7 min) to prevent disengagement. Information that could be easily provided in the text-tailored feedback (e.g., name, body mass index, minutes of MVPA, graphs showing progress over time) but could not be pre-recorded into the videos was presented as text layered on top of the video (i.e., an “overlay”) to make the videos as tailored as possible.

Control group

The control group received access to a limited version of the TaylorActive website that only included the text-based library with generic PA information. The control group had no access to other website components and was regarded as “usual care” condition, as generic PA information is freely available on the Internet.

Procedures

Research officers verified participant eligibility and contact details following participants’ completion of the screening survey. Subsequently, participants completed a 30-min baseline survey administered through Computer-Assisted Telephone Interviewing (CATI), a telephone surveying technique in which the interviewer follows a script provided by a software application. The CATI surveys were conducted by trained and blinded interviewers from Central Queensland University's Population Research Laboratory, Rockhampton, Australia. The CATI surveys were repeated at 3- and 9-month assessments. Once baseline data were obtained, participants were randomized using a randomly generated sequence via www.randomization.com. There was no face-to-face contact with participants at any time during the randomized controlled trial.

Measures

Intervention process data were assessed using the post intervention (3 months) CATI survey measures and Google Analytics. This paper presents intervention process data on intervention usage, acceptability, usability, perceived usefulness, and satisfaction. Socio-demographic characteristics of participants measured in the baseline survey included sex, age, education (in years), relationship status (in a relationship or not in a relationship), urbanization (major city, regional city, or remote or very remote area), occupational category (professional, white collar, or blue collar), employment status (full-time; part-time/casual; or other: unemployed, home duties, student, retired), ethnicity (Caucasian or Indigenous/African/Asian/other), and household income per year (AUD130,000).

Usage

Usage of the TaylorActive website was measured using objective Google Analytics data. Google Analytics is a free tool commonly used in web and app-based interventions to gather usage statistics., Mean minutes spent on the TaylorActive website over the 12-week intervention period was measured using Google Analytics, as was the mean number of sessions and action plans participants completed over the 12-week intervention period. Furthermore, self-reported website usage was measured in the 3-month CATI survey using questions similar to those applied in previous computer-tailored PA interventions., Participants in all groups were asked “How often did you use the TaylorActive website?”; their answers were recorded using a 7-point Likert scale (1 = I didn't use it, 2 = less than once every two weeks, 3 = once every two weeks, 4 = once a week, 5 = 2−4 times a week, 6 = every day, 7 = more than once a day). Based on data distribution, the response options were grouped into ≥2−4 times per week, ≤1 time per week, and ≤once every 2 weeks. Participants in all groups were also asked “How would you prefer to receive personally relevant PA information via the Internet?” Response options included “Video to watch online” and “Text to read and print”. Intervention group participants were also asked: (1) “Overall, how much of the personalized feedback did you read/watch of the sessions that you completed?” Responses were recorded using an 8-point Likert scale (1 = all of it, 2 = most of it, 3 = more than half, 4 = about half, 5 = less than half, 6 = not much of it, 7 = nearly nothing, 8 = nothing), based on data distribution, the response options were grouped into all/most, about half, and less than half; (2) “Did you watch/read the PA information more than once?”; (3) “Did you request new PA information for the same session more than once by completing the online survey again?” Both questions (2 and 3) were rated on a 5-point Likert scale (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always). Based on data distribution, the response options were collapsed into often/always, sometimes, and never/rarely; (4) “Did you share/show your PA information with anybody else?” and (5) “Did you discuss the content of the PA information with someone else?” had response options of yes/no.

Acceptability

Acceptability of the TaylorActive website was measured using questions applied in previous computer-tailored PA interventions., Website acceptability was assessed in all groups using 14 statements rated on a 5-point Likert scale (from 1 (strongly disagree) to 5 (strongly agree)). For illustrative purposes the response options 4 (agree) and 5 (strongly agree) were grouped into a percentage of participants agreeing. Rated statements included, for example, “It was easy to answer the questions prior to receiving personalized feedback”, “The physical activity information was easy to understand”, “The physical activity information was personally relevant”, and “Too much physical activity information was provided per session”. In addition, acceptability questions specific to the intervention condition (video-tailored vs. text-tailored) were also asked. Statements relevant to the video-tailored condition included, for example, “I found the people presenting the information in the videos just right”, “I found the production (technical quality) of the videos poor”, and “I could easily understand everything that was said in the videos (sound quality)”. Statements relevant to the text-tailored condition included “I found the images and illustrations in the text matching well with the content” and “I would have preferred to receive the advice as a personalized video over text”.

Usability

Usability of the TaylorActive website was measured using the standardized Systems Usability Scale (SUS). Website usability was assessed in all groups using 10 statements rated on a 5-point Likert scale (from 0 (strongly disagree) to 4 (strongly agree)). For illustrative purposes the response options agree and strongly agree were collapsed into a percentage of participants agreeing. Statements included, for example, “I thought the TaylorActive website was easy to use”, “I would like to continue to use the TaylorActive website frequently in the future”, and “I found the TaylorActive website very cumbersome to use”. Furthermore, for comparability with similar studies, the overall SUS score was calculated using its established scoring protocol. The score contribution for each of the 10 items ranged from 0 to 4. For Items 1, 3, 5, 7, and 9 (the positively worded items), the score contribution was the scale position minus 1. For Items 2, 4, 6, 8, and 10 (the negatively worded items), the contribution was 5 minus the scale position. The sum of the scores was multiplied by 2.5 to obtain the overall SUS score, which ranged from 0 to 100. The SUS has shown high reliability (α = 0.91) when assessed through Cronbach's α and acceptable construct validity (r = 0.70) when measured through Spearman rank correlation.

Perceived usefulness

Perceived usefulness of the TaylorActive website was assessed using questions derived from a previous online computer-tailored PA intervention. Perceived usefulness was measured across the 3 groups using 5 questions rated on a 5-point Likert scale (from 1 (not at all useful) to 5 (very useful)). Based on data distribution, response options were grouped into very/quite useful, somewhat useful, and not/a little useful. Participants were asked “How useful was the website in helping you to (1) increase your confidence to engage in regular PA over the past 3 months? (2) overcome barriers to participating in PA over the past 3 months? (3) increase the support you receive for participating in PA over the past 3 months? (4) plan for PA over the past 3 months? and (5) stay motivated to participate in PA over the past 3 months?”

Satisfaction

Satisfaction with the TaylorActive website was assessed through 2 open-ended questions: “What did you like about using the website?” and “What did you not like about using the website?” Responses were completed in open text format.

Sample size

The randomized controlled trial sample size (n = 501) was calculated based on the primary study outcome (i.e., Actigraph measured MVPA (min/week), which is reported elsewhere)., Briefly, to achieve 80% power using an α level of 0.05 and accounting for an estimated 30% attrition, 167 participants per group (video-tailored, text-tailored, and control group) were recruited at baseline. At post intervention (3 months), 72 participants in the video-tailored, 83 in the text-tailored, and 104 in the control group completed the CATI survey. Only the post intervention sample (n = 259) was used in the process evaluation presented in this paper.

Data analyses

For the quantitative analyses, categorical data were presented using frequencies and percentages, and continuous data were presented using means and standard deviations. χ tests and one-way analysis of variance were used to assess differences in intervention usage, acceptability, usability, and perceived usefulness by group. All analyses were performed in IBM SPSS Statistics (Version 26.0; IBM Corp., Armonk, NY, USA) with the significance level set at 0.05. Thematic content analysis was used to identify emergent themes in the qualitative analyses of the open text responses to the questions on intervention satisfaction. A pen profile, which is an increasingly utilized technique to present analyzed text data in a diagram, was constructed based on themes identified in relation to “Likes” and “Dislikes” of the TaylorActive intervention. Example verbatim quotations were then extracted directly from the text responses to illustrate a theme. To provide an indication of the prevalence of the themes, the number of times a specific theme was mentioned was also presented.

Results

Baseline data

The socio-demographic characteristics of the full trial participants (n = 501) are reported elsewhere. Table 1 presents the socio-demographic characteristics of participants (n = 259) who completed the post intervention survey at 3 months and are included in the analyses presented in this paper. Participants’ mean age was 44.0 ± 13.1 years (mean ± SD). Most participants were females (70.7%), had 14 or more years of education (81.4%), lived in a major or regional city (92.3%), and were Caucasian (91.1%). Retention rate was 52%, with 259 participants completing the CATI survey at the 3-month assessment (post intervention). There were no significant differences in socio-demographic characteristics between participants retained (n = 259) and those lost to follow-up at post intervention (n = 242) (all p > 0.05).

Table 1

Socio-demographic characteristics of participants (n (%)).a

	All (n = 259)	Video-tailored (n = 72)	Text-tailored (n = 83)	Control (n = 104)
Gender
Male	76 (29.3)	17 (23.6)	28 (33.7)	31 (29.8)
Female	183 (70.7)	55 (76.4)	55 (66.3)	73 (70.2)
Age (year)
18‒44	131 (50.6)	35 (48.6)	48 (57.8)	48 (46.2)
45‒64	111 (42.9)	30 (41.7)	32 (38.6)	49 (47.1)
65 and over	17 (6.6)	7 (9.7)	3 (3.6)	7 (6.7)
Education (year)
≤13	48 (18.5)	13 (18.1)	15 (18.1)	20 (19.2)
14‒20	178 (68.7)	47 (65.3)	59 (71.1)	72 (69.2)
>21	33 (12.7)	12 (16.7)	9 (10.8)	12 (11.5)
Relationship status
Not in a relationship	76 (29.3)	18 (25.0)	23 (27.7)	35 (33.7)
In a relationship	183 (70.7)	54 (75.0)	60 (72.3)	69 (66.3)
Ethnicity
Caucasian	236 (91.1)	65 (90.3)	77 (92.8)	94 (90.4)
Indigenous/African/Asian/other	23 (8.9)	7 (9.7)	6 (7.2)	10 (9.6)
Urbanization
Major city	138 (53.3)	32 (44.4)	49 (59.0)	57 (54.8)
Regional city	101 (39.0)	29 (40.3)	28 (33.7)	44 (42.3)
Remote or very remote	20 (7.7)	11 (15.3)	6 (7.2)	3 (2.9)
Employment status
Full-time	133 (51.4)	37 (51.4)	46 (55.4)	50 (48.1)
Part-time/casual	87 (31.7)	21 (29.2)	28 (33.7)	33 (31.7)
Other	44 (17.0)	14 (19.4)	9 (10.8)	21 (20.2)
Combined household income per year (AUD)
>AUD65,000	46 (19.6)	18 (28.1)	11 (14.9)	17 (17.5)
AUD65,000‒130,000	99 (42.1)	28 (43.8)	29 (39.2)	42 (43.3)
>AUD130,000	90 (38.3)	18 (28.1)	34 (45.9)	38 (39.2)

Note: Percentages add up not to 100% due to rounding.

The post intervention (3 months) sample (n = 259) was used for these analyses.

Usage

Usage of the TaylorActive website in the video-tailored, text-tailored, and control group is presented in Table 2. Google Analytics data showed that participants’ overall time spent on the TaylorActive website during the 12-week intervention period was on average 90.1 ± 71.8 min in the video-tailored group and 77.0 ± 80.0 min in the text-tailored group, compared to 8.5 ± 16.1 min in the control group (p < 0.001). The number of times participants visited the TaylorActive website was on average 7.6 ± 7.2 in the video-tailored group and 7.3 ± 5.4 in the text-tailored group, compared to 1.4 ± 1.9 in the control group (p < 0.001). Out of the 8 intervention sessions, participants in the video-tailored group completed on average 4.8 ± 2.8 sessions, and participants in the text-tailored group completed 5.0 ± 2.8 sessions (p = 0.487). Out of 6 action plans to be completed on the website, participants in the video-tailored group completed on average 2.7 ± 2.1 action plans; similarly, those in the text-tailored group completed 2.7 ± 2.2 action plans (p = 0.940). The post intervention survey data showed that most participants used the TaylorActive website less than once every 2 weeks, with significant differences being observed between intervention and control groups (66.7% video-tailored, 62.7% text-tailored, 87.5% control; p < 0.001). Most intervention participants watched/read the tailored PA advice (77.5% video-tailored, 72.2% text-tailored; p = 0.516). However, the majority of them never or rarely watched/read the tailored PA advice more than once (69.6% video-tailored, 63.3% text-tailored; p = 0.699). Moreover, nearly all intervention participants (98.6% video-tailored, 97.5% text-tailored; p = 0.641) never or rarely requested updated tailored PA advice for the same session by completing the survey again. Few participants in either intervention group (21.7% video-tailored, 29.1% text-tailored; p = 0.306) shared their PA advice with someone else, and less than half of the intervention participants (43.5% video-tailored, 45.6% text-tailored; p = 0.798) discussed the content of the PA advice with someone else.

Table 2

Usage of the TaylorActive website in the video-tailored, text-tailored, and control group.

	Video-tailored	Text-tailored	Control	p
Website usage, measured via Google Analytics
Overall minutes spent on the website during the 12-week intervention	90.1 ± 71.8	77.0 ± 80.0	8.5 ± 16.1	<0.001
Number of website visits during the 12-week intervention	7.6 ± 7.2	7.3 ± 5.4	1.4 ± 1.9	<0.001
Number of intervention sessions completed (out of 8 intervention sessions)	4.8 ± 2.8	5.0 ± 2.8	NA	0.487
Number of action plans completed (out of 6 action plans)	2.7 ± 2.1	2.7 ± 2.2	NA	0.940
Website usage, self-reported				<0.001
≥2‒4 times per week	4 (5.6)	4 (4.8)	10 (9.6)
≤Once a week	20 (27.8)	27 (32.5)	3 (2.9)
≤Once every 2 weeks	48 (66.7)	52 (62.7)	91 (87.5)
Watched/read the personalized feedback				0.516
All/most	55 (77.5)	57 (72.2)	NA
About half	5 (7.0)	10 (12.7)	NA
Less than half	11 (15.5)	12 (15.2)	NA
Watched/read the physical activity advice more than once				0.699
Often/always	8 (11.6)	10 (12.7)	NA
Sometimes	13 (18.8)	19 (24.1)	NA
Never/rarely	48 (69.6)	50 (63.3)	NA
Requested updated tailored physical activity advice for the same session by completing the survey again				0.641
Often/always	0 (0.0)	0 (0.0)	NA
Sometimes	1 (1.4)	2 (2.5)	NA
Never/rarely	68 (98.6)	77 (97.5)	NA
Shared physical activity advice with someone else				0.306
Yes	15 (21.7)	23 (29.1)	NA
No	54 (78.3)	56 (70.9)	NA
Discussed content of the physical activity advice with someone else				0.798
Yes	30 (43.5)	36 (45.6)	NA
No	39 (56.5)	43 (54.4)	NA
Preferred delivery of personally tailored physical activity advice				<0.001
Video to watch online	50 (74.6)	19 (23.8)	29 (31.9)
Text to read and print	17 (25.4)	61 (76.3)	62 (68.1)

Notes: Data are presented as mean ± SD or n (%). Percentages add up not to 100% due to rounding. Bold values are significant.

Abbreviation: NA = not applicable.

Acceptability and usability

Acceptability and usability of the TaylorActive intervention are shown in Table 3. Regarding acceptability, most intervention participants found the PA information interesting (81.2% video-tailored, 74.4% text-tailored, 50.0% control; p < 0.001), credible (94.2% video-tailored, 93.9% text-tailored, 91.3% control; p = 0.805), easy to understand (95.7% video-tailored, 96.3% text-tailored; p = 0.771), and personally relevant (70.6% video-tailored, 67.1% text-tailored, 51.1% control; p = 0.080). More than half of intervention participants (61.8% video-tailored, 63.4% text-tailored) and significantly fewer control participants (14.3%; p < 0.001) said they used the PA information to become more active. However, few intervention participants (32.4% video-tailored, 22.0% text-tailored) and significantly fewer control participants (8.3%; p = 0.009) felt that the PA information was developed for them personally. Furthermore, few participants across all groups (33.8% video-tailored, 27.2% text-tailored, 29.7% control; p = 0.675) changed their opinion about PA because of this study. Regarding usability, most participants across all groups thought the TaylorActive website was easy to use (95.5% video-tailored, 93.6% text-tailored, 91.0% control; p = 0.567) and found the various website features well-integrated (87.5% video-tailored, 72.7% text-tailored, and 74.6% control; p = 0.080). Most participants across all groups thought that people could learn to use the TaylorActive website very quickly (95.5% video-tailored, 96.2% text-tailored, 87.8% control; p = 0.086), and nearly all participants felt very confident using the TaylorActive website (98.5% video-tailored, 92.3% text-tailored, 90.5% control; p = 0.136). However, fewer participants across all groups (52.9% video-tailored, 42.2% text-tailored, 38.5% control; p = 0.174) would like to continue to use the TaylorActive website frequently in the future. The mean SUS scores by group were 77.3 (video-tailored), 75.7 (text-tailored), and 74.1 (control), with no significant between-group differences (p = 0.274).

Table 3

Acceptability and usability of the TaylorActive intervention in the video-tailored, text-tailored, and control groups (n (%) agreeing)a.

	Video-tailored	Text-tailored	Control	p
Acceptability
It was easy to answer the questions prior to receiving personalized feedback.	51 (75.0)	66 (83.5)	NA	0.200
There were too many questions to answer before I got the advice.	18 (26.1)	26 (32.9)	NA	0.365
The physical activity information was interesting.	56 (81.2)	61 (74.4)	23 (50.0)	<0.001
The physical activity information was credible.	65 (94.2)	77 (93.9)	42 (91.3)	0.805
The physical activity information was easy to understand.	66 (95.7)	79 (96.3)	44 (93.6)	0.771
Through the physical activity information, I learned something new about my own physical activity.	44 (63.8)	50 (61.0)	20 (42.6)	0.055
The physical activity information was personally relevant.	48 (70.6)	55 (67.1)	24 (51.1)	0.080
Too much physical activity information was provided per session.	9 (13.0)	14 (17.1)	5 (11.1)	0.613
There were too many sessions.	11 (15.9)	20 (24.4)	5 (12.5)	0.214
The physical activity information met my expectations.	48 (70.6)	59 (72.8)	22 (47.8)	0.010
The physical activity information helped me to meet my goals.	35 (52.2)	45 (54.9)	10 (20.8)	<0.001
I have used the physical activity information to become more active.	42 (61.8)	52 (63.4)	7 (14.3)	<0.001
I have changed my opinion about physical activity because of this study.	23 (33.8)	22 (27.2)	19 (29.7)	0.675
I have the feeling that the physical activity information was developed for me personally.	22 (32.4)	18 (22.0)	4 (8.3)	0.009
Usability
I would like to continue to use the TaylorActive website frequently in the future.	37 (52.9)	35 (42.2)	37 (38.5)	0.174
I found the TaylorActive website unnecessarily complex.	7 (10.6)	5 (6.4)	6 (7.8)	0.650
I thought the TaylorActive website was easy to use.	63 (95.5)	73 (93.6)	71 (91.0)	0.567
I needed the support of a technical person to be able to use the TaylorActive website.	2 (3.0)	1 (1.3)	5 (6.3)	0.226
I found the various features on the TaylorActive website were well integrated.	56 (87.5)	56 (72.7)	53 (74.6)	0.080
I thought that there was too much inconsistency in the TaylorActive website.	4 (6.1)	4 (5.1)	2 (2.8)	0.635
I think that most people could learn to use the TaylorActive website very quickly.	63 (95.5)	75 (96.2)	65 (87.8)	0.086
I found the TaylorActive website very cumbersome to use.	9 (13.6)	17 (21.8)	6 (8.2)	0.060
I felt very confident using the TaylorActive website.	65 (98.5)	72 (92.3)	67 (90.5)	0.136
I needed to learn a lot of things before I could use the TaylorActive website comfortably.	1 (1.5)	0 (0)	3 (3.9)	0.182

Note: Bold values are significant.

For each statement, participants responded on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The response options “4 = agree” and “5 = strongly agree” were grouped into the number and the percentage of participants agreeing.Abbreviation: NA = not applicable.

Acceptability of the specific video-tailored and text-tailored components in the TaylorActive website is shown in Table 4. In the video-tailored group, all participants could easily understand what was said in the videos (i.e., sound quality). Very few participants found the production (technical quality) of the videos poor (5.8%) or the downloading/buffering of the videos slow (7.2%). Few participants (17.1%) would have preferred to receive the PA advice as personalized text over a video. In the text-tailored group, most participants (77.9%) found that the images and illustrations in the text matched well with the content. Some participants (23.8%) would have preferred to receive the advice as a personalized video over text. Across both the video- and text-tailored groups, most participants (73.0%) found the length of the tailored PA advice just right in most sessions. Further, the majority of participants (78.9%) found the graphs displayed in the video/text just right, and most participants (85.7%) thought the tone of the videos was just right (factual and convincing). However, some participants (23.8%) found the personalized text/video boring, and some (20.3%) would have preferred the video/text to be more light-hearted and humorous. A third of intervention participants (33.6%) would have liked to receive printable transcripts alongside the text/videos.

Table 4

Acceptability of the specific video-tailored and text-tailored components of the TaylorActive website.

	n (%) agreeingb
Video only, acceptabilitya
I found the people presenting the information in the videos just right.	65 (94.2)
I found the production (technical quality) of the videos poor.	4 (5.8)
I would have preferred to see someone famous present the videos.	1 (1.4)
I could easily understand everything that was said in the videos (sound quality).	69 (100.0)
The videos were downloading/buffering very slowly.	5 (7.2)
The pace at which new information was presented in the videos was too slow.	11 (16.4)
I would have preferred to receive the physical activity advice as personalized text over video.	12 (17.1)
Text only, acceptabilitya
I found that the images and illustrations in the text matched well with the content.	53 (77.9)
I would have preferred to receive the advice as a personalized video over text.	19 (23.8)
Video and text, acceptabilitya
I found the length of the physical activity advice just right in most sessions.	108 (73.0)
I found the personalized text/video boring.	35 (23.8)
I found the graphs displayed in the text/video just right.	112 (78.9)
The tone of the videos was just right (factual and convincing).	126 (85.7)
I would have preferred the text/video to be more light-hearted and humorous.	30 (20.3)
I would have liked to receive printable transcripts alongside the text/videos.	47 (33.6)

For each statement, participants responded on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree).

The response options “4 = agree” and “5 = strongly agree” were grouped into the number and the percentage of participants agreeing.

Perceived usefulness

Perceived usefulness of the TaylorActive intervention is presented in Table 5. In each group, only a small proportion of participants found the TaylorActive website very/quite useful for increasing their confidence to engage in regular PA (23.1% video-tailored, 22.4% text-tailored, 13.0% control; p < 0.001). More than half of intervention participants (52.3% video-tailored, 57.3% text-tailored) and significantly more control participants (81.1%; p = 0.003) thought the website was not/a little useful for overcoming barriers to engage in PA. Further, most participants across all groups (70.3% video-tailored, 66.7% text-tailored, 80.0% control; p = 0.085) found the TaylorActive website not/a little useful for increasing the support they receive for engaging in PA. Less than half of the intervention participants (43.1% video-tailored, 40.8% text-tailored) and significantly fewer control participants (12.2%; p < 0.001) found the website very/quite useful to plan their PA. Similarly, few intervention participants (33.8% video-tailored, 44.7% text-tailored) and significantly fewer control participants (15.8%; p < 0.001) thought the TaylorActive website was very/quite useful for helping to stay motivated to engage in PA.

Table 5

Perceived usefulness of the TaylorActive intervention in the video-tailored, text-tailored, and control groups.

Perceived usefulness of the website to…	Video-tailored	Text-tailored	Control	p
Increase confidence to engage in regular physical activity over the past 3 months				<0.001
Very/quite useful	15 (23.1)	17 (22.4)	10 (13.0)
Somewhat useful	22 (33.8)	24 (31.6)	10 (13.0)
Not/a little useful	28 (43.1)	35 (46.1)	57 (74.0)
Overcome barriers to engage in physical activity over the past 3 months				0.003
Very/quite useful	12 (18.5)	15 (20.0)	4 (5.4)
Somewhat useful	19 (29.2)	17 (22.7)	10 (13.5)
Not/a little useful	34 (52.3)	43 (57.3)	60 (81.1)
Increase support for engaging in physical activity over the past 3 months				0.085
Very/quite useful	9 (14.1)	12 (16.0)	2 (2.7)
Somewhat useful	10 (15.6)	13 (17.3)	13 (17.3)
Not/a little useful	45 (70.3)	50 (66.7)	60 (80.0)
Plan physical activity over the past 3 months				<0.001
Very/quite useful	28 (43.1)	31 (40.8)	9 (12.2)
Somewhat useful	15 (23.1)	21 (27.6)	13 (17.6)
Not/a little useful	22 (33.8)	24 (31.6)	52 (70.3)
Stay motivated to engage in physical activity over the past 3 months				<0.001
Very/quite useful	22 (33.8)	34 (44.7)	12 (15.8)
Somewhat useful	20 (30.8)	13 (17.1)	7 (9.2)
Not/a little useful	23 (35.4)	29 (38.2)	57 (75.0)

Notes: Data are presented as n (%). Percentages add up not to 100% due to rounding. Bold values are significant. For each statement, participants responded on a 5-point Likert scale ranging from 1 (not at all useful) to 5 (very useful). The response options were collapsed into “1 = Not/a little useful”, “2 = Somewhat useful”, and “3 = Very/quite useful”.

Satisfaction

Satisfaction with the TaylorActive website in the video-tailored and text-tailored group was reported qualitatively through text responses to open-ended questions. Satisfaction was expressed as “Likes” and “Dislikes”, which were then themed and illustrated in a pen profile (Fig. 1). However, given that “Likes” and “Dislikes” with the intervention were assessed through 2 optional open-ended questions, there were limited text responses available. Hence, the frequency with which themes recurred was low (n = 5−21 video-tailored, n = 1−24 text-tailored). Regarding “Likes”, participants mostly liked (n = 21 video-tailored, n = 24 text-tailored) that the TaylorActive was easy to use:

Fig.1

Satisfaction with the TaylorActive website in the video-tailored and text-tailored groups (“Likes” and “Dislikes”).

Easy to navigate and found it simple to use. (male, 33 years)

Participants liked the personalized feedback (n = 12 video-tailored, n = 9 text-tailored) in that it was tailored to the individual information participants had provided, tracked participants progress, and gave them advice. Participants also liked the information provided on the TaylorActive website (n = 5 video-tailored, n = 7 text-tailored), such as tips for becoming more active, and the information accessible through the library:

It was a really informative, fantastic library. It was friendly and approachable. (female, 40 years)

In the video-tailored group, participants (n = 5) liked that the TaylorActive website was motivational:

It reminded me to stay with the project—prompted me to keep going. (female, 67 years)

Some participants (n = 5) also liked the videos and goal setting:

Watching the videos after the questionnaire, and it suits what you've just talked about. (female, 30 years)

The fact that I did set goals and times to be active. (female, 58 years)

In the text group, participants (n = 6) liked the action plan:

It made me make a plan, think about how I would achieve my goals. (female, 42 years)

Regarding “Dislikes”, participants mostly disliked (n = 9 video-tailored, n = 18 text-tailored) the repetitive and tedious aspects of the TaylorActive website. For example, they did not want to fill out the action plan and restate their goals at every session but preferred instead to access their action plan from the previous week and modify it, if necessary:

I had to fill out the action plan every week. Just wanted to put in a plan for the whole time period and work with it. (male, 31 years)

Participants (n = 11 video-tailored, n = 3 text-tailored) also disliked that they had to answer questions every week, and they found the response options too specific, leaving them with too few choices:

I found the optional answers too specific. It forced to make a choice that was not quite right. (female, 60 years)

Technical limitations were also noted in both intervention groups (n = 3 video-tailored, n = 4 text-tailored), such as problems with opening the video, accessing the PA advice from the previous week, or not being able to save the PA advice in a file. Some participants (n = 3 video-tailored, n = 1 text-tailored) disliked the personal trainer:

Images were not realistic. Everyone was young, beautiful, and fit. (female, 56 years)

In the text-tailored group, some participants (n = 5) noted that there was too much to read, and some (n = 3) found the TaylorActive website too time consuming.

Discussion

This study examined the usage, acceptability, usability, perceived usefulness, and satisfaction with a web-based, video-tailored PA intervention for Australian adults (TaylorActive). Examination of process data at 3 months revealed that participants’ usage of the TaylorActive website (i.e., website visits, intervention sessions, and action plans completed) was modest in both the video-tailored and text-tailored groups. In both intervention groups, acceptability and usability of the TaylorActive intervention was mostly rated high. However, perceived usefulness of the TaylorActive website for increasing PA was rated low across the board. Satisfaction with the TaylorActive website was mixed in that participants in both intervention groups liked its ease of use, personalized feedback, and tracking of progress; however, they also found completing action plans and survey questions each week to be repetitive and tedious, and they thought the survey questions were too specific.

The post intervention survey showed that most intervention participants used the TaylorActive website only once or less in 2 weeks. Further, participants in both intervention groups completed just over half of the intervention sessions and less than half of the weekly action plans offered during the 12-week intervention. The TaylorActive website was designed to be used infrequently (i.e., only when a new intervention session was available, which was weekly for the first 4 weeks and then every 2 weeks for the next 2 months). This explains the modest usage findings to some extent. Furthermore, low or declining website usage is a common issue in web-based PA interventions.,43, 44, 45 Another explanation for the modest website usage in this study may be that providing personally tailored PA advice (either video or text) alone was insufficient for participants to more regularly engage with the website. Perhaps, usage would have been higher if the TaylorActive intervention included additional features, such as continuous self-monitoring using activity trackers, social interaction (e.g., online social networking features), and gamification (e.g., rewards), all of which are recognized motivators of PA engagement. Indeed, a subsequent study using the TaylorActive website showed that personally tailored PA advice combined with a Fitbit activity tracker for continuous self-monitoring positively impacted intervention usage, with twice as many participants in the Fitbit group completing all intervention sessions.

Compared to the control group, acceptability of the TaylorActive website was rated relatively high (i.e., interesting, easy to understand, credible) by most intervention participants. However, while most intervention participants thought that the personally tailored PA advice was indeed “personally relevant”, only a third of participants felt that it was “personally developed for them”. This finding is interesting but difficult to explain. Perhaps intervention participants thought that the personally tailored PA advice was generally correct. For example, when advised they are not meeting the PA recommendations, they would agree with that. Nonetheless, participants may have thought that the tailored PA advice did not achieve the appropriate level of personalization and hence may have perceived it as impersonal (i.e., because it did not incorporate contextual factors well enough or because they could tell the advice was produced by a computer). If so, interventions that can more dynamically interact with participants through dialogue (e.g., via an artificially intelligent virtual coach) and use ongoing real-time data to tell whether or not they are on track (instead of a single interaction every week or 2 weeks) may be more appealing. With technological advances in machine learning and artificial intelligence, the delivery of personally tailored PA interventions via an artificially intelligent virtual health coach (i.e., chatbot) has shown promise.

The majority of participants rated the usability of the TaylorActive website high (i.e., easy to use, features well integrated), with no significant differences between the 3 groups. The usability score was also good (i.e., >68, which is the threshold for good usability) and higher than what has been reported in other web-based PA interventions. However, only half of the participants in the video-tailored group and even fewer participants in the text-tailored group stated they would continue to use the TaylorActive website frequently in the future. This is likely due to how the website was organized. For example, for participants who completed all the sessions, there would be little reason to continue to use the website. Providing new or updated content is likely necessary for people to continue using the TaylorActive website.

Perceived usefulness of the TaylorActive website (e.g, for increasing the confidence to be active, planning PA, overcoming barriers, staying motivated) was mostly rated low in both intervention groups, though higher compared to the control group. One explanation may be that the TaylorActive website had very few features beyond the tailoring of PA advice (e.g., no social networking or gamification). Most electronic and mobile health interventions promoting PA incorporate a plethora of activity tracker-, web-, and app-based features as intervention components. However, this makes it impossible to identify which intervention components are effective. The TaylorActive study was designed with a clear objective to isolate the effectiveness of computer-tailoring (video or text) from any other intervention components that may mask its contribution. It appears to be the case that implementing too few intervention components may fail to sufficiently motivate people to become more active. Nevertheless, studies that evaluate intervention components in isolation are needed, as this knowledge will help make future interventions more effective overall.

Participants in both intervention groups found the completion of action plans and survey questions for each session to be repetitive and tedious. This finding is consistent with those from previous computer-tailored PA interventions showing that having to complete many questions prior to receiving personalized PA advice is discouraging for participants., This obstacle could be overcome by using automatically integrated activity tracker data instead of repeated online surveys to generate the personally tailored PA advice. This approach was successfully implemented in a subsequent intervention study using the TaylorActive website.

This process evaluation identified factors that may partially explain why the TaylorActive intervention was ineffective, and it provides ideas for improving the web-based video-tailored PA intervention. The inclusion of a comprehensive set of previously used measures and the use of both self-reports and objective data as part of a large-scale, rigorously conducted, randomized controlled trial are strengths of this process evaluation. However, a significant limitation of this study was the attrition of participants (48% at the 3-month survey), which corresponds to the modest usage and low perceived usefulness ratings revealed in the evaluation. Possible reasons for attrition were discussed in detail elsewhere. A brief interview or survey with participants who dropped out (i.e., an exit survey) might have added more in-depth information to the process evaluation. Another limitation was that the full trial sample (n = 501) was not entirely representative of the target population. For example, the TaylorActive trial targeted adults who did not meet the Australian PA guidelines (i.e., ≤150 min of MVPA per week). Despite screening participants for this during recruitment, more intervention participants (24% video-tailored, 22% text-tailored) than intended met the PA guidelines at baseline when measured using acceleromtery. Moreover, the full trial sample included a high proportion of females (72%) and highly educated (80%) participants.

Conclusion

The process evaluation of the TaylorActive intervention revealed that providing personally tailored PA advice on its own, either through video or text, is insufficient to promote usage, usefulness, and satisfaction with a web-based PA intervention. Strategies to address these issues may include the incorporation of additional components in web-based interventions. For example, using automatically integrated data from activity trackers (e.g., Fitbit, Garmin) instead of repeated online surveys to generate the personally tailored PA advice may improve participant satisfaction and retention. Furthermore, the use of different technologies, such as artificial intelligence and machine learning, may allow for a more personalized dialogue with participants thereby increasing the appeal of personally tailored interventions. The incorporation of social media may also stimulate intervention usage by providing participants with greater opportunities for social interaction. Finally, the availability of a user-friendly mobile version (i.e., smartphone app) in addition to an intervention website may increase the convenience and perceived usefulness of the intervention.