Process Evaluation of the Type 2 Diabetes Mellitus PULSE Program Randomized Controlled Trial: Recruitment, Engagement, and Overall Satisfaction.

BACKGROUND: Men are underrepresented in weight loss and type 2 diabetes mellitus (T2DM) prevention studies.
PURPOSE: To determine the effectiveness of recruitment, and acceptability of the T2DM Prevention Using LifeStyle Education (PULSE) Program-a gender-targeted, self-administered intervention for men.
METHODS: Men (18-65 years, high risk for T2DM) were randomized to intervention ( n = 53) or wait-list control groups ( n = 48). The 6-month PULSE Program intervention focused on weight loss, diet, and exercise for T2DM prevention. A process evaluation questionnaire was administered at 6 months to examine recruitment and selection processes, and acceptability of the intervention's delivery and content. Associations between self-monitoring and selected outcomes were assessed using Spearman's rank correlation.
RESULTS: A pragmatic recruitment and online screening process was effective in identifying men at high risk of T2DM (prediabetes prevalence 70%). Men reported the trial was appealing because it targeted weight loss, T2DM prevention, and getting fit, and because it was perceived as "doable" and tailored for men. The intervention was considered acceptable, with men reporting high overall satisfaction (83%) and engagement with the various components. Adherence to self-monitoring was poor, with only 13% meeting requisite criteria. However, significant associations were observed between weekly self-monitoring of weight and change in weight ( rs = -.47, p = .004) and waist circumference ( rs = -.38, p = .026). Men reported they would have preferred more intervention contact, for example, by phone or email.
CONCLUSIONS: Gender-targeted, self-administered lifestyle interventions are feasible, appealing, and satisfying for men. Future studies should explore the effects of additional non-face-to-face contact on motivation, accountability, self-monitoring adherence, and program efficacy.

RCT Entities:   Population Interventions Outcomes

Diabetes is a global health concern (Guariguata et al., 2014). In 2014, the prevalence of diabetes was estimated at 382 million people (8.3% of the world’s population) and expected to rise to 592 million people (10.1% of the world’s population) by 2035 (Guariguata et al., 2014). Several high-quality trials have demonstrated that lifestyle interventions are able to substantially lower type 2 diabetes mellitus (T2DM) incidence over the medium (3–6 years; Knowler et al., 2002; Tuomilehto et al., 2001) and long term (10 years; Knowler et al., 2009; Lindstrom et al., 2013). Notably, among these seminal trials and more generally in the wider T2DM prevention (Glechner et al., 2014) and weight loss (Pagoto et al., 2012) literature, men are a consistently underrepresented group. This is of great concern given that men appear to be at greater risk of developing T2DM (Chen et al., 2010) and display higher overall prevalence (International Diabetes Federation, 2013), particularly during middle age (The DECODE Study Group, 2003), when compared to women. Effective strategies to improve the recruitment and acceptability of lifestyle interventions for men are required. Previous research has indicated that designing weight loss interventions to specifically cater for the needs and preferences of men may improve recruitment, engagement, and satisfaction (Hunt et al., 2014; Morgan et al., 2009, 2011a, 2011b, 2013, 2014; Robertson et al., 2014). To the authors’ knowledge, there are no previous studies investigating gender-targeted (men only) and tailored lifestyle interventions for T2DM prevention.

In addition, concern exists about the cost of delivery, scalability, and sustainability of lifestyle interventions when translated into community settings (Dunkley et al., 2014; Schwarz et al., 2012). T2DM prevention programs have commonly involved intensive face-to-face approaches, which create substantial participant burden (time and travel), and require significant resourcing (staffing and facility use; Schwarz et al., 2012). Effective programs that employ less time and resource intensive approaches are needed (Dunkley et al., 2014; Schwarz et al., 2012). Self-administered (self-directed) online or print-based intervention delivery modes are suggested as alternatives to intensive face-to-face approaches, as they have great potential to overcome the aforementioned concerns (Dunkley et al., 2014; Schwarz et al., 2012). Importantly, research also suggests men may prefer lifestyle programs that are self-administered (Pagoto et al., 2012), as they are perceived to be more flexible and less time-consuming (Morgan et al., 2011b). To date, there is limited evidence regarding self-administered lifestyle interventions for T2DM prevention (Schwarz et al., 2012).

Given the paucity of information regarding gender-targeted and self-administered lifestyle interventions for T2DM prevention, process evaluation of trials exploring these approaches could provide valuable insights to inform future research and translation. Process evaluation may address aspects of trial design (e.g., recruitment and retention) and intervention design (e.g., acceptability, engagement, adherence, and satisfaction; Moore et al., 2015; Morgan et al., 2014). When paired with outcome data, process information may provide a more comprehensive understanding of complex lifestyle intervention trials (Moore et al., 2015). We recently conducted a 6-month randomized controlled trial (RCT) to examine the efficacy of the T2DM Prevention Using LifeStyle Education (PULSE) Program (Aguiar et al., 2014b, 2016), a gender-targeted, self-administered, multicomponent (weight loss, diet modification, physical activity) lifestyle intervention for men at high risk of developing T2DM. Detailed efficacy data are reported elsewhere (Aguiar et al., 2016) and summarized in Table 1. At 6 months, significant intervention effects were observed for weight (primary outcome; −5.50 kg, 95% CI [−7.40, −3.61], p < .001), glycated hemoglobin (HbA1c; −0.2%, 95% CI [−0.3, −0.1], −2.2 mmol/mol, 95% CI [−3.3, −1.1], p = .002), and several secondary outcomes. The aims of this process evaluation were to examine (a) the effectiveness of the recruitment and selection processes, reasons why men chose to enroll in the trial, and the characteristics of men who enrolled; and (b) the acceptability of the intervention program’s delivery and content in regards to the perceptions, engagement, adherence, retention, and satisfaction of men completing the intervention.

Table 1.

Summary of PULSE Program Intervention Components and Qualitative and Quantitative Outcomes.

Intervention component	Content delivery or self-monitoring	Qualitative evaluation	Objective measure (Aguiar et al., 2016)
15-min program orientation (face-to-face)	Content	NA	NA
The SHED-IT Weight Loss Program
The SHED-IT Weight Loss DVD for Men	Content	Questionnaire	Weight**, waist**
The SHED-IT Weight Loss Handbook for Men	Content	Questionnaire	Weight**, waist**
The SHED-IT Weight Loss Logbook for Men	Self-monitoring	Questionnaire, adherence data	Weight**, waist**
Tape measure	Self-monitoring	Not evaluated	Waist**
CaloriekingTM self-monitoring tool user guide	Content/self-monitoring	Questionnaire	NA
CaloriekingTM “Calorie Fat and Carbohydrate Counter” booklet	Content/self-monitoring	Not evaluated	NA
The PULSE Program
The PULSE T2DM Prevention Handbook for Men	Content	Questionnaire	Weight**, HbA1c**, insulin**
The PULSE Exercise Support Book for Men	Content	Questionnaire	HbA1c*, aerobic fitness*, muscular fitness**
The PULSE Exercise Logbook	Self-monitoring	Adherence data	NA
GymstickTM	NA (exercise equipment)	Questionnaire	Muscular fitness**
Pedometer	Self-monitoring	Questionnaire	Physical activity

Note. Significant between-group differences in changes from baseline to 6 months. NA = not applicable.

p < .05. **p < .01.

Methods

Trial Description

The T2DM PULSE Program trial was a 6-month assessor-blinded, parallel-group RCT (Aguiar et al., 2014b). The trial was conducted at The University of Newcastle, Australia, and was approved by the institution’s Human Research Ethics Committee (H-2012-0232). The trial was registered with the Australian New Zealand Clinical Trials Registry (ANZCTR): ACTRN12612000721808. The design, procedures, and reporting of the trial (Aguiar et al., 2014b, 2016) adhered to the CONSORT statement for parallel group RCTs (Moher et al., 2010; see Additional File 1—CONSORT checklist).

Process Evaluation Design

A process evaluation plan was developed and described a priori in the trial protocol (Aguiar et al., 2014b). Quantitative data regarding recruitment and selection processes were obtained from an eligibility screening questionnaire, participant flow data, and baseline assessments. A process evaluation questionnaire was administered at 6 months (immediate post-program) to gain further information about the trial. This questionnaire used 5-point Likert scales (strongly disagree to strongly agree), with closed and open-ended questions to obtain quantitative and qualitative information. Intervention group participants were asked additional questions specifically addressing the PULSE Program intervention. All statistical analyses described were performed using IBM SPSS version 21. Data are generally presented as mean ± standard deviation (SD) and/or counts and percentages where appropriate. Representative quotes from men are provided to highlight their perceptions regarding specific aspects of the trial. As recommended by the Medical Research Council guidance on process evaluation of complex interventions (Moore et al., 2015), the authors acknowledge that the statistical analyses of process data were conducted after the trial’s efficacy outcomes were determined.

Evaluation of Trial Recruitment and Selection Processes

The target population was men aged 18–65 years, BMI 25–40 kg/m2, and at high risk for T2DM (Australian diabetes risk assessment tool [AUSDRISK]; score ≥ 12; Chen et al., 2010). Men with prediabetes were eligible, whereas men with type 1 diabetes or T2DM were not eligible. Potential participants completed an online eligibility-screening questionnaire, which included the AUSDRISK (Chen et al., 2010) screening tool and a pre-exercise screening questionnaire. This was a pragmatic choice to reduce the time and cost burden of using diagnostic blood tests to screen for prediabetes prior to study entry. Full details of the trial’s eligibility criteria and screening can be viewed elsewhere (Aguiar et al., 2014b). Participants were required to provide written informed consent. The recruitment target was 94 men (37 per group for 80% power to detect a 4-kg difference in mean weight change at 6 months and allowing for 20% loss to follow-up). The retention target was 80% at 6 months, a commonly used target for assessing methodological quality of lifestyle intervention RCTs (Aguiar et al., 2014a; Young et al., 2012).

Men from the Hunter region, New South Wales, Australia, were recruited from August 2012 to March 2013. Recruitment methods consisted of newspaper ads, radio interviews, print flyers, targeted workplace emails, and social media. To improve intervention appeal, recruitment materials and messages were gender targeted and tailored (Morgan et al., 2013; Resnicow et al., 1999; Young et al., 2014) to focus on men’s values. For example, the pitch of advertisements was “Are you a bloke [Australian colloquial term for man] who wants to get fit and lose a few kilos?” and featured an image of an overweight male. Recruitment materials and media interviews also emphasized that the trial was investigating T2DM prevention; involved a self-administered program with minimal face-to-face contact (Morgan et al., 2011b; Pagoto et al., 2012); was focused on fitness, weight loss, and improved health (Hankey et al., 2002); and was university based (i.e., a credible source of information; Smith et al., 2008).

To determine the effectiveness of this gender-targeted and pragmatic recruitment and online selection process, we examined which of the recruitment strategies were most widely seen/heard, reasons for choosing to enroll (e.g., appeal), reasons for ineligibility, time taken to achieve the recruitment target, and characteristics of men enrolled. Chi-square tests were used to determine any differences (significance level, p < .05) in program appeal factors between younger and older men (aged <50 and ≥50 years), overweight and/or obese men (BMI <30 and ≥30 kg/m2), men with HbA1c values in the normal or prediabetes range (<5.7% and ≥5.7% [39 mmol/mol]), and men below or above the median socioeconomic status score (SES, <6 and ≥6).

Evaluation of the Acceptability of the Intervention Program

The PULSE Program intervention is described in detail elsewhere (Aguiar et al., 2014b, 2016). Briefly, the program was a gender-targeted, self-administered (print, video, and online), multicomponent (weight loss, diet, exercise) T2DM prevention lifestyle intervention (Table 1). Men received the Self-Help, Exercise and Diet using Internet Technology (SHED-IT) Weight Loss Program, a gender-targeted, self-administered weight loss intervention for men, which has been rigorously evaluated by our group (Blomfield et al., 2014; Collins et al., 2011; Lubans et al., 2009; Morgan et al., 2009, 2010, 2011a, 2011b, 2012, 2013, 2014; Young et al., 2014, 2015). The additional novel intervention components and program messages (outlined in Tables 1, 3, and 5) were designed to align the overall intervention with current diet and physical activity guidelines for the prevention of T2DM (American Diabetes Association, 2007; Hordern et al., 2012; International Diabetes Federation, 2011). To facilitate and optimize lifestyle behavior change, the intervention program was gender tailored for men (Morgan et al., 2011b, 2013; Resnicow et al., 1999; Young et al., 2014) and grounded in Bandura’s social cognitive theory (Bandura, 1986, 2004). The gender tailoring was designed to cater to the preferences and needs of men, with the aim of improving the receptivity and acceptability of the program. The operationalization of gender tailoring and social cognitive theory within the intervention is reported elsewhere (Aguiar et al., 2014b, 2016). For brevity, we will henceforth refer to the intervention collectively as the PULSE Program.

Table 3.

Men’s (n = 40) Perceptions of the PULSE Program.

	Item score (M ± SD)
The PULSE T2DM Prevention Handbook for Men
• The PULSE T2DM Prevention Handbook for Men increased my understanding of T2DM	4.1 ± 0.6
• The T2DM Prevention Handbook for Men was enjoyable to read	3.9 ± 0.6
The PULSE Exercise Support Book for Men
• The PULSE Exercise Support Book for Men was useful	4.0 ± 0.5
• I found the PULSE pedometer useful	3.7 ± 1.0
• I found the body weight exercises useful	3.5 ± 0.8
• I found the GymstickTM useful	3.4 ± 0.9
• I feel that my endurance fitness has improved since commencing the PULSE Program (i.e., in the last 6 months)	3.6 ± 0.9
• I feel that I have gotten stronger since commencing the PULSE Program (i.e., in the last 6 months)	3.5 ± 0.8
The SHED-IT Weight Loss Program
• The SHEDIT Weight Loss Program provided me with the information I needed to help me lose weight	4.1 ± 0.6
• The SHEDIT Weight Loss Program improved my understanding of physical activity, nutrition, and weight loss	4.2 ± 0.6
• The SHED-IT Weight Loss Handbook for Men was enjoyable to read	3.9 ± 0.6
• The SHED-IT Weight Loss DVD for Men was enjoyable to watch	4.1 ± 0.5
• The CaloriekingTM website was easy to use	3.6 ± 1.1
• Using the CaloriekingTM website to record my food and exercise was time-consuming	3.7 ± 0.8
• The CaloriekingTM website was a valuable tool to help me understand how to lose weight	3.8 ± 0.7
• The CaloriekingTM user guide was useful	3.8 ± 0.6
Effects of the PULSE Program on self and on others
• I believe that my participation in the PULSE Program has decreased my risk of T2DM	4.0 ± 0.9
• As a result of my participation in the PULSE Program other members of my family have started to make healthier food choices	3.4 ± 0.9
• As a result of my participation in the PULSE Program other members of my family have become more active	3.3 ± 0.8
• As a result of my participation in the PULSE Program other members of my family have lost weight	3.0 ± 0.8
• As a result of my participation in the PULSE Program one or more of my friends have lost weight	2.8 ± 0.8
• I have had conversations with friends, coworkers, and/or relatives about the PULSE Program and the strategies I have learned to reduce my risk for T2DM	3.8 ± 0.7
• I would recommend the PULSE Program to my friends	4.4 ± 0.5
Additional feedback
• Being part of a research study has helped me to stick to the program (i.e., healthy eating, exercise, and weight loss)	3.8 ± 1.0
• Knowing that I was accountable (i.e., being followed up by the PULSE team) motivated me to stick to the program (i.e., healthy eating, exercise, and weight loss)	3.6 ± 1.1
• There was too much reading to do for the PULSE Program	2.6 ± 0.8
• I would prefer a program that had more regular face-to-face contact than the PULSE Program	3.5 ± 1.0

Note. Data are reported as mean ± SD of participant responses (n = 40) to the above statements.

1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree.

Table 5.

Associations Between Self-Monitoring Compliance and Changes in Outcomes (n = 30).

	No. SHED-IT Program weekly weight records[a]	No. PULSE Program daily exercise records[a]
Weight (kg)	−0.47**	−0.29
Waist circumference narrowest (cm)	−0.18	−0.32
Waist circumference umbilicus (cm)	−0.38*	−0.37*
Fat mass (kg)	0.46**	−0.30
HbA1c (%)	−0.05	−0.19
FPG (mmol/L)	0.18	0.07
Insulin (mIU/L)	−0.22	0.06
Aerobic fitness (mL/kg/min)	0.3	0.25
Lower body muscular fitness (repetitions)	0.11	−0.0
Upper body muscular fitness (repetitions)	−0.12	−0.07
Physical activity (pedometer steps/day)	−0.02	0.41*
Energy intake (kJ/day)	0.15	0.21

Note. aNumber of SHED-IT Program weekly weight records and PULSE Program daily exercise records from men (n = 30) over the 6-month intervention. FPG = fasting plasma glucose; HbA1c = glycated hemoglobin.

p < .05, **p < .01.

The PULSE Program was presented to intervention participants as a resource pack after their baseline appointment. Each participant received a standardized 15 min orientation to the program resources. No further contact was made for intervention delivery or self-monitoring prompting over the following 6 months. Participants were asked to return program logbooks (self-monitoring components; SHED-IT Weight Loss Logbook for Men and PULSE Exercise Support Book for Men) after 6 months (immediate post-program) for assessment of self-monitoring adherence.

To evaluate the acceptability of the intervention program’s delivery and content, participants were asked a series of questions to evaluate perceptions, engagement and program satisfaction. Men were asked to comment on how they implemented the weight loss, dietary, and exercise messages, strengths and weaknesses of the program, and to provide suggestions for improvement of the PULSE Program. Questions were focused on the novel components of the PULSE Program resources, as the SHED-IT Weight Loss Program has been evaluated previously (Morgan et al., 2011a, 2011b, 2014). Participant retention at 6 months was examined as an indication of program appeal and satisfaction.

Adherence to the intervention was assessed using logbook entries for self-monitoring of weekly weight in the SHED-IT Weight Loss Logbook for Men and weekly exercise in the PULSE Exercise Support Book for Men. Logbook compliance was set a priori (Aguiar et al., 2014b) at 50% (12 weeks) of the 6-month (24 weeks) intervention for (a) recording of weight (n ≥ 12 weekly weigh-in entries), and (b) recording and achieving the physical activity goal (n ≥ 12 weeks achieving 210 min per week). The associations between change in selected outcomes and number of weekly entries for weight and daily entries for exercise over 6 months were assessed using Spearman rank correlations (r; significance level, p < .05).

Results and Discussion

Process Evaluation Questionnaire Data

One hundred and one men were randomized into the trial (Figure 1), with 53 allocated to the intervention group and 48 to the control group. A total of 81 men completed the process evaluation questionnaire at 6 months, comprising 40 men from the intervention group (100% of completers) and 41 men from the control group (97% of completers).

Figure 1.

CONSORT diagram describing flow of participants through the trial including reasons for exclusion and loss to follow-up.

Effectiveness of Participant Recruitment and Selection Processes

Effectiveness of recruitment strategies

More than 250 men expressed interest in the trial. Of these, 218 completed the online eligibility-screening questionnaire. These men were asked to indicate how many of the recruitment strategies they were aware of prior to registering their interest. The most effective recruitment strategies were newspaper advertisements (29% of men), word of mouth (family and friends; 26%), and radio interviews (24%). Recruitment flyers (16%), targeted workplace emails (12%), and social media (4%) were less effective.

Reasons why men chose to enroll (program appeal)

To gain insight into why men enrolled for the trial, participants (n = 81 at 6-month assessments) were asked to indicate how many aspects of the recruitment messages and trial information influenced their decision. Men reported (multiple responses allowed) that the trial was appealing because the messages mentioned: weight loss (76% of men), T2DM prevention (65%), and getting fit (48%). This is consistent with Hankey and colleagues (2002), who reported men seek weight loss primarily because of its association with improvements in health and fitness. Men were also interested because the trial was perceived as “doable” (35%), was specifically tailored for men (31%), and was university based (27%). This information supports previous studies that suggest men prefer programs that are self-administered (Morgan et al., 2011b; Pagoto et al., 2012) and gender targeted (Morgan et al., 2011b), and value credible sources of information (Smith et al., 2008). Interestingly, there were no significant differences (all p > .05) in program appeal responses between younger and older men (aged <50 and ≥50 years), overweight and/or obese men (BMI <30 and ≥30 kg/m2), men with HbA1c values in the normal or prediabetes range (<5.7% and ≥5.7% [39 mmol/mol]), and men below or above the median SES (<6 and ≥6), which indicates the study had broad appeal across these various subgroups of men. Additional support for the appeal of the trial is reflected by the very high retention (88%, Figure 1) in the wait-list control group who were provided with the PULSE Program on completion of the 6-month control period.

Effectiveness of obtaining the target population

The required study sample size was achieved, with an appropriate sample of men (N = 101) at high risk for developing T2DM commencing the study (Figure 1). Recruitment was considerably slower than anticipated, with men recruited in four separate cohorts over a 7-month period to achieve the target sample size. Slow recruitment was partly due to the large proportion of men who were screened as ineligible for the trial. Of the 218 men who completed online screening, 45% were ineligible for the trial (Figure 1). Ineligibility was most commonly due to an AUSDRISK score <12 points (n = 40), already diagnosed with T2DM (n = 10), and BMI >40 kg/m2 (n = 10). Importantly, this suggests that a gender-targeted and pragmatic recruitment and online selection process is also effective in identifying men at low to moderate risk of T2DM and that men already diagnosed with T2DM are interested in participating in a lifestyle intervention.

The characteristics of men enrolled in the study (N = 101) are reported here briefly, and examined more comprehensively in a previous report (Aguiar et al., 2015; Aguiar et al., 2016). Mean (± SD) AUSDRISK score was 18 ± 5 points. Most men were obese based on BMI, waist circumference, and body fat mass (Table 2). In total, 70% of men were above the prediabetes cut points (American Diabetes Association, 2014) for fasting plasma glucose (FPG) or HbA1c. These anthropometric and metabolic characteristics indicate substantial risk for T2DM, and therefore support our use of the AUSDRISK screening tool for identifying men at risk for developing T2DM.

Table 2.

Baseline Characteristics of Men (N = 101) Enrolled in the PULSE Program.

Characteristics	Control(n = 48)		Intervention(n = 53)		Total(N = 101)
	M	SD	M	SD	M	SD
Age (years)	52.2	10.1	52.5	9.5	52.3	9.7
AUSDRISK score	18	5	18	5	18	5
Weight (kg)	103.3	12.7	102.8	13.6	103.0	13.1
Height (cm)	177.8	6.9	178.5	6.6	178.2	6.7
BMI (kg/m2)	32.6	3.3	32.2	3.5	32.4	3.4
Waist narrowest (cm)	110.1	9.7	109.9	9.6	110.0	9.6
Waist umbilicus (cm)	112.9	8.5	112.0	9.0	112.4	8.7
Fat mass (kg)	34.6	8.4	31.9	8.4	33.2	8.5
Fat mass (%)	33.0	5.7	30.6	5.6	31.8	5.8
FPG (mmol/L)	5.1	0.6	5.0	0.7	5.0	0.6
HbA1c (%)	5.8	0.4	5.8	0.5	5.8	0.5
HbA1c (mmol/mol)	40	4.4	40	5.5	40	5.5
Insulin (mIU/L)	8.6	4.0	10.4	7.6	9.5	6.1
Aerobic fitness (mL/kg/min)	38.0	6.7	37.5	6.7	37.7	6.7
Lower body muscular fitness (repetitions)	43	24	44	23	43	23
Upper body muscular fitness (repetitions)	13	6	13	5	13	5
Physical activity (pedometer steps/day)	6368	2643	6699	2613	6544	2618
Total energy intake (kJ/day)	2809	848	2631	751	2715	799

Note. AUSDRISK = Australian diabetes risk assessment tool; BMI = body mass index; FPG = fasting plasma glucose; HbA1c = glycated hemoglobin.

It is acknowledged the study sample may not be entirely representative of men who could benefit from lifestyle intervention, as those who enquired and subsequently enrolled may be more likely to have a basic awareness of their health status and be motivated to improve their health than men who did not respond, that is, a self-selection bias. This may limit the generalizability of the findings of the trial to men who have a basic level of health literacy and who are somewhat self-motivated (Plotnikoff et al., 2013). Also, the requirement to complete online eligibility screening and subsequent online intervention components might have been a barrier for men with no/limited access or computer literacy. Overall, the gender-targeted and pragmatic recruitment and online selection procedures were feasible and effective in identifying the target population of men at high risk for T2DM.

Evaluation of Intervention Acceptability

Overall satisfaction

Men were highly satisfied with the intervention program, with 83% agreeing/strongly agreeing that participation had decreased their risk of T2DM (see Table 3 for mean ± SD).

The program was truly life-changing . . . My overall fitness has improved immeasurably . . . My overall well-being and self-esteem has improved greatly!

(Age 62, Weight loss −6.2%, HbA1c change −0.7%, −7.7 mmol/mol)

The program information reinforced the need to develop a 7 day routine of daily walking and body mass, resistance training and healthier eating. I have continued to lose weight, feel stronger and healthier and enjoy the routine.

(Age 63, Weight loss −4.1%, HbA1c change 0.1%, −1.1 mmol/mol)

One of the strongest indicators of satisfaction was the large number of men (98% agree/strongly agree) indicating they would recommend the program to friends (Table 3). Also, the high level of retention (81% at 6 months; Figure 1) is an indication of satisfaction. Notably, there were no major financial incentives to complete the study or to achieve program goals. Participants received a AUS$10 reimbursement following each assessment time point (maximum of AUS$30) to contribute to the cost of travel and parking.

Engagement with the intervention resources

Men reported a high level of engagement with the gender-targeted and self-administered lifestyle intervention (Table 3), supporting the feasibility and acceptability of these approaches.

The information packs were well presented and made really easy to understand.

(Age 38, Weight loss −0.2%, HbA1c change −0.1%, −1.1 mmol/mol)

Men read the PULSE T2DM Prevention Handbook for Men once (48%), twice (41%), or three or more times (12%), with 90% of men agreeing/strongly agreeing that the document increased their understanding of T2DM (Table 3). Men reported that the document length (14 pages including text, tables, and images) was “about right” (95%), with very few men thinking the document was “too long” (5%). Further, 79% of men agreed/strongly agreed that the document was enjoyable to read. Men read the SHED-IT Weight Loss Handbook for Men once (45%), twice (50%), or three or more time (5%), with 81% of men agreeing/strongly agreeing that the document was enjoyable to read, and 86% of men agreeing/strongly agreeing that the program provided them with the information required to assist them to lose weight. Men watched the SHED-IT Weight Loss DVD for Men once (50%), twice (38%), or three or more times (12%), with 91% of men agreeing/strongly agreeing that the DVD was enjoyable to watch. These findings are consistent with previously published evaluations of the SHED-IT Weight Loss Program (Morgan et al., 2011a; Morgan et al., 2011b; Morgan et al., 2014). While difficult to assess, it is likely that the gender tailoring of the intervention improved the acceptability of and engagement with the intervention resources.

It was directed at blokes [men] and has an easy to follow and laid back approach.

(Age 59, Weight loss −7.0%, HbA1c change −0.2%, −2.2 mmol/mol)

Overall, men were satisfied with the information and exercise recommendations provided in the PULSE Exercise Support Book for Men (86% agree/strongly agree) and indicated they felt fitter (60% agree/strongly agree) and stronger (60% agree/strongly agree) on completion of the program (Table 3). These perceptions support our previously reported improvements in objectively assessed aerobic and muscular fitness (Table 1; Aguiar et al., 2016). Men highly valued the provision of a pedometer to track physical activity (74% agree/strongly agree), but responses were mixed regarding the recommended home-based bodyweight (38% neutral, 55% agree/strongly agree), and GymstickTM (43% neutral, 45% agree/strongly agree) exercise program. This finding may be explained in part by a lack of familiarity, knowledge, or sufficient instruction on resistance training technique and equipment use (Eves & Plotnikoff, 2006); low perceived self-efficacy with resistance training (Hills et al., 2010); or limited mobility/ability to perform resistance training based on previous injury or comorbidities (Hills et al., 2010). Given the potential benefits of home-based resistance training (e.g., cost, time, and removal of barriers in a gym setting) and the paucity of evidence for home-based resistance training (Eves & Plotnikoff, 2006), future research should explore strategies to improve resistance training skill competency, self-efficacy, and satisfaction in the home setting. For example, the inclusion of resistance training demonstration videos, as implemented in a recent trial of the self-administered SHED-IT Weight Loss Maintenance Program for men (Young et al., 2014), or an orientation session where participants trial the exercise equipment under supervision could be provided.

Implementation of program messages

The intervention program’s dietary and physical activity messages were generally well adopted, indicating good engagement with the intervention (see Table 4 for mean ± SD). Men indicated that they tried more often to: “create a colorful plate” (73% agree/strongly agree), eat more low glycemic index foods (84% agree/strongly agree), include more “healthy” monounsaturated and polyunsaturated fats (88% agree/strongly agree), and limit “unhealthy” saturated fats (77% agree/strongly agree). While a pre–post comparison of these behaviors was not possible in the current study, these findings support the previously published efficacy results arising from this trial. In addition, we asked men which of the nine SHED-IT Weight Loss Program tips they used (Morgan et al., 2013). The most commonly implemented tips were “Read food labels” (69%), “Reduce your portion size” (62%), and “Every step counts” (60%). This aligns with a recent mediation analysis of the SHED-IT Weight Loss Program, which identified that approximately 31% of the intervention effect on weight could be attributed to the intervention’s effects on the men’s physical activity behavior and average portion sizes (Young et al., 2015).

Table 4.

Engagement With Diet and Physical Activity Messages (n = 40).

Item	Item score (M ± SD)
(a) I now try to “create a colorful plate” by including a greater variety of foods (e.g., vegetables and fruit) in a meal	3.9 ± 0.8
(b) I now try to eat more low GI foods, e.g., fruits and vegetables, wholegrain breads and cereals, low fat milk, low fat yoghurt, and nuts	4.0 ± 0.8
(c) I now try to include more healthy (monounsaturated and polyunsaturated) fats in my diet, e.g., nuts, oily fish, healthy oils	4.1 ± 0.7
(d) I know try to avoid or limit unhealthy (saturated) fats in my diet, e.g., butter, cream, full cream milk, full fat cheese, fatty cuts of meat	3.9 ± 1.0
(e) I now try to include lean cuts of meat and other sources of protein in my diet	4.0 ± 0.8
(f) I now try to include more fiber in my diet, e.g., wholegrain bread and cereals, lentils, beans, fruits, and vegetables	4.1 ± 0.8
(g) I now try to “move after meals”	3.5 ± 0.9
(h) I usually do 30 min (or more) of aerobic exercise (walk, jog, swim, cycle)	3.2 ± 1.2
(i) I usually do 30 min (or more) of resistance training (body weight, GymstickTM, free or machine weights)	1.8 ± 0.8
(j) I now eat five serves of vegetables	3.2 ± 1.0
(k) I now eat two serves of fruit	3.2 ± 1.1
(l) I now skip meals	1.6 ± 0.9
(m) I now eat breakfast	4.6 ± 0.7
(n) I now pack my lunch or choose healthy options if I have to buy lunch	3.4 ± 1.3
(o) I now avoid eating meals or snacking late at night	3.2 ± 1.4

Note. Data are reported as mean ± SD of participant responses (n = 40) to the above statements.

Questions (a)–(j) 1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree.

Questions (h)–(o) 1 = zero days per week; 2 = one–two days per week; 3 = three–four days per week; 4 = five–six days per week; 5 = seven days per week.

GI = glycemic index.

Engagement and adherence with program self-monitoring tools

Self-monitoring is considered a key component of successful behavior change (Bandura, 2004) as it increases an individual’s awareness of their behaviors (Artinian et al., 2010), provides direct feedback with respect to achievement of lifestyle behavior goals (Artinian et al., 2010), and is associated with improvements in health outcomes (Burke et al., 2011). In the current study, self-monitoring included paper-based (PULSE Exercise Support Book for Men and SHED-IT Weight Loss Logbook for Men) and online (CaloriekingTM website) components.

Adherence to paper-based self-monitoring and associations with study outcomes

Overall, paper-based self-monitoring adherence was poor, a phenomenon previously reported (Stone, Shiffman, Schwartz, Broderick, & Hufford, 2002). Of the 40 men in the intervention group assessed at 6 months, 30 (75% of men) returned their SHED-IT Weight Loss Logbook for Men and PULSE Exercise Support Book for Men. Men achieved better adherence (criteria met for 50% of weeks) in recording their weight in the SHED-IT Weight Loss Logbook at 3 months (68%, n = 27) and 6 months (58%, n = 18) than in recording and achieving the weekly exercise target (210 min/week) in the PULSE Exercise Support Book at 3 months (16%, n = 6) and 6 months (13%, n = 4).

Despite poor paper-based self-monitoring reporting, moderate inverse correlations were observed for self-monitoring and changes in outcomes. Table 5 reports associations between self-monitoring adherence (weekly weight records and daily exercise records over 6 months) and change in anthropometric, glycemic, fitness, and dietary outcomes. Self-monitoring of weekly weight (SHED-IT Weight Loss Logbook) was significantly associated with changes in weight (r = −.47, p = .004), waist circumference (r = −.38, p = .026), and fat mass (r = −.46, p = .006). These associations were consistent with those reported in a previous analysis of the SHED-IT Weight Loss Program (Morgan et al., 2014). Self-monitoring of daily exercise in the PULSE Exercise Support Book was significantly associated with changes in waist circumference (r = −.37, p = .038), and physical activity (r = .41, p = .032). No associations were observed for changes in glycemic markers. Importantly, poor adherence with self-monitoring entries may not necessarily indicate the absence of recommended behaviors, as in the current study self-monitoring reporting of exercise was poor, yet significant intervention effects for aerobic and muscular fitness were observed (Aguiar et al., 2016). Further, although we cannot make conclusions about the self-monitoring adherence of men who did not return their logbooks (n = 10), they had a larger reduction (mean ± SD) in weight (−6.53 ± 8.0 kg vs. −4.61 ± 5.12 kg) and similar reductions in HbA1c (−0.3 ± 0.3% vs. −0.4 ± 0.3%, −3.3 ± 3.3 vs. −4.4 ± 3.3 mmol/mol) compared to men who did return their logbooks, respectively.

Engagement with online self-monitoring

Overall, men were satisfied (71% agree/strongly agree) with the CaloriekingTM online diet and exercise self-monitoring tool (Table 3). When asked which aspects of the PULSE Program they liked, men commented:

Recording food and exercise in the Calorie King website

(Age 47, Weight loss −7.70 kg, HbA1c change −0.9%, −9.8 mmol/mol)

Visual of energy input and energy output bar chart from calorieking

(Age 61, Weight loss −15.5 kg, HbA1c change −0.7%, −7.7 mmol/mol)

As we did not obtain access to CaloriekingTM usage data, we cannot report adherence to online self-monitoring. Just over half the men thought the website was easy to use (58% agree/strongly agree); the remainder either disagreed (25%) or held a neutral position (17%). Dissatisfaction was likely due to the time it took to record diet and exercise behaviors, with most men reporting that using the website for self-monitoring was time-consuming (67% agree/strongly agree).

Part of the variation in satisfaction level and reports of excessive time burden by men could potentially be explained by varying levels of computer/internet competency, as observed by McConnon, Kirk, and Ransley (2009). Additionally, the provision of multiple self-monitoring components (SHED-IT Weight Loss Handbook for Men, PULSE Exercise Support Book for Men, CaloriekingTM online) may have contributed to these mixed findings. Based on this feedback, future studies might consider individually tailoring self-monitoring options for the participant’s preferences (e.g., paper-based, website, mobile phone app). Further, diet and exercise self-monitoring websites/apps that include integration with commercially available physical activity monitors might offer timesaving benefits, providing they are appropriate for the target group (e.g., age, cognitive ability, computer/mobile competency, and ethnic/cultural background). Also, while usability is one consideration, it is also important that self-monitoring tools utilize high-quality food and exercise databases, and provide high-quality information and feedback.

Self-administration versus face-to-face or additional contact

The only face-to-face component of the PULSE Program was the initial 15 min orientation describing the program and the resources. There was no other contact (phone, email, text) with participants other than scheduling assessments. There were no reminders, encouragement, opportunity to ask questions, or support. Slightly more than half of the men (55% agree/strongly) reported that they would have preferred more face-to-face contact (Table 3). This was despite the aforementioned high satisfaction levels and the significant weight loss and improvements in HbA1c observed for the intervention group at 6 months (Aguiar et al., 2016). This finding is consistent with previous evaluations of the SHED-IT Weight Loss Program (Morgan et al., 2011a; Morgan et al., 2011b; Morgan et al., 2014;). Men commented that they struggled with motivation without regular contact, regardless of whether or not they were successful in achieving improvements in key outcomes:

I found it hard to motivate myself.

(Age 36, Weight loss −7.0%, HbA1c change −0.5%, −5.5 mmol/mol)

lack of regular contact, hard to keep up motivation.

(Age 43, Weight loss −0.8%, HbA1c change −0.3%, −3.3 mmol/mol)

A common suggestion from men was to include face-to-face or non-face-to-face contact (e.g., email, SMS, phone, or video messages) in order to maintain participant motivation, engagement, and accountability. In a recent trial of the SHED-IT Weight Loss Maintenance Program, we implemented several non-face-to-face contact strategies (SMS, email, video; Young et al., 2014), but are yet to determine their impact in that study. Men also suggested we explore avenues to increase social support within the program itself.

Maybe have monthly communication either face to face or over the phone

(Age 47, Weight loss −6.5%, HbA1c change −0.9%, −9.8 mmol/mol)

Monthly appointments, weekly check-ups (phone, email), have other participants join in as a group to support each other

(Age 43, Weight loss −0.8%, HbA1c change −0.3%, −3.3 mmol/mol)

Common strategies used to foster social (peer) support in online weight loss programs include discussion forums and chat groups. These features are purported to offer a source of encouragement, accountability, motivation, information, and shared experience that may assist individuals in successful weight loss (Hwang et al., 2010). In a study investigating social support in an internet weight loss community (N = 193), Hwang and colleagues (2010) reported high levels of engagement and overall satisfaction with an online forum for weight loss support. Interestingly, their study population was predominately female (93.8%). In contrast, previous studies by our group have found online forums to be underutilized (Morgan et al., 2011a), with men reporting that they viewed their weight loss as a personal journey (Morgan et al., 2011a). Future research might investigate the influence of gender on social support use (e.g., discussion forums, chat groups) in online weight loss programs.

Strengths and Limitations of This Process Evaluation

This study has several strengths, including a quantitative and qualitative process evaluation exploring the reasons why men, an understudied population in the diabetes and weight loss literatures, enrolled in lifestyle intervention, and their perceptions, engagement, and satisfaction with a gender-targeted, self-administered lifestyle intervention. A limitation of the current process evaluation is that adherence and process data were only collected from men who returned for follow-up at 6 months, resulting in small sample size. Further, the results are not representative of men who withdrew from the study. In addition, logbook reporting of exercise behaviors was poor; therefore, associations between self-monitoring compliance and change in outcomes should be interpreted with caution. Another limitation was that the perceptions, engagement, and satisfaction of men were assessed using a questionnaire that included Likert scale items and open-ended questions. The process evaluation could have been strengthened by inclusion of semistructured qualitative face-to-face interviews, focus group discussions, or telephone interviews.

Conclusions

The aim of this process evaluation of the PULSE Program RCT was to assess the effectiveness of the trial’s recruitment and selection procedures, and the acceptability of the intervention’s content and delivery. The pragmatic approach for the screening of men, which included the AUSDRISK screening tool (Chen et al., 2010), was effective in identifying men at high risk for developing T2DM (Aguiar et al., 2015), but also identified men who were at lower risk and interested in enrolling in a lifestyle intervention. This approach could be readily adopted in community settings to identify men who might benefit from lifestyle intervention, while reducing costs associated with blood biomarker screening for prediabetes. Further, the gender-targeted approach was successful, with men reporting the study was appealing because it focused on weight loss, T2DM prevention, and getting fit, and because the intervention itself was tailored for men and self-administered. Future lifestyle intervention trials targeting men should consider a gender-targeted approach to maximize program appeal.

The intervention program’s delivery and content were considered acceptable based on men’s perceptions, engagement, retention, and satisfaction. Overall, men enjoyed the PULSE Program intervention and believed their health was improved as a result of their participation. The program’s diet and physical activity messages were generally well implemented; however, adherence to paper-based self-monitoring was poor, with men reporting that self-monitoring tasks were time-consuming. Future studies should consider the use of mobile phone apps for self-monitoring of diet and physical activity, provided they are appropriate for the target audience. Also, apps that integrate with wearable physical activity monitors might be preferred, as they may reduce the time burden of self-monitoring and improve adherence. Further, men reported they would have liked more intervention contact, including non-face-to-face communication—for example, SMS, email, and video messaging—to supplement the self-administered program. These low-cost strategies may assist with engagement, motivation, and accountability, while maintaining the advantages of a self-administered delivery mode, such as scalability.