Process and Outcome Evaluations of Interventions to Promote Voluntary Exercise Training Among South Korean Firefighters.

The main aim of the present study is to evaluate reach, dose, fidelity, and outcomes of the interventions for promoting voluntary exercise training among South Korean firefighters. Four interventions for promoting voluntary exercise training among firefighters (i.e., virtual reality exercise system intervention, poster intervention, monitor intervention, and wearable health device intervention) were performed in a fire station located in Seoul, South Korea. To evaluate reach and dose received related to each intervention, participants were asked to answer several simple questions. Three process evaluators completed a 20-item survey to share their impressions related to the quality of intervention delivery. Paired t test was used to examine mean changes in primary (i.e., mean minutes of exercise training per week) and secondary outcomes (i.e., beliefs and intention) between pre- and postinterventions. More than 60% of participants experienced monitor and wearable health device interventions. Process evaluators tend not to agree with a statement saying that the number of the virtual reality exercise equipment was appropriate. Among firefighters who participated in exercise training less than 150 min per week at 1-month follow-up, mean minutes of exercise training per week increased by 67.95 min after interventions. Future studies need to examine whether the monitor and wearable health device interventions effectively increase exercise training participation among firefighters in other fire stations located in Seoul, South Korea.

Introduction

Firefighters need to perform dangerous and difficult rescue operations. They must carry heavy tools and climb stairs and ladders while wearing personal protective equipment. Personal protective equipment imposes a great physiological burden because of its weight, insulative properties, and restrictiveness. Therefore, firefighters must have a high level of physical capacity (Dennison et al., 2012; Smith, 2011). However, in 2018, the average VO2max of firefighters in South Korea was 40.23 mL/min/kg, which is much lower than that (42 mL/min/kg) recommended by the National Fire Protection Association (NFPA, 2007). More than half (55.03%) of South Korean firefighters did not perform 150 min of exercise training per week in 2018 (National Fire Agency, 2018). This is a critical problem because poor physical fitness is strongly associated with low job performance during firefighting activities (Elsner & Kolkhorst, 2008; Michaelides et al., 2008; Rhea et al., 2004) and increased cardiovascular disease risk among firefighters (Davis et al., 2002; Soteriades et al., 2011). A poor physical fitness and a low participation in exercise training of South Korean firefighters are very likely to be caused by a legislation of the Korean National Fire Agency (KFA) specifying that the physical fitness test results contribute only 5% of a total score of the promotion examination (Korea Law Information Center, 2020). Therefore, the effective interventions for promoting voluntary exercise training (i.e., performing exercise training on their own initiative) among South Korean firefighters should be developed and applied.

In January 2020, Lee et al. conducted a semi-structured interview to elicit salient beliefs about voluntary exercise training among South Korean firefighters. They found out three salient beliefs that significantly affect their exercise training behavior (Lee et al., 2020). Based on the results of this study, they developed and applied multiple concurrent interventions for changing each of the three salient beliefs (i.e., “doing exercise training improves my physical ability,” “doing exercise training takes too much time,” and “my colleagues think that I should do exercise training”) to promote voluntary exercise training among South Korean firefighters. This voluntary exercise training promotion intervention for firefighters includes the following components: (a) providing a scientifically proven exercise training program in the fire station using virtual exercise equipment and poster; (b) offering wearable health devices to promote exercise-related communications among firefighters in the same fire station; and (c) installing monitors in the fire station to show firefighters messages emphasizing that their coworkers think that they should do exercise training. These multiple concurrent interventions were quite complex because they included multiple components that target not only individuals but also social and physical environments. As intervention complexity can cause poor implementation of interventions, it is essential to examine whether interventions were delivered or received as planned (Glasgow & Linnan, 2008; Young et al., 2008). This process evaluation allows researchers and practitioners to improve knowledge of how well each intervention was delivered or received and, therefore, increase the ability to make valid judgment about outcomes (Durlak & DuPre, 2008). Moreover, process evaluation, together with outcome evaluation, has the capacity to suggest future directions for interventions in different settings (van de Glind et al., 2017).

Usually, process evaluation aims to capture reach, dose, and fidelity of interventions (Glasgow & Linnan, 2008; Saunders et al., 2005; Steckler et al., 2002). The reach of interventions is related to whether the intended audience participates in each intervention and is typically assessed by measuring attendance. The dose of interventions can be divided into dose delivered and dose received. Dose delivered refers to the extent to which intended units of each intervention delivered or provided by implementers and can be measured using direct observation of intervention procedure. Dose received refers to the extent to which participants interact with, are receptive to, engage with, and use recommended resources or materials or the degree to which participants satisfy with programs and interact with intervention staffs. Dose received is usually assessed by conducting interviews with participants and administering brief satisfaction scales to participants. The fidelity of interventions refers to the quality of implementation and pertains to how well the implementation of interventions reflects the underlying philosophy and theory. Reach and dose received are characteristics of participants, whereas dose delivered and fidelity are functions of interventionists. A thorough evaluation of reach, dose, and fidelity can assist in elucidating why negative outcomes occurred and, eventually, improving outcomes of interventions. The main aim of the present study is to evaluate reach, dose, fidelity, and outcomes of the voluntary exercise training promotion interventions developed by Lee et al. (2020) for South Korean firefighters.

Method

Participants

A total of 175 firefighters were recruited from one fire station located in Seoul, South Korea, for participation. Firefighters whose job is to rescue people in danger, suppress fire, or give first aid to victims were selected because their job is more crucially associated with physical ability than that of firefighters who stay in the office (e.g., administrative staffs). The head of the fire station gave permission for employee participation and execution of interventions in the fire station. The first author trained three graduate students to distribute consent forms to firefighters and perform survey. In August 2018, approximately 81.4% (N = 175) of target population (i.e., firefighters) provided written informed consent for completing a baseline survey. A 1-month follow-up survey was performed to measure voluntary exercise training and the response rate was 97.7% (N = 171). Further information about baseline and 1-month follow-up survey is available in Lee et al.’s study. A 1-year follow-up survey was conducted in November 2019 to evaluate process and outcome of interventions and the response rate was 57.9% (N = 99). The main reasons for dropouts were vacations, reassignment to different fire station, and personal reasons unrelated to interventions. The present study was approved by Yonsei University Institutional Review Board (IRB No. CR318031).

Interventions

The main purpose of the voluntary exercise training promotion interventions was to change each of the three salient beliefs (i.e., “doing exercise training improves my physical ability,” “doing exercise training takes too much time,” and “my colleagues think that I should do exercise training”) that significantly affected firefighters’ exercise training behavior (Lee et al., 2020). To lead firefighters to believe that performing exercise training improves their physical ability, a scientifically proven 15-min exercise training program was developed by five exercise physiology professors and provided to firefighters in the fire station using a virtual reality exercise system that includes video instructions (virtual reality exercise system intervention; Ahn et al., 2018). These instructions were also printed on posters and posted on the walls of the training room located in the fire station (poster intervention). In an attempt to induce firefighters to believe that their colleagues think that they should do exercise training, firefighters were given wearable health devices (wearable health device intervention). Utilizing wearable health devices has been identified to increase participation in exercise by having common exercise goals, exchanging experiences (Stepanovic et al., 2019), or group dynamics (Gorm & Shklovski, 2016) among recipients. Finally, a total of seven monitors were installed in hallway, cafeteria, garage, offices, and training room in the fire station to change all three beliefs mentioned earlier (monitor intervention). The monitors showed firefighters messages emphasizing that their colleagues want them to do exercise training and are currently doing exercise training. The monitors also showed that they can do exercise training during standby periods between emergency calls, their coworkers are using wearable health devices, and there are virtual reality exercise equipment and posters that include instructions for a scientifically proven 15-min exercise training program in the fire station. The wearable health device and monitor interventions were initiated on June 2019. However, the virtual reality exercise system intervention was initiated on September 2019 and the poster intervention was initiated on October 2019 due to delayed development of a scientifically proven 15-min exercise training program.

Data Collection

To evaluate outcomes of the voluntary exercise training promotion interventions for Korean firefighters, outcomes were measured before and after the interventions. The primary outcome (i.e., participation in voluntary exercise training) was assessed in the 1-month and 1-year follow-up survey and the secondary outcomes (i.e., intention to participate in voluntary exercise training and three salient beliefs) were measured at baseline and 1-year follow-up. The questionnaire for measuring primary and secondary outcomes of interventions is described in Lee et al.’s (2020) study.

Intervention reach, dose, and fidelity were evaluated during and after the interventions. To evaluate reach and dose received related to each intervention, participants were asked to answer several simple questions (e.g., did you receive the wearable health device that we provided?) in the 1-year follow-up survey (see Table 1). Three process evaluators (i.e., two female and one male graduate students trained by the first author) visited fire station every 2 weeks during the interventions to evaluate dose delivered and fidelity. These process evaluators completed a 20-item survey to share their impressions related to the quality of intervention delivery. The survey questions were adapted from those used in previous evaluation studies (Robbins et al., 2014; Young et al., 2008) and redesigned by the authors according to the characteristics of each intervention. A 4-point Likert-type scale ranging from 1 (disagree a lot) to 4 (agree a lot) was used to assess each item. The quality of intervention delivery related to each item is considered high if the mean score is above 2.5. The survey questions that the process evaluators answered are listed in Table 2.

Table 1.

Firefighters’ Responses to Yes/No Questions Asking Whether Firefighters Actually Experienced Each Intervention (N = 99).

Questions	Yes (%)
1. Did you receive the wearable health device that we provided?	80 (80.81)
2. Did you use the wearable health device?	61 (61.62)
3. Did you know that instructions for exercise training were printed on posters and pasted on the walls of the training room located in the fire station?	90 (90.91)
4. Did you follow the instructions printed on posters?	48 (48.48)
5. Did you see the messages shown on monitors that were installed in hallway, cafeteria, garage, offices, and training room in the fire station?	83 (83.84)
6. Did you know that the virtual reality exercise equipment that includes video instructions for exercise training are installed in the fire station?	72 (73.47)
7. Did you use the virtual reality exercise equipment?	28 (28.28)

Table 2.

Process Evaluators’ Responses to Questions Asking Whether Interventions Were Implemented as Planned (N = 3).

Questions	M (SD)
1. Were the monitors securely installed in the right places?	4.00 (0.00)
2. Were the monitors properly operated?	3.33 (0.58)
3. Did firefighters appear to like the messages shown on monitors?	3.33 (0.58)
4. Did firefighters carefully look into the monitors?	3.33 (0.58)
5. Were sizes of the monitors appropriate?	3.00 (1.00)
6. Was the number of the monitors appropriate?	3.33 (0.58)
7. Did staffs provide the wearable health devices to all firefighters in the fire station?	4.00 (0.00)
8. Did staffs give clear instructions for using the wearable health devices to firefighters?	4.00 (0.00)
9. Did firefighters appear to like the wearable health devices?	4.00 (0.00)
10. Did firefighters actively use the wearable health devices?	3.00 (0.00)
11. Were the virtual reality exercise equipment securely installed in the right places?	4.00 (0.00)
12. Did staffs give clear instructions for using the virtual reality exercise equipment to firefighters?	4.00 (0.00)
13. Did firefighters appear to like the virtual reality exercise equipment?	3.00 (1.00)
14. Did firefighters actively use the virtual reality exercise equipment?	2.33 (0.58)
15. Was the number of the virtual reality exercise equipment appropriate?	2.67 (1.53)
16. Were the posters securely attached to the right places?	4.00 (0.00)
17. Did firefighters appear to like the posters?	3.33 (0.58)
18. Did firefighters carefully look into the posters?	3.33 (0.58)
19. Were sizes of the posters appropriate?	4.00 (0.00)
20. Was the number of the posters appropriate?	3.67 (0.58)

Note. All responses ranged from 1 (disagree a lot) to 4 (agree a lot).

Data Analysis

Paired t test was used to examine mean changes in primary outcome (i.e., participation in voluntary exercise training) between 1-month and 1-year follow-up. Mean changes in secondary outcomes (i.e., intention to participate in voluntary exercise training and three salient beliefs) between baseline and 1-year follow-up were assessed using paired t test. Participants were divided into two subgroups according to the number of minutes of participation in voluntary exercise training per week at 1-month follow-up (less than 150 min or more than 149 min) and all outcomes were analyzed in two subgroups. Two-tailed tests were used and all p values less than .05 were considered as significant. All analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Results

Reach and Dose Received

A total of 99 firefighters remained in the fire station during the interventions. These firefighters were exposed to wearable health device and monitor interventions, virtual reality exercise system intervention, and poster intervention for 5 months, 2 months, and 1 month, respectively. Reach and dose received were evaluated based on firefighters’ responses to yes/no questions asking whether firefighters actually experienced each intervention during the intervention period (Table 1). More than 80% of firefighters received the wearable health device and approximately 62% of firefighters actually used it. More than 90% of participants knew that there are posters pasted on the walls of the training room and half of the participants followed the instructions printed on posters. About 84% of participants saw the messages shown on monitors. Although 73% of firefighters knew that the virtual reality exercise equipment is installed in the fire station, only 28% of participants used the virtual reality exercise equipment.

Dose Delivered and Fidelity

The results of a 20-item survey showed that all of the interventions, except for the virtual reality exercise system intervention, were delivered with high fidelity (Table 2). One item asking whether firefighters actively used the virtual reality exercise equipment had a mean score lower than 2.5 (M = 2.33, SD = 0.58). Another item asking whether the number of the virtual reality exercise equipment was appropriate had the second lowest mean score of 2.67. The mean scores of all other items were 3.00 or higher.

Outcomes

The mean changes in the outcomes from pre- to postinterventions are presented in Table 3. Firefighters became less likely to think it is bad if participating in exercise training takes too much time (p < .0001). Among firefighters who participated in exercise training less than 150 min per week at 1-month follow-up, mean minutes of exercise training per week increased by 67.95 min (p = .0017). The mean changes in other outcomes from pre- to postinterventions were not statistically significant.

Table 3.

The Mean Changes in Primary and Secondary Outcomes From Pre- to Postinterventions Among Two Groups of Firefighters (N = 99).

Outcomes	≥150 min/week of ET at 1-month follow-up (n = 60)			<150 min/week of ET at 1-month follow-up (n = 39)
	Pre	Post	Change	Pre	Post	Change
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
ET (min/week)	343.00 (211.21)	298.67 (219.22)	−44.33 (267.10)	70.90 (45.56)	138.85 (130.09)	67.95 (125.90)**
Intention a	2.31 (0.82)	2.36 (0.68)	0.04 (0.88)	1.58 (1.32)	1.65 (0.80)	0.07 (1.31)
Belief 1 a	2.60 (0.49)	2.60 (0.79)	0.00 (0.82)	2.33 (0.81)	2.10 (0.75)	−0.23 (1.04)
OE of Belief 1 a	2.57 (0.67)	2.68 (0.50)	0.12 (0.78)	2.49 (0.51)	2.38 (0.67)	−0.10 (0.79)
Belief 2 a	−0.82 (1.61)	−0.25 (1.81)	0.57 (2.28)	0.23 (1.58)	0.15 (1.46)	−0.08 (1.36)
OE of Belief 2 a	−1.70 (1.29)	−0.52 (1.51)	1.18 (1.44)**	−1.74 (1.07)	−0.18 (1.35)	1.56 (1.67)**
Belief 3 a	2.18 (0.54)	2.15 (0.95)	−0.03 (0.92)	2.03 (0.54)	2.00 (0.69)	−0.03 (0.87)
MC with Belief 3 b	4.72 (1.24)	4.62 (1.54)	−0.10 (1.98)	4.85 (0.93)	4.64 (0.90)	−0.21 (1.24)

Note. ET = exercise training; OE = outcome evaluation; MC = motivation to comply; Belief 1 = participating in ET improves my physical ability; Belief 2 = participating in ET takes too much time; Belief 3 = colleagues think that I should participate in ET.

Responses ranged from −3 to 3. b Responses ranged from 0 to 6.

p < .01.

Discussion

The process and outcome evaluations of four interventions for promoting voluntary exercise training among firefighters (i.e., virtual reality exercise system intervention, poster intervention, monitor intervention, and wearable health device intervention) yielded several important findings. More than 60% of participants experienced monitor and wearable health device interventions and these results are comparable with those of previous studies (Barbeau et al., 2007; Wilson et al., 2011). Only 48% and 28% of participants actually experienced poster and virtual reality exercise system interventions, respectively. One of the main reasons for low participation in poster and virtual reality exercise system interventions is the short durations of the interventions. Although the durations of both monitor and wearable health device interventions were approximately 5 months, the durations of virtual reality exercise system and poster interventions were about 2 and 1 months, respectively, due to delayed development of a scientifically proven exercise training program. The short durations of these interventions may have limited opportunities for using virtual reality exercise equipment and posters among firefighters.

Another reason for low participation in virtual reality exercise system intervention is found in the results of a 20-item survey. The second lowest mean score (M = 2.67, SD = 1.53) indicates that process evaluators tend not to agree with a statement saying that the number of the virtual reality exercise equipment was appropriate. The process evaluators also mentioned that two virtual reality exercise equipment were just not enough to cover more than 250 firefighters in the fire station (not shown in the tables). To evaluate the true effects of virtual reality exercise system intervention on voluntary exercise training among firefighters, more number of virtual reality exercise equipment should be installed in the fire station. Applying online reservation system may help to utilize existing virtual reality exercise equipment more efficiently.

After the intervention period, firefighters became less likely to think it is bad if participating in exercise training takes too much time. The monitor or wearable health device interventions may have contributed to this result because reach, dose, and fidelity of poster and virtual reality exercise system interventions were low and the durations of these interventions were short. To underscore the value of doing exercise training, both monitor and wearable health device interventions intended to disseminate several important messages (i.e., messages emphasizing that their coworkers are using wearable health devices, their colleagues, including the head of the fire station, want them to do exercise training and are currently doing exercise training, they can do exercise training during standby periods between emergency calls) to firefighters. These messages may have led firefighters to have common exercise goals (Stepanovic et al., 2019), formulate group dynamics (Gorm & Shklovski, 2016), and, therefore, spend more time participating in exercise training. The mean changes in other secondary outcomes from pre- to postinterventions were not significant. It is possible that a 7-point Likert-type scale was not enough to capture small but significant changes in these variables.

Indeed, mean minutes per week of exercise training increased by 67.95 min (from 70.90 to 138.85 min) after interventions among firefighters who participated in exercise training less than 150 min per week before the intervention period. This result is meaningful because the results of a review of previous studies have reported that the effects of interventions for promoting physical activity among adult populations were only moderate and most of these studies did not succeed in increasing mean minutes per week of physical activity more than 60 min (Foster et al., 2005). Despite of pre–post design without a control group, the tremendous increase in mean minutes of exercise training per week observed in this study strongly suggests that replication research should be performed to see whether the monitor and wearable health device interventions increase participation in exercise training among firefighters in other fire stations located in Seoul, South Korea.

This study has several limitations. First, the use of self-report questionnaires when assessing exercise training can cause recall, social desirability, and response bias. It is more beneficial to use objectively assessed exercise training to increase accuracy of results (Newell et al., 1999). Second, as this study used multiple intervention strategy, it was not easy to find out the precise contribution of each intervention to the increase in exercise training participation among firefighters. However, conducting process evaluation enabled authors to speculate that changes in primary and secondary outcomes were more likely to be caused by monitor and wearable health device interventions rather than poster and virtual reality exercise system interventions. Third, the pre–post design without control group can underestimate the changes in outcomes that would have occurred without interventions. There were no specific events that may have affected exercise training participation among firefighters occurred during the intervention period.

Conclusion

Despite the limitations listed earlier, the process evaluation of four interventions to promote voluntary exercise training among firefighters yielded important findings indicating that the monitor and wearable health device interventions rather than the poster and virtual reality exercise system interventions may have contributed to mean changes in outcomes. The notable increase in mean minutes of exercise training per week observed in this research indicates that future studies need to examine whether the monitor and wearable health device interventions effectively increase exercise training participation among firefighters in other fire stations located in Seoul, South Korea.