Process evaluation of the Restore4stroke Self-Management intervention 'Plan Ahead!': a stroke-specific self-management intervention.

OBJECTIVE: To investigate whether the self-management intervention was implemented as intended. Additionally, we studied involvement in and satisfaction with the intervention among patients, their partners and therapists.
DESIGN: Mixed method, prospective study.
SETTING: Outpatient facilities of hospitals/rehabilitation centres. PARTICIPANTS: Stroke patients, their partners and therapists from the experimental arm of the Restore4Stroke Self-Management study. INTERVENTION: 'Plan Ahead!' is a 10-week self-management intervention for stroke patients and partners, consisting of seven two-hour group sessions. Proactive action planning, education and peer support are main elements of this intervention. MAIN MEASURES: Session logs, questionnaires for therapists, patients and their partners, and focus groups. DATA ANALYSIS: Qualitative data were analysed with thematic analysis supplemented by quasi-statistics. Quantitative data were reported as descriptive statistics.
RESULTS: The study sample consisted of 53 patients and 26 partners taking part in the intervention, and all therapists delivering the intervention (N = 19). At least three-quarters of the intervention sessions were attended by 33 patients and 24 partners. On a scale from 1 to 10, patients, partners and therapists rated the intervention with mean scores of 7.5 (SD1.6), 7.8 (SD.7) and 7.4 (SD.7), respectively. Peer support was the most frequently appreciated element for participants and therapists. The proactive action planning tool was adequately applied in 76 of the 96 sessions.
CONCLUSION: Although the target audience was reached and both participants and therapists were satisfied with the intervention, the proactive action planning tool that distinguishes the current intervention from existing stroke-specific self-management interventions was only partly implemented according to protocol.

Introduction

Self-management can be defined as a person’s abilities to manage the consequences of a condition and its impact on daily life.[1] Most stroke-specific self-management interventions aim either to adjust the participants’ goals or self-efficacy, or to teach reactive strategies for dealing with stroke-related problems only after the problem has occurred.[2-9] Nevertheless, in practice, patients often fail to achieve their goals, hindered by unexpected stroke consequences such as fatigue.[10] Therefore, it seems worthwhile to teach both stroke patients and their partners to anticipate potential barriers during the process of goal-setting. This allows the patients and their partners to think of potential solutions to these barriers before undertaking an activity.

Within the Restore4Stroke programme, we developed a stroke-specific self-management intervention named ‘Plan ahead!’ to enhance stroke patients’ and their partners’ participation in vocational, leisure and social activities by teaching them proactive coping strategies.[11] The effectiveness of this intervention has been evaluated in a randomized controlled trial.[12,13] As many processes influence the outcomes of such trials, it is important to conduct a process evaluation to reveal factors influencing outcomes, providing a correct interpretation and explanation of the intervention effects.[14-16] Moreover, such insights provide opportunities to facilitate intervention implementation.[17]

A process evaluation is a method that enables researchers to look into the black-box of processes underlying the outcomes of a clinical trial.[16] Such a process can provide information about the factors influencing the effectiveness of an intervention,[14,15] the internal and external validity of the trial[17] and the experiences of healthcare professionals and participants exposed to the intervention.[15] Such information can be useful for duplicating the study or comparing it with other studies.[14,15] Moreover, the information provides opportunities for better implementation or improvement of the intervention.[17]

In this article, we present the outcomes of our process evaluation, which was performed alongside the Restore4Stroke Self-Management trial. In this evaluation, we investigated the degree to which the intervention was implemented as intended, as well as the involvement and satisfaction of the target audience (i.e. patients, partners and therapists). The study was based on the following elements of the process evaluation framework proposed by Saunders et al.:[18] (1) reach (i.e. the proportion of target audience that participates in the intervention), (2) dose delivered (i.e. the extent to which the intervention components were delivered to the participants), (3) dose received in terms of exposure (i.e. the extent to which the participants actively engage in the intervention), (4) dose received in terms of satisfaction (i.e. the participants’ and the therapists’ satisfaction with the intervention), and (5) recruitment (i.e. procedures to approach the participants and ensure the participants’ continued participation in the intervention).

Method

In this study, a prospective, mixed-method design was applied, combining qualitative and quantitative data.

Intervention

The stroke-specific self-management intervention called ‘Plan Ahead!’ aims to increase stroke patients’ and their partners’ participation levels in vocational, leisure and social activities. This aim is accomplished by teaching the participants proactive action planning in the context of four themes: ‘handling negative emotions’, ‘social relations and support’, ‘participation in society’ and ‘less visible consequences of stroke’. The intervention lasts 10 weeks and involves six two-hour sessions in the first weeks and a two-hour booster session in the tenth week. The intervention is offered by two rehabilitation professionals at outpatient facilities of hospitals and rehabilitation centres. The groups consist of four stroke patients and, if available, their partners. The partners are regarded as full participants. The intervention is described in more detail in the published treatment protocol.[11]

The therapists received six-hours of training in a group of 8–14 therapists. At the start, the therapists received a workbook. The content and structure of the intervention were explained, and the importance of following the treatment protocol explicitly was emphasized. The therapists were taught to support the participants from a solution-based perspective. That is, they were taught to ask questions that focus on thinking in terms of goals, opportunities and solutions, instead of problems and barriers.[19] In addition, the therapists learned how they could apply this perspective in motivating the participants to work on their proactive action plans.

Participants

The process evaluation study was conducted at the outpatient facilities of three hospitals and five rehabilitation centres in the Netherlands between February 2013 and May 2014. Data were collected from all patients and partners in the experimental arm that took part in at least one session of the self-management intervention ‘Plan Ahead!’ of the Restore4Stroke Self-Management study.[12,13] Inclusion and exclusion criteria for patients and partners, as well as recruitment procedures, are described in the research protocol.[12] In addition to the participants of the self-management intervention (i.e. stroke patients and partners), data were collected from all therapists who delivered the intervention. The therapists were recruited at participating institutes by a rehabilitation physician or manager. The therapist inclusion and exclusion criteria can be found in the treatment protocol.[11]

Data collection

We gathered information about reach, dose delivered, dose received in terms of exposure, dose received in terms of satisfaction and recruitment.[18] Table 1 presents an overview of the measures used to collect this information.

Table 1.

Outcome measures of the process evaluation.

Component	Operationalization	Participant evaluation form	Therapist evaluation form	Therapist session log	Therapist focus interview
Reach	Attendance by the participants			X
Dose delivered	Delivery of the intervention components to the participants			X
Dose received – exposure	Overall engagement, atmosphere and trust in the group			X
	Participants’ goal-setting engagement			X
	Participants’ homework engagement			X
Dose received – satisfaction	Overall opinion about the intervention	X	X
	Opinion about the value of the intervention	X	X
	Opinion about the value of the main elements of the intervention	X	X
	Opinion about the number/frequency/length of sessions		X
	Opinion about the therapists’ training		X		X
	Barriers to implementation				X
	Suggestions for improvement				X
Recruitment	Recruitment procedures				X
	Maintaining participant engagement				X

The participants’ data were gathered using an evaluation form that they received after completing the intervention as part of the post-intervention measurement of the Restore4Stroke Self-Management study. This form was used to assess the participants’ satisfaction with the intervention using structured questions (i.e. a scale question, Likert scales and a multiple selection question).

The therapist data were gathered using a session log, an evaluation form and a two-hour focus group interview. The therapists were asked to complete the session log at the end of each session. These logs were used to assess the session course and content, using open and structured questions (i.e. yes–no or Likert scales). In addition, after the 10-week self-management intervention, the therapists were asked to complete a digital evaluation form. This evaluation form assessed the therapists’ satisfaction with the intervention and the group training using structured questions. That is, the form contained a scale question, Likert scale questions, yes–no questions and a multiple choice question.

At the end of the study, the therapists were invited to take part in a two-hour focus group held at a central site after they had completed the evaluation form. If two or more therapists of one research site could not participate at the central site, a focus group was held at their own site. Such site-specific focus groups were used only if the intervention had been delivered to at least 16 patients at the participating centre. The focus groups were led by a research assistant. The focus groups were recorded on video and audiotape, and notes were taken. There was a structured list of questions about the therapists’ satisfaction with the intervention, their opinion about recruitment procedures and their impression of maintaining participant engagement. Afterwards, one of the authors (NT) transcribed the taped focus groups by repeatedly listening to the audiotaped recordings. If what was said was unclear, the videotapes were watched as well to obtain the correct transcript. Then, the accuracy of the transcription was checked by a researcher (WV).

The patients’ background characteristics were recorded using a self-assessment questionnaire before the start of the intervention, assessing age, sex, education level, employment status, marital status and several stroke characteristics (i.e. months since stroke, stroke history and independence in activities of daily living assessed with the Barthel Index 0–20[20]). The partners’ background characteristics were also recorded using a self-assessment questionnaire before the start of the intervention, assessing age, sex, education level and employment status. The therapists’ background characteristics were recorded using the digital evaluation form, assessing the therapists’ sex, age, work setting, profession, years of experience working with acquired brain injury patients and number of times they had delivered the intervention.

All questionnaires used in this study can be found in Appendices 1 to 3, available online.

Data analysis

The quantitative data from the evaluation and session logs were analysed in terms of descriptive statistics using IBM SPSS Statistics 21. Answers to structured, categorical questions (i.e. multiple choice questions, yes–no questions and Likert-scale questions) from the evaluation forms and session logs were described in terms of percentages.

The qualitative data from the open questions on the evaluation forms and session logs were categorized based on their content and reported if an answer had been given by at least half of the patients, partners or therapists.

The data from the focus groups were analysed based on the grounded theory as proposed by Strauss.[21,22] That is, transcripts were first divided into fragments, and codes were assigned to each fragment, i.e. open coding. Subsequently, the codes were validated for each fragment and adapted where necessary, leading to minor taxonomy modifications, i.e. axial coding. Afterwards, the taxonomy was checked once more for consistency and completeness, i.e. selective coding. The latter phase was carried out using the framework of Saunders et al.[18] Each of these steps was first performed independently by WV and NT, after which the two researchers compared their results. Finally, the results were discussed with CvH, AV and VS until a final coding of the data was formulated.

Results

Response and background characteristics

In total, 58 patients and 28 partners were assigned to the self-management intervention in three hospitals and five rehabilitation centres. Figure 1 (available online) presents the number of patients and partners assigned to the self-management intervention at each institute. All institutes participated in this process evaluation. A total of three focus groups were held, two of which were site-specific.

Session logs were completed by the therapists for 53 of the 58 patients (91%) and 25 of the 28 partners (89%). The characteristics of these patients and their partners are presented in Table 2. No session log data were collected for five of the 58 patients (9%) because they did not attend any of the sessions owing to their physical condition (n = 1) or for unknown reasons (n = 1) or because they quit the study owing to dissatisfaction with the intervention (n = 2) or communicative impairment (n = 1). Session log data were not collected for three of the 28 partners (11%). These partners did not attend any of the sessions owing to other commitments (n = 1), the patients’ inability to take part in the intervention (n = 1) or unknown reasons (n = 1).

Table 2.

Background characteristics of patients and partners.

	Patient (n = 53)	Partner (n = 25)
Demographic characteristics
Sex: males, n (%)	24 (45)	14 (50)
Age in years, mean ± SD (range)	55.5 ±9.1 (36–68)	57.2 ±7.9 (45–70)
Education level: Higher general education or above, n (%)	35 (69)a	19 (76)[c]
Employment status: Employed, n (%)	12 (23)	14 (50)
Participants living with a partner, n (%)	38 (72)	25 (100.0)
Stroke characteristics
Time after stroke in months at baseline (T0), mean ± SD (range)	14.5 ±19.1 (1–113)[b]
Stroke history: more than 1 stroke, n (%)	7 (13.2)[b]
Barthel Index 0–20, mean ± SD (range)	19.0 ±2.5 (4–20)

n = 51.

n = 52.

n = 23.

In total, 52 of the 53 patients (98%) and all 25 partners (100%) returned the participant evaluation form.

Therapist characteristics

All 19 therapists delivering the self-management intervention completed the evaluation form for therapists and the session logs for the 53 patients and 25 partners who completed the intervention. Of the 19 therapists approached for participation in the focus group, nine consented (47%) (i.e. five participated in the central focus group and four participated in a site-specific focus group). The reasons for therapists not to participate in one of the focus groups were holidays (n = 4), another therapist at their institute already participating in a focus group (n = 4), their own physical condition (n = 1) and other commitments (n = 1). The therapists’ characteristics are presented in Table 3, available online.

Reach

In all, 16 intervention groups took place in the Restore4Stroke Self-Management study. Each institute served an average of 2.0 intervention groups (SD 1.1; range 1–4). The groups had a median size of four patients (range 2–5) and two partners (range 0–3).

Table 4 shows the participation rates among the participants in detail as an indicator of intervention reach. Overall, it seems that relatively few participants actually took part in all seven intervention sessions, but most participants did attend at least three-quarters of the intervention, including the booster session.

Table 4.

Participation and attendance rates among participants.

	Patients (n = 58)	Partners (n = 28)
No attendance, n (%)	2 (3%)	3 (11%)
Quit the intervention, n (%)	3 (5%)	0 (0%)
At least one session, n (%)	53 (91%)	25 (89%)
At least three-quarters attended, n (%)	46 (87%*)	24 (96%[a])
All seven sessions, n (%)	33 (62%*)	15 (60%[a])
Attended booster session, n (%)	50 (94%[a])	24 (96%[a])

Percentages in relation to participants (patients or partners) who attended at least one session.

Dose delivered

Table 5 describes several results related to the dose delivered in detail. In 20% of the sessions, the proactive action plan tool was inadequately used according to the therapist sessions logs.

Table 5.

Results related to the dose delivered.

	Number (%)	Notes
Total amount of sessions	112	16 multiplied by 7 intervention sessions
Sessions with one therapist absent	6 (5%)
Sessions containing work on proactive action plan	96	5 of 7 sessions per intervention
Inadequate application of proactive action planning tool, reasons:	20 (20%[a])
- Groups not split up	4 (4%[a])
- Proactive action plans not worked out in sufficient detail	16 (17%[a])

Of the sessions containing work on proactive action plan (96 sessions.)

Dose received: Exposure

Table 6 describes the therapists’ assessments of participant engagement, group atmosphere and trust among the participants in each intervention group. On these three dimensions, no group was assessed below sufficient or acceptable levels, and most groups were assessed to perform beyond these levels.

Table 6.

Therapist assessments of engagement, atmosphere and trust among participants.

Dimension	Assessment	Number of groups (Total = 16) (percentage)
Engagement	Good	13 (81%)
	Sufficient	3 (19%)
	Mediocre	0 (0%)
	Insufficient	0 (0%)
Atmosphere	Pleasant	13 (81%)
	Acceptable	3 (19%)
	Mediocre	0 (0%)
	Unpleasant	0 (0%)
Trust among participants	Very high	3 (19%)
	High	11 (69%)
	Sufficient	2 (13%)
	Low	0 (0%)
	Very low	0 (0%)

Source: Therapist session logs.

Figure 2(A) (available online) presents the percentage of participants engaging in active goal-setting for each session, as reported by the therapists on the session logs. A total of 25 of the 53 patients (47%) worked on their goals during all five sessions requiring goal-setting, as did 13 of the 25 partners (52%).

Figure 2(B) (available online) presents the percentage of participants completing homework assignments for each session, as reported by the therapists on the session logs. A total of 12 patients (23%) did all of their homework assignments, as did seven partners (28%). At session four, the therapists reported a considerable decline in the percentage of patients and partners doing their homework assignments.

Dose received: Satisfaction

On a scale from 1 to 10, the patients, partners and therapists rated the intervention with mean scores of 7.5 (SD 1.6; range 2–10), 7.8 (SD 0.7; range 7–9), and 7.4 (SD 0.7; range 5.5–8.5), respectively.

As regards the structure of the intervention, 13 of the 19 therapists (68%) assessed the number of sessions, 16 (84%) assessed the frequency of sessions and 11 (58%) assessed the length of the sessions as appropriate.

With regard to the intervention content, 47 of the 52 patients (90%) and all 25 partners and 19 therapists considered the intervention to be somewhat useful to very useful (see Table 7). Continued delivery of the intervention after the research project was preferred by 18 of the 19 therapists (95%).

Table 7.

Usefulness of the self-management intervention according to patients (n=52), partners (n=25) and therapists (n=19).

	Patients (n = 52)	Partners (n = 25)	Therapists (n = 19)
Very useful, n (%)	15 (28.8)	8 (32.0)	3 (15.8)
Useful, n (%)	24 (46.2)	10 (40.0)	12 (63.2)
Somewhat useful, n (%)	8 (15.4)	7 (28.0)	4 (21.1)
Not useful, n (%)	5 (9.6)	0 (0)	0 (0)

Figure 3 (available online) presents the elements of the intervention that were most often rated as valuable by the patients, partners and therapists. It was found that generic components, such as ‘peer support’, were most often valued by the respondents (both the participants and the therapists.) The theme of ‘less visible stroke consequences’ was reported as valuable by a larger proportion of the partners than by the therapists and the patients.

A total of 15 of the 19 therapists (79%) attended the group training. All of the therapists reported that the course had helped them to deliver the intervention. The other four therapists were individually trained because they were not able to attend the group training.

Important barriers for intervention implementation mentioned in all three focus groups were existing interventions being already used in a given centre, with overlapping content:

So, maybe [the existing intervention] is not scientifically substantiated, but (…) a team [of therapists] may have already embraced [this intervention]. (Psychologist, focus group 1)

An additional barrier mentioned in two of the focus groups was the challenge of fitting a group intervention into existing timetables of healthcare professionals and facilities:

Well, the organizational planning, that’s sort of our obstruction I think. (…) it has a large impact on our timetables. (Occupational therapist, focus group 1)

Suggested improvements to the intervention content in two focus groups included abandoning the obligatory link between the themes and the proactive action plan, adding one session without a fixed theme and having the booster session at a later moment in time:

I get the reasoning… of on one hand wanting to discuss the subject but on the other wanting to make a link with the roadmap… but sometimes this is very far-fetched… and (…) for the patient that’s often very complicated. (Psychologist, focus group 1)

That was also a shortcoming, that there was no room for a specific theme, that sometimes emerged from the group as well. (Social worker, focus group 2)

So there was a kind of reunion and there were a few weeks between (…) maybe you could do this at a later time (…) that kind of big stick would be nice to have… (Psychologist trainer, focus group 1)

A suggested improvement to the inclusion criteria of the intervention in two focus groups was to include patients with other forms of acquired brain injury:

We were also wondering about (…) whether this would work in other (…) diagnoses… Yes. [for example in] many MS patients. (Psychologist, focus group 2)

Broadening of the inclusion criteria for partners to allow non-cohabiting partners and other relatives to take part in the intervention as well was recommended in all three focus groups:

Well … I don’t think you would necessarily have to make a distinction … so you may be able to … (Psychologist, focus group 1)

Include them, yes, yes… Involve them, yes… (Psychologist trainer, focus group 1)

A suggested improvement to the criteria for therapists in all three focus groups was that the two therapists should have a different professional background, share the preparation and collaborate closely. In two focus groups, it was recommended that at least one of the therapists should have a professional background in the psychosocial domain and that the therapists should receive a more elaborate training course, provided nearer to the provision of the first session.

Recruitment

No barriers regarding the recruitment of patients were reported by the therapists during the focus groups. A barrier in the recruitment of partners reported in all focus groups was a lack of emphasis during the intake procedures on the possibility for partners to take part in the intervention:

(So) there was one partner in our case that was like, oh, so I was supposed to join the patient. Apparently that wasn’t clear to him on forehand. (Occupational therapist, focus group 1)

Other activities of partners, such as work, were reported as important barriers in the recruitment of partners in two focus groups.

Important barriers for maintaining the participants’ engagement reported in all focus groups were the presence of considerable cognitive impairments, a lack of intrinsic motivation to take part in the intervention and a lack of awareness of the active role required of them during the intervention. Other factors mentioned in two focus groups were the participants’ limited experience with the consequences of stroke in daily life and the absence of a therapist during the intake procedure.

Discussion

In this process evaluation, it was found that most participants were only partly reached. A relatively large proportion did not attend all sessions. Inadequate application of the proactive action plan was observed relatively often, mostly because the plan was not filled out in sufficient detail. It was also observed that relatively few participants worked consistently on proactive action planning in terms of goal-setting and doing their homework.

Overall, the patients, partners and therapists were satisfied with the intervention. In this study, we found that the components of ‘peer support’ and ‘invisible consequences of stroke’ were frequently valued by the participants and the therapists. Based on earlier research, the value of peer support may be related to its association with increased awareness of stroke consequences, as well as opportunities for peer comparisons among stroke patients and their partners.[23,24] The interest in invisible stroke consequences can be explained by the distressing impact of such consequences for the patient’s partner.[25] The popularity of these components unrelated to proactive action planning could imply that intervention-specific elements were insufficiently picked up by the participants and the therapists.

The therapists found their preparatory training to be useful, although some indicated that it could have been more elaborate and occurred closer in time to the start of the intervention itself. In addition, earlier research showed that insufficiently trained therapists is an important barrier for successfully implementing self-management interventions.[26] As such, delivering our intervention might have been more complex than expected, possibly leading to therapists not delivering the intervention according to treatment protocol in a considerable percentage of the sessions.

The general implementation barriers observed by therapists predominantly lie in fitting the intervention into the healthcare system. For instance, schedules in the healthcare system are tight. Regarding the intervention itself, the therapists made several recommendations: (a) to abandon the obligatory link between the proactive action planning tool and the four themes discussed in the intervention, (b) to add a session without a fixed theme and (c) to plan the booster session at a later time. In general, the therapists thought that each of the therapists in one intervention should have different backgrounds, share their preparations and collaborate closely. Implementing these results is expected to increase the chances of the intervention protocol being carried out correctly and completely, both for the participants and the therapists.

Regarding recruitment, the therapists suggested that selection criteria should be broadened to include patients with similar brain conditions, as well as non-cohabiting partners and other relatives. As such, it seems to be questionable whether our intervention is suitable for only stroke patients. Furthermore, the therapists reflected that they could have more clearly emphasized the possibility for partners to participate. This lack of emphasis may reflect a traditional therapist tendency towards a patient-centred focus rather than a family-centred one.[26-28] The therapists also thought that they could have further emphasized the active participant role required by the intervention at recruitment. We estimate that the latter suggestion may eventually lead to increased homework assignment completion levels throughout the intervention.

Several strengths of the current process evaluation study are noteworthy. First, a very high response rate was observed, as was the participation of a considerable number of institutes. This response rate is argued to have yielded a reliable reflection of clinical outpatient rehabilitation practice, at least in the Netherlands. In addition, a strength of this study lies in its use of both qualitative and quantitative research methods, because the two methods complement each other. Finally, the outcomes of the process evaluation were not biased by trial outcomes because these were unknown at the time of this process evaluation study.

An important limitation of our study was its lack of attention to the fidelity of the intervention implementation, i.e. the degree to which each element of the intervention was correctly implemented.[18] It is argued that this factor could have been measured only by making observations (researcher in person, or using a camera) during sessions. Such observation was impossible to do within organizational constraints. Furthermore, the outcomes may have been biased by the participants giving socially desirable answers. To reduce this bias, the therapists and the participants were asked to complete the questionnaires in the researcher’s absence; and interviews were conducted by a research assistant who did not take part in the study and who evaluated the effectiveness of the self-management intervention. In addition, the ‘minimum intervention delivery’ requirement used for focus groups might have distorted our results. That is, additional focus groups on sites that did not meet this minimum delivery requirement might have yielded additional barrier insights, for example, regarding recruitment.

In summary, although it seems that the target audience was reached, and the patients, partners and therapists were satisfied with the intervention, there is still room for improvement. In particular, the proactive action planning tool forms the core of what distinguishes the current intervention from existing evidence-based self-management interventions for stroke patients. Therefore, increasing the correct and complete use of this tool, as well as improved therapist training and recruitment procedures, deserve high priority. Based on our study outcomes, researchers and policymakers should be aware that an adequate implementation of interventions such as ours is complex and needs time, because both healthcare professionals and participants still have to become accustomed to the participants’ active role in managing their own situation.

In this self-management intervention, participants find the generic component ‘peer support’ to be most valuable.

Participants’ and therapists’ satisfaction with a stroke-specific self-management intervention does not imply an optimal intervention implementation.

For an adequate implementation of interventions like ours, a thorough training for therapists is needed.