Fidelity protocol for the Action Success Knowledge (ASK) trial: a psychosocial intervention administered by speech and language therapists to prevent depression in people with post-stroke aphasia.

INTRODUCTION: Treatment fidelity is a complex, multifaceted evaluative process which refers to whether a studied intervention was delivered as intended. Monitoring and enhancing fidelity is one recommendation of the TiDIER (Template for Intervention Description and Replication) checklist, as fidelity can inform interpretation and conclusions drawn about treatment effects. Despite the methodological and translational benefits, fidelity strategies have been used inconsistently within health behaviour intervention studies; in particular, within aphasia intervention studies, reporting of fidelity remains relatively rare. This paper describes the development of a fidelity protocol for the Action Success Knowledge (ASK) study, a current cluster randomised trial investigating an early mood intervention for people with aphasia (a language disability caused by stroke). METHODS AND ANALYSIS: A novel fidelity protocol and tool was developed to monitor and enhance fidelity within the two arms (experimental treatment and attention control) of the ASK study. The ASK fidelity protocol was developed based on the National Institutes of Health Behaviour Change Consortium fidelity framework. ETHICS AND DISSEMINATION: The study protocol was approved by the Darling Downs Hospital and Health Service Human Research Ethics Committee in Queensland, Australia under the National Mutual Acceptance scheme of multicentre human research projects. Specific ethics approval was obtained for those participating sites who were not under the National Mutual Agreement at the time of application. The monitoring and ongoing conduct of the research project is in line with requirements under the National Mutual Acceptance. On completion of the trial, findings from the fidelity reviews will be disseminated via publications and conference presentations. TRIAL REGISTRATION NUMBER: ACTRN12614000979651. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

The Action Success Knowledge (ASK) trial’s fidelity protocol is a pragmatic, considered evaluative process, based on the National Institutes of Health Behaviour Change Consortium.

The fidelity protocol sets out the checks and balances in place to monitor and enhance fidelity within delivery of a complex intervention.

One limitation of the ASK fidelity protocol and checklist is that it primarily focuses on how the provider administers treatment to participants with aphasia, as they are the main focus for change.

Treatment receipt and enactment are embedded within the design of the ASK intervention (as demonstrated by the use of accessible aphasia-friendly materials, personalised goal-setting and follow-up telephone reviews), but they are not directly assessed.

Introduction

Treatment fidelity is a complex, multifaceted evaluative process. The aim is to increase scientific confidence in the findings of behavioural intervention studies by monitoring and enhancing the reliability and validity of the intervention(s) delivered.1 2 For studies of interventions for health-related behaviour change, a high level of treatment fidelity is a marker of quality, indicating that end users (ie, researchers, clinicians, healthcare providers and patients) can have confidence in the study’s findings.3 Monitoring and enhancing treatment fidelity forms part of the TiDIER (Template for Intervention Description and Replication) recommendations, a consensus reporting checklist aimed at enhancing the complete description of interventions.4 Without treatment fidelity, conclusions cannot be drawn about treatment effects, leaving open the possibility for type I, type II or type III errors.1 3 5 Within a therapy trial, a common component of fidelity monitoring is to evaluate and enhance the extent to which the therapy provider administers the treatment as planned and how competently treatment is delivered.6–9 The purpose of this paper is to describe the development of a fidelity protocol for the Action Success Knowledge (ASK) study, a current cluster randomised trial investigating an early mood intervention for people with poststroke aphasia (an acquired language and communication disability). The protocol for the ASK trial has been published elsewhere.10 Data collection is currently ongoing; the results of the fidelity monitoring will be reported at the end of data collection, as part of the trial results paper.

Despite the methodological and translational benefits, fidelity monitoring has been used inconsistently within health behaviour intervention studies,11 and no single method has been widely adopted across studies.12 Assessing treatment fidelity is challenging in the context of complex behavioural treatments.13 Such behavioural treatments tend to be situated within an interaction between the therapy provider and the participant and involve a degree of customisation to suit the needs of the individual patient.8 14 Interpersonal interaction is an important contributing factor in delivering a high-quality treatment but creates challenges in training and monitoring treatment administration.13 The need for individual tailoring of the treatment to meet the needs of the patient creates high potential for variation in how the treatment is administered, particularly when the treatment is administered by different providers and across research sites.15 In response to these challenges, a comprehensive framework was developed by a panel of experts who formed the Treatment Fidelity Workshop as part of the National Institutes of Health Behaviour Change Consortium (NIH BCC).1–3 The NIH BCC framework expands on previous concepts of treatment fidelity to provide five domains of guidance for researchers within the broad field of health behaviour treatment (table 1).

Table 1

Comprehensive fidelity framework developed by the NIH BCC (synthesised from 1, 2, 3), with the ASK fidelity protocol mapped alongside these recommendations

Domain	Rationale	NIH BCC recommendations	Fidelity protocol within the ASK trial
Study design	Ensures the study adequately tests its hypotheses	Specify the theoretical model underlying the treatments and define the ‘active ingredients’ of the treatment	Theoretical underpinnings identified and ‘active ingredients’ proposed
		Conduct a pilot study to improve the treatment’s acceptability, feasibility and effectiveness	Pilot study conducted37
		Specify the treatment dose within and across conditions and ensure this is delivered	Treatment dose set a priori and non-adherence is recorded as a protocol deviation
		Plan how to monitor adherence to the protocol for therapy providers	Fidelity plan developed, as reported in the current study
		Plan how to record protocol deviations	Training provided to trial staff to identify protocol deviations, which are reported and recorded in REDCap
		Describe the treatment(s) in a standardised manual	Each treatment described in a standardised manual
		Plan how to manage setbacks, eg, attrition of providers	All sites/clusters receive initial face-to-face training. Subsequent training (eg, for new staff or booster training) is delivered via video conference
Training	Well-trained providers are more likely to follow the protocol and show increased competency	Therapy providers should a) have similar qualifications and experience, and b) ‘buy in’ to key aspects of the study (theory, randomisation, intervention)	All therapy providers are qualified speech pathologists Providers’ ‘buy in’ is not assessed
		Standardise training using the same trainers and materials but accommodate differences in learning styles	Training is centralised (provided by the trial manager) and standardised All training is provided by one of two trainers (trial managers)
		Measure skill acquisition and knowledge following training	Not assessed in the current protocol
		Develop a training plan to ensure providers maintain skills, eg, ongoing coaching and feedback, booster training sessions	Booster training is offered for providers to refresh their knowledge. This training can be accessed at any time at the request of the therapy provider
		Train the providers in study design and the methodology of the study, including preventing contamination across treatment arms	Training includes study design and methodology as well as role-specific information and skills Contamination across arms is reduced by the study design, ie, clusters of sites were randomised to one of two treatment arms
Treatment delivery	Ensures that providers deliver only the target treatment (treatment differentiation); maintain the required skills set (treatment competence) and administer the treatment as intended (treatment adherence) Assessment of fidelity of delivery will a) drive supervision to improve/maintain provider skills and b) can be used in analytical models to investigate the relationship between treatment fidelity and outcome	Develop the willingness and confidence of providers to report protocol deviations	This is not directly assessed within the ASK fidelity monitoring
		Monitor non-specific treatment effects, eg, perceived differences in providers’ warmth and credibility, participant expectations	Fidelity criteria requires therapy providers to demonstrate engagement, rapport and warmth, in line with their familiarity with the participant
		Reduce differences within the same treatment, and maximise differences between treatments	Audio-video recordings of interventions are rated by the fidelity monitor for presence/absence of essential and desirable behaviours Following review of the video-recorded intervention session, the fidelity monitor provides written and verbal feedback and coaching to therapy providers
		Ensure adherence to the treatment protocol including treatment content and prescribed dose	Therapy providers receive the fidelity criteria prior to administering their first session Booster training offered
		Reduce the risk of contamination between treatments	Therapy providers are trained in one trial intervention only, to reduce the risk of contamination
		Enhance adherence to the treatment protocol by audio or video recording treatment sessions	All treatment sessions are video recorded and a selection are reviewed by the fidelity monitor
Treatment receipt	Investigates whether the participant understood the treatment and can demonstrate knowledge of or application of the skills taught in the treatment	Ensure participants understand the information provided in the treatment	This is not directly assessed within the ASK fidelity monitoring. However, both treatments are delivered by qualified speech pathologists who should have skills in supporting people with aphasia to get their message in (ie, comprehension) and get their message out (eg, ask questions, reflect, explain)
		Ensure participants are able to apply the cognitive and behavioural skills taught in the treatment	This is not directly assessed within the ASK fidelity monitoring
		Build in strategies to improve performance of skills	The fidelity monitor provides specific, tailored feedback to the therapy provider on how to target behaviour change more explicitly. However, the current protocol does not directly assess participants’ performance of skills
Enactment	Monitors and supports participants to use treatment-related skills and strategies in real-life situations Acknowledges the difference between what is taught (treatment delivery), what is learnt (treatment receipt) and what is used (enactment)	Assess use and performance of cognitive and behavioural skills in an appropriate setting outside of the treatment	This is not directly assessed within the ASK fidelity monitoring. However, enactment is supported through goal-setting and ongoing reflect on achievement of goals. Furthermore, an essential criteria for the experimental intervention relates to setting homework tasks after each session and reporting on this in the following session

Brief background of fidelity monitoring in aphasia treatment studies

A review of 149 aphasia treatment studies published between 2002 and 2011 found only 14% of aphasia therapy studies reported treatment fidelity.5 A more recent review16 of 42 aphasia randomised controlled trials (RCTs) published since 2012 found that 21% reported on treatment fidelity processes and one article addressed all receommended treatment fidelity components.17

Given the relative scarcity of fidelity monitoring inaphasia treatment studies and in the related fields of psychological treatment,5 18 it was necessary to develop a novel fidelity protocol and tool for the ASK study. The NIH BCC fidelity framework was used to inform and develop a fidelity protocol for the ASK study (table 1).

The ASK trial

The ASK study is an ongoing Australian-based multicentre, cluster-randomised controlled trial with a target recruitment n of 344, funded by the National Health and Medical Research Council (for more information, see the trial protocol10). The experimental treatment targets psychosocial well-being and was developed in recognition of the high incidence of depression in this population, estimated to be 62%–70% and higher than in the general stroke population without aphasia.19 The nature of aphasia makes it difficult for individuals to access usual care psychological treatments, which are typically ‘talking therapies’.20 Speech pathologists are well-placed to provide treatment, but they report feeling uncertain about how to do this.21 The ASK trial trains speech pathologists to deliver either a psychosocial treatment (experimental treatment) or an information-focused treatment relating to secondary stroke prevention (attention control treatment).

Theoretical underpinnings of the ASK interventions

The ASK experimental treatment draws on previous research which explored what factors influence living successfully with chronic aphasia.22 23 Themes identified as being key to living successfully with aphasia included: engaging in meaningful activities; having sufficient social support and positive adaptive strategies. These themes served as a search strategy to identify relevant psychosocial interventions that could be adapted for the population of people with aphasia. That is, the psychosocial intervention literature was explored to identify which interventions mapped onto the themes previously identified as influential in living successfully with aphasia. Those psychosocial interventions with the highest available evidence were incorporated into modules within the ASK experimental intervention. Thus, the ASK experimental intervention is underpinned by theories of positive psychology24 social problem-solving,25 adult learning26 and the life participation approach to aphasia.27 28

The attention control treatment is a secondary stroke prevention information programme that draws on previous research with patients with stroke.29 30 The intervention is underpinned by theories of adult learning,31 the health belief model32 33 and self-efficacy theory.32 This specific intervention was chosen as an attention control as it can be provided in a similar dosage and format to the ASK intervention. Furthermore, the content of a secondary stroke prevention information programme is relevant to all patients with stroke. The materials were adapted in line with aphasia-friendly formatting principles.34 The provision of secondary stroke prevention information has had no known demonstrated effect on the primary outcomes used in the ASK trial.

A central component of both the experimental and attention control interventions are the specific approaches of Supported Conversation for Adults with Aphasia (SCA).35 SCA acknowledges the ‘success’ of a conversation is not wholly dependent on the severity or type of individual’s aphasia; rather, the behaviours of the communication partner can influence the success of the conversation. As such, SCA sets out specific strategies for communication partners (such as family and health professionals) to implement in conversation with the individual with aphasia. These strategies are implemented by those around the individual with aphasia to support message in (understanding), message out (expressing a thought or opinion) and checking that the family member or health professional understood what the individual with aphasia intended (clarification).

The need to develop a specific fidelity protocol for the ASK study

In usual clinical care, speech pathologists do not typically lead administration of either of these treatments adding to the importance of monitoring fidelity within the ASK trial. Fidelity criteria were developed for both the experimental and the attention control treatments, and the same level of monitoring and feedback occurs for both arms. The rationale for each aspect of the ASK fidelity protocol is described below. The study design and training components of the fidelity protocol are monitored separately to the treatment fidelity of sessions delivered by therapy procedures. Criteria for monitoring the fidelity of each treatment session were developed to determine whether treatment fidelity was achieved by each provider, and across participants (tables 2 and 3).

Within the current paper, Section 1 will describe the fidelity processes within the ASK trial and how these processes map onto the NIH BCC framework. The fidelity criteria used to monitor therapy delivery for both the experimental and attention control arms are included (tables 2 and 3). Section 2 will focus specifically on fidelity monitoring of therapy delivery. Here, we describe the procedure for reviewing intervention sessions, such as the procedure for therapy providers to submit audio-video recordings. Figure 1 provides a schematic of this procedure.

Figure 1

Fidelity assessment of the Action Success Knowledge trial experimental and attention control treatments.

Section 1

Overview of the ASK treatments

The ASK treatments consist of weekly face-to-face sessions (minimum of 3, maximum of 8 sessions) followed by monthly telephone reviews until the participant reaches 12 months poststroke. The treatment is delivered by a speech pathologist who has been trained in the trial protocol and the study treatment. The first face-to-face treatment session focuses on goal-setting, when the therapy provider works with the participant with aphasia and their family member to generate relevant goals. Goal development is supported by the Goal Attainment Scale36 and written in a way that is accessible for individuals with aphasia. After the goal-setting session, the therapist is able to use the goals to a) tailor the content of the treatment modules to the participants’ specific needs and b) help the participants to reflect on any perceived changes during the course of the treatment. Within the experimental treatment, homework tasks are agreed at the end of each session, with the aim of supporting the participant to enact the strategies outside of the treatment sessions. Homework tasks are not prescribed in the attention control intervention. The content of the intervention modules are subject to ongoing investigation within the trial; to reduce the risk of unblinding assessors and/or participants the content of treatment will not be discussed in detail in this paper. For further details regarding the protocol of the trial, see Worrall et al.10

Development of fidelity processes within the ASK trial

Study design

The experimental ASK treatment was piloted in a Phase I feasibility study of n=9 participants.37 This development work led to production of a standardised manual for the experimental treatment detailing the theoretical basis of the treatment; supporting information for each module and plans to monitor treatment fidelity, protocol deviations, data management and safety. Clusters (health service districts) provide one trial treatment to minimise potential for contamination between the treatments. Further information can be found in the protocol paper.10

Treatment dose was specified a priori including the length of each treatment session, the number of face-to-face sessions and telephone reviews, and maximum duration of the treatment time. Participants across the experimental and attention control arms receive the same dose and frequency of treatment. A minimum and maximum treatment dose was set to allow for individual preferences of the participants with aphasia (minimum of 3, maximum of 8 face-to-face weekly sessions). Following face-to-face treatment, participants receive monthly telephone reviews. Participants are recruited and complete baseline assessment within 6 months poststroke, and treatment ceases at the participant’s 12-month anniversary of their stroke. For all sessions (face-to-face and telephone), treatment dose within and between groups is monitored via data records submitted by therapy providers. Any changes in dose are recorded as a ‘protocol deviation’, for example, a missed session due to participant illness. While family members are also recruited as participants and are invited to attend the treatment sessions, the treatment dose relates specifically to the participants with aphasia, as they are the primary targets of the treatments.

Provider training

Standardised manuals and materials were developed for each role within the trial, that is, blinded assessor, recruiter and therapy provider. All roles are carried out by speech pathologists who have been trained by members of the trial management team, who are qualified speech pathologists. The assessor manual emphasises the theoretical underpinnings of each of the assessments, instructions on administration, skills required and data management and safety. Blinded assessors are usually contracted as casual employees nominated by the principal investigator at each site as suitable for this role.

Within both arms, the treatment manuals contain information on the theoretical underpinnings of the treatment as well as information on each module, data management and safety. Therapy is provided either by a private therapist employed as a casual employee of the trial, or by speech pathology staff at the participating clusters. In the case of the latter, suitable therapists are identified as those whose usual duties bring them into contact with patients with aphasia for screening, recruitment and administration of the study treatment.

Training for therapy providers across the trial arms is standardised, ensuring that all providers have access to the same content and amount of training. Training is carried out in person or via video conferencing with small groups of therapists, for the duration of 3 hours (for assessors or recruiters) or 6 hours (for therapy providers). Online refresher/booster training is available to all staff; this can be requested by the staff member at any time. Typically, staff access booster training following a period of leave. For therapy providers, the aim of the training programme is to ensure that staff understand the trial procedures, the rationale and theory behind the treatment, and how to tailor the treatment to suit the needs of an individual participant while not deviating from the essential components of the treatment.

Treatment delivery

Fidelity of treatment delivery is defined as the extent to which the therapist administers the treatment as planned and the competency with which the treatment is delivered.6–9 According to Bellg et al,2 assessment of treatment delivery includes monitoring and improving how therapy providers a) deliver only the target treatment (differentiation); b) acquire and maintain the required skills set (competency) and c) deliver the intended treatment components (adherence). ASK therapy providers receive a treatment manual which details the essential components of goal-setting as well as each of the treatment modules. Therapy providers are encouraged to tailor treatment to the individual needs of the participant, for example, through setting personalised, meaningful goals. In the experimental treatment, this tailoring is essential in order to encourage and support the participant to make changes in their behaviour, for example, shifting from potentially negative coping strategies to positive strategies. In the attention control treatment, tailoring ensures participants receive information that is relevant to them and at a pace that is acceptable.

Fidelity of treatment delivery is monitored using video recording of treatment sessions; recording is considered the ‘gold standard’ for monitoring treatment fidelity.2 All assessment and treatment sessions within the ASK trial are video recorded. Each therapy provider submits the audio-video recording of his/her first goal-setting session and first intervention module for assessment by the ASK fidelity monitor. This ensures that any deviation from the treatment is identified early by the fidelity monitor and steps taken to improve adherence to the target treatment. Providers’ behaviours are assessed in relation to content components and competency skills; see the ‘Procedure for monitoring and enhancing treatment delivery’ section. Due to their familiarity with the interventions and as they are unblinded as to the randomisation allocation of each site, the fidelity monitor role is carried out by one of two trial managers. As noted, the trial managers also delivered training to blinded assessors, recruiters and therapy providers within the trial.

Care is taken to engage the providers in the fidelity process and to be sensitive to any potential negative emotions they may feel when receiving feedback on their delivery of the study intervention. During the initial training session, the rationale for video recording treatment sessions is explained along with an outline of the process of fidelity checking including who will watch the video, how it will be assessed and what will happen if criteria are not met. This discussion highlights the importance of mutual respect, transparency and building trust. Therapy providers give their written consent to be video recorded. Before administering their first treatment sessions, therapy providers receive the treatment delivery fidelity criteria. This serves to further emphasise the essential components of the treatment and aims to reduce any provider anxiety relating to what the integrity monitor is assessing in the video recordings of treatment.

Treatment receipt

This category of fidelity focuses on whether the treatment was accurately ‘received’ by the participant (p. 8, 3). Borrelli points out that even if a treatment has been well-designed and competently delivered, the treatment will be ineffective if the participant is unable to understand or use the new skills. The target population of the ASK trial is people with aphasia and their family members. Aphasia can affect understanding, both through auditory and written modalities. Treatment receipt was not directly assessed; rather, it was embedded in the development of the treatment and materials, as well as provider training and treatment delivery. A number of steps were taken to support participants’ comprehension: the treatment materials were developed based on aphasia-friendly guidelines including simplified language, larger font, key words in bold, use of photographs/diagrams/pictures to support the written text and blank space.38 39 The treatment materials were developed by the research team before any participant commenced treatment. Essential criteria in the treatment fidelity checklist (tables 2 and 3) relate to the provider using techniques to support the participant with aphasia to understand (message in) and techniques to support the participant to ask questions, or express their thoughts or feelings, or to demonstrate their interpretation of the conversation (message out). These techniques draw on an evidence base of supported conversation,35 and are essential to ensure that the treatment is being delivered in an accessible way to participants with aphasia.

Enactment of treatment skills

While Bellg et al 2 acknowledge that treatment enactment is difficult both to conceptualise and to implement, they argue there is an important distinction between ‘what is taught (treatment delivery), what is learnt (treatment receipt) and what is actually used (enactment)’ (p. 450). Within the ASK study, enactment of treatment skills is not directly assessed or monitored but is supported in various ways.

For both the experimental and attention control interventions, the therapy provider and participants work together to generate person-specific, meaningful goals. Regular goal review throughout the intervention means that participants can report on how they are progressing and/or update their goals as needed. Goal-setting is assessed by the fidelity monitor and feedback given regarding the appropriateness of goals; goal achievement is not assessed within the fidelity monitoring process.

For both the experimental and attention control interventions, enactment is supported via follow-up telephone sessions for both arms. After attending face-to-face sessions with the therapy provider, participants receive monthly telephone calls from the therapy provider until the 12-month anniversary of their stroke. The purpose of these calls is to reflect on progress related to the participant’s goals, check if the participant requires more information or discussion in relation to these and to explore any obstacles to enacting the skills taught in treatment. The telephone sessions are audio recorded and clinicians submit data on the nature of the discussion and length of the call.

For the experimental intervention only, it is an essential fidelity criterion that therapy providers set homework with each participant. The homework task is set at the nd of each session, reflecting the discussion that took place in the session. The homework task builds on the most recent therapy session and/or generates opportunities for the participant to practise a skill outside of the clinic. In the following session, the therapy provider reviews the homework with the participant. Reasons for non-completion of homework are explored, as these can indicate a lack of understanding or obstacles to implementing the learnt skills. The therapy provider’s behaviours (ie, setting and reviewing homework tasks) are monitored within the fidelity review; the participant’s completion of the tasks as directed is not directly assessed in the fidelity process.

Section 2

Procedure for monitoring and enhancing treatment delivery

All assessment and face-to-face treatment sessions are video recorded. Each therapy provider submits his/her first goal-setting session and first module for assessment by the ASK fidelity monitor. Dependent on resources within the trial, later sessions are selected at random for fidelity checking to ensure skills have been maintained. In particular, assessing the first goal-setting session and the first module ensures that any deviation from the treatment is identified early by the fidelity monitor and steps are taken to improve adherence to the target treatment. The fidelity monitor watches 100% of the recordings of the goal-setting and first modules; this allows the fidelity monitor to get a broad sense of the session, rather than simply focusing on the frequency of specific behaviours.

Providers’ behaviours are assessed in relation to content components and competency skills, and given a score based on whether these behaviours were evident in the recording (see table 2 for the fidelity criteria relating to the goal-setting session, and table 3 for criteria relating to the intervention modules). Once fidelity assessment is complete, the fidelity monitor provides written and verbal feedback to the therapy provider regarding whether they met fidelity criteria and on specific aspects of the sessions. If fidelity criteria are met, the therapy provider continues to video record all subsequent treatment sessions. If the therapy provider does not meet fidelity criteria for the first goal-setting or module sessions, he/she is provided with written and verbal feedback and coaching. Subsequently, the therapy provider is required to submit his/her next session to ensure that the necessary changes have been incorporated. This cycle of submitting a video recording and receiving feedback and coaching continues until the therapy provider meets the fidelity criteria.

Table 2

Fidelity criteria for treatment delivery within the goal-setting session for the experimental treatment and the attention control treatment

Active ingredient	Behaviour	Fidelity component	Experimental treatment	Attention control treatment
Principles of supported conversation	Used appropriate level of communication strategies to support the person with aphasia’s (PWA) receptive language skills For example, adapted resources, writing, gesture, drawing, repetition	Competence	Essential criteria	Essential criteria
Principles of supported conversation	Used appropriate level of communication strategies to support the PWA’s expressive language skills For example, suggested using drawing, writing, pointing, gesture, verification	Competence	Essential criteria	Essential criteria
Collaborative learning Person-centred approach	Did not use preset goals without discussion with the PWA (and family member)	Content	Essential criteria	Essential criteria
Person-centred approach Goal-oriented approach	The therapist delivered only the target treatment as described in the manual and in training sessions	Content	Essential criteria	Essential criteria
Person-centred approach Goal-oriented approach	Gave a rationale for goal-setting	Content	Essential criteria	Essential criteria
Person-centred approach Goal-oriented approach Collaborative learning	Problem solved with the PWA (and family member) to set personally relevant goals For example, guided discussion about the goals and options within the treatment; explained why specific goals do not fall under the study treatment and how to address these concerns; helped PWA (and family member) gain new understanding of problems/goals; helped PWA (and family member) to prioritise goals	Content	Essential criteria	Essential criteria
Collaborative learning	Demonstrated an appropriate level of engagement, rapport and warmth, in line with their familiarity with the PWA (and family member) For example, warm tone of voice; appropriate use of humour; avoidance of criticism; encouragement of communication attempts	Competence	Desirable criteria	Desirable criteria
Collaborative learning	Evidence of therapeutic alliance For example, checked that the goals reflect the PWA’s (and family member’s) needs; avoided dominating the discussion; used active listening; adapted approach to engage the PWA (and family member)	Competence	Desirable criteria	Desirable criteria
			Total score (max 8, min 6.4) Essential behaviours score (min 6)

Table 3

Fidelity criteria for treatment delivery within the treatment modules

Active ingredient	Behaviour	Fidelity component	Experimental treatment	Attention control treatment
Principles of supported conversation	Used appropriate level of communication strategies to support the person with aphasia’s (PWA) receptive language skills For example, adapted resources, writing, gesture, drawing, repetition	Competence	Essential criteria	Essential criteria
Principles of supported conversation	Used appropriate level of communication strategies to support the PWA’s expressive language skills For example, suggested using drawing, writing, pointing, gesture, verification	Competence	Essential criteria	Essential criteria
Behaviour change Person-centred	The therapist delivered only the target treatment as described in the manual and in training sessions For example, spent the majority of the session discussing topics related to the study treatment; for queries/discussions not related to the study treatment, therapist dealt appropriately or sign-posted to an appropriate source of information or advice	Content	Essential criteria	Essential criteria
Collaborative learning	Explained the rationale for the module For example, how the treatment module relates to the participant’s (and family member’s) goals	Content	Desirable criteria	Desirable criteria
Collaborative learning	Demonstrated an appropriate level of engagement, rapport and warmth, in line with their familiarity with the PWA (and family member) For example, warm tone of voice; appropriate use of humour; avoidance of criticism; encouragement of communication attempts; identified what information the PWA (and family member) already had access to in relation to the study treatment and the gaps in information/understanding	Competence	Desirable criteria	Desirable criteria
Collaborative learning	Evidence of therapeutic alliance For example, seeks agreement; avoid dominating the discussion; active listening; adapts approach to engage the PWA (and family member)	Competence	Desirable criteria	Desirable criteria
Behaviour change	Reviewed homework from the previous module and explored reasons for non-completion of homework—experimental treatment only	Content	Desirable criteria	Not targeted
Behaviour change Person-centred	Targeted behavioural change—experimental treatment only For example, asked questions to gain a better understanding of the PWA (and family member); asked questions to prompt the PWA (and family member) to explore the topic or to make it personally relevant to them; used the written materials as a platform to generate further discussion; discussed challenges to achieving the goal for the specific module and worked with the participant to problem-solve; prompted the PWA (and family member) to keep a written note of any activities they wanted to action as ‘homework’	Competence	Essential criteria	Not targeted
Behaviour change Person-centred approach	Effectively engaged the PWA (and family member) in the practical activities—experimental treatment only For example, gave accurate feedback on practical activities; agreed an appropriate homework	Competence	Desirable criteria	Not targeted
		Experimental treatment	Total score (max 9, min 7.2) Essential behaviours score (min 4)
		Attention control treatment	Total score (max 6, min 4.8) Essential behaviours score (min 3)

Development of a tool to assess treatment delivery fidelity within the ASK study

The treatment fidelity criteria were developed from the proposed active ingredients of each treatment module. These criteria reflect the content components and the competency skills necessary to deliver the target treatment (see table 2 for fidelity criteria relating to the goal-setting session; see table 3 for fidelity criteria relating to the treatment modules). Inter-rater reliability is currently being investigated as the trial continues to collect data.

Adherence to content components

The key content components of the treatment were identified for each module (goal-setting module and the therapy modules) based on the underlying theory and evidence.

Adherence to competency skills

Competency skills were defined as the communication strategies used by the provider to support the participant with aphasia to understand the information and to express their thoughts, feelings and questions. Identification of these skills was informed by the extensive literature on supported communication, that is, there is strong evidence to suggest the skill of the communication partner can have a substantial impact on the communication ability of the person with aphasia.40 41

Overall marker of quality of the treatment session

All behaviours (content and competency) are categorised as either ‘essential’ or ‘desirable’. To demonstrate fidelity to the intervention protocol, therapy providers must demonstrate 100% of all essential behaviours. Additionally, providers must also demonstrate a minimum number of desirable behaviours (ie, considered high quality but not essential). Feedback on all behaviours is provided to the therapy providers. Evaluation of adherence to treatment content and competency components involves scoring occurrences of behaviours; a binary system of ‘present’ (1 point) or ‘absent’ (0 points) is applied and the scores counted. Scores for essential and desirable behaviours are combined to give a total maximum score for each session; a higher score is interpreted as a marker of quality of the treatment session. The threshold is set at 100% adherence to essential criteria and an overall minimum score of 80% adherence, that is, therapists need to demonstrate at least 80% fidelity to the protocol to be considered adherent. A score below the minimum results in the conclusion that the therapist has not met the treatment fidelity criteria for that session.

Conclusion

Within health-related behavioural treatment studies, it is crucial that treatment fidelity is monitored, enhanced and reported in order to increase the power to detect treatment effects and to increase confidence in the study’s findings. Monitoring treatment fidelity is still relatively rare in the field of aphasia treatment studies, with a previous review indicating that 14% of aphasia studies reported fidelity5, with a more recent review indicating that 21% of aphasia RCTs reported fidelity. The fidelity protocol and checklist developed within the ASK trial provides a useful template for other aphasia and psychosocial treatment studies. One limitation of the ASK fidelity protocol and checklist is that it primarily focuses on how the provider administers treatment to participants with aphasia, as they are the main focus for change. While family members are recruited as participants and attend intervention sessions, their participation is not included in the current fidelity protocol. Yet, their support is likely to be influential in how the individuals with aphasia receive the treatment and embed the intervention into everyday life. Future studies of intervention involving participants with aphasia and family members could consider adding specific fidelity criteria that relates to the family member. A further limitation is that, while treatment receipt and enactment are embedded within the design of the ASK intervention (ie, accessible aphasia-friendly materials; personalised goal-setting; follow-up telephone reviews), there is no direct assessment of treatment receipt or enactment. In particular, treatment enactment is challenging to directly assess and may rely on self-report from study participants. Future aphasia studies could directly assess treatment receipt and enactment, for example, review whether participants felt they had achieved their goals or collect participant self-reports on whether the strategies learnt in therapy had become embedded into their daily life.

Patient/ and/or public involvement

Patients and/or public were not involved in the development of the ASK fidelity tool.

Ethics and dissemination

The monitoring and ongoing conduct of the research project is in line with requirements under the National Mutual Acceptance with the submission of progress reports, safety and/or adverse event reports and amendments. On completion of the trial, findings from the study and from the fidelity reviews will be disseminated via publications and conference presentations.