Process evaluations in neurological rehabilitation: a mixed-evidence systematic review and recommendations for future research.

OBJECTIVE: To systematically review how process evaluations are currently designed, what methodologies are used and how are they developed alongside or within neurological rehabilitation trials.
METHODS: This mixed-methods systematic review had two evidence streams: stream I, studies reporting process evaluations alongside neurorehabilitation trials research and stream II, methodological guidance on process evaluation design and methodology. A search strategy was designed for each evidence stream. Data regarding process evaluation core concepts and design issues were extracted using a bespoke template. Evidence from both streams was analysed separately and then synthesised in a final overarching synthesis proposing a number of recommendations for future research.
RESULTS: A total of 124 process evaluation studies, reporting on 106 interventions, were included in stream I evidence. 30 studies were included as stream II evidence. Synthesis 1 produced 9 themes, and synthesis 2 identified a total of 8 recommendations for process evaluation research. The overall synthesis resulted in 57 'synthesis recommendations' about process evaluation methodology grouped into 9 research areas, including the use of theory, the investigation of context, intervention staff characteristics and the delivery of the trial intervention.
CONCLUSIONS: There remains no consensus regarding process evaluation terminology within the neurological rehabilitation field. There is a need for process evaluations to address the nature and influence of context over time. Process evaluations should clearly describe what intervention staff bring to a trial, including skills and experience prior to joining the research. Process evaluations should monitor intervention staff's learning effects and the possible impact that these may have on trial outcomes. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

This paper presents the first systematic review that applies a two-stream mixed-evidence synthesis to investigate process evaluations in neurological rehabilitation research.

This review used a rigorous and broad search strategy including a wide range of sources to maximise the capture of relevant literature.

There is a possibility that relevant studies were not identified due to the fact that the term ‘process evaluation’ was not considered a Medical Subject Heading in any of the databases that were searched.

This review does not critically appraise the quality of included process evaluation studies.

Introduction

Although the number of randomised controlled trials (RCTs) of rehabilitation interventions has risen dramatically in recent years, rehabilitation research lags behind other sciences in providing conclusive evidence of its beneficial impacts. As a consequence, the development of innovative interventions and programmes is being slowed down.1 As a ‘broad-based discipline’, the impacts of rehabilitation interventions are difficult to evaluate, and therefore its multidisciplinary nature needs to be addressed when designing research studies.2 Addressing this challenge relies in working towards defining, in detail, rehabilitation treatments in terms of what are their ‘active ingredients’, what is their individual impact and what is the impact of the intervention as a whole.3 Furthermore, rehabilitation is context specific and often defined as the interaction between the individual and the environment.4 Thus, identifying contextual factors (physical, psychological, social, etc) and acknowledging that researchers bring their own personal values into situations is of great importance when thinking about the science of rehabilitation.

A number of models to assist with the development of complex interventions and improve their quality have been published. The UK Medical Research Council (MRC)5 has proposed an approach to the evaluation of complex interventions which includes developing theory-based explanations of how interventions work. This framework has already been used in a number of neurological rehabilitation research projects.6–8 While it describes five phases for intervention development which the research process should follow, this framework does not provide details as to which research methods should be used. The MRC5 framework highlights that a process evaluation, including qualitative data gathering methods, can provide insights into why an intervention fails unexpectedly, has unanticipated consequences or why a successful intervention works and how it can be optimised.9 In 2014, the MRC published the first guidance for carrying out process evaluations in health research.10 This guidance summarises why there is a need for process evaluations alongside current health research, and it then proposes a framework which is highly informed by the MRC guidance on complex interventions.5 It is widely accepted that process evaluations serve a very important role in health services research, not only when checking whether the trial intervention was performed as planned but also in providing detailed insight into the experiences of those exposed to the intervention.9 11 By evaluating processes, an intervention can be improved either during its application, or afterwards, at an implementation stage.12 Finally, trials which include a process evaluation will produce higher quality results that can help clarify the potential generalisability and optimisation of the proposed intervention in routine practice.13 However, to date, process evaluations alongside trials are scarce, and they are even rarer in multidisciplinary therapy research on neurological rehabilitation.

The aim of a process evaluation is to investigate how and why an intervention fails or succeeds at producing the desired outcomes.14 In recent years, there has been an increase in published research on theories and frameworks driving process evaluations for complex interventions. Steckler and Linnan11 proposed a framework for carrying out process evaluations which included a series of programme components that should be measured and evaluated: recruitment (how were participants attracted into the study), context (social, political and environmental factors that could have influenced implementation), reach (proportion of targeted patients that participated in the intervention), dose delivered (proportion of the intervention components that was provided), dose received (level of participant's engagement in the intervention), fidelity (to what extent was the intervention delivered as had been intended by the researchers) and implementation. In order to measure these components, process evaluations may make use of qualitative and quantitative methods.15 However, there is still very limited guidance to help researchers design process evaluations;16 as a consequence, researchers can find the prospect of carrying out a process evaluation daunting and this could lead them to discard the idea of embedding one alongside their proposed trials, especially when looking at complex interventions.

To the best of our knowledge, this constitutes the first systematic review that applies a two-stream mixed-evidence synthesis to investigate the design of process evaluations in neurological rehabilitation research. The overarching question of this systematic review was: How are process evaluations currently designed, what methodologies are used and how are they developed alongside or within neurological rehabilitation research?

Specific research questions were:

What methodologies and methods have been used to carry out process evaluations when undertaken alongside neurological rehabilitation research?

What are the theoretical underpinnings (if any) of process evaluations alongside neurological rehabilitation research trials?

How have the results from process evaluations alongside neurological rehabilitation research trials been used to understand and clarify trial results?

What are the potential barriers and facilitators to carrying out process evaluations alongside neurological rehabilitation research?

What terminology is currently being used in process evaluation research?

Methods

Design—mixed-evidence synthesis

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) was followed in this review. The University of York Centre for Research and Dissemination (York CRD) guidelines were followed at the time of conducting searches and extracting data.17 The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-centre) published methods for conducting systematic reviews18 was used to guide the synthesis of mixed-evidence findings.

Following the CRD's guidance for undertaking reviews in healthcare,17 an initial review of published literature was carried out in order to identify key terms and types of studies reporting on process evaluations. This review identified a variety of quantitative, qualitative and methodological literature. Thus, the decision was made to segregate studies into two different streams of evidence. Studies included in each evidence stream would be analysed and synthesised separately, and following from this, results from each evidence stream would be ‘bridged together’.19 20 The following streams of evidence were agreed. Details on each evidence stream and inclusion criteria are reported in table 1.

Table 1

Evidence streams and inclusion criteria

	Aim and description	Inclusion criteria
Stream I (research evidence)	To identify how process evaluations have been carried out alongside/linked to neurological research trials.Provide data to answer the specific objectives: identifying terminology used in process evaluations, how results of process evaluations are used to understand the trials overall outcomes and what particular methods are mostly used by researchers.	Qualitative and quantitative primary research studies. All study types Reporting on process evaluations linked to neurological rehabilitation research trials conducted around the world. Reporting on one or more process evaluation components.11 (Papers that only reported on impact evaluation were excluded)
Stream II (methodological evidence)	Research studies which were not necessarily primary research.Studies that would provide rich data to help answer methodological and theory-related research questions and aimed to explore frameworks and theory behind process evaluations.	Studies and reports, published in the English language. Exploring methodologies, guidance and opinions regarding process evaluation research methods. Studies in health research reporting on frameworks for implementation fidelity and its components.

Search methods

Stream I searches

Relevant articles to be included in stream I were identified through conducting electronic database searches. All search details (date, search strategy, hits) were recorded in a ‘search log’. All search results were managed using RefWorks. The following databases were searched: CINAHL, Web of Science, MEDLINE, PEDro, SSCI, PsycINFO, ClinicalTrials.gov, HTA Database, Cochrane Central, EPPI-Centre Database, ASSIA and DORIS. The data searches were carried out using search terms and keywords that were a mixture of Medical Subject Headings (MeSH) and non-MeSH terms. Experts in the field were contacted, and manual reference list checks (citation tracking) and hand-searched key journals and trials register websites as well as other specialist websites and internet search engines were carried out.

In consultation with an experienced librarian and informed by three published systematic reviews on process evaluation21–23 searches for stream I were carried out combining three different search arms: ‘process evaluation’, ‘neurological conditions’ and ‘rehabilitation’ (a detailed search strategy is shown in online supplementary table S1). Searches were restricted to studies published in English, between 1977 and February 2015.

Stream II searches

Relevant literature was identified through the use of search engines, such as Google Scholar, and snowball sampling. Thorough manual reference checking of studies included in stream I was carried out. Specialist websites were searched, and experts were contacted in order to assure that relevant studies were not left out. Searches were not limited to the UK; international guidelines and methodological resources were included when considered relevant to the review.

Screening of results

The screening of the results was carried out following PRISMA guidelines, which were adapted for each of the evidence streams. In order to reduce bias, a second researcher joined the screening process and reviewed each study independently. When necessary, agreement on inclusion was reached through discussion.

Data extraction

There are currently no established methods for assessing the quality of process evaluations partly due to the wide variation in their reporting.16 Thus, it was decided not to formally review the methodological quality of the process evaluations. The data extraction process was informed by three published systematic reviews on process evaluations21–23 and a systematic review which included a set of studies that were process evaluations.24 Basic information (publication year, author, title, discipline, study type) was collected from all stream I studies. A bespoke template for data extraction was prepared including components identified in published frameworks on process evaluation such as the one proposed by Steckler and Linnan11 (table 2).

Table 2

Data extraction criteria for stream I evidence

Data extraction criteria

Basic information: author, publication year, title, discipline, study type, design. Recruitment procedures, dose delivered, participant attitudes investigated, adherence/fidelity measured, implementation monitoring, context considered, intervention protocol, aims and objectives, PE study design and rationale, mechanisms to assess adherence to intervention protocol, description of intervention providers, training of intervention providers, learning over time measured, theory informing PE, PE findings, links between PE and outcome results.

In the case of stream II studies, all basic information was first extracted on type of publication, topic, aims, objectives and target audience. Information was collected on how stream II studies had addressed the following aspects (or a number of them) of process evaluation research: theory development, context, recruitment, staff characteristics, staff training, learning over time and adherence to protocol.

As recommended by the CRD,17 data extraction forms were piloted on a sample of 10 included studies in order to guarantee that the relevant information was captured, and resources were not being wasted on extracting irrelevant data. Furthermore, a quality check on data extraction from a random 10% sample of studies included in stream I and stream II evidence was undertaken by two reviewers.

Synthesis of extracted data

Descriptive statistics were used in order to map studies included in both streams of evidence. A separate synthesis was carried out by evidence type.

Synthesis 1

Data extracted from studies included in evidence stream I was analysed and themes were identified using a modified method described by Kavanagh et al.24 Results from this analysis were presented in the form of overarching narrative descriptive themes which were inductively generated.25 26 For quality assurance, initial patterns and subsequent themes were independently reviewed by another member of the research team. Emerging themes were then opened to debate, and disagreements among researchers were solved via discussion.

Synthesis 2

Thomas and Harden's27 method for carrying out thematic qualitative synthesis on primary qualitative research was adapted, as required, in order to carry out the synthesis. The following steps were followed:

Finding ‘descriptive themes’: the researcher read each of the sections which had been extracted during the data extraction phase and coded sections according to its purpose and content. As the coding progressed, a ‘bank of codes’ was generated and new ones created as necessary. The researcher followed an inductive process and looked at similarities between codes in order to group the initial codes. This process resulted in a number of descriptive themes (figure 1 is an example of how one of the descriptive themes, intervention staff factors, was generated).

Figure 1

Example of grouping of initial codes to form broader descriptive themes.

The generation of ‘analytical themes’ was performed by using the descriptive themes that emerged from the inductive analysis to answer the systematic review question. The lead author did this independently, and then, through discussion with the rest of the members of the team, more analytical (abstract) themes began to emerge. These final analytical themes constituted a ‘list of recommendations’ regarding the undertaking of different aspects of process evaluation research.

Synthesis 3

The method described by Harden et al19 and Oliver et al20 was used to structure, compare and bring together findings from both research evidence streams. The themes that emerged from synthesis 1 were then mapped on to the recommendations identified in synthesis 2. As a result of this, potential gaps and strengths defining neurological rehabilitation process evaluation research were identified (figure 2). Synthesis 3 is presented as a list of ‘synthesis recommendations’ on how to best carry out and report process evaluations in neurological rehabilitation research.

Figure 2

Summary of the synthesising process.

Results

Included studies

A total of 3327 studies were found for stream I searches. A total of 1380 duplicates were removed, and after screening titles and abstracts, 1673 were excluded on the basis of not meeting inclusion criteria. The full text of the remaining 274 studies was screened and on complete reading, 124 studies were included (figure 3). The main reason for exclusion at this stage was studies not including any reference to one or more process evaluation components. Some researchers published a separate process evaluation result paper in addition to an outcome evaluation paper and a process evaluation protocol paper; articles describing one intervention were combined and considered as one unit for the purpose of this review. After this grouping, a total of 124 studies reporting on 106 interventions remained for analysis (see online supplementary table S2).

Figure 3

Flow diagram of search strategies and review process based on PRISMA statement. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analysis.

Search strategies for stream II led to a total of 45 studies. Full text of these was screened, and finally 30 studies were deemed eligible for inclusion (see online supplementary table S3).

Synthesis 1—research evidence

The majority of studies (89%) included in stream I were published between 2001 and 2014. Only four and eight studies were published during the 1990s and the year 2000, respectively. In terms of rehabilitation disciplines, 33 (31%) interventions were multidisciplinary, 20 (19%) involved occupational therapy, 21 (20%) were physiotherapy, 9 (8%) were psychological interventions and 4 (4%) were speech and language therapy interventions. A total of 14 (13%) were interventions involving alternative forms of exercise or therapy (eg, yoga, Tai chi, treadmill training).

In terms of research design, half of interventions (50%) were investigated using randomised trial methods, including pilot, multicentre and cluster RCTs. The remaining studies used a range of approaches to investigate interventions, including pre-post one group design, repeated measure three group design and two group non-randomised design. Out of the 124, 10 studies were purely qualitative research studies.

All 106 interventions described in the 124 studies were being investigated for to their effectiveness in treating a range of neurological conditions, such as stroke (28%), Parkinson's disease (PD) (10%), multiple sclerosis (MS) (8%) and traumatic brain injury (TBI) (7%). A total of 24 interventions (23%) were targeting cognitive impairments including Alzheimer's disease and dementia.

Following the method previously described, a total of 10 themes were developed (table 3):

Table 3

Synthesis 1: emergent themes

Emergent themes
Terminology	Training and assessing intervention staff competence in delivering the intervention
Aims and strategies	Tailoring and adherence to intervention protocols
Addressing context	Investigating participants' opinions
Recruitment	A tool to identify barriers and facilitators to implementation
Describing those in charge of delivering the trialled intervention	Linking trial outcomes to process evaluation findings

Terminology

The term ‘process evaluation’ is yet to be widely used to describe the assessment and evaluation that takes place at the time of carrying out research on the implementation of a new or innovative intervention. Only 32 studies of those included in stream I used the term ‘process evaluation’.

More specifically, among all studies included in this review there was a clear lack of consensus regarding terminology used to describe the processes that were being evaluated and their components; this led to confusion and lack of clarity. The term ‘adherence’ was a clear example of this since it was variously referred to as ‘dose’, ‘attendance rate’, ‘compliance’, ‘fidelity’ or ‘exposure’. Neurological rehabilitation studies failed to define terms in a unique and non-interchangeable manner. The term feasibility was widely used; however, it was defined differently across studies. McGinley et al28 in their pilot RCT study looking at the impact of a physical therapy intervention for patients with PD defined it in terms of safety, retention, adherence and compliance measures. Stephens et al29 in their pilot RCT investigating the potential benefits of an exercise intervention for children with fibromyalgia defined it as comprising adherence and recruitment data.

Aims and strategies

A total of 50 studies (reporting on 43 interventions) among those included in stream I identified aims and research questions specific to the process evaluation. Forty-seven studies reported on strategies in place, at times referred to as ‘feasibility’ (12 and 42) or ‘fidelity’ outcomes (67 and 89) to answer those research questions. However, most studies provided a very broad description of these strategies, without much detail. Two studies provided details on specific tools used to investigate a process evaluation component; Alwin et al30 31 described the Patient Perspective on Care and Rehabilitation Process (POCR) instrument to investigate the significance of an assistive technology intervention for the relatives of people with dementia. Khalil et al32 33 used the Intrinsic Motivation Inventory (IMI) to assess how individuals with Huntington's disease perceived (eg, enjoyment, value, usefulness) the activities included in a home-based exercise intervention.

Addressing context

The context in which the neurological rehabilitation intervention implementation and research trial took place was described for 49 (45%) of the included interventions; however, this description often lacked detail and was focused solely on information regarding the trial's setting (eg, 1, 9, 57, 64) without accounting for the wider physical, social, economic, organisation and political environment. Only 11 studies (9%) described in detail the provision (with respect to the trialled intervention) that already existed, and crucially how these contextual factors had changed over time. Out of the 11, 5 studies (16, 17, 18, 93 and 108) described the strategies in place to explore how changes in contextual factors had possibly affected the implementation and/or impacts.

Recruitment

All trials investigating the 106 included interventions described the main trial's participant recruitment procedures, including chosen sites, participants' inclusion/exclusion criteria and diagrams describing the flow of participants throughout the research process; only 21 of them (17%) provided a detailed description of identified barriers and facilitators to initial recruitment and consequent retention and engagement of individuals during the running of the neurological rehabilitation trials. Scianni et al34 provided a detailed account of recruitment procedures and identified lack of transport as the main barrier to patient participation in a gait training study. In relation to recruitment of participants for the process evaluation, 15 studies provided an explanation of what subsample was selected and why.

Describing those in charge of delivering the trialled intervention

A total of 29 studies (comprising 23 interventions) (22%) investigated what were the intervention providers' motivations for joining the research programme or what were their perceptions regarding treatment effects and possible impacts. This was mostly achieved by carrying out in-depth interviews and administering questionnaires (including ranking items and open questions) and less often through focus groups (five studies). Scobbie et al35 carried out a process evaluation to evaluate the implementation of a theory-based action planning framework (G-AP) to guide goal setting practice. In their study, interviews were chosen as a data collection method in order to investigate the experiences of intervention providers and the difficulties they faced. The study by Döpp et al36 is a clear example of the use of a variety of methods in order to achieve this; the authors used web-based questionnaires which included a number of statements regarding barriers to implementation which the occupational therapists had to rate in terms of how much they agreed. They carried out further focus groups and semistructured telephone interviews.

Of those studies included in stream I, only those looking at 10 (9%) of the interventions provided details in regard to the level of experience of intervention staff. A number of studies referred to health professionals experience using expressions such as ‘experienced health professionals’ (51 and 95), ‘multiple years of clinical experience’ (58) or ‘had prior experience’ (75) without providing any further detail. Some studies reported on the years of experience that health professionals had (eg, 28, 32, 49, 60, 71, 100, 103, 107, 108 and 110), the grade of intervention staff (17) or level of education (58 and 101).

Training and assessing intervention staff competence in delivering the intervention

Out of the 106 interventions included in stream I studies, 40 (38%) were delivered by staff who had attended training workshops regarding the neurological rehabilitation intervention, prior to the start of the trial. This training was delivered using a variety of methods such as lectures, role play (14 and 107), practical sessions and group discussions, and it varied in length from 2 hours (58 and 81), 2.5 hours (60), 4 hours (92), 16 hours (107), 40 hours (89) to ½ day (57), 2 days (45), 3 days (23 and 93) and 5 days (50). A study looking at the impact of a structured training programme for caregivers of inpatients after stroke (TRACS study) (16, 17 and 18) relied on ‘cascade training’ as a method of transferring knowledge from one health professional to another.

King et al37 in their feasibility study evaluating the effects of paediatric therapy services in the school setting and Voigt-Radloff et al38 39 in the process evaluation of the RCT looking at the impact of evidence-based community on dementia patients (WHEDA study), both provided a detailed description about the training and expected learning outcomes of staff. However, among the studies that included a training component, only seven defined performance criteria and measured (mainly via observations) the skill acquisition post-training to a minimum standard that would allow the provider to be involved in the delivery of the intervention. Chung40 and Döpp et al36 used quizzes and questionnaires, respectively, to assess intervention staff's knowledge on the trialled dementia intervention. In Morris et al,41 all personnel who had undergone training were required to achieve a score equal to, or greater than, 90% of items correctly executed. Failure to meet this criterion required the tester or trainer to withdraw from the project and to resubmit standardisation videotapes for rating until the 90% or higher criterion was achieved.

Only 11 studies (reporting on 8 interventions) (16, 17, 18, 25, 28, 37, 39, 42, 45, 81 and 92) reported on methods in place to regularly ‘refresh’ intervention staff knowledge on the neurological rehabilitation intervention (eg, due to staff turnover throughout the research process). In the TRACS study (16, 17 and 18), local training sessions were arranged, if necessary, to provide feedback and support. Additionally all centres involved in the trial were offered a local refresher course midway through the trial. A total of 22 studies (reporting on 17 interventions, 16%) described methods in place to maintain intervention staff's skills over time. This was achieved mainly via individual or group supervision (set or available when necessary), led by an ‘expert’ in the field or an advisory group (87) and delivered via a number of ways: meetings, telephone conversations, emails or blogs, among others.

Only 16 studies (regarding a total of 8 interventions) (7, 8, 21, 22, 23, 49, 52, 53, 54, 55, 63, 70, 93, 94, 122 and 123) discussed changes on how intervention staff delivered the neurological rehabilitation intervention over time. However, this was not described in detail. Morris et al,41 in the process evaluation of an RCT looking at an extremity constraint induced therapy intervention, and Østensjø et al,42 in a trial of a goal setting rehabilitation programme, discussed how health professionals improved their standard performance over time. Although these studies referred to learning over time, none of them acknowledged learning curve effects at the time of evaluation.

Tailoring and adherence to intervention protocols

A total of 93 studies (comprising 79 interventions) (74%) reported having protocols/manuals which provided a description of the intervention with a varied level of detail. However, 42 of included studies (34%) also discussed how the intervention remained flexible throughout the research period in order to be tailored to the needs of individual patients. Studies included in this review generally failed to investigate and report whether the research team had reached a consensus regarding standardisation of the intervention, or if health professionals were provided with a brief rationale to help them assess which was the ‘right’ amount of tailoring that should take place. Mayo et al43 and McGinley et al28 describe two different exercise programmes for patients with stroke and PD, respectively. Both studies explain how exercise programmes were tailored to individual needs while remaining within the limits established by the study protocol. Although none of the studies in stream I discusses whether intervention staff were provided with specific guidance and advice on tailoring, 78 of studies (63%) which described a protocol reported results and described a variety of strategies in place to monitor and measure adherence to research protocols. Adherence was assessed via a number of methods: reviews by experts of audiotaped/videotaped sessions of intervention staff with participants (4, 7, 8, 49, 58, 62, 63, 89 and 110), life observations (7, 8, 16, 17, 18, 38, 65, 67, 74, 100, 101, 119, 120, 121, 122 and 123), treatment log books, intervention staff reflexive accounts, therapy evaluation forms, diaries, field notes/case notes and various standardised scales and checklists for fidelity.

Investigating participants' opinion

A total of 84 studies (77 interventions) reported the experience, motivations and opinions of those exposed to the neurological rehabilitation intervention. Out of these, 44 gathered this information via evaluation questionnaires that included either itemised scaled questions, open-ended questions or both. A total of 33 studies used in-depth interviews, and 7 studies carried out focus groups among other methods.

Of all the studies that investigated patient's opinions, 29 described the use of enquiry tools and methods specifically designed to investigate whether participants understood the intervention (eg, goals, outcome measures, rationale) or not. Leuty et al44 and Macht et al45 assessed understanding by adding related questions to the participant opinions questionnaire. Others such as Li et al46 enquired about the level of understanding during an ‘exit interview’. Observations of intervention implementation sessions were also reported as ways of assessing participants' understanding. This was the case for Taylor et al47 who investigated the impact of home-based strength training for young people with cerebral palsy and for Resnick et al48 in their RCT evaluating the impact of an exercise training intervention for patients with stroke (Treadmill Study).

A tool to identify barriers and facilitators to implementation

Studies included in stream I often provided details about how emergent contextual and implementation issues had impacted on the results of the outcome evaluation. Twenty studies (18 interventions) described and discussed barriers and facilitators (enablers and inhibitors) to the implementation of the trialled intervention. Several studies (eg, 40, 41, 57, 75, 81, 96 and 109) used tables or appendices to present data about barriers and facilitators. Van't Leven et al49 and Döpp et al50 used focus groups and telephone interviews with occupational therapists and managers to explore barriers and facilitators to the implementation of an occupational therapy (OT) guideline for older people with dementia and their careers. Scobbie et al35 carried out in-depth interviews with health professionals and participants in order to identify views on implementation and acceptability of a framework for goal setting in community-based stroke rehabilitation.

Linking trial outcomes to process evaluation findings

Only 24 studies (21 interventions) reported the use of a theoretical framework or a research framework to inform the decision-making and design of the process evaluation. As a result, reasoning behind used approaches and methodologies was rarely described. The MRC guideline for reporting complex interventions5 was the most mentioned among neurological rehabilitation trials included in stream I evidence (16, 17, 18, 40, 81, 87, 93, 94, 103, 114 and 117). MacNeil Vroomen et al51 used the Adaptive Implementation Model while Whiting et al52 used the Borrelli et al53 fidelity framework. Vluggen et al54 applied the method of Saunders et al55 which recommends a number of themes to investigate as part of a process evaluation.

Forty-six studies (reporting on 39 interventions) presented and discussed relationships between the results of process evaluations and trials; these studies used the results from the process evaluation to make sense of what had been taking place during the implementation process and for building explanations about the impact on outcome measures. However, only 11 of these studies made use of theoretical frameworks and behavioural theories such as Normalization Process Theory (TRACS study16–18) to guide the ‘explanation building’ process. One example of this is Letts and Dunal56 which developed, through consensus, a logic model in order to plan the implementation and integrate information about process and outcomes of a community rehabilitation intervention for adults with brain injury.

Finally, nearly half of the studies (64; 55 interventions) used process evaluation results to generate suggestions and develop recommendations to counter balance the limitations of the research study. These recommendations were regarding aspects of the intervention that could be adapted or modified in order to increase chances of success at the time of future implementation and further neurological rehabilitation research work.

Synthesis 2—methodological guidance/resources

Among all stream II included studies, 10 (33%) were guidelines and guideline-related studies (G). Eleven (37%) were published studies describing and providing useful resources in order to investigate fidelity alongside research trials (F). Six (20%) were specifically related to process evaluation research and design (P). Finally, 3 (10%) provided information regarding the assessment of learning curves in health research (L).

Following the method previously described, eight analytical themes, in the form of recommendations, emerged. These recommendations were developed including potential strategies, measuring tools and methods to address process evaluation research (see online supplementary table S4).

Recommendation 1

Complex interventions should be clearly defined as such. Complex interventions should be described in terms of their ‘active ingredients’. By defining these, researchers can identify how the intervention works and how these ‘active ingredients’ are exerting their effect. Creating a steering group of experts (eg, researchers, practitioners and stakeholders) is one of the recommended strategies in order to achieve a clear understanding of the intervention and its characteristics.

Recommendation 2

Process evaluations should be theory-based. Researchers should draw on existing evidence, guidance and frameworks in order to understand and theoretically explain what processes they expect will be taking place.

Recommendation 3

Context should be acknowledged and accounted for throughout the research process. The context in which the intervention was developed, implemented and finally evaluated should be clearly defined. It would be necessary to describe and monitor changes in the social, physical, economic, political and organisational context in which the intervention is embedded. Understanding context will help the researchers identify its potential impact on implementation and outcomes.

Recommendation 4

Recruitment strategies and changes in recruitment over time, of the main trial and the process evaluation, should be clearly explained. Researchers need to identify and assess the strategies in place to approach and recruit participants for the research trial and the process evaluation. Barriers and facilitators to recruitment will require close investigation. Interviewing staff involved in the recruitment process, or completing logbooks and questionnaires recording reasons for withdrawal, could be a potential strategy in order to address this issue.

Recommendation 5

Information regarding intervention staff should be analysed in detail. A detailed description of their characteristics should be provided: numbers, background experience, incentives and motivations to join the research and their opinions regarding the potential need for the intervention under investigation.

Recommendation 6

The delivery of the complex intervention should be closely monitored. It is necessary for the complex intervention(s) to be described in a study protocol/manual. This protocol should be a tool that intervention providers can use in order to understand the level of tailoring that is considered appropriate. Furthermore, staff delivering the intervention should be trained in order to increase the chances of standardisation and to brief staff regarding the performance criteria. Process evaluations should have strategies in place to investigate if the interventions are delivered as planned in terms of dose and content.

Recommendation 7

Results from process evaluations should be analysed in detail in order to identify possible links with main trial's outcome results. Data collected and analysed during the process evaluation will be of vital importance in order to avoid type III errors when analysing complex intervention's trials' outcomes.

Recommendation 8

Methodological approaches and data collection methods used in process evaluations should be clearly defined. Chosen terminology and clear aims and objectives should be clearly stated at the start of the process evaluation.

Synthesis 3—overarching findings

Narrative themes from synthesis 1 and recommendations from synthesis 2 were collated to generate 57 ‘synthesis recommendations’ which were then grouped in the following 9 areas (see online supplementary table S5).

AREA 1—complex interventions and theoretical approaches

The use of theory to inform and guide process evaluations is recommended. However, to date most process evaluations fail to do so. Theory should be used to get an in-depth understanding of the neurological rehabilitation under investigation and to identify its components. Researchers working on neurological rehabilitation research currently fail to draw on methodological guidance at the time of designing how they will evaluate the processes taking place.

AREA 2—context

Little attention is currently being given to the in-depth exploration of the contextual systems in which the neurological intervention is embedded. Evaluation literature discusses context extensively with respect to the need to describe it in detail, and the need to understand how it can impact on implementation at different stages of the research process. Neurological researchers should consider moving away from the narrow definition of context as the ‘setting where the research takes place’.

AREA 3—recruitment

Recruitment has been identified as one of the main challenges of rehabilitation research. Understanding the barriers and facilitators that take place during the recruitment period is of vital importance for those attempting to evaluate processes taking place and their potential impact on outcomes. To date, process evaluations alongside neurological rehabilitation research have rarely investigated recruitment with this purpose. Finally, recruitment strategies into the process evaluation should be carefully thought through.

AREA 4—describing intervention staff

A number of emergent themes from synthesis 1 identify how, to date, process evaluations are not focussing enough on understanding issues such as intervention staff's motivations for joining research or their perceptions on the potential benefits of the intervention under investigation. The role of intervention staff's level of experience in shaping potential outcome results is equally neglected. Researchers designing process evaluations alongside neurological rehabilitation research should attempt to record and investigate these.

AREA 5—describing the intervention

To date, process evaluations often fail to investigate and report whether the research team had reached a consensus regarding standardisation of the intervention, or if the health professionals were provided with a rationale to help them assess which was the right level of tailoring that should take place. Study protocols should be detailed enough and guide intervention staff through the research process.

AREA 6—preparing and assessing intervention staff

Process evaluations should attempt to have a clear understanding about how intervention staff were trained in order to start their research role. Although training is often provided, it is important that it includes well-defined performance criteria to guarantee the correct delivery of the intervention. At present, this is rarely the case. Furthermore, staff competence should be assessed at different time points in order to identify any potential changes that could ultimately impact on outcomes.

AREA 7—delivering the trial intervention

To date, process evaluations of neurological research trials often provide information which will help identify barriers and facilitators to the implementation process. However, there is a need for PEs to ‘go deeper’ in order to generate a complete understanding of the quality of intervention delivery. This can be achieved by defining clear strategies to monitor not only the quality but also to measure how much of the intervention was delivered (dose delivered) and how much was ‘received’ by participants.

AREA 8—understanding and interpreting process evaluation results

One clear theme that emerged from synthesis 1 was that at present, results from process evaluations are often not used to make sense of what has been taking place during the research process. Process evaluation (PE) results are required to build explanations about the impacts of the trialled intervention on outcome measures.

AREA 9—thinking about methodology

A process evaluation should be a piece of research in its own right and therefore should be described in a protocol. It should have a clear purpose and clear aims and objectives. To date process evaluations alongside neurological rehabilitation research rarely provide detailed information regarding design and chosen strategies. Ultimately, process evaluation should aim at answering a research question by making use of a variety of methods in order to gather sufficient data.

Discussion

This mixed-evidence systematic review has resulted in the identification of a number of gaps in the evidence informing the undertaking of process evaluations in neurological rehabilitation research. At present, there is no consensus among researchers carrying out and reporting results from process evaluations, regarding which terminology (and definitions) to use. These findings are in line with previous work by Steckler and Linnan11 or Carroll et al.57 They identified a considerable overlap in how terms like fidelity or dose are defined. Among the studies included in this review (stream I and stream II), a clear overlap of terms like ‘adherence’, ‘dose’, ‘attendance rate’, ‘compliance’, ‘fidelity’ or ‘exposure’ was identified. Taking adherence as an example; although it may be simply defined as doing what is required, from a health services research perspective, several authors58 59 have explained adherence as the component of ‘implementation fidelity’ that measures to what extent the intervention that has been delivered is consistent with the way the intervention was originally designed or planned. Carroll et al (ref. 57, p. 3) report that “the measurement of implementation fidelity is the measurement of adherence” which includes the subcategories of content, frequency, duration and dose. However, Steckler and Linnan11 propose this same definition for ‘fidelity’ which they consider a process evaluation component in its own right, in the same way as ‘dose’. The findings here reported therefore reinforce what others have suggested;11 that a clearly defined ‘set of terms’ for process evaluation still needs to be developed, universally recognised and applied in order to allow the number of neurological rehabilitation research studies which include a process evaluation alongside them to increase.

Our findings show that context is often acknowledged. Two previous systematic reviews looking at process evaluations in occupational stress management programmes21 and church-based health interventions23 found that only 9% and 34% of the studies, respectively, included information concerning context. However, context was rarely defined. Neither of these studies assessed the level of detail in which context had been described as part of the process evaluation or which strategies had been used, if any, to assess the impacts that contextual changes over time might have had on outcomes. Our findings show that the way context is currently being assessed as a process evaluation component is not detailed enough, and that the impact of wider contextual changes over time is rarely investigated or even acknowledged in neurological rehabilitation research. This is contrary to the general recognition that context is important in the implementation of interventions and needs to be paid attention to.60 61 Process evaluation should not only aim at identifying and describing contextual factors but also investigate their association with variation in mediating responses to intervention components, and ultimately outcomes.9 Campbell et al62 argue that the investigation of context is ‘all important’ and should include all wider socioeconomic background. They further report that contextual changes over time can influence how an intervention may succeed or fail to show a significant impact. In other words, describing the context in which an intervention takes place is important, but understanding it is ‘crucial’, not only to inform intervention design but also to assess if successful ones might, or might not, work when implemented in different settings and conditions.

A further important point identified in this systematic review is the lack of detailed information describing those delivering the trialled intervention in terms of their previous experience and background and their opinions and perceptions of treatment effects and possible impacts of the intervention. First, although near a quarter of studies included in stream I investigated providers' perceptions towards the quality of the intervention, its perceived effects and possible impacts, this number is relatively low and therefore we suggest more effort needs to be put into this aspect of process evaluations. This is in line with what other authors have suggested regarding staff's perceptions playing a role in influencing outcomes.9 63 64 Second, these results have identified that there is a strong need for process evaluations to both, clearly describe intervention staff skills and experience prior to joining the research and investigate how they might influence outcomes. It is evident that careful recruitment of practitioners with the right level of experience is paramount to the success of implementation.65 However, there is currently a lack of evidence to help researchers decide which should be the essential requirements and optimum experience level of staff in charge of implementing a new neurological rehabilitation intervention.

Training of intervention staff has often been mentioned as a necessary component in order to increase implementation fidelity.65–67 The results from this review show that although training of staff often takes place, process evaluations rarely mention performance criteria or assessments to measure skill acquisition post-training and throughout the research programme. In other words, our results show that process evaluations in this field of research are currently not addressing the role that learning curve effects might play in influencing intervention outcomes. Although a number of studies have looked at learning curve effects in RCTs,68–70 the bulk of these are primarily focused on surgical trials (clinical health technologies) and the implementation of new surgical procedures. Learning curve effects can be defined “as an improvement in performance over time” (ref. 69, p. 421) which indicates that with time, there is generally a change leading to higher quality implementation of the tested intervention.68 A number of hierarchical factors have been reported as influencing learning.69 Cook et al69 reported that ‘professional teams’, the characteristics of the patients undergoing the procedure and the characteristics of the surgeons carrying out the intervention (eg, attitudes, abilities and previous experience), will further impact on the learning. Further consideration should be given to how possible learning curves and their impact on outcomes should be studied within the context of a longer term, complex intervention. The process evaluation of an RCT looking at the impact of an occupational therapy intervention for stroke survivor living in care homes (OTCH)70 described how occupational therapists reported becoming better at implementing the OTCH intervention as time went by. Therapists learnt how to overcome challenges linked to, among others, resource limitations, institutional context and patient and care home staff engagement. These results provided evidence about how learning curves can impact on the quality of the implementation of neurological rehabilitation intervention(s). Process evaluations need to assist researchers in identifying the mechanisms underlying this possible impact. Ignoring the learning effect could potentially lead to non-conclusive results since often the trialled intervention will only have identifiable significant impacts once adequate experience is gained.71 Learning curve effects also have implications for the length of the intervention period within the trial timetable and associated costs.

Several authors72 73 have argued that interventions that have been designed and tested with participants from homogeneous or atypical population groups will have very limited generalisability. There is more to delivering the intervention than simply measuring how many elements were delivered.74 75 There is a need to tailor interventions to patients' limitations and cultural background in order to be able to replicate interventions across settings.76 However, this can lead to tension between tailoring and the need to have fidelity to the original intervention. Song et al76 further explain that tailoring does not mean that the provider may improvise what he/she does, it means that what is standardised will be contrasted and clearly defined and monitored against what is customised (including delivery of unplanned components of the intervention). The way in which this can be accurately performed remains unclear to date. Our findings show that guides for tailoring for staff delivering rehabilitation trial intervention are rare. Song et al76 argue that the assessment of fidelity will have to be standardised and tailored to the actual level of standardisation and tailoring of the trialled intervention. In line with these suggestions, the results from this review have identified a strong need for process evaluations to have strategies in place to investigate and monitor in detail the level of tailoring according to patients' needs that is taking place when providers deliver the trialled intervention. It is only by doing so that researchers can avoid the tailoring process having a negative impact on fidelity of implementation. The recommendations here presented stress the importance of providing a detailed description of trialled interventions and their tailoring. The uptake by researchers of recent published tools such as The Consolidated Standards of Reporting Trials (CONSORT) 2010 statement77 or the more recent Template for Intervention Description and Replication (TIDieR) checklist and guide78 can potentially have a positive impact on the development of this vital component of process evaluations. Contrary to what process evaluation and complex interventions research guidelines strongly recommend, the results here reported have identified that links between process and outcome evaluation results are often not being clearly addressed and that to date, a widely accepted and standardised way to achieve this is yet to be proposed. This finding is supported by previous research16 21 23. In 2006, researchers involved in the process evaluation of the Randomised Intervention of Pupil Peer Led Sex Education (RIPPLE) study analysed process data in two stages. The first stage was carried out before outcome analyses had taken place. This stage generated answers in the form of hypotheses which were then tested during the second stage via statistical analyses that integrated process and outcome data. These included initially on-treatment analyses and then regression analyses followed, where appropriate, by tests for interactions assessing the impact of mediating factors identified in the process evaluation.

The recommendations that this review proposes can be challenging to implement for researchers working with tight funding constraints. Researchers will be faced with choices and trade-offs about what aspects of the intervention and its delivery to focus on in process evaluations. These will need to be carefully designed to maximise their ability to gain an in-depth understanding of the ‘anticipated’ factors that are likely to impact on outcomes. This will avoid wasting resources collecting unnecessary data.

Review limitations

Although database searches, carried out in order to identify studies to be included in stream I, were informed by reliable sources and an expert librarian reviewed the search strategies, there is a possibility that relevant studies were not identified. The main reason for this relies in the fact that the term ‘process evaluation’ is not as yet, considered a ‘MeSH heading’ in any of the databases that were searched. As a consequence, a wide range of term combinations were used.

A high proportion of the evidence included in stream I were process evaluations alongside RCTs. Although the inclusion criteria were broader, the fact that so many of them were RCTs could mean that the findings might have greater relevance to researchers thinking about this type of research design. Finally, only studies written or translated into English were included in this review (because of limited financial resources to translate); there is a chance that, by doing this, a number of relevant studies, written in other languages, were left out.

Conclusions

This review identified the following key findings. (1) There remains no consensus regarding process evaluation terminology, and this provides an excellent opportunity to engage the rehabilitation research community in creating one that reflects the nuances of research in this field; (2) there is a need for process evaluations to address the nature of context, and the role that contextual factors (and their changes over time) can play in influencing outcomes; (3) there is a strong need for process evaluations to clearly describe intervention staff skills and experience prior to joining the research and investigate how these may influence outcomes; (4) process evaluations to date do not investigate learning curves and their potential impact on outcome evaluations; (5) there is a strong need for process evaluations to have strategies in place to investigate and monitor in detail the level of tailoring according to patients' needs that is taking place when providers deliver the trialled intervention; and (6) further research is needed to develop clear and standardised methods for linking outcome and process evaluation results.

This systematic review has provided a valuable insight into the design and quality of process evaluation research in neurological rehabilitation. Further research is needed to promote the use of process evaluation alongside health research trials, and more emphasis on providing specific training in process evaluation research is strongly recommended.