Process evaluation of complex interventions tested in randomised controlled trials in musculoskeletal disorders: a systematic review protocol.

INTRODUCTION: The effectiveness of complex interventions for the management of musculoskeletal disorders has been estimated in many randomised clinical trials (RCTs). These trials inform which interventions are the most effective, however they do not always inform how an intervention achieved its clinical outcomes, nor how and what elements of an intervention were delivered to patients. Such information is useful for translating findings into clinical practice. A few process evaluation studies have been conducted alongside RCTs and a variety of methods have been used. To gain a better understanding of current practices of process evaluation in RCTs in musculoskeletal disorders, this systematic review is designed to answer the following research question: How are process evaluation of complex interventions tested in RCTs in musculoskeletal disorders being conducted? METHODS AND ANALYSIS: We will systematically search seven electronic databases (MEDLINE, SCOPUS, CINAHL, PsycINFO, EMBASE, Web of Science and Cochrane database) from the date of inception to August 2018 for studies on process evaluation of RCTs on non-surgical and non-pharmacological management of musculoskeletal disorders. We will include qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs or separate studies that assessed interventions for musculoskeletal disorders. Two reviewers will screen abstracts and apply prespecified inclusion criteria to identify relevant studies, extract the data and assess the risk of bias within included studies. We will follow recommendations from the 'Cochrane Qualitative and Implementation Methods Group Guidance Series' when assessing methodological strengths and limitations of included studies. We will use a narrative synthesis to describe findings. ETHICS AND DISSEMINATION: Ethical approval is not required as this review will not collect original data. Findings from this systematic review will be presented at a scientific conference and published in a peer reviewed journal. PROSPERO REGISTRATION NUMBER: CRD42018109600. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

One strength of this study is the comprehensive search of published and unpublished literature in different databases and trial registries.

The other strength of this protocol is its scientific robustness. When designing this protocol we followed the Cochrane Qualitative and Implementation Methods Group Guidance Series.

A limitation of this review is that we may not identify other process or outcome evaluation studies that assessed aspects of process evaluation but did not explicitly report them in the title or abstract.

Introduction

Musculoskeletal disorders are the second largest cause of disability.1 Common musculoskeletal disorders include low back pain (the most frequent disorder),1 shoulder pain and neck pain (the second and third most prevalent musculoskeletal disorders)2 and osteoarthritis (the most common joint disorder).1 The individual, societal and economic burden of musculoskeletal disorders is high. For example, in New Zealand, between 2005 and 2013, the direct cost of physiotherapy interventions for shoulder injuries alone was $134 million ($14 million/year).3 The total direct costs of osteoarthritis were greater than $500 million in 2005.4 Together, high direct costs and waiting lists highlight the need for effective and affordable interventions to minimise the growing burden (social and economic) of musculoskeletal pain.

To improve healthcare services, it is crucial to identify which interventions are the most effective.5–7 Randomised clinical trials (RCTs) estimate the effectiveness of different interventions5 8–11 and can be referred to as ‘outcome evaluation’ trials.12 Some RCTs may also include an economic analysis that is run in parallel.13–16 This type of trial is referred to as ‘economic evaluation’ trial.12 While being extremely important for improving healthcare, ‘outcome and economic evaluation’ trials do not inform how interventions achieved their clinical outcomes.12

Complex interventions are defined as interventions with multiple components that may interact with each other, influencing clinical outcomes.17 18 The interaction between the components of an intervention can impact on clinical outcomes.17 18 Examples of complex interventions include exercise therapy, education and behavioural change.13 19 20 Complex interventions represent a challenge for researchers when planning a trial.21 These challenges are caused by the difficulty in standardising the way an intervention is delivered,22 the possible influence of the local context on clinical outcomes,23 logistical challenges at the organisational level and complex interactions between the components of the intervention and the clinical outcome.21 22 24

When assessing the effect of a complex intervention, researchers might need to take into account the interacting elements within the experimental and control interventions, variability of outcomes, behaviour of professionals delivering as well as behaviour of patients receiving the intervention and how much flexibility is permitted for adapting an intervention being tested.18 21 There is no clear threshold for classifying an intervention as simple or complex, but only a few interventions can be considered simple.17 18 Understanding how an intervention achieves its clinical outcomes is particularly relevant when assessing effectiveness of complex interventions.18

Process evaluation studies inform how a complex intervention achieves its clinical outcomes.12 These studies provide information about how and what elements of an intervention were delivered to patients8 11 12 25 and why interventions work (or fail to do so) the way they do.26 Process evaluation studies may inform about what and how interventions are implemented (ie, implementation), how interventions generate change in clinical outcomes (ie, mechanism of impact) and how the context affects the clinical outcomes (ie, context).8 11 27 Such information is useful for translating findings into clinical practice.12 28

Process evaluation methods are still being developed12 and are relatively scarce in musculoskeletal research.29 The Medical Research Council (MRC) (UK) published its first guideline for process evaluation of clinical trials in 2015.12 The number of process evaluation studies conducted alongside RCTs is small, and the approaches used to assess implementation of interventions are varied. Previous reviews have focused on process evaluation of trials in fields other than musculoskeletal disorders, for example, complex interventions for patients with chronic diseases28 and interventions for patients with neurological disorders;30 another review assessed what is being measured in process evaluations for worksite health promotion programmes.31

We planned this review to identify approaches used for assessing process evaluations of trials focusing on complex interventions (non-surgical and non-pharmacological) for the management of musculoskeletal disorders. The systematic review was designed to answer the following overarching research question: How are process evaluations of complex interventions tested in RCTs in musculoskeletal disorders being conducted?

The specific research questions were:

Is there a theory adopted by research teams when conducting process evaluations? If so, which theory is used (eg, theory-based evaluation, realistic evaluation)?

Which study designs were used during the process evaluation (eg, qualitative, quantitative or mixed-method)?

When is process evaluation being performed (ie, phase II - feasibility and piloting; phase III - evaluation; or phase IV - implementation)?

How are results of the trial being integrated with the findings from the respective process evaluation study?

Is the process evaluation independent or does it become independent at some stage in the trial?

What are the barriers and facilitators faced by the authors while conducting process evaluations?

What are the strengths and limitations of the process evaluation methods reported by the study authors?

Methods

We used the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols for planning and reporting this protocol.32 We planned the assessment of process evaluation of RCTs based on Cochrane Qualitative and Implementation Methods Group guidance papers.33–38

Searches

We will search seven databases including MEDLINE, SCOPUS, CINAHL, PsycINFO, Embase, Web of Science and Cochrane database in order to identify relevant studies evaluating process evaluation of complex interventions for the management of musculoskeletal disorders tested in RCTs. To identify any unpublished literature, we will additionally search four clinical trial registries (Cochrane Central Registry of Controlled Trials, ClinicalTrials.gov, EU clinical trials register and Australian New Zealand Clinical Trial Registry). If we identify any project that is likely to meet the inclusion criteria, we will contact the authors for the results related to the process evaluation.

We will combine ‘process evaluation’, ‘ program evaluation’, ‘fidelity’ or other search terms that mean process evaluation with ‘musculoskeletal disorders’ and its related terms. We will exclude all surgical or study protocols by using NOT ‘surgery’ and ‘protocol’ respectively or their substitute words.

Pilot search

Prior to conducting the full search on electronic databases, we conducted a pilot search using MEDLINE, SCOPUS, CINAHL, PsycINFO, Embase and Web of Science databases. This was done for scoping purposes and the pilot search yielded a total of 1695 studies. The terms used during the pilot search are reported in table 1. Following this, we consulted a health sciences librarian for optimising the search strategy. The search strategy will be adapted for each database.

Table 1

Pilot search strategy

Database	Search term 1	Search term 2	Filter	Number of articles retrieved
Medline	MesH: Program evaluation [mh] OR Process assessment (healthcare) [mh] OR Outcome assessment (healthcare) [mh] OR Title/Abstract/Keywords: ‘Process Evaluation’ OR ‘Programme Evaluation’ OR ‘Program Evaluation’ OR ‘Process assessment’ OR Fidelity.	MeSH: Musculoskeletal Diseases [∗diagnosis; *rehabilitation; ∗therapy]; OR Osteoarthritis [diagnosis; *rehabilitation; therapy]; OR Shoulder Pain [diagnosis; *rehabilitation; therapy] OR Back pain [diagnosis; *rehabilitation; therapy] OR Neck pain [diagnosis; *rehabilitation; therapy] OR musculoskeletal pain [diagnosis; *rehabilitation; therapy] OR Arthritis [diagnosis; *rehabilitation; therapy].	Age>18 years Humans Study design: clinical trial (All phases); OR randomized controlled trials; Clinical trials OR qualitative studies.	218
Scopus	Search in title, abstract, keywords: ‘Program evaluation’ OR ‘Process Evaluation’ OR ‘Programme Evaluation’ OR ‘Process assessment’ OR Fidelity.	Search in title, abstract, keywords: Musculoskeletal OR Osteoarthritis OR ‘Shoulder Pain’ OR ‘Back pain’ OR ‘Neck pain’ OR ‘Knee pain’ OR Arthritis.	NOT: (surger*(title, abstract, keywords) OR protocol (title)) Limits: Document type, article; Keyword, Human and humans; Source type, Journals.	756
CINAHL	MeSH: Program evaluation [mh] OR Process assessment (healthcare) [mh] OR Outcome assessment [mh] OR Title/abstract: ‘Process Evaluation’ OR ‘Programme Evaluation’ OR ‘Program Evaluation’ OR ‘Process assessment’ OR ’Fidelity'.	MeSH: (MH ‘Musculoskeletal Diseases’) OR (MH ‘Musculoskeletal Abnormalities’) OR (MH ‘Diagnosis, musculoskeletal’) OR (MH ‘Osteoarthritis’) OR (MH ‘Osteoarthritis, Spine’) OR (MH ‘Osteoarthritis, Wrist’) OR (MH ‘Osteoarthritis, Knee’) OR (MH ‘Osteoarthritis, Hip’) OR (MH ‘Osteoarthritis, Cervical’) OR (MH ‘Low Back Pain’) OR (MH ‘Back Pain’) OR (MH ‘Neck Pain’) OR (MH ‘Chronic Pain’) OR (MH ‘Shoulder Pain’) OR (MH ‘Nociceptive Pain’) OR (MH ‘Patellofemoral Pain Syndrome’) Keywords: Musculoskeletal	Limiters: All adults AND humans AND [therapy(best balance between sensitivity and specificity) OR qualitative(best balance between sensitivity and specificity)]	341
PsycINFO	Subject heading search: Program evaluation Title, abstract, Keyword: ‘Program evaluation’ OR fidelity OR ‘process assessment’ OR ‘process evaluation’ OR ‘process assessment’ OR programme evaluation	Subject heading: Musculoskeletal disorders OR back pain OR pain OR chronic pain OR arthritis Title, abstract, keywords: neck pain OR musculoskeletal pain OR osteoarthritis OR shoulder pain OR ‘knee pain’	Limits: ((‘therapy (maximizes sensitivity)’ or ‘qualitative (maximizes sensitivity)") and adulthood<18+  years>and human)	85
Embase	Emtree search: program evaluation. Keyword: process evaluation, process assessment, fidelity, programme evaluation, Title and abstract: ‘process evaluation’, ‘process assessment’, fidelity, ‘programme evaluation’, ‘program evaluation’.	Emtree search [Diagnosis, Rehabilitation, Therapy]: arthritis OR musculoskeletal pain OR Musculoskeletal disease OR neck pain OR backache OR shoulder pain OR osteoarthritis	Limits: humans AND adults. Additional limits: qualitative study to maximize specificity OR clinical trials (all phases) OR RCTs OR controlled clinical trials.	25
Web of Science	‘Program evaluation’ OR ‘Process Evaluation’ OR ‘Programme Evaluation’ OR ‘Process assessment’ OR Fidelity	Musculoskeletal OR Osteoarthritis OR ‘Shoulder Pain’ OR ‘Back pain’ OR ‘Neck pain’ OR ‘Knee pain’ OR musculoskeletal OR Arthritis		270
Total	1695

Eligibility criteria

Types of study to be included

We will include all qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs and separate reports that assessed process evaluation of RCTs on complex interventions (non-surgical and non-pharmacological interventions) for musculoskeletal disorders. We defined a process evaluation study as any study aimed at understanding the functioning of an intervention by examining the implementation, mechanisms of impact and/or its contextual factors.12

To be included, studies need to indicate in the title or in the aim of the study that they are assessing components of process evaluation (eg, implementation, mechanisms of impact or context). We will include any study that: (1) explicitly indicates that it was a process evaluation study in the title or in the aim of the study, or (2) intends to evaluate process evaluation (eg, fidelity, dose delivered, dose received, reach, recruitment, context, barriers, implementation), without explicitly stating that it is a process evaluation study (eg, qualitative study alongside an RCT).

Condition or domain being studied

We will include process evaluation of RCTs investigating complex interventions for the management of musculoskeletal disorders. For the purpose of this review, we defined musculoskeletal disorders as health problems of the locomotor apparatus (including muscle, tendon, skeleton and ligaments) including for example, low back pain, neck pain, shoulder pain, elbow pain, hip pain, knee pain and soft tissue injuries. We will exclude systemic conditions (eg, gout, rheumatoid arthritis, Raynaud disease, scleroderma and dermatomyositis), osteoporosis, tumours, infections of bones and joints, fibromyalgia, diabetic neuropathy, fractures, ankylosing spondylitis and spinal cord injuries.

Participants/population

Studies should include participants with musculoskeletal disorders who received complex interventions (non-surgical and non-pharmacological) as part of an RCT or clinicians delivering interventions as a part of a clinical trial. To be included in the review, studies must have recruited adult humans (ie, >18 years old).

Intervention

Complex interventions of musculoskeletal disorders in an RCT include, but are not limited to, exercise therapy, physical activity, self-management advice, education and psychosocial interventions. We will not include studies assessing surgical or pharmacological interventions.

Comparator(s)/control

We will include studies that compared complex interventions with appropriate control groups (eg, waiting list, placebo groups, other active interventions). We will exclude studies that compared surgical or pharmacological interventions to non-surgical or non-pharmacological interventions.

Context

We will include studies that assessed effectiveness of interventions in primary care (eg, private practice, home-based interventions and community-based interventions).

Primary outcome(s)

These are in line with our study questions and are as follows: (1) Theory (if any) adapted to conduct the process evaluation; (2) Study designs used for process evaluation; (3) Phase of the trial when the process evaluation was performed; (4) Approach used to integrate the process evaluation with the main results of the RCT; (5) Barriers and facilitators faced by authors while conducting process evaluation of RCTs; (6) Strengths and limitations of the process evaluation methods as reported by the study authors.

Data extraction (selection and coding)

Study selection

Prior to screening, we will remove duplicate articles. Then, two reviewers will independently screen all titles and abstracts following the eligibility criteria and using a standard form (see online supplementary file 1). After the first screening, the two reviewers will meet to assess the agreement on inclusion or exclusion of the studies based on title and abstract reading. Then, both the reviewers will independently screen the full text of the articles for all the articles that meet the inclusion criteria based on the title and abstract or just based on the title, the abstract being uncertain. During title, abstract and full-text screening, disagreements will be resolved by consensus. If consensus is not reached, then a third reviewer will be consulted.

Data extraction

The research team will develop a form for extracting data from the included studies. The data extraction form will be designed based on the ‘MRC of UK recommendation for process evaluations’ and the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’.33–38

Data extraction forms will be piloted by two reviewers using articles that were included after full-text screening. Following recommendations from The University of York Centre for Research and Dissemination, data extraction forms will be piloted on a random sample of 10 included studies. This will ensure that resources are not wasted, and that all relevant information is being extracted from included studies. Once the research team agrees that the form is comprehensive and coherent, two reviewers will independently extract data from studies that were included after full text screening. Disagreements will be resolved by consensus. If consensus is not reached, then a third reviewer will be consulted.

We will extract data regarding: (1) Basic information about the study - publication year, authors, title, study type, aims; (2) Context and participants - study setting, population, participant characteristics, intervention delivered; (3) Methods used - design, methods used for sample recruitment, data collection and analysis, theoretical model used to interpret data and contextualise findings; (4) Process evaluation - rationale for study design adopted, dose delivered, participants’ attitudes and beliefs, approach used to assess participants’ adherence and fidelity to intervention protocol, approach used to assess clinicians’ adherence and fidelity to intervention protocol, description of clinicians, training of clinicians, implementation monitoring, theory supporting process evaluation, process evaluation findings, association between process evaluation and outcome evaluation findings.

Risk of bias assessment

We will assess the risk of bias of the RCT report if process evaluation is reported within the outcome evaluation (ie, RCT study). In that case, we will use the Cochrane Collaboration tool for assessing risk of bias (reference).

When process evaluation is reported as an independent study, we will follow recommendations from the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’ for assessing methodological strengths and limitations of the included studies. In those cases, we will assess the risk of bias of the process evaluation study alone. The following domains will be included in the assessment: (1) Clear aims and research question; (2) Congruence between the research aims/question and research design/method(s); (3) Rigour of case and/or participant identification, sampling and data collection to address the question; (4) Appropriate application of the method, richness/conceptual depth of findings, exploration of deviant cases and alternative explanations and reflexivity of the researchers; (5) We will use the Critical Appraisal Skills Programme (CASP) tool for assessing methodological strengths and limitations of included studies.39 This is the tool most used by systematic reviews focusing on qualitative evidence synthesis.38 As per recommendations from the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’, we may add other tools if we deem that a specific type of study might be at a disadvantage if we use only CASP; (6) We will classify and group interventions using the 10-dimension Complexity Assessment Tool for Systematic Reviews (iCAT-SR).40

Strategy for data synthesis

We will use a narrative synthesis to describe: (1) The theory (if any) adopted by the research teams when conducting the process evaluations; (2) The study designs used during the process evaluation alongside RCTs; (3) The phase in which process evaluation was performed; (4) The way results of the trials are being integrated with findings from the respective process evaluations; (5) The barriers and facilitators faced by the authors while conducting process evaluations; (6) Strengths and limitations of the process evaluation methods as reported by the study authors.

We will use narrative summaries of the individual studies and shared themes to synthesise the findings.

Analysis of subgroups or subsets

Depending on the number of articles included, we will conduct a subgroup data analysis based on: context (eg, indigenous, non-indigenous participants; under-developed and developed countries, healthcare systems) or the category of interventions (as categorised by iCAT-SR) for describing barriers and facilitators, fidelity and adherence to implementation of the planned intervention.33 40

Discussion

Process evaluation studies can help improve translation of research into clinical practice. The information gathered by process evaluation studies is valuable for healthcare professionals, policy makers and researchers. Such evidence can inform whether findings from a small trial should be scaled up or whether findings from a trial need to be modified and adapted to another context.

This review will contribute to the field by identifying methods used for assessing process evaluation of clinical trials assessing the effectiveness of interventions for musculoskeletal disorders. There are no definitive methods or guidelines for conducting process evaluation studies.12 35 This is due in part to the fact that the term ‘process evaluation’ includes different domains: implementation of interventions, the mechanisms of action of interventions and the impact of context factors (ie, how context influences clinical outcomes or is influenced by an intervention). To address each of these three domains, different research methods are required. Findings from this review will identify current practices adopted by musculoskeletal researchers when conducting, analysing and reporting process evaluations studies. Our findings will identify gaps in the literature and inform future research conducted in the area of musculoskeletal disorders and rehabilitation.

This protocol has limitations. We will only include studies that explicitly state process evaluation of an intervention was assessed or that include outcome measures that allow researchers to assess the process evaluation of an intervention (eg, fidelity, or adherence to an intervention). It will not be feasible to screen the full text of all published trials on musculoskeletal disorders. We may not identify other process or outcome evaluation studies that assessed process evaluation but did not explicitly report it in the title or abstract. The advantage of our approach is that it identifies current practices using studies in the broad area of musculoskeletal disorders.

Conclusions

To our knowledge, this will be the first systematic review to assess how process evaluations are currently being conducted in RCTs of non-surgical and non-pharmacological interventions in the management of musculoskeletal disorders. This review will describe current practices on process evaluation of clinical trials and inform future research that will be conducted in this area. Recently, there has been increased encouragement to conduct process evaluation studies to better inform implementation of findings from clinical trials in clinical practice and policy-making. It is reasonable to expect that this review will yield a diversity of methods used by different research groups. Hence, the importance of this review is in identifying best practices for future process evaluation studies tested in RCTs in musculoskeletal disorders.