Process evaluations of primary care interventions addressing chronic disease: a systematic review.

OBJECTIVE: Process evaluations (PEs) alongside randomised controlled trials of complex interventions are valuable because they address questions of for whom, how and why interventions had an impact. We synthesised the methods used in PEs of primary care interventions, and their main findings on implementation barriers and facilitators.
DESIGN: Systematic review using the UK Medical Research Council guidance for PE as a guide. DATA SOURCES: Academic databases (MEDLINE, SCOPUS, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, EMBASE and Global Health) were searched from 1998 until June 2018. ELIGIBILITY CRITERIA: We included PE alongside randomised controlled trials of primary care interventions which aimed to improve outcomes for patients with non-communicable diseases. DATA EXTRACTION AND SYNTHESIS: Two independent reviewers screened and conducted the data extraction and synthesis, with a third reviewer checking a sample for quality assurance.
RESULTS: 69 studies were included. There was an overall lack of consistency in how PEs were conducted and reported. The main weakness is that only 30 studies were underpinned by a clear intervention theory often facilitated by the use of existing theoretical frameworks. The main strengths were robust sampling strategies, and the triangulation of qualitative and quantitative data to understand an intervention's mechanisms. Findings were synthesised into three key themes: (1) a fundamental mismatch between what the intervention was designed to achieve and local needs; (2) the required roles and responsibilities of key actors were often not clearly understood; and (3) the health system context-factors such as governance, financing structures and workforce-if unanticipated could adversely impact implementation.
CONCLUSION: Greater consistency is needed in the reporting and the methods of PEs, in particular greater use of theoretical frameworks to inform intervention theory. More emphasis on formative research in designing interventions is needed to align the intervention with the needs of local stakeholders, and to minimise unanticipated consequences due to context-specific barriers. PROSPERO REGISTRATION NUMBER: CRD42016035572. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is a reflexive thematic synthesis and interpretation of the papers by a multidisciplinary team.

A bespoke quality assessment tool based on the Medical Research Council guidance for process evaluations was used.

Given variations in lexicon, our search strategy may not have exhaustively captured all process evaluations of complex primary care interventions.

Use of the framework and quality assessment tool to synthesise the findings has not been tested previously.

Introduction

An accessible, effective and affordable primary healthcare (PHC) system is needed to equitably reduce the rising non-communicable disease (NCD) burden.1–3 Complex interventions which comprise ‘multiple interacting components (although additional dimensions of complexity include the difficulty in their implementation and the number of organisational levels they target)’ are often used to reduce this burden.4 These interventions often require individual and organisational behaviour change within dynamic health system contexts.5 6 Randomised controlled trials (RCTs) of complex primary care interventions have been conducted, but there is often ambiguity as to what was implemented on the ground.4 7 8 Process evaluations (PEs) are conducted alongside trials to examine if a complex intervention was implemented as intended, and to explore if, for whom, how and why the intervention had an impact.4

A process evaluation is defined by the UK Medical Research Council (MRC) as a study to ‘understand the functioning of an intervention, by examining implementation, mechanisms of impact, and contextual factors’.4 The MRC PE framework and guidance published in 2015 is based on the synthesis of influential frameworks and theories in public health research, and informed by the authors’ PEs.4 The key points are briefly summarised and elaborated on in box 1.

In the guidance, concepts of reach, fidelity and adoption were highlighted as key to examining implementation quality. For example, assessing ‘fidelity’ would help determine whether the research was conducted as per protocol. The MRC guidance also highlighted the value of describing the intervention theory more explicitly, that is, providing the hypothesis relating to how the complex intervention could potentially interact with contextual factors to produce variation in outcomes.9 10 Ideally, the intervention theory would determine the process (qualitative and quantitative) data to be collected and analysed before the RCT outcomes are known. The PE findings could potentially help explain variation in RCT outcome, refine the intervention theory and inform future research priorities. Recognising the need to facilitate implementation of evidence into practice and policy, the MRC guidance also expands on the importance for PEs to be conducted across all stages of research, that is, feasibility/piloting, evaluation of effectiveness and post-evaluation stages. While the guidance was well received, outstanding questions remain in this developing field. For example, what is the role of other theories and frameworks for PEs? What methods can be used and how?11–13 Synthesising the collective ‘experience’ described in published PEs may answer some of these questions.

This review has two primary objectives. First is to review the methods used in published PEs and their alignment with the MRC guidance, and second to identify the key implementation barriers and facilitators reported in these PEs. We used the three key points (as summarised in box 1) from the MRC guidance as a lens to evaluate the published literature, and employed it as a framework for synthesising our findings.

1. Expanded on the functions of the process evaluation.

Implementation: ‘What is implemented and how?’

Implementation process: What did the research team do?

Reach: Did you recruit participants that your intervention was intended to have an impact on?

Fidelity: Did you do what you planned to do as per protocol?

Adaptation: What changes were made in the delivery of the intervention to the local context?

Dose: Is the frequency of the intervention delivered as planned?

Mechanisms of impact: ‘How does the delivered intervention produce change?’

Participants’ experiences of the intervention.

Mediators.

Unexpected consequences and pathways.

Context: ‘How does the context affect implementation and outcomes?’

Contextual factors which shape the theory of the intervention.

Contextual factors that could affect and be affected by implementation, mechanisms and outcomes.

Causal mechanisms within the context that could potentiate the effects.

2. Provided a systematic approach to the design and conduct of process evaluations.

Planning: relationships with stakeholders and degree of separation between evaluators and research team.

Design and conduct: describe intervention and causal assumptions, identify the key process questions, and inform data collection.

Analysis: use of reporting guidelines for methods; transparently report whether process data are analysed and reported prior or after knowing outcomes.

Reporting: intervention theory and how it informed data collection; protocol to link multiple outputs.

3. Described and expanded on the function of process evaluations across the different stages of development, evaluation and implementation.

Development: feasibility and acceptability of implementation strategies and optimising design and evaluation of intervention.

Effectiveness: fidelity of intervention, mechanisms of action and contextual factors.

Post-evaluation implementation: implementation of intervention into usual practice, long-term surveillance.

Methods

The systematic review protocol has been prespecified and published in detail as a separate protocol.14 A summary of the methods is presented here according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see online supplementary appendix 1 for the PRISMA checklist).15

Population: patients with NCDs (diabetes, depression, cardiovascular disease, chronic obstructive pulmonary disease, chronic kidney disease, type 2 diabetes mellitus) and their primary care providers.5

Intervention: complex interventions which comprise ‘multiple interacting components although additional dimensions of complexity include the difficulty of their implementation and the number of organisational levels they target’ within PHC.4

Comparator: the control condition may include treatment as usual, active control or placebo control.

Outcomes for this systematic review: (1) strengths and limitations of each PE using the MRC guidance as a reference point; and (2) identification of implementation barriers and facilitators for the complex interventions.

Timing: published data from 1998 until June 2018.

Design: PE of the included RCTs as defined by the MRC as ‘a study which aims to understand the functioning of an intervention, by examining implementation, mechanisms of impact, and contextual factors’.4 Given that PEs are often not explicitly labelled as such,10 we included studies with comparable aims.

Exclusion criteria: not a journal article, not a report based on empirical research, not reported in English, reviews and not human research.

Search strategy and data extraction: databases reporting academic publications (MEDLINE, SCOPUS, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, EMBASE and Global Health) were searched. Standard systematic review methods were followed for searching, screening and extracting data from eligible studies.14 Our search strategy is included as online supplementary appendix 2. Two reviewers (HL, AM) conducted most of the data extraction, with a third reviewer (MN) assisting in data extraction with some papers and as part of quality assurance, and checked on the data extraction for a 10% sample of the identified papers. Given that a key aim of this study was about PE methodology, we deviated from the published systematic review protocol, by including our interpretation in addition to the study’s strengths and limitations posited by the authors of those papers.

Data analysis and synthesis

Descriptive items (eg, number of positive RCTs) were tallied and synthesised into three tables: (1) Overall characteristics: presenting the studies grouped into different diseases and ordered by year of publication (online supplementary appendix 3); (2) methods table: grouping studies by the stages of the PE (ie, feasibility/piloting, effectiveness, post-evaluation) (online supplementary appendix 4); and (3) quality assessment: findings based on a bespoke assessment tool which was designed based on some of the key recommendations from the MRC guidance and two other papers which had synthesised PE literature before (online supplementary appendix 5).

Extracted qualitative data were coded by HL, and grouped into categories of context, mechanisms and implementation. Inductive derivation of the key themes was done through constant comparison between the findings from the papers within each category and examining the relationships between them. Online supplementary appendix 6 provides illustrative quotations. Using a modified MRC framework, we mapped our methodological and implementation findings to triangulate and synthesise our findings.

Patient and public involvement

While patient and public perspectives were synthesised from published papers, no public and patients were directly involved in this study.

Findings

Characteristics of included studies

We identified 69 studies. The PRISMA flow chart is presented in figure 1. In summary, 66 studies were conducted in high-income countries, 1 study in Zambia, 1 in Malaysia and 1 in India. Cardiovascular disease, diabetes mellitus and depression were the conditions most often investigated, with only six studies on chronic obstructive pulmonary disease and one study on chronic kidney disease. Overall, the complex primary care interventions fit within the general categories of facilitating patient self-management (13 studies), organisational change to include collaborative care (15 studies), facilitating better case management using clinical information systems (eg, telehealth) (15 studies), and the use of decision support and guideline implementation (eg, referral systems) (22 studies). In addition, five studies explored the challenges when conducting trials in primary care, for example, the recruitment of patients.

Figure 1

PRISMA figure. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT, randomised controlled trial.

Only 22 studies were labelled clearly as PEs, although this was more common in recent years. Twenty studies were conducted at the feasibility stage with 5 labelled as PEs, 43 studies at the effectiveness stage with 17 labelled as PEs, and 6 studies at the post-evaluation stage with none labelled as PEs. (The methods used at these different stages are described in greater detail in the next section.) In 35 studies the degree of separation between the process and outcome evaluation researchers was explicit. The cost considerations for the system and stakeholders were mentioned in 10 papers (see table 1 for more details). In figure 2, the context of the studies and an overview of the main methodological and implementation findings are diagrammatically presented in an adapted PE framework.

Table 1

Summary of the characteristics of the included studies

Disease condition	Interventions	Setting	RCT outcomes	Cost considerations (Y/N/NA)
20 studies on depression.	Interventions mostly around collaborative care through increasing expertise of different roles (eg, lay worker, nurse for proactive care, GP for PHC) (16 studies), times to implement practice guidelines (4 studies), and trialling specific interventions such as physical exercise and cognitive–behavioural therapy (2 studies).	9 UK. 7 USA. 1 Sweden. 1 Germany. 1 Australia. 1 India.	11 Positive. 5 Negative. 4 NA.	4 Y. 14 N. 2 NA.
17 studies on diabetes.	The interventions included improving guideline-based referral and treatment (7 studies), patient self-management, community support (7 studies) and telehealth (3 studies).	4 Ireland. 3 UK. 1 Norway. 3 USA 2 Canada (1 of the First Nations). 2 Australia. 1 New Zealand. 1 Malaysia.	6 Positive. 10 Negative. 1 NA.	3Y. 13 N. 1 NA.
25 studies on CVD.	10 studies were about improving the screening and management of CVD using best-practice guidelines (eg, educational materials to improve referral or decision analysis). 10 studies were about organisational change with models of care that incorporated new roles such as a nurse-led clinic, or the use of a lay worker for angina management, and technology (eg, telemonitoring, point of care testing). 5 studies explored trial implementation such as recruitment of patients and providers, and were less about the intervention.	9 UK. 6 Australia. 3 Canada. 2 New Zealand. 2 The Netherlands. 1 Ireland. 1 USA. 1 Zambia.	15 Positive. 5 Negative. 5 NA.	3 Y. 15 N. 6 NA.
6 studies on COPD (2 including other chronic disease) and 1 addressing CKD.	4 studies were about improving self-management of patients through educational materials, or use of monitoring, with support from health providers. 2 studies were about stimulating physical activity through the use of technology. 1 study was about implementing management guidelines in CKD in PHC.	3 The Netherlands. 1 Ireland. 1 UK (Scotland). 1 USA. 1 Australia.	2 Positive. 1 Negative. 4 NA.	0 Y. 5 N. 2 NA.
Overall synthesis of 69 studies in total. 20 depression, 17 diabetes, 25 CVD, 6 COPD and 1 CKD.	Overall, the complex primary care interventions fit within the general categories of facilitating patient self-management (13 studies), organisational change to include collaborative care (16 studies), facilitating better case management using clinical information systems (eg, telehealth) (15 studies), and the use of decision support and guideline implementation (eg, referral systems) (22 studies). In addition, 5 studies were exploring the conduct of trials in PHC, for example, the recruitment of patients.	22 UK. 10 Australia. 12 USA. 6 Ireland. 5 The Netherlands. 5 Canada. 3 New Zealand. 1 Sweden. 1 Germany. 1 India. 1 Norway. 1 Malaysia. 1 Zambia. In addition, 2 studies focused on First Nations peoples in Australia and in Canada. 3 studies were focused on the populations living in disadvantage.	34 Positive. 21 Negative. 14 NA.	10 Y*. 48 N. 11 NA.

*Of note two were full evaluation reports (outcome, process and economic evaluations) in the UK journal of Health Technology Assessments in addressing the question of whether an innovation with limited evidence base in a pragmatic setting (eg, introducing cognitive–behavioural therapy in schools) should be scaled up. Eight papers included descriptions of how cost considerations such as financing incentives/government subsidies impacted on intervention implementation.

CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; CVD, cardiovascular disease; GP, general practices; N, no; NA, not applicable; PHC, primary healthcare; RCT, randomised controlled trial; Y, yes.

Figure 2

Medical Research Council process evaluation framework with tallies of studies, methods and synthesised findings. N/A, not applicable; RCT, randomised controlled trial.

PEs’ strengths and limitations

Description of intervention theory: clear intervention description and clarification of causal assumptions

Thirty papers were assessed as having clear intervention descriptions and clarification of causal assumptions, and in 16 it was unclear because despite clear intervention descriptions the causal assumptions were not described explicitly. An example of a paper that explicitly describes intervention theory is Grant et al, who uses the Template for Intervention Description and Replication (TIDieR) checklist to clearly describe researchers’ assumptions of the intervention’s mechanism as compared with stakeholders’ perspectives.16

Use of existing theories and frameworks

A strength of 22 studies was the use of existing theoretical frameworks to inform their intervention development and/or evaluation (see table 2). Theories and frameworks used are grouped according to Nilsen’s proposed categories.17 This is depicted in box 2, with illustrative examples from the identified studies. In essence, 11 studies used classic theory to inform the development of the intervention theory. In eight studies determinant, implementation theories and evaluation frameworks were used to assist in the synthesis and analysis of qualitative data. The authors of two studies also used their findings and implementation theories to iteratively inform their implementation strategies. The evaluation frameworks were used by study authors to comprehensively evaluate and synthesise their PE data. The MRC framework for complex interventions was used to inform the approach to intervention development in three studies.

Table 2

Summary of the methodology used and quality assessment of the studies

Stage of process evaluation	Methodology and methods	Analysis	Quality criteria
Feasibility/Piloting, 20 studies.	9 studies used theories or frameworks. 18 used interviews. 3 used focus group discussions, 4 used questionnaires or surveys, 2 studies used routine monitoring data, field notes, minutes of meetings and observations.	Thematic analysis, constant comparative approach most commonly used, with some using framework analysis.	Planning: Team description: 11 Y, 6 N, 3 NA. Design and conduct: Purpose: 20 Y. Intervention description and causal assumptions clarified: 5 Y, 6 unclear, 9 NA, 0 N. Justify choice of timing and methods: 19 Y, 1 N. COREQ covered out of the 3 domains (17 applicable studies): 3 domains: 11. 2 domains: 3. 1 domain: 3. Reporting: Clearly labelled as process evaluations: 5. Protocol/full report: 8.
Evaluation of effectiveness, 43 studies.	12 studies used existing theories and frameworks (6 classic theories, 3 evaluation frameworks, 3 implementation theories). 2000–2004: 3 studies documented specific processes of care as part of the process evaluation, which were reported as part of the main trial. 4 studies investigated acceptability of an intervention using surveys/questionnaires. 2005 onwards: 12 studies used interviews alone to explore implementation and acceptability; 20 studies used interviews triangulated with other sources of data (eg, chart audit). 2 studies used routine administrative data to indicate fidelity. 3 studies used questionnaires or surveys.	Descriptive statistics were used for the quantitative data, and thematic, constant comparison and framework analysis for the qualitative data. The studies that used mixed methods used the quantitative data to indicate level of implementation, reach and the dose. This was used to triangulate the qualitative findings on implementation and intervention acceptability. The studies which used evaluation frameworks (eg, RE-AIM) and implementation theories (eg, NPT) used them for the analysis and presentation.	Planning: Team description: 21 Y, 21 N, 1 NA. Design and conduct: Purpose: 43. Intervention description and causal assumptions clarified: 25 Y, 8 unclear, 5 NA, 5 N. Justify choice of timing and methods: 40 Y, 1 N, 2 NA. Report whether the process data are analysed blind to trial outcomes/or post-hoc: 29 Y, 7 N, 7 NA. COREQ covered out of the 3 domains (30 applicable studies): 3 domains: 12. 2 domains: 13. 1 domain: 5. Reporting: Clearly labelled as process evaluations: 17 (of note: 2 before 2008, 6 until 2015 and 9 after 2015). Protocol/full report: 21.
Post-evaluation, 6 studies	1 study used existing theory. 2 studies used interviews, 2 used documentary analysis, and 1 used the administrative data and registry data.	Descriptive statistics, subgroup analysis and thematic analysis.	Planning: Team description: 3 Y, 2 N, 1 NA. Design and conduct: Purpose: 6 Intervention description and causal assumptions clarified: 0 Y, 2 unclear, 2 NA, 2 N. Justify choice of timing and methods: 5 Y, 1 N. COREQ covered out of the 3 domains (3 applicable studies): 3 domains: 1. 2 domains: 1. 1 domain: 1. Reporting: Clearly labelled: 0. Protocol/full report: 1.

COREQ, Consolidated criteria for Reporting Qualitative research; N, no; NA, not applicable; NPT, normalisation process theory; RE-AIM, reach, efficacy/effectiveness, adoption, implementation and maintenance framework; Y, yes.

Theory of planned behaviour. ‘Using the theory of planned behaviour (TPB), we hypothesised that changes in thiazide prescribing would be reflected in changes in intention, consistent with changes in attitude and subjective norm, with no change to their perceived behavioural control (PBC), and tested this alongside the RCT…A strength of this study is its use of a well-tested theory of behaviour operationalized according to best recommended practice to investigate the underlying mechanisms of an implementation intervention’ (Presseau). This theory informed their process evaluation to explore if their intervention of printed educational materials increased practitioners’ intention to prescribe according to recommendations in the guidelines.

Self-determination theory. ‘self-determination theory which proposes that real shifts in behaviour arise through heightened autonomy or personal ownership of behavioural success’ (Chalder). This theory informed their theoretical model underpinning their intervention to improve physical activity for the management of depression.

Grounded theory. ‘This qualitative study was conducted with the objective of better understanding the PP (Prevention practitioner) intervention in the BETTER Trial described above, including the development of the PP (prevention practitioner) role, perceived barriers, facilitators, benefits and disadvantages, and of exploring the feasibility and sustainability of this approach for CDPS (Chronic disease prevention and screening)’ (Manca). This study used grounded theory to better understand their intervention as implemented and to retrospectively describe their intervention theory.

Diffusions of innovation. ‘Key principles, which derive from diffusion of innovations theory, include working initially with practices and clinicians that not only have an interest in the innovation and view it as compatible with their needs, values, and resources, but also have the ability to try it with minimal investment and observe its impact’ (Dietrich). The theory was used to inform their practice change strategy for the sustainability of a chronic care model for depression proven effective in an RCT.

PARIHS as an implementation model. ‘We used the Promoting Action on Research in Health Services (PARIHS) framework as an ‘Implementation model’ to assist clinical partners in adopting the health-coaching intervention. The PARIHS framework posits three interrelated elements that influence successful implementation of evidence-based practices: the (i) perceived strength of the ‘evidence’, (ii) ‘context’ of the environment and (iii) ‘facilitation’ support created for implementation of the intervention…Using a codebook developed a priori from sub elements of the PARIHS framework’ (Naik). This study used PARIHS to inform their participatory approach between research team and primary healthcare teams, and also used it in evaluation of the qualitative data in assessing the building of the partnership to test and implement a health-coaching intervention.

Normalisation process theory. ‘as part of mixed-methods process evaluation, semi-structured interviews were conducted by phone with 27 providers participating in the study. Interviews were audio-taped and transcribed. Thematic content analysis was used to identify themes. Themes were categorized according to the four domains of Normalization Process Theory (NPT)’ (Vest). The authors discuss how the findings are informing their ongoing implementation strategies, for example, clinical mentors for the general practitioners who described a discomfort in their lack of expertise in screening and managing early chronic kidney disease (other papers include Burridge, Coupe, Gask, Hanley, Vest).

Medical Research Council’s (MRC) framework for complex interventions. ‘The MRC framework provided a useful structure through which to examine our theoretical hypothesis and analyse the feasibility evidence’ (Sturt). ‘Guided self-help intervention was developed following a modelling phase which involved a systematic review, meta synthesis and a consensus process…’ (Lovell). The authors used the MRC framework for intervention development. Similarly Byrne et al also used the MRC framework for intervention development of literature review, focus group discussion and modelling and then interviews to refine the intervention.

RE-AIM. ‘The process evaluation followed the RE-AIM (Reach, Efficacy/effectiveness, Adoption, Implementation and Maintenance) framework. Data were collected on attendance and attrition for classroom-based CBT and attention control PSHE (Personal, Social and Health Education) by programme facilitators. An independent observer attended 5% of classroom based CBT sessions to assess treatment fidelity. Feedback was gathered from teachers, young people and facilitators using questionnaires and qualitative interviews’ (Stallard) (other papers include Stallard, Wozniak, Lakeverld).

Realist evaluation. ‘All data assigned to codes relating to the polypill strategy in CVD management were analysed…and the Realist framework of context–mechanism–outcomes utilized to develop the themes’ (Liu). The framework was used to guide the analysis of the qualitative data.

The use of theoretical frameworks seems to enable an indepth investigation of stakeholders’ perspectives of the perceived mechanisms of the intervention, by in a sense providing a checklist of actions and behaviours to be examined.18–28 An illustrative example is the PE of a trial in improving primary care referrals of patients with diabetic retinopathy to specialists through the use of educational printed materials.28 A behavioural theory was used to inform the design and use of a questionnaire to explore the mechanism of the intervention. It was found that the primary care providers’ intention to refer patients was the same before and after the trial, and this may have explained their negative trial results. The authors highlighted that the use of existing behavioural theory enhanced the ‘generalisability and replicability’ of their methods.

Interaction with contextual factors

In 14 papers the interaction of the intervention and contextual factors was explicitly explored. As mentioned above, theoretical frameworks often facilitated a closer and systematic way to consider context. For example, the authors examined if there was ‘contextual integration’, that is, organisational changes necessary to integrate a collaborative model of care for depression into routine practice.29 Otherwise, contextual factors (eg, impact of the introduction of a new policy30) were reported retrospectively in some papers in a more ad hoc manner, for instance being reported as implementation facilitators and barriers, or discussed as possible influences on the outcomes.

Methods used

Assessing the studies based on our bespoke assessment tool, we found that most authors (64/69) clearly justified their choice of methods and stated the studies’ purpose, and most of the qualitative studies (42/50) were of a reasonable quality, with 24 of 50 studies covering all three domains of the Consolidated criteria for Reporting Qualitative research (COREQ) and 18 of 50 studies covering two domains. The methods could be categorised as qualitative studies (eg, interviews, focus group discussions, documentary analysis), quantitative (eg, processes of care, baseline demographics, secondary outcomes), and studies which presented the synthesis of qualitative and quantitative data sources to indicate implementation, acceptability, fidelity and reach (see table 2 and online supplementary appendix 4 for more details).

A strength of some studies was the triangulation of quantitative indicators with the qualitative findings of the acceptability and implementation of the intervention to determine intervention fidelity (ie, whether the intervention was delivered per protocol) (see online supplementary appendix 3 for more details).31 32 The data sources indicating intervention fidelity included routine administrative data, trial/study management logs21 and trial secondary outcomes.33–35 Innovative indicators of e-health interventions included recording process measures such as time logged on by participants.36 Another method to determine intervention fidelity across multiple sites was having independent expert assessors reviewing intervention delivery using standardised forms. Three studies investigated ‘for whom’ an intervention had an impact on with the use of logistical regression of baseline demographics to identify relationships of participants’ characteristics with the primary or secondary outcomes.37

Sampling limitations in the qualitative studies were described as potentially introducing bias in the findings about intervention acceptability/mechanisms.18 19 23–25 28 38–42 For example, authors highlighted that respondents who having agreed to be interviewed may have a more favourable opinion of the intervention.18 38 43–45 Maximum variation sampling (types of participants, sociodemographics, by ‘negative’ baseline of outcome characteristics’), comparing the characteristics of participants who did not partake in the interviews/surveys with the participants who did, and triangulation with other data sources may increase the robustness of such findings.19 20 22 28 38 39 46

PE findings under mechanisms, implementation and context

Does the intervention fit local needs?

Stakeholders were generally motivated to adopt/implement the complex intervention if it addressed the contextual gap in care, that is, intervention fit. For example, a nurse-led secondary prevention clinic was implemented effectively when the health providers perceived it as improving teamwork, care continuity and providing a ‘safety net’ for the patients. In contrast, at other sites, this intervention was poorly implemented by the healthcare providers who viewed it as duplicating the existing model of care.47 As another example, general practitioners reported that training them to manage acute and discrete episodes of depression did not improve their management of depression. This was because this training did not upskill them for the chronic and relapsing nature of depression associated with personality and social problems increasingly seen in primary care.41 48 Similarly, patients’ health literacy about their chronic disease (eg, effectiveness of lifestyle modification for diabetes) was crucial as it affected engagement with the PHC services and their uptake of the intervention.22 25 49–51

Do key actors believe in and adopt their ‘assigned’ roles and responsibilities?

The extent to which key actors believed in and adopted their ‘assigned’ roles and responsibilities as part of implementing the complex intervention was a key theme under the heading ‘Implementation’.21 26 27 42 43 48 52 For example, in a study which used telemonitoring to improve management of patients with chronic obstructive pulmonary disease in the community, there were differing views of the role of the patient. Some health providers described concerns that telemonitoring would reinforce the ‘sick role’ of the patient, and an over-reliance on technology and practitioner support. Therefore, they were less willing to implement this model of care. On the other hand, some patients described that telemonitoring was empowering as it provided knowledge and increased access to health practitioners who could provide reassurance in the management of the disease, and were thereby keen to continue this model of care.22

Facilitators to improve key actors’ uptake of the interventions included the provision of intense training over a transition period prior to the start of the trial, significant research support, and ongoing communication with the researchers to help identify key actors’ concerns and tailor implementation strategies to address them. For example, ensuring adequate communication between nurse practitioners and general practitioners was essential in task-shifting models of care. This facilitated greater trust between nurse practitioners and doctors, which was needed to effectively deliver collaborative services.42 52 Such strategies were especially relevant for collaborative care interventions where new tasks were introduced within established hierarchical systems and interaction between different stakeholders was necessary for effective implementation.

Is the context of the intervention conducive?

Health system structures such as governance, health financing structures and workforce were often mentioned as impacting on intervention implementation. Governance structures were pivotal to the successful adoption of the intervention.23 33 For example, an intervention to enhance referrals to mental health services was implemented well at a site when the intervention was perceived as ‘service delivery’ and directly supported by the mental health trust. In comparison, uptake of the intervention was limited when the intervention was not viewed as ‘service delivery’ and was considered ‘primary research’.33 Similarly, cultivating a strong partnership between researchers and clinicians through the formation of clinical advisory teams facilitated intervention implementation in bureaucratic and geographically complex environments.23 Limited workforce, equipment shortages and inadequate funding structures were reported by several authors as barriers to the intervention adoption. For example, health providers stated that the lack of government reimbursement for allied health services reduced the acceptability of the telehealth model of care for ongoing monitoring of diabetes at home.39 General practitioners reported that time constraints in their busy practices prevented them from using the skills they learnt through an educational intervention to better manage depression.41 Likewise, macro-level context, such as medication being out of stock in rural Zambia, was a barrier to better outcomes, in spite of the provision of an evidence-based intervention to improve clinical assessment and management.32

Importantly, an iterative collaborative approach was described as a facilitator of intervention fit.18 36 45 50 51 53 54 For example, study authors described how early stakeholder involvement identified the key characteristics of the lay worker needed (ie, female, with visibility in the community) for their intervention to improve mental healthcare in India. This preparatory phase in the development of their model of care led to a definitive RCT with positive outcomes. In their PE of the RCT, they found that the provision of a lay worker was not relevant for the primary care practitioners in private practice who had established therapeutic relationships with their patients, but more so for the health providers in the public system who were time-poor. These findings would then inform future scale-up of the intervention within the right context (ie, public health system) for the intervention.53 55

Discussion

Statement of principal findings

To our knowledge this is the first systematic appraisal using MRC guidance on PEs of primary care interventions. Of 69 studies, 66 were conducted in high-income countries, while cardiovascular disease, diabetes and depression were the most frequently studied conditions. There was an overall lack of consistency in the way PEs were conducted and reported. Indeed there was a lack of consistency in nomenclature, with only 47 of the 69 studies identifying as ‘process evaluations’, although their purpose was essentially as such. Few studies (n=30) were underpinned by an intervention theory, description of hypothesised intervention mechanisms of action within local contextual factors. Most studies used robust sampling strategies and frequently triangulated qualitative and quantitative data to better understand the mechanisms of implementation. The MRC PE guidance with its focus on the interaction/configuration between context, implementation and mechanisms of intervention provided a useful framework for the synthesis of the findings. Mapping our findings on the framework also gives our findings and assessment tools greater validity. The findings of these studies can be synthesised into a number of key messages: (1) that often there was a fundamental mismatch between what the intervention was designed to achieve and local needs; (2) that the roles and responsibilities of key actors required to implement the intervention were often not clearly understood; and (3) that health system context—factors such as governance, financing structures and workforce—were often critical to implementation, and as a consequence there were a number of studies where the unanticipated influence of these adversely impacted on implementation.

Comparison with other literature and implications

A key finding is identifying the breadth of literature which fits the MRC definition of PE. This highlights the growing scope in this field to potentially address the evidence to practice gap through greater understanding of the interactions between intervention mechanisms, context and implementation.12 56 57 However, greater consistency is needed in the reporting of PEs—as this would facilitate evidence synthesis, prevent research duplication and enhance transferability of interventions to other settings.58 We note that the consistency in reporting seems to have increased since the publication of the MRC guidance.

An important finding is that theoretical frameworks helped guide a more indepth development of intervention theory, design and implementation.12 59 The MRC PE guidance suggests that PE can help to explain the outcomes variations, and by doing so help refine the intervention theory.17 We note that given the growing focus on self-management for chronic diseases, the theories around behavioural change (eg, empowerment) were most commonly used. Second, the focus on organisational change and the adoption of guidelines in NCDs meant that implementation theories such as the normalisation process theory (NPT) were particularly relevant. Therefore, there should be more consistent use of theoretical frameworks, recognising that different frameworks will be applicable to different settings. In addition, the use of checklists such as the TIDieR or the Standards for Reporting Implementation Studies (StaRI) will ensure consistency in the reporting of intervention theory and implementation, thus reducing research waste.56 60 61 We also note that there were only six ‘post–evaluation’ studies identified. This is not surprising as implementation and translational research is a more recent phenomena, and there may have been less funding for such work and greater difficulties in publishing such findings, a point likewise raised in the MRC guidance report in 2015. We anticipate, however, that more implementation research will be published as this emerging field matures.

We found that the intervention interaction with dynamic contextual factors was often inconsistently reported or reported retrospectively in an ad hoc manner. This gap has been similarly reported in the literature.62 These findings emphasise MRC PE guidance’s value in explicitly appraising context through examining factors ‘that shape theories of how the intervention works’, and affect implementation, and act to ‘sustain the status quo, or potentiate effects’.4 However, this guidance is relatively broad and non-specific, and the question remains as to what should be explored a priori and how best to report such findings. For example, the Context and Implementation of Complex Interventions framework highlights seven domains of context (‘geographical, epidemiological, socio-cultural, socio-economic, ethical, legal and political context’) that could be examined.56 Similarly, the StaRI checklist has context as an item in the methods (ie, ‘consider social, economic, policy, healthcare, organisational barriers and facilitators that might influence implementation elsewhere’) and in the results (‘contextual changes (if any) which may have affected outcomes’).60 These domains are comprehensive, and as a consequence if a study is to examine only a subset of these factors, it is better that this is prespecified in full acknowledgement of the evaluation as a whole. This should be consistently reported and linked through a full report or reference to a protocol.4 As a baseline, a standardised PHC template informed by the questions ‘Does the intervention fit local needs? Do the key actors believe in and adopt their ‘assigned’ roles and responsibilities? Is the health system context (looking specifically at health workforce, governance, health financing structures and availability of medications) conducive?’ and relevant implementation theories (eg, NPT) could be presented for testing in a systematic way. This could be done by PHC researchers engaging with stakeholders at various time points, and iteratively added to.63–65 Such an approach could potentially facilitate a greater shared understanding between stakeholders and greater consistency in the reporting of context.62 64 66–68

Most of these studies were conducted in high-income countries with established PHC systems and universal health coverage (eg, National Health Service in the UK). Therefore, some primary care interventions (eg, improving referrals in collaborative care) may be of limited relevance to low-income and middle-income countries (LMIC) PHC systems given the different context especially with regard to health system structures.69 70 This reinforces the need for more formative research with local stakeholders when developing evidence-based interventions so as to address local needs, and minimise the unanticipated consequences of health system factors.71 72

Strengths and limitations of this study

We were unable to conduct a subgroup analysis of implementation findings by country context (ie, of high-income countries as compared with LMIC) as we identified studies conducted mainly in high-income countries. Some studies conducted in LMIC initially identified in the search were excluded because they did not meet our criteria (eg, not RCTs, not on NCDs), and as such a review with different inclusion criteria may be better suited for this secondary objective. A second limitation is that despite our extended search terms to include components of chronic care model, and similar terms as PEs, we may not have exhaustively captured all PEs of complex primary care interventions. We note, however, that we reached thematic saturation in our analysis across the spectrum of studies, which provides greater validity to our findings. Another limitation is that we appraised the studies using a tool which we developed based on the MRC guidance,4 which has not been tested elsewhere. This was challenging given the heterogeneous studies that were included. For example, we only assessed qualitative methods with COREQ, and did not appraise the quality of statistical methods such as modelling. A strength of this review is having a multidisciplinary team of authors with vast experience in clinical trials and PEs to enable a reflexive thematic synthesis and interpretation of the papers.73

Conclusion

Greater consistency is needed in the reporting of, and the methods used in, PEs. In particular there should be more consistent use of theoretical frameworks to inform intervention theory, and the triangulation of qualitative and quantitative data. Greater emphasis on formative research in designing primary care interventions is needed so that they are clearly aligned with the needs of local stakeholders, that the roles and responsibilities of key actors are better understood, and that unanticipated consequences arising from context-specific barriers to implementation are minimised. We hope this review will inform future PEs and facilitate the sustainability of evidence-based interventions.