Realist evaluation of the antiretroviral treatment adherence club programme in selected primary healthcare facilities in the metropolitan area of Western Cape Province, South Africa: a study protocol.

INTRODUCTION: Suboptimal retention in care and poor treatment adherence are key challenges to antiretroviral therapy (ART) in sub-Saharan Africa. Community-based approaches to HIV service delivery are recommended to improve patient retention in care and ART adherence. The implementation of the adherence clubs in the Western Cape province of South Africa was with variable success in terms of implementation and outcomes. The need for operational guidelines for its implementation has been identified. Therefore, understanding the contexts and mechanisms for successful implementation of the adherence clubs is crucial to inform the roll-out to the rest of South Africa. The protocol outlines an evaluation of adherence club intervention in selected primary healthcare facilities in the metropolitan area of the Western Cape Province, using the realist approach. METHODS AND ANALYSIS: In the first phase, an exploratory study design will be used. Document review and key informant interviews will be used to elicit the programme theory. In phase two, a multiple case study design will be used to describe the adherence clubs in five contrastive sites. Semistructured interviews will be conducted with purposively selected programme implementers and members of the clubs to assess the context and mechanisms of the adherence clubs. For the programme's primary outcomes, a longitudinal retrospective cohort analysis will be conducted using routine patient data. Data analysis will involve classifying emerging themes using the context-mechanism-outcome (CMO) configuration, and refining the primary CMO configurations to conjectured CMO configurations. Finally, we will compare the conjectured CMO configurations from the cases with the initial programme theory. The final CMOs obtained will be translated into middle range theories. ETHICS AND DISSEMINATION: The study will be conducted according to the principles of the declaration of Helsinki (1964). Ethics clearance was obtained from the University of the Western Cape. Dissemination will be done through publications and curation. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

Antiretroviral treatment adherence clubs, aiming at engaging patients and staff in a long-term relationship to improve adherence to treatment, have proven to be effective in pilot settings in South Africa.

Realist evaluation is a methodological approach that allows one to explore how and in which conditions such adherence clubs can be scaled up.

This paper presents the research protocol of a realist research programme that will assess the implementation and effects of facility-based adherence clubs in the Metro area of the Western Cape Province (South Africa).

Through empirical research in five settings, we will develop a programme theory that explains how adherence clubs lead to higher retention in care and better treatment adherence of HIV patients.

Applying a realist evaluation approach can be challenging, and this study will contribute to methodological development by operationalising methods to use the Context-Mechanism-Outcome configuration in the analysis of multiple cases.

Introduction

South Africa is home to the largest number (6.8 million) of people living with HIV/AIDS (PLWHA) in the world.1 The South African government consequently embarked on the fight against the AIDS pandemic through various programmes. As a result, an estimated 3.1 million (32.2%) PLWHA in South Africa have been initiated on antiretroviral therapy (ART) as of April 2015,2 representing the largest ART programme in the world.3 The challenge that the South African ART programme now faces is retaining these patients in care and ensuring that they continue to adhere to their ART medication. In early 2011, the adherence club model was adopted by the Department of Health of the Western Cape Province (WCP) for phased roll-out, initially in the Cape Town Metro to address issues of retention in care and adherence among stable patients on ART.

Background

Adherence—starting, managing and maintaining a given medication regimen at prescribed times, frequencies and conditions—is acknowledged to play a crucial role in determining the success of HIV care and treatment programmes.4 5 Although perfect adherence is recommended for patients using ART, sustained long-term adherence to ART is seldom achieved. According to Bangsberg, with moderate adherence to potent regimens, virological suppression is still possible.6 Nevertheless, achieving even moderate adherence in patients on ART remains challenging. A meta-analysis of adherence studies, with adherence to ART, defined as taking 95% or more of prescribed pills, shows that in sub-Saharan Africa the pooled patient adherence rate is 77%.7 Viral suppression, reduced disease progression and mortality improve with every increase in adherence level.6 Strong evidence suggests that poor adherence to ART leads to potential viral non-suppression, which risks the immediate health of the patient and could contribute to drug resistance.8 Non-adherence is now considered a significant public health challenge, as it can promote the development and transmission of drug-resistant HIV viruses.9 In addition to viral non-suppression, low adherence to treatment has been associated with higher hospitalisation rates, productivity loss, disease progression, low CD4 count recovery rate and death.10

While adherence is crucial to obtaining good clinical outcomes for patients undergoing ART, achieving a sustained engagement of the patients undergoing antiretroviral treatment and care to the care umbrella is equally critical.11 The WHO defines sustained engagement or retention in care as “the engagement in a comprehensive package of prevention, support and care services irrespective of the particular clinic site.”12 For patients who are on ART, retention in care ensures ongoing receipt of ART, assessment of possible medication toxicities, and tracking treatment failure when it occurs in order to take the necessary action.13 Retention in care should also include access to adequate psychosocial support and providing the secondary prevention message that can guide the patient towards adapting their lifelong condition around their lifestyle. Failure to retain patients on ART in care often leads to medication cessation or non-adherence, and immunological failure.

It is observed that non-adherence usually occurs in patients who are not successfully linked to care and treated following diagnosis and in patients who fail to incorporate their treatment into their daily lives.14 A number of complex and sometimes interrelated factors are responsible for this. These factors are categorised as individual, socioeconomic, medication-related and healthcare system (structural) factors.15 Individual factors include forgetfulness, substance abuse, adverse social events, perceived social support, health literacy, self-efficacy and mental health.16 17 Socioeconomic factors affecting adherence include poverty, food insecurity and stigma.18 19 Medication-related barriers to adherence include side effects and dosing.20 With regard to the health system, it is argued that the mainstream (average) clinic often presents challenging treatment circumstances ranging from transportation issues, staff shortages, long waiting times, negative experiences with clinic staff and medication stock-outs.15 21 22 These factors constitute barriers to returning to the healthcare facility for scheduled follow-up consultations and maintaining long-term ART adherence among patients.

Various strategies have been adopted to overcome some of the challenges cited above. While decentralising ART treatment and care services, and shifting aspects of the ART care programme to nurses and other non-clinical staff, including the patients themselves, seemed to work provisionally,23–25 it is anticipated by Luque-Fenandez26 that these strategies are reaching their limits in the face of the foreseeable increase in the number of patients who are being initiated on ART. To provide a more sustainable solution, new models of care for ensuring total adherence of patients to lifelong ART across the sub-Saharan region have been developed. Prominent among these care models are group-based care models, which either operate by recruiting patients into groups, at the clinic (facility-based) or detached from it (out-of-clinic).27 The primary objective of these care models is to improve access to ART medication and to offer psychological and emotional support to patients, which encourages long-term adherence to medication and retention in care.28

Group-based ART treatment care models seek to improve access to medication and retention in care rates, and they strive to achieve this through service decentralisation and task shifting, providing safe and simple ART regimens, and eliminating regulatory or logistical constraints on ART delivery. Therefore, they promise to provide long-term retention in care for patients on ART by providing quick and patient-friendly access to treatment and care while decreasing the burden on overstretched healthcare facilities.29 While the fundamental principles of group-based models of care are the same, they need to be adapted to their context to address the main issues plaguing different communities.28 A review of the literature on the effectiveness of these group-based ART services indicates that most of them contribute to reduced burdens for patients and the health system, increased retention in care and lower service provider costs.30–33

The adherence club intervention

Patients entering the treatment cascade, whether as an early, a delayed or a returning patient, may need support to understand the ramifications of lifelong adherence to medication that can have varying side effects.27 Once their health is stabilised and their immune status is improved, most patients no longer require intensive clinical care and frequent visits to a health facility. These are described as ‘stable’ patients.27 The key elements of care packages for these stable patients are twofold: clinical and operational. The clinical priorities of ‘stable’ patients include sexual and reproductive health, immunisation, nutritional support and tuberculosis (TB) screening, while their operational priorities include retention interventions, viral load monitoring, adherence support and provision of ART.34 35 The adherence club intervention operates by providing the operational services to the stable patients on the HIV treatment and care continuum (figure 1).

Figure 1

The intervention point of the adherence club intervention.

The adherence club intervention could be thus viewed as an ancillary intervention to the standard clinic HIV treatment and care process as it focuses primarily on patients identified as ‘stable’. Patients recruited into the adherence club programme must be 18 years or more, on the same ART regimen for at least 12 months with the two most recent consecutive viral loads of the patient undetectable, and have no medical condition requiring regular clinical consultations more than once a year. These patients are also expected to have a CD4 count value of 200 cells/mm3 or more. When these conditions are met by the patient, programmatically, the patient is described as ‘stable’ and the consulting clinician takes the decision to recruit the patient into the next available adherence club.

An adherence club is limited to 25–30 patients, facilitated by a lay counsellor or community health worker and overseen by a professional nurse (club nurse).32 The club facilitator provides a quick clinical assessment of the patients, performs pill checks, provides emotional and psychosocial support and, where necessary, refers the patients to a clinician.27 Adherence clubs have a number of specific attributes. They enable PLWHA to access a continued ART supply. They create opportunities for establishing collaborations among the group members, leading to peer support.36 Adherence clubs also empower PLWHA through self-management and provide a community network for tracing patients not attending their club. They ensure continued access to clinical care and support through a suitable referral mechanism, which leads to better care, and thus improve retention in care and viral outcomes for the patients.

Various variations have been modified with the goal of taking the care of the patients closer to the homes of PLWHA, to improve service decentralisation. From a health system’s perspective, the adherence club intervention has been shown to be beneficial in producing better retention in care than the standard care,26 reducing the burden that stable patients represent for the health facility,27 diminishing the waiting time of patients in the clinic, and easing the workload of the health workers. Finally, they save the health system and patients' invaluable time and money and improve the rate of identifying defaulters.36

Comparative studies between patients using the adherence clubs and those in the main clinic26 37 showed better retention in care of the patients in the adherence clubs compared to those who remained in the mainstream clinic care. These studies demonstrated the potential effectiveness of the adherence clubs from a quantitative point of view. To enable how to scale up the adherence clubs, however, a proper understanding of how they work, and why (mechanism) and under what circumstances (context) is crucial.

Aim of study

This study will evaluate the implementation and effects of facility-based adherence clubs in the Metro area of the WCP using the realist evaluation approach. The aim is to develop an empirically tested middle range theory that explains the relationships between the dynamics triggered by the adherence club intervention (mechanisms), within different clinical settings (context), and the observed outcomes (overall retention in care and individual adherence to antiretroviral treatment). More specifically, the study objectives are:

To describe the adherence club intervention in five sites

To assess the effectiveness (outcomes) of the adherence clubs in retaining patients in care and promoting individual ART adherence across five primary healthcare centres.

To identify contextual factors that enable or prevent mechanisms to influence the outcomes of the adherence club within selected primary healthcare facilities

To identify the mechanisms through which the adherence clubs achieve the observed outcomes.

Significance of the study

Evaluating the adherence club using the realist evaluation approach will help identify the contextual factors and the mechanisms underlying actors' practices required to generate the desired outcomes. This theoretical understanding is critical for understanding whether the adherence club intervention has been successful in a particular context, and also whether and under what context conditions it can be scaled up or replicated.

Research setting

The Cape Metropole (one of the six subdistricts of the Western Cape Province) is divided into four substructures and each substructure is dichotomised into subdistricts.38 39 Each of the subdistricts has a comparative population. See figure 2 below for the various subdistricts of the Cape Metropole district.

Figure 2

Health subdistricts of the Cape Metropole District.40

On the basis of the HIV prevalence and ART uptake in the City of Cape Town as demonstrated in table 1 below, we purposively selected Khayelitsha, Eastern, Mitchell's Plan, Klipfontein and the Tygerberg subdistricts for inclusion in the study.

Table 1

HIV prevalence and ART uptake in the city of Cape Town39

Subdistrict	Estimated HIV+ Population Total	Estimated HIV+ Population CD4<350	ART service points (October 2009)	On ART (December 2010)
Southern	18 654	9 327	8	5 298
Western	26 201	13 100	12	10 576
Northern	26 164	13 082	4	4 949
Tygerberg	18 532	9 266	4	6 005
Klipfontein	33 423	16 712	4	6 403
Mitchells Plain	20 557	10 278	4	9 574
Khayelitsha	44 772	22 386	11	17 659
Eastern	30 337	15 168	5	7 679
City of Cape Town	218 640	109 319	52	68 143

ART, antiretroviral therapy.

Within the various subdistricts, we purposively selected five primary healthcare facilities. These five facilities are found within four of the eight health substructures of the Metro health district in the WCP. These primary health facilities include the Mitchell's Plain Community Health Clinic (CHC) and Heideveld Community Day Clinic (CDC) from the Mitchell's Plain Health subdistrict; the Crossroads CHC from the Klipfontein health district; the Vanguard CHC from the Western district and the Ubuntu clinic from the Khayelitsha Health District. Being the pilot facility for the adherence club intervention, the Ubuntu clinic will allow examination of how the facility has transitioned from the pilot phase into an implementation phase. It will give us an idea on how adherence clubs work during the pilot phase compared with the implementation phase. The other three facilities, Mitchell's Plain CHC, Crossroads CDC and Heideveld CHC, represent the typical facility in which the adherence club intervention was initially implemented. We wish to explore what other elements could be crucial in determining the success or failure of the adherence club intervention in these facilities. Finally, the Vanguard clinic that implemented the adherence club intervention in September 2014 was selected to allow us to understand the implementation process. Table 2 describes the location of the selected facilities within the various substructures and subdistricts and their characteristics.

Table 2

Study facilities within the different health substructures and subdistricts and their characteristics

Health Substructure	Southern-Western Substructure	Klipfontein-Mitchell's Plain substructure			Southern-Western Substructure
Health Subdistricts	Khayelitsha subdistrict	Mitchell's Plain subdistrict		Klipfontein subdistrict	Southern subdistrict
Facilities	Ubuntu Clinic	Crossroads CDC	Mitchell's Plain CHC	Heideveld CDC	Vanguard Clinic
Characteristics
Adult patients on ARVs in August 2014	±8500	5267	2561	1486	1501
Number of ACs	231	42	39	20	2
Starting date of AC	2007	2012	2012	2012	2014
Number of patients in adherence club care	5900	1401	1309	480	35
Number of ART staff	30	15	11	08	09
Implementation context	Experimental	Roll-out	Roll-out	Roll-out	Roll-out
Predominant catchment population	Black	Black	Coloured	Coloured	Black

AC, Adherence club; ART, antiretroviral therapy; ARVs, antiretrovirals.

Methodology

The methodological approach

This study will be guided by the realist evaluation enquiry. Realist evaluation belongs to the theory-driven evaluation family.41 42 Realist evaluation is an approach grounded in realism,43 a school of philosophy that asserts that both the material and the social worlds are ‘real’ and can have real effects, and that it is possible to work towards a closer understanding of what causes change.

Using the realist evaluation approach, we aim at collating empirical evidence from various sources to develop and test programme theories.44 The realist approach assumes that programmes are ‘theories incarnate’. That is, whenever a programme is implemented, it is testing a theory about what ‘might cause a change’, even though that theory may not be explicit. One of the tasks of a realist enquiry is therefore to make the theories of the actors within a programme explicit in the form of hypotheses or programme theories about how, and for whom, the programmes might ‘work’ (or not). Therefore, the product of each case of the study is thus a refined programme theory, which can be compared and reformulated as a middle range theory, explaining how the programme works (or not), for whom and in what context. A realist evaluation asks ‘what works?’, and also ‘how or why does this work, for whom, in what circumstances?’45 This implies collecting data, not just about programme impacts or the processes of programme implementation, but also about the specific aspects of programme context that might affect programme outcomes, and about the specific mechanisms that might be creating change. The unique nature of realist evaluation lies in its conceptualisation of the central components (context, mechanism and outcome) of a complex intervention.

Realist evaluation uses the term ‘outcome’ to include short-term, medium-term and long-term changes, intended and unintended, resulting from an intervention. For the PLWHA, this could be a change in the experiences of the patients in the programme and a change in their retention in care and adherence behaviour. For the health system, such changes could include changes in the workload of the health workers and the decongested health facilities. Context relates to the circumstances in which the programme is implemented. In realist evaluation, context is considered as the features of the organisation, staffing, history, culture, beliefs, etc that are necessary for the programme to ‘fire’ the mechanism or that prevent the intended mechanisms from firing.46 The context of the PLWHA is key to understanding the development of the ‘generative mechanisms’ and thus the programme theory behind the adherence club programme. With respect to the context, the investigators will look into those context conditions under which the adherence club could potentially achieve greater retention in care rates. Although mechanisms are said to exist independently of the context, a mechanism cannot fire in isolation. Only when mechanisms are activated in a particular context do they exercise their causal power.44 What matters about context in realist evaluation is indeed that it influences the mechanisms to operate. In the realist evaluation perspective, programmes offer resources, opportunities or constraints to the actors.47 The observed outcomes occur because the required resources, opportunities and/or constraints influence the reasoning of the actors (mechanism) within a particular context. In the light of the study, the investigators will explore the various social or psychological drivers that cause the reasoning of both the programme implementers and the users, considering the various resources and opportunities that the adherence club intervention provides.

The relationship between the context, mechanism and outcome is expressed as a context-mechanisms-outcomes (CMO) configuration,48 which is a proposition that can be tested empirically to obtain a refined theory that explains why an intervention works for some but not for others.49

Methods

This study is designed around the realist evaluation cycle as illustrated in figure 3 below. Realist evaluation is method neutral, so we have identified the suitable methods that will be applied in the evaluation process. Our data collection and analysis methods are chosen because they allow the investigators to describe outcome patterns and the context, and identify the mechanisms embedded in the adherence club intervention. They will allow us to uncover the experiences, interpretations and responses of the programme implementers and users involved.51 These methods are described in the section below.

Figure 3

The realist evaluation process and the proposed phases of the study.50 CMO, context-mechanism-outcome.

Phase 1—eliciting the programme theory

The first phase of the realist evaluation involves developing the initial programme theory. A programme theory is defined as ‘a set of explicit or implicit assumptions by stakeholders about what action is required to solve a social, educational or health problem and why the problem will respond to this action.’52 As already mentioned, a programme theory represents a hypothesis that can be tested and further refined.53 Therefore, realist evaluation starts and ends with a theory.

An exploratory qualitative study design will be used in this phase. This will offer us the opportunity to explore and describe the adherence club intervention as conceived by the programme designers and coordinators. Four data collection methods will be used in this phase: a document review of relevant documents on the adherence club programme; key informant interviews, observation of the adherence club, and a review of the evidence on the effectiveness of the adherence clubs.

Through a literature search of various databases and by contacting the Provincial Department of Health, documents will be identified for possible inclusion in the document review. The key informant interviews will be conducted with Médecins Sans Frontières representatives on the adherence club programme, Treatment Action Campaign members, the various HIV/AIDS/sexually transmitted infections (STI) and TB (HAST) unit medical officers (MOs) and adherence club managers; representing the facilities of interest will be included in the study. The observation technique will allow us to see what people are doing at the coal phase of the implementation of the adherence club intervention, rather than rely on what is reported. With regard to the review of available data assessing the effects of the adherence club programme, the investigators are aware of two peer-review articles published on studies that evaluate the adherence club's effectiveness.26 37

The first set of data will be collected by the investigators through a document review process, based on their knowledge on the topic and the relevance of the document to the research aim. Documents such as the adherence club toolkit, reviews on the adherence club intervention, policy documents and implementation guidelines of the adherence club intervention will be consulted. The second data set will be obtained through semistructured interviews conducted by the investigators using interview guides. The interviews will focus on questions related to the design of the adherence clubs, the implementation strategies, why they think the adherence club will resolve issues around adherence and retention in care and the expected outcomes of the intervention. The interview will also seek answers to questions around the primary objective of the adherence club, the resources and dynamics around the interventions. During the observation, we will explore how the adherence club programme is organised and run in practice. This process will allow the investigators to obtain an independent perspective on what is done in the programme. A further review of empirical studies conducted to evaluate the effectiveness of the adherence club intervention will also be performed. This second review will focus on studies that have been conducted to evaluate the effectiveness of the adherence club intervention with regard to the outcomes of retention in care and adherence to medication.

The information from these three sources will be used to formulate the testable programme theories on the adherence club initiative using the CMO pattern configuration.54 The resulting initial programme theory will be ‘tested’ in phase two.

Phase 2—testing the programme theory

Study design

A multiple case study design will be used, in which each healthcare facility is considered a case. The case study design was selected because, first, it is methodologically complementary to realistic evaluation. Second, it allows for the use of multiple methods of data collection, and third, it recognises the importance of context. Within this study design, the convergent parallel mixed method approach of a retrospective cohort analysis and an explanatory qualitative design will be applied to the process of obtaining and analysing the data. While the retrospective cohort analysis will provide an insight into the outcomes (based on routine data captured in the club registers) of the patients receiving care in the adherence club model of care, the qualitative explanatory design will provide the evidence to strengthen every link in the implementation chain. The mixed-method research design will allow for quantitative indicators of the outcomes to be identified and the qualitative exploration of the context and mechanisms that contribute to the observed outcomes.42 The quantitative data will allow the investigators to quantify the outcomes and some elements of the CMO configuration. The qualitative study will allow emergent outcomes and processes to be captured.55

The selection of the study participants will take place at the facility level. During the data collection period at each facility, the goal will be to include all the categories of staff directly involved with running (doctors, nurses, head of the adherence clubs, adherence counsellors) and using (patients) the adherence club programme. At least one programme implementer per facility will be included, and at least five patient interviews will be conducted. The appropriate number of interviews will be determined by thematic saturation.

The data collection process will be complete over an estimated period of 10–15 days per facility. The investigators could revisit any facility to explore issues that would need further exploration through either observation or interviews. The following data collection methods will be applied:

Observation of the activities related to the adherence clubs. The observations and discussions will be recorded in field notes and/or audio recorded as appropriate.

Semistructured interviews with the adherence club implementers (clinicians and lay counsellors of their perspectives of the adherence club intervention and patient interviews about their experiences. Interview topic guides will be informed by the realist framework to elicit information on the following key elements: intervention, actors, outcomes, mechanism and context.

A reflective journal will be kept at the end of each day to capture issues such as the reactions of the interviewees, and other interesting observations that cannot be captured by the audiotapes.

Process tracing56 of the operations of the causal mechanisms in each case will be done through participatory construction of flow loop diagrams during observations.

The data obtained from the above sources will be prepared for analysis accordingly. Field notes will be immediately developed after each interview to describe the context of the interview, the dynamics between interviewer and interviewee(s) and any other impressions of the investigators. The interview recordings will be transcribed professionally and then checked by the interviewer(s).

Data from the provincial Department of Health on the retention in care and adherence outcomes of the patients receiving care in adherence clubs at the selected facilities for the period 2012–2015 will be obtained and prepared for analysis. The focus at this stage will be on obtaining data to describe the defaulting and retention in care behaviours of patients from the adherence clubs (referred back to the mainstream clinic, death or loss to follow-up—unknown outcomes). Data on the time to the first viral rebound (<400 copies/mL) will also be collected on the basis of the clinical attendance records.

Data analysis

The data analysis is divided into three steps: thematic data analysis, identifying the CMO configurations and refining the programme theory.

Step 1: Thematic data analysis

The overall aim of the first-level analysis is to develop themes for classification of data into ‘intervention’, ‘actor’, ‘context’, mechanism’ and ‘outcomes’. The qualitative data collected from the multiple case study will be analysed using the ‘thematic content analysis’ technique for emerging themes as described by Miles and Huberman.57 A deductive analytical approach will be employed, which is suitable if the general aim is to test a previous theory in a different situation or to compare categories at different periods.58 This process will be managed with the use of Atlas.ti V.7.59 A sample of the coding will be checked by the study supervisors to ensure that the mode of enquiry is as balanced as possible and appropriate. The analysis process will entail coding the interviews into themes in terms of intervention, actors, context, mechanisms and outcomes,60 with the aim of identifying the CMO configurations in the next step. The initial programme theory will provide the framework categories (prototype) and analysis will be focused on understanding the ways in which the proposed mechanisms are generated or not generated in practice, identifying alternative mechanisms and explanations. The qualitative data could also identify emergent outcomes and processes in the different contexts. Throughout the coding process, memos will record emerging conceptual links and other observations about the data.

Further information on the outcome of the adherence club programme as an intervention to improve retention in care and adherence will be obtained by conducting a survival analysis on adherence club programme data obtained from the provincial Department of Health. Kaplan-Meier methods will be used to describe the adherence and retention in care rates of patients in the adherence clubs.

Step 2: Identifying CMO configurations

The second-level analysis will start with seeking patterns that will contribute to identifying conjectured CMO configurations.61 62 Configuration analysis will be used on the basis of the CMO typology that was developed by Pawson and Tilley.41 The second-level analysis will be done by grouping the specific outcome, context and mechanism codes to form intermediate-level and high-level codes (conjectured CMO configurations). The context will be analysed in its capacity to enable the conjectured mechanisms to act, or not. This process is informed by the idea that realist evaluation seeks to uncover the underlying generative mechanisms and the context in which this happens to give rise to the observed outcomes.47

Second, the conjectured CMO configurations will be ‘tested’ by confronting them with the data of each case (facility) to check their explanatory power. This will be achieved by representing the data in a tabular form for each higher level outcome under consideration. The patterns (CMO matrices) will focus on what exactly in the programme creates the outcomes, and under what conditions. This is known as in-case analysis.

Step 3: Refining CMO configurations into programme theory

The CMO configurations will be refined in a process in which the CMO configurations obtained in step 2 across the cases are compared and their explanatory power across the cases is examined (cross-case analysis).62 This step involves putting the positive and negative cases in two separate categories. For each active mechanism identified as being associated with a positive outcome, other cases with a positive outcome will be examined, looking for additional elements. In a similar manner, CMOs associated with ‘failed outcomes’ will be grouped together. To ensure the validity of the two final categories of CMO configurations obtained, the investigators will refer back to the detailed case studies and original transcripts for consistency. The final CMO configurations will be verified by employing ‘causal loop thinking’ to map the bigger picture of how the different components interact within the ‘system’ as a whole in order to influence outcomes in terms of retention in care and adherence within the adherence clubs.

Phase 3—translating the refined programme theories into Middle Range Theory (MRTs)

In this phase, the CMO configurations will be compared with the initial programme theory. We will review and compare the results of the individual case studies to see how the initial programme theory can or needs to be modified. The process of moving from the specifics of individual cases to a theory that is more abstract is known as analytical generalisation and it is outlined in figure 4 below.

Figure 4

Analytical generalisation.62

Quality control

Prolonged engagement and persistent observation will be carried out to ensure that the investigators are familiarised with the content and context. Triangulation—using more than one method of data collection and using a wide range of informants and data verification through peer review—will be employed. Credibility will also be ensured by holding debriefing sessions between investigators and superiors and performing member checks of data collected, interpretations and theories formed.

Ethics statement

Authorisation to conduct the study will be sought from the Department of Heath of the Western Cape Province. At the level of the facilities, permission will be obtained from the various facility managers to access the various sites, and finally, consents from the various participants at the sites (doctors, nurses, counsellors and patients) and the key informants (stakeholders) will be sought.

Discussion

Programmes are intended to create change. According to Pawson and Tilley,41 programmes are theory incarnate. In order to create change, programmes make use of this implicit theory about how this change might occur. The task of an evaluator in evaluating a programme, therefore, is to test and understand the programme theory.46 This paper describes the possible use of the realist evaluation methodology for the evaluation of the adherence club intervention that was piloted in 2007 and currently implemented since 2012 in over 400 primary healthcare facilities by 2014 in the metro health district of the Western Cape Province.

During the pilot phase of the adherence club intervention, an evaluation of the 20 facility-based adherence clubs that were established was conducted using a retrospective observational evaluation.26 The investigators compared loss to care and viral rebound in patients receiving the intervention with patients attending routine nurse-led care from November 2007 to February 2011. The study showed that 97% of club patients remained in care compared with 85% of other patients. Another evaluation describing the implementation of community-based adherence clubs at a large public-sector facility in a periurban community in Cape Town, South Africa revealed that after 12 months in the adherence club, 6% of patients were lost to follow-up and fewer than 2% of patients retained an experienced viral rebound.37 These studies provide an insight into the possible effectiveness of the adherence club model; they demonstrate little on the implementation process of the adherence clubs. Therefore, they lack analytical depth and do not present causal explanations.

Arguing for more serious theorising in connection with evaluation, Chen and Rossi63 emphasise the importance of theoretical models in evaluating implementation processes. While ‘black-box’ evaluation methodologies focus on methodological rigour, thus aiming at controlling the contextual factors that might intervene in a study, realist evaluation, on the other hand, explores the role played by the various contexts on the effectiveness of an intervention or a programme.64 65 Realist evaluation systematically tracks outcomes and the actual intervention and explores the contexts in which mechanisms are triggered.18 Realist evaluation thus elucidates the nature of the programmes, the mechanisms that are likely to operate and the contexts in which they might operate to explain how the observed outcomes were attained.41 For the above reasons, the investigators considered the realist evaluation approach as a suitable approach to evaluate the adherence club intervention implemented in the various primary healthcare centres. The selection of the realist evaluation approach was supported by three arguments. First, it provides a framework by which evaluators can systematically deconstruct an intervention into its components and reconstruct it with causal webs that can lead to the observed outcome.50 Therefore, realist evaluation can provide a sound framework by which the context and mechanisms and how they influence the outcome of an intervention could be studied.50 Second, realist evaluation is well suited for the investigation of complex systems (complex adaptive systems) such as health centres where the implementation of the adherence club intervention takes place.66 Third, realist evaluation offers evaluators the opportunity to develop an integrated outcome and process evaluation framework, which can lead to sound decision-making to improve the impact of interventions.67

We recognise that there are some challenges associated with the use of realist evaluation, as pointed out by various authors.49 52 68 Marchal and colleagues found that many researchers using the realist evaluation approach have experienced challenges at various stages. These include debates about the nature of the ‘mechanism’ and the challenge of differentiating between the mechanism and essential context conditions.50 While developing this protocol, the investigators were aware of these challenges. Efforts will be made to address them accordingly.