How does capacity building of health managers work? A realist evaluation study protocol.

INTRODUCTION: There has been a lot of attention on the role of human resource management interventions to improve delivery of health services in low- and middle-income countries. However, studies on this subject are few due to limited research on implementation of programmes and methodological difficulties in conducting experimental studies on human resource interventions. The authors present the protocol of an evaluation of a district-level capacity-building intervention to identify the determinants of performance of health workers in managerial positions and to understand how changes (if any) are brought about. METHODS AND ANALYSIS: The aim of this study is to understand how capacity building works. The authors will use realist evaluation to evaluate an intervention in Karnataka, India. The intervention is a capacity-building programme that seeks to improve management capacities of health managers at district and subdistrict levels through periodic classroom-based teaching and mentoring support at the workplace. The authors conducted interviews and reviewed literature on capacity building in health to draw out the programme theory of the intervention. Based on this, the authors formulated hypothetical pathways connecting the expected outcomes of the intervention (planning and supervision) to the inputs (contact classes and mentoring). The authors prepared a questionnaire to assess elements of the programme theory-organisational culture, self-efficacy and supervision. The authors shall conduct a survey among health managers as well as collect qualitative data through interviews with participants and non-participants selected purposively based on their planning and supervision performance. The authors will construct explanations in the form of context-mechanism-outcome configurations from the results. This will be iterative and the authors will use a realist evaluation framework to refine the explanatory theories that are based on the findings to explain and validate an improved theory on 'what works for whom and under what conditions'. DISCUSSION: The scope for applying realist evaluation to study human resource management interventions in health are discussed.

Introduction

Health worker availability has been associated with better coverage of programmes such as vaccination as well as better outcomes such as reduced child and maternal mortality.1 2 Although the relationship between availability of health service providers and improved mortality outcomes appears straightforward, it is not easy to establish. Issues of health worker performance and their motivation and the contextual factors that shape an enabling environment for health service providers to perform effectively continue to be poorly understood.3 Early studies exploring associations between health worker availability and health outcomes reported results ranging from ‘no significant association with infant mortality’ to positive associations with infant and maternal mortality and even surprisingly, in one study, an adverse association between doctor availability and infant and perinatal mortality, termed ‘doctor anomaly’.4–6 Using improved data and design, more recent cross-country regression-based analysis has shown a positive relation between health worker availability and reduced child and maternal mortality and improved vaccination coverage.7 8

The 2006 World Health Report drew attention to the human element in the delivery of healthcare services by focusing on the health workforce. It identified the forces driving the health workforce (health needs, health systems and contextual factors) and the related workforce challenges (numbers, skill mix, distribution and working conditions).9 A well-performing workforce is considered to be a combination of staff being available (retained and present) and staff being competent (productive and responsive).9 In order to ensure such conditions, the report suggested policymakers to adopt good human resource management (HRM) within the health services. HRM is the management of people in an organisation. It includes the policies, practices and activities at the disposal of managers to ensure the availability of staff in their number, with skills needed to discharge their functions and having the motivation to accomplish the organisation's objectives.10

Suboptimal performance of health workers is a serious issue requiring urgent attention as it is linked to morbidity and mortality, and reviews having shown that health worker performance is critical to achieving good health outcomes across health conditions, age groups and to achieve the health-related millennium development goals.11 12 The world health report suggested four ‘practical and low-cost instruments’ of which supportive, yet firm supervision and lifelong learning are important for a competent and responsive health workforce.

However, the difference made by good HRM in achieving better performance and outcomes of health services is poorly researched. There are serious knowledge and evidence gaps on what kinds of interventions work. This is mainly due to methodological challenges on measuring HRM practices and performance, and the paucity of studies on district-level interventions on health workforce from low- and middle-income countries, where the need for such evidence is most pressing.3 12 On the other hand, several reviews highlight the need for evaluations that can improve our understanding of ‘how’ such interventions work so that HRM interventions may be better designed and implemented.1 3 13 Despite the relevance of this question to policymakers as well as healthcare organisation managers, there are few studies.14

HRM interventions are implemented within existing health systems. Context matters: what works in one setting does not necessarily work in another setting in the same country and may perhaps even not work in the same setting at another moment in time. Evidence on effectiveness of HRM interventions is either scanty or flawed due to poorly designed research.15

Experience from action research in capacity building initiatives in 25 of the (then) 28 Indian states as well as performance reviews of the Indian government's flagship health programme, National Rural Health Mission (NRHM), highlight the need for systemic capacity-building on one hand and scientific evaluations of how interventions work (or do not) on the other.16–18 Paul et al reviewed several studies at both national and local level to identify gaps in the Indian healthcare system; they recommend (among others) ‘…interventions and research to improve decentralised district-level planning in health services’. Given the lack of institutional capacity to use financial or technical inputs especially at the district level, increased health spending even on appropriate services may not lead to actual provision of services.19 Our study intends to address the evidence gap (how do district level training interventions improve performance?) and will contribute to the evidence base for better design of health workforce interventions.

In this paper, we present the protocol of an evaluation of a district-level capacity-building intervention in Karnataka State (India) that aims at responding to the effectiveness question and to the causality question. Inspired by principles of realist evaluation, this study focuses on identifying the determinants of performance of health workers in managerial positions and to understand how changes are brought about.

The capacity-building intervention we assess aims to improve the capacity of health managers to conduct the planning and supervision of health services. These managers are posted at district and subdistrict (taluka) levels (a taluka is an administrative subdivision of a district, with population ranging from 100 000 to 200 000). It does so by combining classroom-based lectures with in-service ‘mentoring’, where trainers and faculty visit participants in their workplace to further build on the classroom teaching and help participants apply the teaching in their working environment.

Methods

Aim

We will carry out an evaluation study of a capacity-building intervention at district level in Karnataka state (figure 1). The aim of the study is to understand how capacity building in district health management works. This study will first describe the structure and nature of the intervention and second design tools to determine whether and how it brought about the changes that it sought to bring about and through what mechanisms these changes were achieved.

Figure 1

Map of India showing Karnataka (shaded red) in south India. Map from Wikimedia Commons/User: Nichalp licensed under Creative Commons Attribution-Share Alike 3.0.

To determine if a district-level capacity-building programme is associated with improvement of planning and supervision practices in Tumkur district, Karnataka state

To identify and describe the plausible mechanisms for changes in planning and supervision practices, if any

To develop recommendations for better design and implementation of capacity-building interventions for health services managers in Karnataka

To contribute to the development of a methodological framework for the scientific evaluation of complex HRM interventions at local healthcare system level.

Research question

Based on these objectives, we framed the following research questions (one main question with three subquestions) to be addressed in the study as follows:

‘How does a training programme for health managers at district level that consists of contact classes and mentoring have an impact on their planning and supervision practices?’

What are the interventions' elements that are associated with improvement of planning and supervision practices?

Was there an association between greater participation in the intervention (classroom training and mentoring) and improved planning and/or supervision practices?

How might a training programme change management practices of health managers with respect to the preparation of annual plans and supportive supervision?

Setting

The study will be conducted in two districts (ie, local healthcare system) of the state of Karnataka in India (figure 2). Karnataka is one of the average performance states in India with respect to health outcomes—the ‘average’ is concealing wide disparities between districts. For instance, in 2008, coverage of immunisation for children was over 90% in Kodagu district, while it was below 50% in Raichur district.20 The study will take place in Tumkur and Raichur districts. Of the 30 districts in Karnataka, Tumkur is the fourth largest in terms of population (total population—2 681 449 people) and the third largest district in Karnataka in terms of size (total area—10 597 sq km) with only 20% urban population and at least half the population recognised as being below the poverty line.21 22 The district has 10 talukas. In view of its large size, average socioeconomic indices and ‘average’ health performance in terms of its outcomes, Tumkur could be considered a typical district of Karnataka. The government classifies Raichur district in northern Karnataka as having several talukas that are ‘backward’, but it ranked 14th among the (then) 27 districts in terms of health indicators. On the same index, Tumkur was ranked ninth.23 These two districts are purposively selected, as they are roughly comparable to each other in terms of health management and outcomes.

Figure 2

Map of Karnataka showing Tumkur district (shaded blue) and Raichur district (shaded green). Map from Wikimedia Commons/User: Planemad licensed under Creative Commons Attribution-Share Alike 3.0.

The intervention

In 2009, Tumkur district was chosen to pilot a capacity-building programme. The programme was implemented by a consortium, Swasthya Karnataka, consisting of five Indian non-governmental organisations, in partnership with the government of Karnataka (see figure 3 for structure of the capacity-building programme, key actors involved and timeline). The programme consists of 12 modules on public health management topics, delivered through classroom teaching for 2 or 3 days/month in a residential training programme for all staff involved in management of health services at taluka and district levels, along with mentoring of these participants on a monthly basis at their workplace. One of the main objectives of the intervention was to improve planning and supervision practices of health managers through providing knowledge of public health planning principles, improving their skills in planning and supervision as well as bringing about a can-do attitude towards organisational change. The programme began in August 2009; the monthly contact classes for health managers ended in January 2011 and mentoring is in progress as of December 2011.

Figure 3

Schematic showing the structure of the capacity-building intervention in Tumkur along with key actors and timeline.

Study design

Marchal24 reviewed the methodological debate around the use of (quasi-) experimental study designs in complex interventions and scientific evaluations in health systems research. He builds a case for using the realist evaluation approach in research on complex interventions in health systems. He presents the results of a realist evaluation of the role of workforce management in well-performing healthcare organisations and identified some mechanisms underlying the better performance of these well-performing hospitals.24 In line with this approach, we will carry out a realist evaluation of the capacity-building programme in Tumkur, using a mix of quantitative and qualitative methods. The characteristics of the intervention that support the choice of realist evaluation are presented in the Discussion section (see below).

Our study design is determined by the following considerations:

Classical controlled (quasi-) experimental designs are limited to answering whether a particular intervention (usually measured as treatment variables) was associated with an observed pre-defined outcome. They do not answer the questions how, why and under what conditions the intervention worked (or did not). Besides enabling an understanding of the changes in planning and supervision practices in course of the intervention, the study design should also generate valid explanations for why and how the results observed were achieved.

HRM interventions are implemented in existing health system settings. Hence, the researcher cannot manipulate all treatment variables for the purposes of testing a priori hypotheses either because the context of the intervention does not support this or for ethical reasons. Although hypothesis testing should be central to discovery of the mechanisms, such hypotheses should be derived from the possibilities permitted by the context within which the intervention is being implemented.

In order to understand whether and how the intervention produces a change in managerial practices at the district level, we will carry out the study in six steps. In figure 4, a schematic shows the sequence of steps (steps A, B1–2, C, D, E and F) with the questions that will be addressed at each step and the corresponding methods.

Figure 4

Study design showing steps A to F.

The various phases of our study design follow the logic presented in the six-step framework developed by Van Belle et al.25 The six steps they describe refer to a theory-driven evaluation where evaluators reconstruct the assumptions based on which the programme was designed (programme theory) in order to refine it through testing and verifying. Based on this process, an improved programme theory is developed, which explains how the intervention and outcome are related. Realist evaluation is a type of theory-driven evaluation26 that generates a theory explaining the mechanisms through which the outcomes were brought about in a given context. We found the steps used by Van Belle et al useful to organise and describe the steps in this study. The steps A–F below refer to the steps in our design as shown in figure 4; the six steps of Van Belle et al are referred to as numbers (steps 1–6, see figure 5). The scope of the evaluation and appropriateness of realist evaluation (corresponding to step 1 of Van Belle framework) is presented in the Discussion section (see below).

Figure 5

Six steps proposed by Van Belle et al.25

The study starts with a reconstruction of the initial programme theory of the intervention (step A in figure 4) corresponding to steps 1 and 2 of the Van Belle framework. A programme theory that may be presented in the form of a logic model is a reconstruction of the assumptions and steps through which the intervention is expected to reach the expected outcomes. An initial programme theory will be the starting point for the study by providing a basis for the questions and tools of the subsequent qualitative and quantitative data collection phases. In figure 6, a simplified hypothetical causal chain based on the programme theory is presented. It links the intervention inputs (contact classes and mentoring) to the expected outputs (improved planning and supervision practices).

Figure 6

Hypothetical pathways to change based on initial reconstruction of programme theory and literature.

In steps B and C, we will use a mix of qualitative and quantitative methods to understand the process of planning and supervision and whether and how it changed in the course of the intervention.27 In step B, we will measure perceptions about training, planning and supervision, organisational commitment, self-efficacy in problem solving and nature of supervision among participants and non-participants through a survey in Raichur and Tumkur districts of Karnataka. Organisational change in health services is an outcome of individual, institutional and contextual factors. Existing theories of behavioural change in health services conceptualise that interventions operate at one or more of these three spheres of influence (figure 7).

Figure 7

Theories of behavioural change in health services in relation to their sphere of influence. Adapted from Rowe et al.12

A hypothetical causal pathway (figure 6) that links the intervention inputs and the outputs and a review of literature (figure 7) on what we know about HRM interventions were used to choose the variables and design the tools for the survey.

In step C, we will use qualitative methods to document and understand the changes in planning and supervision practices before, during and after the intervention in Tumkur district. In this phase, we will also determine the contextual factors that influence planning and supervision in the district, especially other programmes initiated by the state health authorities that have similar or overlapping objectives with the intervention. The NRHM is a nation-wide initiative of the Indian government that seeks to improve district-level planning and supervision and implements this through the creation of a district and taluka programme management unit. The NRHM introduced technical and human resource inputs into the health system in the form of decentralised annual action plans and placement of young management professionals at taluka and district levels for planning and supervision of the plans.

The data from steps B and C will be analysed and interpreted together in step D to understand the relationships between the elements of the initial hypothetical causal chain. This will result in an improved theory linking the inputs, intermediate steps and the effect of contextual factors. We will then formulate—in step E—explanatory context–mechanism–outcome configurations based on the interpretation in step D that will be validated through a fresh round of data collection using qualitative methods. An iterative analysis of findings from steps C, D and E will be conducted so as to build an internally consistent and valid explanation in step F on ‘what elements of the intervention worked, for whom and under what conditions’. The last three steps in our study (steps D, E and F) correspond to the last three steps of the Van Belle framework.

Methods and tools

Realist evaluation is method neutral; it allows for the use of mixed methods, whereby the choice of data collection and analysis methods is determined by the nature of the research questions and of the programme theory.28 The methods and tools for data collection are determined by each step (qualitative or quantitative) and the nature of questions asked at this step (see schematic in figure 4). A summary of the tools and expected outcomes at each step is shown in table 1.

Table 1

Details of the tools, sampling and expected outcomes

Step	Methods/tools	Sampling/selection of respondents	Analysis and expected outcome
Step A: reconstruction of programme theory	Desk review of intervention design, proposal, annual district-level plans, reports and interviews with the people who designed and are implementing the intervention. Review of theories of behavioural change in health services	Not applicable for review of documents; purposive sampling for interviews	Initial programme theory and a hypothetical causal pathway linking intervention inputs and expected outcomes Summary of theories of organisational change in relation to their spheres of influence
Steps B1 and B2: data collection—quantitative (process)	Construct survey questionnaire based on a review of theories of behavioural change in healthcare organisations and reconstruction of initial programme theory from step A	All health managers in intervention and control district who agree to participate (about 100 in all; about 60 in Tumkur and 40 in Raichur)	Key outcome variables for survey Attitudes to training programmes and district planning Organisational commitment Self-efficacy Attitude towards receiving and providing supervision Statistical analysis to determine relationship among variables and effect of exposure to intervention
Step C: data collection—qualitative (context and outcomes)	Assess action plans before, during and after intervention; assess performance and outcomes using routine institutional data and interview participants and non-participants at district and taluka level to understand changes in the course of 3 years	Purposive, based on exposure to intervention	Analysis of the qualitative data to understand how planning and supervision practices changed in the course of the intervention as well as how other contextual determinants influenced these changes
Step D: analysis (context–mechanism–outcome configurations)	Analyse findings from B2 and C to understand the relationship between various elements in the hypothetical causal chain and the contribution of contextual factors to the outcomes observed	Desk review and joint analysis of findings	Further refining of the initial programme theory by the improved understanding from the application of qualitative and quantitative methods
Steps E and F: (validation and refining the theory)	Formulate context–mechanism–outcome configurations and verify through fresh data collection as well as re-looking at the earlier findings (steps B2 and C)	Purposive sampling of participant and non-participant health managers in both districts	An internally consistent and valid explanation of ‘what components of the intervention worked, for whom and under what conditions’

The questionnaire used in the survey (step B) includes six modules (modules B to G in supplementary file 1) to measure attitude towards planning and training programmes, organisational commitment, self-efficacy and supportive nature of supervision. The module on organisational commitment (module C in supplementary file 1) is adapted from two versions of the Meyer and Allen organisational commitment questionnaire that have been tested and validated in public services in south Asian settings.29–31 A 5-point Likert scale is used to grade responses. Self-efficacy in managing conflict situations usually faced by managers of health services is measured with a 10-item scale based on the Bandura scale32 that was developed for use across cultures and has been demonstrated to have cross-cultural equivalence across several languages.33–36 The supportive nature of supervision is measured using 14 items on a 5-point Likert scale. We adopted eight items that measure supportive supervision and four items measuring non-controlling supervision from Oldham and Cummings, which in turn is based on the Michigan organisational assessment package.37 38 We added two items to measure controlling supervision. The questionnaire will be piloted among public health experts and taluka-level health managers. The pilot will be used to improve the understandability of the questions because some of the tools have not been tested earlier among south Indian health services staff. Exposure of participants to the intervention, type of participation and their performance during and immediately after the training programme and mentoring will be captured through analysis of secondary data from attendance records, monthly reports of the training programme and visit notes by mentors.

In step C, we will conduct document review, compile routine health information data on performance, conduct interviews using a semi-structured interview guide (supplementary file 2) and undertake non-participant observation.

Sampling

The survey (step B) will be conducted among all health managers in the district. For the purpose of this study, a health manager is defined as a health worker in the government services, who is managing a facility, team or institutions at the taluka or district level. The questionnaire will be administered among the health managers in the two study districts, Tumkur and Raichur. They will be invited to participate voluntarily in the study. The first author (NSP) or one of two trained data collectors will visit the health managers their place of work after obtaining an appointment at a time convenient to them to ensure good recruitment. The data collectors will be trained to answer questions about the questionnaire and the nature of the study, as well as to clarify doubts arising in the course of filling the questionnaire.

In steps C and E, we will carry out purposive sampling; in step C, we will choose respondents for interviews in order to interview people ranging from no exposure to the intervention to people who have participated most in the intervention. In step E, data collection will be done through participant observation and will be iterative in nature. It will be based on the findings of steps B2 and C. We shall select participant health managers purposively in Tumkur district as well as non-participant health managers with similar outcomes from Raichur district to understand which ones among them achieved organisational change and to what extent this was facilitated (or not) by the capacity-building programme or individual, systemic or contextual factors (see figure 7).

Analysis

The quantitative data from the questionnaire will be examined (step B2) and descriptive parametric measures for organisational commitment, self-efficacy and nature of supervision will be calculated. Participation in training and mentoring (exposure) among the health managers in Tumkur district will be measured through secondary documents (attendance and mentoring notes). We will apply statistical tests of differences between groups to determine the degree of association between exposure to training and the measures of organisational commitment, self-efficacy and nature of supervision.

We will analyse interview transcripts (step C) using content analysis to understand the process of planning at district and taluka levels. We will use triangulation by systematically sorting through the qualitative data from the observation notes, interviews and secondary document analysis to find common themes or categories by eliminating overlapping areas.

The results of the qualitative and quantitative phases will then be analysed together (step D) to develop plausible explanatory context–mechanism–outcome configurations that explain who performs better with respect to planning and supervision in response to a training-mentoring programme in a district. The result from the analysis of participant observation field notes (step E) will be used to validate this framework and refine the initial programme theory. This phase of joint quantitative and qualitative analysis will be iterative—we will refine the framework through purposive participant observation visits and interviews. By taking into consideration the context within which a given outcome was observed and testing and validating explanatory configurations of these three (context, mechanism and outcome), we will explain how the intervention brought about the changes observed in planning and supervision practices.

Ethics

The protocol of this study was approved by the Institutional Review Board of the Institute of Tropical Medicine, Antwerp and by the Institutional Ethics Committee of Institute of Public Health, Bangalore.

All participants shall be made aware of their participation in the study through formal correspondence. They will have the option to decline participation in the study, and it will be ensured that non-participation will not affect further participation in the training programme. In addition, written consent shall be obtained for each interview. The study proposal shall be shared with the state health authority and permission shall be sought to access routine health data, reporting formats and meeting proceedings.

Questionnaires and interview transcripts shall be coded to ensure confidentiality of all ideas/opinions expressed by participants in the course of the study. None of the study outcomes shall identify participants by name or exact designation to avoid potential professional or personal harm to the participants in view of opinions/ideas expressed by them.

The language of interaction with participants will be either English or Kannada (the local language in the state of Karnataka) in function of their preference; this would be established at the beginning of the interaction. Consent forms shall be made available in both English and Kannada (supplementary files 3 and 4), and the participant will have a choice to read and understand the nature of study in the language of their choice and decide accordingly. The content shall also be orally explained to the participant by the trained data collector in the case of the self-administered questionnaire and the interviewer in the case of interviews. All interviews shall be conducted at a time and venue indicated by the participant with prior appointment. The approval for audio recording of interviews shall be sought separately in addition to the consent for taking notes of the interview.

The participant shall have the right to revoke or withdraw consent to part or all of what he has expressed during the study period. In case of collection of any document outside of public domain (eg, privileged communication between district authorities), a permission letter shall be obtained from the authorised official.

There is no interaction with patients in the course of the study.

Quality control

All the data from the qualitative data collection methods will be organised on Nvivo software with clear documentation of the procedures adopted and consistent file naming. Analysis of the interview transcripts, categorisation and analysis will be crosschecked by two researchers.

For each survey respondent, the data collector will check the questionnaire for completeness. Before data entry, a member of the study team will scan all questionnaires for errors. The data will be entered into a spreadsheet using a software for programmed data entry (Epidata) with in-built validity checks and error detection (supplementary file 5).39

Discussion

HRM interventions at the district level are complex; the outputs are produced as a result of interactions between several actors and institutions within a given context resulting in a web of processes, which are difficult to map in a straightforward linear manner. It is being increasingly recognised that such interventions present a methodological challenge.40 41 This study intends to improve our understanding of scientific evaluation of complex interventions in HRM in health. The capacity-building programme in Tumkur has all the features of a complex intervention as described by the new guidance of the Medical Research Council on developing and evaluating complex interventions. The guidance lists some dimensions of complexity—‘the number of and interactions between components within the experimental and control interventions (if identified), number and difficulty of behaviours required by those delivering or receiving the intervention, number of groups or organisational levels targeted by the intervention, number and variability of outcomes and degree of flexibility or tailoring of the intervention permitted’. The latest 2008 guidance of Medical Research Council, while acknowledging the limitations of experimental designs, notes that inclusion of a process evaluation in complex interventions ‘is a good investment to explain discrepancies between expected and observed outcomes, to understand how context influences outcomes and to provide insights to aid implementation’. The recent guidance builds on the experience gained in understanding the limitations of the earlier experimental designs and suggests the use of a ‘more flexible and less linear model of the process, giving due weight to the development and implementation phases, as well as to evaluation’.42 This is further reinforced by Campbell et al40 who emphasise the need to use a mix of qualitative and quantitative evidence that needs to be applied to an (often) iterative process of framework development and testing.

Realist evaluation of HRM interventions

Conduct of trial-based studies in social systems has limitations in view of the lack of ‘control’ over the contextual and operational factors that affect the observations. Although a potentially verifiable causal chain that connects an intervention and a hypothesised outcome linked together through sequential steps is often appropriate for scientific evaluation, the responses of social systems to new approaches are very often difficult to ‘reduce’ to such a testable succession of steps with cause–effect relationships.25 26 43 Increasingly, social programme evaluations have been encouraged to look beyond the ‘successionist’ format of experimental design that is well suited for classical bio-medical research. At the first WHO health systems research symposium at Montreux in 2010, a strong call was made to strengthen the evidence base for capacity development through ‘proper evaluation of capacity development initiatives’ and use of multimethod approaches to overcome the difficulties imposed by the complexity of human resources in health interventions.44 45 Realist evaluation precisely posits that programmes are embedded in social systems and stresses the importance of understanding what works for whom and under what conditions. It offers a framework to design scientific evaluations of human resource interventions. Based on a review of literature on choice of methods for complex interventions, Marchal24 reports that experimental or quasi-experimental designs ‘are indicated when the effectiveness of an intervention should be tested’ and are by themselves inadequate to answer and explain how interventions work, an analysis supported by several other reviews.40 43 46

Health worker practices are complex behaviours that are determined by various individual, systemic or institutional and contextual factors.12 In their review of theories of behavioural change in health services, Rowe et al12 question the premise that poor organisational performance in health is merely due to the lack of knowledge and skills. They encourage studies to move beyond the old paradigm ‘that most performance problems can be solved by training alone’. In the Tumkur capacity-building intervention, a reconstruction of the assumptions of the intervention and how it sought to change planning and supervision practices is established. The outcomes (ie, better planning and supervision practices) are determined by several factors at the individual (improved knowledge and skills), institutional (competence, enabling environment, motivation to apply/change) and contextual (other programmes or interventions with similar objectives and many other contextual factors that may facilitate or discourage organisational change) levels. In order to understand how the programme worked, we will further build and refine these hypothetical pathways based on a review of literature and the study findings to arrive at context–mechanism–outcome configurations.

Realist evaluation presents a scientific approach towards understanding mechanisms through which social interventions work. According to Pawson and Tilley,47 “Programs work (have successful ‘outcomes’) only insofar as they introduce the appropriate ideas and opportunities (‘mechanisms’) to groups in the appropriate social and cultural conditions (‘contexts’)”. By building and testing such Context (C)–Mechanism (M)–Outcome (O) or CMO configurations within the talukas, it is possible to generate an internally consistent and externally valid knowledge of how such interventions work in a given context to produce an observed outcome.26

Existing theories on behavioural change in health services can be divided into those that explain change at or between individual, institutional or contextual levels, and thus, evaluations must consider all these levels while trying to explain behavioural change (figure 7). The variables we chose to measure (attitude towards training, organisational commitment, self-efficacy, nature of supervision) have all been linked to behavioural change, and improvement in organisations and a preliminary desk review of the training reports and documents suggests that these are also linked to the intervention in Tumkur.