Using qualitative comparative analysis and theory of change to unravel the effects of a mental health intervention on service utilisation in Nepal.

BACKGROUND: The integration of mental health services into primary care is essential to improve the coverage of mental health services in low resource settings, but the evaluation of this remains challenging. We used a programme's Theory of Change (ToC) as a conceptual framework to determine what combination(s) of conditions at facility and community level influenced the mental health service utilisation as a result of a district mental healthcare plan (MHCP) implemented in Chitwan, Nepal. In addition, we show how qualitative comparative analysis can be used to provide an integrated analysis of data from a ToC.
METHODS: We conducted a longitudinal case study of 10 health facilities where the MHCP was implemented. We collected data from all facilities at baseline (October to December 2013) and quarterly following the implementation of the intervention (March 2014 to November 2016). The data were analysed using pooled qualitative comparative analysis in fsQCA V.2.5.
RESULTS: The following conditions were necessary for high mental health service utilisation: presence of basic and advanced psychosocial care, evidence-based identification and treatment guidelines (WHO mhGAP), referral to tertiary services and the presence of trained female community health volunteers. Two additional combinations of conditions were also identified as sufficient for a high mental health service utilisation: high medication supply, trained facility staff and either the use of a community informant detection tool or having a larger proportion of the community attend community awareness activities.
CONCLUSIONS: Both supply-side interventions (formalised approaches to health worker detection and treatment, training of health workers, supervision) and demand-side interventions (community awareness and case finding) are important to integrate mental health in primary care. ToC can be used to provide an integrated analysis of data from a ToC, therefore helping to shed light on the black box of complex multilevel interventions.

Multiple intervention components, such as training, supervision and increasing medication supply, are required to integrate mental health into primary healthcare in low resource settings.

The relative importance of these in contributing to an increase in mental health service utilisation is not known.

The study showed that mental health service utilisation was high when a combination of supply-side and demand-side intervention components worked together.

Both supply-side and demand-side intervention components are necessary to improve mental health service utilization.

Background

There is increasing momentum for the integration of mental health services into primary healthcare to address the mental health treatment gap, which is especially marked in low-income and middle-income countries.1 Barriers to integration such as including low investment in mental health, staff resistance to decentralisation, few people trained in mental healthcare and overburdened primary healthcare staff with minimal supervision2 remain.

Integration of mental health services into primary healthcare requires multiple intervention components, including case finding, assessment, and psychosocial and pharmacological treatment.3 4 The role of mental health specialists shifts towards providing supervision for primary care staff, consultation for complex cases and assessment and treatment of cases that cannot be managed at primary care level.3 Despite the evidence for the each of the above components of care in the literature5–8 and the development of evidence-based guidelines in the form of mhGAP to support this integration by WHO,9 there is little guidance for the implementation and effectiveness of these intervention components implemented into routine care in low resource settings.5 9 Developing such guidance is difficult because evaluation of the implementation of multiple complex health interventions into routine health services is challenging, particularly when the implementation relies on health system resources such as staff, medication and space,10 and is constrained by what is feasible within the health system.11

As part of a multiyear mental health services implementation research project, the Programme for Improving Mental Healthcare (PRIME),12 Jordans and colleagues13 developed and implemented a district mental healthcare plan (MHCP) in West Chitwan, Nepal, which focused on the detection and treatment of depression, alcohol use disorders, epilepsy and psychosis. It was developed through comprehensive formative work,14 including the development of a Theory of Change (ToC), and pilot testing in two health facilities13 and is described in detail below. Although most of the intervention components were evidence based, it is not clear which components of the MHCP are essential in what combination for an increase in mental health service utilisation at facility level in Nepal.

Conceptual framework for the evaluation

We developed a ToC to assist with both the development and evaluation of MHCPs15 in the five PRIME countries, including Nepal.13 ToC is a theory-driven planning and evaluation approach that identifies the short-term, medium-term and long-term outcomes of the intervention in order to unpack the ‘black box’ of complex interventions.16 We use the Medical Research Council Guidance definition of complex interventions: interventions which contain multiple interventions, causal strands, outcomes, groups and organisational levels.11 The PRIME ToC outlined the programme theory for how the intervention would lead to improved health, social and economic outcomes for people living with priority disorders and their families and carers in the district.15 17 The PRIME ToC outlines the outcomes required to achieve this impact in relation to political buy-in, capacity building, identification of mental illness and treatment and care at district, facility and community level (online supplementary file 1). Mapped onto these were the intervention components required to achieve these short-term and medium-term outcomes, as well as indicators to measure progress. This process and the resultant ToC is described in detail elsewhere.15 17 A summary ToC is shown in figure 1 with the detailed ToC included in online supplementary file 1.

Figure 1

PRIME Summary Theory of Change at district, facility and community levels adapted from Breuer et al.15 17

Although the ToC provided us with a framework to develop the PRIME MHCPs, results from a recent systematic review found no papers that evaluated process and outcomes simultaneously.18 Qualitative comparative analysis (QCA), a case-oriented method developed in the social sciences for analysis of data in comparative case studies, holds promise as a way to do this.19

In summary, it is important to understand which mental health intervention components result in more people accessing services leading to improved health, social and economic outcomes. However, evaluating complex mental health interventions integrated into routine settings is difficult, particularly when using routine data. The PRIME ToC provides a conceptual framework for the evaluation of these interventions although there is little guidance on how the process and outcomes data from ToC can be combined in one analysis. This paper aims to determine what combination(s) of conditions identified by the PRIME ToC at facility and community level influenced the mental health service utilisation in the PRIME implementation facilities in West Chitwan. In addition, we show how QCA can be used to provide an integrated analysis of data from a ToC.

Methods

We conducted a longitudinal case study based on indicators developed specifically to measure the PRIME ToC indicators,15 20 including the collection of routine service utilisation data.

Setting and intervention

The mental healthcare situation in Nepal exemplifies many of the global challenges to the integration of mental healthcare. Nepal has a recent history of insurgency followed by a delay in the drafting of the constitution.21 Although Nepal’s mental health policy was drafted in 1997 and includes the provision of mental health services in primary healthcare, it has not been implemented.22

This study was set in Chitwan (population 579 984),23 the implementation area for the Programme for Improving Mental Healthcare in southern Nepal. The MHCP was implemented in 10 health facilities and their surrounding areas. Nine of these were health posts which provided outpatient services and one was a primary healthcare centre, which also provides emergency services. Female community health volunteers (FCHVs) who are trained in basic health promotion and referral are based in the community surrounding the health facilities.

Prior to the implementation of PRIME in Chitwan, there had been ad hoc mental health training by NGOs and tertiary centres for primary healthcare staff. Lack of supervision and refresher training meant that in practice, there was no consistent mental health service provided in primary care.21 In addition, there was a limited and erratic supply of psychotropic medications (phenobarbitone and amitriptyline only), no counselling or psychotherapeutic services available and no standardised guidelines or treatment protocols.21

The PRIME MHCP was implemented at the district, facility and community levels. At the district level, engagement and advocacy activities were conducted with health service managers, and ensuring there was a functioning referral system for complex or treatment-resistant cases to specialist mental health services. At the health facility level, the plan included service provider awareness, screening and assessment following the WHO mhGAP intervention guide,24 brief psychosocial support derived from structured interventions adapted from the Healthy Activity Programme (HAP)25 and Counselling for Alcohol Problems (CAP),26 pharmacological management and follow-up for monitoring and treatment adherence. At the community level, the MHCP included community awareness programmes conducted by PRIME community counsellors in coordination with the FCHVs, proactive case detection in the community using the community informant detection tool (CIDT)27 as well as the provision of the HAP and CAP by a community counsellor. The CIDT is a novel vignette-based case finding tool, which was developed for use in Nepal.27 28

Sample and measures

All of the 10 health facilities in Chitwan where the PRIME MHCPs were implemented at each time point were selected as cases. The health facilities are described in table 1.

Table 1

PRIME mental healthcare plan implementation facilities in Chitwan, Nepal

Facility	Facility type	Catchment area		Rural vs urban
		Size (km2)	Population
1	Health post	3	13 929	Semirural
2	Health post	5	11 195	Rural
3	Health post	5	3862	Semirural
4	Health post	3	19 066	Rural
5	Health post	4	6506	Semirural
6	Health post	3	11 500	Rural
7	Health post	10	11 508	Rural
8	Primary healthcare centre	6	7674	Semirural
9	Health post	13	8057	Rural
10	Health post	7	15 071	Rural

The majority of the data used in this study were collected through a quarterly profile of the implementation facilities designed to collect facility-level indicators outlined in the PRIME ToC20 and table 2. A baseline profile was conducted to collect data from October to December 2013. Following implementation of the district mental healthcare plans (beginning in March 2014), the profiles were collected quarterly until November 2016. Data collected included monthly health information system data on mental health patients (which were averaged over the quarter) and quarterly data on trained staff, supervision, medication supply and presence of PRIME intervention components. Data at the community level were collected by the programme implementers.

Table 2

Outcome and condition sets, indicators and calibration for qualitative comparative analysis

ToC outcomes	Condition	Indicators	Data collection	Calibration
Long-term outcome	Primary care service utilisation for mental illness	Monthly mental health patients as a percentage of monthly clinic patients averaged over the quarter	Facility profile	Monthly mental health patients as a percentage of monthly clinic patients was averaged over the quarter.Calibrated in STATA using a ‘direct’ transformation to calibrate levels of deviation from anchors, in terms of log odds. Anchors: 0 (fully out of the set), 1 (mid point) and 3 (fully in the set).
Short-term and medium-term outcomes	Medication supply	Essential medications were available at the facility in the last month	Facility profile	Coded the availability of all medications outlined in the MHCP: all of the time (1), most of the time (0.75), sometimes (0.5), rarely (0.25) and not available (0). An average score was obtained for all medication.
	Supervision (facility)	All facility staff receive supervision on a regular basis as defined by the MHCP and guidelines	Facility profile	Number of monthly supervision visits provided at facility level was calculated as a proportion of the number recommended in the MHCP. This was averaged over the quarter for each type of supervision. The data was naturally calibrated into 0 (fully out of the set), 0.333 (one supervision session every 3 months, more out than in the set). 0.666 (two supervision session every 3 months, more in that out of the set), 1 (fully in the set). Following factor analysis, the data were averaged to create a mean facility supervision score.
	Supervision (community)	All community staff receive supervision on a regular basis as defined by the MHCP and guidelines	Facility profile	Number of monthly supervision visits provided at community level was calculated as a proportion of the number recommended in the MHCP. This was averaged over the quarter for each type of supervision. The data were naturally calibrated into 0 (fully out of the set), 0.333 (one supervision session every 3 months, more out than in the set), 0.666 (two supervision sessions every 3 months, more in that out of the set), 1 (fully in the set). Following factor analysis, the data were averaged to create a mean community supervision score.
	Trained human resources (facility)	Adequate numbers of trained human resources as per the MHCP are available at facility level	Facility profile; process data	Number of trained staff was calculated as a proportion of the government-recommended staff numbers for the cadre of worker. This was initially done for prescribers and non-prescribers but averaged following a factor analysis.Calibrated in STATA using a ‘direct’ transformation to calibrate levels of deviation from anchors, in terms of log odds. Anchors: 0 (fully out of the set), 0.6 (mid point) and 1 (fully in the set).
	Trained human resources (community)	Adequate numbers of trained human resources as per the MHCP are available at community level	Facility profile	Number of trained FCHVs was calculated as a proportion of the government-recommended staff numbers for the cadre of worker. We did not disaggregate between FCHVs trained in detection and referral and those with additional training in home-based care.
	Facility psychosocial interventions	Facility psychosocial interventions available	Facility profile	Average of binary variables indicating the availability (1) or absence (0) of basic and advanced psychosocial interventions for all four mental illnesses outlined in the MHCP.
	mhGAP	mhGAP available	Facility profile	Average of binary variables for availability of mhGAP for all four mental illnesses outlined in the MHCP.
	Referral to tertiary care	Referral to tertiary services available	Facility profile	Binary variable on availability of referral services in the last month.
	Community identification	Increased number of cases detected and referred by community health workers	Facility profile	Average number of community identification detection tool referral forms received at the facility per quarter. Calibrated in STATA by ranking the variables and then standardising them between 0 and 1. Resultant fuzzy value is more out than in the set if there is less than 1 CIDT per month.
	Community awareness and stigma reduction	Proportion of people reached by community awareness programmes	Facility profile	The cumulative proportion of people who had attended community sensitisation sessions was calculated using the cumulative number of people attending the sessions over the population of the area around each health facility. The indicator was calibrated in STATA using a ‘direct’ transformation to calibrate levels of deviation from anchors in terms of log odds. Anchors: 0 (fully out of the set), 0.01 (mid point) and 0.04 (fully in the set).

CIDT, community informant detection tool; FCHV, female community health volunteer; MHCP, mental healthcare plan; mhGAP, Mental Health Gap Action Programme.

Data analysis

We analysed the data using QCA, a case-oriented method developed in the social sciences for analysis of data in comparative case studies.29 It can be used to analyse both quantitative and qualitative data from multiple sources by using set theory and Boolean algebra to identify patterns in data. QCA identifies both necessary conditions, that is, those that are necessary for the outcome to occur but not sufficient, and sufficient conditions, that is, those that lead to the outcomes in some cases.30 Rather than identify single conditions or variables which lead to an outcome, QCA seeks to identify causal pathways or configurations of causes in diverse contexts.31 The relative importance conditions can be determined by (1) whether they are identified as necessary conditions with higher consistency scores indicating a stronger relationship with the outcome and (2) conditions that are included in more than one causal pathway of sufficient causes.

QCA is particularly suited for small numbers of cases where statistical analysis is not possible. Inherent in QCA is equifinality, that is, that there can be multiple causal factors within a pathway or across distinct pathways to an outcome.31 This recognises causal complexity,32 unlike many statistical methods which isolate the effects of one factor on the outcome by controlling for all other factors.

QCA is asymmetrical. This means that the causal pathways shown to lead to a lack of outcome are not merely the inverse of those leading to the outcome but could be completely different causal pathways. For more detail on the QCA approach, we refer to Kane et al.19

QCA outcomes and condition sets

We used the PRIME Summary ToC (figure 1) as our conceptual framework to determine the hypothesised facility-level and community-level pathways which led to increased service utilisation, that is, the ToC outcome: “People with mental disorders are identified and/or diagnosed in the facility”.

We used the section of the ToC leading up to the service utilisation to identify the condition sets, which are the conditions of interest which may or may not result in the outcome. These are listed in the second and third columns of table 2. We included all indicators for the short-term and medium-term outcomes in the subsection of the PRIME ToC as condition sets except for three: (1) programme approved and budget available at district level; (2) programme co-ordinator in post; (3) people with mental disorders are willing to seek treatment. The first two were excluded because the data were collected at district level and therefore did not vary across facilities and therefore could not be included in the QCA. Therefore, we have limited this analysis to understanding the community-level and facility-level pathways outlined in the ToC (as indicated in the grey boxes in figure 1).

The third was excluded because these data were collected qualitatively and the sample size at each of the facilities was not large enough to be disaggregated by facility.

We added two additional condition sets not in the ToC: (1) availability of mhGAP protocols for detection and treatment of mental illness and (2) referral to tertiary care. The first was added to mirror the existing indicator, availability of psychosocial interventions, and the second had been included further along the ToC. However, the availability of referral mechanisms could also contribute to willingness of healthcare workers to identify people with mental illness. Therefore, we included it as a condition set in this analysis.

Data preparation and calibration

In order to minimise the number of condition sets, multiple indicators for one condition were aggregated. For example, we averaged medication supply for all medications outlined in the MHCP to create a summary condition for medication supply. Where we were unsure whether to combine the indicators from a conceptual point of view, we conducted a factor analysis to ensure they loaded on the same factor. For example, supervision for pharmacological and psychological management at facility level were combined into one condition. The ToC indicators and the process used to combine indicators are outlined in table 2, column 3.

Once the outcome and condition sets were finalised, each case was calibrated by assigning a value between 0 and 1 under each condition set to describe set membership or the degree to which the case meets that condition. Zero is fully out of the set, 1 fully in the set and 0.5 neither in nor out of the set.

The data for the outcome and each condition set were calibrated in various ways, using a theoretical understanding of the indicators and calibration commands in the fuzzy package in STATA.33 34 To aid the interpretation of the results, we ensured that the data were calibrated carefully to ensure meaningful interpretation for programmatic and policy outcomes. The outcome, for example, was calibrated as follows: monthly mental health patients as a proportion of all monthly facility patients was averaged over each quarter. The range across the implementation period was 0%–10.69% with a median of 1.79%. These proportions were graphed over time to determine the pattern at baseline, immediately following implementation and over the remaining period of implementation. In most facilities, the percentage of patients attending for mental illness increased to above 2% during the implementation, with 25% reaching 3% and above. We therefore decided that 0% was fully out of the set, 1% neither in nor out of the set and 3% fully in the set. We used a direct transformation in STATA to calibrate levels of deviation from these anchors in terms of log odds and to obtain the fuzzy values.35 The calibration approaches for the conditions are outlined in table 2. The calibrated data can be found in online supplementary file 2.

Data analysis

The data were analysed using pooled QCA36 in fsQCA V.2.537 where data are pooled for all facilities and outcomes and which therefore treats each observation for each facility and each time point as an independent observation.

We conducted each of the following analysis steps for the outcomes ‘high’ and ‘low mental health service utilisation’ as QCA is asymmetrical. We first identified necessary conditions from the list of conditions outlined in table 2 using the Necessary Conditions option in fsQCA and XY plots. Necessary conditions are conditions for which the outcome is almost always a result of a causal condition.19 As outlined by Garson, we defined conditions as necessary when the consistency, or degree to which the outcome was a result of the condition, was greater than 0.90.30

Second, we constructed a truth table with the remaining conditions to determine sufficient causal pathways. A truth table is an intermediate step in the QCA analysis that shows all possible configurations of conditions, and the number of cases that exhibit each causal configuration, together with a consistency score for the causal configuration. The abbreviated truth tables containing only the causal configurations for which data were available for the outcomes of high and low mental health service utilisation are presented in online supplementary file 3. We applied a frequency threshold of 2 and used a standard consistency threshold of 0.830 and the standard analysis option in fsQCA.

fsQCA uses Boolean miniminisation to simplify the combinations of conditions and provide sufficient combinations of conditions associated with the outcome. fsQCA provides three types of solution according to the simplifying assumptions used: (1) a complex solution, which minimises the solution based on the available data only; (2) an intermediate solution, which uses theoretical knowledge to determine which logical remainders to include; and (3) a parsimonious solution, which finds the simplest solution without evaluating the plausibility.38 For the intermediate solution, logical remainders were coded according to the programme theory outlined in the ToC. Logical remainders are the causal configurations in the truth table for which there were no available data.

We calculated solution consistency and coverage based on the findings. Coverage refers to the degree to which the causal condition explains the outcome in the sample.19 There is no specific cut-off for coverage because a lower coverage may indicate a more uncommon causal pathway. We present both the intermediate and complex solutions of sufficient conditions from the truth table analysis in the table and only the intermediate solutions in the narrative.

Permission to collect data was obtained from the managers of all 10 facilities to obtain facility-level data.

Results

The results of the QCA for high and low mental health service utilisation at primary care facilities are outlined in tables 3 and 4, respectively.

In summary, the presence of basic and advanced psychosocial care (consistency 0.97, coverage 0.70), mhGAP (consistency 0.92, coverage 0.77), referral to tertiary services (consistency 1.00, coverage 0.63) and trained FCHVs (consistency 0.97, coverage 0.74) were necessary but not sufficient for the outcome of high mental health service utilisation.

In addition, two causal pathways were identified as sufficient for a high mental health service utilisation in the intermediate solution of the truth table analysis. Mental health service utilisation was high in facilities at times when medication supply was high, facility staff were trained and either the CIDT was used frequently or a larger proportion of the community had attended community awareness activities. The solution consistency was high at 0.85 and with a moderate coverage of 0.74. This indicates that these combinations of conditions lead to the outcome in a high proportion of cases and that a moderate proportion of the cases with the outcome are a result of this combination of conditions.

There were no necessary conditions for low mental health service utilisation at primary care facilities and one causal pathway was identified as sufficient in the intermediate solution in the truth table analysis. Mental health case load was low in facilities at times when there was lack of facility and community supervision, lack of mhGAP services and a low proportion of the community members attending community awareness activities. The consistency was high for this solution (0.90), but the coverage moderate (0.56). This indicates that these combinations of conditions lead to the outcome in a high proportion of cases, but that only a moderate proportion of the cases with low service utilisation are the result of this combination of conditions.

Discussion

This study shows that in Nepal, a combination of formal mental health service intervention components, training, medication and community intervention components lead to increased mental health service utilisation in primary care facilities. In contrast, at times when mhGAP availability, supervision and community awareness activities were low, mental health service utilisation was also low. This indicates that different inputs or intervention components may be important to the establishment and sustainability of mental health services in the community.

The identification of three defined treatment packages (mhGAP, psychosocial services, referral) as necessary conditions over and above inputs such as training and medication for an increase in service utilisation is in line with the existing evidence on core packages of care.3–5 These services were defined as follows: mhGAP services were available when all the required inputs were in place. Specifically, the appropriate staff were trained and medication, mental health outpatient register, protocols, mhGAP flowchart and brochures were available. Similarly, basic and advanced psychosocial services were defined as available when the appropriate staff were trained and the mental health outpatient register, basic psychosocial manual, HAP/CAP manual, brochures and flowchart were available. Our results show that formal guidelines and reporting and referral need to be in place for an increase in detection and treatment by primary healthcare workers. This can be seen in the increase in service utilisation, which occurred following the formalisation of these services in the clinic rather than when training alone was conducted. The results of the truth table analysis also indicated that training needed to be combined with medication and either community awareness activities or case finding in the community. This may give some insight into why the evidence for the effectiveness of training alone on detection of mental illness is mixed, particularly in the long term.39–42

This study highlights the importance of creating demand at community level. Trained FCHVs were a necessary condition of high mental health service utilisation. In the MHCP, FCHVs are responsible for a large proportion of demand creation activities, namely administering the community detection tool (CIDT) and organising community awareness programmes which are conducted by the community counsellors.13 FCHVs also provide home-based care services for patients who have stopped attending the facility. This includes psychoeducation, emotional support and motivation to seek services at the health facility. For high mental health service utilisation, the two sufficient causal pathways included both facility-level conditions (training and medication provision) and either high use of the CIDT or a higher proportion of the community reached with community awareness programmes. The findings in this study of the importance of demand-side factors corroborate the findings of a qualitative study in Chitwan exploring demand and access conducted in the intervention development phase of PRIME.43 Brenman and colleagues43 used framework analysis of qualitative interviews and focus group discussions with stakeholders with the PRIME ToC as their conceptual framework. They found that many of the barriers to demand and access to mental health services were at the community level, including high levels of stigma around mental health, low mental health awareness and lack of awareness about treatment availability. They also suggested that in addition to service provision at the facility level, working at the community level to increase awareness of mental illness and working with local understandings of mental illness was essential to ensure uptake of services. A community-based survey of barriers to care in Chitwan showed that stigmatising beliefs in the community were a barrier to seeking care.44

We found that supervision was important in ensuring that detection and treatment for mental illness occurs at primary healthcare. Low mental health service utilisation occurred at facilities and times when supervision at facility and community level was low, mhGAP was unavailable and low numbers of people in the community were reached by community awareness programmes. The majority of data contributing to this causal pathway is from the initial roll-out of the MHCP when training had been conducted but supervision, mhGAP and community awareness programmes had not reached a substantial amount of people. Understanding the role of lack of supervision and low community awareness and lack of formal mhGAP services is important because integration of mental healthcare into primary care often consists of short training courses and very little supervision at the district level.45 This is due to many factors, including the lack of trained mental health specialists to offer supervision and the lack of evidence for the type and quality of supervision required.46 However, this study shows that supervision at both the community and facility level together may be important to avoid low service utilisation. Structured and ongoing supervision should therefore be seen as a key component of a successful mental healthcare plan together with facility-level and community-level components.

The main contribution of this study is to provide evidence to support a part of the PRIME ToC, which identified specific short-term and medium-term outcomes that work together at facility and community level to improve identification of people with mental illness in facilities in southern Nepal. In this analysis, we showed that all short-term and medium-term outcomes included from the PRIME ToC were either necessary or sufficient (in combination with other conditions) in cases of high mental health service utilisation or absent in cases of low service utilisation. This study did not identify specific gaps in the ToC, although the lower coverage of the intermediate solution for the lack of outcome could indicate that there are other causal pathways that have not been identified in this analysis. These could include, for example, the level of facility manager buy-in to the provision of mental health services, quality of facility management, quality of supervision, competing priorities within the health service or contextual factors.

This study makes a methodological contribution as an example of how QCA can be used together with a ToC to analyse routine service implementation data. There have been some recent applications of QCA to health services research.31 32 47 48 However, to our knowledge, there have been no studies which have used QCA and ToC or QCA for longitudinal data analysis in health services research. ToC provides a transparent approach to develop a contextually relevant conceptual framework and indicators which can be used to quantify the QCA condition sets and outcomes. QCA can then be used to evaluate where and how the different factors identified by the ToC affect the outcome. QCA is particularly suitable for the analysis of routine implementation data because it seeks to understand causal complexity rather than identify individual risk factors,19 uses a relatively small number of cases (in this case health facilities) and can combine data from different sources, including qualitative data. We propose that the combination of ToC and QCA could be expanded beyond health services data to combine process and outcome evaluations for randomised controlled trials. For example, where mixed-method data were available for trial participants, a QCA could be used to understand how fidelity, dose, participant characteristics, therapist characteristics and other contextual factors are related to the outcome.

During the process of applying QCA together with ToC, we learnt several lessons. First, using a conceptual framework, such as ToC, to guide QCA is important to identify the outcome and condition sets as well as providing a theoretical basis for how to treat logical remainders. Second, the number of conditions needs to be kept to around 5–7 to avoid the ‘limited diversity problem’19 which occurs when there are too many conditions in relation to the number of cases which prevent Boolean minimisation of the truth tables and result in causal pathways with many conditions and low coverage. This is similar to the problem in frequentist statistics of including too many explanatory variables in a multivariate linear regression without an adequate sample size. To avoid this problem, we aggregated similar indicators prior to data analysis, for example, proportion of trained prescribers and non-prescribers and the number of trained home-based care workers and FCHVs. Other approaches to avoid limited diversity include (1) a comprehensive approach where all conditions are included iteratively in the analysis until the most parsimonious solution is reached or (2) a two-step approach where an analysis of ths distal conditions is conducted first, followed by the proximal conditions.49 Third, data calibration is one of the key steps in QCA and should be conducted with a theoretical understanding of the values chosen, that is, that 1 is fully in the set, 0 is fully out of the set and that 0.5 is neither in nor out of the set. The calibration of data should be conducted using thresholds that are meaningful to the programme. Although tools exist in STATA and other software to calibrate continuous variables, the distribution of the resulting calibrated data should be checked to ensure that the interpretation of the data is in line with the theoretical understanding. Because of the iterative nature of QCA, it is possible and may be necessary to recalibrate the data following the analysis.

This study has some limitations. Because the district-level effects were the same across the district, we could only assess the facility-level and community-level part of the ToC and could not assess the effect of the engagement, service strengthening and input into the medication procurement system, which was identified in the ToC. This will be expanded on in another paper which investigates the district-level implementation of the Nepal MHCP. In addition, although the PRIME MHCP was implemented in routine health services, a significant amount of support was provided by TPO Nepal, the NGO implementing the study. For example, they provided medication when the government had not yet started supplying this and they provided staff to assist with supervision, and directly implemented the community counselling component. As a result, most of the components of the MHCP were relatively well implemented in most of the facilities. However, as most inputs were available at similar times, it is difficult to determine which ones actually had an effect. Although the MHCP was piloted in two districts and there was regular supervision to ensure that interventions were delivered as intended, there were no formal fidelity assessments conducted prior to this study. This analysis also does not take into account some of the weaknesses of using longitudinal data, such as autocorrelation, non-stationarity and seasonality.50 It may be worth exploring how to combine statistical approaches for longitudinal data analysis with QCA, for example, by using time differencing QCA to look at the change in conditions over the time period of interest and using segmented linear regression to determine the presence or absence of the outcome. Another limitation is the use of pooled QCA which treats each time point as independent and does not take into account the between-clinic variation, and it may be possible to conduct a fixed-effects QCA as software are expanded.36 Lastly, QCA and the ToC did not take into account some of the features of complex adaptive systems such as tipping points, feedback loops, emergence or unintended consequences.51 Despite this, the approach does explore the multiple facility-level and community-level pathways required to achieve the outcome and therefore goes some of the way towards unpacking the black box of complex health interventions.

Conclusions

This study has shown that multiple packages of care are required to integrate mental health into primary care in low resource settings. Specifically, that a combination of formal manualised approaches to health worker detection and treatment, both psychological and pharmacological, referral pathways and trained community workers were necessary for high mental health service utilisation in West Chitwan. In addition, a combination of training, medication and either community awareness or use of community case finding was important. ToC can add value to QCA by identifying the causal pathways and the potential necessary and sufficient conditions which can be explored through QCA. This approach can help to shed light on the black box of complex multilevel interventions. Policy-makers who are planning integrated mental healthcare should ensure that mental healthcare plans are carefully constructed with multiple packages of care. These should include supply-side factors such as formalised approaches to health worker detection and treatment, training of health and community workers, supervision and medication supply. In addition, demand-side factors such as increasing awareness in the community and community case finding should be included.

Table 3

Necessary and sufficient causes for high mental health service utilisation

Conditions		Necessary causes				Sufficient causes
						Complex solution			Intermediate solution
						1	2	3	1	2
Medication							*	*	*	*
Trained staff	Facility					*	*	*	*	*
	Community	*
Supervision	Facility						*	—
	Community					—
Community-level activities	CIDT use							*	*
	Community awareness					*	*	—		*
Services available	Psychosocial care		*
	mhGAP			*
	Referral				*
Condition	Consistency	0.97	0.97	0.92	1	0.94	0.89	0.89	0.85	0.88
	Coverage	0.74	0.70	0.77	0.63	0.44	0.49	0.21	0.46	0.68
Solution	Consistency	1.00				0.87			0.84
	Coverage	0.63				0.68			0.74

Each column under sufficient causes is one causal pathway. Each condition in the causal pathway should be combined with a logical AND. For example: Medication AND Trained Facility Staff AND Absence of community supervision AND Community awareness.

*Indicates presence of condition in causal pathway and — indicates absence of the condition.

CIDT, community informant detection tool; mhGAP, Mental Health Gap Action Programme.

Table 4

Sufficient causes for low mental health service utilisation

Conditions		Sufficient causes
		Complex solution			Intermediate solution
		1	2	3	1
Medication		—	—	+
Trained staff	Facility		—	+
	Community	—	—
Supervision	Facility	—	—	—	—
	Community	—	—	—	—
Community-level activities	Community informant detection tool use	—	—	+
	Community awareness	—	—	—	—
Services available	Psychosocial care available		—	+
	mhGAP available	—	—	—	—
	Referral to tertiary services	+		+
Condition	Consistency	0.91	0.90	0.96	0.90
	Coverage	0.32	0.21	0.08	0.56
Solution	Consistency	0.94			0.90
	Coverage	0.49			0.56

+Indicates presence of condition in causal pathway and — indicates absence of the condition

mhGAP, Mental Health Gap Action Programme.