Impact of the community-based newborn care package in Nepal: a quasi-experimental evaluation.

OBJECTIVE: To evaluate the impact of the community-based newborn care package (CBNCP) on six essential practices to improve neonatal health.
METHODS: CBNCP pilot districts were matched to comparison districts using propensity scores. Impact on birth preparedness, antenatal care seeking, antenatal care quality, delivery by skilled birth attendant, immediate newborn care and postnatal care within 48 hours were assessed using Demographic and Health Survey (DHS) and Health Management Information System (HMIS) data through difference-in-differences and multivariate logistic regression analyses.
FINDINGS: Changes over time in intervention and comparison areas were similar in difference-in-differences analysis of DHS and HMIS data. Logistic regression of DHS data also did not reveal any significant improvement in combined outcomes: birth preparedness, adjusted OR (aOR)=0.8 (95% CI 0.4 to 1.7); antenatal care seeking, aOR=1.0 (0.6 to 1.5); antenatal care quality, aOR=1.4 (0.9 to 2.1); delivery by skilled birth attendant, aOR=1.5 (1.0 to 2.3); immediate newborn care, aOR=1.1 (0.7 to 1.9); postnatal care, aOR=1.3 (0.9 to 1.9). Health providers' knowledge and skills in intervention districts were fair but showed much variation between different providers and districts.
CONCLUSIONS: This study, while representing an early assessment of impact, did not identify significant improvements in newborn care practices and raises concerns regarding CBNCP implementation. It has contributed to revisions of the package and it being merged with the Integrated Management of Neonatal and Childhood Illness programme. This is now being implemented in 35 districts and carefully monitored for quality and impact. The study also highlights general challenges in evaluating the impacts of a complex health intervention under 'real life' conditions. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Adopting a ‘natural experiment’ approach, we used multiple data sources and multiple statistical methods as an important strategy to validate findings.

The two datasets employed, the nationally representative cross-sectional Demographic and Health Survey and the public sector healthcare reporting system Health Management Information System, each has its own strengths and limitations but does not provide representative measures of coverage at population level.

An a priori conceptual framework defined the outcomes of the intervention and guided the analysis; along with other careful measures, such as excluding births taking place during training, this was intended to minimise bias.

Neonatal mortality as the ultimate outcome of interest could not be examined, as the datasets employed were insufficient for examining rare events at district level.

Introduction

While infant and child mortality in developing countries have declined rapidly in the past decades, newborn mortality has decreased much more slowly.1 Nepal has demonstrated impressive reductions in child mortality of 76% since 1990, but over the same time period, neonatal mortality has decreased by only 50%.2 3 With 21 deaths per 1000 live births in year 2016, neonatal mortality now constitutes 54% of under-five deaths.4

Over two-thirds of newborn deaths could be prevented with relatively low-cost, low-tech interventions.5 6 A systematic review based on five randomised controlled trials (RCTs) from South Asia concluded that visits during the antenatal and neonatal periods and home-based treatment for illness reduce the risk of neonatal deaths and improve neonatal care practices, with greater survival benefit when home visits are integrated with preventive and curative interventions.7 Similarly, other South Asian studies employing different programme components and delivery approaches demonstrate improvements in uptake of antenatal care, institutional delivery and newborn care.8–10 Consequently, WHO and the Unicef recommend home visits during the first week of life by appropriately trained and supervised community health workers to promote healthy behaviours and timely recognition of newborn illness, and to provide home treatment for infections and feeding problems.11

Based on global, regional and national evidence, the Ministry of Health (MOH) combined seven community-based and home-based interventions in the community-based newborn care package (CBNCP) to tackle major causes of neonatal mortality.12 This programme comprises (1) behaviour change communication for birth preparedness and newborn care, (2) institutional delivery or clean home delivery through skilled birth attendants, (3) postnatal care, (4) care for low birthweight newborns, (5) management of newborn infections, (6) prevention of hypothermia and (7) recognition of asphyxia, initial stimulation and resuscitation. The programme is delivered through facility-based and community-based health workers as well as the Nepal-specific cadre of female community health volunteers (FCHVs), and comprises training and supervision of the health workforce and provision of essential commodities. The package included 7 days training for facility-based health workers, 5 days training for community-based health workers and 7 days training for FCHVs. Supervision and monitoring mostly uses existing approaches, supplemented with pilot-phase intensive supervision including, for example, monthly review meetings with FCHVs at the health facility level (see online supplementary file, box 1 CBNCP programme components).12 The CBNCP was piloted in 10 out of 75 districts of Nepal in 2009 and 2010 with funding from MOH, the United States Agency for International Development (USAID), Unicef and Saving Newborn Lives (SNL).

The objective of this study was to evaluate the impact of CBNCP on six essential practices to improve neonatal health in pilot districts compared with propensity score-matched comparison districts.

Methods

Study setting and population

Nepal is characterised by three distinct geographies, that is, terai or flatland, hill and mountain areas. The CBNCP was piloted in four hill and six terai districts, constituting the ‘intervention area’, to which we assigned a ‘comparison area’ (figure 1). In both areas, one site was purposively selected for an additional qualitative component of the study; methods and findings of the latter are reported elsewhere.13

Figure 1

Map of Nepal showing intervention and comparison areas and qualitative study sites. Intervention area: four hill (ie, Dhankuta, Kavre, Palpa and Doti) and six terai districts (ie, Morang, Sungari, Parsa, Chitwan, Dang and Bardiya), comparison area: seven hill (ie, Udayapur, Sindhuli, Makawanpur, Lalitpur, Syangja, Baglung, and three terai districts (ie, Jhapa, Dhanusha and Kanchanpur).

The CBNCP targets all women of reproductive age, aiming to increase their interaction with the health system during pregnancy, delivery and the postnatal period. Our study was undertaken among women aged 15–49 years who had a live birth during 30 months preintervention compared with those with a live birth taking place during 7–14 months postintervention in view of Demographic and Health Survey (DHS) data being available for this period.

Study design

This quasi-experimental study uses propensity score matching and multiple data sources to assess the impact of the CBNCP (figure 2). It includes (1) before–after analysis of essential practices in the intervention versus comparison area based on DHS data, (2) before–after analysis of those same practices in the intervention versus comparison area based on Health Management Information System (HMIS) data and (3) analysis of training coverage and knowledge and skills of healthcare providers based on Newborn Health Information System (NHIS) data, which was an integral part of the CBNCP pilot and available in the intervention area only.12 14

Figure 2

Study design comprising quantitative and qualitative components. DHS, Demogaraphics and Health Survey; HMIS, Health management information system; FGD, Focus group discussion; KII, Key informant interviews; RDW, Recently delivered women; FIL, Father-in-laws; MIL, Mother-in-laws; FCHV, Female community health volunteer; CHW, Community health worker.

Drawing on the comprehensive evaluation framework for evaluating the scale-up for maternal and child survival by Bryce and colleagues,15 we developed a conceptual framework, which regards the CBNCP as a complex multicomponent intervention16 17 and graphically presents the presumed causal pathway from CBNCP implementation within the health system (process and outputs) through changed practices of pregnant or recently delivered women (outcomes) to impacts on neonatal health (figure 3). Importantly, while the CBNCP’s main impetus is on training of health workers, supplies of equipment and medicines as well as supervision and follow-up, several of the outputs (eg, taking a urine sample for proteinuria test) and outcomes (eg, postnatal visits) could also be considered as components of the intervention. This conceptual framework was critical in our identification of relevant outcome variables.

Figure 3

Conceptual framework.

Implementation of the CBNCP pilot through training of facility-based and community-based health workers and FCHVs started in May 2009 and was completed in July 2010 in pilot districts (see online supplementary table S1). Training dates were obtained from the MOH to define district-specific preintervention and postintervention periods used in the analysis of DHS and HMIS data; any births taking place during training were excluded from the analysis.

Propensity score matching

Propensity score matching is widely used to estimate the effects of health and other policy interventions, where RCTs are not feasible.18 19 It uses statistical techniques to construct a comparison group that is as similar as possible to the intervention group in an effort to reduce selection bias.20 21

Ten intervention districts were selected by the MOH in consultation with donors, considering development need, donor presence, district interest and ability to implement and monitor the programme (Parashuram Shrestha, personal communication, Nepal Ministry of Health, 2013). To reflect the propensity of a district to be selected for CBNCP implementation, we constructed a propensity score based on (1) the four components of the district Human Development Index value, (2) presence of donors involved in the CBNCP (ie, USAID, Unicef, SNL), (3) percentage rural population, (4) the MOH district performance rank and (5) road density (see table 1 for details).

Table 1

Background characteristics in intervention and comparison areas, based on various data sources

	Intervention area	Comparison area	t	p Value
Propensity score components
Human Development Index: life expectancy (years)*	61.23	62.88	−0.76	0.457
Human Development Index: adult literacy (%)*	51.40	54.38	−0.73	0.475
Human Development Index: school enrolment (%)*	2.77	2.88	−0.33	0.742
Human Development Index: gross domestic product purchasing power parity (PPP US$)*	1293.6	1315.2	−0.15	0.883
Urban population (%)†	16.79	17.85	−0.25	0.803
District performance score (average)‡ (as a proxy for a district’s leadership ability and proactiveness in implementing new initiatives)	74.25	73.77	0.28	0.781
Road density (km/km2)† (as a measure of access and ability to monitor the programme)	0.251	0.258	−0.07	0.941
Donor presence (average number)§	1.3	1.4	0.25	0.806
Population and health infrastructure characteristics¶
Population	4.9 million	4.4 million
Expected pregnancies (n)	142 000	128 000
Number of hospitals	14	11
Number of primary healthcare centres	39	39
Number of health posts	87	89
Number of subhealth posts	435	456
Number of private health institutions	49	38
Number of birthing centres	203	183
Population per birthing centre	24 159	24 330
Number of FCHVs	6903	7378
Population per FCHV	710	603

*UNDP. Nepal Human Development Report, Kathmandu, Nepal, 2004.

†District Profile of Nepal 2007/08: a socio-economic development database of Nepal, Intensive Study and Research Center of Nepal, Kathmandu, 2009.

‡MOH. District Annual Performance Criteria, personal communication, Ghanashyam Pokharel, 2011.

§AIN. Health Mapping Report, Association of International NGOs in Nepal, Kathmandu, 2008.

¶Health Management Information System database, made available on request by Management Division, 2010.

FCHV, female community health volunteer.

As CBNCP implementation was limited to hill and terai districts, mountain districts were excluded. We used the psmatch2 command in Stata Special Edition 1222 to identify suitable comparison districts based on the nearest-neighbour method without replacement. We checked for balance in the distribution of propensity score components (using t-tests) and population and health infrastructure characteristics (using χ2 tests) between intervention (10 districts pooled) and comparison areas (10 districts pooled).

Data sources and variables

Multiple data sources were used to enable as complete an analysis of impact as possible and to triangulate information between sources with different strengths and weaknesses. The DHS provides nationally representative data on fertility, health-relevant behaviours and childhood mortality based on a multistage cluster random sampling strategy.23 The data for the Nepal DHS for 2011 are in the public domain (www.dhsprogram.com). The HMIS, owned by the MOH and primarily based on health facility records, provides information about health service use, morbidity and mortality, treatment outcomes and the availability of commodities. We used data on regular service delivery for 2009–2011, publicly available online (http://www.dohs.gov.np). We also obtained CBNCP-specific NHIS data from the CBNCP secretariat based at the Child Health Division at the MOH.24 These NHIS data were collected by the programme team as part of CBNCP delivery and monitoring, and provided insights about the knowledge and skills of programme-trained health workers and FCHVs.

Neonatal mortality as the ultimate outcome of interest was not feasible to assess given available data sources and sample sizes. Instead, with reference to our conceptual framework (figure 3), we examined changes in six essential practices to improve neonatal health by incorporating relevant contributing practices in combined binary outcomes (coded as ‘better practices’ or ‘poorer practices’). Relevant covariates were identified a priori as family characteristics (ie, wealth quintile, rural vs urban location, caste/ethnicity), maternal characteristics (ie, age at delivery, education and access to media) and child characteristics (ie, sex, parity) (see table 2 for details).

Table 2

Baseline characteristics in intervention and comparison areas (in per cent) for most recent births to women aged 15–49 years in the 5 years preceding the survey based on DHS data

		Intervention area (n=533)	Comparison area (n=347)	χ2	p Value
Family characteristics
Location	Rural	86.0	85.6	0.02	0.929
Wealth index	Poorer*	31.4	51.7	44.09	0.003
Caste and ethnicity	Disadvantaged†	74.0	70.6	1.05	0.673
Maternal characteristics
Education	No education‡	36.5	45.0	24.82	0.072
Age at delivery	Higher risk age group§	31.9	23.0	6.92	0.022
Access to media	No¶	51.4	65.4	14.34	0.101
Child characteristics
Sex	Female	45.7	49.0	1.98	0.187
Parity	Higher risk parity**	56.5	51.1	2.12	0.211
Essential practices to improve neonatal health
Birth preparedness	Better practices††	6.2	4.9	0.63	0.568
Antenatal care seeking	Better practices‡‡	33.7	26.4	4.39	0.218
Antenatal care quality	Better practices§§	36.0	29.0	3.87	0.195
Delivery by skilled birth attendant	Yes¶¶	46.7	31.2	17.61	0.007
Immediate newborn care	Better practices***	74.4	64.3	8.63	0.091
Postnatal care within 48 hours	Yes†††	33.7	26.8	3.97	0.097

*Poorer: includes poorer and poorest quintiles, that is, lowest 40% in wealth ranking based on selected household assets.

†Disadvantaged caste and ethnicity: includes hill dalit, terai dalit, hill janajati, terai janajati, other terai caste and Muslim.

‡No education: includes illiterates and those without any formal education but may have some literacy classes.

§Higher risk group: those who delivered before 20 years or after 35 years.

¶No access to media: those reporting not listening or watching any public health radio or television programme in the last month.

**Higher risk parity: first or more than third parity.

††Birth preparedness: combined variable including saving money, organising transportation, finding a blood donor, identifying a health worker to assist with the delivery and purchasing a safe delivery kit; coded as ‘better practices’ if at least two items are fulfilled.

‡‡Antenatal care seeking: combined variable comprising number of antenatal visits (four or more), taking iron supplements (>90 tablets) and having been vaccinated against tetanus (at least two doses); coded as ‘better practices’ if all items are fulfilled.

§§Antenatal care quality: combined variable comprising whether the woman had her blood pressure taken, a urine and/or blood sample collected, and was told about pregnancy complications and where to go in case of complications; coded as ‘better practices’ if at least four items are fulfilled.

¶¶Delivery by skilled birth attendant: defined as delivery by a doctor, nurse or midwife at home or at a health institution.

***Immediate newborn care: combined variable comprising delayed bathing for 24 hours, drying, wrapping, placing the baby on the mother’s breast or belly, applying chlorhexidine or nothing on the umbilical cord, and initiation of breastfeeding within 1 hour of birth; coded as ‘better practices’ if at least three items are fulfilled.

†††Postnatal care within 48 hours: defined as any newborn examination by a health worker or female community health volunteer within 48 hours of birth.

DHS, Demographic and Health Survey.

Analysis

Difference-in-differences analysis estimates the change in outcome for the intervention area over a given time period by subtracting any change in outcome for the comparison area over the same time period. All outcomes were assessed as combined outcomes, that is, as the percentage of pregnant or recently delivered women adhering to ‘better practices’.25 Analyses for individual outcomes are provided as background information in online supplementary table S2.

For DHS data, difference-in-differences analysis using ordinary least squares regression was conducted for births occurring preintervention and postintervention. Where a woman had given birth more than once during the preintervention or postintervention period, only the most recent birth was included in the analysis to avoid non-independence of observations and to minimise recall bias. For HMIS data, a similar approach was adopted; however, tests of significance were not possible as the data were available only in aggregate at the district level. We also conducted logistic regression analysis of DHS data to examine if any differences between intervention and comparison areas persist after adjustment for all a priori identified covariates; here, the outcome was assessed at the individual level as either adhering or not adhering to ‘better practices’. All analyses were undertaken in Stata Special Edition 12.22

Findings

Baseline characteristics

Table 1 shows that intervention and comparison areas are balanced for propensity score components as well as relevant population and health infrastructure characteristics.

Using preintervention DHS data, 533 and 347 births took place in the intervention and comparison area, respectively. Table 2 compares outcome variables and covariates for the most recent births in the 5 years preceding the DHS survey. In both areas, a majority of children are from rural locations, disadvantaged families and born to a mother with at least primary education. While respondents from intervention and comparison areas are largely comparable, there are statistically significant baseline differences in relation to family wealth status, maternal age at delivery and delivery by a skilled birth attendant even after matching.

Intervention coverage

In the 10 pilot districts, a majority of health providers were trained, that is, 1615 facility-based health workers, 902 community-based health workers and 7072 FCHVs. Overall, knowledge and skills as reported or demonstrated were fair with some variation by type of provider; availability of drugs and commodities was also good (table 3). All of these, however, showed much variation between districts, pointing to concerns with respect to quality of training, supervision and logistics (see online supplementary table S3).13

Table 3

Intervention process indicators, based on NHIS data

	Unit	Facility-based health worker	Community health worker	Female community health volunteer
Training coverage
Number of individuals trained	Number	1615	902	7072
Knowledge
Knowledge of immediate newborn care messages (ie, thermal care, clean cord, skin-to-skin contact, immediate breastfeeding and delayed bathing)	% (SD)	70 (17.6)	62 (12.4)	57 (24.3)
Knowledge of correct dose of co-trimoxazole paediatric tablet	% (SD)	88 (11.5)	91 (5.6)	82 (16.5)
Skills
Ability to demonstrate hand washing correctly	% (SD)	81 (9.8)	68 (17.1)	60 (14.3)
Ability to demonstrate resuscitation steps correctly using a doll	% (SD)	53 (19.6)	37 (17.0)	27 (17.7)
Availability of drugs and commodities
Co-trimoxazole paediatric tablet	% (SD)	99 (1.6)	87 (12.6)	89 (10.2)
Gentamicin	% (SD)	95 (5.1)	78 (16.9)	–
Thermometer	% (SD)	–	–	85 (9.9)

NHIS, Newborn Health Information System.

Difference-in-differences analysis

Table 4 presents findings from the difference-in-differences analysis of DHS data. With the exception of birth preparedness (no change) and postnatal care within 48 hours (increase in intervention area, decrease in comparison area), improvements were observed but to a similar extent in both areas with no statistically significant differences. For all six essential practices, the percentage of pregnant or recently delivered women adhering to better practices was lower in the comparison area at both points in time.

Table 4

Difference-in-differences analysis for six essential practices to improved neonatal health (combined outcomes in per cent), for most recent births to women aged 15–49 years in the 5 years preceding the survey based on DHS data*

		Intervention area			Comparison area			Difference in differences	p Value
		Before (n=533)	After (n=168)	Difference	Before (n=347)	After (n=104)	Difference
Birth preparedness	Better practices	6.2	8.4	2.2	4.8	6.0	1.2	1.0	0.810
Antenatal care seeking	Better practices	33.7	49.7	16.0	26.4	33.2	6.8	9.2	0.383
Antenatal care quality	Better practices	47.4	59.9	12.5	34.8	37.8	3.0	9.5	0.290
Delivery by skilled birth attendant	Yes	46.7	57.7	11.0	31.2	37.6	6.4	4.6	0.577
Immediate newborn care	Better practices	74.4	85.9	11.5	64.2	79.9	15.7	−4.2	0.605
Postnatal care within 48 hours	Yes	33.7	44.6	10.9	26.8	17.4	−9.4	20.3	0.036

*See table 2 for details on variables.

Similarly, difference-in-differences analysis of HMIS data showed improvements in both intervention and comparison areas for most of the practices assessed13; HMIS does not provide information on birth preparedness or immediate newborn care practices. Table 5 compares findings based on DHS and HMIS data, showing congruent trends for all essential practices despite differences in the specification of some indicators. The contradictory finding that iron supplementation decreased postintervention in the HMIS (which collects data from public service providers) but not in the DHS analysis (which reflects households seeking care from both public and private providers) is explained by government health facilities having run out-of-stock in October and November 2011.

Table 5

Comparison of difference-in-differences analysis for selected antenatal, delivery and postnatal indicators (in per cent) between DHS and MIS data

Essential practices to improve neonatal health*	DHS					HMIS
	Intervention		Comparison		Difference-in-differences	Intervention		Comparison		Difference-in-differences
	Before	After	Before	After		Before	After	Before	After
Birth preparedness (combined)	6	8	5	6	1	–	–	–	–	–
Antenatal care seeking: antenatal care contact (at least one)	63	70	53	64	−4	69	81	73	78	7
At least four ANC visits	52	64	41	56	−3	36	43	35	46	−4
Iron tablet taken	78	87	77	80	6	74	62	73	58	3
Antenatal care quality (combined)	42	45	41	41	3	–	–	–	–	–
Delivery by skilled birth attendant	47	58	31	38	4	27	38	25	36	0
Immediate newborn care	74	85	69	79	1	–	–	–	–	–
Postnatal care within 48 hours	34	45	27	17	21	44	54	41	45	6

*See figure 3 for details on variables.

ANC, antenatal care; DHS, Demographic and Health Survey; HMIS, Health Management Information System.

Logistic regression analysis

The unadjusted ORs suggest statistically significant improvements in antenatal care quality (OR 1.8, 95% CI 1.1 to 2.9), delivery by a skilled birth attendant (OR 2.0, 95% CI 1.2 to 3.3) and postnatal care within 48 hours (OR 2.7, 95% CI 1.1 to 2.6) but not in the other three essential practices (figure 4). However, when adjusted for a priori identified covariates none of the changes in essential practices remained statistically significant.

Figure 4

Impact of CBNCP on six essential practices to improve neonatal health, based on logistic regression analysis of DHS data. ANC, antenatal care; CBNCP, community-based newborn care package; DHS, Demographic and Health Survey; PNC 48 hr, postnatal care within 48 hours; SBA, skilled birth attendant. *Adjusted for wealth quitile, location, cost, ethinicity, maternal age at delivary, maternal education, access to media, child sex and parity.

Discussion

Key findings and their explanation

Nepal’s CBNCP was developed based on existing studies, mostly from Nepal and South Asia, to ensure relevance to the country-specific or region-specific epidemiology, demonstrating effectiveness for a majority of the intervention components.14 The choice of interventions for integration within the package was driven by both effectiveness and feasibility considerations. However, there was no evidence for the effectiveness of the package as a whole,12 and the additional feasibility challenges of implementation at scale were probably not given sufficient attention.

The analysis of DHS and HMIS data suggests that the CBNCP did not have a significant impact on essential practices to improve neonatal health above a generally increasing trend in these practices. These findings must be interpreted with caution, given the relatively short time period between training health workers and FCHVs, which ranged from 7 to 14 months depending on the district, and assessment of relevant outcomes among programme beneficiaries. In light of the complex nature of the programme, where multiple components are intended to improve a whole range of health provider and population behaviours throughout pregnancy, delivery and the postpartum period, the present evaluation represents a very early assessment of potential impact.

Several factors are likely to interplay in explaining this current lack of impact.

Packaging of multiple interventions

The CBNCP bundled a range of specific measures in a complex package and implemented this across a large geographical area with an implementation modality largely dependent on the existing health system. In Nepal, the health system suffers from a number of problems and there is strong reliance on FCHVs. In contrast, prior studies, concerned with efficacy or effectiveness under real-world conditions, usually examined a single and relatively simple component (eg, chlorhexidine for cord care26) in a limited geographic area (eg, Mother and Infant Research Activity (MIRA)),27 implemented through a dedicated cadre of higher-level service providers (eg, Society For Education, Action and Research in Community Health (SEARCH28)) or undertaken as a distinct research project (eg, resuscitation29). It is therefore not surprising that the effectiveness of these interventions is diluted when merged in a package that is delivered by a lower-level service provider under ‘real life’ conditions. Indeed, a similar reduction of effectiveness when moving from research studies to large-scale implementation has been observed elsewhere.16 30 31 When going to scale, programme management, effective high coverage and a good match between community-based and facility-based service improvements are seen as critical.32–34

Healthcare providers and their training

The CBNCP was implemented through training of the existing cadre of facility-based (7 days) and community-based (5 days) health workers in the government system as well as FCHVs (7 days) with limited subsequent supervision and follow-up. Supervision is one of the most important elements of successful programmes, but also one of the most challenging programme elements to implement and assess. As a general indication, the Nepal Health Facility Survey35 reported that nearly 7 in 10 health facility-based workers received any kind of supervision visits during the previous 6 months. Comprehensive information on the extent and content of supervision in the context of the CBNCP is lacking, but anecdotal reports indicate concerns with respect to the frequency and effectiveness of supervision visits. While evidence from Nepal suggests that community health workers and FCHVs can identify and manage maternal and newborn health problems, this requires frequent training and mentoring.36 This study suggests much variation in programme performance across districts (see online supplementary table S3), generally indicating better results in areas where the CBNCP is implemented with more intensity. In addition, the qualitative component showed that service providers perceived the training as insufficient for them to be able to apply their skills confidently and to retain them over prolonged periods of time.13 Therefore, following the argument made by Kumar et al37 that the effectiveness of an intervention is constrained by the weakest link in the causal-intervention pathway, the amount of training and subsequent supervision for this complex intervention package are likely to have been insufficient to promote meaningful behaviour change. Moreover, in a setting where medical shops are perceived to be more convenient than government health facilities,35 38 a programme that does not involve private providers is likely to show limited impact. In relation to antenatal services, private providers often provide specific components of those services (eg, iron folic acid supplement) and on-call services.

Other relevant health initiatives

In the last decade, Nepal has witnessed a host of programmes to improve maternal and child health, with many of these directly or indirectly impacting neonatal health.2 As adjustment for other relevant ongoing initiatives was not feasible in design or analysis of this impact study, the observed trends in essential practices to improve neonatal health and the lack of CBNCP impact in intervention relative to comparison areas are in part likely to be due to the high level of background activity.

Strengths and limitations

Study design

The CBNCP is a complex intervention, where multiple components are intended to improve a whole range of health provider and population behaviours throughout pregnancy, delivery and the postpartum period. As its implementation was outside of the control of the researchers, randomisation was not feasible and we had to adopt a ‘natural experiment’ approach. While matching largely achieved balance between intervention and comparison areas, some baseline differences persisted. Moreover, we did not match individual intervention and comparison districts but intervention and comparison areas. A major strength in addition to propensity score matching is this study’s use of multiple data sources to assess impact.

Data

The DHS is a cross-sectional survey with retrospective recording of all pregnancies and births as well as relevant behaviours; it is thus subject to recall bias. DHS data are designed to be representative at the national level—for rare events, they are not necessarily representative at the district level and, consequently, assessment of impact on neonatal mortality was not feasible. The number of births covered is also limited, especially postintervention, as exposure time to the intervention was short (ranging from 7 to 14 months) to reflect true changes between areas. It is possible that changes in the behaviour of pregnant and recently delivered women will only become manifest after longer periods of time once health providers have internalised recommendations and implement them on a regular basis. The HMIS provides valuable information about healthcare use, knowledge and skills of service providers, and availability of key commodities and supplies in the health system. However, HMIS data are only available for the public sector and thus do not provide representative measures of coverage at population level, as many people rely on healthcare from informal and private providers.

Analysis

Use of multiple data sources and multiple statistical methods was an important strategy to validate findings or lack thereof. Difference-in-differences calculations are subject to limitations, as adjustment for confounders was not possible with the information available at district level. Filtering of births for analysis (ie, before, during and after implementation) was customised by district, and the analysis excluded births taking place during training as a conservative strategy. We used an a priori conceptual framework to define the outcomes of the intervention and to guide the analysis.

Implications for research and practice

Overall, this study highlights that the design, piloting and implementation of a complex intervention such as the CBNCP must be carefully planned and evaluated. In fact, the assumption that combining a large number of intervention components, even where their individual effectiveness has been proven, will yield an effective intervention package that can be successfully implemented at scale does not hold. Importantly, evaluating under ‘real life’ conditions is not necessarily straightforward and may require the use of limited-quality routine data in combination with innovative study designs. Even though the CBNCP, as assessed through our study, was conceived as a pilot, rigorous assessment through the MOH and donors was lacking; despite increasing concerns about the quality of CBNCP implementation and a potential lack of impact, implementation continued and was rapidly extended beyond pilot districts.

The findings presented here, supported by those of the qualitative component of the study,13 suggest that the programme may need a repackaging and tightening of content as well as a revision of its implementation modality. Components with high burden and greater effectiveness (eg, infections and care for low birthweight babies) should be strengthened, whereas components with lower burden and less effectiveness (eg, asphyxia) should be removed especially for FCHVs. With respect to implementation modality, more emphasis must be placed on focused, high-quality training of all involved healthcare providers and ongoing supervision and support.

The CBNCP has been scaled up to 39 districts of Nepal. The findings presented here, which were previously shared with CBNCP stakeholders, and a move towards more integrated approaches to improve child survival prompted a removal of selected components and integration of CBNCP interventions with the Integrated Management of Neonatal and Childhood Illness (IMNCI) programme. The IMNCI programme is currently being implemented in 35 districts and monitored in terms of programme coverage, quality and impacts on behaviours, health and equity.