Impact of home-based family planning counselling and referral on modern contraceptive use in Karachi, Pakistan: a retrospective, cross-sectional matched control study.

OBJECTIVES: To assess: (1) the impact of a reproductive health program on modern contraceptive use from baseline to program close; (2) the sustained impact from baseline to follow-up 36 months later; and (3) the exposure-adjusted impact at program close and follow-up.
DESIGN: Retrospective, cross-sectional matched control study.
SETTING: Karachi, Pakistan. PARTICIPANTS: 2561 married women aged 16-49 years.
INTERVENTIONS: The Willows Program, a community-based family planning counselling and referral program implemented from 2013 to 2015. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was community-level modern contraceptive prevalence rate (mCPR), measured for January 2013 (baseline), June 2015 (program close) and at follow-up 36 months later. A secondary outcome was exposure-adjusted mCPR (among women reporting a family planning home visit) at program close and at follow-up.
RESULTS: There was no significant effect on community-level mCPR at program close (2.4 percentage point increase in intervention over comparison; 95% CI -2.2 to 7.0) or at follow-up (1.9 percentage point decrease; 95% CI -6.7 to 2.8). Only 18% of women in the intervention area reported receiving a family planning visit in the preceding 5 years. Among those reporting a visit, we observed a significant 10.3 percentage point increase (95% CI 4.6 to 15.9) from baseline to close, and a non-significant 2.0 percentage point increase (95% CI -3.8 to 7.8) from baseline to follow-up, relative to matched women in the comparison area. The cost per new modern method user was US$1089, while the cost per user-year during the intervention period was US$455.
CONCLUSIONS: The program had a positive short-term effect on women who received a family planning visit; however, this effect was not sustained. Program coverage was low and did not significantly increase community-level family planning use. Findings highlight the need to increase community coverage of high-quality counselling and contextually relevant interventions for family planning demand generation. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We used the contraceptive calendar method to assess historical contraceptive use; this method is routinely used in population-based surveys but is subject to recall bias.

Our measure of program exposure relied on women’s self-report, and we do not have information on women who moved away from study areas either before or after program close.

Despite these limitations, our study had several innovative strengths, including use of a matched comparison site and coarsened exact matching, which is likely the most robust strategy available to estimate causal effects retrospectively.

Few family planning program evaluations include cost-effectiveness estimates; this study helps to fill this gap.

Few studies have evaluated the impact of interventions to generate demand for family planning; this study is an exception.

Introduction

Improving access to family planning (FP) is critical to achieving the Sustainable Development Goals (SDGs), particularly as they relate to women’s empowerment and gender equity, maternal and newborn health, and quality education.1 As of 2017, approximately 214 million women in the Global South who wanted to avoid pregnancy were not using a modern contraceptive method.2 Sub-Saharan Africa and Southern Asia account for 39% of all women in developing regions who want to avoid pregnancy and 57% of those with an unmet need for modern contraception.2 To address unmet need, it is essential to improve both access to, and uptake of, modern contraceptive methods, which reduce unintended pregnancies and promote healthy timing and spacing of births.

Pakistan’s unprecedented population growth combined with increasingly scarce resources has serious health, economic and social implications for the country, including rising rates of poverty and potentially dire environmental consequences. In the 1960s, Pakistan was one of the few Asian countries to implement a national FP program, although its implementation has been inconsistent.3 4 More recently, as a signatory to the Family Planning 2020 pledge, the government renewed its commitment and increased investments in FP considerably. Despite these commitments, Pakistan’s progress on FP has stalled and fertility decline has been slower than in most neighbouring countries.5 Nearly one in five married Pakistani women have an unmet need for FP, and the modern contraceptive prevalence rate (mCPR) remained static between 2012 and 2018 (26% and 25%, respectively).6

In urban Pakistan, modern contraceptives are widely available and most methods can be accessed free of charge through government facilities,7 yet contraceptive use remains low; in urban Sindh province, mCPR is 28%.6 Given this supply-uptake gap, community-based FP interventions that provide home-based information, education and counselling may be a pivotal strategy to generate and meet the latent demand for modern contraceptives.8 While such initiatives hold promise, few have been rigorously tested.

Previous evaluations provide some evidence that demand generation interventions contribute to increased uptake of modern contraception in low and middle-income countries (LMIC), but generally such evaluations are scarce, vary in quality and show mixed results.9 10 Programs diverge greatly in aspects of their implementation, including intended beneficiaries, intervention components and implementation length. In addition, evaluation indicators and outcome measures are not standardised. Reviews call for more robust study designs to identify the most effective types of interventions,9 10 including more experimental and quasiexperimental designs with comparison groups to identify causal relationships and potential pathways to impact.

A common demand generation approach in LMICs is to train local community health workers (CHW) to deliver FP services through household visits. Deployment of CHW as a ‘task-shifting’ strategy has been implemented in LMIC for decades, and a large body of research indicates that CHW can effectively provide basic reproductive, maternal and child health services, including some FP services.11–14 A 2015 review found that a majority (93%) of CHW-led FP initiatives increased the use of modern contraception, and 83% improved knowledge and attitudes regarding contraceptives.14 However, the same review points to a gap in research on CHW FP interventions implemented in urban areas; of the 56 studies reviewed, only five were conducted in exclusively urban settings.14

A more recent systematic review of FP counselling approaches, specifically, found that structured counselling for women initiating injectable contraception, long-acting reversible contraception and sterilisation increased contraceptive continuation in most sites studied.15 However, of the 61 studies included in the review, only four were community-based interventions, and most targeted only current contraceptive users. Consequently, little is known regarding the effectiveness of counselling-focused FP interventions that focus on non-users; and few programs have developed dynamic strategies that respond to women’s changing contraceptive needs and behaviours over time. A 2016 economic evaluation of FP interventions in LMIC suggests that relative to other strategies to reduce maternal morbidity and mortality, meeting the unmet need for FP is the most cost-effective strategy available.16 However, in a review of demand generation interventions in LMICs, Belaid and colleagues report that economic analysis was nearly absent in the literature, and none of the reviewed studies assessed cost-effectiveness.9

The Willows Program, a large-scale, community-based reproductive health intervention implemented in more than 60 sites across Turkey, Tanzania, Ghana and Pakistan, has the potential to fill some of these gaps. In this paper, we present results from a retrospective impact evaluation of the Willows Program in Korangi Town, Karachi, Pakistan.

Description of the Willows Program

Willows International’s mission is to ‘support women through improved knowledge and increased access to high quality services and products to help them achieve their reproductive goals’.17 The Willows Program aims to improve women’s access to modern contraceptive methods and safe abortion services in resource-limited, urban/periurban communities by training and deploying field educators, a cadre of paid workers recruited from the communities they serve, to provide door-to-door education, counselling and heath facility referrals to women of reproductive age. Willows selects intervention sites, in consultation with local health officials, based on community need for such services—densely populated areas with low modern contraceptive use despite availability of contraceptive services at nearby health facilities. Since Willows is primarily a demand generation intervention, field educators do not provide FP products directly to clients. This counselling and education-focused approach differs from other community outreach programs, a majority of which include both supply and demand-oriented activities.14 In addition, Willows staff liaise with local health authorities and service providers to ensure that FP services are available and accessible to clients who receive referrals, but unlike other demand generation programs,18 they do not provide clients with direct financial incentives such as cash transfers or vouchers. However, field educators do accompany women for referral, when requested.

In Korangi Town, Karachi, Pakistan, 35 field educators and seven supervisors were trained to deliver the Willows Program. Field educators had to be 18 years or older with a minimum of 12 years of schooling, living in the intervention area and able to communicate in local languages. All field educators were females, Muslim and a majority were below the age of 40. Selected field educators participated in a 2-week training workshop on reproductive health and FP, including modules on the effectiveness, benefits and side effects of various contraceptive methods. The training also covered counselling and communication strategies, and how to collect and report client information. Each field educator was designated a particular geographical area within Korangi Town based on the density of clients residing in the area. Supervisors provided support to field educators and reviewed data collected during household visits for accuracy and completeness. The program was implemented in Korangi Town between April 2013 and September 2015.

A hallmark of the Willows Program is its community-wide approach. The initial stage of the program involves a registration period in which field workers aim to enrol all eligible women living in the defined intervention area. In Korangi Town, all married women aged 15–49 years were eligible, and the program registered over 45 000 women between September 2012 and April 2013 (prior to the launch of field educator visits). In Pakistan, it is not socially acceptable to approach unmarried women to discuss contraception, thus the program enrols only married women.

Another key feature of the program is its prioritisation of women through routine data collection and use of a central system that organises clients into ‘prioritization’ categories. Willows uses their baseline registration data to prioritise women according to their reproductive and contraceptive status and childbearing goals and to plan the timing and frequency of future field educator counselling, visiting women with the greatest risk for unplanned pregnancy first. For example, women not using any contraceptives or those using traditional methods ineffectively are considered highest risk for pregnancy and thus receive highest priority for education and counselling visits (online supplemental figure S1).

During counselling visits, field educators provide clients with information on modern contraceptive methods, and refer those interested in contraception to local health facilities to consult with healthcare providers and select the method most suitable for their needs. They also collect data on women’s self-reported pregnancy status, current contraceptive use and satisfaction with their current method, and report this information back to the central Willows office on client information forms. Field educators document changes in clients’ contraceptive use at each visit and this information feeds back into Willows’ central database to create up-to-date prioritisation groups on a weekly basis. The central database also includes health facility data on clients’ referral completion. Field educators stop visiting a client if she switches to a permanent contraceptive method (tubal ligation or vasectomy).

In addition to counselling visits, field educators regularly conduct shorter ‘drop in’ visits with women who have recently adopted a modern method to ask if they are satisfied with the method. If a woman is not satisfied with the method, or has discontinued the method, a follow-up counselling session with the field educator is scheduled. These regular, repeated visits with prioritised clients are considered a key driver of contraceptive behaviour change. Although all enrolled clients are eligible to receive visits on a rolling basis, those considered at high risk for unintended pregnancy at baseline are prioritised for visits. Women who are not initially prioritised (eg, those using a modern contraceptive method) may change their contraceptive use following the baseline registration period. Since they are not required to report their changing status directly to field educators (ie, outside of a home visit), it is possible that the program misses some changes in clients’ contraceptive behaviour.

When deciding where to implement, Willows prioritises neighbourhoods where no other FP programs are present; however, near the end of the intervention in Korangi Town, the Aman Foundation’s ‘Sukh Initiative’ began working in some of the same neighbourhoods. The Sukh Initiative’s model is similar to Willows’, but they provide some FP supplies directly to women through a collaboration with the government’s Lady Health Worker (LHW) program.19 To minimise overlap of the two programs, Willows and Sukh staff coordinated the timing of their implementation in Korangi Town. Sukh implemented activities incrementally, and its field staff began making household visits in some areas covered by Willows in July 2015—only 3 months prior to Willows Program close.

In this study, we assessed the impact of the Willows Program in Korangi Town. Our objectives were to assess: (1) the impact of the Willows Program from baseline to program close; (2) the sustained impact of the program from baseline to follow-up 36 months later; and (3) the impact at close and follow-up among those reporting exposure to the program. We hypothesised that mCPR would be higher in the Willows intervention area compared with the comparison area at program close and 36 months after close.

Methods

Study design

We used a retrospective, cross-sectional matched control study design. Three years following the close of the Willows Program (between August and December 2018), we conducted a cross-sectional survey with married women in the Willows intervention site (Korangi Town) and matched comparison sites (PIB Colony and Dalmia/Shanti Nagar). We estimated both the immediate and sustained effects of the Willows Program with the understanding that Sukh activities may have partly contributed to any long-term effects observed. Online supplemental figure S2 summarises the Willows Impact Evaluation (WIE) survey timeline, including the timing of program implementation, data collection and outcome (mCPR) assessment.

Study setting

Korangi Town is a periurban neighbourhood located in East Karachi, and is home to an ethnically diverse population including Sindhi, Baloch, Urdu-speaking Muhajir, Pashtuns, and Gilgiti and Balti people. We selected comparison sites based on the following criteria: (1) no previous or ongoing FP programs in the area; and (2) similar socioeconomic characteristics as Korangi Town. To limit the risk of contamination, we selected non-contiguous comparison sites located approximately 16 km north of Korangi Town. PIB Colony and Dalmia/Shanti Nagar are periurban areas located in Karachi’s Gulshan Town. Separate health facilities serve intervention and comparison sites for FP and reproductive health, which further limits the possibility of contamination (online supplemental figure S3). A majority of residents in both intervention and comparison sites identify as Muslim.

Participants and procedures

We used a multistage cluster random sampling approach to create a representative sample of women in the intervention and comparison sites. First, we used geographic information system (GIS) mapping to outline the boundaries of each site, and constructed clusters of approximately equal size (about 80 households per cluster on average). This activity yielded 548 clusters in the intervention area and 160 clusters in the comparison area. Of these, we randomly selected 110 clusters from each site. Within selected clusters, we conducted a complete household listing to identify every woman of reproductive age living in each household, and randomly selected 25 households. If more than one eligible woman resided in a selected household, we randomly selected one for the study. Women were eligible for inclusion in this study if they met all of the following criteria: (1) 16–49 years of age at the time of the survey, (2) reside in the study area, (3) spoke either Urdu, English, Pushto or Sindhi, (4) currently married, and (5) not mentally/physically incapacitated.

We used an electronic tablet-based survey to collect current (2018) and retrospective (pre-Willows implementation; January 2013) sociodemographic data. We also collected data on women’s reproductive history, including their monthly contraceptive use for 5 years prior to the survey using the internationally validated contraceptive calendar approach.17 The contraceptive calendar module (Pakistan Demographic and Health Survey, PDHS) documents every birth, pregnancy and termination, as well as contraceptive use between events to generate a comprehensive monthly record of contraceptive history over a specified period. In our survey, the 5-year time frame covers the period from 3 months prior to the Willows Program launch (baseline) to approximately 3 years following program close.

The survey instrument was translated from English to Urdu and back-translated. The questionnaire was programmed using Dimagi’s CommCare application, and underwent extensive internal testing and multiple rounds of field piloting and refinement. Trained enumerators collected data under the supervision of senior researchers. We implemented a robust quality assurance protocol with built-in, automated checks to ensure consistency, identify erroneous entries in real time and minimise missing data. In rare cases, enumerators made follow-up visits to collect missing information from study participants. We also held weekly meetings with the field team throughout data collection to review errors and conduct refresher trainings where appropriate.

Variables and outcomes

The primary outcome was mCPR, expressed as the percentage of married women using a modern contraceptive method among all married women of reproductive age (16–49 years). Consistent with the PDHS definition, modern contraceptive methods included male or female sterilisation, intrauterine device (IUD), implants, injectables, contraceptive pill/oral contraceptives, male condoms, lactational amenorrhoea method and emergency contraception.

We calculated mCPR based on women’s self-reported use of a modern contraceptive method, expressed as a binary variable (not using or using) for the month of January 2013 (baseline), June 2015 (Willows close) and at WIE survey. While the Willows Program officially closed in September 2015, we redefined the close date to June 2015, the month before Sukh Initiative began implementation. This allowed us to isolate the immediate effect of the Willows Program alone. We also collected data on potential confounding variables including women’s characteristics in 2013: age, education, religion, ethnicity, work status, parity and modern contraceptive use at baseline.

Primary and secondary analyses

In our primary analysis, we restricted the sample to a panel of married women aged 16 years or older at baseline who lived in the study sites continuously from January 2013 until the follow-up survey in both intervention and comparison sites (n=2561). This isolates the effect of the intervention on women living in the intervention area throughout the program period who had the highest likelihood of exposure to intervention, as intended by Willows. Therefore, we excluded from this analysis women who migrated in at any time after the launch of the program. Primary analysis (hereafter referred to as ‘community-level analysis’) followed an intention-to-treat approach, whereby all intervention-area women in the panel sample were considered ‘exposed’ to the program, which reflects the stated intention of Willows to reach all women of reproductive age, and contribute to community-level changes in modern contraceptive use.

In a secondary analysis (hereafter referred to as ‘exposure-adjusted analysis’), we measured program exposure based on women’s reports of having any FP visitor to discuss pregnancy prevention or termination of unwanted pregnancy in the past 5 years. We consider this a conservative approach to measure program exposure, since it is possible that women recall having a visitor but not the name of the organisation, or they may not recall either.

Statistical analysis

Due to the non-experimental nature of our study design, we used statistical matching to control for observable individual-level baseline characteristics that might confound the relationships between exposure and outcome. Since the Willows Program itself was not randomised, assessing the main effects on a matched sample was critical to ensure balance between the intervention and comparison sites.20 To do this, we employed coarsened exact matching (CEM) techniques to assess the effect of the Willows Program on mCPR between baseline and Willows close (immediate effect) and baseline and follow-up (sustained effect). The CEM procedure compares each ‘treated’ (ie, living in intervention area) woman with a matched ‘control’ woman from the comparison area.20 Recent research suggests that CEM improves balance between groups and reduces observable bias,21 and is thus preferable to other more commonly applied strategies, such as propensity score matching, which tends to reduce balance and increase bias.22

We matched on the following observable baseline (year 2013) characteristics: age group (six categories), work status (yes/no), education level (none, primary or higher), religion (Muslim vs Christian or other), ethnicity (Urdu, Sindhi, Punjabi, other), parity (0–1 or 2+ children); and modern method use (using/not using). We selected these variables a priori because they are well-known predictors of contraceptive uptake and use. We dropped observations that had missing data for any of the aforementioned variables (n=8). We then coarsened the data, creating groups of women for each subcategory of these variables, and created a matched data set using these groups. To assess mCPR, we used multivariate logistic regression, with intervention and comparison respondents matched via CEM. We conducted weighted descriptive analyses and logistic regressions, and adjusted SEs for clustering.

Our study was powered to detect a 5 percentage point difference in mCPR between intervention and comparison sites, allowing for 20% attrition. For all analyses, a two-tailed test with significance at the 0.05 probability level was considered a statistically meaningful effect. We performed all analyses using STATA V.15 software.

Cost-effectiveness analysis

We obtained a detailed record of program costs incurred from Willows International and used this information to calculate cost-effectiveness. We used two main outcomes for effectiveness. One is the number of new users of modern contraceptive methods due to the Willows intervention at the close of the intervention. We calculated the number of new users by multiplying the population of eligible women in the intervention area and the treatment effect on those eligible women. We used the data from the household listing to estimate the total population of eligible women in the intervention area. Additionally, we estimated the number of new users of modern contraceptive methods by multiplying the eligible population by the proportion visited and the treatment effect on the visited women.

A second approach is to calculate the effectiveness by the number of modern user-years generated. This allows for the possibility of a sustained effect of the intervention, creating benefits even after the project had ended, and possibly improving cost-effectiveness by taking into account these longer term gains. We calculated the number of user-years by summing the number of new users each year over the 6 years of data collected in the follow-up survey. We calculated all cost-effectiveness estimates for two samples: (A) the panel sample of eligible women living in the intervention area throughout the study period; and (B) the sample of women who reported an FP visit.

Patient and public involvement

No patients were directly involved in this study. However, study tools were piloted extensively with women from local communities to ensure appropriate language, literacy levels and cultural interpretations of concepts and question phrasing. Key community stakeholders will have the opportunity to attend research dissemination meetings.

Ethical considerations

At the time of the survey, there were a number of active political campaigns in Karachi, which raised security concerns among residents of our study areas. As a result, participants were reluctant to provide a signature or thumb print, especially on electronic devices. In this context, signing was considered a risk for identity exposure. Due to these concerns, women provided verbal consent to participate in the study. Enumerators signed consent forms to indicate that women provided verbal consent.

Results

The household listing in both intervention and comparison sites identified 14 400 households with 5384 eligible women. The overall sample included 4197 women (2064 in the intervention and 2129 in the comparison site) with contact and consent rates of 83% and 94%, respectively. The main reason for non-contact was an inability to locate women after three attempts (usually due to extended travel or relocation). The main reason for non-consent was women’s reluctance to take part in FP discussions. The reduced (panel) sample used for analysis included 1210 women in the intervention site and 1351 women in the comparison site (figure 1).

Figure 1

Sampling and study flow diagram. C, comparison; I, intervention.

Table 1 summarises the reported characteristics of study participants as of January 2013, that is, before the launch of the Willows intervention (also referred to as ‘baseline’) and mCPR at each time point. Age distribution, religion, region of birth, parity and baseline modern contraceptive use did not differ significantly between intervention and comparison sites. There were significant differences in education level, with more women in the comparison area having no education, but equal percentages with secondary or higher. Ethnicity also differed between the two sites, with a larger proportion of Urdu-speaking women residing in the intervention site (66% vs 40%).

Table 1

Baseline (2013) characteristics and modern contraceptive use of panel sample: married women aged 16–49 and living in study areas continuously since January 2013

	Intervention (n=1210)	Comparison (n=1351)
Age group
<20	45 (3.7%)	53 (3.9%)
20–24	234 (19.3%)	230 (17.0%)
25–29	293 (24.2%)	333 (24.6%)
30–34	319 (26.4%)	321 (23.8%)
35–39	182 (15.0%)	236 (17.5%)
40–44	137 (11.3%)	178 (13.2%)
Education†
None	335 (27.7%)	445 (32.9%)
Primary	175 (14.5%)	167 (12.4%)
Middle	164 (13.6%)	141 (10.4%)
Secondary	351 (29.0%)	285 (21.1%)
Higher	185 (15.3%)	313 (23.2%)
Religion
Islam	1144 (94.5%)	1252 (92.7%)
Christian or other	66 (5.5%)	99 (7.3%)
Ethnicity†
Urdu	793 (65.5%)	543 (40.2%)
Sindhi	49 (4.0%)	134 (9.9%)
Punjabi	172 (14.2%)	181 (13.4%)
Other	196 (16.2%)	493 (36.5%)
Region of birth
Sindh province	941 (77.8%)	1011 (74.8%)
Other region	269 (22.2%)	340 (25.2%)
Working
No	1077 (89.0%)	1209 (89.5%)
Yes	133 (11.0%)	142 (10.5%)
Parity
0	46 (3.8%)	62 (4.6%)
1	73 (6.0%)	92 (6.8%)
2	190 (15.7%)	255 (18.9%)
3	316 (26.1%)	308 (22.8%)
4+	585 (48.3%)	634 (46.9%)
Modern method use (mCPR)
January 2013 (baseline)	350 (28.9%)	355 (26.3%)
June 2015 (program close)*	468 (38.7%)	469 (34.7%)
August to December 2018 (follow-up)	505 (41.7%)	534 (39.5%)

*P<0.05.

†P<0.01.

mCPR, modern contraceptive prevalence rate.

Community-level analysis

Online supplemental figure S4 plots the sample-weighted mCPR estimates at baseline, program close and follow-up in 2018 before adjusting for differences between study sites using CEM. In the intervention site, mCPR increased from 27.9% at baseline to 37.1% at program close and to 40.7% at follow-up. In the comparison site, mCPR increased from 25.8% to 32.9% at close, and to 38.7% at follow-up. After performing CEM, we found no significant effect on mCPR from Willows baseline to close, or at WIE follow-up (table 2).

Table 2

Community-level CEM results

Sample	Sample size/effect
Intervention	1210
Comparison	1351
Number of women matched (intervention)	1166 (96%)
Number of women matched (comparison)	1256 (93%)
Number of strata	243
Number of strata matched	149
L1 statistic	0.09
mCPR—baseline to close	2.4 percentage point increase in intervention over comparison(95% CI −2.2 to 7.0, p=0.31)
mCPR—baseline to follow-up	1.9 percentage point decrease in intervention over comparison(95% CI −6.7 to 2.8, p=0.43)

CEM, coarsened exact matching; mCPR, modern contraceptive prevalence rate.

Exposure-adjusted analysis

Only 18% of women in the intervention site reported receiving any FP visit within the 5-year period preceding the survey, compared with 0.44% of women in the comparison site. Since program exposure was lower than expected, we repeated the same analysis on a subsample of women to examine the effects on those who reported receiving an FP visit compared with women in the comparison area who did not report a visit. Online supplemental table S1 summarises the baseline characteristics of study participants in this subsample.

Online supplemental figure S5 plots the sample-weighted mCPR estimates at baseline, program close and follow-up in 2018 among women exposed to the program, before adjusting for differences between study sites via CEM. Among women who received an FP visitor, mCPR increased from 28.7% at baseline to 45.0% at program close and to 47.2% at follow-up. In the comparison site, mCPR increased from 25.7% to 32.7% at close, and to 38.7% at follow-up. After performing CEM, we found a significant effect at program close and no significant effect at follow-up (table 3). CEM results from both community-level and exposure-adjusted analyses are summarised in figure 2.

Table 3

Exposure-adjusted CEM results

Sample	Sample size/effect
Intervention	222
Comparison	1344
Number of women matched (intervention)	212 (96%)
Number of women matched (comparison)	916 (68%)
Number of strata	216
Number of strata matched	75
L1 statistic	0.12
mCPR—baseline to close	10.3 percentage point increase in intervention over comparison(95% CI 4.6 to 15.9, p<0.001)
mCPR—baseline to follow-up	2.0 percentage point increase in intervention over comparison(95% CI −3.8 to 7.8, p=0.50)

CEM, coarsened exact matching; mCPR, modern contraceptive prevalence rate.

Figure 2

Estimated effect sizes based on matched (CEM) analyses from baseline to close, and from baseline to follow-up. Effect sizes were calculated at the community level (blue), and after adjusting for program exposure (green). CEM, coarsened exact matching.

Using the effect size for the community-level sample (all women), the cost per new modern method user was US$1089, while the cost per user-year during the intervention period was US$455. The long-term cost per user-year (ie, until the follow-up survey in 2018) was US$484. Using the effect size for the sample of women who reported an FP visit, the cost per new user was US$1412, while the cost per user-year during the intervention period was US$672. The long-term cost per user-year was US$378 (online supplemental table S2).

Discussion

This evaluation found that the Willows Program fell short of reaching all eligible women in the community and had no significant effect on community-level modern contraceptive use in Korangi Town, Karachi, Pakistan. However, among women who reported an FP visit, we found a significantly positive, immediate effect at program close. Although the program intended to reach all women of reproductive age in the intervention area, only a small proportion (18%) of women surveyed reported receiving an FP visit. This suggests that challenges during implementation may have prevented the program from achieving its intended reach, and ultimately, its potential impact.

Collecting and analysing data on program implementation, including detailed documentation of context, field procedures and intervention fidelity, is critical to understanding program impact, interpreting outcomes and generating recommendations for replication in other settings.9 Because this was a retrospective study, we were unable to conduct a process evaluation at the time of program implementation. It is therefore difficult to determine the degree to which barriers to implementation may have influenced our results, and what aspects of the intervention were implemented well. Future evaluations would benefit from qualitative interviews with program staff, field educators and program participants to understand these mechanisms.

It is possible that Willows encountered difficulties identifying women who were available and willing to participate, particularly if women themselves, their husbands or their in-laws were hesitant to discuss FP matters or had reservations about contraceptive use.23 24 Previous research suggests that involving partners and/or other family members during counselling sessions may enhance the impact of FP counselling.9 25 Alternatively, it may be that additional human resources (field educators, supervisors) were required to cover the entire intervention area efficiently, or that field educators needed additional incentives and/or more supportive supervision to stay motivated.26–28 Lastly, it is likely that some women who were using a contraceptive method at baseline (and therefore deemed ‘low priority’) discontinued using contraception later. By prioritising women according to their status at baseline, and conducting counselling visits only with those classified as ‘high priority’ at that time, the Willows Program may have missed opportunities to respond to women’s changing contraceptive needs throughout the intervention period. Repeated assessment of women’s contraceptive status would address this issue.

The limited community coverage and high cost per user raises questions about the feasibility of the Willows Program. However, our finding that women who received an FP visit had higher modern contraceptive use at program close provides some evidence that provision of individualised, home-based counselling and referral by trained field workers may be an effective short-term behaviour change strategy in this setting. We did not detect a significant effect at follow-up despite subsequent implementation of a similar FP program (the Sukh Initiative) in several neighbourhoods of our intervention site, which raises questions about the sustained impact of such programs. It would have been useful to assess the individual and combined effects of Willows and Sukh, but we did not design this study to assess both programs. The Sukh Initiative’s midline (2016–2017) evaluation reports a ‘significant association between door-to-door counselling and the use of contraceptive methods’. While these findings support the strategy also used by Willows, the study had a cross-sectional pre-post design, with no comparison, and therefore results cannot be attributed solely to the Sukh Initiative. Additionally, the study does not factor in the potential effect of exposure to other programs.

Our evaluation differs from previous studies of FP initiatives in Pakistan in two important ways. First, most emphasise other demand generation strategies, such as social franchising and/or vouchers for contraceptive services/methods.29 30 For example, the Marie Stopes Society’s ‘Suraj’ program trained private providers to administer contraceptive methods, trained field workers to conduct community mobilisation activities and implemented marketing and branding strategies along with a voucher scheme for prospective clients.31 Azmat et al30 report a 19.6% increase in CPR and a significant increase in uptake of IUDs, which were promoted with vouchers. The same study found a 22.7% increase in modern contraceptive use as a result of the Suraj program, thus authors advocate for multipronged approaches to improve contraceptive uptake.

Second, since the Willows Program focuses solely on door-to-door information, counselling and referral, our evaluation offers an opportunity to isolate the effect of a single demand generation strategy. Some research suggests that multicomponent health interventions can have more than the additive impacts of individual components32; however, when resources are limited, it can be helpful to disentangle intervention components to determine which specific strategies are most effective.9 14 Most FP studies report the overall impact of multiple intervention components. For example, a program in Rwanda trained CHWs to deliver FP counselling, trained facility-based nurses on long-acting reversible contraceptives (LARCs) and provided logistical support in government clinics.25 The overall intervention was associated with increased and rapid uptake of LARCs, but it is unclear which component was the bigger driver of contraceptive uptake. A national evaluation of Pakistan’s LHW program found that rural women served by LHW delivering door-to-door maternal and child health services (including FP) were more likely to use a modern reversible contraceptive method than women in communities not served by the program.33 However, because LHWs have multiple responsibilities (eg, counselling, referral and direct provision of condoms and oral contraceptive pills), it is difficult to isolate the impact of counselling and referral services alone. An exception is a quasiexperimental trial in Bangladesh, which isolated the impact of adding FP services to a broader maternal and newborn health program.34 Researchers demonstrated that integrating FP services into home-based visits by CHWs improved both postpartum contraceptive use and birth spacing outcomes.34

Substantial evidence demonstrates the effectiveness of community-based health workers in delivering a range of preventive and curative services related to reproductive, maternal, newborn and child health.11 12 35 However, a host of challenges have been documented, including poor integration into national healthcare systems, and an over-reliance on donor-driven management and funding that threaten the sustainability of CHW-focused strategies.36 These challenges may also apply to the Willows Program, which is ultimately funded and overseen by donors. When non-governmental organisations train their own cadre of field workers, independent of the existing health system, short-term gains may be achieved at the expense of meaningful long-term impacts. To ensure sustained outcomes of Willows and similar initiatives, organisations must partner more closely with existing government-based systems, such as the LHW and community midwife programs in Pakistan.

This study had some limitations and may not be generalisable beyond other low-income, urban areas that share similar socioeconomic characteristics to our study sites. First, we used the contraceptive calendar method to assess historical contraceptive use over a 5-year period (2013–2018). While this is standard practice in population-based survey research, it is subject to recall errors, and the validity of participants’ retrospective reports is uncertain, particularly for women with complex reproductive histories.37 The potential for recall errors likely increases with time. Second, our measure of program exposure relied on women’s self-report, thus it is possible that some women received a visit from Willows but did not remember it, especially if their most recent memory was that of a Sukh visitor and they reported as such. Third, we do not have information on women who were registered and exposed to the Willows Program, but moved out either before or after program close. Fourth, while we adjusted for known differences between the intervention and comparison areas by matching on these variables, as with all matched analyses, it is possible that other unmeasured factors influenced the outcome. Lastly, due to the retrospective design of our study, we do not have reliable measures of counselling quality. Future studies could observe field educators via direct observation, and/or interview women shortly after home visits to assess clients’ satisfaction with the counselling, and their recollection of the content covered.

Despite these limitations, our study had several innovative strengths, including a cost-effectiveness analysis, and use of a matched comparison site and CEM techniques. It is typically infeasible to randomise large programs, and very few have built-in prospective evaluation designs; therefore, using matching techniques to estimate causal effects may be the most robust strategy available. We are aware of only one other study—a social franchising evaluation in Kenya—that applied CEM techniques to assess FP outcomes.21 Second, the intervention and comparison sites were relatively similar, there were no other FP programs operating in the comparison site and baseline measures of the main outcome (mCPR) were similar in both sites. Recent work cautions that when baseline outcome estimates differ between treatment and comparison groups, there is a risk of regression to the mean bias.38 This was not an issue in our study. Third, the comparison area is geographically distinct from the intervention area, thus the chance of contamination between study sites was low. Lastly, this study adds to limited existing research on the impact of FP demand generation programs among the urban poor. With urbanisation rates projected to increase in Pakistan and in other LMICs, identifying effective strategies to address reproductive health inequities in urban centres will be paramount to achieving the ambitious targets laid out in the SDGs.

Conclusion

The program had a positive short-term effect on women who received an FP visit; however, this effect was not sustained. Program coverage was low and did not significantly increase community-level FP use. We recommend that future evaluations occur prospectively, and include complimentary, comprehensive process evaluations to document barriers to, and facilitators of, community-wide implementation. Findings provide strong rationale for investing in ways to increase community coverage of counselling initiatives as one component of broader intervention packages targeting improved contraceptive access and uptake.