Process Evaluation of a Large-Scale Community-Based Nutrition Program in Malawi.

BACKGROUND: Global attention to the study of nutrition program implementation has been inadequate yet is critical for effective delivery and impact at scale.
OBJECTIVES: The objective of this mixed-methods process evaluation study was to measure the recruitment, fidelity, and reach of a large-scale, community-based nutrition program in Malawi.
METHODS: The nutrition program delivered a small-quantity lipid-based nutrient supplement (SQ-LNS) and social and behavior change communication (SBCC) to improve infant and young child feeding (IYCF) and water, sanitation, and hygiene (WASH) practices in households with children aged 6-23 mo. Program monitoring and evaluation data were used to measure program recruitment, reach, and fidelity. Structured direct observations and knowledge questionnaires with program volunteers measured quality aspects of program fidelity. The number of times activities were done correctly was used to tabulate proportions used to represent program functioning.
RESULTS: Half (49.5%) of eligible children redeemed program benefits by 8 mo of age during the first 4 y of program implementation. Implementation of training activities for SBCC cadres exceeded program targets (100.6%), but the completion of certain modules (breastfeeding and complementary feeding) was lower (22.9% and 18.6%, respectively). Knowledge of IYCF, WASH, and SQ-LNS messages by volunteers was >85% for most messages, except ability to list the 6 food groups (35.7%). Structured direct observations of SQ-LNS distributions indicated high fidelity to program design, whereas those of household-level counseling sessions revealed lack of age-appropriate messaging. Program reach showed participation in monthly distribution sessions of 81.0%, group counseling of 93.3%, and individual-level counseling of 36.9%.
CONCLUSIONS: This community-based nutrition program was implemented with high fidelity and quality, with specific interventions requiring further attention. The documentation of implementation contributes to our understanding about how program impacts were achieved.

Introduction

Child stunting affects 149 million children globally and is representative of a number of complex, interrelated factors that contribute to a chronic deficiency of essential nutrients during critical periods of growth and development (1). Global progress toward the reduction of child undernutrition has been extremely slow, especially in sub-Saharan African countries like Malawi, where, until 2015, stunting prevalence was ∼50% for >2 decades (2–4). In 2011, Malawi joined the Scaling Up Nutrition (SUN) movement, demonstrating its political commitment to reducing undernutrition among its vulnerable populations (5). Part of these efforts were later exemplified in the rollout of a large-scale community-based nutrition program in January 2014 that included the blanket distribution of a daily ration of a small-quantity lipid-based nutrient supplement (SQ-LNS) and social and behavior change communication (SBCC) messages to all participating mothers of children aged 6–23 mo and to the community about optimal infant and young child feeding (IYCF) and water, sanitation, and hygiene (WASH) practices. Both of these selected interventions have demonstrated an impact on child anthropometry and intermediate outcomes including improved feeding and care practices in similar settings (6–11), but whether such an impact can be sustained in this specific geographic context and population, and when delivered as a package within a district-wide public health program, required further evaluation.

The impact of the program on child nutrition outcomes (anthropometry and key feeding and WASH behaviors) was measured by an independent impact evaluation led by Johns Hopkins University (Christian P, Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, unpublished results, 2017; Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, Christian P, unpublished results, 2016). After 4 y of implementation, the cross-sectional impact evaluation found no differences in mean length-for-age z-scores or prevalence of stunting, but mean weight, weight-for-age z-score, and midupper arm circumference (MUAC) were all significantly higher in children who had received the intervention (Christian P, Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, unpublished results, 2017; Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, Christian P, unpublished results, 2016). In a longitudinal subsample of the impact evaluation, length-for-age z-score was also higher among children who received the intervention (Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, Christian P, unpublished results, 2016). The program also contributed to improvements in child morbidity, IYCF practices (including dietary diversity), and hand-washing practices in the program compared with comparison district (Christian P, Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, unpublished results, 2017; Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, Christian P, unpublished results, 2016).

This process evaluation study was conducted under the umbrella of the impact evaluation, to unpack the why and how interventions did or did not achieve their impact (12). Process evaluation, which measures the extent to which planned program components are actually implemented (13), is especially important and relevant in implementation research and in nutrition given the complexity of social and behavioral interventions, and the various and interrelated determinants of nutrition (14), which render nutrition impact variable and difficult, and especially under programmatic conditions (15, 16). The use and value of process evaluation in nutrition has been exemplified by the Alive & Thrive Initiative in Bangladesh, Ethiopia, and Vietnam, which identified implementation bottlenecks and challenges throughout the life of the respective programs (16, 17). Although limited, these examples magnify the role of process evaluation in making program adjustments, in understanding which program components were implemented with high (or low) fidelity to the planned protocol, and how they were received by program staff and beneficiaries alike. In the context of an impact evaluation, process evaluation goes 1 step further in clarifying whether measured impact was truly a result of the program's implementation (18).

The objective of this study was to use the principles of implementation science, “the study of methods to improve the uptake, implementation, and translation of research findings into routine and common practices” (19), to measure the program's conformity to its original design using process measures of program recruitment, fidelity, and reach.

Methods

The community-based nutrition program

The community-based nutrition program was launched in the district of Ntchisi in January 2014 by the Government of Malawi, with the technical and financial support of the World Food Programme (WFP) and the Children's Investment Fund Foundation, respectively. World Vision was responsible for day-to-day implementation and logistics, including training and supervision of program volunteers and conduct of SQ-LNS distributions. The program included 4 intervention components, of which only the first 2 were evaluated in this process evaluation study: 1) the monthly blanket distribution of a daily ration of SQ-LNS (20 g, 110 kcal; Nutributter; Nutriset), to all children aged 6–23 mo in the program district; 2) the monthly delivery of SBCC messages to mothers and the community on optimal IYCF and WASH practices; 3) scaling up of the coverage of community-based management of acute malnutrition to 100% of health facilities in the program district; and 4) nutrition-sensitive actions such as promotion of small livestock and home gardens among agriculture leaders in the program district to improve access to nutrient-dense foods being promoted by the program. The monthly distribution of SQ-LNS was done through 80 distribution points across the district, which were either health centers or “extended distribution points” (EDPs) (i.e., health posts, churches, schools, or other community structures).

The program's SBCC campaign fell under the larger, countrywide SUN (5) initiative, which included the use of nationally vetted and standard communication materials and messages that were based on rigorous formative research (20). Nutrition education for the program was implemented under a care group model (Supplemental Figures 1 and 2), which included monthly one-on-one counseling by care group volunteers (CGVs), group counseling by promoters and health surveillance assistants (HSAs) at monthly SQ-LNS distributions, and community-level activities such as nutrition days (once per year) and mass media messaging utilizing billboards and radio messages that featured community members. Training of CGVs was done on a rolling basis, covering 1 module every 2 wk, with the entirety of training modules requiring approximately 4 mo. Refresher trainings were also conducted throughout the duration of program implementation dependent on need.

Process evaluation study

This process evaluation study was conducted under the umbrella of the community-based nutrition program's impact evaluation, which was registered at clinicaltrials.gov (NCT #0,298,5359) and is described in detail elsewhere (21). A mixed method design was used to define and test the implementation theory of the nutrition program's first 2 interventions () using process indicators of recruitment, fidelity, and reach (13). Implementation theory was defined as the details of how a program is implemented: the inputs, process, and delivery of the program's main components that “intend to bring about the desired interactions with the target population and provide the planned services” (22). The program's third and fourth components were not included in either this process evaluation or the broader impact evaluation study because they were facility-based (as opposed to community-based) interventions (component 3) or were introduced in a second phase of program implementation (component 4).

FIGURE 1

Implementation theory for the Malawi community-based nutrition program. CGV, care group volunteer; CIFF, Children's Investment Fund Foundation; IYCF, infant and young child feeding; LNS, lipid-based nutrient supplement; MoH, Ministry of Health; SBCC, social and behavioral change communication; SQ-LNS, small-quantity lipid-based nutrient supplement; SUN, Scaling Up Nutrition; WASH, water, sanitation, and hygiene; WFP, World Food Programme; WVI, World Vision International.

Various sources of data were used to examine the quality of program implementation and conformity to its original design (). The timing of data collection for each of these activities is described here briefly; additional detail is available in a publication of the program's monitoring and evaluation systems (21).

TABLE 1

Summary of data collection methods, sample, and indicators used for the process evaluation of the Malawi community-based nutrition program

Type of process variable	Objective	Data source	Sample size	Measurement/indicators
Program recruitment	To clarify procedures and methods used to recruit participants, and identify reasons for nonparticipation	Review of available program design and implementation documentsInformal discussions with program design and implementing staff	N/A	Description of procedures used to attract and recruit participants
		Program M&E systems: SCOPE	n = 31,2562	• Eligible beneficiaries registered by 6–7.9, 8–11.9, 12–17.9, and 18+ mo of age, %
Program fidelity: quality	To evaluate the extent to which the program is being delivered as it was intended, including an assessment of the quality and integrity of its delivery	Direct observations of SQ-LNS distributions	n = 143	• Functioning of card scanning technology, %
				• Duration of the distribution session, mean h
				• Adequate stock of SQ-LNS at the distribution, %
				• Correct distribution of SQ-LNS, %
				• Courteous treatment of mothers at distribution, %
				• Communication of SQ-LNS usage messages, %
				• Communication of IYCF and WASH messages, %
				• Receipt of group counseling at distribution, %
		Direct observations of CGVs delivering SBCC	n = 14	• Communication of IYCF and/or WASH messages, %
				• Communication of SQ-LNS usage messages, %
		Knowledge assessments with promoters	n = 14	• Training completed by promoters and CGVs, %
				• Promoter and CGV knowledge of SQ-LNS messages
		Knowledge assessments with CGVs	n = 70	• Promoter and CGV knowledge of IYCF messages
				• Promoter and CGV knowledge of WASH messages
		Program M&E systems: PDM surveys	n = 4320	• Waiting time at distribution point, median h
Program fidelity: dose delivered	To measure how much of the program's intended activities were delivered to the target beneficiaries	Program M&E systems: APR	N/A	• Health staff trained in program delivery, %
				• Training of SBCC/volunteer cadres, %
				• Open nutrition days conducted, %
				• Billboards erected, %
				• Promotional campaigns on IYCF at health facilities, %
Program fidelity: dose received	To measure the extent to which all program components are received and utilized by target beneficiaries	Knowledge assessments with CGVs	n = 70	• Households visited per month by CGVs, mean
		Program M&E systems: PDM surveys	n = 4320	• Target beneficiaries who attended a community event, %
				• Target beneficiaries who heard a radio message, %
				• Mothers who recognized ≥3 nutrition messages, %
				• Consumed 1 sachet of SQ-LNS in last 24 h, %
				• SQ-LNS shared, %
				• Households experiencing a gap in SQ-LNS supply, %
				• Gap days among households experiencing a gap in supply, mean
Program reach	To measure the % of eligible beneficiaries who are reached by the program	Program M&E systems: SCOPE	n = 41,617	• Target beneficiaries attending monthly SQ-LNS distribution sessions, %
		Program M&E systems: PDM surveys	n = 4320	• Target beneficiaries who received a CGV home visit, %
				• Target beneficiaries who received a group counseling, %

APR, annual performance review; CGV, care group volunteer; EDP, extended distribution point; IYCF, infant and young child feeding; M&E, monitoring and evaluation; N/A, not applicable; PDM, postdistribution monitoring; SBCC, social and behavioral change communication; SCOPE, World Food Programme tracking and registration system; SQ-LNS, small-quantity lipid-based nutrient supplement; WASH, water, sanitation, and hygiene.

This number excludes the 12,769 children who had their first redemption in the first quarter (Q1) of year 1 of the program (January, February, March 2014) because many of the children enrolled during that period were part of a calendar- rather than age-based mass enrolment period.

An imbalance in observations of Nutributter distributions at the health center and EDP resulted from a misunderstanding of distribution dates, which yielded 1 more EDP-based observation than planned, and 1 less health center–based observation than planned. Nevertheless, the total number of observations of Nutributter distributions (n = 14) remained as planned.

Program design and implementation documents

Available program design and implementation documents, including the program concept note, logical framework, qualitative study reports, monitoring and evaluation reports, and peer-reviewed publications (5, 23, 24), were reviewed and used to learn about the program's implementation theory. Information-gathering meetings and informal discussions with key program personnel (i.e., WFP and World Vision staff) were used to understand and clarify program procedures needed for the development of the program's implementation theory.

Program monitoring data

Monthly program monitoring data from the WFP's tracking and registration system (SCOPE, the World Food Programme tracking and registration system) was used to measure the program's recruitment, fidelity, and reach (coverage). SCOPE collected individual-level attendance and redemption data from all program beneficiaries who had ever participated in the program from January 2014 to December 2016 in the program district (n = 44,025), which were used to measure program reach of the SQ-LNS component. SCOPE data were also used to assess program recruitment (age at first SQ-LNS redemption) for the first 3 y of program implementation (January 2014 to December 2016) (n = 31,256).

Quarterly postdistribution monitoring (PDM) surveys (n = 12) were used to measure program dose delivered and dose received under the umbrella of program fidelity. PDM surveys used a 2-stage stratified sampling strategy to identify a new, representative sample of pregnant and lactating women, and mothers of children aged 6–23 mo using district population and program registration lists (21). PDM survey data were based on interviews with selected beneficiary households using self-reported receipt of SQ-LNS, SBCC messaging, and SQ-LNS consumption by the target child; sampling strategy for PDMs is described in detail elsewhere (21). The WFP provided their data from 12 PDM surveys for this analysis, each with the following sample sizes: PDM1 (n = 368); PDM2 (n = 386); PDM3 (n = 390); PDM4 (n = 369); PDM5 (n = 358); PDM6 (n = 344); PDM7 (n = 340); PDM8 (n = 346); PDM9 (n = 340); PDM10 (n = 353); PDM11 (n = 379); and PDM12 (n = 347). The total, pooled sample size used as the denominator of process measures pulled from PDM surveys was n = 4320.

In addition to SCOPE and PDM, data specific to SBCC activities were obtained from the WFP annual performance review documents that presented a summary of staff trained and implementation of SBCC activities.

Direct observations

Structured direct observations of SQ-LNS distributions and one-on-one household counseling sessions by CGVs were used to capture elements of quality within the realm of program fidelity. Observations were conducted during December 5–17, 2016 by 2 trained research assistants who fluently spoke the local language, Chichewa, and who were familiar with the district, the nutrition program, and the impact evaluation.

Stratified purposive sampling was used to select direct observation sites to represent each of the district's 7 traditional authorities (TAs). The criterion was to have 1 health center and 1 EDP represented; these were randomly selected within each TA using a random number generator in Microsoft Excel, for a total of n = 14 observations of monthly distributions and n  = 14 observations of household counseling in an area near the selected distribution site. A randomly selected subsample of CGVs who participated in knowledge questionnaires were selected for observations of one-on-one counseling.

Structured guides were used to conduct both types of direct observations to capture dimensions of program fidelity, such as having an adequate stock of SQ-LNS at distributions and tailoring individual counseling sessions according to household characteristics (see Supplemental Materials 1 and 2). Standardization of direct observations for the 2 research assistants was obtained by conducting paired observations (research assistant and supervisor) prior to and during the first half of data collection, followed by debriefing as pairs and as a team to discuss what was observed and how certain features of the observation were interpreted and why.

Knowledge questionnaires with program staff

Knowledge questionnaires with frontline program staff (nutrition promoters and CGVs) measured program quality, according to completion of training modules and knowledge of IYCF, WASH, and SQ-LNS provision and use messages on which they were trained, as reported by respondents. Questionnaires were designed to test the knowledge of workers as presented in the 4 training modules and lessons used in the program, which included: 1) breastfeeding; 2) complementary feeding; 3) WASH; and 4) SQ-LNS use.

Sampling of promoters and CGVs was based on the sampling scheme used for the direct observations of SQ-LNS distributions described above. Using a district-wide roster of field-level program staff, promoters were identified at each health center and EDP at which an observation of SQ-LNS distribution was conducted. When >1 promoter was present, 1 promoter was randomly selected to participate in the knowledge questionnaire (total promoters n = 14). Subsequently, 1 of every 2 CGVs who fell under the selected promoter's catchment area was randomly selected from the staff roster and asked to participate in knowledge questionnaires (CGVs per promoter = 5; total CGVs n = 70). One of every 5 CGVs per SQ-LNS distribution area and who participated in knowledge assessment interviews was randomly selected for observations of one-on-one counseling sessions. Questionnaires were administered during the same data collection period as the direct observations, during December 5–17, 2016.

Measurement and analysis of program implementation variables

The program's conformity to design was assessed using process measures of recruitment, fidelity, and reach, largely based on Linnan and Steckler's (2002) definition (13), with slight modifications, as presented in  and described below. Each of the indicators used to measure implementation and delivery was selected based on the program's implementation theory (Figure 1).

TABLE 2

Key process evaluation components included in this study

Process evaluation component	Definition
Recruitment	Procedures used to approach and attract participants. Recruitment often occurs at the individual and organizational/community levels
Fidelity	The extent to which the intervention was delivered as planned. It includes 3 critical elements of quality, dose delivered, and dose received
Quality	The quality and integrity of the intervention as conceived by the developers
Dose delivered	The number of intended units of each intervention or each component that are delivered
Dose received	The extent to which the target audience (of the dose delivered indicator) actively engages with, interacts with, is receptive to, and/or uses materials or recommended resources
Reach	The proportion of intended target audience that participates in an intervention, often measured by attendance. Reach is a characteristic of the target audience

Source: Adapted from reference 13 with permission.

Program recruitment

Program recruitment refers to the procedures used to attract, recruit, and retain participants. Program recruitment was collected by SCOPE and measured as the age at which eligible beneficiaries had their first redemption, and disaggregated by the following age groups: <8 mo (n = 15,202), 8–11.9 mo (n = 10,053), 12–17.9 mo (n = 4201), and 18+ mo (n = 1800). Data on child's age at first redemption came from SCOPE and were analyzed for the program's first 4 y of implementation (n = 31,256); children who first redeemed in the first quarter of year 1 (January through March 2014) were excluded due to mass enrolment activities that were calendar- and not age-based (n = 12,769).

Program fidelity

Program fidelity encompasses the extent to which the program was delivered as it was intended, including elements of quality and integrity of delivery, as well as dose delivered and received by the target population(s) for each activity. The measurement of program fidelity is critical because it can act as a moderator in the relation between the interventions being delivered and their intended outcomes (25). The focus of this process evaluation on the quality and dose elements of fidelity was based on 2 factors: 1) available data, and 2) later addition of a program component (program adaptation). The quality component of program fidelity was measured by observations of SQ-LNS distributions and one-on-one counseling sessions (n = 14 each, for total n = 28), from which the number of times specific activities were done correctly was used to calculate proportions used to represent program functioning. Activities included: functioning of the card scanning technology; adequate stock of SQ-LNS at the distribution site; correct distribution of the SQ-LNS to beneficiaries (e.g., provision of 35 sachets); treatment of mothers; communication of IYCF, WASH, and SQ-LNS messages; and receipt of group counseling during distributions. Duration of the distribution session was recorded for all distributions and presented as mean ± SD.

Knowledge assessments measured staff (promoters and CGVs) knowledge of IYCF, WASH, and SQ-LNS messages based on self-reported modules they were trained on. The proportion of CGVs (n = 70) and promoters (n = 14) who provided a correct answer to each question was calculated based on total number of staff who answered that question. Mean knowledge scores were calculated for each module among program staff who had been fully trained on that module (i.e., answered all questions in the module): 1 point was assigned to each correct answer and summarized for each module to create a knowledge score, for CGVs and promoters. Highest possible scores for the modules were 27 for breastfeeding (completed by n = 16 CGVs and n = 7 promoters), 34 for complementary feeding (completed by n = 13 CGVs and n = 5 promoters), 5 for WASH (completed by n = 70 CGVs and n = 14 promoters), and 9 for SQ-LNS (completed by n = 65 CGVs and n = 14 promoters).

Process indicators of dose delivered and dose received were grouped under program fidelity and mirrored each other, in that dose delivered referred to units of each intervention that were delivered, and dose received represented how and how many of those units were received by the target population for that indicator, which in this case could be program staff and/or mothers of eligible children. Dose delivered was measured by the number of health staff trained in program delivery, training of SBCC cadres/volunteers, billboards erected, and promotional IYCF campaigns conducted at the health facility level, against the targets set by the program to generate proportions. Dose received was measured as the proportion of volunteer program staff (promoters n = 14 and CGVs n = 70) who participated in knowledge assessments and had completed training on the SBCC modules, as well as the proportion of target beneficiaries who attended a community event, heard a radio message, recognized ≥3 nutrition messages, and consumed SQ-LNS as directed, as collected by PDM surveys.

Program reach

Often referred to as coverage, program reach provides the proportion of the target audience that participated in each of the intervention's components. We measured program reach in the program's SQ-LNS component as the proportion of target beneficiaries who attended monthly distribution sessions from January 2014 to December 2016, provided by WFP's SCOPE system. Reach of the SBCC component was obtained from PDM data, measured as the proportion of target beneficiaries who received a CGV home visit and group counseling. Remaining program recruitment indicators were informed by the program's 12 PDM surveys conducted over 3 y of program implementation. PDM data were pooled (n = 4320) and mean proportions were calculated for each indicator presented.

Ethical approval

This process evaluation was approved by institutional review boards at the Johns Hopkins Bloomberg School of Public Health, and at the College of Medicine Research and Ethics Committee in Malawi. Informed consent was given by all staff members who participated in knowledge questionnaires and households who participated in direct observations.

Results

Program recruitment

In the first 4 y of program implementation (2014–2017), the proportions of children who had redeemed their first ration of SQ-LNS by 8 mo of age were 53.1%, 47.0%, 46.2%, and 51.8%, respectively (). In those same 4 y, a mean of 81.3% of children had redeemed their first ration of SQ-LNS by 12 mo of age.

FIGURE 2

Proportion of children who redeemed program benefits (SQ-LNS) in the first 4 y of program implementation, by age at first redemption. SQ-LNS, small-quantity lipid-based nutrient supplement.

Program fidelity

Based on observations of SQ-LNS distributions, the quality component of program fidelity was deemed to be high (). Although the card scanning technology worked most of the time (85.7%), in some cases the system failed during the distribution (n = 1) or was not used at all even if it was functioning (n = 1) (data not shown). In all observations, ID card numbers were written down by HSAs and/or promoters, which was a major source of delay, but described as necessary by program staff during the conduct of observations, due to previous experiences with loss of electronic data. The waiting time at the distribution point as measured by maternal report in PDM surveys was a median of 1.0 h, whereas the mean (± SD) duration of the distribution as observed was 2.0 ± 0.8 h. In all observations, the stock of SQ-LNS was adequate to serve all attending mothers, as was the number of sachets distributed. Staff managing distribution sessions were courteous and respectful toward beneficiaries attending distributions, and mothers were engaged during messaging sessions, and compliant and patient during distribution of the SQ-LNS. Multiple rounds of counseling to accommodate late-coming mothers were conducted only in half (n = 7) of the distributions observed. This resulted in only 28.6% of distributions having all mothers in attendance receiving group counseling. Finally, all distributions included messages on SQ-LNS use, whereas 57.1% and 28.6% of distributions included IYCF and WASH messaging, respectively.

TABLE 3

Program fidelity: quality of implementation of SQ-LNS distributions (n = 14) and one-on-one counseling sessions (n = 14)

Program activities	Value1
SQ-LNS distribution
Card scanning functioning and used at distributions	12 (85.7)
Duration of distribution session, mean ± SD, h	2.0 ± 0.8
Adequate stock of SQ-LNS at distribution	14 (100.0)
Correct number of sachets distributed	14 (100.0)
Courteous treatment of mothers
Most of the time	2 (14.3)
All of the time	12 (85.7)
SQ-LNS use messages communicated	14 (100.0)
IYCF messages communicated	8 (57.1)
Water, sanitation, and hygiene messages communicated	4 (28.6)
Receipt of group counseling at distribution
Some of the mothers	2 (14.3)
Most of the mothers	8 (57.1)
All of the mothers	4 (28.6)
One-on-one counseling
SQ-LNS usage messages communicated	2 (14.3)
IYCF messages communicated	9 (64.3)
Water, sanitation, and hygiene messages communicated	9 (64.3)

Values are n (%) unless otherwise indicated. IYCF, infant and young child feeding; SQ-LNS, small-quantity lipid-based nutrient supplement.

Demographic, employment, and workload characteristics of promoters and CGVs who participated in knowledge questionnaires are provided in Supplemental Table 1. Knowledge questionnaires with CGVs highlighted that knowledge of key breastfeeding, WASH, and SQ-LNS messages was ≥90%, except for how to overcome breastfeeding challenges, which only 47.8% could answer correctly (). Complementary feeding knowledge was lower, both among CGVs and promoters, especially regarding messages around the 6 food groups, which only 22.9% of CGVs could list correctly. Trends in knowledge among promoters mirrored those seen among CGVs, with low knowledge of messages about overcoming breastfeeding problems (69.2%) and being able to name the 6 food groups (35.7%). A comparison of mean knowledge scores among CGVs and promoters who had completed each of the 4 modules further demonstrated these similarities. Mean ± SD (range) knowledge scores among CGVs and promoters were, respectively: 23.3 ± 2.0 (19–26) and 24.0 ± 1.3 (22–26) for breastfeeding; 24.4 ± 3.6 (19–28) and 26.0 ± 3.5 (21–29) for complementary feeding; 4.9 ± 0.3 (4–5) and 4.6 ± 0.8 (2–5) for WASH; and 8.7 ± 0.6 (7–9) and 8.6 ± 0.5 (8–9) for SQ-LNS ().

TABLE 4

Knowledge of promoters and CGVs on 3 select key messages per training module (breastfeeding; complementary feeding; and water, sanitation, and hygiene, SQ-LNS), based on knowledge questionnaires with program staff

	CGVs (n = 70)	Promoters (n = 14)
Module 1: Breastfeeding
1. Begin breastfeeding within 1 h after birth	63 (95.5)	14 (100.0)
2. One instruction on how to overcome breastfeeding challenges	22 (47.8)	9 (69.2)
3. Not to give child water or other liquid or food before 6 mo, even in hot weather	62 (91.2)	14 (100.0)
Module 2: Complementary feeding
4. Foods other than breastmilk (complementary feeding) should start at 6 mo of age	60 (85.7)	10 (71.4)
5. Can list the 6 food groups	16 (22.9)	5 (35.7)
6. Three foods that can be added to porridge to improve its nutritional value	58 (85.3)	12 (92.3)
Module 3: Water, sanitation, and hygiene
7. Three critical time points for washing hands	69 (98.6)	12 (85.7)
8. Use of water and soap to wash hands	69 (98.6)	13 (92.9)
9. Handwashing is important for preventing disease	70 (100.0)	14 (100.0)
Module 4: SQ-LNS
10. SQ-LNS is for children aged 6–23 mo	64 (98.5)	14 (100.0)
11. Children should eat 1 packet of SQ-LNS/d	61 (93.9)	14 (100.0)
12. SQ-LNS should not be shared	61 (93.9)	12 (85.7)

Values are n (%). Proportions presented are based on the number of staff who stated having completed the lesson to which the question corresponds. Messages 1, 2, and 3 were completed by n = 66, n = 46, and n = 68 CGVs, respectively. Messages 4, 5, and 6 were completed by n = 70, n = 69, and n  = 68 CGVs, respectively. Messages 7, 8, and 9 were completed by n = 70 CGVs. Messages 10, 11, and 12 were completed by n = 65 CGVs. All messages except message 6 (n = 12) were completed by all promoters. CGV, care group volunteer; SQ-LNS, small-quantity lipid-based nutrient supplement.

FIGURE 3

Program fidelity: mean ± SD and maximum knowledge scores of care group volunteers and nutrition promoters who completed breastfeeding, complementary feeding, water, sanitation, and hygiene, and SQ-LNS training modules under the social and behavior change communication activities of the Malawi community-based nutrition program. Dark grey bars indicate maximum possible scores. BF, breastfeeding; CF, complementary feeding; SQ-LNS, small-quantity lipid-based nutrient supplement.

Program quality indicators showed that many of the training and mobilization activities, such as training of health staff (90.0%) and of SBCC cadres (>100%), implementation of open nutrition days (95.2%), and installment of promotional billboards (100.0%) were conducted as planned and in some cases exceeded targets originally set by the program (). Despite taking these steps needed to implement SBCC training activities, completion of training by frontline promoters and CGVs remained low at the time of data collection (Table 5). Only 21.4% of promoters and 5.7% of CGVs had completed all 4 training modules. For both CGVs and promoters, completion of the complementary feeding module was the lowest (18.6% and 35.7%, respectively), followed by completion of the breastfeeding module by 50.0% and 22.9%, respectively. Conversely, 100% of both CGVs and promoters had completed the WASH module, whereas 92.9% and 100.0% of them had completed the SQ-LNS module, respectively. Self-reported receipt of community-level SBCC by mothers, such as attendance at community events and exposure to radio messages, was 20.3% and 47.5%, respectively. The ability of mothers to recall ≥3 nutrition messages was 99.3%, likely due to receipt of messages from SQ-LNS distributions (group counseling), which 93.3% of mothers reported as a source of information (Tables 5 and ). Similarly, adequate SQ-LNS usage practices, such as consumption of 1 packet by the child in the last 24 h (76.2%) and relatively low self-reported product misuse (sharing of 10.9%), further highlight that program messages were being received by mothers.

TABLE 5

Program fidelity: dose delivered, program quality, and dose received for the Malawi community-based nutrition program

Dose delivered	Program target, n		Achieved, n (%)
Health staff trained on project delivery	638		574 (90.0)
Training of SBCC/volunteer cadres	1105		1112 (100.6)
Open nutrition days conducted	21		20 (95.2)
Billboards erected	21		21 (100.0)
Promotional campaigns on IYCF at health facilities	60		44 (73.3)
Program quality: training of nutrition promoters (n = 14)		n (%)
Training completed by promoter		3 (21.4)
Breastfeeding module		7 (50.0)
Complementary feeding module		5 (35.7)
Water, sanitation, and hygiene module		14 (100.0)
SQ-LNS module		14 (100.0)
Program quality: training of care group volunteers (CGVs) (n = 70)		n (%)
Training completed by CGVs		6 (8.6)
Breastfeeding module		16 (22.9)
Complementary feeding module		13 (18.6)
Water, sanitation, and hygiene module		70 (100.0)
SQ-LNS module		65 (92.9)
Dose received: target beneficiaries (n = 4320)		n (%)
Target beneficiaries who attended a community event		877 (20.3)
Target beneficiaries who heard a radio message		2052 (47.5)
Mothers who recognized ≥3 nutrition messages		4290 (99.3)
Children aged 6–23 mo who consumed 1 packet of SQ-LNS in the last 24 h		3292 (76.2)
SQ-LNS shared		471 (10.9)
Households that experienced a gap in SQ-LNS supply		1642 (38.0)
Gap days, mean ± SD		10.3 ± 2.0
Households visited per month by CGVs, mean ± SD (n = 70)		10.5 ± 2.7

IYCF, infant and young child feeding; SBCC, social and behavior change communication; SQ-LNS, small-quantity lipid-based nutrient supplement.

TABLE 6

Program reach (coverage) of SQ-LNS distributions and SBCC activities in the Malawi community-based nutrition program

Process indicator	N	n (%)
Target beneficiaries attending monthly SQ-LNS distributions	44,617	33,710 (81.0)
Exposure to SBCC activities	4320
Target beneficiaries who received CGV home visit		1594 (36.9)
Target beneficiaries who received group counseling		4031 (93.3)

CGV, care group volunteer; SBCC, social and behavior change communication; SQ-LNS, small-quantity lipid-based nutrient supplement.

Program reach

Over 3 y of program implementation, 81.0% of beneficiaries had attended ≥1 monthly distribution session (). Participation in the monthly SQ-LNS was high throughout the program period assessed (January 2014 to December 2016), with monthly participation of eligible children well above 70% most months (Supplemental Figure 3). The proportion of beneficiaries who reported having been exposed to counseling by home visits from CGVs and group counseling was 36.9% and 93.3%, respectively.

Discussion

In this process evaluation, we measured and described the implementation of a district-wide community-based nutrition program that aimed to reduce the prevalence of stunting among children aged 6 to 23 mo. A variety of data sources and research activities were used to assess 2 of the program component's conformity to original design. Through the various measurements presented, this evaluation highlighted that the program component that dealt with monthly SQ-LNS distributions functioned well. Nevertheless, enrollment of program beneficiaries at or by age 8 mo and variations in counseling strategies during SQ-LNS distributions to reach all participating mothers could benefit from improvement. SBCC activities were more variable in their implementation. Though many of the community mobilization and training activities were implemented by the program as planned, completion of training modules in a small sample of frontline staff was low and might have affected downstream processes, such as exposure to individual counseling and community events by target beneficiaries.

Our program fidelity data captured aspects of quality, integrity, and dose of implementation, which was found to be highly adequate for the SQ-LNS distribution component. Distribution sites were plentiful and conveniently located, and distributions were run efficiently. The implementation of the SBCC components was observed to be both more challenging to implement, as well as its success more difficult to measure. The program fully established a network of approximately 500 CGVs and achieved a complete rollout of SBCC activities as reflected in the saturation and geographic distribution of promoters and CGVs in a manner sufficient to cover all households in the district. After 2 y of work to establish this system, however, challenges to maintain it were evidenced by the low (36.9%) proportion of mothers who reported home counseling by a CGV as a source of SBCC messaging, and the variable completion of training modules by promoters and CGVs and delivery of messages by CGVs. The lack of completed training by frontline staff, however, was not seen as a reflection of trainings not being conducted as planned. On the contrary, this study found that all care groups were formed according to plan, with corresponding training activities of ≥90% for health staff and SBCC cadres. The low proportion of promoters and CGVs having completed training, as well as the limited implementation of home visits, could thus rather reflect the realities associated with depending on a network of volunteers, who have competing priorities or other responsibilities that prohibit them from conducting or attending training sessions according to schedule, yielding an uneven completion and delivery of lessons and modules. Similarly, the elapsed time between the completion of the module and the knowledge assessment could have contributed to gaps in knowledge observed in this study. Nevertheless, and likely balanced by the high attendance recorded at distribution sessions, which also included a counseling component, nearly all mothers were able to recall ≥3 nutrition messages. This could be verified by the >90% of mothers who reported having received messages from group counseling. The use of a comprehensive and multisource messaging scheme with other community-level SBCC activities and mobilization has been demonstrated in other contexts to yield maximum benefits (26) and could have also contributed to a high exposure and recall of nutrition and sanitation messages.

The observed fidelity to program design for both program components and high levels of exposure to nutrition messages by target beneficiaries likely contributed to the positive impacts of the program on children's nutritional status, dietary diversity, and morbidity (Christian P, Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, unpublished results, 2017; Hurley K, Phuka J, Kang Y, Ruel-Bergeron J, Buckland A, Chirwa E, Mitra M, Wu L, Klemm R, West Jr K, Christian P, unpublished results, 2016 ). The lack of impact of the program on stunting might have been related to several factors beyond program implementation, such as evaluation design and program interventions that did not directly address causal determinants of linear growth retardation and stunting (27). This process evaluation also demonstrated that throughout the program's duration, half of eligible children had first received SQ-LNS at or by 8 mo of age, affecting their total duration of exposure to SQ-LNS, which is important for improving linear growth (8). Under the original design, children were intended to receive SQ-LNS for a period of 18 mo, but delays in enrolment, in addition to unanticipated events that could have caused a child to miss ≥1 mo of supplementation throughout the period of eligibility, could minimize both the total months of supplementation and, subsequently, the potential impact of SQ-LNS on linear growth.

This study has several strengths. The use of various methods and sources employed in this study promoted the triangulation of information to more accurately understand program operations and procedures, as well as product usage by eligible beneficiaries. The use of direct observations was also considered a strength because it allowed us to see, experience, and assess program implementation rather than to depend on reported implementation (28). Lastly, the use of a systematic approach to study implementation also contributed to the robustness of the findings; the detailed documentation of the various study elements included promotes the replication or adaptation of this method for measuring program implementation in nutrition in other similar settings and programs.

There are, however, a few limitations of this study. This process evaluation was dependent on a variety of sources, including data that relied on self-report of practices and/or observation, which are prone to social desirability bias (29) and reactivity to the observer (30), respectively. Social desirability bias is the systematic error that occurs in self-reported measures resulting from the “desire of respondents to project a favorable image to others” (29). Data specific to fulfillment of CGVs’ work responsibilities, and home usage of SQ-LNS, for example, were both based on self-report by the CGVs and mothers, respectively, and could therefore have been systematically less reliable or reflective of actual practices for fear of being caught doing the wrong thing or not following instructions. One example of this lies in the discrepancy observed across various sources of reported sharing of SQ-LNS by mothers of beneficiary children.

Another limitation of this study is the timing of data collection on program functioning (direct observations) and quality (knowledge questionnaires), which was conducted only once and late into the program's implementation, and was therefore unlikely to inform program improvements. Repeated measurements that started earlier in the program's implementation would have been valuable for course correction as well as for providing a basis for comparison over time. For example, process evaluations in Bangladesh, Ethiopia, and Vietnam under the Alive & Thrive Initiative collected data on program implementation early on and throughout the life of the program to identify and correct bottlenecks and challenges in a timely manner (16, 17). Nevertheless, the use of various sources of existing data and the availability of well-established and continuous program monitoring data collected at different times since the program's initiation, promoted the efficiency with which this process evaluation was conducted, and allowed for data-driven triangulation of findings and conclusions.

Lastly, the failure of this process evaluation to measure the implementation of all 4 components of the program prevented us from distinguishing the pathways through which an impact on certain outcomes was achieved. For example, given the lack of assessments of the component on strengthening facility-based management of acute malnutrition (component 3), it was impossible to disentangle whether improvements in child weight, weight-for-age z-scores, and MUAC resulted from this or the SQ-LNS component. Similarly, quantitative attributions to improvements in children's dietary diversity to SBCC activities without measurement of the contributions of the nutrition-sensitive interventions (component 4) that were added to the program at a later stage of implementation, could not be made. The addition of program components represents the tension between balancing and incorporating program adaptation into program fidelity measurement (31).

In conclusion, to our knowledge, this study is among the few process evaluations of a large-scale nutrition program delivering a combined package of a lipid-based nutrient supplement and nutrition to all children aged 6–23 mo with a SBCC campaign on optimal IYCF and WASH practices in a district of rural Malawi. The measurement of this program's processes and interventions, and how they can be modified to achieve their objectives more efficiently and effectively (12), adds to the growing body of evidence on how a food supplement can be delivered to large numbers of geographically dispersed rural beneficiary populations frequently and effectively in combination with group and individual social behavioral change initiatives, and the challenges that arise in doing so. This study also demonstrates that, for this community-based nutrition program in Malawi, most aspects of the program were implemented at very high quality, with some identified as requiring improvement. The program was effective in reaching a large proportion of the target population with many of its planned activities, which likely contributed to the positive impact on child growth and dietary diversity in the program district. Strengthening the timely enrollment of participants and improving household-level counseling by CGVs could have potentially promoted an even greater impact on children's linear growth beyond what was achieved.

Click here for additional data file.