An adequacy evaluation of a 10-year, four-country nutrition and health programme.

BACKGROUND: Evaluations of large-scale health and nutrition programmes in developing countries are needed for determining the effectiveness of interventions. This article critically analyses a non-governmental organization (NGO)-led large-scale, multi-country, 10-year micronutrient and health (MICAH) programme with an 'adequacy evaluation', that is, a documentation of time trends in the expected direction.
METHODS: MICAH was implemented from 1996 to 2005 in selected areas of Ethiopia, Ghana, Malawi and Tanzania, reaching >6 million people with numerous health and nutrition interventions. Coverage and impact were monitored through surveys at baseline, midpoint and end of funding. The data were subjected to post-hoc methods of quality determination, and, if of suitable quality, included in the adequacy evaluation.
RESULTS: Most collected data were of moderate or high quality and therefore included in the adequacy evaluation. There were moderate to large improvements in vitamin A status in Ethiopian school-age children, children <5 years of age in Tanzania and Ghana and mothers in Ghana. Iodine status improved in Malawi and Tanzania. Anaemia rates and malaria prevalence decreased in women, pregnant women and pre-school children in Ghana, Malawi and Tanzania, but anaemia increased in Ethiopian women. Large increases were reported for rates of exclusive breastfeeding and immunization. Child growth improved to the maximum that would be predicted with the given interventions.
CONCLUSIONS: Numerous nutrition and health impacts were observed in the intervention areas, often of a magnitude equal to or larger than observed in controlled interventions or trials. These results show the value of integrated long-term interventions.

Introduction

A recent series of publications in the Lancet has described the scale of malnutrition in developing countries, the possible effects of successful nutrition interventions on malnutrition and some of the financial, bureaucratic and institutional hurdles preventing implementation of these interventions. One of the key hurdles is the limited number of evaluations of the effectiveness of large-scale multiple-intervention nutrition programmes. The lack of effectiveness evaluations discourages investment in nutrition programmes and precludes well-informed decision making.

The efficacy of some single nutrition interventions has been frequently and thoroughly evaluated (e.g. iron supplementation, vitamin A supplementation, salt iodization). However, the effectiveness (i.e. the efficacy in ‘real world’ settings) of large-scale integrated health and nutrition programmes has not been thoroughly evaluated., Victora et al. and Habicht et al. argue that evaluations of large-scale programmes are necessary for determining the effectiveness of interventions, and thus for deciding upon resource allocation and public health programming: ‘In summary, there are important restrictions to the external validity of RCTs [Randomized Controlled Trials] for complex public health interventions. The likelihood of effect modification implies that one cannot take for granted that interventions that are proven efficacious in controlled trials can be generalized to other settings’ (p. 403), although such assumptions are in fact made. Thus, there is a need for high-quality evaluation of large-scale programmes. However, such programmes are not usually led by capable researchers with the budget and mandate to evaluate programme effectiveness, but rather by programme implementers without the budget, mandate or skills to conduct the evaluation. As a result there is a paucity of literature documenting the nutrition outcomes of large-scale integrated nutrition and health programmes.

The objective of this article is to provide evidence of the effectiveness of a large-scale nutrition and health programme in Africa. For this, we evaluated a non-government organization (NGO)-led large-scale, multi-country, 10-year comprehensive micronutrient and health (MICAH) programme, assessed with an ‘adequacy evaluation’. Victora et al. define an adequacy evaluation as, ‘documentation of time trends in the expected direction, following introduction of an intervention’ (p. 404). Adequacy evaluations have less rigorous designs than ‘plausibility evaluations’ and ‘probability evaluations’, which provide the evaluator with a more confident assessment of the programme’s impact. Included within adequacy evaluation are ‘performance evaluations’, which encompass evaluations of provision, utilization and coverage, and ‘impact evaluations’, which evaluate health outcomes and behaviours. Most of the evaluations within this article are ‘adequacy impact evaluations’, although immunization coverage should be considered an ‘adequacy performance evaluation’.

Prior to the evaluation, it was necessary to determine the quality of the evaluation data. Two specific questions were considered: (i) are programme evaluation data, collected by trained surveyors but with less rigour than would be expected in a randomized control trial, of sufficient quality to monitor programme impacts? And if so, (ii) were the interventions effective in improving the nutrition and health of beneficiaries?

The MICAH programme was launched in 1995 by World Vision Canada. The goal of the programme was to improve the nutrition and health status of women and children through the most cost-effective and sustainable interventions, in line with international targets set at the 1990 World Summit for Children for the virtual elimination of vitamin A and iodine deficiencies, and the reduction of iron deficiency anaemia in women by one-third.

MICAH was implemented from 1996 to 2005 in five African countries, four of which are reported here (Ethiopia, Ghana, Malawi and Tanzania), reaching 4 million direct beneficiaries, and more than 6 million indirect beneficiaries., The fifth country, Senegal, was of different structure and duration, consisting of two 4-year interventions conducted in different areas, rather than one longer intervention, and is therefore excluded from this evaluation.

In this article, we evaluate the impact of MICAH through a detailed assessment of programme evaluation data, and we consider the implications for future large-scale nutrition and health interventions.

Description of the MICAH programme interventions

Considering baseline assessment of the vitamin A, iron and iodine deficiencies epidemiology, and following national policies, context-specific programme plans were developed for each MICAH country within a programme-wide framework of objectives and strategies. Multiple interventions, ranging from community-based supplement distribution to fortifying and diversifying foods, and to national-level advocacy for national policy change, were conducted to address the deficiencies and target groups identified. Interventions were integrated into existing systems, structures and services, wherever possible, to increase potential for sustainability. The interventions not only had a shared purpose, but also had many gross as well as nuanced differences based on country context. Table 1 summarizes the MICAH activities: all countries used multi-pronged integrated approaches to address various aspects of nutrition and health. Not all countries had the same target groups and activities, but all had multiple targets and activities. Programme areas within countries were poor rural communities, where few or no other major development organizations were operational. In Malawi and Ethiopia, proposals were invited from other NGOs, government organizations and local World Vision units (‘Area Development Programmes’). In both countries, programme areas were selected based on quality of the proposals, evidence of community need and applicants’ organizational capacity. In Ghana, the Kwahu South district was selected as it was known to have a high goitre rate, and World Vision was working in the area on a water and sanitation project. In Tanzania, the eastern zone was selected by World Vision’s National Office, with the agreement of the Ministry of Health, as an area in need of micronutrient interventions. Further details of each country programme are available in the Supplementary data available at IJE online (see Appendix 1).

Table 1

Summary table of MICAH interventions

			Ethiopia	Ghana	Malawi	Tanzania
	Approximate number of ‘direct’ beneficiariesb		1.8 million	150 000	1.8 million	255 420
	Total implementation cost, US$ million		16.6	3.3	14.5	3.2
	Total cost of evaluation, US$		559 345	147 866	1 293 018	172 243
	Total cost, US$ millionc		17.0	3.5	15.5	3.3
	US$ per direct beneficiary per year (Phase I/Phase II)		1.39/0.83	3.37/2.14	1.06/5.06	1.85/1.28
Objective	Strategy	Target group/activity	Ethiopia	Ghana	Malawi	Tanzania
Increase intake and bioavailability of micronutrients (iron, iodine and vitamin A)	Vitamin A supplementation	Pre-school children	D, T, M	P, D, T, A, M		D, T, M
		School-age children	D, T, A, M
		Post-partum women	D, T, M	P, D, T, A, M		D, T, M
	Iron supplementation	Pre-school children		P, D, T, A, M	P, D, T, A, M	P, D, T, M
		School-age children		P, D, T, A, M
		Women of childbearing age		P, D, T, A, M	P, D, T, A, M
		Pregnant women	P, T, M	P, D, T, A, M	P, D, T, A, M	P, D, T, M
	Fortification	Iodized salt promotion	P, D, T, A, M	P, D, T, A, M	T, A, M	T, A, M
		Small-scale flour fortification			P, D, T, A, M	P, D, T, A, M
	Dietary diversification	Small animal rearing	P, D, T, A, M	P, D, T, M	P, D, T, A, M	P, D, T, M
		Vegetable gardens	P,D,T,A,M	P, D, T, M	P, D, T, A, M	P, D, T, M
		Fruit tree cultivation	P, D, T, A, M	P, D, T, M	P, D, T, A, M	P, D, T, M
	Infant and young child feeding	Promotion of optimal breastfeeding and complementary feeding	T, A, M	T, A, M	T, A, M	T, A, M
Reduce prevalence of diseases that affect micronutrient status (diarrhoeal, parasitic and vaccine-preventable)	Water and sanitation	Provision of clean water	P, D, T, M	P, D, T, M	P, D, T, M	P, D, T, M
		Latrine construction	P, D, T, M	P, D, T, M	P, D, T, M	P, D, T, M
		Garbage disposal construction	T, M	T, M	T, M	T, M
	Malaria control	ITN distribution	P, D, T, M	P, D, T, M	P, D, T, M	P, D, T, M
		Chemoprophylaxis to pregnant women	P, D, M	M	P, D, T, M	P, D, T, M
		Malaria treatment to pre-school children	P, D, M	T	P, D, T, M	P, D, T, M
	Treatment of worms and parasites	Deworming of pre-school children	P, D, T, M	P, D, T, M	P, D, T, A, M	P, D, T, M
		Deworming of school-age children	P, D, T, M	P, D, T, M	P, D, T, A, M	P, D, T, M
		Schistosomiasis treatment	P, D, T, M	P, D, T, M	P, D, T, A, M	P, D, T, M
		Immunization	Support EPI campaigns	P, D, T, M	P, D, T, M	P, D, T, M	D, T, M
		Diarrhoea treatment	Promotion of ORT	T	T	P, D, T	T
		HIV/AIDS prevention	IEC re: HIV and AIDS	T	T	T	T
Build local capacity for delivery systems to improve micronutrient status	Education	Health and nutrition IEC to communities	T, M	D, T, M	D, T, M	D, T, M
		Staff and partner training	T	T	T	T
	Advocacy	Influence national policies on nutrition issues	A	A	A	A
		Use of media to communicate nutrition and health messages	T, A	T, A	T, A	T, A

P: procurement; D: delivery/distribution (includes both transport to distribution sites such as health centres and direct delivery to beneficiaries); T: training (includes both community education and training of implementing staff/volunteers); A: advocacy; M: monitoring; ITN: insecticide-treated bednet; EPI: expanded programme on immunization; ORT: oral rehydration therapy; IEC: Information, Education and Communication.

aBlank cells indicate that MICAH did not work on that target group/activity in that country.

bBased on target population in MICAH programme communities; not including the significant number of indirect beneficiaries (e.g. 4.7 million in Malawi) of MICAH’s national advocacy and intervention efforts (e.g. iodized salt coverage, EPI and vitamin A supplementation (VAS) campaigns).

cCosts are in US$ and based on exchange rates applicable at the time of purchase. World Vision Canada technical support and programme management costs, as well as overhead costs at country and Canada levels, are included.

A comparison of MICAH data with the Demographic and Health Survey (DHS) data for national rural samples (see Table 5) suggests that the selected areas were similar to or worse off than the rural average in each country at baseline. Ghana’s MICAH baseline in 1997 was worse than the 1998 DHS rural national numbers for immunization coverage, access to protected water source (but not access to latrines) and exclusive breast-feeding rates, with mixed results for child growth status (better growth in height, worse growth in weight). Tanzania’s baseline in 1997 was worse than the 1996 DHS rural national numbers for access to protected water source and latrines, and for child growth status. In Ethiopia and Malawi the DHS data were available for the follow-up year (2000) but not for the baseline year. If the DHS data are extrapolated back to the baseline year, assuming a linear trend, then 1997 MICAH indicators in Ethiopia were worse than DHS for vitamin A capsule (VAC) coverage in children <5 years of age, and better for measles coverage and latrine access. In Malawi, 1996 (extrapolated) DHS indicators were better than MICAH for all available indicators, except child growth, for which DHS and MICAH were similar.

Table 5

Programme coverage rates and outcome indicator prevalences at baseline (1996 or 1997), follow-up (2000) and endline (2004), and comparable available DHS data

When coverage rates and outcome indicators are discordant (changes not in expected directions) cells are shaded grey.

When the effectiveness is similar to or higher than previously observed efficacy, the text is shown in bold.

All DHS data were from http://www.statcompiler.com/ accessed on 9 October 2009.

Methods

Sampling and data collection

Details of the methods for sampling, data collection and data management are provided in the project documents. In brief, cross-sectional surveys were conducted in each programme area, at baseline (1996/97), the end of phase I (2000) and the end of phase II (2004). The surveys were conducted in the same month of the year in each country. Two-stage cluster sampling was employed, in which clusters were randomly selected using probability proportional to population size. Key indicators were assessed through structured interviews with a standardized questionnaire, and collection of biochemical, clinical and anthropometric data.

The sample sizes per country were between 900 and 4801 randomly selected households per survey. Sub-samples for clinical and biochemical indicators were randomly selected from the parent sample. When there were multiple individuals of a given age group in a household, one was chosen using standard criteria, except for the immunization and anthropometric indicators in which case all children <5 years of age were included. Data were collected by trained enumerators and clinical staff (varying by survey, but including laboratory technicians, clinicians, nurses, anthropometrists). More than two-thirds of immunization data were from child health cards, otherwise as reported by the child’s parent. Child growth was measured using calibrated Salter scales and locally made length boards. Questionnaires were filled out and anthropometrics taken at the subject’s house. In some cases, the biological samples were collected from the selected children when they were assembled at local clinics or schools rather than at their homes.

Haemoglobin was measured using a portable haemoglobin photometer: HemoCue B-Haemoglobin photometer (HemoCue AB, Angelholm, Sweden) in Ethiopia (adjusted for altitude), Ghana and Malawi; and Hemo-Control (EKF-diagnostic GmBH, Barleben, Germany) in Tanzania. Urine was collected from school-age children for measuring urinary iodine. Samples were transported to the laboratory, where they were frozen until analysis with the Sandall–Kolthoff reaction following digestion with ammonium persulfate. For breast-milk retinol, breast milk was expressed from both breasts and a 20-ml sample was collected in a foil-wrapped container, transported with ice packs and then stored at –25°C until laboratory analysis by high-performance liquid chromatography (HPLC). Goitre was assessed by palpation by trained team members or clinical officers, and Bitot’s spots were observed by trained nurses, physicians or technicians. For malaria parasitemia, capillary blood samples were smeared and stained according to standard methods and read by trained technicians, who assigned a malaria parasites score of ‘present’ or ‘absent’.

Children’s heights and weights were converted to standardized height-for-age Z-score (HAZ) and weight-for-age Z-score [WAZ; the number of standard deviations (SDs) the child’s height or weight is from the reference population mean for children of the same age], using EpiInfo 6.04 (Centers for Disease Control and Prevention, Atlanta, Georgia).

Data were entered by data clerks using standard software (e.g. EpiInfo, MS Access, SPSS or CSPro); 10% of the data were double entered and checked for accuracy. For key indicators with >10% data entry error, data entry was manually checked. Initial analyses were performed by statisticians using EpiInfo or other statistical software. Additional cleaning and analyses were done by World Vision Canada using MS Access (various versions, Microsoft Corporation) and SPSS (versions 10 and 11, Chicago: SPSS Inc.). The results presented in this article are from this additional cleaning and analyses.

In all countries, the MICAH programme received ethics clearance from the Ministry of Health. In addition, local authorities, community leaders and respondents were informed of the purpose of the evaluation and consent was requested from each individual respondent.

Quality control

MICAH evaluation survey data were collected by trained enumerators, with field supervision by trained programme staff and overall supervision and guidance by an international team of epidemiologists, nutritionists and statisticians. Accepted practices for field-level quality control were followed. However, systematic repeat measures, repeat sampling and inter-lab sampling were not available for quality control of the MICAH data. Therefore alternative, post-hoc methods were used for evaluating the quality of data collected. Some of these methods have been used previously, whereas others were developed for the purpose of this evaluation. The methods were adapted for each specific indicator, but are outlined in general in Table 2. Not all variables were amenable to all types of quality control checks.

Table 2

Methods used for quality control assessment

Method	Expectations of high-quality data
• Comparison of magnitude of SDs of continuous variables to SDs in other, well-controlled studies	• SDs of MICAH will be similar to othersb
• Consistency of relationships between coverage and outcomes; congruency in chain of results	• Increased VAC coverage, iodized salt use and iron supplementation coverage will lead to decreased nightblindness and higher breast-milk retinol, higher urinary iodine, and, possibly, reduced anaemia, respectively
• Comparison of magnitude of apparent MICAH effect to observed effects in efficacy trials	• Magnitude of MICAH effects will not greatly exceed maximum magnitude of effects in efficacy trials
• Pattern of cross-sectional growth curves	• Average HAZ and WAZ of infants <3 months of age will be near 0; Z-score nadir reached between 12 and 24 months
	• Change in average Z-scores by month of age will not fluctuate widely
	• Sample size at each age (by month) will be approximately equal
• Pattern of immunization coverage	• BCG > DPT3 ∼ OPV3 ≥ measles

aThese methods were developed post hoc. The acceptable levels (e.g. what constituted ‘similar’ or ‘widely’) were decided after the analyses were done. Limits of this method are described in the ‘Discussion’ section.

bThis method of comparing SDs with reference populations has been recommended for anthropometrics. We assume that common levels of variations will exist for other variables.

BCG, Bacille Calmette-Guérin Vaccine; DPT3, 3 doses of Diptheria, Pertussis, Tetanus Vaccine; OPV3, 3 doses of Oral Polio Vaccine.

Adequacy evaluation

The magnitude of the change for each key indicator from baseline to follow-up to final survey was compared with that observed in published controlled trials and reports of other large-scale programmes (Table 4). If improvement was of comparable magnitude to the high end observed in controlled trials, the impact was considered ‘high’; if the improvement was of a range common in other programmes, the impact was considered ‘moderate’; if the change was smaller than other programmes but greater than zero the impact was considered ‘low’. Testing of differences from baseline to follow-up was done by chi-square for categorical variables and t-tests for continuous variables.

Table 4

The maximum efficacy found in the literature for clinical, biochemical and biological indicators tracked in MICAH

	Maximum efficacy observed
Indicator	Controlled trials	Large-scale programmes
Vitamin A
Night blindness in children <5 years of age	From 1.2 to 0.1%45 From 6 to 1.3%46	From 2.3 to 0.6%47
Bitot’s spots in children <5 years of age	From 0.8 to 0.5%45 From 6 to 1.3%46	From 1.1 to 0.5%47
Low breast-milk retinol	From 70 to 36%,48 from 25.6 to 15.2%,49 similar and slightly different results observed elsewhere36,50
Iodine
Total goitre rate	Although it is not uncommon to have no change in goitre prevalence,34 marked reduction2,28 and near elimination2,51 of goitre through long-term programmes is achievable
Low urinary iodine	From ∼25 to 0%52	From 39, 56, 77 and 1.5% to ∼0% 34
Iron
Anaemia in women 15–49 years of age	From 7 to 1.6%;9 46 to 19%53,54	From 73.3 to 25.4%54
Anaemia in pregnant women	As high as from 70 to 10%;4 73% reduction (RR = 0.27) in pooled analysis of iron folate supplementation2	From 62 to 52%,4 from 55 to 32%55
Anaemia in children <5 years of age	As high as from 72 to 19%;4 from 38 to 62% reduction in non-malarial areas; from 6 to 32% in malarial hyperendemic areas2	From 85 to 68%;4 from 54 to 14%56
Malaria
Malaria in pregnant women	ITNs have protective efficacy of 0.2657 for malaria parasitemia
Malaria in children <5 years of age	Average of protective efficacy of 0.13, maximum of 0.57 with stable malaria and 0.42 with unstable malaria;58 48% reduction in risk of clinical malaria through intermittent preventive treatment (IPT)2
Anthropometrics
HAZ < –2	Average increase in Z-score was ∼0.3, but single studies as high as 1.75.59–61 When food supplements delivered to food-insecure populations HAZ increased by an average of 0.41.2 Increase of 0.3 Z-score would lead to reduction in stunting of ∼10–15%.
WAZ < –2	Average increase in Z-score ∼0.3(60, 61). Increase of 0.3 Z-score would lead to reduction in underweight of ∼10–15%.
Exclusive breastfeeding (EBF)
EBF for 6 months	0.6–7.9%;25 Average odds ratio (OR) of ∼3.5, maximum OR of >90 (40% in intervention group, 0% in control)2	As high as 46–68%31

RR, Relative Risk.

Results

Comparison of SDs

In most cases for which SDs could be calculated, the MICAH values were within range of observed SDs in published trials. Furthermore, the SDs were usually relatively constant between survey years (Table 3). Unusually high SDs (20-fold of that reported elsewhere) were observed for breast-milk retinol in Tanzania. The SDs for haemoglobin in pregnant women in Ghana in 1997, Tanzania in 2000 and Ethiopia in 2000 as well as for children <5 years of age in Ghana in 1997 were outside the range of SDs reported elsewhere by –40 to +15%.

Table 3

SDs of continuous variables in MICAH surveys in baseline (1996 or 1997), follow-up (2000) and endline (2004) compared with examples from the literature, for quality control purposes

	Ethiopia			Ghana			Malawi			Tanzania
	1997	2000	2004	1997	2000	2004	1996	2000	2004	1997	2000	2004	Examples
Breast-milk retinol (µmol/l)					1.4	0.5					22.8	13.6	0.70, 0.4835 1.19, 0.9936
Urinary iodine—school-age children (µg/l)				25.4	140.0	132.1		76.9	98.0		20.5	38.9	145, 80,37 48.1,38 9.2–40.7,39 40.440
Haemoglobin— women (g/dl)	1.8	1.1	1.6	1.9	1.9	1.4		1.7	1.6	2.1	1.9	2.0	1.941
Haemoglobin— pregnant women (g/dl)	1.7	1.1	1.5	2.2	1.6	1.5	1.7	1.6	1.5	1.6	2.3	1.7	1.5,421.8,43 1.644
Haemoglobin— children <5 years of age (g/dl)				2.3	1.8	1.5	1.8	1.8	1.7	1.8	1.9	1.6	1.941
HAZ		1.5	1.4	1.3	1.2	1.2	1.5	1.5	1.2	1.5	1.5	1.3	1.8423
WAZ		1.3	1.1	1.3	1.2	1.2	1.4	1.5	1.1	1.4	1.4	1.3	1.50 23

Unexpected or unusual values are indicated in bold.

Magnitude of change compared with other programmes and trials

The magnitude of change in the MICAH indicators was compared with what has been achieved in trials and large-scale nutrition or health interventions (Table 4). The Table 4 data were drawn largely from a recent review and the references therein. The objective was to document the maximum impacts observed, not necessarily the expected impacts, and so only the highest efficacies are reported. No data were found for nightblindness and Bitot’s spots in school-aged children, or for malaria parasitemia in non-pregnant women. Data were available for all other indicators.

The maximum impacts observed in efficacy trials were then compared with the changes observed in MICAH (Table 5). Changes in MICAH, equal to or greater than those observed in reviewed efficacy trials, are shown in bold. Changes as large as those observed in efficacy trials may initially be considered ‘questionable’—were there sampling biases in either the efficacy trials or the MICAH sample? And if not, can impacts observed in effectiveness studies be as large as those observed in efficacy trials? There were questionable results in all four countries but there were no indicators for which the results were questionable in more than 3 countries. Within countries, there were as few as three and as many as six questionable results. Whether these are false results due to poor assessment or excellent results due to strong programming is considered in the ‘Discussion’ Section.

Relationship between coverage and outcome

Changes in coverage data for vitamin A and iron supplementation and iodine fortification correspond to expected changes in outcomes in most cases (Table 5, cells with non-concordance between programme coverage and outcome indicator are shaded). Increased coverage with VACs from one survey to the next corresponded to decreased nightblindness in Ethiopia and Tanzania and decreased prevalence of low breast-milk retinol in Ghana. An increase in the percentage of households with iodized salt corresponded to decreased prevalence of low urinary iodine in children in Malawi and Tanzania but not Ghana. In Ethiopia, goitre rates dropped despite the decrease in iodized salt in the households and in Ghana goitre rates did not decrease despite the increase in households with iodized salt. An increase in iron tablet supplementation corresponded with a decrease in anaemia prevalence in seven of eight cases (that is, across countries for pregnant women, non-pregnant women and children <5 years of age, where supplementation and anaemia data are available for ≥2 years). In all five cases where malaria parasitemia data were available, an increase in iron tablet supplementation and a decrease in parasitemia corresponded to a decrease in anaemia. In all four countries, an increase in access to a protected water source and latrines corresponded to decreased intestinal parasite infection.

Pattern of cross-sectional growth curves

For each country survey, average HAZ and WAZ and sample size at 1-month intervals were plotted against child age, and the plot was examined for the characteristics described in Table 2. The data were then scored as low quality (do not use), medium quality (use with caution) or high quality. The Ethiopian data were considered medium quality as many of the child ages were reported by mothers, and they appeared to be very often rounded to the half-year and year. The 2004 data for age > 48 months were low quality (Z-scores higher than considered feasible). The data from Ghana and Malawi were low quality at age <6 months as there was a gain of ∼1 Z-score from birth to 4 months of age that is considered unlikely; otherwise the data were of high quality in all three surveys. Tanzanian data were high quality in all three surveys.

Immunization coverage

In all cases, except Ethiopia 2004, the MICAH immunization coverage rates followed the pattern of BCG > DPT3 ≈ OPV3 ≥ measles. Furthermore, the ratio of coverage of BCG:measles was between 1.0 and 1.3 in all cases. Overall, the MICAH immunization data were considered reliable and useable for evaluation purposes.

The MICAH programme impact in each country is summarized in Table 6, where the change in each indicator over the course of the intervention (appearing in Table 5) is graded as high, moderate, low, none or negative, and the quality of the data supporting the claim is rated as high, medium or low.

Table 6

Summary of impact of MICAH, by indicator and country

	Ethiopia	Ghana	Malawi	Tanzania
Vitamin A
Night blindness in children <5 years of age	Higha	Lowa
Bitot’s spots in children <5 years of age	Higha			Highb
Night blindness in school-age children	Moderatea
Bitot’s spots in school-age children	Moderatea
Low breast-milk retinol		Highb		Not assessedd
Iodine
Total goitre rate	Lowa	Nonea	Higha	Highc
Low urinary iodine		Lowa	Higha	Not assessedd
Iron
Anaemia in women 15–49 years of age	Negativea	Higha	Lowa	Moderatea
Anaemia in pregnant women		Higha	Higha	Lowa
Anaemia in children <5 years of age		Higha	Higha	Lowa
Malaria
Malaria in women		Higha	Higha
Malaria in pregnant women			Higha	Highb
Malaria in children <5 years of age		Higha	Higha	Highb
EBF for 6 months	Highc	Highc	Highc	Lowc
Immunization	Higha	Lowa	Moderatea	Lowa
Anthropometry in children
HAZ < –2	Noneb	Lowa	Higha	Higha
WAZ < –2	Lowb	Higha	Higha	Higha

High, moderate, low, none and negative indicate magnitude of change in indicators. High: similar to upper end of impact of controlled trials; moderate: less than trials and of a range common in other programmes; low: small effect, but greater than zero. t-tests conducted for continuous variables and chi-square tests for categorical variables. Results with P < 0.05 indicated with bold text. Quality of data: ahigh quality; bmoderate quality; clow quality; dtoo poor quality to assess impact.

There were moderate improvements in vitamin A status in Ethiopian school-age children, and high improvement in vitamin A status of Ethiopian children <5 years of age and Ghanaian mothers. Nightblindness dropped to zero in Tanzanian children <5 years of age (P > 0.05). Iodine status improved in Ethiopia, Malawi and Tanzania. Anaemia rates decreased in women, pregnant women and pre-school children in Ghana, Malawi and Tanzania, but increased in Ethiopian women. Malaria prevalence dropped in the same groups. Large increases were reported for rates of EBF and immunization rates and child growth status also improved in all countries.

Discussion

Integrated nutrition programmes are commonly recommended for developing countries, but they are not commonly implemented and even more rarely evaluated. A recent review identified 30 large-scale nutrition programmes, most of which did not have strong designs for monitoring and evaluation.

The MICAH programme appears to be unique in that it was an NGO-led multi-year, multi-country integrated nutrition and health intervention with monitoring of numerous clinical, biochemical and behavioural indicators. The data were collected for results-based management purposes, and as part of the contractual requirements of the donor, and have been reported in detail to the donor. In this study, we considered whether the evaluation data were of sufficient quality to evaluate programme impacts, and, if so, the direction and magnitude of the impact—an adequacy evaluation.

Post-hoc assessment of quality of data collected

Most of the collected data were considered of good quality, according to the methods used in this article (Table 2). The important exceptions included anthropometric data in Ethiopia, anthropometric data for children <6 months of age in Ghana and Malawi, breast-milk retinol analysis and urinary iodine in Tanzania and EBF data in all countries. There may have been poor laboratory practices in Tanzania for breast-milk retinol and urinary iodine, and other published evaluations of the quality of retrospective EBF data cast doubt on all retrospective exclusive breastfeeding data. In addition, breast milk retinol data and urinary iodine data in Tanzania were of very low quality and were excluded from the adequacy evaluation.

Based on the quality checks carried out in this study, it appears that the following micronutrient indicators are of sufficient quality when collected by programme-based organizations (such as NGOs) if adequate training and quality control measures are taken during the data collection: for vitamin A status—clinical indicators (night blindness, Bitot’s spots); for iodine status—goitre; and the biochemical indicator of anaemia as a proxy for iron status. In addition, anthropometric indicators can also be reliably collected for children between 6 and 59 months of age. Health indicators such as malaria and immunizations coverage also appeared to be of adequate quality.

Although the quality checks employed here (Table 2) cannot replace standard methods of repeat measures, inter-laboratory comparisons, inter-observer reliability tests and others, they do offer a relatively inexpensive and easily applied means of post-hoc data quality assessment. These quality check analyses allow the assessment of the impact of MICAH to be conducted with more confidence. Furthermore, the analyses give confidence that future surveys carried out by World Vision, or by other similarly staffed and resourced NGOs, can be done with sufficient accuracy and reliability for an adequacy evaluation.

However, these post-hoc methods are limited by their subjectivity, such as choosing the cut-offs used for magnitude of SDs, deciding what constitutes a consistent relationship between coverage and outcome, the maximum feasible magnitude of change for each indicator and the definition of an acceptable level of fluctuation in average Z-score between months. These were all determined after the analyses were completed and there are no existing guidelines. Further research is needed to provide standards and guidelines on such post-hoc methods. Having standards and guidelines could increase the reliability of data collected by non-research and/or programme institutions without adding significantly to the cost of data collection.

Magnitude of impact

Based on the finding that most of the indicators were of good quality, an evaluation of the change in indicators over time (adequacy evaluation) revealed many positive results in the MICAH programme (Tables 5 and 6).

MICAH programme staff reported five aspects of the programme that they believe uniquely contributed to the positive results: (i) results-based management identified programme activities that were not producing positive outcomes and allowed for mid-stream corrections, such as the decision in Malawi (following the 2000 evaluation) to reduce the geographic spread and intensify the intervention in a smaller area to ensure all participants received all interventions; (ii) provision of regular, intensive technical support in the form of regular email correspondence, monitoring and support visits by World Vision technical staff and expert consultants, and annual training workshops; (iii) use of a broad-based integrated package of interventions rather than relying on a single ‘magic bullet’; (iv) community participation in programme design, implementation and monitoring and evaluation, such that some interventions could be tailored to suit community preferences, especially regarding animal husbandry where existing practices differed from community to community; and (v) regular supervision of staff in communities. Other aspects of the intervention, such as having competent staff, effective supply chains and proper financial controls are always prerequisites for success, though not unique to MICAH’s success.

In Malawi there was a positive impact on all indicators, perhaps a result of higher investment per direct beneficiary compared with the other countries, as well as MICAH’s support for the National Micronutrient Coordinator position housed within the Ministry of Health and Population that led to concurrent positive developments in national nutrition policy and action. In Ghana and Tanzania, improvements were observed in vitamin A, iron, malaria, EBF, immunization and child growth status. There were fewer positive results in Ethiopia, which may be because the intervention efforts were ‘diluted’ over a larger number of beneficiaries. However, there was positive impact on vitamin A status in school-age children—an uncommon target and an uncommon success.

It has been estimated that ∼US $5−10/head/year is a workable level of expenditure in nutrition programmes, and Malawi, the one MICAH country with costs that fell within this range, had the broadest range of positive results and overall impact appeared correlated with per capita expenditures.

Vitamin A

Decreases in clinical symptoms of vitamin A are similar to those found in other large-scale programmes. The xerophthalmia results in Ethiopia were better than observed previously (Table 4), but considering the 10-year duration of the intervention the large effect reported is feasible.

The logistics of breast-milk collection and storage were difficult, as was measuring retinol in the samples. The breast-milk retinol data collected were considered of moderate quality (Ghana) or too poor quality to use (Tanzania). Breast-milk retinol data should perhaps be reserved for research/controlled trials, with other indicators being used for programme monitoring.

Coverage of children <5 years of age with vitamin A supplements increased greatly in Ethiopia, Ghana and Tanzania (from 9 to > 70%), although it dropped in Ghana in 2004 to 26% when a novel method of distribution was tested. Coverage with supplements to new mothers increased in Ethiopia, Ghana and Tanzania from ∼10 to 40–83%, an effect likely related to the introduction by MICAH of community-based distribution alongside facility-based distribution. There was a concomitant improvement in vitamin A status in Ethiopian children <5 years of age (P < 0.001) and of school age (P < 0.001), and in Ghanaian post-partum women (P < 0.001), but not in children <5 years of age in Ghana (P = 0.53) and Tanzania (P < 0.001) (although observed xerophthalmia dropped to zero).

Iodine

There were improvements in iodine status in Malawi, consistent with improvements observed elsewhere when successful long-term salt iodization programmes are in place.,, In Malawi, the MICAH programme had a particularly strong focus on increasing usage of iodized salt (increased household coverage from 59% in 1996 to 88% in 2004) through efforts at multiple levels (i.e. national, regional and community), as well as ensuring provision of iodized oil capsules in areas where iodine deficiency disorder was a severe public health problem. National-level advocacy facilitated full implementation of salt iodization legislation. District-level customs and border control staff and port authority health inspectors implemented testing of salt at border crossings. Intensive community education and regular monitoring of salt at the vendor and household levels increased consumer awareness and demand for iodized salt. In contrast, although iodine deficiency was a major concern of MICAH Ethiopia, there was decreasing iodized salt available at household level (i.e. 48% in 1997 decreased to 2% in 2004) due to conflict with neighbouring Eritrea, their traditional supplier of iodized salt, and lack of identification of adequate alternate sources of iodized salt. Neither Ghana nor Tanzania had major public health problems with iodine deficiency at the beginning of the programme, based on goitre rates (i.e. 2 and 9%, respectively). In Ghana, a reliable iodized salt supply was difficult to secure, as there are many small-scale producers in the country and existing salt iodization legislation is difficult to enforce, although household supply increased from 33 to 63% during the programme. This was due to strong programme advocacy efforts at the national level and collaboration with one of the major iodized salt suppliers in the country to provide increased supply to the targeted district.

In Tanzania, the prevalence of goitre was reduced to 0%, perhaps due to the programme emphasis on ensuring communities’ access to iodized salt (i.e. household salt coverage increased from 41% in 2000 to 72% in 2004).

Iron

In project areas of Ghana, Malawi and Tanzania, where anaemia was identified as a severe public health problem, programme resources were focused on decreasing anaemia and subsequent decreases in anaemia prevalence were observed. This is a particularly important finding, as anaemia is often very difficult to address—although this magnitude of change is not unprecedented (Table 4), it is uncommon. The improvements observed in MICAH are likely a result of the use of multiple strategies (supplements, animal source foods, fortification, malaria control, water quality improvement, sanitation, deworming) over a long intervention period. Furthermore, treating not only pregnant women but also non-pregnant adult women with iron supplementation allows for improved iron status before pregnancy, making it easier to prevent anaemia in pregnancy and may explain the unusually large improvement in anaemia rates in pregnant women of Ghana and Malawi despite high prevalence of HIV/AIDS in the latter country.

EBF

The large increases reported in EBF are relatively high compared with other interventions (of much shorter duration)., Self-reports on duration of exclusive breastfeeding have been shown to have low to moderate accuracy (whereas duration of any breastfeeding has higher accuracy)., Although over-reporting of EBF duration may be expected as promotional messages sensitize mothers to the ‘right’ answers, this does not necessarily occur. Although acknowledging the potential weakness of EBF data, MICAH’s focus on breastfeeding communication through community-based interventions for behaviour change [e.g. through women’s groups, community health workers (CHWs) and volunteers, traditional birth attendants, agricultural extension workers] in all countries and partnerships in Ghana and Malawi with the Baby Friendly Hospital Initiative—in which baby friendly practices were promoted not only in the hospitals but also in health centres that provided outreach services by trained nurses to pregnant women and new mothers in rural communities—is likely to have contributed to improved infant feeding practices.

Immunization

MICAH supported the delivery system (cold chain equipment, monitoring, per diems), and put extensive efforts into social mobilization and community education. This likely contributed to improvements in immunization coverage in all four countries (high in Ethiopia, moderate in Malawi, low in Ghana and Tanzania).

Anthropometry

Perhaps the most important indicator of all is child growth status, as it integrates the health effects of all the interventions. Stunting and underweight were reduced in all countries by ≥10%, except Ethiopia. Improved dietary diversification was promoted for children <2 years of age, utilizing food demonstrations and nutrition education to caregivers to promote improved infant and complementary feeding practices, including a revolving fund to increase household access to animal source foods in Ghana, Malawi and Tanzania. Infant and young child feeding was also an emphasis of the programme in Ghana, Malawi and Tanzania, through women’s groups, education through CHWs and volunteers, and agricultural extension workers (Malawi), as well as through supporting the Baby Friendly Hospital Initiative (Ghana and Malawi), and forming mother support groups in communities (Ghana and Tanzania). The smaller impact on child growth in Ethiopia is consistent with less emphasis placed on infant and young child feeding. Reduced illness frequency and parasitic infection also likely contributed to the improvement in growth, given the programme’s emphasis on water, sanitation and deworming.

Bhutta et al. developed a model predicting the reduction in stunting with a large-scale programme comprised of eight types of interventions: balanced energy–protein supplementation, intermittent preventive treatment of malaria, multiple micronutrient supplementation in pregnancy, breastfeeding promotion, complementary feeding and other supportive feeding strategies, vitamin A and zinc supplementation and hygiene intervention. If coverage of all eight interventions was 99 or 70% then the relative reduction in prevalence of stunting at 12 months would be 33 and 23%, at 24 months 36 and 24% and at 36 months 36 and 24%, respectively. Over the 10-year MICAH programme (which included four of the eight interventions in the Bhutta model, plus other components) this theoretical maximum reduction in prevalence of stunting of 24–36% after 3 years was approximated in Malawi (15%), Ghana (28%), and Tanzania (36%). This is most encouraging and points again to the potential impact of multiple, integrated strategies.

Attribution

The question of ‘attribution’ of improvements is not easy to answer. An adequacy evaluation is not designed to enable attribution, as other confounding factors have not been controlled for: a plausibility evaluation would be required. In MICAH’s case, for example, concurrent activities and inputs by other international (e.g. UNICEF) and national (e.g. Ministry of Health) actors would also have contributed to improvements; also, there were some improvements in average socio-economic status of beneficiaries over time, which, independent of MICAH, could have contributed to improved water and latrine access, ITN usage and child growth status. However, the MICAH programme was implemented in predominantly rural areas where existing government nutrition and health services were generally weak. In an attempt to estimate the impact of MICAH independent of other local and global influences, comparisons were made with DHS data for the national, rural samples. The change in those indicators for which there were both MICAH and DHS data at baseline or follow-up and endline is shown in Table 7 as the difference of the differences (the difference between the MICAH difference between endline and baseline and the DHS difference between endline and baseline). For most indicators in all four countries, MICAH areas outperformed rural areas of the country as a whole. Although the comparison groups are not perfectly suited as control groups (different years, different baseline conditions, other differences not related to MICAH), given the MICAH performance compared with the DHS along with the general concordance between coverage and outcome indicators in the MICAH samples, it does support the interpretation that the improvements in programme beneficiaries are greater than the general trends, and greater than would have occurred if MICAH was not implemented.

Table 7

Difference in differences of coverage rates, EBF and prevalence of stunting, wasting and underweight between MICAH surveys and DHS surveys

	Ethiopiab	Ghanac	Malawid	Tanzaniae
VAC coverage, children <5 years of age	16
VAC coverage, post-partum women	10
Measles coverage	44	–5	13	7
Access to protected water source	41	20	–2	–4
Access to latrines	–10	2.5	2	–7
EBF for 6 months		9	14
Stunting (HAZ <–2)	–4	0	3	9
Underweight (WAZ <–2)	–6	8	10	11

aCalculated in general as (EndlineMICAH–BaselineMICAH) – (EndlineDHS – BaselineDHS) for those variables where ‘higher’ is ‘better’. For stunting and underweight, where ‘lower’ is better, calculated as: (EndlineDHS – BaselineDHS) – (EndlineMICAH–BaselineMICAH). A positive value indicates that MICAH villages improved more than the rural, nationally-representative, samples. DHS data used when available within 1 year of timing of MICAH survey; DHS data not available in baseline years for Ethiopia and Malawi, therefore mid-line year (2000) used as ‘baseline’ in calculation.

b(2004MICAH-2000MICAH) – (2005DHS – 2000DHS).

c(2004MICAH-1997MICAH) – (2003DHS – 1998DHS).

d(2004MICAH-2000MICAH) – (2004DHS – 2000DHS).

e(2004MICAH-1997MICAH) – (2004DHS – 1996DHS).

Was this an adequacy evaluation?

Although this evaluation has documented the trends over time, it still perhaps falls short of a full adequacy evaluation according to the prerequisites proposed by Victora et al.: ‘(i) the causal pathway must be relatively short and simple, (ii) the expected impact must be large and (iii) confounding must be unlikely’. These prerequisites are vague and open to interpretation.

The first criteria proposed by Victora et al. would not be met, as in the MICAH programme the causal pathways are variable, ranging from the very short and direct (e.g. MICAH-funded staff supervising the distribution of MICAH-purchased VAC to school-aged children in MICAH communities to decrease clinical symptoms of vitamin A defieciency among school-aged children), to long and indirect (e.g. MICAH partners using MICAH-financed training materials to advocate for salt iodization to increase availability of iodized salt at the household level to decrease goitre rate), where the health impact, if any, cannot clearly be attributed to the availability of training materials. In addition, the use of multiple, integrated strategies precludes the evaluation of any one intervention’s causal pathway.

The second criterion, that expected impacts be large, is also not consistent, as expected impacts are variable in the MICAH programme. It is not uncommon for salt iodization programmes to virtually eliminate low urinary iodine. On the other hand, improvements in growth status in previously published research are, on average, ∼0.3 Z-scores, which may be about one-third to one-sixth of the population standard deviation. However, the sample sizes chosen provided adequate power in most cases such that statistical tests of programmatically important changes had results with P < 0.05.

The third prerequisite, that confounders are unlikely, is most certainly not met, as confounding (i.e. effects from multiple sources) is likely for many of the indicators. However, it is interesting that programme success was observed in those indicators most open to confounding (growth status, anaemia) as well as those least open to confounding (e.g. nightblindness, urinary iodine).

Thus, perhaps not a ‘perfect’ adequacy evaluation, the analyses and interpretation presented here are useful in assessing the quality of MICAH monitoring and the level of impact, and strongly suggest important positive changes of micronutrient status during the MICAH programme.

Conclusion

In conducting an adequacy evaluation we documented trends in health indicators following introduction of the MICAH programme. In doing so we have established World Vision’s capacity for undertaking such evaluations and we expect other similarly positioned NGOs to be able to implement similar evaluations. The challenge in future assessments will be to determine the most appropriate key indicators to measure impact, to improve the quality of the data collection and management so that all of the data are of high quality and to strengthen the study design to enable ‘plausibility’ and ‘probability’ evaluations.

The rating of the magnitude of changes as high, moderate or low was, of course, subjective. Considering the specific characteristics of MICAH, it is a challenge in the interpretation to determine if the ‘questionable’ changes (i.e. improvements greater than observed in efficacy trials) were real results due to excellent programming or false positives due to limitations in the data collection methods. The duration of MICAH was longer than almost all the efficacy trials reviewed, and MICAH simultaneously addressed numerous aspects of nutrition and health, thus results larger than those reported in the published literature on single interventions are feasible. Therefore, unless there were specific, known problems, the results are considered to be valid, whereas it is acknowledged that there may be false positives. Further research on broad-based, long-term interventions is required to quantify maximum magnitude of impacts that may be feasible through long-term interventions. Presumably with a very successful, broad-based long-term intervention, the end points can be similar to the nutrition and health situation in developed countries (e.g. no goitre, anaemia prevalence <5%).

In conclusion, we have identified three lessons from this work. First, post-hoc methods appear reasonable and feasible for quality control assessment, but further research is required for standardization of the cut-offs and limits for the methods described in Table 2. Secondly, NGOs that are capable of executing large-scale health interventions can also be capable of conducting adequacy evaluations, using similar levels of support as MICAH, if planned for from the beginning. Thirdly, for programme implementation, integrated nutrition and health interventions can produce excellent results, perhaps even greater than randomized controlled trials. Further operational research quantifying the magnitude of the effectiveness of integrated programmes is required.

Supplementary Data

Supplementary data are available at IJE online.

Funding

Canadian International Development Agency (CIDA) (694/M3544, M-011372) and World Vision Canada.

This article complements the sparse literature on evaluations of large-scale nutrition and health programmes, through the ‘adequacy evaluation’ of an NGO-led, large-scale, multi-country, 10-year nutrition and health programme.

Post-hoc assessment of data quality indicated most data were of moderate or high quality and therefore suitable for an adequacy evaluation.

Results varied by country, with one or more countries achieving improvements in vitamin A and iodine status, decreases in anaemia rates and malaria prevalence, higher rates of EBF and immunization and improved child growth.

The magnitude of the nutrition and health impact was often larger than observed in controlled interventions or trials.