Heterogeneity in the validity of administrative-based estimates of immunization coverage across health districts in Burkina Faso: implications for measurement, monitoring and planning.

BACKGROUND: Data aggregation in national information systems begins at the district level. Decentralization has given districts a lead role in health planning and management, therefore validity of administrative-based estimates at that level is important to improve the performance of immunization information systems.
OBJECTIVE: To assess the validity of administrative-based immunization estimates and their usability for planning and monitoring activities at district level.
METHODS: DTP3 and measles coverage rates from administrative sources were compared with estimates from the EPI cluster survey (ECS) and Demographic and Health Survey (DHS) carried out in 2003 at national and regional levels. ECS estimates were compared with administrative rates across the 52 districts, which were classified into three groups: those where administrative rates were underestimating, overestimating or concordant with ECS estimates (differences within 95% CI of ECS rate).
RESULTS: National rates provided by administrative data and ECS are similar (74% and 71% for DTP3 and 68% and 66% for measles, respectively); DHS estimates are much lower. Regional administrative data show large discrepancies when compared against ECS and DHS data (differences sometimes reaching 30 percentage points). At district level, geographical area is correlated with over- or underestimation by administrative sources, which overestimate DTP3 and measles coverage in remote areas. Underestimation is observed in districts near urban and highly populated centres. Over- and underestimation are independent of the antigen under consideration.
CONCLUSIONS: Variability in immunization coverage across districts highlights the limitations of using nationally aggregated indicators. If district data are to be used in monitoring and planning immunization programmes as intended by decentralization, heterogeneity in their validity must be reduced. The authors recommend: (1) strengthening administrative data systems; (2) implementing indicators that are insensitive to population mobility; (3) integrating surveys into monitoring processes at the subnational level; (4) actively promoting the use of coverage information by local personnel and district-level staff.

The similarity of national coverage rates provided by administrative and population-based sources masks a large heterogeneity at the district level, with administrative data at times overestimating or underestimating real coverage.

Administrative data should be treated cautiously in areas affected by migration and mobility across catchment areas and in areas where fertility levels differ significantly from the country average.

With decentralization, responsibility for planning and monitoring is increasingly being transferred to health districts. Therefore, this is the level at which critical information, such as accurate district-based estimates of immunization coverage, is urgently needed.

There is a need not only to strengthen administrative data systems and expand existing efforts to improve the collection, management and communication of information, but also to include indicators that are insensitive to the mobility of targeted populations and to integrate coverage surveys into monitoring processes at subnational level. Most of all, there is a need to promote effective utilization of the information directly at the district level.

Introduction

Worldwide, administrative data tend to be the primary source used by health information systems to estimate immunization coverage and to guide decision-makers and Expanded Programme on Immunization (EPI) managers in planning and monitoring immunization programmes and assessing their performance. They are also the primary source used by national authorities to fill in the WHO/UNICEF joint reporting forms (JRF) on vaccine-preventable diseases, a major source of information on global coverage and immunization systems performance (WHO 2006).

Because resource allocation processes tend to be more intimately linked to systems performance, and because intervention requires estimates of coverage levels, administrative data play a key role in determining the allocation of government and donor-supported initiatives (such as the GAVI Alliance) aimed at strengthening countries’ immunization activities. However, the validity of administrative data has been repeatedly questioned (Crabb 2003; Murray et al. 2003; Lim et al. 2008). The immunization Data Quality Audit (DQA) was developed and implemented with the support of the World Health Organization (WHO) and GAVI to help overcome these limitations and improve the usefulness of information systems for immunization data (WHO 2003). DQAs help diagnose specific weaknesses in health information systems through systematic process analysis at the national, district and health facility levels. The quality, accuracy, timeliness and completeness of immunization data is audited, and the DQA provides information about which level of the reporting system contributes most to inaccuracies (Ronveaux et al. 2005). Recommendations for change are made and the quantitative indications of reporting consistency and accuracy provided by DQAs can be further used to monitor the impacts of strategies implemented to improve the quality of health information systems.

While it provides a comprehensive assessment of process quality, as well as evidence on current practices to support quality assurance strategies, the DQA does not provide indications of the validity of existing immunization coverage estimates per se. For this, administrative-based estimates have to be compared with population-based estimates, as Murray et al. (2003) did, for example, in 45 countries using the household Demographic and Health Survey (DHS), and Lim et al. (2008) did more recently using DHSs, Multiple Indicator Cluster Surveys (MICSs) and country-specific surveys. Both studies concluded that administrative data tend to overstate the number of immunized children. Overestimation of immunization coverage by administrative data has also been reported in Burkina Faso (Zuber et al. 2003) and Cameroon (Guyer and Atangana 1977), as well as in a sample of case study countries including Uganda, India and the Philippines (UNICEF 1996), while administrative data have underestimated coverage rates in surveys done in Panama (Huezo et al. 1982) and Zimbabwe (Borgdorff et al. 1988).

The gap thus measured between population-based and administrative sources provides valuable information on the outcomes of processes of information production and management. It is important to realize, however, that the gap observed in any given country is an aggregate that can be represented as a mean estimate of variations observed within regions and health districts. The validity of coverage estimates can, in fact, vary considerably from one district to another. This heterogeneity is well reflected by DQA indicators, which tend to vary substantially between districts, notably in poorly performing countries like Burkina Faso (Bosch-Capblanch et al. 2009), and national estimates of the verification factors have such wide confidence intervals (CIs) that this alone compromises their usefulness at the subnational level. For example, nothing prevents national estimatesshowing concordance between administrative- and population-based sources from actually masking heterogeneity at the district level, with administrative data sometimes overestimating and sometimes underestimating real coverage.

Because the district is the first level of data aggregation in countries’ national information systems, and because decentralization has given districts a lead role in health planning and management, it is at least as relevant to get evidence on the accuracy of administrative-based estimates at the district level and on inter-district variations, as it is to obtain evidence of overall health information systems performance.

Using an EPI cluster survey conducted in Burkina Faso in 2003, the authors previously showed that the national average hides wide variations between districts in immunization coverage (Bicaba et al. 2009). A consultation undertaken with EPI officials and district health managers in Burkina Faso revealed substantial underutilization of existing administrative data for planning and monitoring immunization activities, ascribable to shared apprehensions about their quality. District heterogeneity also emerges from a recent comparative study across African countries which demonstrated, among other things, that one-third of the districts had not achieved 50% DTP3 coverage by the end of 2004 (Arevshatian et al. 2007). Significant heterogeneity in immunization outcomes between states and municipalities has also been reported, advocating convincingly for the need to look beyond the aggregated national or state pictures (Pande and Yazbeck 2003; Gaudin and Yazbeck 2006).

However, neither the research we have been able to consult to date nor the available literature allows us to document variations in the quality of administrative-based estimates of immunization coverage across districts. This can be explained by the limitations of available surveys. The DHSs used by Murray et al. (2003) or Zuber et al. (2003), for example, only make it possible to provide regional estimates, at best. Another probable explanation is the intense attention focused by national decision-makers and cooperation agencies on national aggregates, which are more directly applicable and more immediately convincing than are measurements that take into account territorial distribution.

This article aims to provide evidence on the validity of administrative-based estimates of immunization coverage across 52 health districts in Burkina Faso and their usability for planning and monitoring immunization activities at the district level. The analysis is followed by a preliminary exploration of factors that might account for inter-district differences in the significance and size of inconsistencies observed between administrative-based and population-based estimates. The article takes advantage of the existence of a cluster survey carried out nationwide in 2003 using the standard EPI methodology. Contrary to the DHS, whose sampling design only allows regional-level estimates, the EPI cluster survey (ECS) provides district-level representative estimates, thus offering a unique opportunity for assessing the quality of administrative coverage data beyond the national average.

Methods

Data sources

We used data from the administrative registers of the 52 health districts of Burkina Faso and from the ECS conducted in 2003. We also drew upon the DHS conducted in 2003 to add another source of comparison at the regional level and address some methodological issues.

Administrative estimates of coverage (2002 and 2003)

The numerator is based on the count of the number of children 0–11 months old who have received each antigen in the previous month, over one calendar year. Each health unit is required to send monthly information to the corresponding health district on the number of children who have received each antigen. The information is collected at the health unit in the following way: for every child vaccinated, a mark is written in the vaccination log. No individual information on the child is registered. At the end of the month, an activity report is written based on the compiled daily immunization logs of all registered antigens. The denominator, the district population of children <12 months of age, was estimated as 4.21% of the total population, based on the 1996 census, using a projection of the 0- to 1-year-olds updated with an annual population growth factor of 2.6%.

2003 ECS estimates of coverage

The ECS, conducted nationwide in May–June 2003, was based on the 30-cluster methodology of the EPI used worldwide (WHO 2005). The study was organized by the Ministry of Health under the supervision and with the support of WHO and UNICEF. (The national coordinator was from WHO, and his assistant coordinator was provided by the Ministry of Health.) Thirty clusters of seven or eight children were selected in each of the 52 health districts. Confidence intervals of 95% have an average width of ±6% around the point estimate of coverage. Children 12–23 months old who had lived in the sampled cluster for at least 3 months were targeted, resulting in a sample size of 11 353. The children’s vaccination status was recorded either from the immunization card or the mother’s report. For our study, we adopted a conservative approach and derived immunization information from immunization cards rather than from the mother’s declaration, because the validity of the latter approach is considerably more doubtful and susceptible to potentially significant recall biases (Suarez et al. 1997; Murray et al. 2003; Bishai 2008; Burton et al. 2009).

2003 DHS estimates of coverage

Conducted between June and November 2003, the DHS provides representative estimates for urban/rural areas as well as for the capital and each administrative region of the country. The sample was selected using a stratified, two-stage cluster design. Each region was divided into urban and rural areas to make up the strata. At the first stage, 400 clusters were selected, 90 in urban and 310 in rural areas. At the second stage, 26 and 23 households were selected in each urban and rural cluster, respectively (Institut National de la Statistique et de la Démographie 2004). Households and, within them, all resident or visiting women aged 15–49 years and their living children under age 5 years were targeted. Children in the 12–23 month age bracket made up a sample of 1840. Coverage information was obtained from maternal recall and immunization cards. Again, only vaccination status as per the immunization cards was considered.

Comparability of estimates

Child cohorts targeted by the two population surveys and by administrative data are not fully overlapping because, on one hand, the surveys are not conducted concomitantly and, on the other, the age groups targeted by surveys and administrative data differ (Figure 1). The DHS was carried out between June and November 2003 and thus covers children who, depending on the survey areas, were born between August 2001 and November 2002. The ECS was conducted between May and June 2003 and thus covers children born between July 2001 and July 2002. The two surveys therefore do not overlap perfectly, but we can reasonably assume they concern cohorts of children that are relatively similar. Administrative data provided in 2003 relate to the vaccinal status of children born in 2002, and the data provided in 2002 relate to children born in 2001. To minimize differences and enhance comparability across child cohorts, we compare immunization estimates from the 2003 ECS and the 2003 DHS with the mean of administrative rates throughout 2002 and 2003.

Figure 1

Data collection periods and targeted children cohorts, by source of data: administrative, DHS and ECS

Variables potentially associated with the quality of administrative reporting

District components that may modify the quality of administrative reporting were selected according to the literature and evidence provided by studies conducted in Burkina Faso (Bicaba et al. 2009; Haddad et al. 2009). We hypothesized that reporting systems would perform better in urban areas (Ouagadougou and Bobo-Dioulasso) and semi-urban areas (11 regional capitals) than in rural areas (rest of the country, including border zones and remote areas); and that they would vary with the district population density (more errors in estimating the denominators of coverage in remote, sparsely populated areas) and health resources as indicated by the physician/patients, nurse/patients and immunization points/population ratios. These variables are available at the district level in the national health information system.

Comparison of immunization estimates across data sources and territorial units

First, we compared DTP3 and measles coverage from administrative sources with estimates from both the ECS and the DHS at both national and regional levels. Then we compared ECS immunization estimates with administrative rates across the 52 districts. In each district, the proportion of children vaccinated against DTP3 and measles according to administrative data was compared with the proportion estimated by the 2003 ECS.

At the district level, agreement between administrative and ECS coverage rates was calculated as follows. If the proportion of coverage according to administrative data exceeded the 95% CI around the ECS estimate, the district was considered to be overestimating. If, on the other hand, the proportion of coverage estimated by administrative data was lower than the 95% CI around the ECS estimate, the district was considered to be underestimating. Therefore, health districts were classified into three groups: those where administrative estimates were found to be underestimating (<95% CI of ECS rate), overestimating (>95% CI of ECS rate) or concordant with the ECS rates (differences within 95% CI of ECS rate). Bivariate tests for associations of district characteristics with over- and underestimating (chi-squared statistics) were carried out using SPSS, release 13.0 (SPSS Inc. 2006).

Results

According to the ECS, coverage varies appreciably between districts within a given region. There are >54 percentage points of difference between the worst and the best performing districts in the country (33% versus 93% for DTP3 immunization rate and 34% versus 88% for measles coverage) and an average gap of 24 and 25 percentage points within each region for DTP3 and measles coverage, respectively (Figure 2).

Figure 2

Coverage gap between the worst and the best performing districts in each region (EPI cluster survey 2003)

Thus, when comparing regional averages, this substantial inter-region variation must be kept in mind. There are at least 30 percentage points of difference between the region showing the highest DTP3 coverage and that with the lowest, regardless of the data source. For measles, the gap ranges from 20 to 40 percentage points depending on the data source (Table 1). Table 1 and Figure 3 also show that DHS estimates are consistently more conservative than those of the ECS across regions and antigens.

Table 1

Administrative and ECS estimates (% and CI), by antigen and district

Antigen	DTP3 coverage							Measles coverage
Source	Adm. (2002)	Adm. (2003)	Adm. (2002–03)	EPI Cluster Survey (ECS, 2003)				Adm. (2002)	Adm. (2003)	Adm. (2002–03)	EPI Cluster Survey (ECS, 2003)
DISTRICT			(1)	(2)			(1) − (2)			(1)	(2)			(1) − (2)
			Average	Average	CI inf	CI sup	Sign of diff*			Average	Average	CI inf	CI sup	Sign of diff*
PISSY	57.02	64.75	60.9	83.0	77.3	87.8	–	51.88	56.18	54.0	77.4	71.1	82.8	–
KOSSODO	47.2	70.72	59.0	77.8	71.9	83.4	–	38.78	51.66	45.2	68.4	61.5	74.4	–
SECT.30	69.7	75.89	72.8	79.0	72.9	84.2	–	57.02	62.49	59.8	69.2	92.5	75.3	–
PAUL VI	79.2	74.64	76.9	83.2	77.2	87.6	–	68.3	64.99	66.6	75.7	69.7	81.5	–
DEDOUGOU	61.2	78.6	69.9	57.8	51.4	64.8	+	67.05	73.34	70.2	61.7	55.0	68.3	+
BOROMO	77.9	79.73	78.8	73.7	67.5	79.7	0	69.71	68.57	69.1	71.4	65.1	77.5	0
NOUNA	58.3	66.92	62.6	52.1	45.2	59.0	+	64.13	70.25	67.2	57.8	50.8	64.6	+
SOLENZO	94.5	100.09	97.3	84.5	79.5	89.3	+	76.6	86.49	81.5	83.2	77.6	87.9	0
TOUGAN	76.5	80.13	78.3	74.1	68.3	79.1	0	71.77	71.59	71.7	39.4	33.5	45.5	+
TOMA	90.2	92.73	91.5	69.0	62.3	75.2	+	98.09	90.63	94.4	66.2	59.4	72.6	+
MANGA	93.0	86.11	89.6	92.9	88.5	96.0	0	81.4	75.7	78.6	87.8	82.5	91.8	–
PO	71.3	58.72	65.0	84.2	78.5	88.6	–	65.24	50.35	57.8	84.2	78.5	88.6	–
SAPONE	80.8	73.09	76.9	77.7	72.4	82.8	0	77.68	63.41	70.5	72.0	66.0	77.2	0
KOMBISSIRI	52.8	76.08	64.4	81.6	76.3	87.0	–	46.58	57.4	52.0	57.1	50.3	64.0	0
KOUPELA	79.4	88.37	83.9	83.6	78.5	88.7	0	67.31	74.71	71.0	80.8	74.9	85.9	–
TENKODOGO	63.9	90.9	77.4	71.6	65.6	77.9	0	65.6	80.3	73.0	66.1	59.3	72.3	+
ZABRE	72.3	90.86	81.6	72.6	66.6	78.8	+	76.67	85.53	81.1	72.1	65.6	78.0	+
OUARGAYE	56.3	80.91	68.6	56.5	50.2	63.5	+	52.52	77.06	64.8	60.1	53.3	66.6	0
KOUDOUGOU	58.6	66.01	62.3	62.5	56.0	69.4	0	57.7	60.99	59.3	56.7	49.9	63.6	0
NANORO	58.5	83.38	70.9	79.5	74.2	85.3	–	55.74	77.76	66.8	75.3	68.8	80.7	–
REO	51.1	67.62	59.4	67.0	60.3	73.2	–	51.17	56.8	54.0	61.7	55.0	68.4	–
LEO	79.75	82.82	81.3	62.5	56.4	69.3	+	71.77	75.15	73.5	42.8	36.2	49.4	+
KAYA	77.7	80.3	79.0	91.5	87.5	95.3	–	73.59	75.75	74.7	88.7	83.6	92.6	–
BARSALOGO	79.3	90.78	85.0	87.7	83.1	92.2	0	80.57	86.34	83.5	79.2	73.2	84.5	0
BOULSA	72.09	88.08	80.1	82.9	77.3	87.7	0	61.46	80.23	70.8	75.9	69.8	81.5	0
KONGOUSSI	92.8	108.85	100.8	88.8	83.8	92.7	+	79.45	99.57	89.5	82.8	77.1	87.6	+
BANFORA	59.1	77.3	68.2	62.0	55.0	68.4	0	55.53	70.13	62.8	63.4	56.4	69.7	0
SINDOU	95.6	87.4	91.5	91.5	87.5	95.3	0	84.58	81.49	83.0	84.0	78.4	88.7	–
SECT.22 B	67.8	67.89	67.85	84.1	78.4	88.6	–	69.34	61.54	65.4	73.4	67.0	79.1	–
SECT.15 B	58.6	75.0	66.80	72.6	66.7	78.8	0	54.4	67.0	60.7	73.0	66.7	78.8	–
DANDE	48.6	72.19	60.40	68.9	62.3	75.1	–	51.69	64.85	58.3	66.5	59.9	72.9	–
HOUNDE	71.2	74.56	72.88	69.5	63.3	75.9	0	68.06	68.54	68.3	63.8	57.1	70.3	0
ORODARA	50.6	75.32	62.96	55.9	49.2	62.9	+	53.33	81.16	67.2	3.9	46.8	60.6	+
DORI	77.9	90.79	84.3	50.7	44.2	58.1	+	69.64	85.21	77.4	45.1	38.3	52.0	+
DJIBO	47.9	50.48	49.2	33.3	26.9	39.9	+	48.52	56.67	52.6	34.7	28.2	41.4	+
SEBBA	80.7	71.24	76.0	56.9	49.9	63.6	+	64.22	78.85	71.5	59.2	52.2	65.8	+
GOROM	63.9	67.76	65.8	54.9	48.3	61.2	+	63.87	64.95	64.4	51.2	44.5	57.5	+
GAOUA	79.5	102.99	91.2	52.4	44.9	58.8	+	65.64	86.59	76.1	54.7	47.7	61.5	+
DIEBOUGOU	85.2	116.44	100.8	77.1	70.6	82.4	+	74.72	106.94	90.8	72.4	65.6	78.1	+
DANO	93.7	92.69	93.2	75.7	69.9	81.7	+	72.82	81.77	77.3	60.0	52.9	66.4	+
BATIE	101.4	104.96	103.2	37.3	30.7	44.0	+	79.72	79.17	79.4	34.4	28.1	41.2	+
ZORGHO	49.9	60.67	55.3	51.6	44.9	58.7	0	42.57	46.03	44.3	42.7	35.8	49.4	0
BOUSSE	82.8	79.58	81.2	77.7	72.0	83.4	0	66.24	71.05	68.6	71.6	65.2	77.5	0
ZINIARE	70.5	79.84	75.2	67.8	61.6	74.3	+	62.92	75.9	69.4	55.9	48.9	62.5	+
FADA	68.0	62.25	66.6	45.6	39.1	52.7	+	79.33	74.2	76.8	54.9	48.2	61.8	+
DIAPAGA	94.2	101.98	98.1	78.5	72.6	84.1	+	82.66	85.27	84.0	75.1	68.6	80.7	+
PAMA	66.4	103.76	85.1	54.2	47.0	60.7	+	76.38	102.04	89.2	53.7	47.0	60.7	+
BOGANDE	66.6	78.35	72.5	82.5	77.3	87.7	–	63.98	75.25	69.6	78.8	72.8	84.0	–
OUAHIGOUYA	71.6	68.23	69.9	73.0	67.1	78.6	0	68.94	59.77	64.4	72.0	65.8	77.5	–
SEGUENEGA	78.7	80.31	79.5	83.9	78.6	88.7	0	72.51	76.46	74.5	83.4	77.6	87.9	–
YAKO	59.8	69.46	64.6	75.0	68.8	80.7	–	48.61	65.62	57.1	58.8	52.1	65.6	0
TITAO	73.5	81.68	77.6	61.3	53.9	67.5	+	74.78	67.6	71.2	63.7	56.8	70.1	+

– = administrative estimate inferior to the lower limit of the 95% CI of the ECS estimate.

+ = administrative estimate superior to the upper limit of the 95% CI of the ECS estimate.

0 = administrative estimate within the 95% CI of the ECS estimate.

Figure 3

Difference in estimates provided by ECS and DHS by region and antigen

When regional administrative data are compared against ECS and DHS data, administrative figures show large discrepancies. Differences between administrative and survey figures can be as much as 40 and 30 percentage points for DTP3 and measles coverage, respectively (Figures 4 and 5).

Figure 4

Percentage gap between administrative rates (mean 2002–03) and survey rates—DTP3

Figure 5

Percentage gap between administrative rates (mean 2002–03) and survey rates—measles

Substantial heterogeneity is also found in the direction and size of discrepancies between administrative and ECS district coverage rates (Table 1). Administrative figures for DTP3 and measles coverage exceed survey figures in 22 districts. Administrative estimates of DTP3 and measles coverage are lower than corresponding ECS proportions in 13 and 16 districts, respectively. The remaining health districts show concordant estimates of coverage.

Overestimation occurs in remote and border areas, with 82% and 76% of border districts overestimating DTP3 and measles coverage, respectively. Underestimation is observed in districts located around the two major urban centres of Ouagadougou and Bobo Dioulasso and other predominantly urban areas (Figures 6 and 7).

Figure 6

Level of concordance of estimates: DTP3 coverage rate according to administrative data (mean 2002–03) and to the EPI survey data (2003)

Figure 7

Level of concordance of estimates: measles coverage rate according to administrative data (mean 2002–03) and to the EPI survey data (2003)

Overestimation and underestimation are independent of the specific antigen under consideration. Areas overestimating DTP3 coverage tend to coincide with areas overestimating measles coverage. The same correspondence is also seen for areas that tend to underestimate (Figures 5 and 6). This suggests that these errors are inherent to the reporting system.

Among the district variables, only the type of geographical area is statistically associated with over- and underestimating (χ2 = 29, df = 6 and P = 0.000 for DTP3; χ2 = 29.3, df = 6 and P = 0.000 for measles). The ratio of physicians or nurses to population, the ratio of immunization points or services to population, and population density show no statistical relation with over- and underestimating.

Discussion

This study takes advantage of the unique opportunity offered by the availability, in a given year, of immunization coverage rates from three distinct sources: a standard EPI cluster survey, a DHS and routine administrative information systems. Gaps between administrative and survey coverage estimates at the national and regional levels show wide variations. The two surveys themselves provide differing estimates across regions, with DHS rates consistently more conservative than those of the ECS. Case definition and the method of obtaining data may differ between surveys, and other differences in survey methodology may also result in divergent estimates for the same population (Boerma et al. 1990).

For methodological and practical reasons, we considered the ECS to be the reference database and information source for producing standard coverage estimates against which to assess the validity of administrative data. First, the ECS design and methodology have been specifically developed for the purpose of estimating immunization coverage, whereas the DHS is more focused on demographic events and women’s health. In fact, some studies suggest the DHS might produce biased estimates of certain child health outcomes (Manesh et al. 2008). Second, when it comes to assessing children’s immunization coverage, the DHS sample is much smaller than that of the ECS. DHS sampling is expressly designed to gather information on households; children are not specifically targeted, as they are in the ECS. Third, and this is a key practical consideration, only the ECS provides reference estimates of immunization coverage by district, which we required to meet our objectives; the DHS provides estimates at the regional level only. Thus, even if ECSs are not always carried out as meticulously as might be hoped, we believe—contrary to what Lim et al. (2008) suggest—that the DHS cannot be considered the reference method for estimating national coverage, and even less for regional coverage. Ultimately, however, what is important is that the variations between the DHS and the ECS were relatively consistent, such that they had no tangible impact on either the conclusions drawn from the inter-district comparisons or the identification of those regions showing the widest gaps between values from the administrative databases and from the surveys.

Comparing administrative-based with survey-based coverage estimates in the 52 health districts reveals variations in the validity of administrative estimates across the country, with deviations from survey rates varying in extent and significance between different geographical areas. National coverage estimates from administrative data and the ECS are quite similar (74% and 71% for DTP3 and 68% and 66% for measles, respectively). However, this masks huge regional differences; gaps between regional administrative rates and ECS estimates can range from above 30% to below –15%. Administrative data tend to overestimate immunization coverage in border areas and underestimate it in areas near major urban centres, a geographical pattern that is also consistent across antigens.

To assess which factors might account for this geographical variation in administrative data quality, we turn to the literature for indications on what factors might affect the validity of the two terms of the ratio. Table 2 illustrates numerator and denominator biases that can result in underestimating or overestimating actual coverage. Several studies have shown that numerators or denominators can be inflated or deflated because of errors in recording, storing or reporting information (Freund and Kalumba 1985; Kumar 1993; Onta et al. 1998; Mavimbe et al. 2006). Such errors may be related to a lack of training and supervision (Freund and Kalumba 1985; Kumar 1993), but erroneous reporting can also be intentional, which is most likely when immunization targets are used as indicators of performance or are linked to incentives (Onta et al. 1998; Mavimbe et al. 2006). According to Lim et al. (2008), the incentives created by GAVI’s funding model, which ties the allocation of an organization’s resources to the performance of national immunization programmes, have led to pronounced overestimation of progress in countries’ official reports and in WHO and UNICEF estimates. Other studies address how the mismatch between catchment and attraction areas produces unreliable coverage estimates (Borgdorff et al. 1988; Zuber et al. 2003). Because the numerator for a given area records immunizations of children who have consumed a service located in that area, and the denominator estimates the number of children served, in theory, by that area’s services on grounds of residency, the validity of coverage indicators depends on the extent to which the attraction and catchment areas overlap.

Table 2

How numerator and denominator biases might be reflected in under- or overestimated ratios

	Resulting ratio
Terms of ratio	Underestimated	Overestimated
Numerator	Targeted users seek service outside the catchment area	Users coming from outside the catchment area1
	Lack of supervision2–3	Intentional inflation of figures in response to the pressure to achieve fixed targets4–5
	Inadequate incentives for health workers2–3	Intentional inflation of figures related to financial or non-monetary incentives for health workers or supervisors6
	Lack of procedures for handling late reports7	Doses administered to children aged ≥12 months are recorded as doses administered to infants
		The number of doses distributed are counted as the number of doses administered, ignoring wastage that occurs7
Denominator	Migration of targeted users to other catchment areas8	Under-enumeration of very young children in census1–8
	Decreased fertility inside the catchment area8	Immigration into the catchment area8
		Increased fertility inside the catchment area8

Notes: 1: Borgdorff et al. 1988; 2: Freund et al. 1985; 3: Kumar 1993; 4: Onta et al. 1998; 5: Mavimbe et al. 2006; 6: Murray et al. 2003; 7: Ronveaux et al. 2005; 8: Zuber et al. 2003.

These circumstances should be considered in assessing the sources of observed geographical variation. We examined two specific factors more closely for their role in biasing administrative coverage proportions: migration and fertility. First, the existence of nomadic populations along the borders of Mali and Niger and in the Sahel region is likely to be a source of inflation of the estimates, by contributing to the numerator of the ratio while not being accounted for in the denominator. Conversely, underestimation in areas around and outside the capital, Ouagadougou, and the second major town, Bobo Dioulasso, is probably explained by the outflow of target users to these two better served and resourced urban centres. These users might go to the towns for better access to vaccination services or to seek other treatment. In the latter case, health workers might decide also to provide vaccination, in line with recommendations for not missing opportunities, illustrated elsewhere (Borgdorff et al. 1988).

Second, the denominator is sensitive to the estimation of fertility, such that variations in fertility across the country also affect its precision. In Burkina Faso, the health information system estimates the target population of children aged 0–11 months to be 4.21% of the total population based upon the 1996 General Population and Housing Census. The target is updated annually using a 2.6% growth factor calculated on the basis of fertility and infant mortality rates estimated in the 1996 census (Ministère de la Santé du Burkina Faso 2003). The same rate is applied in all districts, so denominators are based on a mean value that does not reflect locally observed demographic variations arising from migration flow or particular changes in fertility or infant mortality (Zuber et al. 2003). Internal and cross-border migration have different effects on district population figures. All else being equal, the denominator is more likely to be inflated (lowering the ratio) in districts with below-average fertility rates and deflated (raising the ratio) in districts with above-average fertility rates. These errors would eventually diminish with the implementation of more effective free-of-charge birth registers in every municipality. Where reliable estimates of denominators and/or of fertility levels are not possible, some suggest not using denominators in assessing the performance of vaccination programmes and, instead, using indicators to ascertain coverage such as the number of DTP3 vaccinations or the numerical difference between infant BCG and infant DPT3. As useful as they may be, these alternatives remain stop-gap measures, inferior alternatives whose informative value cannot compare with good estimators of coverage that allow us to assess the extent to which the needs of target populations have been met.

Discrepancies in the accuracy of reports on numbers of immunized children might also explain the inter-district variation in the quality of administrative estimates. Errors may be due to lack of training of health personnel responsible for data management at both the health post and the district level (Loevinsohn 1994). While our data do not enable us to address this concern, the DQA has provided some indications on this issue. Data verifiability has significantly improved between 2002 and 2005, with a verification factor rate rising from 0.57, below the recommended GAVI rate, to 0.97 (WHO 2003; Gaudin and Yazbeck 2006; Bosch-Capblanch et al. 2009). We might therefore speculate that such improvement in the reporting system would be reflected in more reliable numerators.

Conclusions and recommendations

With decentralization, responsibility for planning and monitoring is increasingly being transferred to health districts. Therefore, this is the level at which critical information, such as accurate district-based estimates of immunization coverage, is urgently needed. Even in a small country such as Burkina Faso, we observe considerable inter-district heterogeneity in immunization coverage, confirming the limitations of nationally aggregated indicators for planning and monitoring immunization activities (Balraj et al. 1993; Gaudin and Yazbeck 2006).

In most areas of the country, health district authorities can use administrative data with a certain level of confidence; such data are accurate in about one-third of the districts. If overestimation is considered the least favourable outcome and underestimation is less of a threat for effective planning, then administrative data can be considered to provide adequate estimates in two-thirds of the districts—obviously an encouraging result. However, information appears to be inappropriate in one-third of the districts, which together make up more than half of the country’s population. Therefore, for decentralization to succeed, more must be done to improve the quality of the information needed for proper monitoring and planning. The ‘Reach Every District Strategy’ promoted by WHO could be an effective means to reach that objective (Vandelaer et al. 2008). Audits can also play a formative role in the districts where they are carried out, but can only really have a significant effect on information if they are part of a larger process that extends beyond those districts to include the whole territory. So there is a need for a national strategy to support districts in strengthening administrative data systems.

Administrative rates should be considered with great caution in areas affected by migration and mobility across catchment areas and in areas where fertility levels differ significantly from the country average. In these areas, an inaccurate denominator can markedly affect the validity of immunization coverage estimates. The use of denominator-free indicators focusing on production does not capture the programme’s ability to meet the needs of the target population, and is thus only a stop-gap measure. Thus, efforts must be made to improve the validity not only of numerators, but also of denominators. More effective free-of-charge birth registers and local initiatives aimed at improving the estimation of the target population should be encouraged.

However, where there is significant population mobility, improving the accuracy of denominators will not be enough. Urbanization, improved communication systems in rural areas and access to better information have increased population mobility and reduced people’s dependence on the service providers to whom they theoretically ‘belong’. The population of users tends increasingly to differ from the reference target populations, and the usual indicators of coverage, particularly in urban and border zones, are largely biased (Bicaba et al. 2009). This explains why indicators of coverage in these areas commonly exceed 100%. Health authorities and international agencies need to adapt their measurement tools so that they can produce indicators based on counts that distinguish between users from within and outside the health facilities’ catchment areas.

The validity of administrative data varies considerably and is sensitive to various types of incentive. It should be possible to verify, from time to time, the validity of the indicators produced by coverage surveys, as well as the potential impact of any actions aimed at improving practices in health information systems. As emphasized by Lim et al., administrative data will never be a perfect substitute for, nor a promising alternative to, data from independent and incentive-free surveys; so there is a strong need for ‘empirically-based replicable assessments of trends in immunization coverage’ (Lim et al. 2008, p. 2043). Moreover, only coverage surveys make it possible to eliminate distortions introduced by denominator mis-estimation and the mobility of the service user population. Therefore, we recommend integrating coverage surveys into monitoring processes at the district level.

In times of decentralization, improving health information systems is critical to help districts plan, manage and properly monitor their immunization activities. We have argued that such an improvement requires the strengthening and expansion of administrative data systems, implementing and using indicators that are insensitive to population mobility, and integrating surveys into monitoring. However, these measures are essentially technical and could eventually prove inadequate if they are not part of broader processes aimed at promoting information utilization at the level of the districts themselves. Teams still see themselves too often as simply collectors of information for the higher levels, and not as actors who are the rightful receivers and users of this information (Haddad et al. 2009). Until such time as the discussion about decentralized management translates into localized and participative management practices, efforts to strengthen the adequacy and quality of information systems are likely, at best, to continue to run up against the indifference and passivity of the field teams, who constitute the first link in the chain of information production.

Funding

As well, we wish to thank the reviewers for their helpful comments. Funding was provided by the International Development Research Centre (IDRC), grant number 102172–009, as part of the Canadian International Immunization Initiative Phase II. We also thank Donna Riley for translation and editing support.

Conflict of interest

The authors declare they have no conflicts of interest.