Evaluating the experience of GAPS--a methodology for improving quality of mass immunization campaigns in developing countries.

The existence of pockets of under-vaccinated persons has allowed outbreaks of disease in countries that have achieved high levels of vaccination coverage. A field-based methodology--GAPS (Geographic Assessment of Planning and Services)--was developed to predict, in advance of an immunization campaign, the sites of which are most likely to have a pocket of unvaccinated persons and then use this information to improve planning, supervision, and evaluation of the campaign. At this time, there have been two applications of GAPS (Nepal and Ethiopia). The purpose of this paper was to evaluate these two applications of GAPS and make recommendations regarding its future use. Structured, expert interviews were conducted with at least three campaign organizers to evaluate each application of GAPS using purposive sampling. An evaluation of an individual campaign was considered positive when at least two of the three campaign organizers considered GAPS to be useful and worthwhile. The three campaign organizers interviewed following the GAPS application in Ethiopia responded that GAPS was useful and worth the effort. In Nepal, all four campaign organizers responded that GAPS was useful and worth the effort. Some suggestions for improvement were also identified. Although this evaluation was limited in the number of applications evaluated, GAPS appears to have promise as a practical method to help improve the quality of mass immunization campaigns. And even if no pockets of unvaccinated persons are found, the method may serve as a rapid quality-check of administrative estimates of coverage. Further applications in different settings are needed to confirm these findings or under what circumstances GAPS might best be used. GAPS may also be considered for improving other types of health campaigns, such as distribution of insecticide-treated bednets, vitamin A capsules, and deworming medications.

INTRODUCTION

Achieving a high percentage of vaccination coverage, while necessary, may not be sufficient for the elimination or eradication of disease. The existence of pockets of under-vaccinated persons has allowed outbreaks of disease in countries that have achieved high levels of vaccination coverage (1–7). For this reason, information about coverage levels in the general population is not sufficient to determine if a population is at risk of importation or continued transmission of disease. Information about whether or not pockets of under-vaccinated persons exist would also help understand the risks of disease transmission.

Weiss et al. identified factors that predict whether or not a child will receive vaccine during a mass campaign (8). These factors can help us predict and/or identify the existence of pockets of susceptible children. For example, if one or more predictive factors are geographically clustered, the population living in this geographic cluster is at risk of having a pocket of unvaccinated persons. We can use this information before, during, and after the campaign to help us take steps to prevent and/or address the development of a pocket of unvaccinated persons.

A way to predict likely geographic clusters of unvaccinated persons may be useful when planning, supervising, and evaluating mass campaigns (8). With this information, priority can be given to assess the quality of planning for such sites. Priority can also be given to supervision and post-campaign evaluation of these sites. If indicated, steps can be taken to prevent problems that may lead to geographic clusters of unvaccinated, or to address any problems identified. This paper describes an evaluation of the experience to date with one such field-based methodology, which we call GAPS (Geographic Assessment of Planning and Services) in two settings (Ethiopia and Nepal). Before presenting the evaluation, a brief description of GAPS is provided below [The GAPS manual with the questionnaire and interview guides and instructions, and more detailed reports of the two applications of GAPS are available from the authors].

Description of GAPS

We carry out GAPS before, during, and immediately after a mass vaccination campaign. Following this pattern, GAPS has three phases: (a) improve planning before the campaign begins, (b) improve implementation/monitoring of the campaign, and (c) carry out the post-campaign evaluation and follow-up. We describe below the action-steps we take in each phase.

GAPS Phase 1: Improve planning before the campaign begins

As a first step during Phase 1, we identify a geographic area—usually a district or province that will participate in a mass vaccination campaign. The second step is to identify sites within that area that warrant special attention through expert interviews. The interview format is semi-structured that follows a guide available in the manual. During the interview, local experts are asked if certain factors—those identified during the past studies as factors that predict missed vaccination in mass immunization campaigns (8)—are clustered in any sites within the area. Sites mentioned several times by each expert during the interview are selected as geographic areas that warrant special attention (a form of triangulation). The third step is to review campaign plans for the sites selected for special attention. The fourth and final step is to address any gaps identified in the campaign plan for sites we have designated as high-priority. In our actions, we are limited to solutions that can be carried out with available resources; these resources may come from government, UN agencies, health programmes of non-governmental organizations (NGOs), or community-based organizations. For example, if there is a gap in social mobilization activities in a site, an NGO might conduct a community meeting or other health-promotion activities, such as a skit, rally, or parade. By these actions, before the campaign begins, we may prevent problems in the campaign.

GAPS Phase 2: Improve implementation and monitoring of the campaign

During this phase of GAPS, the first step is to decide—from information gathered during Phase I—which sites will be given priority attention for supervision during the campaign, given available resources for supervision (time, vehicles, and number of persons). For example, if available resources only allow supervision-visits to five sites, we decide what we believe to be the first five sites we should visit in order of priority. The second step is to identify if there are any problems in implementing the campaign that may result in a site having a pocket of unvaccinated persons. Using a supervision checklist (usually one that is in use nationally, although an example is provided in the manual), we observe immunizations provided at any fixed posts and by house-to-house vaccination teams within the sites we supervise. During supervision, we observe the quality of vaccine, equipment, and supplies and assess knowledge and practices of vaccinators using a convenience sample. We also visit homes in a site using a convenience sample to check the vaccination status of persons in these homes and inquire about reasons for any missed vaccination. A convenience sample is used for allowing more vaccination sites and homes to be visi-ted and allowing for more flexibility in follow-up of problems identified than a more structured sampling method would allow. The third and final step of GAPS Phase 2 is participation in quality-improvement activities, as indicated by findings of supervision, to prevent or address problems before the end of the campaign. We present our findings to organizers of the campaign (e.g. district health officer, district immunization programme manager, immunization officers of World Health Organization [WHO] or United Nations Children's Fund [UNICEF]) and then participate with them in follow-up actions depending on available resources.

GAPS Phase 3: Carry out the post-campaign evaluation and follow up

The first step in GAPS Phase 3 is to decide what sites will be visited during the post-campaign evaluation. This decision is based partly on observations of the campaign by GAPS participants and campaign officials, partly on discussions with campaign organizers prior to the campaign, and partly on resources (manpower, time, money) available. Therefore, the list of sites we decide to visit during Phase 3 may be a mix of sites; some that we suspected even before the campaign as likely to have problems, and some where problems were identified only during observations of the campaign. The second step is to design a hypothesis test of the post-campaign evaluation. The hypothesis is that the campaign coverage in a site to be evaluated is unacceptable because the site has a pocket of unvaccinated persons (defined as 20% or more of the target population unvaccinated in that site). The null hypothesis is that the campaign coverage in the site is acceptable because the site does not have a pocket of unvaccinated children. We used lot quality assurance sampling (LQAS) techniques to classify whether or not the campaign coverage in a site is acceptable or not (9,10). In contrast to more typical uses of LQAS to assess the vaccination coverage in a larger geographic area, LQAS is used here for testing a hypothesis that a site has a cluster of unvaccinated children (11). The following LQAS design was used in each application evaluated in this paper (9–12): (a) Target value of 80% with a specificity greater than 90% (this also is the upper threshold); (b) Lower threshold of 50%, with a sensitivity greater than 90%; (c) Random sample size of 19 households, with one or more children aged less than five years; and (d) Decision rule of 13.

The expected coverage of a mass immunization campaign for polio is close to 100% (13–19). Given this, we recommend a target value of 80%. We considered 80% sufficiently lower than the typical coverage achievement as to warrant special action and resources to improve the coverage in any site found to be below the target. The unit of analysis was a lot of 19 randomly-selected households, with a child aged less than five years in each site. If less than 13 of the 19 households have all children aged less than five years vaccinated, the coverage is classified as unacceptable, the null hypothesis is rejected, and the evaluation site is judged as having a pocket of unvaccinated persons. Conversely, if 13 or more households have all children aged less than five years vaccinated, the coverage is classified as acceptable, the null hypothesis is not rejected, and the evaluation site is considered NOT to have a pocket of unvaccinated children.

The third step in GAPS Phase 3 is to carry out the hypothesis testing in sites selected. We select each household at random according to the process des-cribed by Valadez et al. (10). In each household selected, we interviewed caretakers of children aged less than five years. During the interview, we identified the vaccination status of each child aged less than five years in the household. We also identified any self-reported reasons for why a child was missed being vaccinated, if indicated. The final step of GAPS Phase 3 is to participate in quality-improvement activities as indicated by the findings of the post-campaign evaluation. We present the evaluation information—about successes and problems and main causes of problems—to organizers of the campaign (e.g. district health officer, district immunization programme manager, WHO or UNICEF immunization officers). We then participate in discussions with them about what can be done—with available or obtainable resources—to correct any deficiencies identified (and who would be responsible for these corrective actions). Following these discussions, we participate in qualityimprovement activities to address problems before the next campaign as indicated by the discussions mentioned above.

GAPS is one of several rapid assessment methods that have been developed to identify clustering or pockets of health problems in a quick, simple and inexpensive way. Rapid assessments have been developed to assess elimination of neonatal tetanus, the pattern and severity of trachoma, the endemicity of onchocerciasis, and other health problems (20–33). All these methods, including GAPS, include some method for deciding with sites that are at an increased risk of having a health problem and, therefore, deserve priority attention for study (and action if a problem is confirmed). GAPS is distinct from these because it seeks to prevent health problems, not simply to address them once identified. GAPS typically uses smaller sample sizes than these other rapid assessment methods. Alternative methods of assessment typically include prevalence surveys that are costly and require more time.

GAPS experience to date

At this time, there have been two complete applications of GAPS to date: (a) in one woreda (district) in the Southern Nations Region of Ethiopia and (b) in one terai district in Nepal that shares a border with India [Detailed reports of these applications are available from the authors]. Both the applications of GAPS in this paper were carried out at the district level for improving the quality of mass polio-vaccination campaigns, although GAPS can be used for other types of campaigns (vaccine, vitamin A, bednets). In both the applications, teams comprising district-level representatives from governmental and non-governmental organizations worked together to carry out the GAPS process. These representatives easily learned and applied the GAPS methodology within their own worksettings to improve an activity they were responsible for organizing and/or supporting. In both the settings, we observed some procedural problems, with little indication that more than just a few children were being missed and no indication of pockets of unvaccinated children. These were our findings, although we assessed sites considered by local experts (government and NGOs) to be the most likely sites in our districts to have problems. Mass immunization campaigns that achieve a high coverage and do not leave pockets of unvaccinated persons are important for interrupting the transmission of disease. This is probably one explanation for why Ethiopia and Nepal have had a few reported wild polio cases since 2000/2001, although cases have been reported along the border of these countries in subsequent years.

Since the GAPS process is flexible and since the process can be adapted to match the availability of local resources, there were differences between the two applications reported here. In the Nepal application, the investigator applied GAPS in a district where an NGO was receiving project funds from the United States Agency for International Deve-lopment (USAID), Washington, DC to support the Government's polio- eradication efforts (mass campaigns, surveillance, etc.). In Nepal, the NGO had dedicated staff for this effort at the district level and resources for a vehicle and had hired independent monitors (in addition to the investigator). In Ethio-pia, GAPS was carried out in a district where no NGO was receiving project funds to support polio-eradication efforts. In this zone, NGOs did support the government polio-eradication efforts but not with specific project funds. The campaign in Nepal was implemented on two days (Saturday and Sunday) whereas the campaign in Ethiopia was carried out over seven days due to terrain and the layout of households within communities. In Nepal, district health staff involved in the GAPS application had access to vehicles whereas, in Ethiopia, the district health staff relied on the investigators’ vehicle to participate. The implication of these differences was that more resources were available on a daily basis in Nepal compared to Ethiopia, and requirements of resources were higher in Ethiopia. Typically—during the campaign and post-campaign evaluation—we were able to visit many sites each day in Nepal but only one site per day in Ethiopia. It was also more difficult for staff to participate in the GAPS application in Ethiopia compared to Nepal because the campaign lasted for five days more than the Nepal campaign. More persons participated in Nepal—some coming from government and UN offices in Kathmandu—because it was easier to find the time to participate. One effect of the differences mentioned above is that we were only able to carry out the post-campaign evaluation in one site in Ethiopia.

Following these two applications of GAPS, several questions remained to be answered, among others. First, does GAPS accomplish the objectives it was designed to produce? Second, are there any suggestions for the improvement of the GAPS metho-dology? This paper describes an evaluation of the current experiences with GAPS to date to answer these two questions. In the following section, we describe the evaluation methodology. Then, the results and conclusions of the evaluation—about GAPS—are provided.

MATERIALS AND METHODS

The evaluation of GAPS described below assessed how GAPS was used and thought about by persons who were involved in organizing or evaluating the two mass polio campaigns at the national and local levels (herein called campaign organizers). The long-term utility of GAPS in any site will be determined by campaign organizers’ beliefs about the method and how these persons use the information generated by GAPS. Specifically, we purposively sampled government, NGO and UN representatives to participate in the evaluation to provide both a district/operational-level perspective and a national-level perspective. At the district level, the district EPI manager or his/her assistant is typically responsible for evaluating the campaign. In addition, surveillance officers of WHO or programme officers of NGOs working in the district may be supporting the organization and evaluation of the campaign. We consider these persons to be campaign organizers at the local level. At the national level, there is usually someone working in the immunization programme or child health programme of the Ministry of Health, who is res-ponsible for organizing and analyzing routine evaluation data about the campaign. Alternatively, there are often one or more persons at the WHO or UNICEF, who help the Government organize and evaluate national immunization campaigns. We consider these persons to be campaign organizers at the national level.

The evaluation of GAPS described here assessed only short-term outcomes, such as reactions to the method and immediate use of information that GAPS provides. It would be ideal to look at sustainability of the method several years after its introduction in a country and/or district. However, this was considered to be impractical due to the time required and the time limits of this research. To evaluate GAPS, the experience of GAPS in two separate campaigns was assessed. If the first two evaluations were not similarly positive, a third evaluation (in a different campaign and site) would have been carried out. For the purposes of this evaluation, we considered GAPS to have successfully accomplished its objectives if the evaluation of at least two campaigns demonstrated positive results.

The evaluation of each individual campaign was carried out using semi-structured, expert interviews with at least three campaign organizers who were purposively sampled. We tried to interview respondents so that we have a mix of NGO, government and UN backgrounds and district-level and national-level experience [A GAPS manual that includes instructions and data-collection instruments is available from the authors]. An evaluation of an individual campaign was considered positive when the responses of at least two of the three campaign organizers were consistent with expected outcomes detailed as follows:

In cases where GAPS identified at least one geographic area as having many unvaccinated children, at least two of three campaign organizers:

stated that they agree that GAPS is useful for identifying geographic areas requiring special immunization attention;

recommended that the problems identified by GAPS be addressed (or stated that they have already decided to do so); and

stated that carrying out GAPS was worth the effort.

In cases where GAPS did not identify at least one geographic area as having a few or zero unvaccinated children, at least two of three campaign organizers:

stated that they agree that GAPS is useful for confirming the effectiveness of campaign planning and implementation in such areas (or stated that GAPS is useful for comparing with routine information collected during campaigns);

recommended continuing the current campaign strategies (or stated that they have already decided to do so); and

stated that carrying out GAPS was worth the effort.

A semi-structured questionnaire developed for each of the two cases described above included close-ended and open-ended questions, such as recommendations for improving the method or explanations of responses to close-ended questions [This and other sets of questionnaire are included in the manual available from the authors]. The close-ended questions are tallied by hand for analysis of frequency distributions. The open-ended questions were read for understanding better the responses to the close-ended questions and learning what suggestions of the respondents for improving the method were.

Ethics

The GAPS application and evaluation methodolo-gy was reviewed and given a certificate of exemption by the Johns Hopkins Bloomberg School of Public Health, Committees on Human Research under no. 45 CFR 46.101. No identifiers were collected during interviews with experts or household members. Letters of collaboration from local NGOs and ministries of health were obtained instead of local IRB approval since the evaluation involved assessing quality-improvement efforts of an ongoing government health intervention (in this case, mass immunization campaigns to eradicate polio).

RESULTS

We evaluated two applications of GAPS within one week following each application. The first application was in Ethiopia and the second one in Nepal. The results of each evaluation of GAPS are presented below in turn beginning with Ethiopia.

First application of GAPS in Sidama zone, Ethiopia, by phase 1–3

During 13–28 October 2003, the first-ever application of GAPS was conducted around a Subnational Immunization Day (SNID) in Sidama zone, Southern Nations Region, Ethiopia. The date of this SNID was 17–23 October [A description of the GAPS method and results for this application are available from the authors]. Of major importance is that this GAPS application did not identify any pockets of unvaccinated children following the SNID. The evaluation of this GAPS application is described below.

We carried out expert interviews with a purposive sample of three persons. The GAPS evaluation form we used and tailored—for a situation where no evaluation sites failed the post-campaign evaluation—is provided in the GAPS manual. The first person we interviewed was the health coordinator who provided technical backstopping to the only NGO health programme that was supporting (voluntarily) the immunization campaign in the district in which we carried out GAPS. This respondent provides technical backstopping to several health programmes located in the southern region of Ethiopia, including the district in Sidama zone where we had carried out the first GAPS application. Earlier, this respondent participated as a team member during the GAPS application and was able to observe the process and discuss the results first hand. This respondent was selected for an expert interview because we felt that he was the only person who could bring an NGO perspective at an operational level, who had experience with evaluating and supporting vaccination campaigns, and who was very knowledgeable about GAPS due to his participation on the GAPS team in this very application

The second person we interviewed was a health official for the district/woreda where the first GAPS application was carried out. This respondent has major oversight responsibilities for the vaccination programme in the woreda, and he was intimately involved in planning, implementation, and evaluation of the SNID in this district. This respondent also participated as a team member during the GAPS application in his district. We selected this person for an expert interview because he was the only person who could bring a government perspective, who had experience in evaluating and supporting vaccination campaigns at an operational level, and who was also very knowledgeable about GAPS due to his/her participation on the GAPS team.

The third person interviewed was a health officer for a UN agency with substantial responsibility for supporting the planning, implementation, and evaluation of vaccination programmes and other child-health programmes. This respondent primarily provides technical support to government child-health programmes and is a technical officer of her agency. This respondent did not participate as a GAPS team member in Sidama zone. She did attend both pre- and post-campaign orientations and debrief on the GAPS application, however, before the interview. We selected this person for an expert interview because she could bring a UN agency- and national-level perspective, was experienced with many types of evaluations, would have responsibility for supporting future vaccination campaigns, was fairly knowledgeable about GAPS due to her participation in pre- and post-GAPS orientations, and expressed interest in the methodology during both the orientations. All the three respondents have experience or responsibilities with vaccination programmes, with evaluations, and with GAPS (at a minimum they experienced a detailed orientation about GAPS). In addition, both national-level and operational-level perspectives are included, as are NGO, government, and UN perspectives, as designed.

A summary of the results of the Ethiopia GAPS evaluation is provided in Table 1. We consider the results of this evaluation positive due to the following: (a) all the three evaluators agreed that GAPS is useful for confirming the effectiveness of campaign planning and implementation in areas where a few or no children were missed by the vaccinators; (b) two of the three recommended continuing the current campaign strategies in those areas; and (c) when asked if carrying out GAPS was worth the effort, all the three respondents said ‘yes’. When asked an open-ended question to explain their responses, we received the following replies:

Table 1.

Results of evaluation of GAPS, Ethiopia, October 2003

Name of campaign	SNID Sidama zone, Ethiopia
No. of evaluation sites	1
No. passed/no. failed	1 / 0
Campaign organizer/respondent	1	2	3	Summary calculations
Level of responsibility	District	District	National
Type of organization	NGO	Government	UN/WHO
State that GAPS is useful for confirming the effectiveness of campaign planning and implementation (and/or state that GAPS is useful for comparing with routine information collected during campaigns)	Yes	Yes	Yes	Yes–3 No–0
State that they recommend (or have already decided) continuing campaign strategies	No	Yes	Yes	Yes–2 No–1
State that carrying out GAPS was worth the effort	Yes	Yes	Yes	Yes–3 No–0
Most frequently-mentioned negative things about GAPS	No.	Idea		Frequency
	1	Staff turnover requires continued access to training opportunities		1
	2	Need for policy change to support use of such techniques		1
	3	Does not provide the cause of problems identified		1
Most frequently-mentioned ideas for making the evaluation more useful or practical		None		NA

GAPS=Geographic assessment of planning and services; NA=Not applicable; NGO=Non-governmental organization; SNID=Sub-National Immunization Day; UN=United Nations; WHO=World Health Organization

(GAPS) provided skills to monitor activities …. We can see the quality of health services (Government, district-level respondent).

The GAPS experience provided knowledge and skills … to find the effectiveness of some programmes (NGO, district-level respondent).

(GAPS) is cost-effective time-wise and practical … it is an effective rapid assessment (UN, national-level respondent).

In sum, the respondents provided responses about GAPS that are consistent with its design for use and follow-up action. The respondents also mentioned several barriers to implementation, although no two respondents mentioned the same thing: one respondent stated that staff turnover requires continuous access to training in such a methodology, and another respondent mentioned that there was a need for policy change to support the use of such techniques. Another limitation of GAPS mentioned by a respondent was that GAPS can identify problems but may not provide sufficient information by itself to understand the causes of problems identified.

Second application of GAPS in Dhanusa district, Nepal, by phase 1–3

During 18–27 February 2004, we carried out the second application of GAPS during a National Immunization Day (NID) in a terai district of Nepal; terai districts border India with endemic transmission of wild poliovirus just kilometres from the border. The date of this NID was 21 and 22 February [A des-cription of this GAPS application is available from the authors]. Similar to the findings of the GAPS application in Ethiopia, this GAPS application also did not identify any pockets of unvaccinated children remaining following the NID. The evaluation of this GAPS application is described below.

We carried out expert interviews with four persons using a purposive sampling process. Similar to the Ethiopia GAPS application, the GAPS evaluation form we used was for a situation where no evaluation sites failed the post-campaign evaluation. The first person we interviewed was a health official for the district where this GAPS application was carried out. This respondent has major oversight responsibilities for the vaccination programme in the district, and he was intimately involved in planning, implementation, and evaluation of the NID in the district. This respondent also participated as a team member during the GAPS application in his district. We selected this person for an expert interview because he was the only person who could bring a local-government perspective, who had experience in evaluating and supporting vaccination campaigns at an operational level, and who was also very knowledgeable about GAPS due to his participation in the GAPS team.

The second person we interviewed was the project manager of an NGO health programme office in the district where we applied GAPS methodology. This respondent managed and provided technical backstopping to several health programmes located in the district, including a polio-eradication project. Earlier, this respondent participated as a team member during the GAPS application and was able to observe the process and discuss the results first hand. This respondent was selected for an expert interview because he was the only person who could bring an NGO perspective at an operational level, who had experience with evaluating and supporting vaccination campaigns, and who was very knowledgeable about GAPS due to his participation in the GAPS team.

Our third interview was with a high-level health official of the Government of Nepal. This person had a leading responsibility for supporting the planning, implementation, and evaluation of all vaccination programmes in Nepal. This respondent also participated as a GAPS team member in this terai district. We selected this person for an expert interview because he was the only person who could bring a national-level government perspective, who was responsible for helping make major decisions about the vaccination programme in Nepal, who would have responsibility for organizing future vaccination campaigns, who was experienced with many types of evaluations, and who was very knowledgeable about GAPS due to his participation in this GAPS application.

The fourth person interviewed was a health officer for a UN agency with substantial responsibility for supporting the planning, implementation, and evaluation of vaccination programmes in Nepal. This respondent primarily provides technical support to government vaccine programmes and is a technical officer of his/her agency. This respondent also participated as a GAPS team member in this terai district. We selected this person for an expert interview because he was the only person who could bring a UN agency- and national-level perspective, who was experienced with many types of evaluations, would have responsibility for supporting future vaccination campaigns, and who was very knowledgeable about GAPS due to his participation in this GAPS application. All the four res-pondents have experience or responsibilities with vaccination programmes, with evaluations, and with GAPS. As per the study design, we obtained the perspectives of persons working at both nationallevel and operational level and persons working with an NGO, the government, and the UN.

The results of the Nepal GAPS evaluation are presented in Table 2. We consider the results of this second evaluation positive for the same reasons as described above in the Ethiopia application. When respondents were asked to explain their positive evaluation of GAPS, we received the following replies:

Table 2.

Results of evaluation of GAPS, Nepal, February 2004

Name of campaign	2004 Nepal NID 2nd Round (February)
No. of evaluation sites	3
No. passed/no. failed	3 / 0
Campaign organizer/respondent	1	2	3	4	Summary calculations
Level of responsibility	District	District	National	National
Type of organization	Government	NGO	Government	UN/WHO
State that GAPS is useful for confirming the effectiveness of campaign planning and implementation (or state that GAPS is useful for comparing with routine information collected during campaigns)	Yes	Yes	Yes	Yes	Yes–4 No–0
State that they recommend (or have already decided) continuing campaign strategies	Yes	Yes	Yes	Yes	Yes–4 No–0
State that carrying out GAPS was worth the effort	Yes	Yes	Yes	Yes	Yes–4 No–0
Most frequently-mentioned negative things about GAPS	No.	Idea			Frequency
	1	Not all potentially-problematic sites can be evaluated			2
Most frequently-mentioned ideas for making the evaluation more useful or practical	1	Adapt GAPS and LQAS for other uses			2
	2	Visit potential problem sites that were never visited before (overcome any possible ‘Hawthorne’ effect)			1

GAPS=Geographic assessment of planning and services; LQAS=Lot quality assurance sampling; NID=National Immunization Day; UN=United Nations; WHO=World Health Organization

(GAPS provided) more return than investment … We are now more confident that the strategy worked, and the coverage was high … and we also have information about process problems that can be addressed (a government, district-level respondent).

Because the cost of not finding an existing pocket may be higher (an NGO, district-level respondent).

The resources spent have a positive result which is a value … (the process) also gives the community awareness of the importance of vaccination, an appreciation of our interest in what happens there (a government, national-level respondent).

A minimum of time was needed to do this (a UN, national-level respondent).

As with the Ethiopia application, the responses provided by the Nepali respondents are consistent with the purpose of GAPS. The respondents also mentioned one limitation: due to resource constraints, it is not possible to visit each potential problem site for a post-campaign evaluation. Two primary suggestions for the improvement of the GAPS methodology were provided: (a) adapt elements of the GAPS methodology for other vaccination programme activities and (b) during the post-campaign evaluation, visit potential problem sites that were never visited during the campaign or just prior (overcome any possible ‘Hawthorne’ effect of the attention sites receive because of GAPS).

DISCUSSION

The respondents participating in the evaluation of GAPS stated that GAPS was useful for confirming the administrative estimates of campaign coverage, indicating that the campaigns were effectively planned and implemented. These responses are consistent with the design of GAPS in the situation where no pockets of unvaccinated children were found—the case in both the applications in Ethio-pia and Nepal. This is one important additional value of GAPS—a rapid method for checking the quality of mass campaigns rather than relying on administrative estimates of coverage alone. Administrative estimates of coverage are based on dividing the number of persons vaccinated by the number of persons in the target group (e.g. children aged less than five years)—a number obtained from a census that may be old and/or poorly implemented leading to greater than 100% coverage when underestimated. By systematically including GAPS (or at minimum the final post-campaign evaluation phase of GAPS) in mass campaigns, campaign organizers are able to compare administrative estimates of coverage with another population-based source of data.

Similar to other rapid assessment methods, particularly those designed to identify pockets of disease, GAPS is designed to test hypotheses about clustering or pockets of health problems in a relatively-rapid, simple, and inexpensive way. Each of these methods has a process for deciding with sites that are at an increased risk of having a health problem and, therefore, deserve priority attention for supervision (and action if a problem is confirmed). In contrast to these other methods, GAPS requires a smaller sample size that may lead to less time and cost to carry out. This suggests that it may be feasible for managers of mass campaigns to use GAPS routinely to improve the planning, implementation, and evaluation of the campaigns. In addition, GAPS seeks to prevent health problems before they occur by improving planning and to identify problems that may be addressed before the end of a campaign—something these other rapid assessment methods do not include.

The consistent responses from the respondents in the evaluations provide for a positive evaluation of GAPS in each of the two settings it was applied. Based on the evaluation design, a positive evaluation of two specific GAPS applications (and no negative evaluations) provide for an overall positive evaluation of the GAPS as a complementary method to help improve the quality of mass vaccination campaigns and for cross-checking or triangulating routinely-collected administrative estimates of vaccination coverage. This positive evaluation suggests that further applications of GAPS with concurrent evaluation studies should be carried out to confirm the findings here and to examine the usefulness of GAPS in other settings.

Limitations of evaluation

The outcome of any single application of GAPS is not predictable. While we applied GAPS in two very different settings (Ethiopia and Nepal), the outcomes were very similar: no sites evaluated in the post-campaign phase had many unvaccinated children. We did not evaluate an application of GAPS in the situation where one or more sites had many unvaccinated children during the post-campaign phase. This is one limitation of the study. It is possible that expert opinions about GAPS would not be much positive in this second situation. We could have evaluated more applications of GAPS. However, adding more applications of GAPS to this evaluation study would not guarantee that we would be able to evaluate GAPS where one or more sites were identified as having a pocket of unvaccinated children. Because of this and because of time and resource constraints, we did not include more applications of GAPS in the evaluation study once two applications demonstrated positive results.

Another limitation of this evaluation study was that there were a limited number of people who qualified for our expert interviews. We did not interview a large representative sample of campaign organizers to quantify the proportion with positive statements about GAPS. Instead, we purposively sampled the small number of people who we considered relevant experts because of their experience in planning, implementing and evaluating vaccination programmes; involvement in mass immunization campaigns; and had participated in an application of GAPS in the field. We also selected experts with various perspectives: NGO, government, UN, district and national levels. In any one setting, there were only a handful of persons who met these criteria, and we did not want to weigh one perspective heavily over others. For this reason, 3–4 expert opinions about GAPS in each setting was considered to be appropriate.

Recommendations for improving the GAPS methodology

A detailed description of the GAPS methodology is available from the author. Sites visited in the post-campaign phase in both the applications were those sites that were considered most likely—among the other sites in the district—to have a pocket of unvaccinated children. The decisions about sites to visit in the post-campaign phase were based primarily on observations and feedback of health workers during the campaign. Therefore, all sites visited in the post-campaign phase of both the applications had received some kind of supervision visit during the campaign. One recommendation we received during our expert interviews was to also include, during the post-campaign phase, a visit to a potential problem site that had received no or minimal supervision during the campaign. The purpose of this recommendation was to check a site where there was less a chance that performance reflected a reaction to increased supervision attention during the campaign. Another recommendation was to adapt the GAPS methodology, particularly the LQAS elements, for use in testing hypotheses about the quality of other aspects of vaccination programmes, such as the quality of documentation, the cold-chain and supplies, and vaccinator's techniques. The methodology can be adapted to identify specific sites within a programme area with performance significantly lower than expected.

Another potential improvement includes regular updating as we learn about additional factors that may predict the existence of geographic pockets of unvaccinated children. For example, an outbreak of wild polio in 12 polio-free countries occurred during the field-testing of this method as a result of stoppages in polio vaccination in several states in northern Nigeria. The stoppages occurred after political and traditional authorities in these limited geographic areas raised concerns about the safety of the oral polio vaccine. The potential of authorities stopping the campaign was not considered a factor that might predict a pocket of unvaccinated children during the field-testing as this issue did not appear in the literature previously. This factor was included in updates of the GAPS method.

An additional recommendation is to apply GAPS to other types of mass campaigns. For example, currently, there are campaigns taking place around the world for the measles vaccine, vitamin A supplements, deworming medications, tetanus toxoid vaccine, insecticide-treated bednets, meningitis vaccine in emergencies, ivermectrin, and others. GAPS can be applied to these other types of campaigns after appropriate adaptation to both identify pockets of persons missed by the campaign and provide a population-based source of data for confirming administrative estimates of coverage.

In summary, although this evaluation was limited in the number of applications evaluated, GAPS appears to have promise as a practical method to help improve the quality of mass immunization campaigns in two main ways: (a) testing hypotheses about pockets of unvaccinated persons and (b) providing a population-based source of data to help triangulate administrative estimates of coverage. Further applications in different settings will be needed to confirm these findings and/or under what circumstances GAPS might be used for complementing the existing methods and procedures to improve mass immunization campaigns. GAPS may also be considered for improving other types of health campaigns, such as distribution of insecticide-treated bednets, vitamin A capsules, and deworming medications.

ACKNOWLEDGEMENTS

Funding for field work was supported, in part, by the United States Agency for International Deve-lopment (Cooperative Agreement No. HRN-A-00-98-00053-00).