Reducing missed opportunities for vaccination in selected provinces of Mozambique: A study protocol.

Background: A missed opportunity for vaccination (MOV) refers to any contact with health services by an individual who is eligible for vaccination, which does not result in the person receiving the vaccine doses for which he or she is eligible. A consortium of partners, including VillageReach, the Ministry of Health in Mozambique and the World Health Organization, will implement a strategy to reduce MOV in Mozambique. The strategy involves demonstrating the magnitude of missed opportunities and their causes, and exploring tailored health system interventions to reduce them, with the aim of increasing vaccination coverage and timeliness of vaccinations.
Methods: A mixed-methods approach will incorporate both quantitative and qualitative tools. The assessment will target caregivers of children between the ages of 0-23 months who attend a health facility in the selected districts on the day of the assessment. Caregivers who are at least 18 years old will be eligible for inclusion. Another component of the assessment will target all health workers in the selected health facilities on the day of the assessment. A sample of 30 health facilities in different regions of the country will be assessed, with a target sample size of 600 caregiver exit interviews, 300 health worker interviews and focus group discussions with both caregivers and health workers. Data collection will commence late 2017, and the data will be electronically captured, managed and analyzed. Thematic analysis of data from the qualitative aspects of the assessment will be conducted, presenting the scope of interviews, representative verbatim quotes and key conclusions. Conclusions: A concerted effort to reduce or eliminate MOV could increase vaccine coverage by up to 30% and may contribute to wider improvements in efficiencies of service delivery beyond the immunization program. In addition, the findings could contribute to a better understanding of MOV in similar settings.

Introduction

According to a report by the World Health Organization (WHO), global vaccination coverage has stagnated below the 90% target for more than a decade, meaning that a significant proportion of the global annual birth cohort remains unvaccinated with DTP3. Similar to many other developing countries, Mozambique has experienced challenges with increasing immunization coverage. The recently identified cVDPV polio case and pockets of unvaccinated children in one province indicates an urgency to strengthen the Expanded Program on Immunization (EPI). There is a growing need to understand the persisting barriers and the degree to which these issues affect eligible children who could potentially be vaccinated in the absence of other challenges. Thus, a consortium of partners, including VillageReach (a global health non-profit organization), the Mozambique Ministry of Health (MISAU) and the World Health Organization (WHO), are planning to implement a strategy to reduce Missed Opportunities for Vaccination (MOV) in Mozambique during the last quarter of 2017. According to the Planning guide to reduce missed opportunities for vaccination, MOV include any contact with health services by a child (or adult) who is eligible for vaccination (unvaccinated, partially vaccinated or not up-to-date and free of contraindications to vaccination), which does not result in the individual receiving all the vaccine doses for which he or she is eligible. For our purposes, the focus will be only on children below two years of age.

The MOV strategy as described in the Methodology for the assessment of missed opportunities for vaccination will aim to demonstrate the magnitude and identify causes of these missed opportunities, followed by tailored health system interventions with the objective to reduce them, leading to an increase in vaccination coverage and timeliness of routine vaccinations.

While it is widely accepted that some of the missed children (commonly referred to as the 5 th child) have limited access to health services, previous studies reported in the Planning Guide suggest that a proportion of the missed children may already be accessing treatment and vaccination services. Unfortunately, for a variety of reasons, many health services continue to miss the opportunity to vaccinate these eligible children. Therefore, the MOV strategy emphasizes reducing MOV at the health facility level on a day-to-day basis and consequently increasing immunization coverage at existing vaccination sites (health centres, hospitals, outreach/mobile services). Furthermore, reducing MOV will also improve timeliness of vaccination, improve the efficiency of health service delivery in general, and promote synergies between treatment services and preventive programs at the health facility level. This is in line with other previous recommendations that have been made on the same to address this widely prevalent issue [1].

There has been an increasing momentum at global, regional and country levels to conduct MOV assessments in different settings. During the 2014 review of the Global Vaccine Action Plan, the Strategic Advisory Group of Experts on immunization (SAGE) recommended MOV studies based on evidence indicating a globally pooled prevalence of 32% [2]. According to the Planning Guide, previous MOV assessments suggest several reasons for MOV linked to service delivery, vaccine shortages or other logistics barriers and demand-related barriers. That said, the findings tend to be country-specific and preclude standardized solutions [2].

Study objectives

In spite of partner support to the EPI program in Mozambique, the program continues to under-perform by several measures. This study will assess the magnitude and causes of MOV in three provinces that have received varying degrees of vaccine delivery strengthening interventions (e.g. optimized vaccine supply chain) and short-term interventions (e.g. immunization campaigns). This study will allow for some comparisons by province or type of barrier (e.g. vaccine shortages or other logistics barriers; the failure or inability of health providers to screen patients for eligibility; perceived contraindications to vaccination on the part of providers and parents).

Key questions

The MOV strategy answers three important questions:

How many opportunities for vaccination are missed at existing vaccination sites?

Why are opportunities for vaccination missed at the different vaccination sites?

What can be adjusted or done differently (e.g. policies, behaviours, structural or organizational changes) in order to eliminate MOV?

Estimate the extent of MOV and examine different correlations between MOV and other explanatory and demographic factors;

Estimate the proportion of children with missed opportunities, the proportion of visits with missed opportunities and the number of eligible doses missed, by antigen;

Classification of MOV by cause (e.g. logistics-, care-giver- or health provider-related).

Study design

This study will employ a mixed-methods approach, incorporating both qualitative and quantitative tools. A mixed-methods approach according to Creswell (2003) is one where researchers tend to base knowledge claims on pragmatic grounds (e.g., consequence-oriented, problem-centred, and pluralistic) [3]. The researcher bases the inquiry on the assumption that collecting diverse types of data best provides an understanding of a research problem [3].

Caregivers (mother or any other person responsible for the child) of children between the ages of 0–23 months;

Health workers (both preventive and curative department staff).

Study population

Inclusion/exclusion criteria

The MOV assessment targets caregivers of children between the ages of 0–23 months who attend a clinic in the selected districts on the day of the assessment. Regardless of the reason for visiting the health facility, their place of residence or relationship to the child, all caregivers are eligible to be interviewed. The caregiver may be the person who gave birth to or adopted the child, or is otherwise taking care of the child, such as an aunt, grandmother, or father. Caregivers should be at least 18 years of age. The assessment also targets all health workers (both preventive and curative department staff) that work in the selected health facilities on the day of the assessment.

Sample size

According to the Methodology Guide, a sample size of 600 caregiver exit interviews and 300 health worker questionnaires will be targeted in a purposive sample of 30 health facilities.

Sampling criteria

Provincial, district and health facility sampling

We have selected three out of the eleven provinces based on the EPI team’s perceptions of performance, reported coverage rates, variability of vaccine delivery interventions received and region/location (North, South, Central). The three selected provinces are: Niassa (Northern region), Zambezia (Central region) and Maputo (South).

On average, each province has about 15 districts. We will aim for two districts per province (N=6), selecting a well performing and a poorly performing district. The criteria for selection of districts will take into account the following indicators: administrative coverage, stock out rates and cold chain equipment functioning. Data from two electronic systems SELV and SISMA will be reviewed for a period of three months (April to June 2017). For the three indicators, the vaccination coverage (children completely vaccinated), stock out and fridge functioning will be analyzed. Good performing districts will be defined as having more than 80% of vaccination coverage, less than 10% of stock out and more than 95% of fridges functioning. Poorly performing districts will be defined as having less than 50% of vaccination coverage, more than 10% of stock out and less than 95% of fridges functioning.

In each of the six districts, we will select five health facilities (N=30). The health facilities will comprise of a mix of hospitals, health centres and clinics. We will assign an identification number to each health facility and apply a randomization technique.

Participant sampling

At each health facility visited, the field team will conduct 20 sequential exit interviews and 6–10 Focus Group Discussions (FGDs) with caregivers. Field teams will be made aware of the following core principles:

Aim to conduct a fixed number of interviews at each health facility selected (‘quota sampling’):

10 exit interviews with caregivers of infants 0–11 months old;

10 exit interviews with caregivers of children 12–23 months old;

Aim to speak with caregivers attending the facility for different purposes, i.e. if possible, an equal mix of caregivers seeking immunization vs. other services;

Whenever possible, data collection should be spread across several different health facilities (for example, no more than 20 exit interviews in any one facility)

If smaller health facilities are selected, it may not be possible to interview up to 20 caregivers. In such instances, additional health facilities will be visited to make up the sample quota for each field team. In these situations, the additional facilities will have similar characteristics to the one originally sampled, e.g. size/patient flow, district of location.

The assessment will also target all health workers (both preventive and curative department staff) that work in the selected health facilities on the day of the assessment for interviews and FGDs. 6–10 FGDs for health workers will be scheduled. Each FGD will have 6 to 10 participants for effective dialogue to take place. For health workers, it might be difficult to mobilize an FGD if they are too few or are not available to all meet at the same time. In such instances, a few in-depth interviews may be considered.

Finally, a purposive sample of senior staff and health administrators at multiple levels of the health system will be selected.

Methods

Table 1 presents an overview of the proposed tools for this study. The detailed tools are available from the methodology guide on the WHO website.

Table 1.

Tools for assessment of missed opportunities for vaccination (MOV).

Tools	Type of data to be collected
1. Health facility exit interviews (interviewer-administered)	-  socio-demographic information; -  vaccination history (routine immunization) *; -  awareness of opportunities for routine immunizations; -  home-based record (vaccination card/health passport) availability and retention; -  reasons for non-vaccination; and -  quality of the vaccination service received.
2. Health worker knowledge, attitudes and practices survey (interviewer-administered)	Knowledge about vaccination, including antigens, immunization schedules, and contraindications to vaccination.
3. Focus group discussions (with caregivers and health workers)	Exploring causes of MOV, and potential solutions and barriers to implementation of proposed interventions.
4. In-depth interviews (with senior staff and health administrators)	Interviews will be conducted with individuals who are insightful or influential about the health facility or the community, such as the health facility heads or in-charges, directors, matrons and administrators. This facilitates triangulation with other data elements and provides information regarding the informants’ perception of the community and health facility dynamics affecting immunization services, informant support or opposition to immunization. Key informants will also be asked to explain health worker and caregiver behaviors/responses and suggest or validate/refute previously proposed interventions to reduce missed opportunities.
Brainstorming sessions	Present research findings to immunization stakeholders and formulate an action plan with proposed interventions.

* Vaccination status will be documented from the information on the home-based records or health facility registers, where possible. The data collection team will not rely on the caregivers’ recall if the required record has been lost or is otherwise unavailable. This is particularly important because the date of vaccination is critical in assessing MOV.

Study procedures

Facilitators will meet 1–2 days prior to the beginning of field team training to finalize the remaining logistic details for the assessment (such as finalizing the list of sampled facilities, transport arrangements, training logistics) and to finalize the data collection tools.

All data collectors will be centrally trained over a period of 4 days to prepare them for data collection.

All caregivers exiting the health facility with a child that appears to be between 0–23 months are eligible and will be asked to participate. Each potential participant will be pre-screened on age of child only. The interviewer will explain the process and seek for written consent to participate in the interview. The questionnaire will be administered to consenting caregivers of eligible children, or any other consenting adult member (defined as being ≥18 years of age) accompanying the child, who is knowledgeable about the mother’s background and the vaccination status of the child. If an adult is accompanied by more than one child, the interview will focus on the youngest child. Caregivers who attend the facility for a variety of purposes (i.e. immunization as well as other services) will be targeted. In situations where the EPI clinic is located separately from the outpatient department, the interviewers will move around to interact with caregivers exiting at different points of the facility complex. For the health worker interviews, all health workers (both preventive and curative department staff) working in the selected health facilities on the day of the assessment are eligible to be interviewed.

The MOV assessment is designed to be completed in less than 10 days, including training, data collection and preliminary data analysis. Preparation of field teams through training and pilot testing of the data collection tool will commence in November 2017. Field teams will spend a maximum of two days in each health facility. Data will be collected using an electronic data platform where each fieldworker will be given a tablet computer. Standardized paper interview questionnaires will be available for back-up purposes.

As outlined in the Planning Guide, field teams (interviewers and supervisors) will be drawn from MOH staff and other in-country immunization partners since they are knowledgeable about the EPI program. In addition, we will involve a few independent researchers with experience in conducting similar work. This assessment will utilize a three-person team for province.

Data management and analysis

The use of an electronic platform for data collection will facilitate preliminary analysis immediately after data collection for presentation at the brainstorming and debrief sessions. All data on the electronic data platform will be exported into Excel, and then to Epi Info™ 7 for data cleaning, data management and analysis. Initially, a simple analysis of quantitative and qualitative data will be carried out. Preliminary results will consist mainly of simple frequencies and bivariate analyses of quantitative data and a preliminary interpretation of key themes of the qualitative data. The results will be presented at the opening of the brainstorming session with stakeholders. These analyses coupled with other anecdotal information from the field will be used to inform the development of an action plan to reduce MOV.

In the weeks following fieldwork, a more detailed analysis will be carried out once all the data are compiled. The detailed analysis will estimate the extent of MOV as well as examine different correlations between MOV and other explanatory and demographic factors. Estimates of the proportion of children with missed opportunities will be calculated, as well as the proportion of visits with missed opportunities and the number of eligible doses that were missed, by antigen. Other analyses may include home-based record availability and vaccine coverage by antigen. These analyses will be stratified using other explanatory variables in the dataset, as appropriate. Where possible, the results will be compared to other recent surveys in the country, including Demographic Health Surveys and Multiple Indicator Cluster Surveys.

Thematic analysis of data from the qualitative aspects of the assessment will be conducted and included in a final report, presenting the scope of interviews, observations, representative verbatim quotes and key conclusions. This thematic analysis will be conducted using standard qualitative data analysis software (Atlas TI 8.0). Where possible, community and key informant recommendations for reducing missed opportunities will be included along with other study conclusions.

There will be minimal collection of personally identifiable information (PII); however, where necessary any electronic datasets with PII will be password-protected. All electronic databases will have PII removed before any off-site analysis. The tablets will automatically record the time of the interview as well as the location where the forms were completed. Supervisors will review all data collection forms for completeness and accuracy prior to submission to the Assessment Coordinator (GJ). Interviewers will upload data every evening, so that further data quality checks can be conducted by BM, SS, IO. To enable the best quality data entry, the data entry forms will be designed using value constraints and data checks to the extent possible.

Ethical considerations

The Nacional de Bioetica (National Bioethics Committee) in Mozambique granted ethical clearance for this study (Reference number 1008). Furthermore, each of the provinces provided permission for the study.

Respondents in this assessment (caregivers and health workers) will be requested to provide written consent prior to the interview. They will be informed that participation is voluntary and optional and that their responses will in no way affect their ability to access services or threaten their employment in the case of health workers.

Limitations

The MOV assessment process is subject to some limitations and biases inherent in the simplified sampling methodology and the convenience sample of all children available on the day of the study, including selection bias. Also, the health facility staff could modify their practices on the day of the assessment. To minimize the impact of any changes in practices, the administration of health worker KAP questionnaires will take place in the afternoon, after the vaccination clinic sessions for the day have been completed and caregivers should only be interviewed upon exiting the facility, after they have received the service. Similarly, the FGDs will be conducted with different health workers (at a different health facility) than those that completed the health worker questionnaires. The assessment results will need to be interpreted with caution, as the sample will not necessarily be nationally representative.

Dissemination

Summary reports will be developed for each district and province. Preliminary results will be presented and discussed at brainstorming sessions and debrief meetings with immunization stakeholders. Results will also be reported to the district, provincial and national-level immunization program staff, using regular feedback and training meetings. Results may be presented at professional meetings, and national, regional or international scientific meetings. To accelerate cross-country and peer-to-peer learning, results from the MOV assessment be written up for publication in peer-reviewed journals.

Conclusion

A concerted effort to reduce or eliminate missed opportunities, especially by addressing persistent barriers, could result in increased vaccine coverage by up to 30%. In addition to using data to improve local vaccination coverage in Mozambique, the findings could contribute to a better understanding of MOV in similar settings.

Detailed comments

This research protocol is based on resources from the recently developed WHO strategy for reducing missed opportunities for vaccination (MOVs) ( http://www.who.int/immunization/programmes_systems/policies_strategies/MOV/en/), and in particular the MOV Methodology. The protocol is clearly written and follows the WHO guidance closely. It is based on formative research which gives priority to public health action and not solely to more technical aspects of the assessment of the prevalence of MOVs. As such it should be expected to lead to a participatory process of investigation of the importance of MOVs and the development of strategies to reduce their occurrence. The WHO strategy is tailored to in-country public health professionals with limited time and/or research experience, aiming to provide field-friendly, practicable tools. This study will provide further experience with the use of the tools and contribute to knowledge of how assessment of MOVs and their causes can lead to effective public health action. It should be noted, however, that the study is not a rigorous statistical measurement of the prevalence of MOVs.

Rationale and objectives

The introduction summarizes key reasons why assessment of MOVs is considered important at the global level and notes that vaccination coverage appears to have stagnated in Mozambique. Previous work on MOVs in Mozambique is not, however cited, and the authors are encouraged to review the relevant literature from their country.

In the abstract and conclusions, the authors suggest that by avoiding MOVs, coverage could be increased by up to 30% but provide no data on which to base this claim. Indeed, given that according to WHO-UNICEF estimates, coverage of 3 rd dose of pentavalent vaccine is 80%, it is difficult to see how coverage could increase by another 30%!

The study objectives, key questions and results measurement indicate that the authors wish to measure the frequency of MOVs in 3 provinces; to assess variation in MOV frequency according to variation in potential contributing factors such as stockouts; and to classify the causes of MOVs. Most importantly, the study aims to determine what can be done differently to reduce MOV occurrence.

The study design does not, however, lend itself to meeting all these objectives in a statistically rigorous way. This should be made clearer in the protocol – the WHO methodology sacrifices statistical rigour for programmatic utility and while the trade-off is likely to be appropriate, it needs to be clearly acknowledged.

Study design

These health-facility-based studies, conducted in a purposive sample of facilities in only a few districts on a single day per facility, give an indication of how common MOVs are on the day of the visit as well as qualitative insights into the causes of MOVs, but are not designed to give a representative estimate of MOV frequency.

By contrast, data from high-quality household surveys can be analysed to assess the frequency of one type of MOV which is non-simultaneous vaccination – the failure to administer all vaccines for which the child is eligible at a visit when at least one vaccine is administered (Jani et al., 2008, Cutts et al., 1989). The 2015 update of the WHO coverage survey guidelines provides information on how to analyse coverage surveys (including multiple-indicator surveys such as DHS or MICS) to obtain estimates of the frequency of MOVs (and gives detailed guidance on different potential definitions of MOVs). Unfortunately, it appears that there have been no recent national vaccination coverage surveys in Mozambique – the most recent one used in the WHO-UNICEF estimates was the 2011 DHS (see section 4.6 “coverage survey data” on http://www.who.int/immunization/monitoring_surveillance/data/en/) – thus this approach may not be feasible at national level at present. The authors could, however, look for data from subnational surveys that could be re-analysed to measure the frequency of MOVs at the population level.

Details of study methods

The study methods follow closely the WHO guidance.

It is unclear how the authors decided on just 2 districts per province. The choice of districts is said to be based on those with less than 50% admin coverage, and those with more than 80% coverage as well as indicators of cold chain and vaccine management. They do not provide information on the electronic data systems which provide these data for selection of districts. How do they define fully vaccinated – usually this is not an indicator which is included in routine reports. Are there districts having less than 50% pentavalent 3 coverage? According to the WHO website ( http://apps.who.int/gho/data/view.other.immu-admin2-dtp3?lang=en) the lowest district coverage is just over 60%.

Sample size – as recommended by WHO, they aim for 600 exit interviews and 300 health worker interviews at 30 health facilities. However, this does not take into account their desired aim of comparing MOV frequency with various barriers in the different provinces. Indeed, the issue of how to calculate sample size (and how to calculate confidence intervals when analysing data) is difficult in the absence of a true probability sample and it might be best to accept that this is a semi-quantitative study that does not aim to provide statistically valid results.

The authors note the potential problems that may occur if there are very few children attending the facility on the interview day. An alternative sampling scheme whereby facilities are selected using probability proportional to estimated size (where size is average patient load per day or per month, depending on data availability) rather than simple random sampling could help to avoid this problem.

The authors do not specify whether facilities will only be visited on days when vaccination is scheduled or irrespective of vaccination scheduling. In many countries, health facilities do not offer vaccination daily but only schedule one day per week or per month etc. for vaccination. The value of including hospitals is unclear – do hospitals in Mozambique provide vaccination services? If so, more information about how work would be organized in hospitals would be helpful. What is the difference between a health facility and a clinic? Will outreach sites be included? Outreach is often done to more remote areas, and these are areas where MOVs may be particularly important as it may be several weeks before the next health service contact, thus leaving children at risk of disease for longer.

The need for 300 health worker interviews is not explained. This component of the study is likely to be time-consuming for data collection and data analysis and yet many of the health workers interviewed are unlikely to work directly with vaccination services. Perhaps one or two questions about referral guidelines for vaccination of children attending primarily for curative care could be asked as part of key informant interviews at facilities instead.

This protocol, and the WHO guidelines, do not include the collection of retrospective data on MOVs, nor any assessment of the quality of data recording on home-based records (HBRs) and facility registers. It would be straightforward to add these components to the study and need not prolong the duration of data collection (especially if fewer health worker interviews are conducted). For children with HBRs, dates of all vaccinations ever received can be recorded, allowing analysis of past MOVs which would not be subject to any potential bias from the presence of the study team on the day of interviews (and WHO has software pre-coded which can facilitate this). For these same children, dates of vaccination can also be extracted in the afternoons from facility registers, and compared with the dates on the HBRs to assess the quality of data recording – a type of data quality self-assessment. If more time is available, facility registers could be a rich source of data for a larger sample, potentially selected randomly, which would allow assessment of the frequency of MOVs at that facility over a period of time not just the day of the interview and would be more likely to provide statistically meaningful results. It would be very helpful to pilot such additions and determine the time and resources needed to add them to the WHO Methodology.

The section on selection and training of data collectors is short. It can be very helpful to include health workers from other districts or provinces in the data collection as they will gain insights into MOVs by participating in the assessment. Training should include enough time in the field to determine how to organise work at the facility, in order to be able to conduct sequential interviews without making mothers wait too long and without missing those who may leave via multiple exit points. Work at various types of facilities should be included in the practical training. There is no detail on training for focus groups which require different skills from those for exit interviews.

The protocol and WHO guidance says that the study should be complete in 10 days including training and analysis. This appears to be unduly short and likely to jeopardise data quality. It would seem wise to extend this, at least to allow additional time for training, 10 days for data collection, and additional time for data cleaning and analysis including, ideally, analysis of previous MOVs as well as those on the day of the visit.There will be a lot of qualitative data collected from all these interviews and focus groups, that take substantial time to compile and interpret.

Datasets

Which electronic platform will be used for data entry?

If both paper and electronic data collection are used, how will discrepancies be resolved? Or will paper forms only be consulted if data are lost from the electronic platform?

For analysis, the authors might like to review the guidance in the WHO coverage survey manual on calculation of the frequency of MOVs using different definitions and, if interested, contact WHO for further information and guidance on appropriate software.

The authors aim to measure the proportion of MOVs occurring on the day of the visits but do not define clearly how this will be done – these analyses are not as straightforward as might initially be thought. First, definitions of eligibility for each vaccine-dose combination is needed, e.g. what is the minimum age that will be accepted for 1 st dose pentavalent vaccine – exactly 42 days or a few days before and if so, what lower age bound is acceptable? What will be done if the date of birth has been entered incorrectly on the HBR or register? For intervals between doses, likewise, will there be any flexibility below the recommended 4 weeks (28 days)? What, if any, contra-indications will be considered valid contra-indications to each vaccine? What is the national policy for vaccinating children who do not present with a home-based record and cannot be traced in the register. These details need to be worked out and documented before the analysis.

The authors propose further analyses that are beyond the remit of this formative research. The study is not designed to measure coverage hence “analysis of vaccine coverage by antigen” is not meaningful –a convenience sample of 10 children in each age group at purposively selected facilities on one day does not represent coverage. It is not meaningful to try to compare results from this sample with those from surveys such as DHS or MICS. It is definitely helpful to analyse DHS and MICS data to obtain population-based measures of MOVs but these are not comparable to this sample.

Overall, this should be a useful study that helps to generate important discussion of strategies to reduce MOVs and address other barriers to vaccination which the interview results will suggest. It could be substantially enriched at relatively little marginal cost by including the assessment of past MOVs and the quality of data recording at the health facilities visited. It is important to remember that this is formative research and not try to use the data as if it were from a statistically rigorous sample.

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Please provide more information on the selection of data collectors, particularly the qualifications needed

More information is needed on the method for the brainstorming sessions that take place after the assessment but are considered integral to the MOV assessment methodology. At what level do they take place (national or also subnational); who should be involved—is it just “immunization stakeholders” when there are other stakeholders (particularly at subnational level) who have important roles to play in reducing MOV; what is the process.  Such information is vital to ensuring that the real purpose of the assessment – namely to reduce MOV – is served.

Abstract: Is the expected 30% coverage increase in relative percent or absolute percentage points.

Introduction: Make it clearer that you are referring to valid (not invalid) contraindications.

32% prevalence of MOV needs a little more clarification for it to be understandable.

Mention the limitation that health facilities that do not offer vaccination services (even though they could) are not included in the methodology and also that mothers who do not visit health facilities are excluded.

Sampling criteria:  “less than 10% of stockout” is not understandable.

6-10 FGD at EACH health facility is probably highly unlikely to implement, given the size of the health facility staff, assuming that a focus group is understood to be different than an interview with a single individual.

It would also be useful to interview staff responsible for managing the vaccine supply chain to gain a possible additional perspective.

Study procedures: Be prepared with rules when the adult is not in fact knowledgeable about the vaccination status of the child. Be careful to recruit interviewers and supervisors from staff who are not directly responsible for the program in the sampled health facilities. They should also be able to speak the local language.

Data management and analysis: Given their lower power and different geographic focus, it is unclear how these results can be compared to large recent surveys in the country, including Demographic Health Surveys and Multiple Indicator Cluster Surveys.

Limitations: This statement, affecting the methodology, was not at all clear earlier: “The FGDs will be conducted with different health workers (at a different health facility) than those that completed the health worker questionnaires.”

Conclusion: Is the expected 30% coverage increase in relative percent or absolute percentage points?

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.