Limitations of immunization registers at community health centers for measuring immunization coverage: a case study of the Japanese encephalitis mass immunization program in Bali Province, Indonesia.

OBJECTIVE: The aim of this study was to compare the coverage of Japanese encephalitis (JE) immunization obtained from a recall survey and immunization registers at community health centers (CHCs) in Bali Province, Indonesia.
METHODS: A population-based survey was conducted, and random 2-staged selection of clusters of sub-villages was performed. The sample consisted of households with children aged 9 months to 15 years old. Interviews were carried out with carers to recall JE immunization status. The recall immunization status was considered valid when name, date, and confirmation of immunization were available in an immunization register at a CHC. Descriptive analysis was performed. The completeness of the information within immunization registers at CHCs was assessed.
RESULTS: The coverage of JE immunization obtained from the recall survey was 93.8% (95% confidence interval [CI], 92.8-94.9). It decreased to 74.9% (95% CI, 72.8-77.2) after being validated against immunization registers. The recall coverage of JE immunization was significantly higher than immunization register data suggested. This discrepancy varied from 6.5% to 36.4% across 6 districts; however, none of these districts achieved the recommended target coverage of 95%. The quality of immunization registers varied across CHCs.
CONCLUSION: The use of an immunization register may result in underestimating the true coverage of vaccination programs, and its utilization for measuring immunization coverage requires further consideration.

Graphical abstract

Introduction

Vaccination is critical for the prevention and control of infectious diseases. Children in Indonesia now compulsorily receive at least 5 vaccinations to prevent 8 diseases (1-time hepatitis B [HB]-0, 1-time Bacillus Calmette–Guérin [BCG], 3-time diphtheria-tetanus-pertussis [DTP]-HB/DPT-HB-HiB, 4-time polio or 3-time inactivated polio vaccine [IPV], and 1-time measles injections), which together comprise the basic immunization program [1].

Discrepancies between the vaccination rate as reported by the state and as recorded in independent survey data are common for basic immunization programs [2]. The Indonesia Health Profile in 2018, which collected data from immunization registers at community health centers (CHCs), reported that the rate of complete coverage for basic immunizations was 90.6% [1]; however, the National Basic Health Survey (RISKESDAS) reported a coverage rate of only 53.8% to 57.9% [3,4]. Similarly, the Demographic Health Survey (DHS) reported low coverage for basic immunizations (48.0% in 2012 and 59.4% in 2017) [5]. This is also the case for Bali, the province with the highest rate of basic immunization coverage in Indonesia. The Indonesia Health Profile in 2018 reported that the rate of complete coverage for basic immunizations in Bali was 99.6% [1]. This figure decreased to 80% to 92% when compared against the 2018 National Basic Health Survey [5,6].

Both the RISKESDAS and DHS measure immunization status using either carers’ recall or immunization registers at CHCs. No survey in Indonesia has ever examined the possible causes of discrepancies regarding coverage between data reported by the program (based on immunization registers at CHCs) and data obtained using recall surveys. A low immunization rate due to underreporting might trigger unnecessary interventions to improve coverage.

World Health Organization (WHO) guidelines suggest that immunization coverage can be best measured by combining both approaches by using carers’ verbal reports and validating them against immunization registers at CHCs [7]. While a 2-staged approach suits countries with integrated health information systems [8], such an approach may not be a good fit for the Indonesian context. First, as is also the case for many countries with fragmented health information systems, the quality of immunization registers at CHCs in Indonesia is poor (e.g., incomplete data, frequent errors, and double recording) [9,10]. Second, immunization registers at CHCs are predominantly paper-based and are vulnerable to missing records. Third, the availability of health resources across geographical regions influences the quality of recording and reporting at CHCs, meaning that wealthy regions are likely to have better recording and reporting systems. Arguably, using immunization registers as the standard for measuring the rate of immunization coverage might lead to underestimating the actual coverage.

Household surveys based on information from home immunization cards can still result in many issues such as incomplete filing, inaccurate immunization dates, or even misplacement of the card [7,11]. A systematic review found that the sensitivity and specificity of carers’ recall for measuring the immunization status of children varied, ranging from 41% to 98% and 12% to 80%, respectively [12]. Household surveys and carers’ recall are vulnerable to recall bias, a phenomenon in which a carer may have received or has come to believe an inaccurate report of the immunization status of their children. Additional vaccination programs to improve uptake, typically provided through supplementary immunization activities, can further complicate recall bias.

The Indonesian government introduced the Japanese encephalitis (JE) mass immunization program in 2018, and Bali was selected as the first province to implement it. The program aimed to improve JE vaccination coverage among children aged 9 months to 15 years old, followed by a regular vaccination program for children aged 10 months [13]. The program was conducted in 2 phases in March and April 2018 at elementary and junior high schools and health immunization posts. The program used 1 dose of the Chengdu SA14-14-2 live-attenuated JE vaccine [13] and aimed to reach the estimated target set by Indonesia’s Ministry of Health of 890,050 children [14,15]. The Bali Provincial Health Office reported that coverage resulting from the JE immunization program based on the estimated target of 890,050 children was 101.78% [14], exceeding the initial target of 95%.

In this study, we conducted a population-based survey to estimate the immunization coverage at district and provincial levels in Bali using the 2 approaches recommended by the WHO [7,11]. We compared the immunization rate for JE in Bali that resulted from a recall survey to the rate that resulted from data collected from immunization registers at CHCs. We also compared sociodemographic characteristics between districts using both methods and with recall survey data only. Given that the survey was performed soon after the vaccination program, we assumed that recall bias would be minimal. Thus, we examined the usefulness of immunization registers at CHCs for determining the coverage of the immunization program. In addition, we evaluated the quality of immunization registers at CHCs.

Materials and Methods

A 2-staged probability-based cluster and systematic sampling survey was conducted across 9 districts in Bali Province, Indonesia. The survey followed the 2015 and 2018 WHO guidelines [7,11] for estimating immunization coverage at the district level and defining immunization status.

Bali Province consists of 9 districts, with the highest population density and economic and tourist development located in the southern area, particularly in the districts of Denpasar and Badung. Almost half of the population of Bali lives in urban areas, and more than 80% is Hindu [16]. The population migration rate between districts within Bali and from other Indonesian provinces to Bali is high, particularly in Denpasar and Badung [16,17]. Within the 9 districts of Bali, there are 57 sub-districts, 667 villages, and 4,450 administrative sub-villages (banjar) [18]. A sub-village is the smallest administrative unit from which we obtained household listings.

Samples were calculated for each district, comprising 9 strata. Within each stratum, we used an expected immunization coverage of 90%, with a desired precision of 0.05, alpha (0.05), and design effect (2.04), yielding an effective sample size of 216. We assumed we would collect data from an average of 9 respondents per cluster (sub-village or banjar), and assumed an intra-cluster correlation coefficient of 0.13. We estimated that an eligible child would be located in an average of each 1.3 homes visited. We assumed that as many as 10% of families with eligible children would either not be at home when the survey team visited or would refuse to participate in the survey, so we used an inflation factor of 11% to account for likely non-responses. The estimated total number of respondents who would complete questionnaires was 3,966, while the estimated total number of households that would be visited was 5,729. We distributed the number of targeted clusters proportionately based on the population size of each district.

Details of the location and name of each banjar were obtained from district health offices and CHC officers, and the household listings in each cluster were obtained from the Civil Record Offices and the village and sub-village head offices. No sketch mapping or household mapping was performed [7,11] before data collection. We conducted 2 stages of cluster and systematic random sampling for each district to select 441 clusters and 13 households from each cluster. Only households with eligible children aged between 9 months and 15 years old at the time of the JE immunization program were included. To interview carers, a prioritized system was designed to first target the mother, who most commonly provides childcare, to be interviewed. When the mother was not available, the next family member in terms of priority was the father, then the grandfather/grandmother, and then any other relative who might be aware of the immunization status of the child, such as an aunt, uncle, or sibling. Field staff were required to explain the purposes and procedures of the study, confidentiality, and voluntary participation prior to the interview. Only respondents above 17 years old who signed a consent form were interviewed. An additional visit was conducted if no one was available at the targeted households for the first visit.

A web-based closed-ended questionnaire was developed using Epicollect5 (Imperial College London, England). The questionnaires were piloted with 10 respondents to check the content and flow of the questions. Some revisions were made to replace questions that were found to be too technical. Field surveyors used smartphones to interview respondents, and data entry was linked directly via data transmission to a central location. A paper-based questionnaire was provided for back-up when internet access was limited. One person was assigned to be the data manager.

The survey was administered by 6 teams for data collection, with each team being led by 1 study coordinator (SC) supported by 5 or 6 interviewers. SCs were responsible for validating immunization status obtained through the recall survey against immunization registers by visiting CHCs where records on JE immunization were stored. SCs were accompanied by interviewers to assist in the validation process. Interviewers played a critical role in determining children’s immunization status by recall. A 3-day intensive training was provided for interviewers to ensure the quality of the data collection. Fieldwork took place from June 2, 2018, to August 25, 2018.

Information on immunization status was obtained via a recall survey taken by carers, which was then cross-checked against immunization registers from CHCs. The immunization status from a recall survey was deemed valid when the name, date of immunization, and tick mark or other signs as evidence of immunization were available within the register. Immunization data on each child were photographed to verify the date of immunization during data cleaning. Immunization status was deemed invalid when a child’s name could not be found or the SC could not confirm immunization status.

We also evaluated the quality of each immunization register for additional data collection. The average score of the register was obtained from a qualitative assessment by the SC during their visit to a CHC. Availability and completeness, management of the register, the patient name recording system, and straightforwardness of finding subjects’ names were assessed. The completeness of 3 indicators (name, date of immunization, and mark or sign of immunization) was noted. The cooperativeness of the health staff was also qualitatively recorded by the SC. This measure was based on the ease of communication with the SC as well as the staff’s intention and motivation to support the process of validation. Scores for negotiations with SC or with SC and enumerators who visited particular CHCs were rated on a scale of 1 (poor) to 10 (excellent). Completed questionnaires were uploaded to the Epicollect server and backed up regularly. Daily paper control forms were used to record information on households in each cluster and were used for the purposes of data checking and processing.

Immunization coverage was measured using a recall survey and cross-checking immunization registers. The numerator was the number of children who had been immunized with the JE vaccine according to the recall survey or from checking immunization registers. The denominator was the total number of eligible children who participated in the study (including children whose immunization status was invalid or unconfirmed). Analyses were stratified by district. We also compared the sociodemographic characteristics between districts using both methods and with results from the recall survey only. Data from validation processes were analyzed descriptively after being weighted by sampling design and non-responses. Calculations of coverage were provided in terms of point prevalence, lower and upper estimates of prevalence using 95% confidence intervals [CIs], and 99% CIs to account for possible random errors [9]. All statistical analyses were carried out using STATA ver. 12 (StataCorp., College Station, TX, USA).

Results

We obtained information on children’s immunization status through both a recall survey and checking immunization registers in 6 districts (Tabanan, Gianyar, Denpasar, Badung, Klungkung, and Karangasem). Due to time constraints, data from 3 additional districts (Bangli, Buleleng, and Jembrana) were obtained through a recall survey alone.

Survey Characteristics

The percentage of clusters surveyed per district and the percentage of clusters with eligible subjects were both above 95% (Table 1). Out of a total of 5,733 households targeted for sampling, 5,630 (98.2%) were visited, of which, 2,193 (39.0%) were eligible for inclusion (Table 1). Of these households, field staff successfully interviewed someone from 2,075 of them (94.6%), but 118 respondents (5.4%) refused to participate. A total of 97 households (1.7%) were visited a second time without finding eligible subjects, or were not visited again due to the name of subjects not being recognized by the head of the sub-village, respondents having moved to other places, or the survey period having ended.

Table 1.

Cluster and household characteristics

District	Total cluster	Description of selected clusters			Description of households
		Planned	Surveyed (%)	Cluster with eligible subjects (%)	Households with eligible subjects (%)	Children
Bali	4,450	441	436 (98.9)	431 (98.9)	2,193 (39.0)	3,331
Tabanan	792	40	40 (100)	40 (100)	221 (42.5)	309
Gianyar	565	40	40 (100)	40 (100)	240 (46.2)	348
Denpasar	442	91	87 (95.6)	85 (97.7)	400 (36.0)	623
Badung	553	62	62 (100)	61 (98.4)	351 (44.0)	537
Klungkung	285	30	30 (100)	29 (96.7)	130 (33.3)	189
Karangasem	570	47	46 (97.9)	46 (100)	179 (29.9)	301
Bangli	353	30	30 (100)	30 (100)	178 (45.8)	273
Buleleng	620	71	71 (100)	71 (100)	369 (40.3)	552
Jembrana	270	30	30 (100)	29 (96.7)	125 (32.1)	199

Table 2 shows the differences in weighted coverage based on carers’ recall and validation from immunization registers. It shows that the lowest rate of recall was from mothers (93.2%), while the highest rate of recall was from other relatives or children (98.7%). After validation, the rate of coverage was lowest when immunization status was reported by the grandfather/grandmother (72.3%), while the rate of coverage was highest when immunization status was reported by other relatives or children (81.8%). The lowest and highest differences in percentage between immunization status according to carers’ recall and immunization registers at CHCs were, respectively, from other relatives or children (17.1%) and grandfathers/grandmothers (25.2%).

Table 2.

The gap in weighted coverage of the Japanese encephalitis immunization program by recall and by validation according to carer

Carer	Total sample (%)	Immunized children	By recall alone (%)			After validation (%)			Difference in percentage (%)
			Point	95% CI	99% CI	Point	95% CI	99% CI
Bali	3,331	3,146	93.8	92.8-94.9	92.4-95.2	74.9	72.8-77.2	72.1-77.9	20.1
Relationship with childrenb)	3,316	3,132	93.8	92.8-94.9	92.5-95.2	75.1	72.8-77.3	72.1-78.0	19.9
Mother	2,110 (63.6)	1,980	93.2	91.8-94.6	91.4-95.0	75.0	72.2-77.9	71.3-78.8	19.5
Father	933 (28.1)	887	94.3	92.4-96.1	91.8-96.7	74.7	70.4-78.9	69.1-80.3	20.8
Grandfather/grandmother	160 (4.8)	154	96.6	93.7-99.6	92.8-100	72.3	62.2-82.4	58.9-85.7	25.2
Other relative	113 (3.4)	111	98.7	96.7-100	96.1-100	81.8	71.3-92.2	67.9-95.6	17.1

CI, confidence interval.

Calculated based on children’s data, since 1 household may have more than 1 child.

Missing=15.

Gaps of Weighted Immunization Coverage Obtained through Recall and Record Methods

Table 3 shows the estimated coverage of the JE immunization program using the recall survey alone and after being validated against immunization registers. The weighted coverage of JE immunization obtained through the recall survey in Bali was 93.8% (95% CI, 92.8–94.9; 99% CI, 92.4–95.2). It decreased to 74.9% (95% CI, 72.8–77.2; 99% CI, 72.1–77.9) after being validated against immunization registers at CHCs. There were minor differences in estimates between the 95% CI and 99% CI. Six of the 9 districts achieved coverage of above 95% based on the results of the recall survey. One of the 3 remaining districts was above 90%, and the other 2 districts had coverage rates slightly below 90%. However, after being validated against immunization registers at CHCs, JE immunization coverage substantially dropped. None of the 6 districts achieved the 95% target according to immunization registers, and only 1 district was found to have a coverage rate above 90% (92.1%), with the lowest rate of coverage being in the capital city of Denpasar (58.1%). We identified that the coverage gaps between the 2 methods ranged from 6.5% in Klungkung district to 36.4% in Denpasar.

Table 3.

The gap in weighted coverage of the Japanese encephalitis immunization program by recall and by validation according to district

Districta)	Total sample	Immunized children	By recall alone			After validation			Difference in percentage (%)
			Point (%)	95% CI	99% CI	Point (%)	95% CI	99% CI
Bali	3,331	3,146	93.8	92.8−94.9	92.4−95.2	74.9	72.8−77.2	72.1−77.9	20.1
Denpasar	623	571	91.3	88.7−93.9	87.8−94.8	58.1	53.4−62.8	51.9−64.3	36.4
Badung	537	518	95.8	93.8−97.8	93.1−98.5	86.8	83.0−91.0	81.8−91.7	9.4
Tabanan	309	296	95.4	92.6−98.3	91.6−99.2	70.4	64.7−76.2	62.8−78.1	26.2
Gianyar	348	336	97.1	95.3−98.8	94.7−99.4	80.1	75.2−84.9	73.7−86.5	17.5
Klungkung	189	184	98.5	96.8−100	96.2−100	92.1	87.8−96.4	86.5−97.8	6.5
Karangasem	301	287	95.6	93.0−98.1	92.2−98.9	72.5	66.7−78.3	64.8−80.2	24.2
Bangli	273	268	98.8	97.6−99.9	97.3−100	-	-	-	-
Buleleng	552	506	89.5	86.5−92.6	85.5−93.5	-	-	-	-
Jembrana	199	180	88.7	83.3−93.9	81.6−95.7	-	-	-	-

CI, confidence interval.

Recall coverage was calculated from 9 districts, while the validation coverage was calculated from 6 districts.

Table 4 shows the demographic characteristics of respondents and children who participated in the study. It shows that the characteristics of respondents in terms of age, sex, and their relationship with the child, as well as the number of children aged 5 to 15 years old, were similar between districts with and without the validation process. However, respondents in districts where the reported immunization status was validated tended to be more educated, work in offices, and have more children aged 9 to 59 months. Meanwhile, children’s characteristics between districts with and without the validation process were similar in terms of age at immunization, sex, and education level. However, we found a higher rate of children born fourth to tenth in districts without validation processes, indicating that households in these districts had higher parity and more children.

Table 4.

Comparison of sociodemographic characteristics of respondents and children in districts with and without validation in Bali

Characteristic	District with validation	District without validation	Total	p
Respondenta)	1,430	645	2,075
Age (y), median (interquartile range)	39 (13)	38 (14)	39 (13)	0.064
Sex				0.364
Female	960 (67.1)	446 (69.1)	1,406 (67.8)
Male	470 (32.9)	199 (30.9)	669 (32.2)
Relationship with children				0.167
Father	871 (60.9)	415 (64.3)	1,286 (62.0)
Mother	411 (28.7)	182 (28.2)	593 (28.6)
Grandfather, grandmother/other	148 (10.4)	48 (7.4)	196 (9.4)
Education				<0.001
No schooling yet	53 (3.7)	31 (4.8)	84 (4.1)
Elementary school	244 (17.1)	236 (36.6)	480 (23.1)
Junior high school	245 (17.1)	156 (24.2)	401 (19.3)
High school	618 (43.2)	187 (29.0)	805 (38.8)
College or higher	270 (18.9)	35 (5.4)	305 (14.7)
Occupation				<0.001
Farmer	147 (10.3)	205 (31.8)	352 (17.0)
Seller	91 (6.4)	44 (6.8)	135 (6.5)
Housewife	390 (27.3)	172 (26.7)	562 (27.1)
Private staff	306 (21.4)	32 (5.0)	338 (16.3)
Entrepreneur	277 (19.4)	75 (11.6)	352 (17.0)
Labor	73 (5.1)	54 (8.4)	127 (6.1)
Other	146 (10.2)	63 (9.8)	208 (10.0)
Childb)	2,307	1,024	3,331
Age at immunization				0.591
9–59 mo	1,670 (72.4)	732 (71.5)	2,402 (72.1)
5–15 y	637 (27.6)	292 (28.5)	929 (27.9)
Sex				0.083
Female	1,092 (47.3)	518 (50.6)	1,610 (48.3)
Male	1,215 (52.7)	506 (49.4)	1,721 (51.7)
Education				0.682
No schooling yet	601 (26.1)	287 (28.0)	888 (26.7)
Preschool	195 (8.5)	84 (8.2)	279 (8.4)
Elementary	976 (42.3)	426 (41.6)	1,402 (42.1)
Junior high	535 (23.2)	227 (22.2)	762 (22.9)
Order in the family				<0.001
First-born	935 (40.5)	357 (34.9)	1,292 (38.8)
Second-born	838 (36.3)	346 (33.8)	1,184 (35.5)
Third-born	390 (16.9)	194 (18.9)	584 (17.5)
Fourth-born	108 (4.7)	80 (7.8)	188 (5.6)
Fifth-born	19 (0.8)	31 (3.0)	50 (1.5)
Sixth- to tenth-born	17 (0.7)	16 (1.6)	33 (1.0)

Data are presented as n (%) unless otherwise specified.

Calculated based on respondents’ data.

The Quality of Immunization Registers at CHCs

Overall, the number of visits to CHCs for the validation process ranged from 1 to 4 visits. The number of days required to complete validation ranged from 1 to 7 days per CHC. Register completeness scores ranged from 2.5 to 10 with an average of 7. Staff members were cooperative, with an average score of 7.6 out of 10.

Table 5 shows the number and characteristics of invalid records for each district. Two districts were found to have a small proportion of invalid records. However, the remaining districts were found to have a high proportion of invalid records, ranging from 14.1% to 34.5%. The proportion of missing registers was particularly high in 2 districts—Karangasem and Gianyar, followed by Klungkung and Denpasar. Missing names were a major issue in 4 districts (Karangasem, Gianyar, Badung, and Denpasar), while a missing immunization/vaccination mark was very common in Tabanan, Karangasem, and Denpasar. Many registers were unable to be evaluated due to CHC staff being unavailable or refusing to provide the register before the time for data collection had ended.

Table 5.

Characteristics of invalid records for each district in Bali

District	Child	Invalid records, n (%)	Missing register (%)	Missing name (%)	Missing date (%)	Missing mark (%)	Missing date and mark (%)	Cannot be determined (%)
Tabanan	309	82 (26.5)	2.4	2.4	9.8	52.4	6.1	28.0
Klungkung	189	12 (6.3)	16.7	0	8.3	0	0	75.0
Karangasem	301	64 (21.3)	20.3	26.6	3.1	23.4	0	32.8
Gianyar	348	49 (14.1)	26.5	20.4	8.2	18.4	2.0	30.6
Badung	537	37 (6.9)	5.4	35.1	2.7	5.4	0	56.8
Denpasar	623	215 (34.5)	11.2	14.4	6.5	0.9	0	67.9

Discussion

The JE mass immunization program in Bali is the first to be conducted in Indonesia. We found that the weighted coverage according to carers’ recall was 93.8%, still below the 95% target [13]. Following validation against immunization registers at CHCs, however, coverage decreased significantly to 74.9%. The greatest decrease in coverage was observed in Denpasar (58.1%).

WHO guidelines recommend the use of immunization registers at CHCs to validate immunization status obtained from carers’ recall to measure the true coverage of the vaccination program [7,11]. Previous studies have documented weaknesses in using carers’ recall alone for measuring immunization coverage. This suggests that written documents such as immunization registers can slightly increase the rates obtained from carers’ recall [12,19]. However, our findings suggest that the use of immunization registers at CHCs reduced the reported coverage of JE immunization by 6.5% to 36.4% when compared to the recall survey.

We argue that carers’ recall provides a more reliable estimate of the true coverage of JE immunization in Bali. Central to this assertion is the assumption that a recall-based coverage estimate does not lead to overestimation, as would occur if carers inaccurately reported the immunization status of children under their care. Over-reporting of immunization is unlikely in this context since our study examined a single mass vaccination program after an extensive and large-scale health promotion campaign prior to implementation that involved many stakeholders both from health and non-health sectors [13]. As a result, JE immunization was remembered by carers as a special event and was unlikely to be forgotten within the short period of time between vaccination and the study’s commencement, regardless of who reported a child’s immunization status. The survey was conducted 2 months after the vaccination program—a relatively short timeframe during which parents were unlikely to forget if their children were immunized—thus decreasing the possibility of recall bias. Moreover, Bali has been cited as having the highest coverage of basic and supplementary vaccinations in Indonesia [3,5,6,20]. As such, we strongly believe that the uptake of JE vaccination should have equally high coverage, if not higher.

Our study suggests that a validation process using immunization registers at CHCs is time-consuming, complicated, and may result in bias. In this study, SCs needed several days and multiple visits to CHCs, as well as help from enumerators, other SCs, and CHC staff, to complete the validation. We also found that immunization registers at CHCs were often fair or poor in quality, with some indicators and elements missing from the register. The use of unstandardized, paper-based immunization records across districts and health immunization posts [21,22], failure to use methods for recording immunization status recommended in WHO guidelines (name, date of immunization, and mark as evidence of immunization), and missing registers [7,22] are factors that can reduce the quality of immunization registers, making them less reliable for use as the standard for measuring the true coverage of an immunization program. Furthermore, some CHCs only report aggregate data in their immunization registers, and individual data cannot be traced [21]. Despite its subjectivity as an evaluation, we found that the cooperativeness of health staff at CHCs is one of several crucial factors that determined the success of the validation process. This could be another difficulty when evaluating the quality of immunization registers.

Poor immunization registers have been reported in other studies [23-28], including by the WHO [21]. The use of the 2-staged method recommended by the WHO to measure immunization coverage can lead to underestimation of the true immunization coverage, as suggested by our study. Other studies have shown that agreement between 2 measures of immunization status does not guarantee a more accurate analysis of immunization status [29]. Our study location was Bali, which is 1 of the most densely populated islands in Indonesia, with Denpasar City being its most populous district. The other 8 districts in Bali have varying population densities. This reflects the typical makeup of provinces across Indonesia, where most people are concentrated in major cities. In addition, the socio-demographics of Bali and its health programs mirror the national situation. With regard to vaccination programs, the overall and health system–specific contexts of Bali represent the Indonesian situation. However, Bali is considered a progressive province in Indonesia, and its residents have better access to health services than those in many other provinces. This suggests that many provinces in Indonesia will likely have similar or worse reporting and recording systems for health data, including low-quality immunization registers. Home-based immunization records in Indonesia have also been shown to be unsuccessful, with a high percentage of loss (63%) [30]. Another study found that the ownership rate of the Maternal and Child Health Handbook as a home-based immunization record in Bali was also very low (14.78%) [31]. Home-based records are also not reliable for the validation of immunization status by recall. These circumstances can affect the reported or estimated immunization coverage. Other countries across Asia and Africa may face similar challenges to Indonesia in terms of the quality of their health information systems.

An electronic immunization recording system has been developed and implemented in some countries [21,28]. However, such a system was implemented only as of recently in Indonesia (in 2019). Some challenges for implementing electronic recording and reporting systems have been documented in other studies [25,28,32]. A study from China reported that the use of electronic records still underestimated the true coverage of an immunization program compared to a recall survey due to human error in adding data to the electronic system [33]. The same situation has been observed in the USA regarding electronic recording and reporting systems for immunizations [25], as well as reported in 4 pilot studies in low- and middle-income countries [9]. Given these constraints, in a country with less developed health information systems, the 2-staged method suggested by the WHO for measuring immunization coverage requires further consideration. There should be alternatives for measuring immunization coverage tailored to the capabilities of local health systems and the quality of immunization registers to produce accurate estimates of immunization coverage.

A study in Pakistan examining the validity of carers’ recall against immunization registers showed that the recall method had fair sensitivity for detecting immunization status. However, this study supports the assertion that carers’ recall is a viable alternative for measuring immunization coverage [34], especially when a country’s health information systems are fragmented. Recall, however, cannot be avoided as an ascertainment method in surveys. Further study is required to better formulate recall questions and how to best implement adjustments for areas with recall data only [22].

In districts where the validation process was successfully conducted, respondents and children tended to be more educated, work in an office (as opposed to working in a blue-collar job), and have a higher quantity of children aged 9 to 59 months. Several studies have found that the level of education of a mother or carer is an independent factor for higher immunization coverage [35-38], while a carer working and having more children could decrease coverage [35,36]. In addition, these districts were mainly urban, where the overall uptake of vaccinations tends to be lower than in rural areas [35,39]. To some extent, those factors may indirectly influence the accuracy of an immunization reporting system due to, for example, failure to report immunizations or losing immunization cards [31,37]. We found a very large discrepancy between immunization status according to carers’ recall and immunization registers at CHCs when the immunization status was reported by grandfathers/grandmothers (25.1%). However, such instances accounted for a small portion of our sample (5.2%). This situation was not observed among other carers where the discrepancies were very narrow/small. At this stage, we assume that potential information bias among carers is low. These data suggest that, on the district level, registers may play more of a role in explaining these coverage discrepancies than individual factors. Therefore, validation data obtained from these 6 districts may not adequately represent the coverage across Bali.

Our study has several limitations. Location mapping prior to data collection was not conducted due to time constraints [9]. In addition, we encountered numerous cancellations of home re-visits and limited supervision during the validation process. Transient residents may also have not been included in the denominator data, which may have affected the results of our study. In addition, the register validation process might have provided better results if the study period was longer. There were also differences in the characteristics of districts with and without validation, the effects of which were beyond the scope of our study.

The coverage of JE immunization in Bali obtained using a recall survey was higher than was recorded in immunization registers. This discrepancy is likely due to inadequate recording and reporting systems at CHCs. Our study is not unique to Bali or Indonesia, but reflects the ongoing challenges of measuring the coverage of vaccination programs in many developing nations. The results of our study underscore the importance of developing and maintaining integrated recording and reporting systems to generate accurate estimates of immunization coverage. Recall surveys need to be carried out shortly after immunization to reduce the possibility of recall bias so the data can be used to assess the validity of immunization registers.