Digital health for real-time monitoring of a national immunisation campaign in Indonesia: a large-scale effectiveness evaluation.

OBJECTIVE: To assess the contribution of a digital health real-time monitoring platform towards the achievement of coverage targets during a national immunisation campaign in Indonesia.
INTERVENTIONS: A digital health platform was introduced to facilitate real-time reporting and data visualisation. Health workers submitted reports of children immunised each day by geolocation using mobile phones. Automated reports were generated for programme managers at all levels to enable early responses to coverage gaps.
METHODS: Risk profiles were generated for each district to assess precampaign immunisation programme performance. Digital health platform use and progress towards targets were monitored continuously throughout the campaign. Study outcomes were total coverage and time to achieve full (100%) coverage. Kaplan-Meier, Cox and linear regression analyses were used to estimate the associations and outcomes after adjusting for district risk profiles. A complementary qualitative assessment explored user experiences and acceptance through interviews with vaccinators and programme managers in provinces and districts selected through multistage random sampling.
RESULTS: Between August and December 2018, 6462 health facilities registered to use the digital health platform across 28 provinces and 395 districts. After adjusting for precampaign district risk profile and intracampaign delays due to vaccine hesitancy, districts with greater platform utilisation demonstrated higher coverage overall (R2=0.28, p<0.0001) and a shorter interval to achieving full coverage (>75% reporting compliance; Risk Ratio 15.4, 95% CI 5.8 to 40.6). Stronger effects were observed among districts experiencing implementation delays due to vaccine hesitancy. Results from 106 key informant interviews conducted in 6 provinces and 18 districts suggest high degrees of acceptability, ease of use and satisfaction.
CONCLUSION: A digital health platform introduced for real-time monitoring of a national immunisation campaign in Indonesia was feasible, well liked and associated with improved problem solving and programme performance, particularly among districts affected by vaccine hesitancy. TRIAL REGISTRATION NUMBER: ISRCTN10850448. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This large-scale effectiveness evaluation assessed the contribution of a digital health intervention to the achievement of immunisation coverage targets.

The assessment took place throughout the Indonesian archipelago and included areas of wide socioeconomic and geographical diversity with varying levels of infrastructure, capacity and connectivity.

The evaluation made use of both quantitative and qualitative data collected before, during and after the immunisation campaign, and accounted for baseline performance of district immunisation programmes.

While unanticipated issues of vaccine hesitancy affected progress in some areas and prolonged the campaign, they were overcome in most instances and adjusted for in the analyses.

Introduction

Indonesia has large numbers of unimmunised children. Vaccine preventable diseases including measles and rubella (MR) remain serious public health challenges.1 Prior to 2018, coverage with the first dose of measles vaccine had plateaued at 85%, with 60% of children having received the second dose.2 Measles outbreaks occurred frequently, with over 12 000 cases reported annually in the preceding 5 years.3 With the rubella vaccine not yet introduced, an estimated 15%–20% of pregnancies remained vulnerable to congenital rubella syndrome (CRS).4

Indonesia has committed to eliminating measles and controlling rubella and CRS by 2023. To achieve this ambitious aim, the country undertook a nationwide MR vaccination campaign that targeted over 67 million children between 9 months and 15 years old.5 The first phase of the campaign was conducted in 2017 on the densely populated Java island with 35 million children across six provinces.5 In 2018, the second phase, which is referred as the ‘MR campaign’ in this paper, aimed to cover the remaining 31.9 million children across 28 socioeconomically and geographically diverse provinces, comprising 395 districts over an estimated 6000 inhabited islands.5–8

To support implementation and overcome challenges associated with paper-based reporting in a wide geographical area, the Ministry of Health with the support of partners introduced a digital health platform known as RapidPro. The term digital health refers to the systematic application of information and communications technologies, computer science and data to support informed decision-making by individuals, the health workforce and health systems, to strengthen resilience to disease and improve health and wellness for all.9 During the MR campaign, the platform was used by health workers to send a text message/SMS (short message service) report of the number of children vaccinated each day using mobile phones. For each message sent, an automated response was sent to vaccinators and managers to report local progress against immunisation coverage targets. Programme staff were also able to visualise intracampaign coverage by facility, district or province through an online dashboard to identify areas where corrective action may be required.10

The digital health platform was initially introduced on a limited scale during the first phase of the national campaign. A process evaluation found it to be a useful tool to track progress, identify gaps and intensify outreach.11 A large-scale effectiveness evaluation12 was conducted during wider deployment of the platform in the second phase of the MR campaign with the aim of assessing its potential contribution towards immunisation coverage and programme performance.

Methods

Intervention description

The digital health platform was developed using UNICEF-designed open source software (RapidPro) that supports interactive messaging and allows information collected through mobile phones to be entered directly into a database. To ensure simplicity and limit smart phone requirements, the platform was designed to accept data through text message/SMS. Vaccinators used personal mobile phones to register their health facility which was linked to a known geolocation. At the end of each reporting day, vaccinators submitted a text message to report the number of children immunised. Data from all facilities were aggregated and visualised with an online dashboard at the district, provincial and national levels, respectively. Vaccinators and programme managers received an automated text message at the end of each day with feedback on progress towards coverage targets for their administrative level. The platform was administered at no cost to users (figure 1).

Figure 1

Real-time monitoring innovation used during the second phase of Indonesia’s MR campaign. MoH, Ministry of Health; MR, measles and rubella; SMS, short message service.

Provincial immunisation managers received a short orientation to the digital health platform at a national coordination meeting in advance of the campaign. District immunisation managers were subsequently introduced to the platform through face-to-face engagement in their respective provinces. Cascade training took place for health facility staff, supplemented by a 2 min instructional animated video disseminated through the commonly used social media platform, WhatsApp. Further details of the intervention are reported elsewhere.13

Study design

A large-scale effectiveness assessment was conducted using prospective observational design. Data collected during the campaign (1 August–31 December 2018) on platform use and study outcomes were analysed at the district level comparing varying degrees of implementation.

The trial was registered with the International Standard Randomised Controlled Trial Network.

Outcomes

The primary outcome of interest was immunisation coverage, including achievement of full coverage and time to achieve target thresholds. The secondary outcomes included the potential contribution of the platform towards early problem identification and the taking of corrective action to address challenges during the campaign.

Quantitative assessment

Table 1 profiles the range of quantitative indicators assessed with the source of data.

Table 1

Indicators included in the quantitative analysis

Indicators	Definition	Data source
District profile
Risk profile	District risk (low, medium, high and very high) based on assessment of immunisation coverage, surveillance quality and vaccine preventable disease case load.	WHO
Affected by vaccine hesitancy	List of districts recorded to ever stop MR campaign activities due to vaccine hesitancy.	MoH list
Digital health platform utilisation
Reporting days	Number of days when health facilities submitted their report to digital health platform during the first 2 months of the campaign (1 August–30 September 2018), averaged by district.	Digital health data
Reporting compliance	Proportion of health facility’s reporting days against eligible reporting days, averaged by district. Eligible reporting days are total days health facilities were expected to report in order to reach 100% coverage. Total days during the whole campaign (1 August–31 December 2018) are eligible reporting days for districts below 100% coverage.	Digital health data
Perceived usefulness
Help to achieve target	Survey respondents’ perception of digital health usefulness to achieve coverage (yes/no).	Mobile phone-based survey
Useful for problem identification	Survey respondents’ perception of digital health usefulness for problem identification (yes/no).	Mobile phone-based survey
Useful for corrective action	Survey respondents’ perception of digital health usefulness for corrective action (yes/no).	Mobile phone-based survey
Coverage
Total coverage	Proportion of children immunised against target children to be immunised by district.	Digital health data
Time reaching full (100%) coverage	Number of days districts take to achieve full (100%) coverage—the official target set by MoH to all participating districts during the second phase of the MR campaign.	Digital health data

MoH, Ministry of Health; MR, measles and rubella.

Reporting and coverage data from all registered users were obtained from the digital health platform through continuous monitoring during the campaign. To assess use, numbers of health facility’s reporting days and reporting compliance—defined as the ratio of observed over planned reporting days—were monitored. Coverage data were validated against information collected through manual reporting systems throughout the campaign. All submissions from registered facilities were included in the analysis.

Risk profiles were generated for each district using a standardised WHO Measles Risk Assessment Tool.14 The tool combines measures of immunisation coverage, surveillance activities and case load. These district risk profiles were used to adjust for potential confounding related to precampaign programme performance, including supply-side and demand-side factors, such as management, financing, numbers and capacity of health worker numbers, as well as commodity availability.

The potential contribution of the digital health platform towards identifying and addressing challenges during the campaign was assessed using mobile phone-based surveys. Requests for participation were sent to all registered users in the final month of the campaign through the same platform used to report immunisation coverage. Three yes–no questions were asked, namely perceived usefulness of the platform to achieve targets, whether the platform assisted in the identification of problems during the campaign, and whether the platform facilitated corrective action (table 1).

Qualitative assessment

A complementary qualitative assessment was conducted in parallel to generate insights on user experience and acceptability. This included indepth interviews with vaccinators and programme managers in 6 selected provinces and 18 districts using a multistage random sampling framework. Provincial stratification was based on (1) levels of mobile connectivity, (2) prior experience with digital health applications and (3) district immunisation programme risk profiles.14 Two provinces from each group were selected at random using probability proportional to size (number of districts by size). For each selected province, three districts were selected at random, with three health facilities chosen at random from four coverage groupings.

Indepth interviews were conducted face-to-face near the midpoint of the campaign in local language with respondents responsible for immunisation monitoring and implementation. These included programme managers at central, provincial and district levels, as well as vaccinators at health facilities. Six informants at the global level were also interviewed from key campaign partners including UNICEF, WHO, and Gavi, the Vaccine Alliance. The interview guide was designed to explore overall impressions of the digital health platform, ease of use and use-as-intended, and whether data collected through the platform informed problem identification and corrective action. Initial development of the guide was in English, with back-to-back translation to Bahasa Indonesia by local researchers. Informed verbal consent was obtained prior to all interviews. All interviews were audio-recorded, transcribed and coded.

Analysis

With high levels of decentralised governance in Indonesia, districts were the core administrative units responsible for implementation and reporting on campaign performance. To assess statistical associations between platform use and primary coverage outcomes, Kaplan-Meier survival analysis, Cox regression and linear regression were employed with the district as the unit of analysis. To assess the potential contribution to time to reach full coverage, survival analysis was calculated in reverse15–17 with achieving 100% coverage as the event. To address issues of vaccine hesitancy that emerged during the campaign,8 analyses were conducted separately for districts experiencing delays.

To assess associations between secondary outcomes (problem identification and corrective action) and coverage, a t-test was performed on individual responses from mobile phone-based surveys. Similarly, estimates were conducted separately for districts affected and unaffected by vaccine hesitancy. All analyses were performed with SPSS V.23.

The qualitative analysis followed a grounded theory approach.18 We avoided possible loss of meaning in translation by developing the initial codebook in English, which was then translated to the Indonesian language. Prior to coding of all transcripts, codes were tested on a subset of transcripts by two local native researchers. Discrepancies in codes and their scope were discussed, with updates to the codebook made accordingly. Transcripts were coded and analysed using NVivo (V.12, 2018; QSR International). Responses were triangulated across different types of informants and across sites to assess validity through the convergence of information.

Patient and public involvement

This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results.

Results

Utilisation of the RapidPro digital health platform

Between August and December 2018, a total of 6462 health facilities across 28 provinces and 395 districts were registered and reported at least once into the RapidPro platform. Reporting compliance was highest during the initial months (August–September 2018), with a gradual decrease towards the end of the campaign (December 2018) as targets were achieved. A total of 72.7% of facilities reported at least weekly into the system. Reasons for irregular or non-reporting included decisions by some local areas to conduct the campaign intermittently (only on select days) and/or to upload the data periodically; the fact that not all initially registered facilities were responsible for reporting; and mobile connectivity challenges. Additional irregular reporting resulted from 61 of 395 districts and 14% of facilities which stopped participating in the campaign at some point due to vaccine hesitancy (demand-related delays in vaccine acceptance).8 There was a high level of concordance between coverage reports through the digital health platform and final reports received through paper-based systems (<1% difference).

Associations between platform use and primary coverage outcomes were assessed from August to September 2018—the original timeline of the campaign. After adjusting for district risk profile, associations were observed between coverage and reporting frequency across all reporting districts (R2=0.28, p<0.0001) (online supplemental table A-1). Figure 2 profiles the correlations separately for districts affected and unaffected by vaccine hesitancy. While significant associations were observed in both groups, stronger correlations were observed among districts reporting ever stopping campaign activities due to vaccine hesitancy (R2=0.65, p<0.001) relative to those unaffected (R2=0.06, p<0.001).

Figure 2

Correlation of the average number of reported days/health facility and district coverage during the first 2 months of the campaign (1 August–30 September 2018). Y = district coverage and x= average number of reported days/puskesmas in the district.

Strong relationships were also observed between reporting frequency and compliance with the likelihood of achieving full coverage and time to achieve coverage thresholds. Districts with higher average reporting compliance were more likely to reach 100% coverage. The likelihood was significantly increased in districts with average health facility reporting compliance of 75% and above (Risk Ratio [RR] 15.4, 95% CI 5.8 to 40.6) (figure 3). Cox regression analysis yielded similar results, where the likelihoods of reaching full coverage were strongly correlated with reporting compliance (online supplemental table A-2). These relationships were also not influenced by district risk profile.

Figure 3

Reporting compliance and districts reaching 100% coverage by 31 December 2018.

Qualitative findings

A total of 106 interviews were conducted with 114 informants from 82 institutions. Overall, vaccinators cited high levels of acceptance. Training was deemed sufficient and the manual and short video guides were clear reference tools. The platform was easy to use, with mobile connectivity identified as the most common challenge. While this may have led to delays in reporting, most facilities found ways to overcome this. In some instances, facilities accumulated the data and reported when travelling to an area with better connectivity. Others informed their respective district managers who reported on their behalf.

The timeliness and ease of use of the platform were preferred over paper-based reporting and phone calls used in past immunisation campaigns and programmes. Health facility informants appreciated the daily reminders, which they said improved their motivation to report, creating a sense that their work was recognised and appreciated. Reports submitted through the platform went directly to programme managers at district, province and central levels. For some, this digital health ‘cross-check’ reinforced the importance of ensuring manual reports were correctly submitted. Informants also perceived RapidPro submissions reduced workloads as updates were automatically provided to relevant authorities. Data on numbers of immunised children recorded on mobile phones were also perceived as a useful backup system. Illustrative findings are highlighted in table 2.

Table 2

Informant perceptions of the digital health monitoring platform

Themes	Quotes
Overall utilisation
Integration into existing campaign reporting scheme	“We haven’t had a clear idea how well people in the field are able to operate and enter the data. But I think someday we can do it. Because there will be a verification process from District Health Office. They would open the data and identify what’s missing. Provincial Health Office would also be able to see districts’ performance. They would say ‘what’s wrong with your data?’ then the District Health Office would verify it.”*
	“Sometimes, their number was different between our recap and what we report in RapidPro. We would know the difference. For instance, we have 10 in RapidPro, while in manual it is written 15. From that data, we could re-check where we made the mistake.”*
	“Yes, it once occurred (difficulty in sending SMS). Perhaps it was due to lack of network, so it wasn’t successful. Sometimes the network isn’t available. It happened, but not every day. It was only one time, if I’m not mistaken. Then once the signal was available, I sent the accumulation.”†
Effects on reporting motivation	“With the help from RapidPro, first, we were able report quicker. Secondly, it motivated us to work faster. The data all had to be completed in certain hours. For us, it is more like a motivation. I would say to my colleagues, ‘Come on, the report all should be collected.’ It’s different with using the manual data where they usually procrastinate. They would say, ‘there’s still time, it can wait’. With this, we are becoming more well-organized and able to report faster.”*
Use as intended
Understanding of RapidPro purpose	“I think this RapidPro made it easier to understand information within the immunization campaign. So, we are able to compare the numbers with what facilities reported. This also facilitated the immunization officers in facilities to do daily reporting and to see the coverage.”*
	“[purpose of RapidPro] to report number of children immunized.”†
Use on problem identification, target achievement and corrective action	“See in this case, we obtained this data from immunization coverage in a province. It says 14 districts and 197 health facilities. This is [the number] that came into us. Meanwhile, we know exactly that we have 304 health facilities in the province, but only 197 reported. To confirm the others that had not reported, we gave them feedback through a WhatsApp group.”*
	”For that case, it’s what I said about the benefit, in MR campaign, we all have targets to achieve. We can see our achievements through RapidPro. Then we can analyse our progress. Automatically, we can give District Health Office officers feedback. So, the result is also coming rapidly. And when our coverage only achieved half the target, we give them time, say, 2 months to achieve the target. At least this has become our monitoring tool, whereas manual reporting takes a longer time. While RapidPro is able to provide us data daily rapidly.”*
	“Coverage achievement would depend on the health workers. On reporting, they are indeed helped [by RapidPro].”*
	“The challenge here is lack of coordination and support from cross sectoral stakeholders. Lack of support is what I mostly feel, especially from our own Regent (Bupati). Sometimes I envy districts that receive full support, where even the Regent is willing to see the condition directly in the field. Here, it’s difficult to gain cross-sectoral commitment when it comes to a new program. We held a meeting and invited them, but it was difficult to bring in the key person. Some of them just sent delegates. So, it is quite challenging to achieve the target when the pressure is minimal from the Regent and above level.”*
Satisfaction
Satisfaction	“On the last immunization campaign, we did not use RapidPro…I got confused at times. Sometimes program officers would say, ‘I have the data.’ But with RapidPro, I can do it (check the coverage) everywhere, even when I’m away…I’d still have the control.”*
	“If we have enough data to do validation, we can map our achievement based on villages, schools, and integrated health services post. We identify what is the target, how much we have achieved, what are the remaining numbers to be achieved, and the percentage. We could map them if the data is available. We currently don’t have those details in RapidPro. We are not able to do the mapping. There’s only the global data. It could only map coverage in health facilities. But below that, we could not identify where the coverage is low. We could not read that.”†

*Informant type: programme manager.

†Informant type: vaccinator.

SMS, short message service.

Both health facility vaccinators and programme managers highlighted that daily RapidPro updates on immunisation coverage triggered discussions through established social media networks. These updates improved local coordination, which better facilitated problem solving. This was reinforced by the online data visualisation platform, which fostered greater accountability within and between districts. Programme managers also found daily coverage updates from their districts to be helpful in regular progress reviews of campaign performance. However, informants also noted corrective action was not taken in all instances. Some emphasised that data were just one ingredient of success, with the achievement of targets dependent on coordinated efforts to address bottlenecks. While the platform was simple, some respondents would have liked to see more detailed information included on age and sex breakdown, as well as the ability to include reporting on perceived reasons for delays.

Quantitative effects: perceived usefulness of the platform and coverage

A total of 981 responses were received from mobile phone-based surveys on the usefulness of the platform (response rate ~10%). Overall, respondents felt that RapidPro helped to achieve targets (90.7%), improve problem identification (73.7%) and facilitate corrective action (72.6%). For districts unaffected by vaccine hesitancy, associations were observed between mean end-of-campaign coverage and the perceived contribution of RapidPro towards achieving targets and its use for corrective action (table 3). No association with problem identification was observed. Among districts affected by vaccine hesitancy, where campaign efforts had stalled, no associations were observed between mean coverage and secondary outcomes.

Table 3

Perceived usefulness of digital health platform and associations with end-of-campaign district immunisation coverage among districts affected and unaffected by vaccine hesitancy

	Total (N=981 respondents)	Responses	Residing in unaffected district (n=850 respondents)	District coverage
		Residing in affected districts (n=104 respondents)		Affected districts (n=44 districts)	P value	Unaffected districts (n=274 districts)	P value
Help to reach target
No	30 (3.1)	1 (1.0)	28 (3.3)	–	–	98.9 (80.4 to 117.4)	0.066
Yes	890 (90.7)	94 (90.4)	772 (90.8)	44.0 (35.2 to 52.8)		82.9 (80.4 to 85.4)
No responses	61 (6.2)	9 (8.6)	50 (5.9)
Helpful for problem identification
No	181 (18.5)	17 (16.4)	161 (18.9)	41.5 (15.6 to 67.3)	0.848	80.5 (70.4 to 90.5)	0.494
Yes	724 (73.8)	73 (70.2)	629 (74.0)	44.1 (34.4 to 53.7)		83.5 (80.9 to 86.1)
No responses	76 (7.7)	14 (13.4)	60 (7.1)
Helpful for corrective action
No	173 (17.6)	16 (15.4)	154 (18.1)	32.1 (22.6 to 41.6)	0.313	71.1 (59.4 to 82.7)	0.003
Yes	712 (72.6)	72 (69.2)	619 (72.8)	45.6 (35.4 to 55.8)		84.4 (81.9 to 86.8)
No responses	96 (9.8)	16 (15.4%)	77 (9.1)

Response data are n (%), with respondent as unit of analysis. Coverage data are mean % (95% CI), with district as unit of analysis.

No significant mean differences found among different risk profile and hence not displayed.

Discussion

We evaluated the contribution of a digital health platform deployed to support real-time reporting and feedback on the performance of a national immunisation campaign in Indonesia. The introduction of the platform at-scale was feasible, with high levels of user acceptance and ease of use. Feedback of local coverage data to vaccinators and programme managers created a sense of shared accountability. A high proportion of respondents felt the platform helped to identify and address local obstacles. While the campaign experienced challenges related to vaccine hesitancy, strong associations were observed between platform use and the achievement of coverage targets. These relationships were stronger in districts where vaccine hesitancy was an issue, suggesting that feedback from the platform stimulated corrective action to help districts overcome challenges. The observed contributions were independent of the precampaign performance of local immunisation programmes.

We opted for a large-scale effectiveness evaluation for several reasons. First, Indonesia’s MR campaign was an intense and time-bound national effort, requiring implementation and monitoring systems to be highly standardised. This afforded limited political or practical flexibility to adopt randomised evaluation approaches. Second, this design takes advantage of the continuous monitoring of intervention and outcome indicators generated through the platform. With large numbers of districts across a range of contexts, the dose–response analysis generated strengthens the plausibility of findings. Finally, the availability data on precampaign performance of district immunisation programmes allowed for important potential confounding factors to be addressed in the analysis.

While the platform deployed in this evaluation was the first application of a digital health application at-scale in Indonesia, the approach leveraged prior learning from recent small-scale pilots. These included digital innovations to improve access to immunisation services in urban slums, to improve access to healthcare among poor populations,19 to monitor malaria bednet distribution and to assess the status of water and sanitation programmes. Importantly, routine reporting in the health sector remains largely paper-based with a reliance on periodic submissions of aggregate data. There is currently no single national platform that supports seamless data entry, visualisation and feedback of critical health information to improve programme planning, budgeting and performance.20 Ongoing support through UNICEF has been helpful to embed digital innovations within the national data architecture, which is important for sustainability and wider application. The introduction of this system was intended to be complementary to paper-based reporting, and not a substitute.

A number of factors contributed to the adoption and effective use of the platform. First, as highlighted in a previous review,21 high-level endorsement by the Ministry of Health was a central prerequisite to deployment during the campaign. This support was informed by positive experience during 2017 across a more limited geographical area.11 Second, immunisation campaigns are by nature costly, time and resource-intensive and disruptive to routine services. In such a context, rapid problem identification with immediate course corrections is critical to achieving targets. Our findings suggest real-time data enhanced local coordination, with nearly three-quarters of respondents reporting the platform facilitated problem identification. There were also strong associations between taking corrective action and overall coverage among districts unaffected by vaccine hesitancy. Finally, Indonesia is highly digitally connected, with at least 72% of the population currently using mobile phones22 and 56% connected to the internet and social media.23 While poor network strength is a known barrier to digital health effectiveness in low-resource settings21 24–26 and was identified as an issue for some areas in this assessment, these challenges were generally overcome.

During Indonesia’s MR campaign, issues of vaccine hesitancy were a serious concern.8 The rapid identification of districts experiencing delays allowed for targeted social and political mobilisation, which facilitated campaign resumption in most instances. The availability of decentralised real-time data and visualisation tools was important for advocacy and an improvement over slower paper-based systems which aggregate data at the provincial-level only. Nonetheless, hesitancy was not overcome in several districts, highlighting the importance of parallel accountability and problem-solving mechanisms. These findings are echoed in a recent review suggesting immunisation programmes take time to accommodate digital innovations; that acceptance and quality improve progressively with experience; and that digital strategies can only be optimised once embedded within a wider set of mutually reinforcing system improvements.26

The adoption of digital health platforms by health providers and programme managers has sometimes been hampered by perceptions of dual reporting burdens and the potential for an increased workload by front-line workers. Similar to experience elsewhere, our research suggests health workers found simplified submissions via mobile platforms with real-time feedback to enable improved performance, allowing more convenient updates relative to paper-based systems.26 27 This reinforces important lessons from other low-income and middle-income contexts, where digital health platforms have been demonstrated to increase the timeliness and completeness of surveillance reports and immunisation records.26–28

This assessment had several limitations that are important to underscore. First, the study protocol did not anticipate issues of vaccine hesitancy which resulted in campaign extensions. Despite being adjusted for in the analysis, this was not specified in advance. Second, our analysis was restricted by data availability. Use and coverage data from the RapidPro platform facilitated nationwide district-level analysis. However, mobile phone-based survey data were only available for a subsample of respondents. Respondent numbers limited our ability to conduct district-level analyses on secondary outcomes, with mobile surveys also subject to potential recruitment bias. Finally, coverage estimates are subject to the availability of accurate denominators for eligible children. While we used both projections from the national census as well as locally generated denominators, there were discrepancies in some instances.

Conclusion

In summary, this study provides encouraging evidence that a digital health platform to monitor a national immunisation campaign in Indonesia was feasible, well liked and associated with improved programme performance and problem solving, particularly among districts affected by vaccine hesitancy. Its successful application across a diverse range of contexts with varying range of data literacy and connectivity underscores its generalisability within the country and elsewhere. Future applications of this approach to other campaigns, to surveillance activities and to routine programming are ongoing areas of work. Finally, digital innovations that combine supply-side decision support for health workers with efforts to improve demand-side awareness among programme beneficiaries are important areas to pursue in future programmes and research.