Data for decision making: using a dashboard to strengthen routine immunisation in Nigeria.

Availability of reliable data has for a long time been a challenge for health programmes in Nigeria. Routine immunisation (RI) data have always been characterised by conflicting coverage figures for the same vaccine across different routine data reporting platforms. Following the adoption of District Health Information System version 2 (DHIS2) as a national electronic data management platform, the DHIS2 RI Dashboard Project was initiated to address the absence of some RI-specific indicators on DHIS2. The project was also intended to improve visibility and monitoring of RI indicators as well as strengthen the broader national health management information system by promoting the use of routine data for decision making at all governance levels. This paper documents the process, challenges and lessons learnt in implementing the project in Nigeria. A multistakeholder technical working group developed an implementation framework with clear preimplementation; implementation and postimplementation activities. Beginning with a pilot in Kano state in 2014, the project has been scaled up countrywide. Nearly 34 000 health workers at all administrative levels were trained on RI data tools and DHIS2 use. The project contributed to the improvement in completeness of reports on DHIS2 from 53 % in first quarter 2014 to 81 % in second quarter 2017. The project faced challenges relating to primary healthcare governance structures at the subnational level, infrastructure and human resource capacity. Our experience highlights the need for early and sustained advocacy to stakeholders in a decentralised health system to promote ownership and sustainability of a centrally coordinated systems strengthening initiative.

Poor routine immunisation data quality has been a long-standing problem for Nigeria. The District Health Management Information System version 2 (DHIS2) Routine Immunisation Dashboard Project was implemented to address this challenge on a countrywide scale.

The DHIS2 RI DashboardProject contributed to improvement in completeness of routine health management information systems reports from 53% to 81% over a 3-year period. Timeliness of the reports improved substantially across the country in the same period.

Consistency in the implementation of agreed policies and strategies at subnational levels has a strong potential for stimulating an enduring improvement in health systems performance.

Background

The quality of data available for healthcare decision making has direct implications for the life and well-being of people.1 2 Availability of reliable data has for a long time been a challenge for Nigeria.3 4

Routine health data are widely considered unreliable and of poor quality.5 Poor investment in health information systems have led to challenges like weak human resource capacity; data infrastructure deficits; weak monitoring and supervision; and non-existent or ineffectual feedback on data quality and use.6 7

The uncertainty around health data in Nigeria is most stark with routine immunisation (RI) data. Decision makers and stakeholders are often confronted with conflicting administrative coverage figures for the same vaccine across different routine data reporting platforms and surveys as shown in table 1. Coverage numbers vary considerably between District Vaccination Data Management Tool (DVDMT), District Health Information System version 2 (DHIS2) of the National Health Management Information Systems (NHMIS) and surveys.8–10

Table 1

2016 National coverage for Penta 3, OPV3 and measles vaccine across major data sources

Vaccine	2016 coverage by source (in percentages)
	NICS 2016–2017(children age12–23 months)10	DHIS2(children under12 months)	DVDMT(children under12 months)
Third dose of pentavalent vaccine (Penta 3)	33	74.2	105
Third dose of oral polio vaccine (OPV3)	33	78.5	104
Measles vaccine	42	78.5	104

DVDMT, district vaccine data management tool; NICS, national immunization coverage survey.

The need to streamline reporting systems and platforms led to Nigeria adopting DHIS2 as the platform for routine health data reporting through a resolution of the 56th session of the National Council of Health (NCH) in 2013.11 The resolution meant that all health programme needed to transit to DHIS2. At the time, RI data were being reported on the DVDMT—a Microsoft Excel-based tool developed by the WHO—while DHIS2 was being used for reporting NHMIS or primary healthcare (PHC) data.

The NCH 2013 resolution coincided with the development of the National Routine Immunization Strategic Plan (NRISP) 2013–2015.8 NRISP was developed to improve RI service delivery. To monitor the implementation of NRISP, the Accountability Framework for Routine Immunization in Nigeria was developed.12 13 The accountability framework had several indicators, including those relating to immunisation sessions, vaccines and logistics, which were not available on DHIS2.

The aforementioned challenges, among others, led to the initiation of DHIS2 RI Dashboard Project. The overarching purpose of the project was the improvement of data quality and use of data for decision making. Specific objectives were to:

Develop, pilot and scale-up an RI dashboard on DHIS2 for monitoring of key RI indicators.

Accelerate national deployment of DHIS2 for PHC data management and use.

Provide information for better targeting of enhanced supportive supervision.

Improve use of good quality RI data for decision making at all levels.

Process

Figure 1 is a pictorial representation of the project phases and associated activities.

Figure 1

Project process and timelines. DHIS2, District Health Information System version 2; LGA, local government area.

Preimplementation

Formation of a District Health Information System version 2 Technical Working Group

At the start of the project, a Technical Working Group was formed to oversee day-to-day implementation activities. The TWG, which was cochaired by representatives of the National Primary Health Care Development Agency (NPHCDA) and the Federal Ministry of Health had representatives of development organisations as members to ensure interagency collaboration. One of the initial tasks embarked on by the TWG was to develop an implementation framework for the project. The framework outlined preimplementation, implementation and postimplementation activities.

Advocacy and prerapid assessment

Given the fundamental changes this project would bring to RI data management system in Nigeria, it was necessary to secure the broadest stakeholder engagement possible prior to implementation. At the national level, these engagements were made through the existing working groups including the Routine Immunization Working Group and the Monitoring and Evaluation (M&E) Working Group. In order to translate this to subnational levels, multistakeholder advocacy and needs assessment missions were incorporated into preimplementation activities. These missions had the objective of securing commitments and promoting ownership and leadership by the states and local governments prior to any implementation activity.

After the state-level meetings, a joint needs assessment visit with each state team was conducted to four local government areas (LGAs) and four health facilities (HFs) within each LGA. LGAs were selected on the basis of their last full year performance on the national instance of the DHIS2 on domains such as RI coverage, dropout rates and reporting rates. Two of the best performing and two of the worst performing LGAs on those criteria were selected. Within each selected LGA, two high and two low patient-volume HFs were selected for assessment. The LGA and facility visits were necessary to tailor interventions to the needs of each state.

Review of routine immunisation tools

The NHMIS currently uses both paper-based tools and electronic forms to report data. At the start of the project, the NHMIS data tools could not be used to report on all vaccines, especially vaccines introduced after the last review of NHMIS tools in 2013. In addition, there were no tools available to record other core indicators that support the RI activities such as supportive supervision; funding for RI; cold chain and logistics; and sessions planned versus those conducted for fixed and outreach sessions. To overcome these challenges, the NHMIS Supplementary Form (version 2014) and Health Facility Monthly Vaccine Utilization Summary Form were introduced.

District Health Information System version 2 routine immunisation dashboard development

It was identified that supportive supervision was not being regularly carried out largely due to inadequate funding.14 To efficiently use limited funds, there was a need to prioritise the HFs and LGAs to be visited for supervision. A customised RI dashboard for the states and LGAs was developed to closely monitor the indicators and, by extension, the progress of RI activities at LGA, state and national levels. This dashboard was meant to be reviewed regularly to identify problematic HFs in the LGAs for further corrective action and follow-up.

Human resource capacity building

Capacity building of healthcare workers was central to the project. Health worker training within the project was set up in a way that more experienced personnel were first trained and subsequently deployed to train others. Prior to project commencement, key personnel from both government and partner organisations were identified and trained at DHIS2 academies to enhance their capability to render technical support. This group formed an initial pool of facilitators for the training at subnational levels during implementation. They also formed part of the project monitoring team.

Pilot phase

A pilot phase was conducted for the purpose of refining processes and providing evidence for advocacy to funders and stakeholders. For DHIS2 RI Dashboard Implementation Project, Kano State was selected for the pilot. Activities at the pilot included planning, logistics, development of a training module and a training of trainers. The Kano pilot was conducted between November 2014 and February 2015. The lessons learnt during the implementation of each component helped to improve the scale-up to other states.

After the initial training in Kano, the state was monitored for a period of 3 months to check for improvements in the reporting rates for completeness and timeliness. Thereafter, PHC workers in the state were trained on data quality and use supportive supervision. This was intended to build their capacity to assess and improve the quality of data generated at the facility level and use the same for decision making.

Implementation

Training module development

A draft training module, which was developed for the pilot phase, was expanded and further refined with lessons from the pilot phase. This iterative process continued well into the implementation phase to adapt it appropriately to the peculiarities of the states as they came on board

National training of trainers

During the implementation phase, several rounds of national training of trainers (NToTs) were conducted to increase the pool of facilitators. The first NToT had two participants from each of the six geopolitical zones of Nigeria, as well as participants from the national level and development organisations. Subsequent NToTs had the same category of participants as the first and two additional state personnel: one from the state ministry of health and the other from the state PHC Board.

State and local government area training

Health workers responsible for immunisation records and PHC data management at different levels were trained on the revised RI data tools and DHIS2. To keep the training focused, health workers were trained only on aspects of the project that had direct relevance to their daily responsibilities, whereas federal, state and LGA personnel were trained on both the revised RI data tools and DHIS2, health facility staff were trained only on the revised RI data tools. Data entry on DHIS2 in most states is configured in such a way that data is collected with paper forms at the health facility and monthly summaries are sent to the LGA for entry onto DHIS2 by the LGA M&E officer with support from other LGA health team members.

Postimplementation

To sustain support for the states after training has been conducted, a dedicated DHIS2 Implementation Officer was deployed to each state for a period of 3 years. The IO is to work to promote state ownership of the project and ensure that local capacity is built to sustain the gains beyond the period of support. In addition to technical support, the LGA teams were provided laptop computers to aid the data entry process. Postimplementation activities also include data quality and use workshops as well as targeted supportive supervision using data from the dashboard.

Project outcome

The DHIS2 RI Dashboard Implementation project was a massive undertaking. As shown in table 2, as of the end of 2016, nearly 34 000 health workers of different categories and administrative levels had been trained.

Table 2

Number of personnel trained on the revised RI data tools and DHIS2 from 2014 to 2016 across national, state and LGA

Year	State	National training of trainers	State-level training	LGA-level of health facility workers	Development agency personnel	Total
2014	Kano	24	132	3036	102	3294
2015	Enugu and BauchiAkwa-Ibom	102	224	6155	58	6539
2016	Nasarawa, Oyo,Sokoto, TarabaNiger, AbiaOsun, ImoDelta, PlateauRivers and KadunaKwara	528	941	22 431	117	24 017
Total trained across all levels						33 850

DHIS2, District Health Information System version 2; LGA, local government area; RI, routine immunisation.

Since the inception of the DHIS2 RI dashboard project in 2014, there has been a substantial improvement in completeness and timeliness of reports from HFs across Nigeria. Most of this improvement is directly attributable to the project, which was the largest singular initiative for promoting uptake of DHIS2 for reporting routine health data in Nigeria during the period.

Completeness of reports measures the proportion of HFs submitting their routine HMIS reports on the DHIS2 platform. Low completeness rates hamper the reliability and utility of the data because decision makers cannot trust it to be a true reflection of the situation in the location and the period covered. As shown in figure 2, completeness of reports before the commencement of the project was 53%. By 2017, completeness of reports had increased to over 80%.

Figure 2

National report completeness rate on District Health Information System version 2, 2014–2017, Nigeria.

Timeliness is a measure of the proportion of HFs submitting their HMIS reports at the stipulated time on DHIS2. Figure 3 shows the status of timeliness of reports in the second half of 2014 with only one state scoring 80% or more and 14 states scoring 50%–80%. By the first half of 2017, 10 states scored 80% or more, while just six states scored below 50% (figure 4).

Figure 3

Timeliness of reporting for June–December 2014, all states, Nigeria.

Figure 4

Timeliness of reporting for January–June 2017, all states, Nigeria.

Regarding RI, the project led to a marked improvement in the availability of coverage data from HFs in every region of Nigeria. Figure 5 shows that the national composite coverage reports for the third dose of pentavalent vaccine (Penta 3) improved from 56% in 2014 to 84% in 2017.

Figure 5

National Penta 3 coverage on District Health Information System version 2, 2014 and 2017, Nigeria.

The deployment of the DHIS2 RI Module has led to improved ownership of RI data by states and LGAs as they now have real-time access to their data rather than the previous situation where they had to obtain RI data from the DVDMT through the WHO. Use of RI data for decision making has also improved through regular monthly review of the DHIS2 RI dashboard, which summarises the performance of LGAs or HFs on selected indicators. The review helps managers prioritise HFs for supportive supervision and other support to improve overall performance.

Overall, there has been an improvement in availability and timeliness of actionable health information to decision makers.

Challenges

Governance

Implementing a large-scale national initiative such as the DHIS2 RI Dashboard Project in a development context is usually fraught with challenges. As project implementation was centrally coordinated, it was necessary to secure the commitment of subnational governments (states and LGAs) at all stages. While some states were slow to prioritise the project, implementation in others was impeded by the evolving healthcare governance landscape brought about by the Primary Health Care Under One Roof (PHCUOR) initiative.15 PHCUOR required that the responsibilities for PHC be transferred from the ministry of health, local government health departments and ministry of local governments to a state PHC development board in each state. The transition meant that lines of authority were not always clear: there were problems around who made the decisions on who got trained and which agency of government would lead the process. The foregoing led to delays. Sustained high-level advocacy to the states ensured that these challenges were eventually resolved.

Internet access

The project was designed in such a way that participants at the LGA-level training are required to conduct real-time data entry into the web-based DHIS2 platform. The availability of stable internet access was a major constraint in many training locations. In those locations, training had to be conducted on a demo platform. Post-training, without stable internet connectivity, data clerks may have difficulties with data entry and analysis. This challenge could cause delays in data reporting and ultimately discourage continuous use of the dashboard.

Human resources

Availability of personnel with sufficient skills in computer use posed a challenge to the project. A substantial part of training resources were expended on providing basic computer appreciation training to participants. In addition, project scale-up had to be phased due to the inadequacy of facilitators at the national level to support the state level trainings. Furthermore, labour disputes resulting in strikes by health workers disrupted the training in some states. The foregoing led to delays and frequent adjustments to the project implementation timeline. The location of many HFs in hard-to-reach communities and the lack of incentives for timely delivery of completed summary forms to the LGAs for entry on the electronic platform mean that timeliness of reports and promptness of feedback will continue to be a challenge.

Funding

Funding for the project was largely donor dependent. Even though efforts have been made to ensure project sustainability, there are concerns regarding the capacity and willingness of subnational governments to continue investing in supportive supervision, training and infrastructure maintenance that are vital for the project. A planned midterm review on the project could not happen due to funding constraints and scheduling difficulties.

Policy recommendations

The experience and challenges of implementing the DHIS2 RI Module Project require some policy changes or modifications that would fundamentally impact the quality of data generated and its use for decision making. The following are recommendations for improving the impact of this project:

Full implementation of Primary Health Care Under One Roof policy

The PHCUOR policy is intended to bring PHC under one state-level governance structure (the state primary healthcare board (SPHCB)) to institutionalise management integration and decentralisation.16 At present, no state has fully implemented the PHCUOR agenda.17 In some states, rather than have the state PHC board located within the health ministry, it is located in the office of the state governor. It is necessary for NPHCDA and other stakeholders to sustain advocacy and dialogue to ensure that the principle of one management; one plan and one M&E for PHC is maintained in all states of Nigeria. Because SPHCBs are expected to be responsible for all aspects of immunisation programming at the state level, their proper establishment is vital to ensure ownership and sustained improvement in use of DHIS2 RI data for decision making.

Implementation of accountability framework

The existing accountability framework for RI in Nigeria is yet to be implemented, largely due to lack of political will at all levels.13 The framework was intended to institutionalise a reward and sanctions mechanism for outstanding work and poor performance/fraud, respectively. The reluctance to implement the framework reinforces the current situation where health workers are not held to account for their actions. It is expected that full implementation of PHCUOR would ease the process of implementing the accountability framework as all PHC workers in each state will be brought under one management body. This will make measures to improve data quality—like the DHIS2 RI dashboard Project—easier to implement and manage.

Appointment of computer-literate personnel as local government area monitoring and evaluation officers

The evolution of the data management landscape has brought about increasing computerisation of processes. Many M&E officers especially at the LGA level have been slow to acquire computer literacy and are struggling to cope with the trends. Going forward, it is important to ensure that all personnel appointed as M&E officers are skilled in the use of computers. A memo to this effect may need to be presented to the National Council on Health to institutionalise the recommendation (table 1).

Conclusion

Through the DHIS2 RI Dashboard Project, Nigeria has deployed an innovative approach to enhancing PHC immunisation data and using the results for decision-making and health systems performance improvements.

Like many complex health system interventions, the sustainability of the gains of the DHIS2 Dashboard Project depends on the extent to which frontline health workers and managers value, adopt and own the tools and procedures introduced. Decision makers must also be willing to invest in the infrastructure and the continuous development of the human resources that are central to the project. For every last child to be immunised, every immunised child has to be correctly counted.