Effectiveness of the Diagnose-Intervene- Verify-Adjust (DIVA) model for integrated primary healthcare planning and performance improvement: an embedded mixed methods evaluation in Kaduna state, Nigeria.

OBJECTIVES: This study evaluates the real-world effectiveness of Diagnose-Intervene-Verify-Adjust (DIVA), an innovative quality improvement mode, in improving primary healthcare (PHC) bottlenecks impeding health system performance in Kaduna, a northern Nigerian state.
DESIGN: An embedded mixed method study design involving participant observation.
SETTING: PHCs in 23 local government areas of Kaduna state, Nigeria. PARTICIPANTS: 138 PHC managers across the state (PHC directors and programme managers in the 23 local governments). INTERVENTION: DIVA is a four-step improvement model in which 'Diagnose' identifies constraints to effective coverage, 'Intervene' develops/implements action plans addressing constraints, while 'Verify/Adjust' monitor performance and revise plans. PRIMARY AND SECONDARY OUTCOME MEASURES: The model, as adapted in Nigeria, is designed to evaluate and improve the availability of health commodities, human resources, geographical accessibility, acceptability, continuous utilisation and quality of four PHC interventions (immunisation, integrated management of childhood illnesses, antenatal care and skilled birth attendance).
RESULTS: 183 bottlenecks were identified by local government teams across all interventions in 2013. 41% of bottlenecks concern human resources. Geographical access and availability of commodities ranked least. Availability of commodities was the most improved determinant although among the least constrained, probably indicating skewed implementation of operational plans. 1562 activities were planned to address identified bottlenecks in the state, of which only 568 (36%) were completely implemented
CONCLUSION: Our study demonstrates that PHC planning using the DIVA model can potentially improve health system performance. However, effective implementation is critical and may require some central government oversight. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study uses an uncontrolled before and after design for assessing change in identified bottlenecks.

The design is constrained in attributing observed improvement to the intervention but was, however, most suitable given that all local governments in the state are implementing Diagnose-Intervene-Verify-Adjust.

Participant observation methods are prone to the Hawthorne effect.

Given that this is an embedded real-world continuous improvement intervention, the Hawthorne effect is a desirable if positive and sustained.

We found this design very useful and easy-to-use for lower level operational managers in low- and middle-income countries implementing/evaluating health system improvement initiatives, particularly in district/decentralised health systems.

Background

Introduction

Attaining universal health coverage (UHC) requires systems capable of ensuring that required interventions and services are effectively delivered to populations in need. Primary healthcare (PHC) provides an appropriate platform for integrating health system components towards UHC.1 While selective PHC services usually offer a quick win approach to disease control through verticalised interventions such as immunisation and malaria control programmes, integrated health system strengthening is being advocated as a more efficient and effective strategy for improved health system performance.2 3 Substantial knowledge exists on causes of ill-health alongside interventions to address them, however how to implement these interventions is far less understood.4 This is typical of complex health systems characterised by many actors and interest groups (governmental and non-governmental) operating at different levels.5 6 Lower levels of decentralised health systems are known to pose implementation challenges in low- and middle-income countries (LMICs) largely due to weak capacity for governance as well as political constraints.6–8 Several methods have been implemented to improve PHC performance in LMICs. Most common of these approaches include supportive supervision, mentoring, tools and aids, quality improvement methods and coaching.9 These approaches have largely focused on health facilities and are mostly vertical (focusing on one intervention).7 9 A systematic review of quality improvement strategies for improving maternal and child health in LMICs revealed a limited number of studies involved integrated approach to PHC.10 In this paper, we examine the effectiveness of an innovative model for improving integrated PHC performance as implemented in Kaduna, a northern Nigerian state.

The Diagnose-Intervene-Verify-Adjust (DIVA) model was designed for systematically identifying and resolving health systems bottlenecks, particularly in decentralised LMIC contexts.11 The intervention theory, as described in the methods section, postulates that implementation of DIVA should improve health systems performance by facilitating prompt identification and response to health systems bottlenecks. However, in real-world contexts, the extent to which this can be achieved is not established. Implementation theorists believe that the effectiveness of interventions is subject to contextual variations, thus similar results may not be expected across all settings.12 Consequently, this paper evaluates the effectiveness of DIVA as a model for improving health system performance through integrated PHC operational planning in Kaduna, Nigeria.

The DIVA model was introduced by Unicef in nine sub-Saharan African countries (Botswana, Democratic Republic of Congo, Mali, Malawi, Niger, Nigeria, Sierra Leone, Uganda and Zambia).13 While there has been some success in terms of DIVA providing evidence for policy making in these countries,11 13 there has been no effectiveness study of DIVA implementation to our knowledge. ‘Effectiveness’ here refers to how the intervention affects health systems outcomes when implemented in real-world (uncontrolled) environment. Thus, it reflects how contextual factors (such as the political economy, implementer capacity and discretion, stakeholder influences, etc) may enhance or inhibit the expected outcome of the intervention.14 15 Given the contextual diversities of these countries particularly with respect to political governance (all except Nigeria have a centralised government),8 our study provides unique insight to implementing DIVA in a politically decentralised system. Thus, lessons from this implementation research may be used to inform adaptation of DIVA in other similar politically decentralised LMIC contexts.

Study context

Nigeria’s health system is anchored on a PHC structure known as the Ward Health System (WHS).16 The WHS is an adaptation of the WHO’s District Health System where political wards of the local governments (LGs) (the smallest political governance units) double as the operational units for PHC.17 This design aims to strengthen community participation in health system governance as envisaged in the Alma-Ata PHC declaration of 1978.18 The WHS operates a primarily community-based management structure which functions through the Ward Development Committees (WDCs) and Village Development Committees (VDCs) at the ward and village levels respectively. Each WDC consists mostly of chairmen of the various VDCs under its jurisdiction while the VDCs consist of respectable members of the communities and representatives of various community-based interest groups and strata (eg, women, youth and occupational groups).19 The functions of these committees include identifying, planning interventions and mobilising resources to meet health and social needs of the communities, supporting and monitoring the implementation of work plans and activities of community-based and health facility workers, support maintenance of the Ward Health Centres as well as serve as the linkage between the communities and the government/other partners.17 Although the DIVA model is designed to strengthen this model,11 in practice community participation has been rather very passive.20

This study was conducted in Kaduna, one of Nigeria’s 37 states/territories. Kaduna is home to about 7 915 48721 people distributed across 23 local government areas and 255 political wards. About 1523 (98%) of the 1560 health facilities in Kaduna are PHC facilities.22

Methods

The DIVA model

DIVA is a variant of the Plan-Do-Study-Act cycle designed to improve performance in complex macro-health systems, particularly the district health system.11 23 The model aims to build the capacity of LG health managers to assess, analyse, act and be accountable for equitable service delivery and to strengthen decentralised health systems. Four principles guide the implementation of DIVA- strong government leadership and ownership; flexibility and adaptability to local programmes, building on what exists; community involvement and engagement and supportive (technical and financial) process from civil society, private sector and development partners.11 Kaduna state implements DIVA in quarterly cycles for each LG and annual reviews are conducted at the state level which feed into existing health sector annual budgeting and planning cycles. These phases overlap in practice.24 The phases of DIVA are described below:

Diagnose phase

A modified Tanahashi model, described by O’Connell and Sharkey, is used during the diagnose phase of the PHC reviews to assess health systems bottlenecks at the LG level.13 25 Six determinants of quality or ‘effective coverage’ of interventions are measured: availability of essential health commodities, availability of human resources, accessibility of distribution points for the interventions (geographical access), initial utilisation (acceptability) of interventions, continuous utilisation (contact) of interventions and quality of interventions delivered. The first three determinants represent the supply side while the others are demand side determinants. Health systems bottlenecks are identified as constraints to full functioning of each determinant which impacts on effective/quality coverage of the intervention.13

A pre-implementation training of LG PHC management teams on bottleneck analysis methodology was conducted in 2012.26 Thereafter, during the PHC reviews we provide technical guidance to the teams to carry out this diagnosis using routine data (and data from other sources eg demographic and health surveys) analysed on a MS-Excel based Bottleneck Analysis tool developed by the National PHC Development Agency (NPHCDA).24 The LG teams are then supported to identify bottlenecks inhibiting optimal effective coverage with respect to each tracer intervention. This training and technical support was embedded into the DIVA process to ensure optimal implementation fidelity by mitigating lower-level discretion commonly observed in multi-level governance contexts.7 27–29

The tool calculates coverage for each determinant using routine data inputted from the health management information system as numerators and the national population census data as denominators. Graphs are automatically developed for analyses. The difference between target coverage and observed coverage for each determinant is identified as a measure of the bottleneck.13 24 25 30

Using techniques and tools like brainstorming, ‘5 Whys’ technique,31 affinity and driver diagrams, the teams are guided to identify immediate, proximate and distal causes of identified bottlenecks (S1). This step is known as root cause/causal analysis (figure 1).

Figure 1

Framework for implementation of the ‘Diagnose’ phase (adapted from Unicef and Management Science for Health (MSH)11).

Intervene phase

Following step 1, LG teams are guided to identify plausible solutions to these bottlenecks with specific quality/coverage targets for each quarter. Thereafter, proffered solutions are converted into action plans (S1) which are incorporated in annual health operational plans and budgets following stakeholder consultation (eg, the State PHC Development Agency chief executive, LG chairperson and ministries of health and finance). This is followed by implementation of the planned activities.

Verify phase

During this phase, the implementation of planned activities is monitored through existing supportive supervision, monitoring and evaluation mechanisms year-round. This helps early detection of deviation or lag while also ensuring implementation fidelity of planned activities. Quarterly diagnostic cycles further enhance verification. Supportive supervision includes quarterly visits from state to LGs and from LGs to health facilities to supervise and aid frontline health workers as they implement activities planned out for each intervention.

Adjust phase

At any point during verification, implementation challenges identified are addressed to ensure interventions are carried out as planned and are on track towards attaining targets within stipulated time frames.

DIVA implementation process in Kaduna

Eight state and LG teams (PHC managers) were trained on the methodology of and guided on the Diagnose and Intervene steps to obtain a baseline in 2013. These managers include the PHC directors and programme managers at both levels. Thereafter technical support was provided during the annual reviews conducted in March each year. The reviews focused on determinants for availability of health commodities, human resources for health, geographical accessibility, initial utilisation, continuous utilisation and quality coverage of four PHC tracer interventions (immunisation, Integrated Management of Childhood Illnesses (IMCI), antenatal care and skilled birth attendance). These tracer interventions were selected by stakeholders in the health sector using the following criteria: ’Data is available for the six coverage determinants; the tracer is an internationally recommended intervention with proven and quantified efficacy; the tracer is representative of other indicators within its intervention group in terms of facing similar health systems constraints at the chosen service delivery level, for accurate assessment of costs in overcoming bottlenecks’.32

LG health management teams (PHC directors and programme managers) were coached and supervised as they utilised routine data collected from their LGs to analyse health systems constraints, develop and implement evidence-informed plans to address them and improve performance. Each of Kaduna state’s 23 LGs have six PHC managers (one PHC director and five programme managers) involved with DIVA. A total of 138 across the state therefore constituted the study population. Field notes were taken, and reports developed from each LG detailing bottlenecks identified, causal analysis and action/operational plans for each year. Draft reports were circulated to all LGs for confirmation and inputs towards the final documents (which were also endorsed by all LGs).

Study design

We evaluated the PHC reviews in Kaduna using an embedded a mixed method approach. The mixed approach was complementary such that qualitative data provided context and content to the changes in indicators measured using quantitative methods.33 Embedded research involves a partnership between researchers and decision-makers to facilitate co-production of knowledge towards evidence-informed policy and practice.34 35 A moderate participant observation approach was employed such that the embedded researchers mostly acted as process facilitators for the state and LG teams.36 EAE and NI were played dual roles as embedded researchers and national-level policy makers between 2013 and 2016. The process aligned with Howell’s participant observation phases such that the researchers first established rapport with the participants (subnational PHC managers), then embedded themselves in the field as process facilitators, recorded observations as fieldnotes and finally conducted framework and narrative analyses of the information gathered.37 Detailed findings from these analyses are beyond the scope of this paper, but are published in other sub-studies within this research series.7 20 26 However, collated outputs from the process inform the process description below as well as the ‘causal analyses and planned action table’ in the results section.

Furthermore, we quantitatively evaluated the effectiveness of DIVA by measuring improvements on selected indicators (table 1) developed by the NPHCDA and from the Health Management Information System.24 32 Each LG team was instructed to identify two priority bottlenecks (constrained determinants) per intervention (one supply side and one demand side) using the MS-Excel bottleneck analysis tool described earlier. We followed up on actions planned to tackle these identified bottlenecks and their impact on reducing them between 2013 and 2016. This was done by extracting data from programme reports on how the planned activities were implemented. The level of implementation was categorised in a three-point ordinal scale (not implemented, partly implemented and completely implemented). The implementation data collated from all LGs was analysed using a pie-chart.

Table 1

Selected process and outcome measures used for the primary healthcare reviews

Determinants	Immunisation	IMCI - malaria	Antenatal care	Skilled birth attendance
Commodity	Percentage of health facilities without stock out of pentavalent vaccines in the reporting period	Percentage of health facilities without stock out of ACTs in the reporting period	Percentage of health facilities offering ANC services without stock out of iron-folate supplements in the reporting period	Percentage of health facilities offering delivery services experiencing no stock out of delivery kits in the reporting period
Human resources	Percentage of health facilities with at least two trained vaccinators	Percentage of health workers trained in the management of childhood illnesses	Percentage of ANC service providers who have been trained in focused antenatal care	Percentage of maternity staff trained in basic emergency obstetric care
Geographical access	Percentage of population living within 5 km radius from immunisation service points (health facilities and outreach locations) *	Percentage of population living within 5 km radius of health facilities offering services for management of childhood illnesses*	Percentage of population living within 5 km radius of health facilities offering ANC*	Percentage of population living within 5 km radius of health facilities offering basic delivery services*
Utilisation (initial)	Percentage of children under the age of one who received pentavalent vaccine dose 1	Percentage of children under the age of 5 years having fever and using health facility services for the management of childhood illnesses	Percentage of pregnant women attending at least one ANC service	Percentage of deliveries in health facilities
Continuity (continuous utilisation)	Percentage of children under the age of one who received pentavalent vaccine dose 3	Percentage of children under the age of 5 years having fever and that were treated with ACT	Percentage of pregnant women who attended four ANC visits †	Percentage of mother/infant pairs who received at least one follow-up home visit within the first month after delivery
Quality (effective coverage)	Percentage of children fully immunised (with all doses of routine vaccines) before their first birthday	Percentage of children under the age of 5 years having fever, who were tested with rapid diagnostic tests and treated with ACT	Percentage of pregnant women who had four ANC in a timely manner in accordance with focused ANC guidelines	Percentage of deliveries receiving postnatal check-up within 48 hours at health facilities

*Nigeria’s government policy aims to provide services within 5 km/1 hour walking distance of communities.8 50

†Nigeria’s policy (in line with WHO) recommends that all pregnant women attend at least four ANC sessions.50 51

ACT, artemisinin-based combination therapy; ANC, antenatal care; IMCI, integrated management of childhood illness.

Coverage gap analyses were conducted on the prioritised bottlenecks to monitor improvements. The median performance values of each constrained determinant identified across the 23 LGs was determined both at baseline and post-implementation. Median values (rather than mean) were used as measures of central tendency to eliminate the effect of isolated extreme performance in some LGs. The change in bottlenecks are presented in bar charts. Health system performance measures were monitored using time-series charts which show trends in monthly coverage of selected outcome measures between 2013 and 2016. A trend analyses was conducted to monitor for possible improvements. The selected measures were based on proximate indicators from Nigeria’s National Strategic Health Development Plan considered to have indirect but proximate relationships with the DIVA indicators.38 These are trends in measles vaccination coverage, management of febrile Illnesses using rapid diagnostic testing (RDT) and artemisinin-based combination therapy (ACT) for confirmed malaria cases, and attendance at antenatal care clinics.

Patient and public involvement

This study did not directly involve the patients, but rather an ecological approach using data from the Health Management Information System. The participatory action research approach involved the health system managers and planners who utilised the routine (secondary) data for decision making.

Results

The results from this evaluation represent findings aggregated from 1523 out of 1560 health facilities distributed across all 23 LGs in Kaduna state.

Trend of selected performance measures based on routine data

First, we present trend analyses of selected outcomes; vaccination: measles vaccine coverage, antenatal care (ANC): coverage and average visits per client, IMCI: fever cases tested with RDT and clinical malaria treated with ACT. These outcomes were selected because they are proxy indicators of health systems performance associated with the various interventions in the national PHC review guidelines.24

Figures 2-4 show that although improvement is observed with some outcome indicators (measles vaccination and fever cases tested with RDT), decline was observed with others.

Trend - coverage of measles vaccine (monthly).

Trend - outcome of management of febrile illnesses (monthly). ACT, artemisinin-based combination therapy; RDT, rapid diagnostic testing.

Trends - outcomes of antenatal care.

Bottleneck analysis and action plan implementation

We now present findings from the baseline diagnosis in 2013 with priority ranking of bottlenecks and root cause analysis across the various interventions and causal analysis. Thereafter we demonstrate on bar charts how these bottlenecks have changed between 2013 and 2016. Lastly, we illustrate implementation gaps in action plans developed to address the bottlenecks.

A total of 183 bottlenecks were identified by the 23 LG teams across all tracer interventions in 2013. Figure 5 shows the distribution of these bottlenecks across the modified Tanahashi determinants. Most (41%) bottlenecks identified relate to human resource challenges, while geographical access and availability of commodities ranked least (4% respectively). Figure 6 disaggregates these findings per tracer intervention using Pareto charts.

Figure 5

Distribution of identified bottlenecks across determinants of health system performance.

Figure 6

Pareto charts showing priority ranking of determinants with identified bottlenecks for each tracer intervention.

Table 2 shows the common bottlenecks for each intervention as well as their probable causes as identified by the LG teams. Online supplementary material S1 contains an example of integrated evidence-informed operational plans developed by the teams during the PHC reviews to address these constraints.

Table 2

Causal analysis of commonly identified bottlenecks and planned actions by local governments

Interventions	Bottlenecks identified	Immediate causes	Underlying causes	Structural causes	Managerial causes	Planned actions
Immunisation	Difficult access to vaccination sites	Closure of some health facilities, dispersal of some communities	Insecurity	Inadequate security for health workers and communities in conflict areas	Poor response from the policy makers on security matters.	Advocacy to policy makers and security outfits
	Sub-optimal demand for vaccination services	Lack of information on the importance and safety of vaccines	Sociocultural apathy to vaccination	Weak functionality of community development committees	Weak community engagement	Community mobilisation
Integrated management of childhood illnesses	Inadequacy of health workers trained in the management of childhood illnesses	Lack of training of health personnel	Poor release of funds for planned training	No sustainable funding structure for primary healthcare	Weak prioritisation of health workers' capacity building by policy makers	Advocacy to policy makers on prioritising health workers’ capacity building
	Inadequate ACT treatment of children under 5 years diagnosed with malaria	Inadequacy of health workers trained in the management of childhood illnesses; poor adherence to clinical guidelines	Limited release of funds for planned training	Inadequate distribution health facilities providing ACT treatment across communities	Limited commitment and support to implementation of work plans by higher level of government	Advocacy to policy makers on prioritising health workers’ capacity building
	Low demand for services by the communities	Patients' preference for self-medication	High ACT stock-out rates, diminishing confidence of community in availability of quality services at PHC facilities	No sustainable funding structure for primary healthcare	Inadequate supervision of health workers	Community sensitisation on the demerits of self-medication, supportive supervision for PHC; advocacy for sustainable funding mechanism
	Inequitable distribution of health facilities providing IMCI	Inadequate siting of facilities given population distribution	Weak planning of PHC expansion projects		Political influence on health facility siting	Advocacy to policy makers on prioritising equity in PHC expansion
Antenatal care	Limited coverage of ANC services	Low quality of services	Inadequacy of health workers (re)trained on focused ANC, inadequacy of (free) MCH commodities		Inadequate funding for training	Advocacy to policy makers on prioritising health workers’ capacity building
	High ANC dropout rates	Negative staff attitudes towards patients	Little or no training on service delivery	Weak supportive supervisory system	Inadequate funding for supervisory visits	Advocacy for improved funding
Skilled birth attendance	Weak delivery of BEOC	Weak technical capacity of frontline workers in BEOC	Inadequacy of health workers trained in BEOC; inequitable distribution of trained health workers	Weak funding mechanism for training	Weak political will	Advocacy for improved funding
	Few follow-up home visits of mother/infant pairs who within the first month following delivery	Lack of commitment from the health personnel	Weak motivation of health workers	Unavailability of technical and financial support for health workers' home visits	Weak supportive supervision	Advocacy for improved funding
	Limited patronage of health facilities for delivery services	Preference for home delivery	Sociocultural	Inequitable geographical access (distance) to facilities providing 24 hours skilled delivery services		Health education and support to community; advocacy for equitable distribution of health facilities

ACT, artemisinin-based combination therapy; ANC, antenatal care; BEOC, basic emergency obstetric care; IMCI, integrated management of childhood illness; MCH, Maternal and Child Health; PHC, primary healthcare.

Figure 7 measures the difference between 2013 (baseline) and 2016 median values of determinants for each intervention in LGs that identified them as priority constraints. These values exclude performance in LGs that did not identify priority constraints in the respective determinants (table 3). Thus, the measurements (figure 7) represent improvement (or otherwise) on identified bottlenecks following development of action plans to address them. While variable levels improvement can be observed in most determinants, geographical access to routine immunisation and antenatal services reduced. This may have resulted from the closure of facilities and dispersal of communities following security challenges in some LGs as identified in table 2.

Figure 7

Bar charts showing changes in median values of indicators for each intervention between 2013 and 2016* (* some determinants were not identified by the teams as having priority bottlenecks in 2013).

Table 3

Local governments that identified priority constraints in the respective determinants for each intervention in 2013 (n [%])

Intervention	Determinants	Number of LGs identifying bottlenecks (n [%])
Immunisation	Commodities	4 (17%)
	Human resources	13 (57%)
	Geographical access	5 (22%)
	Initial utilisation	2 (9%)
	Continuity	7 (30%)
	Quality	15 (65%)
Integrated management of childhood illnesses	Commodities	1 (4%)
	Human resources	20 (87%)
	Geographical access	1 (4%)
	Initial utilisation	2 (9%)
	Continuity	12 (52%)
	Quality	9 (39%)
Antenatal care	Human resources	22 (96%)
	Geographical access	1 (4%)
	Initial utilisation	2 (9%)
	Continuity	5 (22%)
	Quality	16 (70%)
Skilled birth attendance	Commodities	2 (9%)
	Human resources	21 (91%)
	Initial utilisation	16 (70%)
	Continuity	5 (22%)
	Quality	2 (9%)

LG, local government.

Figure 8 shows a gap in implementation of activities aimed at addressing bottlenecks. A total of 1562 activities were planned to address identified bottlenecks in the state, in which only 568 (36%) were completely implemented. Only 6 out of 23 LGs fully implemented at least 50% of planned activities. Implementation gaps occurred across all interventions, however, as shown on table 2, implementation was not a priority bottleneck with routine immunisation, which is the most government and donor supported intervention.

Figure 8

Implementation of planned activities.

Discussion

Our findings reveal that the DIVA model was helpful in identifying bottlenecks and developing evidence-informed action plans to address them. Interpretation of the findings are, however, guided by the understanding that while DIVA is implemented at LG level, these outcomes are measured at state level (which are aggregates of possible effects in all LGs).

Challenges were observed, particularly regarding execution of planned activities to address the bottlenecks. Funding constraints resulting from poor political will/government ownership (particularly at the LG level), was identified as a major factor underpinning implementation gaps. These mirror implementation challenges with DIVA in Ghana and Uganda.30 39 In addition to these, data availability to populate the Tanahashi determinants constrained implementation in Uganda. Generally, health systems bottlenecks identified were similar across these countries, save for some context specific constraints discussed below.

Human resources for health was identified as the most constrained determinant across all interventions except routine immunisation where it ranked second to quality. Common human resource challenges identified by the LG teams include weak capacity to deliver the interventions as well as poor motivation and commitment of frontline workers to service delivery. In addition to funding and management constraints, this may have contributed significantly to the poor quality of services which ranked second. In some LGs skilled human resources, where available, were inequitably distributed (being concentrated in larger urban or semi-urban facilities).

Geographical access and availability of commodities were the least constrained determinants identified. Security challenges (including terrorism and ethno-religious conflicts) constrained geographical access to services (particularly immunisation and IMCI) in some communities. Whereas Kaduna is not known as a hotspot for terrorism, findings from this research have shown that isolated security challenges have significant effects on the health system. Most responses to terrorism (including health sector) are currently concentrated in the North-Eastern states of Adamawa, Yobe and Borno. Consequently, there is need for a more widespread assessment and response to the effects of terrorism and conflicts in Nigeria beyond the known hotspots.

Inequitable siting of health facilities was also cited as a constraint. A well-known reason for this is the tendency for politicians to influence the siting of developmental projects (including health facilities) based on quest for political capital as against equity needs of the population. Thus, while health services are relatively inaccessible to some communities, others with greater political influence have more facilities than needed even though many of these function sub-optimally.8

Utilisation of services was mainly constrained by demand side factors such as sociocultural norms and poorly functional community development committees (which are supposed to provide a link between facility managers and the users of services). Efforts have been made by the national government to revitalise these committees through community Participatory Learning and Action programmes,40 however a more sustainable framework for operationalising these committees will require states and LGs to play more significant roles in engaging communities in recognition of their vital role as part of the health system.6 Studies in Kenya, Tanzania and Nigeria have shown that communities can play active roles in service delivery and project implementation, health promotion and advocacy and resource mobilisation and management.41–43

Planned actions to tackle these bottlenecks mostly revolved around advocacy and community mobilisation, indicating that majority of the solutions were beyond the immediate responsibility or capacity of the LG health teams. This probably explains the weak implementation of planned activities. Across all interventions, improvement in human resources was minimal even though it was the most constrained determinant. The challenges with human resources for healthcare are not new in Nigeria. However, most efforts to address them have focused on increasing their numbers and distribution, particularly Community Health Extension Workers.44 This approach is not unusual in LMICs given that maldistribution and high attrition rates are common human resource challenges in these countries.45 For example, maldistribution of health workers following punitive transfers and lobbying for posting are common concerns that have affected equitable access of clients to services in India, Pakistan, Mali and Kenya, which staff audits and re-distribution have attempted to solve.33–36 In addition to these, our findings suggest that attention should also be given to the capacity of available health workers to deliver specific services effectively. Thus, it is not sufficient to have adequate cadre of staff, efforts should be made to ensure these frontline workers get trained and retrained on the various PHC interventions to ensure quality. Clinical mentoring has also been employed successfully in Malawi, Uganda and Tanzania to strengthen human resource capacity.9

Availability of commodities was the most improved determinant even though it was among the least determinants cited as constrained. This pattern probably indicates skewness in implementation of operational plans as government and donors may prefer to intervene in this determinant to sustain supplies as against human resources, which require high costs and locally sensitive training. Availability of commodities was not identified as major constraint for ANC. Furthermore, there was improvement in IMCI geographical access in the LGs that identified it as being a bottleneck. The reverse was the case with ANC and routine immunisation where access worsened. This may have been due to insecurity which led to closure of some health facilities in some LGs. These two interventions were probably more affected because they also include community-based outreach services, which declined as reported by participants.

Whereas some indicators for interventions like malaria and routine immunisation (which have strong donor support) improved, ANC indicators slightly declined. This may indicate that managers skew implementation towards donor interests by not prioritising other less supported interventions like ANC. Consequently, action plan implementation tends towards verticalisation contrary to the design of DIVA.

The PHC reviews are yet to be fully institutionalised as implementation is largely dependent on external support and contextual factors such as security, financing and political will. The successful implementation of the Verify-Adjust phase is hinged on a strong monitoring and evaluation system. However, our findings reveal that this system is weak, as reported in previous studies.7 Integrated supportive supervision has been relatively stronger with vertical programmes like malaria and HIV/AIDS due to donor funding requirements.46 Although donor support has contributed to strengthening health systems in LMICs and improving PHC services in some cases, concerns have been raised about their negative impact on integrated systems strengthening.47 For example, increase in HIV/AIDS funding has been observed to stagnate funding for reproductive health. Further, accusations have arisen that scarce personnel are being drawn off other health services by offers of better-paying opportunities in HIV/AIDS programmes.47 48 We thus recommend a robust government coordination of PHC to improve the effectiveness of DIVA.

While many of the findings are not new, particularly as relates to human resources for health being a major health system challenge in LMICs, including Nigeria,8 44 45 the PHC review presents a new approach to evidence-informed solutions using DIVA. Successful implementation of DIVA is guided by four principles - strong government leadership and ownership; flexible and adaptable to local programmes, building on what exists; community involvement and engagement and supportive (technical and financial) process from civil society, private sector and development partners.11 Whereas, Nigeria’s integrated governance structure may have played an important role in sustaining implementation to date,7 dependence on donors as primary technical and funding sources for implementation may threaten future sustainability of the initiative.

A limitation of our study is that the magnitude of change observed may not reflect the magnitude of impact DIVA has on health system performance. This is because DIVA implementation is at the LG while this study aggregates findings to state level, consequently, outstanding results from high performing LGs are potentially obscured by low performing ones. However, given that our study focuses on state-level effectiveness of implementing the model (as against efficacy or impact evaluation), our findings are reflective of the potential of DIVA to improve health system performance within a decentralised context. Further, our analysis utilised median values as against means, thus eliminating the effect of possible extreme values from some LGs. While the use of routine data may be considered a limitation of DIVA owing to data quality concerns, our study focuses on change in coverage as against actual coverage thus potentially eliminating some known quality deficits. For example, the denominators for coverage in routine data are based on expected values projected from Nigeria’s last census in 2006, but observed coverages sometimes exceed 100% signifying a possible change in population dynamics over the past 10 years. However, this is unlikely to affect our findings as we analyse changes in coverage rather than the absolute coverage. Further, Nigeria’s electronic routine data platform and the PHC bottleneck analysis tool possess internal quality control mechanisms.24

This study uses an uncontrolled before and after design for assessing change in identified bottlenecks. Consequently, it is constrained in attributing observed improvement to the intervention. This design was, however, most suitable given that all LGs in the state are implementing DIVA. Using LGs in other states as controls will potentially introduce complex confounders given the dissimilarities in contexts, governance and other characteristics across states. However, the time series charts used in analysing health system outcomes support the interpretation of our findings. Further, a follow-up evaluation of contextual influences on DIVA outcomes using the Model for Understanding Success in Quality has been conducted to develop a theory of attribution that takes into account other factors.20

We found this design very useful and easy-to-use for lower level operational managers in LMICs implementing/evaluating health system improvement initiatives, particularly in district/decentralised health systems. Although desirable, we were unable to get sufficient reliable data to conduct a time series analysis for facility deliveries as proxy outcome indicator for skilled birth attendance due to gaps in routine data collection. As with all participant observation research, this study is prone to the Hawthorne effect. However, given that this is an embedded real-world continuous improvement intervention, the Hawthorne effect is a desirable as part of the intervention if positive and sustained.

Conclusions

Our study demonstrates that PHC planning using the DIVA approach model can potentially improve performance in decentralised health systems like Nigeria’s if implemented well. However, effective implementation will require some degree of central oversight and ownership. While an integrated approach to improving health system performance is ideal, this may not always be practicable owing to the complexity of the health system. However, our study has shown that integrated planning and some aspects of implementation are possible and potentially effective if structural and managerial bottlenecks are addressed. We view that while integrated financing structures recommended by Nigeria’s ongoing PHC under one roof reforms policy may improve effectiveness of DIVA, political considerations have constrained its implementation to date.46 49 However, to improve health systems performance, a governance model that empowers the LGs technically, politically and financially will have to evolve, requiring the engagement of stakeholders at different levels of government as well as different sectors of the economy given that most challenges to performance observed are outside the immediate jurisdiction of the LG health authorities. This inference is supported by our follow-up context and process evaluations of DIVA implementation.20 26

We recommend the tools used in this study (S1) for real time diagnosis, improvement and evaluation of health systems performance when implementing this model.