Monitoring maternal and newborn health outcomes in Bauchi State, Nigeria: an evaluation of a standards-based quality improvement intervention.

OBJECTIVE: This study assessed the correlation between compliance with set performance standards and maternal and neonatal deaths in health facilities.
DESIGN: Baseline and three annual follow-up assessments were conducted, and each was followed by a quality improvement initiative using the Standards Based Management and Recognition (SBM-R) approach.
SETTING: Twenty-three secondary health facilities of Bauchi state, Nigeria. PARTICIPANTS: Health care workers and maternity unit patients. MAIN OUTCOME MEASURES: We examined trends in: (i) achievement of SBM-R set performance standards based on annual assessment data, (ii) the use of maternal and newborn health (MNH) service delivery practices based on data from health facility registers and supportive supervision and (iii) MNH outcomes based on routine service statistics.
RESULTS: At the baseline assessment in 2010, the facilities achieved 4% of SBM-R standards for MNH, on average, and this increased to 86% in 2013. Over the same time period, the study measured an increase in the administration of uterotonic for active management of third stage of labor from 10% to 95% and a decline in the incidence of postpartum hemorrhage from 3.3% to 1.9%. Institutional neonatal mortality rate decreased from 9 to 2 deaths per 1000 live births, while the institutional maternal mortality ratio dropped from 4113 to 1317 deaths per 100 000 live births.
CONCLUSION: Scaling up SBM-R for quality improvement has the potential to prevent maternal and neonatal deaths in Nigeria and similar settings.

Background

About 289 000 maternal deaths occur annually worldwide, and Nigeria accounts for about 14% of these [1]. In Nigeria, the estimated maternal mortality ratio (MMR) of 576 deaths per 100 000 live births reported in 2013 was higher than the 545 deaths per 100 000 live births recorded in 2008 [2] and fell far short of the Millennium Development Goal of 250 deaths per 100 000. Neonatal mortality declined from 48 deaths per 1000 live births in 2003 to 37 in 2013 [2]. However, the burden remains unnecessarily high.

One of the critical strategies for reducing maternal and newborn morbidity and mortality is ensuring that every woman is assisted during labor by a skilled birth attendant (SBA) who can prevent or treat life-threatening complications or facilitate referral to a higher level of care. Evidence-based interventions to avert maternal and newborn mortality are known, but achieving optimal coverage of these interventions has been challenging [3]. In Nigeria, only 36% of births occur in a health facility, and only 42% of postpartum women and 16% of newborns receive a checkup [2]. Quality of maternal and perinatal care is among inter-related safety and quality concerns existing in developing countries, indicating need for recognizing safety and quality gaps and devising strategies for collecting quality data for evidence-based interventions [4].

Tara Tancred et al. [5] while evaluating perceived quality of maternal and newborn care in Southern Tanzania reported that, though 93% of surveyed women were self-assured in availability of health care workers in health facilities, less than 50% described interactions with them as supportive and respectful.

Lack of trust in providers and poor quality of service have been cited as important reasons for not delivering in a health facility or using other maternal and child services [6], reflecting the need to improve perceived quality of care. Poor quality of care at health facilities is a challenge to maintaining patient safety and is associated with increased rates of maternal and newborn mortality and morbidity [7-9]. Few quality improvement studies for maternal and newborn health (MNH) have been published from Nigeria or similar settings. In Tanzania, lack of quality care at health facilities has been argued to be a major barrier to receiving services, outweighing lack of transportation, mother's lack of knowledge or difficulty getting to a facility [10].

Quality improvement includes (i) selecting the most appropriate interventions that are evidence-based and tailored to address specific reasons for a failure to provide quality services and (ii) creating a governance structure that assigns tasks and monitors selection and implementation of specific interventions and evaluation [8, 11]. Though the processes of producing good-quality health services are essential to the services’ effectiveness, such processes are poorly documented, affecting promotion of high-quality services [12]. According to Kinney et al. [13], ‘Gaps in measurement of quality of care also affect the ability to identify and reduce such quality gaps’.

Studies that focus on establishing the relationship between assessment of the actual performance of health facilities and use of various quality improvement strategies are believed to be crucial [14]. Raven et al. [15], after review of various quality improvement strategies for MNH in developing countries, found insufficiency of a particular tool, methodology or approach alone to achieving expected quality of care improvement and emphasized the need for studies that document processes and provide evidence of the effectiveness of such interventions.

The quality of maternal and newborn care services in health facilities of Bauchi state, northeast Nigeria is below standard, while MMR and neonatal mortality rate (NMR) are among the highest in the country [16]. This article documents the experience of applying the Standards-Based Management and Recognition (SBM-R®; Jhpiego, Baltimore, MD, USA) approach to quality improvement for MNH services in Bauchi state. It examines whether increased compliance with set performance standards was associated with improved maternal and neonatal outcomes.

Methods

Through the support of the USAID-funded Targeted States High Impact Project (TSHIP), the Bauchi state Ministry of Health (MOH) introduced Jhpiego's SBM-R approach at all 23 secondary health facilities in the state.

Setting and sample

All 23 secondary health facilities from the 20 local government areas (LGAs) in Bauchi state were included. We examined trends in (i) facility achievement of SBM-R performance standards, (ii) the use of evidence-based service delivery practices and (iii) related MNH outcomes. The facilities are evenly distributed across the state and are all government-affiliated. Annual delivery caseloads ranged from 120 to 1440 per facility and each facility had at least one medical doctor and four midwives.

Intervention

The SBM-R process follows four basic steps (Fig. 1); more detailed information and comparison with other quality improvement initiatives are available elsewhere [17]. Each standard is divided into a series of specific tasks, known as verification criteria. The MOH, TSHIP, Abubakar Tafawa Balewa University Teaching Hospital, College of Nursing and Midwifery and MNH implementing partners in Bauchi state adapted Jhpiego MNH performance standards for Bauchi. The standards were pre-tested and finalized. An SBM-R training of 69 frontline health workers (at least two from each intervention site) was conducted in stages. The trained staff members, with the support of TSHIP/USAID and MOH staff, oriented facility managers to SBM-R and trained and coached other service providers on the tools and desired outcomes. A five- to seven-member team was formed at each facility to lead the SBM-R process. The team, with technical assistance from TSHIP, conducted a baseline assessment, analyzed results, then analyzed each identified gap or problem using the ‘Why Tree’ approach by asking team members ‘why’ the gap or problem developed and turning the answer into another ‘why’ question. The findings guided activity planning and implementation to address gaps identified. Each assessment was followed by development of a detailed action plan to improve service quality. The plan included identified problems, interventions, persons responsible and an expected timeline of completion.

Figure 1

The four steps of the SBM-R® approach.

Each health facility's team monitored progress toward the standards through follow-up assessments, monitoring trends in service statistics and health outcomes. Program staff members recognized and rewarded achievements when a facility achieved targets for SBM-R standards, and this further incentivized performance improvement and contributed to sustainability. The intervention was grounded in a logical framework for achieving desired MNH outcomes based on the performance improvement framework developed by the Performance Improvement Consultative Group, a collaborative group of representatives of USAID and USAID-funded cooperating agencies (Fig. 2).

Figure 2

Steps in the performance and quality improvement process.

Ethical considerations

The Bauchi State Ministry of Health and the Bauchi State Health Research Ethics Committees (HREC) approved this study. Providers and clients gave their informed consent to participante.

Data sources

The assessment and monitoring tool used for this study included a mix of record reviews, observations of client–provider interactions, structured interviews and direct observations on provision of evidence-based practices such as essential newborn care (ENC), active management of third stage of labor (AMTSL) and management of complications using live patients and/or by model-based simulation. Follow-up assessments and measurement of trends for the variables were conducted every 12 months.

During both baseline and follow-up assessments, service data were collected from the 23 secondary health facilities for analysis. Data were extracted from existing maternity registers, supplemental facility data collection forms, the national health management information systems and observations. The same data collection team participated in both assessments. For health facility achievement of SBM-R performance standards, we calculated the percentage of performance standards for which all the verification criteria were achieved in a particular area.

Output and outcome variables

The program measured the level of compliance with set SBM-R performance standards to assess areas including focused antenatal care (nine standards); normal labor, childbirth and immediate newborn care (17 standards); postpartum care (six standards); management of antenatal, intrapartum and postpartum complications (13 standards); infection prevention (23 standards); health facility management (16 standards); and drug supplies management (six standards). The assessment tool is filled out immediately by filling in ‘Yes’ in the corresponding columns if the item exists or is performed as it is described, ‘No’ if the item does not exist or is performed incorrectly or incompletely, ‘Not Applicable’ when the item requires a condition that does not exist, or ‘Not Observed’ when the standard was not observed or not performed. In the comments column, potential reasons why the item does not exist or is performed incorrectly or incompletely are summarized. To achieve a performance standard, a health facility must meet all of the verification criteria for that standard. The mean percentage of MNH standards achieved by facilities implementing SBM-R was calculated. Among facility births, we calculated the percentage of women receiving AMTSL and newborns receiving ENC.

Health outcomes (impact)

We assessed the following outcomes for mothers and newborns at the facility level: Postpartum hemorrhage (PPH) rate, defined as the percentage of women delivering at participating facilities who experienced a PPH; eclampsia rate, defined as percentage of women delivering at participating facilities who were managed for eclampsia; maternal death rate, defined as the number of maternal deaths divided by the number of deliveries at participating facilities; and early neonatal death rate, defined as the number of neonatal deaths before discharge divided by the total number of live births at participating facilities.

Data quality considerations

To assure adequate data quality, the intervention improved the registers, printed and distributed the new register to health facilities, and trained of health service providers in correct record-keeping. All personnel involved in data collection, tabulation and use were trained to understand the quintessence of data collection. At the time of routine data collection, staff checked a sample of facility forms for validity and completeness and verified them against the compilation summary. In addition, the intervention included a supportive supervision checklist with a focus on the data quality. This checklist provided the opportunity to verify data in the different service registers and the monthly facility summary forms, including cross-checking predetermined indicators between the two sources.

Data analysis

The data were summarized for five periods, including the baseline period.

To estimate the trends in maternal and child health indicators over time and account for clustering within facilities, for each health indicator we fitted a marginal generalized linear model with log-link, poisson distribution, and exchangeable working correlation structure. The models were fit using generalized estimating equations (GEEs) with robust variance [18]. Time was represented with four indicator variables for Periods 1 through 4 in the model; the exponentiated beta coefficient for each period represents a relative rate of the outcome for that period compared to the baseline. Time periods for which there were no data or zero count in the denominator (for example, for total number of deliveries or live births) for a given facility were excluded from the analyses. For SBM-R scores, simple descriptive statistics, including frequencies, percentages and means, were calculated using Excel.

Results

The percentage of SBM-R standards for MNH achieved increased for 3 years in succession after the implementation of SBM-R at all 23 facilities. At the baseline assessment in 2010, the 23 secondary health facilities in Bauchi state achieved only 4% of SBM-R standards for MNH, on average. This increased to 35% in 2011, 69% in 2012 and 86% in 2013.

Figure 3 shows provider compliance with set criteria for MNH performance standards, disaggregated by area. For example, provider compliance with the interventions for managing life-threatening complications of pregnancy and childbirth increased from 8% at baseline in 2010 to 92% in 2013. Similar increases were observed for the other areas. During the reporting period, the institutional MMR declined from a high of 4113 maternal deaths per 100 000 live births in 2010 to 1317 in 2013 and 705 in 2014. Overall, the model supported a significant reduction in maternal mortality over time (P-value = 0.006).

Figure 3

MMRs and mean SBM-R scores.

Table 1 presents the estimated MMRs and their 95% confidence intervals (CIs) by time period; 15 data points (13% of the data) are missing. It shows a significant change in incidence of PPH and eclampsia (P-value < 0.001), and presents their estimated rates and 95% CIs by time period. Nine data points (8% of the data) were missing for each.

Table 1

Trends of events and mortality for baseline and follow-up assessments

Year	MMR, deaths per 100 000 live births, (95% CI)		PPH rate per 100 (95% CI)		Eclampsia rate per 100 (95% CI)		NMR, deaths per 1000 live births, (95% CI)		Administration of a Uterotonic for AMTSL (%)
		P-value		P-value		P-value		P-value		P-value
2010	4113 (1432.5–11811.3)		3.3 (2.2–5.2)		6.0 (3.9–9.3)		9 (4.7–16.37)		11 (9.75–11.82)
2011	3352 (669.1–16792.7)	0.843	2.0 (0.9–4.8)	0.343	2.9 (1.3–6.3)	0.095	5 (1.52–14.82)	0.408	49 (48.92–49.03)	<0.001
2012	1413 (654.9–3047.6)	0.124	0.9 (0.4–2.2)	0.013	0.3 (0.1–0.6)	<0.001	3 (1.00–8.99)	0.119	76 (76–76)	<0.001
2013	1317 (676.7–2563.1)	0.075	1.9 (1–3.6)	0.132	2.8 (1.3–5.9)	0.028	2 (1.01–4.9)	0.007	95 (95–95)	<0.001
2014	705 (396.9–1251.8)	0.006	1.1 (0.7–1.6)	<0.001	1.2 (0.8–1.6)	<0.001	2 (0.91–4.61)	0.008	100 (100–100)	<0.001

The PPH incidence in the hospitals declined from 3.3% in 2010 to 1.1% in 2014, and the eclampsia incidence declined from 6.0% in 2010 to 1.2% in 2014. Table 1 also shows significant change in administration of uterotonic for AMTSL over time (P-value < 0.001) and presents the estimated rates of administration of uterotonic for AMTSL and their 95% CIs; eight data points (7% of the data) were missing. Uterotonic for AMTSL significantly increased from 11% in 2010 to 95% in 2013 and 100% in 2014.

Table 1 and Fig. 4 also show a significant change in NMRs in the health facilities over time (P-value = 0.008) and present the estimated NMRs and their 95% CIs by time period (18 data points [16% of the data] on NMR are missing). Overall, NMRs declined from 9 deaths per 1000 live births at baseline in 2010 to 2 in each of 2013 and 2014.

Figure 4

NMR and mean SBM-R scores.

Qualitative improvements

Teams engaged providers from the initial stages in conducting baseline assessments, developing action plans and working together to implement activities to address some of the quality issues. They met regularly to review their action plans and performance. Teams obtained resources to renovate boreholes and staff quarters; purchase water reservoirs and wastebins and benches for women visiting clinics; repair ambulances for some health facilities where this was needed; establish or strengthen drug revolving funds; build incinerators for medical waste. In some cases, this involved community engagement; Pic. 1 shows an incinerator built by the community. Teams also equally advocated for posting of doctors and midwives to health facilities.

Picture 1

Incinerator built by a community.

Discussion

Our findings have shown that as health facilities achieve more compliance with MNH quality of care performance standards, the use of evidence-based delivery practices increases, leading to decreases in maternal and neonatal mortality.

Baseline performance was low, with no health facility achieving greater than 16% compliance with standards. Similar baseline scores were observed at 13 ACCESS-supported hospitals in Nigeria, while higher baseline scores were seen in Malawi [19]. In this study, significant and consistent increases in compliance with SBM-R standards were observed over time. Comparable results have been recorded in other studies [20].

There is ongoing discussion about significance and appropriateness of maternal and newborn outcome parameters, such as maternal and perinatal mortality and morbidity, as indicators of quality in low-resource countries. However, such outcomes have value for assessing and monitoring quality improvement interventions over time [9]. Long-term implementation and recorded achievements of quality interventions in industrialized nations have shown that quality improvement strategies can result in improved outcomes [21]. Studies linking quality improvement and outcomes in low-income countries, and specifically in Nigeria, are limited. In Malawi, Rawlins et al. [22] demonstrated effectiveness of the quality improvement process and recommended use service delivery data and measurement of outcome parameters to assess quality of care. In Ghana, a quality improvement process was associated with trends toward increase utilization of maternal child health services [23]. Mathai stressed the need for quality improvement, alongside increasing coverage, of evidence-based interventions to achieve maternal mortality reduction and reported improvement of maternal health with introduction of quality improvement approaches in Tamil Nadu [24].

Only 10% of the women who delivered in the surveyed facilities benefited from timely and correct administration of uterotonic for AMTSL during baseline assessment, which is comparable to findings in Guatemala, Mali, Niger and Afghanistan [25]. In this study, SBM-R led to universal coverage of uterotonic for AMTSL with a corresponding drop in the incidence of PPH. A similar inverse relationship between AMTSL practice and PPH incidence was also observed after the quality improvement interventions in Niger and Mali [25]. Similar results were also observed in studies of hospitals in Tanzania [23] following quality improvement interventions [26].

Our study also observed increases in the proportion of newborns receiving essential care according to set standards, similar to findings in Guatemala between 2007 and 2011 [25]. We observed a correlation between increased provision of ENC and early neonatal mortality. Singh et al. [23] similarly reported a notable trend in reduction of neonatal mortality after evaluation of impact of a quality improvement intervention on maternal and child health outcomes in Northern Ghana.

Study strengths and limitations

The study has three key strengths. First, the study included a baseline assessment followed by annual assessments, to allow for progress monitoring over time. Second, the tracked facility achievement of compliance with the set SBM-R standards, the use of evidence-based service delivery practices and MNH outcomes, which provided an opportunity to establish linkages between quality improvement activities and morbidity and mortality. Third, all SBM-R follow-up assessments were conducted by external evaluators, who were likely to be more objective than teams from within the health facilities.

Limitations to the study design include the absence of a control group. However, it is unlikely that the statistically significant improvements observed in this study could be attributed to secular trends, and we are unaware of other factors outside the intervention that could explain the changes. Measurement of the achievement of performance standards was based on a relatively small sample of observations, and for this reason we have not reported statistical tests of these indicators. Cause-specific maternal mortality would be a better measure of the impact of this intervention, but this would require a larger sample and higher-quality data on causes of death, both of which were unavailable.

Finally, this study describes only supply side interventions, whereas demand side interventions are also needed in this context, including efforts to improve health care financing [27]. It is necessary to improve both coverage and quality in this state, where only 17% of deliveries occur in a health facility [2]. The study only presents facility-based data and therefore cannot estimate population-level changes in facility birth or mortality rates.

Conclusion

Introduction of an SBM-R approach, which involved intensive and mutually reinforcing intervention activities, has led to significant increases in compliance with set performance standards and improved quality of MNH services and outcomes. Scaling up use of SBM-R for quality improvement and tracking selected outcome indicators alongside of improvement in standards is recommended.