Development and testing of a composite index to monitor the continuum of maternal health service delivery at provincial and district level in South Africa.

INTRODUCTION: The continuum of care is a recommended framework for comprehensive health service delivery for maternal health, and it integrates health system and social determinants of health. There is a current lack of knowledge on a measurement approach to monitor performance on the framework. In this study we aim to develop and test a composite index for assessing the maternal health continuum in a province in South Africa with the possibility of nationwide use.
MATERIALS AND METHODS: The composite index was computed as a geometric mean of four dimensions of adequacy of the continuum of care. Data was sourced from the district health information system, household surveys and the census. The index formula was tested for robustness when alternative inputs for indicators and standardization methods were used. The index was used to assess performance in service delivery in the North West province of South Africa, as well as its four districts over a five-year period (2013-2017). The index was validated by assessing associations with maternal health and other outcomes. And factor analysis was used to assess the statistical dimensions of the index.
RESULTS: The provincial level index score increased from 62.3 in 2013 to 74 in 2017, showing general improvement in service delivery over time. The district level scores also improved over time, and our analysis identified areas for performance improvement. These include social determinants of health in some districts, and access and linkages to care in others. The provincial index was correlated with institutional maternal mortality rates (rs = -0.90, 90% CI = (-1.00, -0.25)) and the Human Development Index (r = 0.97, 95% CI = (0.63, 0.99). It was robust to alternative approaches including z-score standardization of indicators. Factor analysis showed three groupings of indicators for the health system and social determinants of health.
CONCLUSIONS: This study demonstrated the development and testing of a composite index to monitor and assess service delivery on the continuum of care for maternal health. The index was shown to be robust and valid, and identified potential areas for service improvement. A contextualised version can be tested in other settings within and outside of South Africa.

Introduction

Maternal health outcomes in South Africa (SA) remain poor despite national investments toward their improvement [1, 2]. The maternal mortality ratio (MMR) was estimated between 138 and 158 deaths per 100,000 live births in 2015 [3]. Pregnancy related and facility-based rates of maternal mortality are also high, estimated in 2016 at 536 per 100,000 and 135 per 100,000 respectively [2, 4]. Maternal mortality refers to deaths occurring during pregnancy up to 42 days post-partum or termination of pregnancy, from a pregnancy-related cause; and pregnancy related deaths include all deaths in that period, where cause of death is not identified [5, 6]. The major causes of maternal mortality include HIV infection, obstetric haemorrhage, and hypertensive disorders [7]. The prevailing challenges in maternal health in SA, which also contribute to death, include: inequalities in health service access [8], poor coverage and quality of essential interventions [9], inadequate system wide improvements in quality of care [2] and weak community health services [2]. As such, health systems, health worker and patient related factor are important determinants of individual morbidity and mortality.

One of the key strategies to address maternal health challenges in SA has been to strengthen service provision at all levels of care, from the community to the regional and tertiary hospitals [10-12]. The continuum of care for maternal and child health is a public health framework for outlining the essential interventions and addressing service delivery challenges [13, 14]. The framework has been developed for low- and middle income countries (LMICs) [13] and adapted by national health system stakeholders to the South African context, as illustrated in Fig 1.

Fig 1

The continuum of care framework for maternal, new-born and child health in South Africa [10].

The framework outlines interventions from pre-pregnancy to childhood; the maternal health interventions encompass reproductive health, antenatal, delivery and postnatal care. The framework for SA also outlines “intersectoral factors”, which represent social determinants of health such as housing, nutrition, water and sanitation, and education (Fig 1). The implementation of the framework is expected to improve health outcomes by improving coverage and comprehensiveness of services, quality of clinical care, mitigating duplication of resources, and improving integration of health services [13, 15, 16].

A crucial barrier to the implementation of the continuum of care framework in SA and many LMICs is the lack of a comprehensive monitoring tool for service delivery [11, 17]. The current discourse in SA focuses on the importance of integrated delivery of services, quality of care, strengthening community health systems, and multisectoral collaboration to improve outcomes [1, 11, 15, 18, 19]. Maternal death audits have emphasized the importance of referral linkages, women’s empowerment, quality of care (including quality antenatal care, prevention of pregnancy associated hypertension etc.), and post-natal follow-ups to improve maternal health outcomes in the country [20]. However, gaps remain in measuring community and social factors influencing maternal health outcomes [21]. Assessment of the continuum of care requires consideration of a broad set of indicators beyond antenatal, birth and postnatal care [17].

Previous studies by these authors reviewed and evaluated available indicators for tracking services on the continuum of care for maternal health in South Africa [22, 23]. Another study by these authors proposed an analytical approach emphasizing assessment of access and utilization, quality of care, linkages of care and social determinants of health [17]. Multidimensional assessment is often carried out with composite indices that summarize the performance of multiple interventions on the continuum. Composite indices have been used to track continuum of care performance at subnational and global levels [24-26], while a gap remains in broader integration of quality and social determinants of health [17, 27]. In this study we explore an approach to combine a broad set of indicators for maternal health interventions on the continuum of care through development of a composite index. We explore if the index can be used to assess service delivery at subnational levels in SA and the implications for future monitoring efforts to support implementation of the framework.

Methods

Setting

The North West province is one of the nine SA provinces and consists of four districts: Dr Kenneth Kaunda, Ngaka Modiri Molema, Dr Ruth Segomotsi Mompati, and Bojanala Platinum District Municipalities. This province is among the worst performers with regards to maternal health outcomes and health system indicators [28]. However, the province was also one of the pioneers of primary health care quality improvement and health system strengthening initiatives, such as the Ward Based Outreach and the primary level Ideal Clinic realization programs [29, 30]. Thus, the province was expected to have a broader set of available indicators across the continuum of care compared to others.

Design

We used a step-wise approach to develop and test the proposed index, based on current methodological guidelines [31, 32]. The main steps include: i) defining a theoretical/conceptual framework, ii) selection of variables/indicators, iii) imputation of missing data, iv) multivariate analysis, v) scaling of indicators, vi) weighting and aggregation, vii) checking for robustness, and viii) validation [31].

Conceptual framework

A critical interpretive synthesis of current measurement and monitoring approaches in LMICs found a gap in multi-dimensionality of sets of indicators currently used to assess the continuum of care (COC) for maternal health [17]. The adequacy construct was therefore defined, which outlines four important dimensions to consider: 1) access and utilization of care; 2) quality of care; 3) linkages between levels and packages of care; and 4) social determinants of health [17] (Fig 2). The adequacy construct complements the COC framework by adding elements of quality of and linkages to care, and proposing that all four dimensions be monitored, not just access and or utilization. Indicators of service delivery across all dimensions should therefore be sought from local data sources, with consideration for their relevance, feasibility, validity and quality of routinely collected data such as the District Health Information System [22, 23].

Fig 2

Dimensions and types of indicators used to develop the continuum of care index for maternal health.

Selection of variables/indicators

Our previous research has described the systematic process for selection of indicators and assessment of suitability and measurement gaps [22, 23]. The indicators were aligned with the interventions along the continuum of care framework as illustrated in Fig 1, thus reflecting mainly inputs, processes and outputs in service delivery (outcomes indicators were used during validation steps of the index development process). Selection was also based on their availability within the South African data sources. These indicators were extracted for the North West province and districts for the period 2013–2017. Health service indicators were sourced from the National Indicator Data Set (NIDS) of the District Health Information System (DHIS). The DHIS is used to report and monitor facility level data for health services to support policy and planning [33]. The DHIS provided indicators for access and utilization, linkages, and quality of care dimensions of the continuum of care. This was done with consideration for the limitations of DHIS data due to poor quality of routinely collected data in South Africa and other LMICs contexts.

Social determinant indicators were sourced from the annual General Household Survey [34] (GHS) (2013–2017) [34], Census 2011 [35] and Community Survey (CS) 2016 [36]. The census and CS enabled assessment at district level, even though they offer fewer social determinants of health indicators than the GHS. The census and CS provided indicators of literacy, housing, access to electricity, water, and sanitation at the district level. Additionally, the CS also assesses dietary behaviour and empowerment, but this was not included in the final analysis of performance to allow district level comparison with the census indicator set. There were no comparable indicators of dietary behaviour and empowerment across the data sources at the district level, so only those applicable across were included. A description of all indicators used in this study is provided in S1 Table. Indicator data were extracted, cleaned, and analysed in MS Excel 2010, R v3.6.1 and STATA 14.0.

Imputation of missing data

Health service indicator data may be missing due to lack of services and under-performing systems for data collection and reporting which affects data quality. In the period between 2013–2017 reporting rates gradually improved from a median under 20% to over 75% across most districts. Missing data was also common in the beginning of the period and improved gradually over time. These are systematic issues that are considered to affect availability of data for indicators completely at random. While there is a need to strengthen the health information system to improve data quality, the authors did some corrections for missing data in order to conduct this analysis. We conducted single value imputation using the indicator value observed from the adjacent year [37]. In the Results section we discuss the impact of the remaining data gaps on the index findings. Single value imputation was also applied to indicators from community survey and census to allow calculation of index at district level.

Multivariate analysis

We used exploratory factor analysis to assess dimensionality of the data, in order to compare the statistical and conceptual groupings of indicators [31]. We assessed whether the data fitted the four dimensions of continuum of care proposed by the adequacy construct. The output of the exploratory factor analysis is assessed in the Results section.

Scaling of indicators

We developed a scale transformation approach of scoring indicator values on a scale between 0 and 100 [31, 38, 39] (Eq 1).

The indicator score is calculated on a scale between 0–100; the ideal score is the maximum attainable score, which is a 100; the target is the ideal performance of the indicator; and performance is the observed value of the indicator during a given time period. Targets may consist of a range of values and in such a case we calculated the median score to represent indicator performance. Targets were also based on national policy documents and global technical or scientific guidelines [31, 32, 38]. Targets were set to the conservative maximum (100%) where guidelines were unavailable. The difference between target and observed performance is multiplied by 100 because indicators are originally measured as percentages/proportions. Using targets for performance improves the meaningfulness of the index and its’ role in policy discourse [39].

Weighting and aggregation

The comprehensive continuum of care for maternal health index (C3MH index) was computed as a geometric mean of equally weighted sub-indices reflecting the four adequacy dimensions. We chose equal weighting since this was estimated the most reasonable approach, and based on lack of evidence on the relative importance of each sub-index, lack of theoretical structure to justify a differential weighting scheme, and inadequate statistical and/or empirical knowledge, among others [31, 32, 40, 41].

Simple indices, based on arithmetic and geometric means, can be robust and give valuable information about public health or health system performance [25, 26, 40, 42, 43]. Unlike the arithmetic mean, the geometric mean allows for a degree of non-compensation of performance of one indicator by another [31, 32]. Each sub-index (e.g. access to care) was also formulated as a geometric mean of its indicator scores.

Where a, b, c are individual indicators and n = number of indicators within the sub-index.

Validity and robustness

We ran sensitivity analyses comparing index performance with different indicator combinations and normalization methods [44]. We tested if z-score standardization leads to a shift in district ranks [31]. Index performance was also compared after removal of indicators that were considered outliers (performed close to 100%), missing data, or indicators that could be represented by a proxy (e.g. syphilis treatment measured with one indicator instead of the three across the treatment cascade, see S1 Table Indicators 4–6). Index aggregation by arithmetic and geometric mean was also compared. We assessed the median absolute difference in district ranks, and its inter-quartile range, testing alternative approaches to index formulation [44]. External validation of the index was conducted by exploring its relationship with indicators of public health performance and maternal health outcomes, particularly the Human Development Index (HDI) and maternal mortality rates [38, 45, 46]. Confidence intervals for correlations were calculated by bootstrapping methods in R v3.6.1.

Results

Performance at provincial level

In the North West province, we combined 12 indicators of access and utilization to care, two quality of care, two linkages of care, and 9 social determinants of health indicators to measure the C3MH index (Table 1).

Table 1

The continuum of care for maternal health index, sub- indices and indicators for North West Province, South Africa, in the period 2013–2017.

		Indicator Performance					Indicator Scores
Indicators	Targets	2013	2014	2015	2016	2017	2013	2014	2015	2016	2017
Cervical cancer screening coverage	100%	58%	62%	66%	66%	69%	58	62	66	66	69
Antenatal 1st visit before 20 weeks rate	100%	48%	53%	59%	64%	64%	48	53	59	64	64
Antenatal 1st visit coverage	100%	78%	79%	75%	76%	78%	78	79	75	76	78
Syphilis positive pregnant female receive Benz-penicillin 1st dose rate	100%	na	na	57%	57%	78%	na	na	57	57	78
Syphilis positive pregnant female receive Benz-penicillin 2nd dose rate	100%	na	na	60%	60%	57%	na	na	60	60	57
Syphilis positive pregnant female receive Benz-penicillin 3rd dose rate	100%	na	na	57%	57%	50%	na	na	57	57	50
Antenatal client starts on antiretroviral therapy rate	100%	63%	88%	89%	93%	93%	63	88	89	93	93
Delivery by Caesarean section rate	5–15%	18%	21%	22%	21%	24%	92	89	88	89	86
Delivery in facility rate	100%	65%	67%	69%	69%	72%	65	67	69	69	72
Mother postnatal visit within 6 days rate	80–100%	75%	76%	71%	73%	77%	85	86	81	83	87
Couple year protection rate	50–100%	42%	54%	50%	59%	55%	67	75	75	75	75
Termination of pregnancy 0–12 weeks rate	100%	97%	96%	95%	95%	96%	97	96	95	95	96
Access sub-index							70.9	76.0	71.5	72.5	74.1
Antenatal client HIV re-test rate	100%	47%	64%	78%	100%	100%	47	64	78	100	100
Average ideal clinic status (score)	70–100%	na	55%	55%	65%	66%	na	70	70	80	81
Quality sub-index							47	66.9	73.9	89.4	90.0
Emergency rural obstetric response under 40 minutes rate	75–100%	na	na	na	61%	61%	na	na	na	74	74
Emergency urban obstetric response under 15 minutes rate	75–100%	na	na	na	41%	41%	na	na	na	54	54
Linkages sub-index							na	na	na	62.7	62.7
Domestic water compliance rate	100%	72%	53%	62%	76%	63%	72	53	62	76	63
% women 15–49 who are literate	100%	83%	84%	83%	83%	82%	83	84	83	83	82
% women 15–49 in households with adequate water infrastructure	100%	82%	82%	78%	80%	80%	82	82	78	80	80
% women 15–49 with basic sanitation facility	100%	71%	69%	70%	71%	71%	71	69	70	71	71
% women 15–49 living in adequate housing	100%	52%	44%	48%	54%	55%	52	44	48	54	55
% women 15–49 living in formal housing	100%	85%	85%	82%	82%	83%	85	84	82	82	83
% women 15–49 with access to electricity	100%	95%	95%	94%	94%	95%	95	95	94	94	95
% women 15–49 who have adequate food access	100%	62%	62%	62%	64%	64%	62	62	62	64	64
Mean Household Dietary Diversity Score (women 15–49) (converted to 100)	100%	62	61	62	62	62	62	61	62	62	62
SDoH index score							72.6	68.6	69.9	73.0	71.8
CoC (maternal health) Index							62.3	70.4	71.7	73.8	74.0

SDoH = social determinants of health; CoC = continuum of care.

The C3MH index at the provincial level ranged from 62.3 in 2013 to 74 in 2017, showing a general trend of improvement (see Table 1). The two sub-indices that substantially contributed to this increase were:

Improvement in access and utilization of care indicators, particularly cervical cancer screening, timely antenatal care, and antiretroviral drug provision.

Improvement in facility performance on quality of care measures. The quality sub-index improved from 66.9 in 2014 to 90 in 2017. However, this sub-index is only based on two indicators, one of which represents a drastic improvement in HIV program processes. A gap exists in maternal health care safety and patient experience of care indicators for measurement of quality in the North West province.

Little overall improvement was made in the social determinants of health during that period, which may point to a slow pace of development in the province. Data was unavailable for the period (2013–14) to monitor treatment of sexually transmitted illness (syphilis), emergency obstetric transport (2013–2015), and quality of care (Ideal Clinic) (2013–2014, while the program was under conceptualization and testing).

Monitoring performance at district level

There was an overall improvement in the index at district level over the period 2013–2017, as illustrated in Fig 3.

Fig 3

Comprehensive continuum of care index (C3MHindex) scores by districts over a five-year period 2013–2017.

Overall, Dr Ruth Segomotsi Mompati (RSM) district performed better than other districts on the index, while Bojanala Platinum performed generally poorer. We also compared sub-index performance to demonstrate effect on overall scores, using 2016 as a reference year (Fig 4).

Fig 4

Sub-index and C3MH index scores by districts in 2016.

BN = Bojanala Platinum District, KK = Dr Kenneth Kaunda District Municipality, RSM = Dr Ruth Segomotsi Mompati District Municipality, NMM = Ngaka Modiri Molema District Municipality, SDOH = Social determinants of health.

In 2016, Dr Kenneth Kaunda district scored relatively higher than other districts on the social determinants of health and quality of care sub-indices. But due to poor performance on access and linkages of care, the district scored second best in overall performance in 2016. Comparatively, the Ruth Segomotsi Mompati district had relatively high scores across sub-indices, and thus ranked highest in 2016. Thus, the balance of good performance across all sub-indices improved the overall index.

Robustness

There was no significant difference in district ranks between index scores calculated with linearly scaled indicators (our method) and z-score standardization (rs = 0.83, 95% CI = (0.49–0.95)) (Table 2). There was also no significant difference between index scores when geometric and arithmetic aggregation techniques were used (rs = 0.95, 95% CI = (0.78–0.99)). The median absolute difference in index rankings at district level when linear and z-score standardization were compared was 2 ranks with an interquartile range (IQR) of 0–3. There was no difference in rankings (IQR = 0–1) observed at district level when indices computed with arithmetic and geometric means were compared.

Table 2

Spearman rank correlation between alternatives for indicator standardization and aggregation at district level.

	Base case	z-score (districts)	arithmetic mean
Base case	1.00
z-score	0.83	1.00
arithmetic mean (d)	0.95	0.84	1.00

Base case is based on linear scaling (our method) and geometric mean aggregation.

All the index values across alternative indicator selections were highly correlated (Table 3).

Table 3

Spearman rank correlation coefficients of index values when dropping one indicator at a time to compute index.

	Base case (all)	No syphilis 2&3	No termination	No syphilis & no termination
Base case (all)	1.00
No syphilis 2&3	0.98	1.00
No termination	0.99	0.97	1.00
No syphilis & no termination	0.98	0.99	0.98	1.00

Validation

The C3MH index had a positive correlation (r = 0.972, 95% CI = (0.63, 0.99)) with the Human Development Index (HDI) in the North West province for the period 2013–2017. The HDI measures healthy life outcomes, education and standard of living [47]. The index also increased with decreasing rates of institutional maternal mortality (iMMR) at the provincial level (rs = -0.90, 90% CI = (-1.00, -0.25)). The correlation between the index and iMMR at district level, was not statistically significant (r = -0.13, 95% CI = (-0.58, 0.39). There are no data for HDI scores at district level to allow comparisons with the COC index.

Results of multivariate analysis

Parallel analysis in exploratory factor analysis suggested one main underlying factor for the data (Fig 5), although a three-factor model may be possible.

Fig 5

Parallel analysis scree plot for indicators of the continuum of care for maternal health in North West province South Africa.

A one factor model accounted for 0.52 proportion of variance of the data. A three-factor model accounted for cumulative variance of >0.9: the majority of factor 1 indicators related to the health system or facility based care, factor 2 contained both health system and social determinants of health, and factor 3 contained social determinants (Table 4). A two-factor model accounted for 0.72 cumulative proportion of variance of the data but did not reveal any informative conceptual groupings–all factor loadings were relatively high for one factor. The variables for linkages of care and one variable for quality of care were also not included in the results of the model due to missing data.

Table 4

Exploratory factor analysis of the items of the continuum of care service delivery framework.

	Factor 1	Factor 2	Factor 3
Cervical screening	0.93	-0.25	0.20
Timely antenatal visit	0.90	-0.33	0.31
ARTs during antenatal care	0.95	-0.25	-0.21
Caesarean section delivery	0.91	-0.08	-0.02
Delivery in facility	0.91	-0.12	0.25
Couple year protection rate	0.86	-0.09	-0.05
HIV retest rate	0.91	-0.24	0.33
Adequate food	0.77	0.14	0.47
Ante natal care coverage	-0.06	0.99	-0.14
Postnatal visit	0.18	0.98	0.06
Timely pregnancy termination	-0.59	0.75	0.15
Water infrastructure	-0.45	0.83	-0.13
Type of housing	-0.52	0.78	-0.35
Electricity access	-0.33	0.91	-0.23
Water safety compliance	-0.23	-0.26	0.79
Literacy	-0.37	0.02	-0.77
Sanitation	0.20	0.08	0.98
Housing condition	0.25	0.01	0.97
Household dietary diversity	-0.04	-0.45	0.86

Notes: Extraction method–ordinary least squared/minres; Rotation–varimax; Loading larger than 0.5 are in bold.

Discussion

This study demonstrated the development and testing of a comprehensive and multidimensional index to assess the continuum of care for maternal health at subnational levels in SA. The C3MH index measured health and non-health sector components of service delivery for maternal health, as guided by the continuum of care framework for SA [10]. The multi-sectoral perspective of the index is increasingly important in current public health and health system performance assessment [11, 48, 49]. The index was comprehensive in that it allowed monitoring of myriad interventions, summarized through four sub- indices representing dimensions of the continuum of care. The comprehensive and multisectoral character of the index contrasts with the “silo” or vertical program approach that singles out single interventions to address maternal health outcomes [50].

Our findings suggest that the index can be used as a monitoring tool to compare subnational performance over time, and as a basis for recommendations on areas of service delivery improvement. For instance, our findings show that improvements in access and linkages to care will enhance Dr Kenneth Kaunda district’s overall score and improve its ranking. On the other hand, in Dr Ruth Segomotsi Mompati district, poor performance on the social determinants of health affected the index. Thus, while the C3MH index can be used to compare and rank districts, an analysis of sub- indices indicates areas that may proportionally affect overall performance. The relevance and utility of sub-components of composite indices for public health policy and action is important to consider as well, beyond the monitoring application of the overall index [32, 39].

Our findings also indicate that the index was robust and not much influenced by outlying scores for specific indicators. In other words, the index values the parts differently than the whole, as well as good performance over several indicators and not just a few outlying values. Further research is needed to compare the index to other comprehensive standards for the integrated care approach it seems to reflect. Alternative methods for computing the index and standardising the indicators produced comparable results. The simple geometric approach allows future integration of missing data, while maintaining the conceptual grounding of the index. Our approach also accommodates the expected refinements of indicators over time [51]; the index should undergo recurrent assessment and validation to remain useful [52]. Additionally, exploratory factor analysis indicates a distinction between at least 2 factors, with one factor covering health systems indicators and the other factor(s) social determinants of health. This reflects the multidimensional nature of the index and underlines the need to include social determinants as a dimension of continuum of care in maternal health.

Future research should also reflect emerging insights on determinants of health and wellbeing in maternal health, in order to continue to refine the framework and index. These include partner dynamics that influence a range of outcomes during pregnancy [53-57], as well as maternal mental health, gender power relations and intersectional factors such as age and (dis)ability as found in our previous research [23].

Limitations

This was a case study of five subnational geographical areas over a five-year period. We recommend more research across other provinces/districts to allow further comparison. In other countries, the same approach using a comprehensive set of available indicators can be used to develop a contextualised version of the index. The composition of the index in this study was affected by gaps in data availability common in the SA health system [58]. We recommend health information system improvements in monitoring the quality and availability of data so that better estimates of the index can be made in the future. In addition, the lack of comparability of provincial and district level associations between the index and maternal outcomes warrants further investigation. There may be systemic issues with maternal mortality estimation in the country [59], and we also recommend use of the GHS as a source of data at district level. Other indicators could also be considered as proxies based on their shown reciprocity with maternal health indicators, such as neonatal and child health outcomes [25, 26, 60, 61].

Conclusion

This study shows the feasibility to monitor and assess service delivery for the continuum of care for maternal health using indicators from different sectors with a composite index. The index allows monitoring of performance over time and across geographical areas. From our analyses, we concluded the index to be robust and valid, with potential to guide policy and planning to improve maternal health outcomes and service delivery from a multisectoral perspective. Future steps include engagement of health system and other relevant management stakeholders to discuss the current method and results, in order to continually refine the approach and inform any future implementation. More comprehensive monitoring of social determinants at district level and health information system strengthening can further improve and extend the use of this index. The index is amenable for testing with data from different South African and international contexts.

Indicators used for measurement of the continuum of care index in North West province and districts 2013–2017.

(DOCX)

Click here for additional data file.

18 Mar 2021

PONE-D-20-15418

Development and testing of a composite index to monitor the continuum of maternal health service delivery at provincial and district level in South Africa

PLOS ONE

Dear Dr. Mothupi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Two experts in the field reviewed your manuscript. Please ensure to thoroughly address all reviewre comments.

Please submit your revised manuscript by Apr 29 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Susan Hepp

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.  We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript seems scientifically sound, written in standard English language, results and findings are presented in appropriate fashion and concluded appropriately. This must be a product of years of handwork of investigators. Moreover, the study is novel and generated strong scientific evidence for the policy change and reform in maternal health in Africa and other similar countries of the world.

Reviewer #2: please see additional comments attached

Data from the DHIS is confidential and this point is made by the authors

Major issues regarding selection of indicators, lack of comment on quality of the DHIS data etc

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Hari Prasad Kaphle, MPH, PhD, Assistant Professor (Public Health) and Coordinator (Master of Public Health Program), School of Health and Allied Sciences, Pokhara University, Kaski, Nepal. Email: harikafle07@gmail.com

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: MH_PONE-D-20-15418_reviewer.pdf

Click here for additional data file.

Submitted filename: Development and testing of a composite index to monitor the continuum of maternal.docx

Click here for additional data file.

24 Apr 2021

Reviewer 1

Thank you for your time and input into our manuscript.

Reviewer #1: The manuscript seems scientifically sound, written in standard English language, results and findings are presented in appropriate fashion and concluded appropriately. This must be a product of years of handwork of investigators. Moreover, the study is novel and generated strong scientific evidence for the policy change and reform in maternal health in Africa and other similar countries of the world.

� Thank you very much for your review!

Reviewer 2

Thank you for your time and input into our manuscript. Please see the changes requested in the ‘Revised Manuscript with Track Changes'. We start here with general comments then the line by line comments made below.

This is a very important study and potentially adds to the literature and can be very useful to programme managers. It’s a brave attempt especially given issues related to data availability and quality in South Africa and in many LMICs.

Thank you, indeed data availability and quality is an issue in LMICs and the ways in which it has affected the measurement of the index and the future improvements are noted in the manuscript.

In their review of the literature I would request the authors to pay more attention to issues related to quality of care and outcomes (these issues are reflected in my notes in the text of their manuscript).

Thank you, we have responded to the notes in the line by line comments below.

There are areas of concern with the methodology that I wish to flag for the considerations of the authors. The author should be add a critical review of the use of linear transformation (see Greco et al, 2019 for example). In addition, the selection of indicators used (especially those from the DHIS) and their decision not to include mortality (maternal and neonatal mortality) as outcomes should be clearly stated. The latter is only noted in the section on limitations of the study. Finally, the quality of the DHIS data (in additional to imputation of missing data) should be reflected on.

Thank you, these issues were captured in the line notes made by the reviewer and we have responded accordingly to each in the line by line responses below.

On the practical use of the index – the authors should inform the reader if the study and its approach, results and conclusions were discussed with the local managers and if any of their recommendations were adopted/implemented. I am aware that this is not usually requested by reviewers but practical application of such studies are important to improve quality of care.

Thank you, we have added in the conclusion a plan for further engagement of stakeholders (Line 354-356).

I would therefore encourage the authors to strengthen the aspects reflected above and as reflected in my notes in their manuscript.

Thank you, please see below on pages 3-8 of this letter how we have responded to and addressed the reviewers’ comments citing lines where changes have been made in the manuscript. We believe they were helpful and strengthened the manuscript.

Major issues regarding selection of indicators, lack of comment on quality of the DHIS data etc.

� Thank you we respond below to these points as they are raised in the specific lines in the manuscript and your letter.

Abstract:

Line 25: the focus was one province with the possibility of use nationwide.

Please see Line 25 and 26 for the change that reflects focus on one province with possibility of nationwide application.

Background:

Line 56: given this difference maybe add the distinction between these two measures.

Please see lines 57 to 60 added to distinguish between maternal mortality ratio and pregnancy related death ratio.

Line 57: add administrative causes as well (patient and healthcare worker related factors)

We have modified Line 62-66 to show that these other factors contribute to deaths as well. We regarded health care worker related factors as related to the healthcare access and quality issues.

Line 63: also need to focus on regional and tertiary hospitals

Please see Line 66-67 we have modified to include.

Line 70: maternal nutrition is also a major contributor to good pregnancy outcomes.

Indeed, and it is featured in the framework as well, we have stated it more explicitly now in Line 78.

Line 72: The co-production of health and wellness (with in this case the pregnant woman and the partner) is surfacing as a new approach, consider adding this to the literature review?

In the paragraph we focus on explaining elements of the framework as it stands. We have discussed co-production as a new approach that could possibly inform future iterations of the framework in the Discussion (Lines 329-333). The literature on co-production of health and wellness seems to suggest partner dynamics (including abuse) that influence a range of pregnancy related attitudes, behaviours and outcomes such as pregnancy wantedness (Kroelinger & Oths, 2000), smoking cessation (Flemming et al., 2015), exclusive breastfeeding (Moraes et al., 2011), postnatal depression (Ludermir et al., 2010), alcohol use (Van der Wulp et al., 2013) among others. These were also cited by South African experts in our previous research and recommend to include them in this revision (Lines 329 - 333 in the Discussion section).

Line 75: need to include quality of clinical care as well.

Please see Line 80 quality of clinical care added.

Line 78: add quality of care

Please see Line 84-85 quality of care added

Line 80: quality antenatal care, improvement of clinical quality of care as well, such as ESMOE, prevention of pregnancy associated hypertension etc.

Please see Line 87 - 88 for this addition

Line 124: I would add quality of data, especially routinely collected data like the DHIS

Please see this addition on Line 131-132

Line 133: you may wish to comment on the quality of DHIS data

Please see Line 342 under Limitations, and Line 147 to 149 under Methodology where we include this reflection on DHIS data quality and make the limitation more explicit in the relevant section.

Line 140: these are important indicators, please include reasons why they were not included

Please see Line 157-158 for explanation – “There were no comparable indicators of dietary behaviour and empowerment across the data sources at the district level, so only those applicable across were included.”

Line 144: with respect to poor quality data, was any data cleaning done?

The Department of Health Northwest gave us already tabulated data on specific indicators, not raw data. The only corrections made were in the rare cases where proportions were over 100%, which were revised back to 100. The DOH also reported data quality issues around data capturing rates at primary health care and hospital level in the different districts of the North West. For 2013, these figures were not there for most districts and the reported figures pointed to low data capturing rates. From January 2014 it was reported in more districts, the rates generally improved over time and by 2017 median rates were over 75%. Completeness also improved over time. With continual improvement in reporting, completeness, and data availability over time – i.e., a strengthened health information system, we believe measurement of the index will be more reliable. The authors can control for missing data through a transparent approach shared in the paper, and other elements recommendations made on strengthening the health information system.

In line 157-158 we mention underperformance in data collection and reporting. Please see lines 158-164 for additional integration of the statements on data quality as stipulated in this comment.

Line 158: It will be useful to add a short review of the benefits and challenges of using linear transformation, see for example Greco et al 2019.

We thank you for this useful reference. Greco et al 2019 provide interesting guidance on weighting and aggregation. We added a more detailed justification regarding the choice of our weighting procedure (i.e. equal weights) (see lines 197-199). For aggregation, we used geometric aggregation rather than linear aggregation as Greco et al. suggest being the better choice in our case.

The linear transformation we referred to was used for the scale transformation.

We have modified line 180 to clear the potential confusion with linear aggregation methods and specify that it is a variable/indicator scale transformation approach, developed by the authors based on the logic of similar models with the authors’ modifications: Similar methods are referred to as linear scaling transformation (Booysen et al 2002) (doi: 10.1136/bmjqs-2018-007798), linear interpolation (Barclay et. al. 2019 – doi: 10.1136/bmjqs-2018-007798), and scale transformation (OECD 2008 10.1111/jgs.13392). We tested the approach we developed against an established method for normalization, which is z-score standardization to assure its' robustness.

Line 200: Why were no outcome measures such as maternal mortality and neonatal mortality rates included? Surely this is the real test of performance of MH services – that pregnant women survive and deliver healthy neonates? This would have been more useful than including say cervical cancer screening coverage – the point is the authors should make a case for selection of indicators.

Thank you. Maternal mortality and neonatal mortality rates are considered outcome indicators that do not correspond directly to process, inputs, outputs etc. in service delivery. The continuum of care framework as it is designed is concerned with comprehensive delivery of specific interventions at different levels and across the health needs of mothers. Please see Figure 1. Thus the focus of the framework is the access, utilization and quality aspects of interventions. The outcomes of maternal and neonatal mortality are indeed important, and they are used by the authors to validate the measurement of performance with the service delivery indicators. The outcome indicators thus go hand in hand with the index, where one of the validation steps prescribed by the authors looks at the correlation between the index and maternal health outcome indicators. Please see Lines 275-282 (Validation section of Results). Additionally, we make a recommendation that neonatal and child health indicators be tested in future research, please see Lines 349 - 351 of the Discussion section.

We also include this explanation in the updated section on Selection of Indicators, please see lines 138-142.

Line 210: The more usual spelling is antiretroviral

Thank you, please see the change on Line 234

Line 213: This is why a careful explanation of indicator selection is important

In the selection of variables/indicators section, we have added lines 136-142 to more clearly explain indicator selection. This follows along the explanation given above for non-selection of outcome indicators, as well as a point about availability. In line 136-137 we also referred to two other publications where we documented the systematic process of selecting indicators: Mothupi MC, Knight L, Tabana H. Review of health and non-health sector indicators for monitoring service provision along the continuum of care for maternal health. BMC Res Notes. 2020;13(151) and evaluating whether they were suitable: Mothupi MC, Knight L, Tabana H. Improving the validity, relevance and feasibility of the continuum of care framework for maternal health in South Africa: a thematic analysis of experts’ perspectives. Heal Res Policy Syst [Internet]. 2020 Feb 26 [cited 2020 Oct 10];18:28. Available from: https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-020-0537-8

Line 219: was this programme implemented in 2013/14?

The program was undergoing early conceptualization and testing during this time, please see the updated Line 243-244. Even though the program was not being widely implemented, we mention this data gap to explain why data is not reflected for our period of interest 2013-2017.

Line 321: Not including outcome measures to me is the biggest limitation of the study

Outcome measures were treated as an important indicator for validation of the index – please see the methodological explanation of how outcome indicators were used in Line 217-219, and Lines 278-281 (Validation section of Results) for findings of the correlation between the index and maternal health outcome measures. The indicators aligned with the continuum of care framework used in this study reflect intervention/services and thus inputs, process and outputs and not health outcomes. Please see the added lines 138-142 on selection of indicators.

Line 328: was the study shared with the provincial and the local management? How did they interpret the results? Was any action taken or considered by these managers on the basis of this study or will this be done should this study be published?

The preliminary steps of the study, which are selection and assessment of indicators was done through consultations with some Department of Health and other relevant sectors’ stakeholders, which gave opportunity to get their feedback on the framework on which the index development detailed in the manuscript is based. Those findings were published elsewhere and shared with the provincial management. As follow-up and concerning this particular manuscript, we have added the engagement that still needs to happen to further refine, get interpretation, and build on the approach in Lines 358-360. The publication of the study is intended for the publication of the methodology but we acknowledge the engagement that still needs to happen to further refine the approach and get stakeholders’ further input in it.

In their review of the literature I would request the authors to pay more attention to issues related to quality of care and outcomes (these issues are reflected in my notes in the text of their manuscript).

Thank you we have taken consideration of these and made a line by line response above.

-

Thank you again for the opportunity to revise this manuscript; and we hope that the changes made will be satisfactory and have improved it significantly.

Yours sincerely,

Mamothena Carol Mothupi

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

12 May 2021

Development and testing of a composite index to monitor the continuum of maternal health service delivery at provincial and district level in South Africa

PONE-D-20-15418R1

Dear Dr Mothupi

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Yogan Pillay

Guest Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

17 May 2021

PONE-D-20-15418R1

Development and testing of a composite index to monitor the continuum of maternal health service delivery at provincial and district level in South Africa

Dear Dr. Mothupi:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Yogan Pillay

Guest Editor

PLOS ONE