Monitoring of birth registry coverage and data quality utilizing lot quality assurance sampling methodology: A pilot study.

BACKGROUND: Effectively monitoring the coverage and quality of data in low-resource settings is challenging. Lot quality assurance sampling (LQAS) is a method to classify coverage as adequate or inadequate. The aim of this pilot study is sought to determine the coverage and quality of a birth registry in a rural district in Pakistan.
METHODS: This survey was conducted in 14 clusters of Thatta, Pakistan. LQAS methodology was used to monitor the birth registry from December 2015 to February 2016. We randomly selected 19 villages from each cluster. We used a short questionnaire to review the quality of data collection for select variables. Frequency and percentages were reported for categorical variables. For data validation, Kappa statistics (κ) were applied to assess the agreement between categorical observations, and the Bland-Altman test was used to assess agreement for continuous data.
RESULTS: Of the 14 clusters sampled, 12 clusters had adequate coverage. Agreement of hemoglobin performance between the women's response and information in birth registry data was good (κ = 0.718) (95% confidence interval [CI]: 0.58-0.82); agreement on birth outcome recorded by the workers in the registry and as mentioned by women was very good (κ = 1.0); and agreement whether birth weight was assessed within 48 h of delivery was good (κ = 0.648) (0.37-0.92).
CONCLUSION: LQAS is a powerful tool to monitor coverage and data quality of the birth registry maintained by the global network for women's and children's health in Pakistan and potentially for data from other surveillance systems.

Introduction

The global network for women's and children's health research (global network) is a multicountry research network which supports the conduct of clinical trials in resource-limited countries by pairing foreign and US investigators, with the goal of evaluating low-cost, sustainable interventions to improve the health of women and children.[1] Accurate reporting of births, stillbirths, neonatal deaths, maternal mortality, and measures of obstetric and neonatal care is critical to efforts to develop strategies for improving pregnancy outcomes in resource-limited settings.[1] Because most of the regions in the global network have weak birth registration systems, the global network developed the Maternal Newborn Health Registry (MNHR), a prospective, population-based registry of pregnancies to provide more precise data on health outcomes and measures of care in the regions associated with study sites.[2]

Lot quality assurance sampling (LQAS) methodology is gaining popularity as a tool for monitoring public health interventions due to its simplicity and low sample size requirement.[3] LQAS was initially introduced by industry to determine whether a lot of a product was ready to be sent to the market for sale.[45] In 1980, this method was adopted for public health interventions to determine coverage of interventions and dissemination of public health messages to the communities.[5] LQAS is a cross-sectional study design that classifies exposure or outcome as high or low and adequate or inadequate. This study design has been shown to have <10% error rate and high sensitivity.[6] Results from LQAS can help local supervisors to assess coverage so that they can make informed decisions for allocation of resources to improve coverage as well as data quality.[6]

Monitoring the health workers’ work to ensure good coverage and high data quality is especially challenging in the rural communities in low-resource settings, such as Thatta, Pakistan. In the District Thatta of Sindh Province, the population is scattered over large geographical areas, making supervision and monitoring for any program very difficult.[2] Thus, ensuring high quality of the data from the region may require specialized tools.

LQAS is a relatively rapid and inexpensive approach to data collection and has primarily been used for monitoring and evaluation purposes. The LQAS method enables managers to assess whether program objectives and targets have been achieved within a specific unit of interest (a geographical area, a facility, an organization, or any other catchment area). To determine the potential utility of LQAS for the global network, we conducted a pilot study of LQAS for the MNHR in Thatta District. The objective of this pilot study was to determine the coverage in the geographical area by the Thatta registry health workers for enrollment of pregnant women and recording pregnancy outcomes. The second objective of this pilot was to assess the quality of data collected by the registry workers at field sites.

Subjects and Methods

The MNHR is a population-based registry of all births in defined study clusters, which are geographically defined regions with approximately 300–500 births annually. The MNHR is a multicountry research study, which is supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development Global Network for Women's and Children's Health Research.[27] In the Pakistan study site, the MNHR is being conducted in 14 study clusters of the District Thatta, Sindh Province.[8] For the MNHR, women are enrolled during pregnancy, visited following delivery and at 42 days postpartum. The variables that are collected include maternal height at the enrollment visit, birth weight at the delivery visit and maternal and infant status at 42 days postpartum.

From December 2015 to February 2016, the LQAS survey was conducted in 14 study clusters of district Thatta. The estimated population of these 14 clusters is approximately 200,000. Thatta is predominantly a rural district bordering the cities of Hyderabad and Karachi. The global network's MNHR was first established in 2008. The primary objectives were to record fetal and maternal outcomes, including morbidity and mortality and their temporal trends. In Thatta, 14 female birth registry workers collect the MNHR data on standard forms under the supervision of 4 field supervisors.

Since we only have information on the number of villages in each cluster and a rough estimate of the population and number of households in each village (through government records), we used a multistage sampling strategy. For the first stage, a total of 19 villages were randomly selected from each supervision cluster; for the second stage, for selection of the random selection of the first household from the selected village, spinning a bottle at a central point in the village was done. It was predefined that the research team will follow the direction where the mouth of the bottle will face and the fifth house falling on the left-hand side in this direction will be selected for the interview. Where we had to select three households from a village, the bottle was spun a second and third time in the area farthest and opposite to one selected before, and the same strategy of having the fifth house on the left-hand side was carried out. For the third stage, if there were more than one eligible woman in the household, one was randomly selected by a simple draw method. One to three women were interviewed from a village based on proportion to the population size of the village. For LQAS methodology, a sample of this size results in classification errors (i.e., alpha and beta errors) of <10%.[9] We selected a decision rule of 16 to capture a coverage benchmark of ≥95% [Figure 1].

Figure 1

Lot quality assurance sampling sample size, decision rules, and coverage benchmark

Women of reproductive age, residing in a cluster for at least 6 months who consented to participate in the study were included in this pilot survey. Women living in an area for <6 months and those who refused consent were excluded from the study. The research coordinators with the help of the field coordinators collected the data. Verbal consent was obtained from all the participants because of its high acceptability by women in the community. Information was collected on a brief questionnaire specially designed for the purpose. Coverage was defined as “a birth registry worker” visiting a household at least once in the past 42 days either to enroll a pregnant woman in the birth registry, to weigh the newborn within 48 h of the birth or to visit the woman in the postpartum period, i.e., at 42 days after delivery to collect information about the woman and her infant.

Enrollment of the woman in the MNHR was verified by asking for the copy of the consent form kept with the woman when she was enrolled and by noting down her unique ID mentioned on the form. To assess the quality of data collected for the birth registry, enrolled women were asked several questions. The questions assessed parity, whether her hemoglobin was measured (“yes” or “no” response), whether the weight of the baby was taken within 48 h of delivery (as “yes” or “no” response), the pregnancy outcome (live birth, stillbirth, miscarriage), and the sex of the baby. In order to assess the skills of registry workers, we selected the measurement of maternal height as an indicator. For this assessment, the monitoring team measured the height of the respondent which was later compared with the height measurement recorded in the MNHR database by the registry worker.

The data were entered in Microsoft Excel for Windows® version 2010 and analyzed using SPSS v. 10 (SPSS Inc., Chicago, IL, USA) in the data management unit of MNH birth registry office in Karachi, Pakistan. The data were entered by a data input operator and analyzed by the data supervisor of the central data management unit. Frequency and percentages were reported for categorical variables. For data validation, Kappa statistics (κ) were applied to assess the agreement between categorical observations. A value of <0.2 was considered to be poor agreement, 0.2–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as good, and 0.81–1.0 as very good agreement.[8] We constructed 95% confidence intervals (CIs) for Kappa. The Bland–Altman test was used to assess agreement for continuous data, i.e., height measurements which were recorded by the data monitoring team and the birth registry worker. In the Bland–Altman plot, the differences between observation pairs are shown against their mean and the mean difference and its 95% confidence limit lines are shown.[10]

Results

A total of 284 houses were approached and of these, 266 (93.6%) were surveyed, with 19 from each cluster.

As described, the predefined rule of 16 respondents saying “yes” for the home visit was considered to be acceptable coverage. In two clusters, <16 of the 19 women responded that a health worker had visited their home during the past 42 days. In 12 out of 14 clusters, the coverage was ≥95% [Figure 2] and the cluster was considered as adequately covered by the registry worker. Overall, the weighted population coverage of the birth registry for the whole study area by the workers was 88% (95% CI: 84%–92%). As assessed through Kappa statistics, agreement of hemoglobin performance was “good” (κ = 0.718 95% CI 0.58–0.82); agreement on pregnancy outcome recorded by the workers in the registry and LQAS team was “very good” (κ = 1.0); and agreement, whether the birth weight was assessed within 48 h of delivery, was “good” (κ = 0.648, 95% CI 0.37–0.92). The Bland–Altman plot showed no evidence of bias or measurement inconsistencies in the measurements collected in the registry compared to those collected by the LQAS auditors [Figure 3].

Figure 2

Coverage of birth registry by cluster

Figure 3

Bland–Altman plot of agreement for height as assessed by the lot quality assurance sampling team and the birth registry worker

Discussion

LQAS is a simple and powerful study design for monitoring projects and for determining the appropriate allocation of resources. LQAS also empowers local supervisors to make informed decisions. Furthermore, while alternative sampling methods may allow for more precise CIs, LQAS provides a greater sample spread and requires less time for data collection, analysis, and resources. It also facilitates real-time decision-making by the supervisors. To the best of our knowledge, this is a first study that uses LQAS for monitoring coverage in a population-based birth registry. However, LQAS is a popular and widely accepted method of coverage of vaccination and especially polio vaccination coverage.[11]

Coverage of birth registry and data quality

The results of this pilot survey showed the overall weighted coverage in the birth registry catchment population was 88% (95% CI-84–92%). Of the 266 households surveyed, birth registry personnel did not visit 34 houses in the last 42 days in 2 clusters. In follow-up discussions, one of the reasons for the low coverage was the resignation of two workers, which left the respective areas uncovered for some time.

Across all clusters, good agreement between the LQAS and the birth registry results was found in the assessment of hemoglobin, birth outcome, birth weight, sex of the baby and the height measurement of the mothers. This reassured the supervisors that registry workers are generally well-trained and doing their job adequately.

Strengths and limitations of the study

Data analysis at the project level (overall coverage) is relatively simple and supervisors can manually tally the data at the community level to make timely decisions. However, to assess the quality of data collected by the registry workers and to assess their skills, we compared the data collected by the LQAS auditors to those collected by the MNHR data collection staff using standard quality control metrics. This information was found to be useful for the senior managers of the project, who could then plan for refresher training.

We experienced several challenges/limitations when we applied this design to our birth registry. In geographically scattered populations, random selection of villages and then households was challenging and time consuming. We learned that it is important to have an understanding of the geographical boundaries of the clusters. For this reason, we believe that for areas where a birth registry is maintained, a recurring household survey is needed (perhaps once a year) to obtain or update information on the obstetric history of women in the reproductive age group and the age and sex structure of the population. This information can be a good resource for LQAS sample selection and for comparison of LQAS and registry data.

For LQAS, more than one cycle could be carried out in a year, and questions could also change during each cycle to develop a broader picture of the quality of the study data. These iterations can be done according to the needs of a project.

Conclusion

LQAS methodology is a useful tool for monitoring coverage and data quality for community-based projects. In this study, two low performing areas were identified according to our decision rule.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.