Validity and reliability of maternal recall of pregnancy history and service use among signing Deaf women: a cross-sectional descriptive study from South Africa.

INTRODUCTION: There is little credible quantitative data on pregnancy histories and outcomes for disabled women in low-income and middle-income countries. The purpose of this study, based in Cape Town, South Africa, was to test the reliability and validity of maternal recall of pregnancy history and service use among a sample of Deaf women who use South African Sign Language (SASL).
METHODS: We interviewed 42 signing Deaf women of childbearing age (18-49 years) in SASL using a structured questionnaire in July 2016. To assess reliability, seven participants (16% of the sample) were reinterviewed by different interviewers under the same conditions after 10-30 min. For the analysis we used (1) Cohen's kappa, an inter-rater statistical method, and (2) overall percentage agreement. Validity was explored by comparing the participants' pregnancy history to the Western Cape Provincial Health Data Centre (PHDC) database.
RESULTS: The reliability results showed that out of 19 questions 14 demonstrated substantial to perfect agreement kappa scores (kappa between 0.61 and 1) and 5 had the lowest kappa agreement scores (kappa <0.61). With respect to percentage agreement, participants provided identical responses in 87% cases. Overall, women provided more reliable responses to pregnancy outcomes compared with demographic information. Validity results showed that 29 out of 35 Deaf women provided survey responses that matched or nearly matched (83% agreement) the PHDC database for birth history and delivery location.
CONCLUSION: This study suggests that for this sample of signing Deaf women recall of pregnancy history and service use is reliable and valid. Extending this approach to other similar populations will require further research, but it is important that methods to access hard-to-reach disabled populations are developed so that health system responsiveness to marginal populations can be based on robust evidence. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study focuses on a vulnerable hard-to-reach population that is under-researched.

Its findings suggest possible methods to generate reliable and valid data for Deaf women as a hard-to-reach population using simple tools suited to this population.

Not all pregnancy-related outcomes were available to explore in the analysis.

Assessment of reliability was based on a relative short interval between testing.

Reliability may have been reduced if there had been a longer time gap between the test and retest interviews.

Introduction

It can be reasonably estimated that disabled women in low-income and middle-income countries (LMICs) are at disproportionate risk for poor pregnancy outcomes. According to the United Nations Development Program, 99% of all maternal deaths occur in LMICs1 where 80% of persons with disabilities reside.2 Studies from the USA and Europe indicate that women with disabilities are at an elevated risk of preterm and low birthweight infants,3–5 yet these findings have not been confirmed in resource-limited settings. To our knowledge, there is little credible quantitative data on pregnancy histories and outcomes for disabled women nor on their use and experiences of antenatal care and childbirth services in LMICs.

Disabled persons pose a challenge for obtaining credible quantitative data in that they are considered an example of a hard-to-reach population.6 7 As a hard-to-reach population, they remain largely hidden and inaccessible for research and healthcare.6 8 In particular, Deaf populations are increasingly left out of research due to barriers of communication, mistrust of researchers and inaccessible procedures.8 The result is that there are few methods that provide valid and reliable data and a representative or probability sample that allows extrapolation to the wider population.

Despite the challenges, credible data are required to ensure the needs of hard-to-reach disabled women are addressed by policymakers. For example, South Africa is currently embarking on a National Health Insurance (NHI) plan. Valid and representative data on maternal health status and use of maternity services are needed to ensure their access to healthcare under the NHI and advance their sexual and reproductive human rights.

Although maternal recall is often used to characterise reproductive histories, studies examining validity through agreement between maternal recall and routine hospital records have largely been confined to high-income countries.9–11 Gichane et al 12 carried out the first study in Cape Town, South Africa, to assess pregnancy outcomes and maternity service use in a sample of signing Deaf women. Deaf (capitalised) refers to those permanently, sensorily disabled people with congenital or early-onset deafness and whose first language is signed, referred to in this country as South African Sign Language (SASL). The study aimed to provide a quantitative profile of Deaf women (aged 18–49 years) by (1) maternal health status, (2) use of maternity services, (3) experiences of the maternity services and (4) women’s recommendations for improvements. The overall results showed that Deaf women differed in key pregnancy outcomes. For instance, the sample fertility rate of 1.72 was lower than the South African population rate of 2.40 (T. Moultrie, personal communication, March 18, 2016). The study also showed a higher rate of miscarriage of 31% for Deaf women versus 16% found in a population-based study in South Africa.13 These findings12 are consistent with other studies of pregnancy history in South Africa which shows that most women have received antennal care during pregnancy14 but that there is a delay in seeking care beyond the first trimester.15–17 This delay in seeking antenatal care increases the risk of adverse pregnancy outcomes.18

This paper is therefore a follow-up of Gichane et al 12 to assess the reliability and validity of the data collected by questionnaire. Reliability refers to the ‘consistency of a measure’19 over time and place and between interviewers; validity refers to the ‘extent to which a concept or concepts (in our case pregnancy history and pregnancy outcomes) are accurately measured.19 If the questionnaire previously used in the study by Gichane et al 12 is valid and reliable, this tool could be explored in other settings to generate information for programmes and policies to improve maternal and child health for this hard-to-reach population.

Methods

The methods for the main descriptive survey have been described in detail elsewhere.12 In brief, the target population was signing Deaf women, of childbearing age, residing in Cape Town and aged between 18 and 49 years.

The sampling selection and recruitment strategies were based on a range of snowballing techniques that have been adapted to local context, including Deaf people’s use of various forms of communication technology.16 17 Participants were primarily recruited via short message service and WhatsApp messages sent to a database of Deaf adults in Cape Town developed using non-probability snowball sampling. The database was originally developed to advertise medical interpretation services. Seven people representing a range in age, gender and residential address were recruited as initial seed participants. These individuals were tasked with soliciting phone numbers from their Deaf peers, as well as asking each contact for additional referrals of people in their social networks. Each referral was contacted to explain the purpose of the database and to provide consent to be included. A total of 220 contacts were collected and included in the final database.

Data collection and administration of the questionnaire took place on two Saturdays in July 2016 at two well-known gathering spaces for the Deaf community of Cape Town. After informed consent, the questionnaire was administered by trained SASL interpreters and interviewers trained in survey administration and research ethics. During each interview, the interpreter signed the question and the participant signed their response. The interpreter then voiced the response in English which was captured on an online form by a research assistant.

The questionnaire

The questionnaire, described in Gichane et al,12 included 22 closed and open-ended questions on demographic measures and maternal health service usage and pregnancy outcome measures. Closed-ended questions related to individuals’ characteristics, such as their education, employment, number of pregnancy; while for open-ended questions we asked questions regarding (1) Deaf women’s experiences of maternity healthcare services and (2) their recommendations on how maternity healthcare services—antenatal and delivery—should be provided for them. Questionnaires, in addition to SASL, were made available in all three local languages spoken in the Western Cape Province (Afrikaans, English and isiXhosa). Participants were also asked permission to access their medical records within the Health Department as part of the consent procedure.

Assessing the pilot study reliability

All participants (42) were asked whether they were available for a second interview. A total of seven participants (16% of the sample) were reinterviewed by different interviewers under the same conditions after 10–30 min to assess reliability of the questionnaire. The first four participants were interviewed twice on the first day of the data collection, while on the second day three participants were randomly selected to be reinterviewed.

The Cohen’s kappa,20 21 an inter-rater method, and overall percentage agreement were used to assess the questionnaire’s reliability. The Cohen’s kappa results can vary from −1 to 1; results equal to 1 imply perfect agreement, while results equal to −1 imply no agreement and the distributions are subject to random chance (table 1). The percentage absolute agreement was calculated by dividing the number of the participants answers that are in agreement by the total number of participants (n=7). Only questions that capture individuals’ characteristic, such as their marital status, education, employment, number of pregnancy and number of children, were used in this analysis. Therefore, three open-ended questions were excluded from this analysis. In total, 19 questions were included in the reliability analysis.

Table 1

Cohen’s kappa Interpretation

Interpretation	Kappa
Complete agreement	1
Almost perfect agreement	0.81–1.0
Substantial agreement	0.61–0.80
Moderate agreement	0.41–0.60
None to slight agreement	0.01–0.20
No agreement	0

From refs.20 21

Assessing the pilot study validity

Validity was measured by comparing the women’s answers regarding their pregnancy history to the health data available from the Provincial Health Data Centre (PHDC). According to the Western Cape Department of Health (WCDH), the PHDC is an initiative of WCDH which capitalises on the durable investment in both a single hospital information system and a patient master index over the past decades in order to consolidate all person-level clinical data in a single environment. The PHDC functions as a nascent health information exchange, combining standards-based interoperability solutions with bespoke data take-on processes to consolidate data from most of the sources on a daily basis’ (N. Zinyakatira, personal communication, November 3, 2017). The PHDC data therefore served as the gold standard for assessment of patient response validity for maternity service attendance in the Western Cape.

Data were requested from the PHDC database for each participant on the number of pregnancies, number of children, whether the individual had a miscarriage and/or termination, latest due date or child’s birthday, if the youngest child was delivered in the WC, health facility attended when pregnant with youngest child and health facility at which youngest child was delivered.

Once permission was granted by the WCDH, a secured list with the participants’ Identity Numbers (IDs) and names was sent to the PHDC. Only the names and IDs of women who gave written informed consent in the main study were used. The PHDC provided the health data, by participant’s name and ID, which was then cleaned and reorganised into tables by the first author of the paper, MFM, for the analysis.

Out of 42 Deaf women that consented to participate in the study and have their records checked, 7 were excluded from the validity analysis because two participants did not have children; while 5 participants names or IDs collected during the interviews were not the same used in the WCDH facilities. In total, 35 Deaf pregnant women were included in the validity analysis. The results were grouped into the following categories:

‘Match’ refers to those answers collected in the questionnaire that matched completely the information provided by the PHDC.

‘Nearly match’ refers to small differences between answers given in the questionnaire and the information provided by the PHDC. These differences included typographical errors that could not be 100% confirmed or a missing record that was likely to have been present, as explained in the ‘Results’ section.

‘No match’ refers to those cases where there were answers from our questionnaire responses did not match information in the PHDC database.

Patient and public involvement

MH and LL have a long-standing (>20 years) research and advocacy relationship with the Deaf community of Cape Town. Their needs have informed the research question and outcome measures for this study. Potential participants were involved in the recruitment—via their WhatsApp groups. However, study participants had no involvement in the study design or conduct of the study. The findings from this study will be disseminated to the participants at their regular gatherings, such as ‘Third Sunday’ at a well-known non-governmental organisation in Cape Town for feedbacks.

Results

Participants

In total, there were 42 signing Deaf women who met the eligibility criteria and who consented to participation. Participants ranged in age from 18 to 49 years and 57% were married or lived with a partner (57%). The majority of participant’s highest level of education was between grades 7 and 12. Over half were unemployed (59%) and received a monthly disability grant. Most women (62%) had between one and two pregnancies. Thirty-one per cent had at least one miscarriage and 19% had had an abortion.

Reliability

With respect to percentage agreement, of the 133 items for which repeat measures were available (7 participants × 19 questions each), participants provided the same response to both interviewers in 87% of cases (n=116 items in agreement). The analysis of kappa values, shown in table 2, showed that of the 19 questions, 8 achieved complete agreement (kappa=1), 3 achieved almost perfect agreement (kappa=0.81–1.0) and 3 achieved substantial agreement (kappa=0.61–0.80). Answers to questions on monthly income, education, termination of pregnancy and number of months (or weeks) at first booking when pregnant with the youngest child had the lowest kappa agreement scores (kappa <0.61). The nature of discrepant answers for income were (1) ‘between R4000 and R10000’ versus ‘refused to answer’; (2) ‘disability grant’ versus ‘

Table 2

Distribution of kappa scores for 19 questionnaire items

Kappa range (interpretation)		Questionnaire items, n
Complete agreement	1	8
Almost perfect agreement	0.81–1.0	3
Substantial agreement	0.61–0.80	3
Moderate agreement	0.41–0.60	3
None to slight agreement	0.01–0.20	1
No agreement	0	1

Table 3

Reliability results

Question	Kappa	Percentage agreement	Nature of discrepant answers
In which of the following languages do you prefer to read or write?	1.00	7/7	None
Marital status	0.70	6/7	Widow versus divorced
What is your highest level of schooling/education?	0.46	6/7	Below grade 7/standard 5 versus don’t know
What is your employment status?	0.61	6/7	Unemployed versus employed
Monthly income	0.15	3/7	Between R4000 and R10 000 versus refused; disability grant versus <R4000; <R4000 versus between R4000 and R10 000
How many times have you been pregnant?	1	7/7	None
How many children do you have?	1	7/7	None
Did you ever have a miscarriage?	1	7/7	None
Did you ever lose a baby at birth?	1	7/7	None
Did you ever lose a child later on?	1	7/7	None
Did you ever have a termination of pregnancy?	0	6/7	Yes versus no
How many months pregnant were you when you went to book at clinic with your youngest child?	0.45	3/7	4 months versus <2 months; 5 months versus 4 months; did not book versus 6 months
Which clinic did you attend when you were pregnant with the youngest child?	0.82	6/7	Did not attend versus Eastern Cape
Did you attend a hospital when you were pregnant with the youngest child?	1	7/7	None
If you attended hospital, what was its name?	0.81	6/7	Tygerberg versus Eastern Cape
How many times did you attend clinic when you were pregnant with your youngest child?	0.70	6/7	Did not attend versus once
In which province was your youngest child born?	1	7/7	None
In which type of healthcare facility did you deliver the youngest child?	0.59	6/7	Hospital versus Midwife Obstetric Unit
What was name of the hospital or clinic?	0.81	6/7	Gugulethu, KTC versus Tygerberg

Validity

The validly was explored through comparing the study participants’ pregnancy history to the Western Cape PHDC database. In total, for 16/35 participants records given in the questionnaire matched completely the information provided by the PHDC; while 13/35 ‘nearly match’ (table 4). For the ‘nearly match’, the small differences included the following: 6 out of 13 Deaf women who reported a pregnancy experience had a record confirmed in the Provincial Database of going into labour but had no confirmation of the actual birth of the child in the PHDC database, for reasons related to failure to register the child. This was considered (1) ‘near match’ since the provincial record confirmed an episode of labour which likely ended in a delivery and (2) another ‘nearly match’ was a note of a different birth facility attended. Because a participant might have gone into labour at one facility but have been transferred to another facility due to complications of labour, this may not be picked up in the routine database.

Table 4

Validity results

Validity	Match	Nearly match	Not match
	16	13	6

Validity is measured by comparing the instrument answers to the Western Cape Provincial Health Data Centre database. n=35.

All the six Deaf women who did not match claimed to have had a child but had no record from the PHDC database to confirm this history. Our focus on linkage was on pregnancy history, therefore a missing record on the PHDC database could reflect that no pregnancy-related encounter with health facilities took place in the province or that participants’ pregnancy history was not captured by the PHDC database (eg, old records).

Overall, table 4 validity results show that 29 out of 35 (83% agreement) Deaf women had their survey answers matched or nearly matched to the PHDC database.

Discussion

This study tested the reliability and validity of answers to a questionnaire to assess pregnancy experience, utilisation of maternity services and pregnancy outcomes reported elsewhere among signing Deaf women in Cape Town, South Africa.12 To our knowledge, this is the first study to use a health system database to assess the validity of Deaf women’s recall of their pregnancy experiences in LMICs. The results suggest that the piloted methods provided reasonably reliable and valid data on this hard-to-reach population.

Overall, the data appeared highly reliable. Participants provided identical responses in 87% cases. Demographic indicators that showed lower levels of inter-rater reliability included data on income and, to a lesser extent, education. This is similar to findings in other studies that suggests that questions on income might not be a reliable source and sensitive questions might be misreported.22 Further, because interviews were conducted face to face, and demographic questions were asked at the outset of the interview, participants may have been especially hesitant to disclose this information. In order to improve reliability for such questions, survey methodologists suggest asking sensitive information later in the interview once the participant and interview have built rapport.22

For the reproductive history, the main outcomes (parity, miscarriages) achieved a high level of reliability. However, reliability was much lower for timing of first antenatal visit which requires more complex and sometimes long-term recall. The most recent pregnancy of some of the women who were reinterviewed may have been several years ago which may explain their different responses within a short period of time. Recall bias is especially common when the health condition or event occurred a long time ago, is frequent or if it was not particularly salient to the individual.23 It is possible that women who had healthy pregnancies or already had multiple pregnancies may have more of a challenge remembering the initiation of antenatal care because it felt like a routine event. Timing of antenatal initiation is an important outcome as it can have significant impacts on the health of the mother and infant.18 Developing methods to aid complex recall, can better improve our understanding of the maternal health of Deaf women.

The validity, measured as agreement between women’s report of their pregnancy history compared with the health data available from the PHDC, was found to be accurate (83% sensitivity). Women reported key pregnancy events including termination, miscarriage and birth location with high accuracy. These results confirm many studies comparing maternal recall with medical or government records of pregnancy outcomes.24–26 On average, mothers are able to recall characteristics of their pregnancy, birth outcomes and mode of delivery.13 24 One study found that women are able to recall salient pregnancy events including complications and pregnancy outcomes up to 15 years post pregnancy.24 Future studies should assess validity of other pregnancy outcomes including birth weight and medical complications of Deaf women.

Most participants who used the public healthcare system in the Western Cape were matched. However, the assumption that the WC Datacentre is the gold standard may not be the case because it is fairly new, having only been implemented in 2013. Use of electronic medical records in sub-Saharan Africa is just starting to increase traction.27 Clinics and hospitals in the Western Cape started capturing the records electronically at points in time such that earlier data might have been less consistent. The six Deaf women that had their pregnancy records ‘no match’ said that they delivered in WC but the records could not be found in the PHDC database. This linkage failure could be due to several issues: discrepancies in the names supplied during the research study and those used in health facilities, WC clinics and hospitals failed to enter patient information in PHDC database, or old records may be missing (for instance, some women gave birth in 1988 and 1996). Further investigation of the quality control measures employed by the PHDC database are necessary. Nevertheless, the results suggest that reports from Deaf women provide reliable and valid data on which to describe their pregnancy history.

Implications

Findings from this study have several implications for research with Deaf populations. First, the high level of validity self-reported outcomes is particularly promising given the challenges of obtaining medical records in LMIC.27 The Western Cape of South Africa has more resources and a more robust healthcare system compared with other regions of South Africa.14 In more rural areas, it would be nearly impossible to find a database of health information such as the Western Cape PHDC database. Establishing the validity of self-report with other health outcomes with Deaf populations is an important next step.

Second, measures and methods need to be developed to improve the complex recall of pregnancy-related events. Some methods to improve recall include memory aids or adjusting the recall period.23 Creating tools that are culturally relevant and specific to Deaf populations is necessary.

Limitations

The following limitations should be considered in the interpretation of the results. First, there were many pregnancy-related markers which were not explored in the analysis, therefore we cannot generalise the results to all pregnancy history measures. Second, inter-rater reliability was measured with a small sample over a short period of time (10–30 min); thus, we cannot comment if participants’ responses would have remained consistent if there would have been a longer time gap between the test and retest interviews.

Conclusion

Findings from this study indicate that the protocol used in Gichane et al 12 yielded valid and reliable results. This suggests that it is possible to develop methodologies that will produce reliable and valid data for Deaf women using simple tools that are suited to this population in question. Extending this approach to other populations will require further research, but it is important that methods to access hard-to-reach populations are developed so that health system responsiveness to marginal populations can be based on robust evidence.