Using mixed methods to evaluate perceived quality of care in southern Tanzania.

OBJECTIVE: To compare perceived quality of maternal and newborn care using quantitative and qualitative methods.
DESIGN: A continuous household survey (April 2011 to November 2013) and in-depth interviews and birth narratives.
SETTING: Tandahimba district, Tanzania. PARTICIPANTS: Women aged 13-49 years who had a birth in the previous 2 years were interviewed in a household survey. Recently delivered mothers and their partners participated in in-depth interviews and birth narratives. INTERVENTION: None. MAIN OUTCOME MEASURES: Perceived quality of care.
RESULTS: Quantitative: 1138 women were surveyed and 93% were confident in staff availability and 61% felt that required drugs and equipment would be available. Drinking water was easily accessed by only 60% of respondents using hospitals. Measures of interaction with staff were very positive, but only 51% reported being given time to ask questions. Unexpected out-of-pocket payments were higher in hospitals (49%) and health centres (53%) than in dispensaries (31%). Qualitative data echoed the lack of confidence in facility readiness, out-of-pocket payments and difficulty accessing water, but was divergent in responses about interactions with health staff. More than half described staff interactions that were disrespectful, not polite, or not helpful.
CONCLUSION: Both methods produced broadly aligned results on perceived readiness, but divergent results on perceptions about client-staff interactions. Benefits and limitations to both quantitative and qualitative approaches were observed. Using mixed methodologies may prove particularly valuable in capturing the user experience of maternal and newborn health services, where they appear to be little used together.

Introduction

The widely used Donabedian model suggests that information about the supply of quality of care is best drawn from learnings around structure (the setting in which care is delivered), processes (the interactions between providers and clients) and outcomes [1]. Because of its links with care seeking [2-7], also important are user perceptions of quality. With respect to maternal and newborn health (MNH), perceived quality is multi-dimensional, focussing on: treatment with respect and dignity; being provided information and education; having physical comfort; involvement of social supports like friends and family as needed; courtesy and availability of staff; trust in provider treatment; client autonomy and participation in decision-making; and reliance on confidentiality [8-11].

Measurement of multi-dimensional quality outcomes is challenging. Qualitative methods used to assess perceived quality of care commonly rely on focus group discussions and in-depth interviews [12-18]. Although often providing rich insights, they are time-consuming and cannot provide population-level measures that can be tracked over time and that can represent different population subgroups. Quantitative population-level survey methods, often populated with a number of scales linked to dimensions of quality, are widely used to gain measures of perceived quality of MNH from users [19-26]. However, heterogeneity between settings limits the ability of structured surveys to capture context-relevant dimensions of quality of care, and respondents may lack a reference point with which to benchmark their responses [27, 28]. Finally, Batchelor et al. note that surveys tend to yield disproportionately positive outcomes in terms of patient satisfaction with various measures of their care [29].

The World Health Organization's Every Woman, Every Newborn initiative and large-scale efforts by organizations like the White Ribbon Alliance have emphasized the importance of gaining the user's experience of MNH services as part of the drive to improve survival of women and newborns around the time of birth [30, 31]. The suggested literature around perceived quality or client satisfaction from the World Health Organization is dominated by surveys in clinical settings. Focus group discussions are mentioned occasionally, but the use of quantitative and qualitative measures together is not emphasized [25, 32–36], and there are few examples of mixed methodologies in evaluating perceived quality of MNH services in Sub-Saharan Africa [37-40].

Here we present findings from both quantitative and qualitative approaches to evaluate perceived quality of care among users of MNH services. We highlight where findings were similar, where they differed and suggest how overall measurement of perceived MNH quality of care could be improved.

Methods

We used a mixed methods study design in which quantitative and qualitative data around perceived quality of care in the same locality were independently collected and analysed.

Study setting

The study setting has been described in detail elsewhere [41]. Briefly, Tandahimba district in southern Tanzania has a population of 227 500, the majority of whom are rural-dwelling cashew farmers from the Makonde ethnic group [42]. The coverage of antenatal care and facility delivery is high, but the area has persistently high maternal (712 deaths per 100 000 live births) and neonatal (31 deaths per 1000 live births) mortality [43-45]. There are 34 health facilities, including 1 district hospital, 3 health centres and 30 dispensaries [46, 47].

Quantitative data collection

Quantitative data were generated as part of a continuous household and health facility survey [48]. The cross-sectional continuous household survey ran between February 2011 and November 2013. Independent probability samples of household clusters were drawn each month to represent outcomes at the district level. A modular structured questionnaire was designed to represent outcomes for women aged 13–49 years who had a recent birth. During this period, a total of 11 473 of 11 937 sampled households consented to participate (96% response rate). Twenty-five a priori measures of perceived quality of care were defined, derived from the literature reporting measurement of service quality perceptions, especially those carried out in African or low-income country contexts [19, 20, 38, 49–53]. These were then integrated within the structured questionnaire and pre-tested prior to data collection. Most questions had binary yes/no responses. Questions covered access to care and barriers to seeking care (8); confidence in the availability of staff, drugs and equipment (3); type of facility last accessed and reasons for seeking care (3); interaction with the provider (for example, ‘When you were [at the health facility], did the health worker(s) talk to you politely?’) (5); facility infrastructure (for example, ‘Did the facility seem clean to you?’) (4) and payment for care (2).

Qualitative data collection

Between May and October 2013, qualitative data were collected during 30–60 min semi-structured in-depth interviews (12) or birth narratives with mothers (23) who had recently given birth and fathers (13) whose partners had recently given birth. Although men were not interviewed in the continuous household survey, we felt they may have important contributions to make on the topic of quality of care. Two female research assistants, both very familiar with MNH practices in Tandahimba district, led interviews and birth narratives. Birth narratives were open, giving participants the opportunity to share their experiences from the start of their—or their partner's—labour until the post-partum period. In both semi-structured in-depth interviews and birth narratives, participants were probed about the care that was received throughout each step, what they did and did not like and how they felt health services could improve, if it was believed they should. From four villages across one division of Tandahimba district, respondents were purposively selected to reflect a broad range of perspectives, including different age, parity, place of the most recent child's birth (home, dispensary, health centre or hospital) and socioeconomic status. Mothers' ages ranged from 16–44 years. Mothers' parity ranged from 1–6. Fathers' ages ranged from 21–60 years. The number of children for each father ranged from 1–8.

Analysis

To align temporally with the qualitative data, continuous household survey data were restricted to the period April 2011 to November 2013 and included 1338 interviews with resident women aged 13–49 years who had a recent live birth and who reported having accessed health services for themselves or their newborns in the past 12 months. Data were summarized and descriptive statistics about participants and their responses to questions about their most recent experience of care within the past 12 months for them or their child were tabulated using the svy command in Stata 13 to account for the clustered survey design. Evidence of statistical difference in perceived quality outcomes by the level of health facility was determined using a weighted Pearson's χ2 test.

All transcripts from in-depth interviews and birth narratives were recorded, transcribed verbatim and translated from Swahili to English. Qualitative data were analysed thematically using constant comparison, in which data collection tools were adjusted to further explore emerging themes or divergent cases. Familiarization with all scripts was carried out and data were coded line by line and higher level themes were generated using NVivo 10 software. Representative quotations have been selected to indicate the most prominent themes.

To enhance the comparison of findings from the two data sources, both quantitative and qualitative responses were organized around the four categories of Framework for Evaluation of Quality of Care in Maternity Services (see Box 1): contact with human and physical resources; cognition; respect, dignity and equity; and emotional support [10]. Qualitative and quantitative findings were compared side by side to determine which findings were the same and which were different—i.e. to ascertain the convergent validity of the data [54].

(1) Contact with human and physical resources: impression of the state of the infrastructure, cleanliness etc., contact time with staff, impression of treatment and sense that staff are competent enough to provide care

(2) Cognition: information is conveyed in an understandable way, using acceptable language, and questions have been answered; women know their options and have real informed choice; reasons for care are explained; and information about post-partum care is effectively conveyed

(3) Respect, dignity and equity: women feel they have been treated with respect; women do not undergo unnecessary and humiliating procedures; cultural practices that do not interfere with quality are respected; women face no discrimination and services are priced appropriately for the catchment area

(4) Emotional support: women can maintain self-control and preserve their self-esteem; women choose their social support—typically who will be with them during labour; women are treated with honesty, kindness and understanding; staff are aware of their supportive role; and processes exist where providers can identify and respond to user expectations.

Ethical considerations

Both the quantitative and qualitative studies received favourable review from the ethics committees at the London School of Hygiene and Tropical Medicine (UK), Ifakara Health Institute (Tanzania) and the National Institute for Medical Research (Tanzania). Participant anonymity and confidentiality were respected throughout, and all participants underwent an informed consent process.

Results

In the household survey, 1338 women aged 13–49 years who had a live birth in the past two years and had accessed health services in the past 12 months prior to the survey were interviewed. The majority were aged 20–39 years, were married and were Muslim (Table 1). Most (36%) of women accessed services for their child, 26% went for a routine check-up for themselves, 9% of respondents went because they were sick, 8% went for reproductive health services and the remainder visited the facility for other reasons.

Table 1

Characteristics of respondents accessing health services in the past 12 months from the survey date

Participant characteristics	N	%b
Age
13–19	166	12
20–29	592	44
30–39	437	33
40–49	143	11
Total	1338
Marital status
Currently married	1001	75
Previously married	208	16
Unmarried but living with partner	30	2
Never married	99	7
Total	1338
Religious background
Christian	21	2
Muslim	1316	98
Totala	1337
Facility type most recently accessed
Hospital	193	14
Health centre	162	12
Dispensary	983	73
Total	1338

aOne missing value.

bPercent does not always add up to 100 due to rounding.

Contact with human and physical resources

From the survey, reported confidence in finding staff available when assessing a health facility was high (93%, 1244/1338, 95% CI 91–94). However, only 61% (817/1338, 95% CI 59–64) were confident that the facility would have sufficient drugs and equipment.

However, qualitative data suggested that concerns about both issues affected perceived quality. Particularly in reference to trying to access dispensaries, many participants pointed out that there were few staff, which further contributed to poor quality of services.

There are two only two attendants in this centre, so the service availability is very low. Even if there was improvement done earlier, still there is a need to improve the staff so that there will be quality service at the health centre. (Father, 55)

Many respondents also expressed frustration at the lack of drugs and equipment at the health facilities. Vaccinations and other medications were commonly raised as things that may not be available at the health facility.

You are told there are no vaccines and that they are not available, and if you are sick the medicine is not available. They prescribe it and you go and buy it at the pharmacy. (Mother, 16)

Survey questions about the client experience during last healthcare visit revealed that at least 70% of respondents were satisfied with the infrastructure of facilities. However, there was statistical evidence to suggest that infrastructure in hospitals was perceived to be of lower quality than at other levels of care, with just 60% of respondents reporting that drinking water was easily accessible when they visited a hospital (P = 0.002), and 62% reporting that they had perceived the hospital toilet to be clean (P = 0.002) (Table 2).

Table 2

Contact with resources: user-reported accessibility of drinking water and perceived facility and toilet cleanliness

	n/N	%	95% CI
Drinking water was easily accessible
All women	940/1338	70	67–74
Level of facility accessed
Dispensary	707/983	72	68–76
Health centre	118/162	73	67–78
Hospital	115/193	60	52–67
Facility perceived to be clean
All women	1312/1338	98	97–99
Level of facility accessed
Dispensary	963/983	98	97–99
Health centre	159/162	98	95–99
Hospital	190/193	98	95–99
Toilet (if accessed) at facility perceived to be clean
All women	250/314	80	73–85
Level of facility accessed
Dispensary	178/208	86	80–90
Health centre	29/37	78	58–91
Hospital	43/69	62	48–75

Similarly, a lack of access to water in hospitals was mentioned in the qualitative data: water was not always available and access was restricted to certain times during the day. During childbirth, for example, whoever accompanied the woman to the health facility may be expected to collect water from elsewhere or bring it from home.

Imagine that you need water in the morning and you are told to wait until 3 p.m.; a new baby has come and you need water for washing, etc. How can you force someone to wait until 3 p.m.? (Father, 38)

Cognition: understanding care and being aware of options

In the survey, 95% of respondents reported that the health worker listened them to carefully and 88% suggested that they understood all aspects of their care. The one dimension that was reported less positively was on sufficient time given to ask questions of health workers, to which only 51% of respondents responded “yes” (Table 3).

Table 3

Cognition: user-reported experiences asking questions, being listened to and understanding providers

	n/N	%	95% CI
Respondent given enough time to ask questions
All women	684/1338	51	48–54
Level of facility accessed
Dispensary	498/983	51	47–54
Health centre	86/162	53	45–61
Hospital	100/193	52	44–60
Health worker listened carefully to respondent
All women	1268/1338	95	93–96
Level of facility accessed
Dispensary	933/983	95	94–96
Health centre	152/162	94	89–96
Hospital	183/193	95	92–97
Diagnosis, and/or advice and/or treatment understood
All women	1171/1338	88	86–89
Level of facility accessed
Dispensary	861/983	88	86–89
Health centre	144/162	89	82–93
Hospital	166/193	86	79–91

In contrast, using qualitative methods, both positive and negative interactions with providers were mentioned by participants. If clients received thorough explanations of their care and were given education, they spoke highly of their interaction with the provider. When services met women's expectations of care, they were satisfied and deemed the quality of care to be high. However, it was often indicated that no explanation or education was provided, or the client felt ignored. There were instances where women described having a vaginal exam, a catheter inserted or being given oxytocin and generally not understanding why.

The nurse put on gloves and inserted her hand in the vagina. [Interviewer: What was she looking for, did she tell you?] She didn't tell me anything. (Mother, 29)

They didn't educate me on how to give birth, so I didn't know anything. And there wasn't any doctor who told me, ‘you are supposed to do it this way’; I was just suffering there until the [birth] happened spontaneously. I was alone and there was no one there. (Mother, 26)

Respect, dignity and equity

Survey measures of respect, dignity and equity were limited to perceptions about health worker politeness and out-of-pocket payments (used as a proxy for equity). On the basis of their last experience of accessing healthcare, women reported a universally high degree of health worker politeness (95%) (Table 4). Eighty-eight percent of respondents did not have to make any out-of-pocket payments for care (excluding transportation and food), but there was statistical evidence that they were more likely to have to make payments at health centres (21%) and hospitals (20%) than at dispensaries (9%) (P < 0.005).

Table 4

Respect, dignity and equity: user-reported politeness of provider and out-of-pocket payments

	n/N	%	95% CI
Health workers polite
All women	1271/1338	95	94–96
Level of facility accessed
Dispensary	932/983	95	94–96
Health centre	158/162	98	94–99
Hospital	181/193	94	88–97
No out-of-pocket payment (other than for food or transport) was made
All women	1174/1338	88	86–90
Level of facility accessed
Dispensary	892/983	91	89–92
Health centre	128/162	79	69–86
Hospital	154/193	80	70–87

Qualitative findings centred on the instances of harassment or abuse that women reported during their care.

[The health facility staff] don't have good language. I don't know whether it's because of being tired or it is their behaviour, for example, during delivery one is tired and cannot do anything, but they become furious and abusive, accusing us that we are lazy. (Mother, 36)

On the note of equity, care should be affordable to individuals in the catchment area of a facility. According to national policy in Tanzania, services and medications for pregnant women and children under five are free of charge, yet many women reported that health staff had recommended they purchase items, which reflected particularly badly on perceived quality. It was acknowledged that, if the family lacked the capacity to buy what was needed, the client would simply suffer and would not be treated as she should be. With the need to make out-of-pocket payments, care may be inequitable, with some clients being precluded from care due to financial constraints.

If you don't have means, you just accept the situation, and if you don't have money to buy drugs you just leave [without receiving services]. (Mother, 26)

Emotional support

Overall, survey respondents found staff helpful, with 91% reporting positively on this measure.

Qualitative data also highlighted some positive interactions between staff and clients. Those in which staff were gentle or spoke very kindly to clients were remembered and contributed to the perception that care was good.

[The nurse] cared for me a lot. I had no [food] … she went out of the gate to call [my mother] to come and give me [food] and then she brought me [food] where I was. (Mother, 19)

Negative interactions were also reported, including a sense that staff generally did not care about clients or have a ‘heart’ for the work that they were doing was raised by a number of participants.

Frankly speaking, the nurses whom we have, they don't have that good heart … When you go to the facility, they don't care about you … they can come at their own time, and once she comes she will use harsh words. (Father, 60)

Quantitative and qualitative findings compared

Table 5 highlights some of the key findings from both quantitative and qualitative data collection methods, as organized around the four domains: (i) contact with human and physical resources; (ii) cognition; (iii) respect, dignity and equity; (iv) emotional support.

Table 5

Comparison of learnings about perceived quality of care using quantitative versus qualitative methods

Dimension of perceived quality of care	Learnings from quantitative data	Learnings from qualitative data
Contact with human and physical resources	Overall, 93% of respondents felt confident that staff would be present, and only 61% felt confident that required drugs and equipment would be present30% of respondents reported difficulty accessing drinking water, which was particularly true at hospitals (40%)	Frustration at a lack of staff was expressed predominantly by participants seeking care from dispensaries. There was also a widespread sense that drugs and equipment could not be reliably found and would have to be purchasedGenerally, participants had a good sense of what services they should be receiving, and if those expectations were met, they were satisfied. More than half of participants reported that services rendered met their expectations, despite a third of these participants also commenting on being ignored or harassedOnly respondents who were accessing hospitals commented on lack of water
Cognition	91% of respondents found health facility staff to be helpful, and 88% felt that they understood their diagnosis and treatment; however, only 51% of respondents felt they had enough time to ask questions	Almost half of the participants spoke of specific instances in which they were ignored, a procedure was carried out without them being given any information, or that they had asked for information and were dismissed
Respect, dignity and equity	95% of respondents felt that health facility staff were polite and that they were listened to by health facility staff. Respondents aged 13–19 years were also more likely to report that provider attitudes were a barrier to seeking care (41%), compared with 23% of women aged 30–39 years and 30% of women aged 40–49 years (results not shown above)12% of respondents had to pay out-of-pocket for care, which was higher in health centres (21%) and hospitals (20%). Overall, 60% of these respondents paid an amount they were expecting, with 49% of those receiving care from a hospital and 53% of those receiving care from a health centre paying an unanticipated amount, compared with only 31% of those accessing care at a dispensary	Half of the participants mentioned the harassment and disrespect of clients, many elaborating on examples of abuse to them or their spouse. Women giving birth in the hospital in particular mentioned that they had laboured almost entirely on their own, with a health worker providing assistance as the baby was almost fully—or was fully—outMany respondents indicated that they accrued many unexpected expenses and stressed the unfairness they felt in having to pay anything at all
Emotional support	Overall, 91% of respondents felt that the facility staff were helpful.	Almost all women who delivered at a health facility described being with only a health facility staff during delivery, with their social support allowed to see them afterAmong participants who did not report harassment or being ignored, some specifically indicated when staff had been particularly kind or gentle with them or their partners

Discussion

Qualitative and quantitative findings were similar regarding the lack of confidence in available drugs and equipment, the need for out-of-pocket payments and difficulty accessing water. However, as found in other low-income country settings [19, 52, 55, 56], survey responses around staff politeness (95% positive), helpfulness (91% positive), listening (95% positive) and understanding care (88% positive) had very homogenous responses using quantitative methods. Conversely, 46% of participants in in-depth interviews and birth narratives highlighted harassing or disrespectful care and 38% reported being ignored or having their queries dismissed. Likewise, in other settings, negative reports of quality of maternal or newborn care seem to be largely derived from qualitative methods [12-14]. A lack of clear benchmarking—what is ‘helpful’? Compared with what?—within quantitative surveys may explain more homogenous results. Furthermore, a recent review of determinants of user satisfaction in maternal health suggested that very high satisfaction ratings by women might reflect a lack of awareness and exposure, especially among women in low-income country settings [57].

Honest reporting of perceived quality requires confident participation. An important consideration is that there may be a different relationship between participants and different types of data collector. Survey enumerators may be perceived as more closely linked to the government or to health facilities directly, which may lead participants to censor their responses. However, using qualitative methods, developing trust and openness with the participant to the greatest extent possible is essential.

There are, of course, well-documented ways to use mixed methods to draw on the strengths of each, possibly using qualitative methods to provide formative research that can inform the creation of context-specific quantitative tools that optimally measure what they set out to, or using qualitative research to explore and elaborate on quantitative research findings [54, 58, 59]. Others have used this approach in measuring perceived quality of care with success [37-40]. However, the use of such mixed approaches to explore user perspectives within the confines of maternal and newborn care, particularly in a Sub-Saharan African context, is limited.

Limitations

Although attempts were made to align the continuous survey with a Tanzanian context, a lack of previous examples from reported literature meant that questionnaires applied in other low-income country settings were used to inform the development of the survey module on perceived quality of care: as a consequence, some context-specific information may have been lost. Using qualitative methods, the majority of data come from birth narratives, in which mothers and fathers were given much more flexibility to discuss what mattered most to them in their care, and were not necessarily guided to speak to the same measures of quality of care that the survey addressed.

Conclusion

There are benefits to both quantitative and qualitative research methods when assessing perceived quality of care. Population-level estimates that can only be achieved through quantitative methods may be of more value to policymakers. However, these methods require a priori assumptions about what constitutes quality of care in a given setting. Qualitative research methods are time-consuming and can be resource-intensive, and although generating transferrable results, cannot produce the generalizability that researchers and policymakers often desire. Using mixed methodologies to evaluate perceived care may produce valuable population-level estimates with rich description and nuance.