Developing an Evaluation Model for Maternity Care: A Mixed-Method Study from Iran.

BACKGROUND: Maternity care is an integral part of primary health care (PHC) systems worldwide. This study aimed to develop a new model for evaluating the maternity health program (MHP) in Iran.
METHODS: In this mixed-methods study, first, the challenges of MHP were surveyed through systematic review and expert interviews. Next, to identify the existing shortcomings in MHP evaluation system, the SWOT technique, cross-sectional study and comparative analyses were used. Finally, the Delphi technique was used to reach consensus on developed evaluation standards.
RESULTS: The final developed evaluation model contains five dimensions including reproductive health/family planning, maternity health, health records, evaluation, and resources management. Overall, this model has 32 standards and 289 measures. The scores obtained for the sum of the measures in two importance and applicability criteria were 8.24 and 7.85, which these scores are estimated to be equal to 91.55 and 87.22 percent of the highest possible scores, respectively.
CONCLUSION: Considering the comprehensiveness of the obtained model, it is hoped that it could lead to performance improvement of the PHC centers in the area of maternity health.

Introduction

Maternity care was defined as “all types of care associated with pregnancy, delivery and postpartum period” (1). The studies were showed that receiving timely, adequate and high-quality care can reduce the mortality and morbidity rates in mother and child (2, 3). The mothers who do not receive care during pregnancy are eight times more likely to have premature infants compared with those who receive these services (4), and mothers who start receiving care in the third trimester of the pregnancy have a higher risk of delivering a low-weight baby (5).

The occurred social and economic changes in the twentieth century led to improved maternity care (6) so that the overall maternity mortality rate (MMR) from 1990 to 2013 dropped by 45% (7), but the distribution of this decline is not proportionate worldwide (7). Unfortunately, maternity health indicators, especially MMR are not satisfactory in developing countries (8). Annually, four million infants and half a million mothers die globally due to pregnancy-related problems, and 99% of these deaths occur in low and middle-income countries (LMICs) (9).

Despite the considerable achievement of MHP in Iran in recent decades, it is now facing some major challenges including the undesirable quality of provided cares, inappropriate economic and geographic access to some services, high rate of preventable mortality, mothers’ dissatisfaction from provided care, inequalities in health outcomes in different geographical regions, high number of premature infants with low birth weight (LBW), high infant mortality rates (10–13). Undoubtedly, these problems can be preventable with high-quality care (14).

Given that focusing on performance evaluation with quality-related indicators leads to effective and safe care (15, 16), one of the basic shortcomings of maternity care in LMICs is lack of comprehensive evaluation model (17). Due to the destructive impacts of this shortcoming on health outcomes (18), several studies have been conducted to develop guideline-based indicators in maternity care worldwide (19), but no effective action has been taken in this regard in Iran yet (20). Accordingly, this study aimed to develop a new model for evaluating MHP in Iran.

Methods

This mixed-methods study was conducted in 2017–2019. First, to identify the challenges of MHP in Iran, the published articles were reviewed using systematic review and some interviews were conducted with midwives, which occupied in MHP too. Next, to assess internal (strengths and weaknesses) and external (opportunities and threats) environment of the MHP evaluation system, the SWOT analysis was used. Then, existing monitoring instruments of MHP were assessed comparatively based on the context, input, process and product (CIPP) model. Moreover, across-sectional study was conducted for assessing the healthcare provider’s perspective regarding the identified challenges in pervious steps. After that, the improvement strategies for MHP evaluation system were achieved using expert panel. Finally, based on technical requirements and the identified challenges/strategies, the evaluation standards/measures were developed and approved from experts using Delphi technique.

The systematic review was conducted in September 2017 in Scopus, PubMed, Web of Science, Embase, SID, Magiran and Irandoc databases/search engines using the following search strategy:(((((maternity) OR maternity)) AND ((medical) OR health)) AND ((care) OR service)) AND Iran. There were two inclusion criteria: being published in English or Persian languages, and publication after the year 2000. In addition, to avoid publication bias, the grey literature was reviewed. Overall, 2438 articles were recognized and 107 articles were included finally (Fig. 1). In continue, the identified challenges were summarized in extraction tables and were concluded (21–23).

Fig. 1:

The process of searching and applying the inclusion and exclusion criteria of referees in the systematic review stage of this study

The interviews were conducted for identifying the challenges of MHP and improving strategies with participation of 27 midwives and 14 MHP professionals. The purposeful sampling was used and the interviews continued until data saturation. To analyze obtained data, the content analysis was used in which the concepts and existing themes in the data are extracted, interpreted and reported using a systematic approach. After recording the participant’s perspective, the coding phase was done and final themes and sub-themes were extracted from obtained codes (24). For enhancing the study rigor, the interview questions was determined by key informants, the data analysis phase was conducted by an external researcher in parallel with research team, and the results of interview analysis were approved by study participants (24).

In SWOT phase, the required data was obtained using a semi-structured qualitative questionnaire. The study participants comprised 47 experts in MHP field countrywide with at least 10 years of executive and managerial experience. The opinions provided by these experts were also categorized based on the content (25). In continue, the existing monitoring tools in MHP was assessed based on CIPP model. Accordingly, 9 monitoring checklist and clinical information system (SIB) were assessed and compared. Finally, the strengths and weaknesses of instruments were identified and discussed.

In the quantitative phase, 756 healthcare professionals, officers and general practitioners involved in MHP was assessed through cluster random sampling. Accordingly, the sample size was obtained by achieving a difference of 0.05 with an estimated standard deviation of 0.527 and finite population of 402 people. Considering the effect of the plan, 1.5 and 25 percent of the population was removed of the plan and the final sample size was 756. The study was conducted using a researcher-made questionnaire whose validity and reliability had been approved by experts. In the first step of questionnaire development, in addition to obtaining information from previous phases, the existing literature was also reviewed. Subsequently, the content and face validity of the questionnaire were examined by 15 experts in the field of MHP in Iran. Accordingly, all the questionnaire items were assessed in four scales in five aspects of relevance, transparency, simplicity, necessity and measurability; then Content Validity Ratio (CVR) and Content Validity Index (CVI) were examined; based on the below formula:

In this formula, the nE is referred to the number of experts who select the positive options; and N is referred to the total number of experts. In all cases, the acceptance score of 70% was the criterion for decision-making due to the responses of 15 experts. During this process, content validity of the questionnaire for CVI and CVR indexes (26) was approved with the scores 0.91% and 0.94%, respectively. Face validity of the questionnaire was also achieved by applying the opinions of the experts about the wording, way of expressing and appearance the questions. The reliability of the questionnaire was also confirmed by data from an initial sample of 50 participants from the study and the Cronbach’s alpha (α) was calculated to be 0.89. Initially, the deceptive analysis was done on data, then the analytical phase was done using ANOVA, t-test and Pearson correlation coefficient.

The Delphi technique was the last method used to reach a consensus on developed evaluation standards/measures. At this stage, initially developed measures were presented to 20 experts to be reviewed based on two “importance” and “applicability” criteria on 9-degree scale. The standards with a lower median score of 4 are eliminated, the standards with a median score of 7 to 9 are accepted and the standards with a median score of 4 to 7 enter the next Delphi round (27, 28). In accordance with the principles of Delphi technique and to maintain the value and credibility of the study, the response rate of the experts in each Delphi round should be more than 70 percent; and if the change rates in the scores given by the experts in two consecutive rounds are lower than 15%, it means a consensus (27,28) (Fig. 2).

Fig. 2:

The study implementation process as a flowchart

In this study, the participants were free to contribution, the privacy of participants was respected, the informed consent was gained from participants, and the study was approved by the ethics committee of Islamic Azad University of Semnan (IR.IAU.SEMNAN.REC.1397.001).

Results

The systematic review showed that the provided maternity care and its health outcomes faced fundamental challenges categorized in 15 dimensions. These dimensions and their related problems are briefly investigated in Table 1.

Table 1:

Identified problems associated with the MHP from the perspective of published articles

Dimension	Identified Problems
Safety	High rate of medical errors; birth injuries; high-risk pregnancies; lack of safe delivery and safety culture; complaints from caregivers; and adverse health results for mothers such as blood pressure, cardiovascular disorders and respiratory problems
Technical	Inappropriate referral system; inadequate and with delay pregnancy visits; poor compliance of provided cares with the clinical guidelines standards; providing low quality cares, inability to manage pregnancy-related illnesses; undesirable health outcomes
Quality	Lack of support groups, timely care, early prevention and diagnosis of health problems, continuity of care, respect, safety, access to the services, initial facilities, empathy, responsiveness, reliability, confidentiality, and accountability; poor education of mothers and involving them; lack of access to blood bank; poor clinical equipment and physical environments
Quality of the services	High maternal mortality rate due to bleeding, high blood pressure, eclampsia and pre-eclampsia, embolism, puerperal pyrexia and mother’s underlying diseases
Maternal mortality	High infant mortality rate (and its inequity and inequality) due to congenital anomalies, prematurity, respiratory problems, blood infections, low birth weight, embryo problems, and …
Infant Mortality	Inappropriate gain weight during pregnancy than what is recommended due to bad nutrition, inappropriate intake of nutritional supplements, inadequate educations regarding nutritional habits, inappropriate pregnancy age, and anthropometric features of mothers
Nutrition status	High rates of cesarean due to cultural issues, previous cesarean deliveries, induced demand by physicians, lack of physiological birth facilities, and inadequate maternal education
Cesarean section	High rate of LBW inappropriate family planning, oligohydramnios, preterm labor history in mother, bleeding during pregnancy, cesarean, mother’s illness and employment, and living in rural areas
Low birth	High rate of gestational diabetes due to pregnancy age, history of gestational diabetes in mothers and relatives, the status of BMI, history of abortion and history of macrosomia
weight	High rate of premature delivery due to pre-eclampsia, placental disorder, idiopathic reasons, mother stress, violence and trauma in mother, smoking and drug use in mother, poor prenatal care, high number of pregnancies, diabetes, thyroid problems and heart diseases in mother
Gestational diabetes	The incompetence of the personnel working in MHP in areas such as managing pregnancy and delivery problems, maternal and child care, family planning, pregnancy-related diseases, managing the chronic diseases and exposure to traumatic events, and referral system
Early	High rate of psychological problems such as depression, fear of pregnancy and delivery, postpartum and delivery psychiatric disorders; and high rate of violence against mothers
delivery	Inappropriate health education provided to mothers particularly regarding physiological changes, risk factors and possible problems during pregnancy; importance of oral health and physical activity; and dangers of smoking cigarette and hookah
Healthcare providers competencies	Inequalities in health outcomes such as birth of premature infants and LBW, mothers low weight, maternal and infant mortalities, injuries, babies’ disabilities, births spacing, pregnancy at unsuitable age, breastfeeding, caesarean; and inequity in access to cares among different social and economic groups, geographical areas, and migrant
Psychological	High rate of problems such as oral and dental problems; anemia; quality of life; backache; headache; fatigue; infectious diseases; respiratory problems; retinopathy associated with prematurity; abdominal colitis; bleeding; and bronchitis

Next phase of study was dedicated to examining the problems of MHP from the midwife’s perspective. In this phase, 9 themes, 16 sub-themes and 39 items were identified. The main identified themes were human resources, data management, service continuity, cultural barriers, legal and administrative barriers, care environments, medical equipment, monitoring and evaluation, and geographical/physical accessibility (Table 2).

Table 2:

Identified problems associated with MHP from the perspective of midwives

Main themes	Sub-themes	Relevant Codes
Human resources	Quality of human resources	- The limited capability of non-midwifery staff in maternity cares - Physical weakness of many midwives due to a relatively high age - Insufficient motivation of many midwives due to low salaries -Lack of human resources in this area, especially midwives - Lack of sufficient number of gynecologists in some public health centers - The large number of duties in MHP -The high work pressure on midwives, especially during peak hours - Excessive workload on midwives caused by other health personnel - The high number of unnecessary questions included in the care system - A system that is not user-friendly - Non-compliance of the system with protocols in some professional activities - The dual role defined for the midwives in the system - Entering false and unrealistic services into the system - The low speed of care system - Creation of false data and statistics due to entering unrealistic cares - No specific aim to use the produced data in the system -Inaccessibility of the produced data to the health personnel - The lack of paper documents -The lack of alternative paper-forms to be used during system errors and problems - The absence of a caring midwife at the time of delivery - Lack of coordination between private and public centers - The problem of early detection of pregnant mothers in cities - The poor coordination of hospitals with health centers in referral cases - Unnecessarily referrals of patients by physicians - Low tendency of mothers to receive services from public centers - Unrealistic expectations of mothers from midwives, especially in rural areas - Using the title “health care provider” for the midwives - Taking a quantitative approach towards maternity care - Lack of special examination rooms for the midwives -A large number of stairs in the buildings and the lack of an elevator - Poor physical condition of many public centers - Inadequate number of medical equipment in some private health centers such as weighing scales and barometers -Using defective equipment such as barometer etc. - Limited laboratory facilities for pregnancy cares - Focusing on quantitative evaluation indices and disregarding qualitative ones - The high number of indices and defined norms for evaluation -Poor verification of the provided care statistics - Poor supervision of cares, especially in private health centers - Inappropriate geographical location of some health centers
	Quantity of human resources
	A wide scope of duties
Data management	Initial design of the system
Clinical information system	Entering data into the system
	Production and distribution of the data in the system
	Paper documents
Continuity of services	Continuity and integrity of services
	Referral system
Cultural barriers
Legal and administrative barriers
Care environments
Medical equipment
Monitoring and evaluation	Evaluation indices
	Evaluation process
Geographical accessibility

In SWOT analysis phase, various internal weaknesses and major external threats were identified to the evaluation system of MHP. The notable weaknesses were related to poor evaluation processes, tools, and surveyors; and lack of required financial and physical resources. Moreover, the main threats were lack of national evaluation/accreditation organizations, lack of evaluation/accreditation experts particularly in MHP, existing deficiencies in funding of the PHC and MHP system.

In assessing existing monitoring tools, the tools have substantial deficiencies including disregarding half (50%) of the CIPP dimensions and lack of proper evaluation measures to assess these dimensions too. Other deficiencies were subjectivity of the evaluation measure, and failure to provide appropriate and categorized measures (Table 3).

Table 3:

Coverage status of the dimensions of Iranian MHP in the current evaluation tools

Evaluation dimensions Evaluation tool	Context	Input	Process (cares)			Output	Outcome	Impact
			After pregnancy	During pregnancy	Pre-pregnancy
Checklist of monitoring the “physician” in urban health centers	✓	-	-	✓	✓	-	-	-
Checklist of monitoring the “midwife” in urban health centers	-	-	✓	✓	✓	-	✓	-
Checklist of monitoring the “healthcare provider” in urban health centers	-	-	-	✓	∞	✓	✓	-
Checklist of monitoring the “rural centers”	-	-	✓	✓	✓	✓	✓	-
Checklist of monitoring the “health house”	-	✓	✓	✓	-	-	-	✓
Seasonal performance monitoring form of the “midwife of health team”	✓	✓	✓	✓	✓	-	-	-
Checklist of “monitoring center of maternity health”	-	✓	-	✓	-	-	-	-
Checklist of monitoring “Maternity facility unit”	∞	✓	∞	✓	∞	-	-	✓
Checklist of monitoring “city headquarter”	-	✓	-	✓	-	✓	-	✓
SIB clinical information system	✓	-	✓	✓	✓	✓	-	-

✓: Existence of measures related to the evaluation dimensions in the examined tools

- : Lack of measures related to the evaluation dimensions in the examined tools

∞: No need to address this evaluation dimension in the checklist due to the nature and purpose of the related care

The cross-sectional phase which aimed to gain the perspective of MHP care providers countrywide showed that among the various MHP evaluation area, the “SIB clinical information system” with a score of 2.38 and “inter and intra sector collaboration” with a score of 3.29 has the worst and best status, respectively. Moreover, according to the defined statistical cut-off points the “SIB clinical information system” dimension has poor status and other dimensions have moderate status. The total status of this evaluation system has moderate status too, with score of 2.74 (Table 4).

Table 4:

The status of the MHP Evaluation system in various dimensions

Evaluation areas	Minimum	Maximum	Mean	Standard deviation
Evaluation process	2.09	4.73	2.92	0.68
Evaluation tools	1.75	5	2.90	0.78
SIB clinical information system	1.78	4.06	2.38	0.41
Human resources	1.75	4.50	2.83	0.64
Financial and physical resources	1.57	4.29	2.81	0.67
inter and intra sector collaboration	1	4.5	3.29	0.64
Total score	2.21	4.13	2.74	0.50

In Delphi technique phase, the response rate was 90% and 94.44% in first and second rounds, respectively. From the 294 evaluation measures, 274 and 15 measures were accepted in first and second rounds respectively; and 8 measures were rejected by experts. Moreover, the average scores obtained for sum of the measures in importance and applicability criteria were 8.24 and 7.85, respectively. The final model consisted of 5 areas, 32 standards and 289 measures. The area of reproductive health (family planning) had 2 standards and 15 measures; the maternity health area had 20 standards and 207 measures; the health records area had 3 standards and 24 measures; the evaluation area had 2 standards and 10 measures; and the resource management area had 5 standards and 33 measures. The content of this model is shown in Fig. 3. This model was developed based on management cycle, CIPP model, and improvement/PDCA cycle.

Fig. 3:

The content of final evaluation model as satellite figure

Discussion

This study aimed to develop a comprehensive evaluation model for MHP. The resulting final model consisted of 5 areas. The area of reproductive health/family planning focuses on correct implementation of family planning processes and obtaining desirable outcomes. Maternity health area is dedicated to providing proper care in pre-pregnancy, pregnancy, delivery, postpartum, mother’s education regarding baby care, maternal mortality cycle, and obtaining desirable outcomes. Health records area is including creating appropriate software and hardware infrastructure for keeping health records, proper design of electronic process, proper completion of documents, accurate data analysis, achievement desirable outcomes, and proper use of health records. The evaluation area emphasizes on development of mechanisms for correct implementation of evaluation processes. The resource management area also focuses on proper execution of processes in human resource management, financial resources, physical facilities and therapeutic/non-therapeutic devices.

Assessing excellent accreditation models worldwide show that most of them have avoided addressing maternity health due to their functional nature. An interesting point is that provided accreditation models in the Eastern Mediterranean Region (EMRs) have been designed as combination of functional and departmental standards (21). Assessing the evaluation/accreditation programs in EMRs shows that only Egypt and Saudi Arabia have considered maternity care; but their evaluation standards/measures are so limited (29, 30).

This study has notable strengths including using a wide range of research methods, including existing MHP clinical guidelines in standards/measures development, using key informants as experts, high expert’s response rate, and high obtained scores to evaluation measures in the Delphi phase. The first notable study limitation is lack of similar studies to compare and discuss. In addition, using systematic review as a research approach has limitations including its inability to overcome methodological problems and biases in included studies; or controversies in interpretation and summarization of results due to diversity in types, design, and quality of the included studies. Likewise using the Delphi technique has limitations such as potential bias in selecting experts.

The researchers suggest that a pilot study be conducted to evaluate and improve this model in real fields; also recommended to develop a scoring, grading/ranking and accreditation system after conducting pilot study. As well, it is valuable to expand developing evaluation/accreditation models for all fields of Iran’s PHC system.

Conclusion

The MHP evaluation system in Iran is facing numerous shortcomings in designing, implementing and effectiveness; and it requires improvement scientifically based on stakeholders’ and experts’ perspectives. Considering using proper research methods in this study and unique comprehensiveness of obtained model, the researchers hope that implementation of this model can be helpful in performance improvement of MHP.

Ethical considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.