Evaluation of Timeliness, Simplicity, Acceptability, and Flexibility in Child Mortality Surveillance System for Children Aged 1-59 Months in Iran.

BACKGROUND: Child mortality surveillance system (CMSS) for children aged 1-59 months is a critical issue in the prevention of mortality. This surveillance system like other health programs needs to be evaluated. Therefore, this study aims to evaluate CMSS in Iran.
METHODS: This evaluation was performed from March 2015 to March 2016 based on selected criteria for assessing the public health surveillance system proposed by the Centers for Disease Control and Prevention. Selected criteria examined in this study included timeliness, simplicity, acceptability, and flexibility. These criteria were evaluated in two ways. First, it included the use of a researcher-made questionnaire. The questionnaires were completed by 100 experts on CMSS. Second, to perform a more exact evaluation of these criteria, 24 of these experts were selected for the focus group.
RESULTS: In this study, the response rate was 91% (42% hospital-based and 49% primary care-based). In the timeliness section, 49% of the experts believed that approvals of the child mortality committees have not been sent within the designated time frame; hardware, software, and questionnaires were reported as effective factors in this respect. The structural and administrative problems were effective in simplicity domain and the experts of mortality registration and mood of relatives were effective in acceptability domain. The flexibility of the system was high and appropriate.
CONCLUSIONS: The findings of the present study reveal that CMSS has some limitations and problems in the timeliness, simplicity, and acceptability criteria, which can be resolved. But this program has an appropriate situation in terms of flexibility. Copyright:

Introduction

The child mortality surveillance system (CMSS) for children aged 1–59 months in Iran started nationally in 2007 onwards across Iran's medical universities with the goal of reducing the mortality rate of children. To achieve this goal, the system uses appropriate interventions to review the history of each case of child death from the onset of the first symptom of danger until death and to identify affecting and intervening factors. This system was implemented as a hospital surveillance system and as the primary health surveillance system (titled “out of hospital”). In general, the CMSS consists of four main activities: 1) collection of children's mortality data; 2) examining the causes of death; 3) design and implementation of interventions; 4) monitoring and evaluation. Children's mortality is an indicator of the population health, which is associated with various factors such as socioeconomic status, maternal health, quality and access to health services, and public health functions.[12] According to the eight Millennium Development Goals, committed by all countries in 2000, the child mortality rate should be reduced by 75% in 2015 as compared with that in 1990.[3] Surveillance helps in detecting diseases, supporting health organizations, and interventions, estimating the impact of a disease or injury and portraying the natural history of diseases.[45]

Evaluation of a public health surveillance system gives a clear description of the purpose, design, management, and operational characteristics of the system and provides evidence-based information, which could be used for strengthening the reporting mechanism and enhancing implementation of public health action.[6] Therefore, CMSS examines the history of each child's death from the onset of the first symptom of danger to death in terms of quality, quantity, and ease of access. It also identifies effective factors in child mortality and designs appropriate interventions for it. Therefore, regular evaluation of the information provided by the surveillance system is essential.[678]

The assessment of the CMSS for children aged 1–59 months in Iran was carried out using selected indicators of the Centers for Disease Control and Prevention (CDC), which included simplicity, flexibility, acceptability, and timeliness so that it shows the current status of the CMSS to national authorities and policymakers and, if necessary, immediate interventions can be designed to improve the characteristics of the CMSS.

Methods

This study was approved by the Vice Chancellor at the Research and Ethics Committee for Research of Iran University of Medical Sciences with number code 26212/2015.

In this cross-sectional survey, definitions, goals, and applications of CMSS in Iran were first summarized, and then the evaluation of the CMSS was performed in 2014 based on selected criteria for assessing the public health surveillance system proposed by the CDC. Selected criteria examined in this study included timeliness, simplicity, acceptability, and flexibility. These criteria were evaluated in two ways. The first one included the use of a researcher-made questionnaire. The questionnaires were completed by 100 experts on CMSS for children aged 1–59 months in Iran. The experts were randomly selected from those participating in an annual conference on “Prevention of child injury and death.” The components of the questionnaire included questions about the timeliness at each stage, simplicity, flexibility, and acceptability of the surveillance program. The questionnaire was designed in in-hospital and out-of-hospital parts. Responses on a five-point Likert scale (I totally agree, agree, have no idea, disagree, completely disagree or very simple, simple, so-so, difficult, very difficult). The reliability and content validity of the questionnaire were examined for these domains. The face validity was evaluated by pediatricians, experts working in CMSS of Ministry of Health, and Medical Education of Iran and epidemiologists to find the level of difficulty, the degree of inappropriateness, and the ambiguity in the expressions or in the meaning of the words. Their views were applied as minor changes in the questionnaire. The reliability of the questionnaire was also evaluated using Cronbach's alpha method for each domain. Cronbach's alpha coefficients of 87%, 94%, 95%, 91%, and 81% were obtained for in-hospital and out-of-hospital timeliness, simplicity, flexibility, and acceptability, respectively.

The second part: To perform a more exact evaluation of these criteria, the focus group was formed using experts in the CMSS. This focus group participated in two groups of 12 people. The participants in each group were later asked questions on the qualitative features of the surveillance system, including timeliness, simplicity, flexibility, and acceptability.

The data of the questionnaires were entered into SPSS software (version 22), and the frequency distribution was calculated for each item of timeliness, simplicity, acceptability, and flexibility. The content analysis method was used to extract the theme, category, subcategories, and code of the information collected from the focus group. The views of 24 experts of CMSS were reviewed and summarized and a single text (a unit of analysis) was later formed. Then, the summarized text was separated into units of meaning. And, they were eventually compared according to their differences and similarities and classified into subcategories and categories. Initial categories were discussed by the researchers. Differences eventually led to an agreement on the categorization of subcategories, and the main meaning and theme were finally constituted within a theme by categories.

Questionnaires

Questionnaire A: It was designed to investigate the general specifications of family and health records. Questionnaire B: It was designed to review the medical outpatient services provided to a deceased patient. Questionnaire C: It was designed to investigate records of medical procedures. Questionnaire D: It was a checklist for investigating the causes of death.

Results

Timeliness

In the timeliness domain of the out-of-hospital CMSS, 49% of the experts believed that the instructions of the CMSS review committees would not be sent to the executive level at a given time and this process would take longer time. Overall, out-of-hospital experts believed that timeliness set by the CMSS for reporting is far beyond the time limit. The main problem of the in-hospital CMSS, according to experts of this system, is related to holding a monthly committee to monitor child mortality; 56% of experts believed that such a committee is not held monthly [Table 1]. Regarding the timeliness domain, results showed that most of the experts in the focus group pointed out the structural and implementation problems of the surveillance program for the expected time period. Therefore, the expected time (24 hours) for the initial registration of a child's death is very limited. According to the experts’ opinion, there are some problems that cause limitation including hardware problems of the surveillance program (e.g., old computers and lack of appropriate computers), software problems (e.g., lack of access to the Internet and lack of timely access to CSO software), budget and financial problems, high volume of questionnaires and lack of timely cooperation of the deceased's family to receive the deceased's information; also, a waiting time of 24 hours has significantly limited the reporting and recording of the death cases.

Table 1

Distribution of experts surveillance system’s response to the timeliness in out-of-hospital and in-hospital child mortality surveillance system (CMSS)

Timeliness in out-of-hospital section
Questions	Completely agree	Agree	No idea	Disagree	Completely disagree
Timely reporting of all basic mortality information in <24 h after death	7 (14.28%)	14 (28.57%)	6 (12.24%)	20 (40.82%)	2 (4.08%)
Complete child death questionnaires up to two weeks after the death	7 (14.28%)	12 (24.49%)	9 (18.3%)	18 (36.7%)	5 (6.12%)
Extraction of the proposed interventions (questionnaires, children’s health programs, files, etc.) before the death committee was formed	7 (14.58%)	6 (12.4%	13 (27.08%)	19 (39.58%)	3 (6.25%)
Child mortality review committee to hold regular meetings with the main members every 2 months.	8 (16.33%)	10 (20.41%)	19 (18.37%)	18 (36.73%)	4 (8.16%)
Reporting the results of decisions, interventions, and activities of the committee to the response levels up to a week after the committee is held	6 (12.4%)	9 (18.37%)	10 (20.41%)	20 (40.82%)	4 (8.16%)
Reporting of all the results of the county committee and detailed activities within a maximum of two months	6 (12.4%)	9 (18.37%)	9 (18.37%)	22 (44.9%)	3 (6.12%)
Extraction and analysis of information related to child mortality indicators at the end of each year and use the results	8 (16.33%)	10 (20.41%)	8 (16.33%)	17 (42.86%)	2 (4.08%)
Reports and feedback from the Ministry of Health are received promptly at the appointed time by the experts	5 (10.2%)	10 (20.41%)	13 (26.53%)	18 (36.73%)	3 (6.12%)
Timeliness in out-of-hospital section
Timely reporting of all basic mortality information in <24 h after death	7 (16.28%)	11 (25.58%)	6 (13.95%)	16 (37.21%)	3 (6.98%)
Complete child death questionnaires up to a week after the death	8 (18.65%)	17 (3953%)	3 (6.98%)	11 (25.58%)	4 (9.3%)
Hospital child mortality review committee to hold regular meeting with the main members every month	4 (9.3%)	10 (23.26%)	5 (11.63%)	18 (41.86%)	6 (13.95%)
Reporting the results of decisions, interventions, and activities of the committee to the response levels up to a week after the committee is held	5 (11.63%)	13 (30.23%)	5 (11.63%)	16 (37.21%)	4 (9.3%)
Reporting of all the results of the committee and detailed activities within a maximum a month	7 (16.28%)	16 (37.21%)	2 (4.65%)	15 (34.88%)	3 (6.98%)
Reporting of completed death information to responsible persons for the death of children at the appointed time and designated	9 (20.93%)	16 (37.21%)	3 (6.98%)	12 (27.91%)	3 (6.98%)
Reports and feedback from the Ministry of Health are received promptly at the appointed time by the experts	8 (18.6%)	16 (37.21%)	7 (16.28%)	9 (20.93%)	3 (6.98%)

Simplicity

Different items of simplicity domain of the CMSS were evaluated and the results showed that 54.58% of the experts have referred to the incomprehensibility of “E” questionnaire directions in the surveillance system as the main problem. Also, 48.35% of surveillance system experts referred to the difficulty in distinguishing between the main cause and the secondary cause of death as one of the major problems in this domain of the CMSS, on the basis of a questionnaire survey [Table 2]. In addition, the experts of the focus group of CMSS indicated that the surveillance system experts pointed to structural problems of the program and educational problems as the main problems of the system. Also, the problems with coding diseases in the International Classification of Diseases, 10th revision (ICD10), the existence of “other” option for the disease classification in the CSO software, the inability to receiving CSO reports, and the complexity of the CSO software were reported as the main reasons for the complexity of the surveillance system.

Table 2

Distribution of experts surveillance system’s response to the simplicity of CMSS

Simplicity	Very simple	Simple	Moderate	Difficult	Very difficult
Simple completion of the monthly reporting form of child mortality	10 (10.99%)	21 (23.08%)	33 (36.26%)	22 (24.18%)	5 (5.49%)
Simplicity of distinguishing the underlying cause and the main cause of child mortality in the form of reporting	8 (8.79%)	17 (18.68%)	22 (24.18%)	33 (36.26%)	11 (12.6%)
Simple completion of Questionnaire B for deceased	1 (1.1%)	30 (33%)	30 (33%)	22 (24.2%)	8 (8.8%)
Simple and understandable instructions for completing the questionnaire “B” for employees’ care system	3 (3.3%)	19 (21.1%)	29 (32.2%)	29 (32.2%)	10 (11.1%)
Simplicity of completing questionnaire “D” for deceased children	8 (8.79%)	15 (16.48%)	27 (29.67%)	29 (31.78%)	12 (13.19%)
Simple and understandable instructions for completing questionnaire “D” for employees care system	9 (9.89%)	18 (19.78%)	36 (39.56%)	19 (20.88%)	9 (9.89%)
Staff awareness of child mortality surveillance system for grouping ICD10	8 (8.89%)	18 (19.78%)	32 (35.16%)	25 (27.47%)	8 (8.79%)
Experts have sufficient knowledge of the medical terminology used in the questionnaire mortality surveillance system	5 (5.49%)	19 (20.88%)	31 (34.07%)	26 (28.57%)	10 (10.99%)
Simplicity in completing the questionnaire “E” for deceased children	6 (6.59%)	22 (24.18%)	24 (26.37%)	33 (36.26%)	6 (6.59%)
Simple and understandable instructions for completing the questionnaire “E” for employees care system	5 (5.49%)	14 (15.38%)	22 (24.18%)	40 (43.96%)	10 (10.99%)
Simple completion of the questionnaire “C” for deceased children	6 (6.59%)	15 (16.48%)	35 (38.46%)	27 (29.67%)	7 (7.69%)
Simple and understandable instructions for completing the questionnaire “C” for employees care system	4 (4.4%)	24 (26.37%)	29 (31.78%)	27 (29.67%)	14 (14.9%)
Easy access to Internet portals child mortality surveillance system for the expert care system	5 (5.49%)	12 (13.19%)	35 (38.46%)	27 (29.67%)	14 (14.9%)
Simple as entering data in CSO software	4 (4.4%)	15 (16.89%)	25 (28.09%)	37 (41.57%)	8 (8.99%)
The simplicity of extracting information required for children’s mortality CSO software	5 (5.56%)	17 (17.78%)	27 (30%)	31 (34.44%)	11 (12.2%)

Flexibility

With regard to the flexibility domain of the CMSS, the experts were asked some questions about a new list of diseases to the CMSS in such a way that this question showed the flexibility of the CMSS program in terms of cost and time. And according to this question, 52.75% of the experts believed that adding a new list of diseases to the ICD10 grouping or adding new features from the deceased child to the CMSS is affordable in terms of time. Only 17% of the experts opposed the change in the CMSS in terms of time and believed that it would be difficult to make these changes in the CMSS due to time constraints. Also, the results of investigation about making changes in the CMSS from the point of view of workload and the associated problems showed that 48.35% of the CMSS experts believed that it was very easy to make changes in this system.

Also, 46.15% of the CMSS experts believed that adding a new list of diseases to the ICD10 grouping in CMSS is affordable in terms of time [Table 3]. Some discussion was done in the focus group section on the flexibility of the CMSS in terms of the characteristics of time, workload, and cost in various parts of the system. Most of the childcare system experts believed that making changes such as disease encoding, improving CSO software; reporting formats; and the methods of diagnosing the cause of death by doctors, and adding new items to the program would be affordable for the surveillance system program in terms of time, workload, and cost.

Table 3

Distribution of experts surveillance system’s response to the flexibility of CMSS

Flexibility	Completely agree	Agree	No idea	Disagree	Completely disagree
Affordable in terms of “cost” to add a new disease to the surveillance system	11 (12.09%)	31 (34.07%)	29 (31.78%)	16 (17.58%)	4 (4.4%)
Affordable in terms of the “difficulty” of adding new disease list and the new and additional features of the deceased to the surveillance system	15 (16.48%)	29 (31.86%)	26 (28.57%)	17 (18.68%)	4 (4.4%)
Affordable in terms of “time” to add a new disease list to the surveillance system	19 (20.88%)	29 (31.87%)	27 (29.57%)	13 (14.19%)	3 (3.3%)

Acceptability

Regarding the acceptability domain of the CMSS, the results showed that 44% of experts believed that the surveillance system questionnaire was filled by patients’ experts and 44.5% of the experts believed that the deceased child's families did not have the necessary cooperation to complete the questionnaire information [Table 4]. In the focus group, it was referred to as relatives’ unwillingness to cooperate in completing the information of the deceased's questionnaire.

Table 4

Distribution of experts surveillance system’s response to the acceptability of CMSS

Acceptability	Completely agree	Agree	No idea	Disagree	Completely disagree
The cooperation of the deceased children’s family in completing the questionnaire information	4 (4.4%)	21 (23.08%)	25 (27.47%)	32 (35.2%)	9 (9.89%)
Doing the programs and interventions recommended by parents of children	1 (1.1%)	23 (25.3%)	23 (25.27%)	34 (37.3%)	10 (10.99%)
Staff collaboration and the use of oral statements by staffs to complete child information	3 (3.3%)	39 (42.86%)	17 (18.68%)	25 (27.47%)	6 (6.59%)
The patience of the death register experts to complete all the questionnaires in the surveillance system	9 (9.98%)	32 (34.07%)	20 (21.09%)	25 (27.57%)	6 (6.59%)

Discussion

In order to improve the timeliness status in surveillance systems, a report should be made of the public health status, the data collection procedure, the reporting monitoring process, and the time intervals for monitoring the surveillance system.[9] In addition, the information system of the surveillance system should pay attention to the timeliness data, since timeliness is one of the most important determinants of decision-making and policy-making in the health system.[101112] Timeliness in the CMSS was influenced by many factors such as the overall structure and appearance of the program, the time required to submit mortality reports, and the timeliness of feedback and reports. In the meantime, most experts believed that the timeliness (24 hours) for immediate death reporting was very low due to limited possibilities and existing problems. Most experts believed that limited hardware facilities such as computer and software systems were the main problems in delaying the immediate reporting of death. Also, most experts believed that the data obtained from questionnaires B, C, and D could not be completed within one week due to their size. In-hospital and out-of-hospital committees are not held at regular intervals because the committee's main members do not attend sessions, so if in-hospital and out-of-hospital committees are not held, there would be no useful outcome for the surveillance system. Yoo et al., in 2009,[13] evaluated the timeliness of nationally notifiable diseases surveillance system in KORE based on their study results. The proportion of cases reported in time was lower for disease groups with a recommended time limit of 1 day compared with 7 days. Time from disease onset to diagnosis generally contributed most to the delay in reporting,[14] which is consistent with our study results. While in He et al.'s study, which was conducted to assess the Australian Pediatric Surveillance Unit (APSU) from 2000 to 2007 on the basis of the CDC guideline, an average 96% of monthly report cards were returned per annum since 2000. Contrary to the results of this study, the timeliness of APSU was timely and acceptable. Feedback is sent to the users of the APSU in a timely and regular basis. This can be explained by differences in hardware, data quality, and feedback in both systems.[8]

Our study results are consistent with other previous studies on infectious diseases.[131516] Timeliness has been reduced for most diseases in recent years.[17] Timeliness in the CMSS has been improved in terms of the immediate reporting of infectious diseases as compared with the previous studies.[14181920] Another study showed that the timeliness of the diseases is low based on the national reporting of the disease surveillance system.[21] The results showed that 28% of doctors reported the delay in completing surveillance system questionnaires, and the main reason for this delay was attributed to the volume of questionnaires, which is consistent with the results of Toprani et al.'s study.[17] The results of these studies are consistent with those of the present study regarding the surveillance system.[22] Overall, based on the results of this study, there is limited timeliness for immediate reporting, holding children's death committees on time, and completion of questionnaires and the intended measures are not carried out in the specified time duration.

Proposed practical solution for the timeliness domain of the CMSS: Based on the results of this study, the deadline for the initial report of death and reporting was very limited due to the lack of feedback from the Ministry of Health to beneficiary universities and the failure to report child deaths in the form of a general report; it is suggested that the existing timeframe is increased and child death reports are available to stakeholders and the general public on an annual basis.

Usually, people involved in the CMSS in Iran have not undergone any special training for this purpose and doing the related activities while providing other health services. As a result, the surveillance system should be simple and capable of achieving its goals at the same time.

According to the results of this study, the majority of experts believed that the overall structure of monthly reporting is complex and difficult. According to the reports and results obtained from this study, the experts of this program believed that the program was highly complex due to structural and executive reasons and factors such as inability to store information in CSO software, complexity of ICD10 classification, contradictory questions in questionnaires, lack of reporting capabilities, novice experts, and complexity of the CSO software were mentioned as the most important problems related to the simplicity of the surveillance system. This result is consistent with that of the current study.[23] Toprani et al., in 2014, evaluated the simplicity of the abortion reporting system in New York City. According to the results of other studies, 95% of the experts believed that reporting was very constructive and useful, while 52% of the experts referred to the complexity of the reporting questionnaires and believed that completing the questionnaires requires training and is a complex process and also stated that the certificate of death and the mentioned causes of death are vague,[17] which is consistent with the result of the present study. Unlike the results of this study, the study by He et al.[8] showed that the Australian childcare system was very simplistic. This can be explained by differences in hardware, the presence of specialized and trained experts in both the systems.

Proposed practical solution for the simplicity domain of the CMSS: reducing the volume and questionnaire questions, reviewing and revising CMSS questionnaires based on the expert opinion of this system, training of care system experts, organizing virtual training courses for experts in the care system, and assessing the extent to which experts are mastered using virtual test to confirm the certificate for care system experts.

If changes occur in the CMSS, it will be possible to implement flexibility at a low cost and effort so that, for instance, it would be possible to add a disease or new individuals’ profiles to the previous surveillance system in less time and with little human resources. In the current surveillance system, adding a new list of diseases or new profiles of the deceased child or making changes in the surveillance system implementation procedure should be affordable in terms of time, cost, and practicality of the program. The results of our study are consistent with the results of the study conducted by He et al., which reported that the flexibility of the APSU is high and acceptable.[8] In this study, time, cost, and operationalization of these processes were examined; most experts believed that making changes in the surveillance system was affordable in terms of time. They also believed that these changes could be applied in the surveillance system within a short period of time. A total of 49% of the experts considered these changes to be practical and 37% of them considered the changes to be positive in terms of cost. These results are consistent with the results of the previous study.[17]

The proposed practical solution for the flexibility domain of the CMSS: Make changes to the CMSS if needed.

The acceptability domain refers to the willingness of the deceased child's families to report and also to participate in the CMSS, which will guarantee the success of the surveillance system. Most of the experts believed that the deceased children's families were not able to respond due to their child's death and, in some cases, the deceased's relatives did not cooperate due to the possibility of legal conflict and fear of conviction. Therefore, this program does not enjoy good acceptability among the deceased family, and the experts believed that parents’ cooperation status would be improved if parents were given more time to complete the questionnaires and experts were provided with training courses and good counseling principles. Therefore, they will not have a positive effect on reducing the mortality of children. Other studies on the acceptability of the surveillance system such as the infectious[23] disease surveillance system[2425] and the vaccination surveillance program[22] showed that this program enjoys high acceptability compared to the CMSS in Iran. This difference may be justified due to the difference in the subject matter.

The proposed practical solution for the acceptability domain of the CMSS: Publication of timely national, regional, and provincial reports (feedback) by the certified medical management safety specialist (CMMSS) and the Ministry of Health experts in the CMSS and publication of annual child deaths by the Ministry of Health for public access.

Study strengths and weaknesses: Our study had a strong point to be noted, the evaluation of the CMSS was done according to the CDC guidelines. To increase the accuracy of the results, CMSS experts were randomly selected from all centers under the care system. Another strength of this study is that data collection was done by questionnaires and holding focus groups. The only major limitation of this study was the subjectivity of the data.

Conclusions

The findings of this study reveal that CMSS has some limitations and problems in the timeliness, simplicity, and acceptability criteria, which can be resolved according to the executive experts in Iran's medical sciences universities. But this program has an appropriate situation in terms of flexibility.

Financial support and sponsorship

The results described in this paper were part of student thesis. The present study was supported by a grant from Iran University of Medical Sciences.

Conflicts of interest

There are no conflicts of interest.