Evaluation of Two Malaria Surveillance Systems in Yemen Using Updated CDC Guidelines: Lessons Learned and Future Perspectives.

Yemen is classified as high malaria endemic area with two-thirds of population at risk. Currently, the National Malaria Control Program (NMCP) uses two malaria surveillance systems: the Integrated Malaria Surveillance System (IMSS) and the Early Disease Electronic Warning System (eDEWS). This study aimed to assess and compare the usefulness and attributes of the two malaria surveillance systems. The systems were evaluated according to the US Centers for Disease Control and Prevention (CDC) updated guidelines. Data were collected from 10 stakeholders through interviews and from 10 districts' coordinators and 20 health facilities' focal points using semistructured questionnaire. The score of the system attributes were interpreted as very poor, poor, average, good, and excellent according to the mean percent score. Both systems were found to be useful. The IMSS overall performance score was poor where flexibility was average and simplicity, acceptability, representativeness, and stability were poor. For eDEWS, the overall performance score was good where data quality, acceptability, and flexibility were excellent; simplicity was good; representativeness was average; and stability was poor. In conclusion, although the IMSS was found to be useful for assessing the burden of malaria, response to outbreak, and future planning, the overall performance was poor. While the eDEWS overall level of performance was good, it was found to be useful only for outbreak detection. Therefore, both surveillance systems need to be integrated for the advantages of both systems to be maintained.

What do we already know about this topic?

While the Integrated Malaria Surveillance System (IMSS) and the Early Disease Electronic Warning System (eDEWS) are functioning in Yemen, their performance is not known.

How does your research contribute to the field?

Evaluation of the ongoing malaria surveillance systems helps to ensure that they provide timely, reliable, complete, and up-to-date information on malaria to guide case management, control, and preventive measures.

What are your research’s implications toward theory, practice, or policy?

The two malaria surveillance systems in Yemen need to be integrated in one surveillance system where the advantages of both systems can be utilized.

Introduction

Malaria is a life-threatening disease with a high burden in terms of morbidity and mortality especially in regions with limited resources.[1] It is commonly caused by Plasmodium falciparum parasites, which are transmitted through a bite from an infected female Anopheles mosquito.[2] In 2016, 216 million people were infected with malaria and almost half a million died mostly due P falciparum. Since 2000, the incidence was declined by 37% and the number of deaths was declined by 60% in many regions of the world as a result of multiple interventions including vector control, preventive chemotherapy, and improved diagnosis and treatment.[3]

Despite reduced malaria mortality in children, malaria accounted for 5% of below 5 deaths in 2015.[4] The Eastern Mediterranean Region (EMR) has been ranked at the third position among World Health Organization (WHO) regions in term of malaria burden.[2] In 2014, about 276 million people in 8 countries in the region were at some risk of malaria with 108 million at high risk.[2] Six countries have areas of high malaria transmission (Afghanistan, Djibouti, Pakistan, Somalia, Sudan, and Yemen). Most cases are due to P falciparum, except in Afghanistan, Iran, and Pakistan, where Plasmodium vivax predominates.[2]

According to the WHO, Yemen is classified as a high malaria endemic zone. Malaria remains to be one of the top 10th priorities causing a high morbidity and mortality in Yemen.[2] About 68% of the population is at high risk to develop the disease. Approximately 163,000 infections and 72 deaths were estimated in 2012.[5] The overall incidence rate was 5/1000 a year and two-thirds of malaria cases were reported from coastal governorates such as Al-Hodeida, Hajjah, Sadah, and Mahwit governorates.[5,6] In 2009, the Integrated Malaria Surveillance System (IMSS) was established. It was formed from the integration of 3 malaria surveillance systems: The National Malaria Control Program (NMCP), Directorate of Health Information System (DHIS), and Directorate of National Central Disease and surveillance (NCDS). The main objectives of the IMSS are to provide timely, reliable, complete, and up-to-date information on malaria to guide case management, control and preventive measures, and timely detection and response to malaria epidemics within the first two weeks of its occurrence.[5] Another surveillance system is the electronic Disease Early Warning System (eDEWS) which, in addition to its other activities, is responsible for malaria surveillance in Yemen. The eDEWS was launched in March 2013 with an objective to detect and respond rapidly to alerts that could indicate outbreaks and clusters of epidemic-prone diseases.[7]

Objective of the Evaluation

This study aimed to assess and compare the usefulness and attributes of the two malaria surveillance systems, through that we will identify strengths and weakness of both systems to provide recommendations for improvement.

Methods

Evaluation Design and Approach

The 6 steps of CDC updated guidelines on evaluation of a public health surveillance system were followed.[8] The public health importance of IMSS and eDEWS was described by measuring morbidity and mortality as well as their purpose and operation. Quantitative and qualitative assessment of usefulness and attributes of both systems was conducted. Five qualitative and one quantitative system attributes were assessed with set of questions under each attribute. The qualitative attributes included simplicity, flexibility, acceptability, representativeness, and stability. The quantitative attribute was timelines. Each attribute was scored accordingly as excellent, good, average, poor, and very poor.

Sample

The systems’ stakeholders were engaged in this study including NMCP and eDEWS managers and staff, Ministry of Public Health and Population (MOPH) staff, WHO staff, and data providers at the 4 levels: central, governorate, district, and health facilities levels.

Data Collection

Desk review was conducted for the main IMSS and eDEWS documents such as strategic plan, guidelines, annual reports, surveys, and pertinent literatures. In-depth interviews were conducted with 12 relevant stakeholders of both systems at central and governorate levels to determine the level of usefulness of each system. At district and health facility levels, a semistructured questionnaire was used to collect data from coordinators and focal points using face-to-face interview. The questionnaire consisted of items assessing the performance attributes according to the activity of malaria surveillance at the different levels.

➢ To determine the level of usefulness, the system was considered useful if it has addressed at least one of its objectives and/ or one of its planned uses. The level of simplicity, acceptability, and so on of the system were assessed by using Likert scale and level of agreement with each positive indicator on a 5-score scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree).

➢ For each indicator, the score percent was calculated as the following:

Analysis

➢ The overall attribute score percent was calculated as the following:

Every indicator and overall attribute score percent were expressed as rank as following: Excellent (attribute score >90%), Good (attribute score 80%-90%), Average (attribute score 60%-80%), Poor (attribute score 40%-60%), and very poor (attribute score <40%). Timeliness was calculated by dividing the number of received reports by total reports expected. The overall scoring of each system was calculated by averaging all percent scores of the attributes. Data were analyzed using Epi Info version 7.2.1.

Results

Description of IMSS and eDEWS

Table 1 describes the main components of IMSS and eDEWS and the purpose and operation of each system. Figure 1 describes the IMSS dataflow and feedback mechanism. Figure 2 describes the eDEWS dataflow and feedback mechanism.

Table 1.

Main Components of the National Malaria Surveillance System IMSS and eDEWS.

Components	IMSS	eDEWS
System description	IMSS is an integral part of the NMCP malaria surveillance that provide reliable, complete, timely, up-to-date morbidity & mortality of malaria data to be used as an evidence-based tool to guide program intervention measures, detection and response to malaria epidemic within the first 2 weeks.	eDEWS is an electronic system for data collection, compilation, and analysis from the health facilities to detect outbreaks at an early stage and take necessary response measures to prevent or limit its occurrence.
Objectives of the system	The main purpose and objectives of the IMSS are to provide timely, reliable, complete, and up-to- date data to be used by the NMCP as an evidence-based tool to guide routine malaria case management, control and preventive measures, and to a guide the timely detection and response to malaria epidemics within the first 2 weeks of its occurrence.	The only and main objective is to detect and respond rapidly to alerts that could indicate outbreaks and clusters of epidemic-prone diseases.
Population under surveillance	81 out of 334 public and private health facilities are covered (cover 24% of the population in Sana’a City).	Covers 150 public and private health facilities.
Time period of data collection	Data are collected and reported on monthly basis.	Data are collected and reported on immediate alert within 24 hours after detection and data are aggregated by health facilities on weekly basis.
Type of the system	IMSS is a health facility–based passive morbidity and mortality surveillance.	eDEWS is a health facility–based passive morbidity and mortality surveillance.
Type of data collected	Individual malaria cases and deaths recorded on line-listing.	Data mortality and morbidity of 31 diseases recorded in the health facility registration book.
Reporting source of data	Malaria registration book at health facility.The register contains the following variables:ID, age, sex (Female). For women if pregnant or not, residence, diagnosis: simple uncomplicated Malaria case (either probable or confirmed), severe complicated Malaria case (either probable or confirmed).Investigations: RDT (positive or negative), microscopic (positive or negative), type of Parasites species: (Plasmodium falciparum, Plasmodium ovale, Plasmodium vivax, and knowlesi), antimalarial drugs used and outcome	eDEWS reporting form present at each health facility. The form consist the following variable: Disease (28 notifiable disease): No. of cases (Male/Female), Cases <5 years, cases >5 years, No of death (in both <5 years, and >5, No of Slides (Positive and Negative), Type of Plasmodium: Plasmodium falciparum, Plasmodium ovale, Plasmodium vivax, and Plasmodium .Mensoni, Investigations: RDT (positive or negative), microscopic (positive or negative)
Data management	Data entry and edits are conducted at the governorate level. Data are stored in MS excel spreadsheet at the national level. Data transfer is done manually from health facility to the governorate level, and electronically thereafter. There is no standard data error check mechanism.	A focal point in the health facility has a username and password for sending eDEWS weekly reports. Weekly reports are sent through mobiles or Internet.
Data analysis	Limited simple descriptive analysis by time, place, and person at the governmental level. Complete descriptive analysis of malaria surveillance indicators at the national level on biannual and when needed by the sponsor organization.	Automatically data analysis during using eDEWS program at presence of alert by time, place, and person at the central level.
Information dissemination	Information is disseminated through summary report on annual basis and when needed by the sponsor organization and stakeholders.	At the national level, information is disseminated through weekly bulletin to all the sponsor organization and stakeholders.
Patient privacy & data confidentiality	Patient names and other identifiers are recorded only in the health facility case-based line listing and shared only with the district level. Aggregated counts of malaria cases without names of patients are shared with the other concerned people beyond the district.	Aggregated counts of cases without names of patients recorded in the health facility registration book only.
Data link to action	The data are linked to actions related to routine control and preventive measures; but not linked to address the timely detection of malaria epidemics.	The data are linked to detect and respond rapidly to alerts that could indicate outbreaks and clusters of epidemic-prone diseases so that control measures can be implemented as soon as possible to prevent further cases and death.

Note. IMSS = Integrated Malaria Surveillance System; eDEWS = Early Disease Electronic Warning System; NMCP = National Malaria Control Program; RDT = rapid diagnostic test.

Figure 1.

Data flow of Integrated Malaria Surveillance System.

Figure 2.

Data flow of Early Disease Electronic Warning System.

Characteristics of the Respondents

Twelve interviews (6 for each system) were conducted with relevant stakeholders at central and governorate levels. Only 2 (17%) of them were females. Thirty-one respondents (18 nurses, 9 lab technicians, and 4 general practitioners) at the districts and health facility levels filled the semistructured questionnaire of each system. Half of the respondents were females.

Performance of the Malaria Surveillance Systems

Usefulness

All eDEWS stakeholders at central and governorate levels stated that the system objective of identifying malaria/disease outbreak was fulfilled. For the IMSS, all the stakeholders stated that the IMSS has achieved its objective of using data to monitor the trend. Furthermore, 83% and 67% of stakeholders stated that the IMSS has achieved the objective of using data to identify high-risk groups/areas and to estimate the magnitude, incidence, and mortality rates. Therefore, both systems are rated useful.

Simplicity

Four (80%) of both systems’ managers stated the systems have taken less than 1 year to integrate with other systems. Out of 31 health facility workers and district coordinators of IMSS, 29 (94%) stated that there is no standard written malaria case definition. For the eDEWS, 29 (94%) stated that there is a standard written malaria case definition and 22 (71%) stated that the case definition is very easy to be applied. While 21 (68%) of IMSS participants stated that the data collection forms were not clear and not easy to be applied, 27 (87%) of eDEWS participants stated that the data collection form is clear and easy to be filled. For IMSS, 24 (77%) reported that they did not receive any feedback from the higher levels. However, 25 (81%) of eDEWS participants reported that they did. While 26 (71%) of eDEWS participants stated that they had received training on malaria surveillance, 21 (68%) of IMSS participants reported that they did not receive training.

Based on the scoring system, the overall mean score of simplicity was 48% for IMSS and 85% for eDEWS indicating a poor level of simplicity for the IMSS and a good level of simplicity for eDEWS.

Flexibility

Although all IMSS stakeholders agreed that IMSS could adapt easily the change in malaria case definition and include other disease (eg, Dengue), they reported that such change needs cost and efforts (eg, training staff on such change). On the contrary, all eDEWS managers and governorate coordinators agreed that eDEWS could adapt very easily to the change in malaria case definition and can include other diseases. In addition, they stated that the staff can accommodate the data change with minimum cost and efforts.

The overall mean flexibility score was 71% (ie, average) for the IMSS and 100% (ie, excellent) for eDEWS.

Stability

Of the total 6 participants of governorate and IMSS managers, 5 (83%) stated that the number of unscheduled outages of the computers had occurred more than 3 times and that the electricity was powered off during the preceding month. All participants stated that the system could collapse if the fund withdrawn. For eDEWS, all participants stated that the number of unscheduled outages of the computers had not occurred at all, but 3 (50%) persons stated that the electricity was powered off occasionally during the preceding month. All participants agreed also that the system could be collapsed if the fund is cut. Based on the scoring system, the overall stability of the IMSS was poor (46%) while it was average for eDEWS (67%).

Acceptability

Questions related to acceptability were only asked for district coordinators and health care workers. About 58% said that they are willing to participate in the IMSS system and to report up to the next level in a timely manner. However, the sites monthly completeness-reporting rate for the preceding month was zero. For the eDEWS, the majority of respondents (95%) said that they are willing to participate in the system and to report up to the next level in a timely manner. The sites monthly completeness-reporting rate for December 2016 was 100%. The mean acceptability score of the IMSS was 34% (poor) compared with 97% (excellent) for the eDEWS.

Representativeness

Reviewing the IMSS documents revealed that 75% of the public health facilities and only 29% of private facilities are covered by the system Table 2. However, eDEWS program covers 97% of the public health facilities and 46% of the private facilities.

Table 2.

Number of Health Facility Covered by IMSS and eDEWS, Yemen, 2016.

Type of health facility	IMSS			eDEWS
	Total	Covered by IMSS system	%	Total	Covered by eDEWS system	%
Public	64	48	75	68	66	96
Private	368	105	29	184	84	46

Note. IMSS = Integrated Malaria Surveillance System; eDEWS = Early Disease Electronic Warning System.

The mean score of representativeness for IMSS was 52% (poor) compared with 72% (average) for eDEWS

Data quality

Timeliness reflects the speed between steps in a public health surveillance system.

For eDEWS, timeliness and completeness by default are 100% that indicates excellent performance level for the eDEWS. We could not calculate the timeliness and completeness for the IMSS as the reporting system had collapses since 2014.

Overall scoring for both systems

Overall, the mean percent score for all system attributes was 42% for IMSS system and 87% for eDEWS system, indicating that the IMSS has poor performance level and the eDEWS system has good performance (Table 3).

Table 3.

Comparison of Attributes of IMSS and eDEWS in Yemen for Monitoring Malaria.

Attributes	IMSS		eDEWS
	Attribute score and interpretation	Justification	Attribute score and interpretation	justification
Usefulness	Useful	IMSS data are useful for planning and monitoring, control & prevention tasks but are less useful for detecting malaria outbreaks	Useful	The system is useful to detect a disease epidemic
Simplicity	Poor48%	● There is no standard written malaria case definition● The data collection forms are not clear and not easy to be applied● Don’t receive any feedback from the higher levels● Don’t receive training on malaria surveillance subjects	Good85%	● There is standard written malaria case definition● The data collection forms are clear and easy to be applied● Receive feedback from the higher levels weekly● Receive training on disease surveillance subjects and data entry
Flexibility	Average71%	The staff could not accommodate data change with minimum cost and efforts	Excellent100%	The staff can accommodate data change with minimum cost and efforts
Acceptability	Poor34%	The sites monthly completeness-reporting rate for December 2016 was 0%.	Excellent97%	The sites monthly completeness-reporting rate for December 2016 was 100%.
Representativeness	Poor52%	75% of the public and 29% of private health facilities are covered by the system	Average72%	● 97% of the public and 46% of the private health facilities are covered
Stability	Poor46%	● Unscheduled outages of the computers had occurred more than 3 times during November 2016.● Electricity was powered off occasionally during the last week in the same month.● All participants stated that the system could be collapsed if the fund is disappears.	Average67%	● All participants stated that the number of unscheduled outages of the computers had not occurred at all during November 2016.● The electricity was powered off occasionally during the last week in the same month● All participants agreed that the system could be collapsed if the fund is disappears.
Data quality	Not applicable		Excellent100%	Timeliness
Total scoring	42%		87%

Note. IMSS = Integrated Malaria Surveillance System; eDEWS = Early Disease Electronic Warning System.

Discussion

Evaluation of any surveillance system is the corner stone for its improvement and ensuring proper morbidity and mortality monitoring. It is crucial to identify weakness and strengths of the surveillance system and provide decision makers with evidence-based data to decide on its future and continuity.

We found that the IMSS is useful for malaria control and prevention activities (eg, monitoring trend). This finding is consistent with the findings of a study in Nigeria.[9] However, IMSS was found to be not useful to address the early warning program objective (ie, outbreak detection). This leads malaria epidemics to be detected only on an ad hoc basis and from channels other than the surveillance system. This finding is in agreement with the findings of a previous evaluation in Yemen in 2014.[5,10] On the contrary, eDEWS was found to be useful as it met its objective, which is early outbreaks detection and response. This finding is support by the findings of a previous evaluation in Yemen and Pakistan.[11,12] However, eDEWS needs to update its objective to meet other NMCP objectives.

Based on the findings of this study, areas of weakness were identified in the IMSS system as indicated by some of the system attributes. Regarding simplicity, absence of standard written case definition and multiple forms for data collection makes the IMSS system more complex both in its structure and in operation. This finding disagrees with the findings in previous studies in Nigeria and Bhutan.[9,13] This also was constrained more by lack of training. Such lack of training also could negatively impact other attributes, eg, acceptability and data quality. On the contrary, eDEWS was found to have a good simplicity, as there is clear case definition, simple form and regular staff trainings. A similar finding was reached by a previous study in Yemen and Pakistan.[11,12] However, eDEWS collect data only on limited malaria variables that may not respond to all NMCP needs.

Furthermore, the acceptability attribute in IMSS was poor as reflected by collapse of the reporting system, due to the discontinuation of the staff monthly incentives due to suspending of Global Fund budget. This finding is in agreement with the finding of a previous evaluation in Nigeria.[14] Lack of irregular feedback could be also a contributing factor. In comparison, eDEWS has an excellent acceptability as reflected by regular feedback, refreshing training, and most importantly paying reporting cost through mobile bills remuneration. This finding is in agreement with the finding of a previous study in Yemen[11] but it disagrees with that in a study in French Guiana.[15]

The average flexibility attribute may be due to inability of IMSS to adapt new database that will need software upgrading which needs both human and financial resources that constrained by sole dependence on donor funding.[9,14] In contrast, the eDEWS system is flexible and able to adapt and accommodate changes when there is need for additional information. This finding is supported by another study.[15]

The poor representativeness for IMSS is due to the low coverage rate where only 75% of public and 29% private health sectors are covered in Sana’a City. This finding has been reported by previous studies.[12,14] Completeness of data could not be evaluated due to the absence of reports needed for counting data missing due to collapse of the reporting system since 2014 that was parallel with suspending funds. In contrast, eDEWS has expanded its coverage and is now collecting data from 150 facilities, which accounts for 96% of health facilities in Sana’a city. The eDEWS system does not accept any missing data and therefore the completeness is 100%.

The very low stability of the IMSS system resulted from its failure in 2013-2014, where the system reliability (data collection, reporting, analysis and dissemination) and availability had hugely affected. This could be explained by the solely reliance of the system on outside funds. Although the stability of eDEWS was found to be average, its long-term sustainability could be negatively affected if donor support is suspended as it happened with IMSS.[11]

The eDEWS was found to be excellent regarding timeliness because it provides timely information through daily recording and weekly reporting and feedback. This finding is supported by previous studies in Pakistan.[12] However, timeliness could not be calculated for IMSS as the reporting system has collapsed since 2014.

In conclusion, although the IMSS was found to be useful, the level of performance was poor with weak attributes that need improvements. On the contrary, the overall eDEWS performance was good as the system had met its objectives (ie, outbreak detection), data quality was 100% by default, and simplicity was fostered by availability of written case definition and regular staff training. Therefore, both surveillance systems need to be integrated in one surveillance system where the advantages of the eDEWS can be utilized, eg, simplicity, data quality, representatives, and outbreak detection, while the IMSS data usage to address the burden of malaria in the country, response to outbreak, and for future planning should be maintained. Furthermore, innovative methods to learn from previous experience of both systems to ensure the sustainability of the integrated new surveillance system need to be considered.