Methodological Considerations for Use of Routine Health Information System Data to Evaluate Malaria Program Impact in an Era of Declining Malaria Transmission.

Coverage of malaria control interventions is increasing dramatically across endemic countries. Evaluating the impact of malaria control programs and specific interventions on health indicators is essential to enable countries to select the most effective and appropriate combination of tools to accelerate progress or proceed toward malaria elimination. When key malaria interventions have been proven effective under controlled settings, further evaluations of the impact of the intervention using randomized approaches may not be appropriate or ethical. Alternatives to randomized controlled trials are therefore required for rigorous evaluation under conditions of routine program delivery. Routine health management information system (HMIS) data are a potentially rich source of data for impact evaluation, but have been underused in impact evaluation due to concerns over internal validity, completeness, and potential bias in estimates of program or intervention impact. A range of methodologies were identified that have been used for impact evaluations with malaria outcome indicators generated from HMIS data. Methods used to maximize internal validity of HMIS data are presented, together with recommendations on reducing bias in impact estimates. Interrupted time series and dose-response analyses are proposed as the strongest quasi-experimental impact evaluation designs for analysis of malaria outcome indicators from routine HMIS data. Interrupted time series analysis compares the outcome trend and level before and after the introduction of an intervention, set of interventions or program. The dose-response national platform approach explores associations between intervention coverage or program intensity and the outcome at a subnational (district or health facility catchment) level.

BACKGROUND

With a renewed interest in achieving malaria elimination and funding available from a variety of sources for malaria control, many malaria endemic countries have successfully increased coverage of malaria prevention and control interventions as part of their national strategic plans.[1] As countries consider approaches to sustain these gains or progress toward elimination, there is a continued need for rigorous evaluation to demonstrate the impact of interventions delivered by national control or elimination programs, and to advocate for continued investment in malaria control and elimination.

Impact evaluations of public health interventions typically attempt to attribute changes in a given health outcome to a particular program, intervention, or set of interventions. Common across impact evaluation recommendations of major donor agencies is the need for a counterfactual, allowing one to measure or estimate the change in outcome indicator had the intervention, set of interventions or program never been implemented. However, there are differences between agencies in the requirements for demonstration of causal attribution of outcome to intervention.[2-4] Impact evaluations can use a variety of inferences to explore attribution of intervention to outcome.[5] Although experimental designs using randomization have[4] been viewed as gold standard in evaluation,[6,7] these methods may not be ethical or appropriate for evaluation of routine activities involving interventions previously demonstrated to be effective under controlled conditions. Quasi-experimental designs allow evaluation of intervention impact under routine program conditions, demonstrating the plausibility of an intervention being causally linked with an outcome.[5,8] Considering the complex causal pathways and multiple contributing factors involved in evaluation of public health interventions, as well as the lack of true contemporaneous control groups, plausibility evaluation designs may be more appropriate than probability designs under these circumstances.[6,9]

For the purpose of this paper, we define impact evaluation as the potential contribution of one or more interventions delivered under routine operational conditions to change malaria associated morbidity. Additionally, malaria burden is defined for this paper as either malaria case count or malaria incidence, estimated using routine health management information system (HMIS) data.

Although the HMIS structure has already been established in many countries at either national scale or across malaria-endemic settings to routinely collect and report the number of suspected and laboratory-confirmed malaria cases identified at health facilities, these data have typically been overlooked in favor of population-based cross-sectional surveys as the primary data source for impact evaluations. Objections to use of HMIS data are largely due to potential for measurement error and confounding in malaria burden estimates generated from HMIS data, and concern that these limitations may bias subsequent impact estimates.[10-12] This concern was particularly valid before the expansion of malaria diagnostic testing that has occurred in the past 5 years; when most HMIS data from high burden countries were based solely on clinical cases without confirmation. Consequently, the majority of evaluations assessing impact of malaria interventions applied under routine conditions by national malaria control programs have used population-based, nationally representative cross-sectional survey data (e.g., Malaria Indicator Survey [MIS]) to assess the association between malaria interventions and malaria outcomes.[13-15] Unfortunately, these data are intermittently gathered after relatively long periods of 2–3 years, limiting our ability to assess impact on longitudinal trends in indicators of malaria morbidity.

Considering the substantial investments already made in HMIS systems, the underutilization of routine HMIS data has been described as an unacceptable inefficiency in resource constrained countries.[12,16] HMIS data provide a source of longitudinal data for facilities, enabling estimation of malaria incidence rates over time. Furthermore, widespread adoption of malaria rapid diagnostic tests (RDTs) has increased the availability of parasitologically confirmed malaria case data in HMIS.[1] Adoption of electronic reporting systems such as the District Health Information System 2 (DHIS2) has been shown to improve completeness and timeliness of routine data reporting.[17] HMIS data have particular value for impact evaluation in low transmission settings, where periodic large-scale cross-sectional surveys may have inadequate power to assess trends in malaria outcomes over time, especially at subnational levels.[18] Amending these cross-sectional survey designs to have adequate power for such national and subnational examination of trends in low transmission areas would require a large increase in sample size and cost.

The Global Technical Strategy for Malaria 2016–2030 has emphasized the importance of high-quality routine HMIS data by redefining surveillance as a core intervention in malaria control and elimination.[19] It is therefore likely that the pace of improvements in HMIS data completeness, quality, and use will accelerate in line with the new global strategy. As a result of increasing access to parasitologically confirmed, complete, and representative HMIS data, it is timely to reconsider the applications of these data in impact evaluation.

In this study, we review the literature to describe malaria impact evaluation designs where the primary outcome of malaria burden was generated using routine HMIS data. Limitations and advantages of the various methodologies used are discussed. Cross-cutting methodological issues such as bias and confounding are discussed in relation to generation of outcome estimates from HMIS data, as well as in generation of impact estimates using various analysis approaches.

METHODS

Literature search method.

The literature was searched to identify malaria impact evaluations that used routine health information system data. Although not systematically reviewed, the papers were used to describe the range of methodologies used to assess trends in malaria morbidity or impact of interventions and programs on malaria morbidity, when the primary outcome was generated from routine HMIS data. In addition, the approaches used in the literature to address bias in outcome indicators and bias in impact estimates were investigated, with additional recommendations and examples drawn from the wider literature. A selection of malaria impact evaluations using HMIS data are presented in Table 1, to illustrate the range of methodologies identified in the literature. Since only a minority of malaria deaths occur in health facilities, use of HMIS inpatient data to assess trends in malaria mortality may be biased,20 unless it can be demonstrated that inpatient mortality trends are representative of mortality trends in the population. As such, HMIS-derived indicators of malaria mortality are not considered in this study.

Table 1

Papers identified during literature search presenting malaria trends or impact evaluations using routine HMIS data

Reference	Data extent	Analysis approach	Contextual factors included in analysis interpretation
Pre–post comparison of rates
Chanda and others 2012[29]	30 Districts, 2 years’ data. Confirmed malaria: severe and deaths among < 5 seconds, case fatality rate	Pre–post comparison. Logistic regression model including population and % intervention coverage to account for between district differences.	Change in first-line treatment, consequent changes in treatment seeking
Comfort and others 2014[25]	Two hospitals, 6 years’ data. Confirmed malaria outpatient cases and inpatient admissions	Pre–post comparison. Descriptive comparison of mean outcome level pre–post intervention.	All-cause outpatient and inpatient numbers
Louis and others 2015[96]	4 Years’ data from one district. Clinical malaria	Pre–post comparison. Descriptive comparison of outcome level pre–post intervention.	Nonmalaria illnesses diagnosed at health facilities, health-seeking behavior, reporting completeness, service coverage and quality, community-based health insurance scheme, rainfall
Maes and others 2014[97]	Data from one district, over two separate 3 year periods. Confirmed outpatient malaria and malaria admissions	Pre–post comparison. Descriptive comparison of outcome level pre–post intervention.	Temperature, rainfall, flooding, malnutrition, rift valley fever, IRS campaigns and LLIN distribution, larval source management
Masaninga and others 2013[98]	11 Years’ data, national scale. Confirmed malaria admissions and deaths	Pre–post comparison. Descriptive comparison of outcome level pre–post intervention.	Increased funding for malaria control, LLIN and IRS campaigns
Sarrassat and others 2008[30]	5 Years’ data, one facility. Clinical and confirmed malaria.	Pre–post comparison. Descriptive comparison of outcome level pre–post intervention.	Rainfall (qualitative), facility attendance, change from presumptive treatment to confirmatory diagnosis, population movement
Thang and others 2009[99]	3 Years’ data from two districts. Clinical and confirmed malaria	Pre–post comparison. Incidence rate ratios from Poisson regression to describe change in outcome. No covariates	Rainfall, village health worker program
Willey and others 2011[67]	2 Years’ data from 11 health facilities. Confirmed malaria	Poisson regression comparing incidence between intervention and comparison areas, rather than pre–post intervention.	Vaccination coverage, microscopy quality assurance
Yapabandara and others 2015[100]	7 Years’ data, national scale. Clinical and confirmed malaria	Pre–post comparison. Descriptive comparison of outcome level pre–post intervention.	Health system strengthening, targeting of interventions according to microstratification, change in first-line treatment, adoption of RDTs, IRS chemical.
Descriptive analysis of trends
Alba and others 2011[36]	4 Years’ data from 14 health facilities. Clinical malaria.	Poisson regression models with village-level random effects to describe trends in outcome over time. No other potential confounders included in model.	Change in first-line treatment and estimated vector control coverage. Internal and external validity of data assessed.
Bhattarai and others 2007[81]	7 Years’ data from 13 facilities, clinical malaria and malaria admissions.	Descriptive analysis of indicators over time and Pearson correlation coefficients assessing linear relationship between rainfall and outcomes.	Speed of ACT rollout, climate, vector control interventions
Ceesay and others 2008[34]	Five health facilities, 7–9 years’ data. Slide positivity, proportion of admissions, and deaths due to malaria.	Trends in outcome tested using χ2, and linear regression (without covariates) of log-transformed case count.	Decreasing sales of antimalarial medicines at pharmacies, change in first-line antimalarial. Rainfall data. Changes in socioeconomics, communications, access to education.
Dhimal and others 2014[35]	50 Years’ national data, additional analysis for district-level data from most recent 9 years. Confirmed malaria	Descriptive analysis of trend over period of intervention scale up using linear regression of log-transformed case count. Tested for temporal autocorrelation. No covariates.	Trends in climate variables assessed, not included in models. Changes in vector ecology, insecticide resistance, change in first-line treatment, community passive detection posts, population movement
		Additional pre–post intervention descriptive comparisons.
Kamuliwo and others 2013[37]	6 Years’ data, national scale. Clinical malaria throughout, confirmed malaria for final 3 years only	Poisson regression models with case count and intervention coverage level (categorised), with district-level random effect. No covariates.	Insecticide resistance, population movement.
Konchom and others 2003[101]	11 Years’ data from 30 border-area districts. Annual parasite incidence, ratio of Plasmodium falciparum to Plasmodium vivax	Simple descriptive analysis of indicator levels. t Tests comparing incidence in border and nonborder districts over time.	Population movement, change in first-line treatment
Maude and others 2014[33]	10 Years’ national data from facilities and community-level. Clinical and confirmed malaria.	Simple descriptive analysis of indicator levels and correlations.	Change in RDT, expansion of village malaria worker program, vector control, migration, deforestation
Mufunda and others 2007[102]	6 Years’ data, national scale. Clinical malaria	Simple descriptive analysis of indicator levels. Linear regression to explore association interventions with outcome.	Change in first-line treatment
Mukonka and others 2014[103]	7 Years’ data, from 11 facilities in one district. Clinical and confirmed malaria.	Simple descriptive analysis of indicator levels over time	Insecticide resistance, population movement, changes in access to care and case reporting
Ngomane and others 2006[38]	9 Years’ data from one district. Confirmed malaria, from passive and active surveillance. Malaria mortality.	Descriptive analysis of indicator levels and χ2 test for trend over time. ARIMA model fitted to assess effect of climate on outcome, accounting for temporal autocorrelation.	Changes in first-line treatment, population movement, behavioral factors in some age groups, agricultural practices
Nyarango and others 2006[39]	5 Years’ data, national level. Clinical and confirmed malaria	ARIMA model testing association between interventions and outcome, accounting for temporal autocorrelation. Rainfall included in model as covariate.	Change in first-line treatment, diagnostic quality assurance, health-seeking behavior, quality of case management
Okiro and others 2007[40]	8 Years’ data from hospitals. Confirmed malaria inpatient admissions	Seasonally adjusted linear regression models to examine trends. Models accounted for temporal autocorrelation. Rainfall, nonmalaria admissions and seasonality included as covariates.	Per capital ITN distribution, change in first-line treatment
Okiro and others 2011[41]	10 Years’ data from five hospitals. Confirmed malaria inpatient admissions.	ARMAX models to examine long-term trends in outcome. Models accounted for temporal autocorrelation, rainfall and nonmalaria admissions included as covariates.	Changing diagnostic practices, abolition of user fees, differences in access to effective treatment between districts
Okiro and others 2013[42]	11 Years’ data from four hospitals. Suspected malaria inpatient admissions	ARMAX models to examine long-term trends in outcome. Models accounted for temporal autocorrelation. Rainfall and nonmalaria admissions included as covariates.	ITN distribution timeline, change in first-line treatment, adherence to national treatment guidelines
Otten and others 2009[26]	7 Years data from 13 facilities in one country, 7 years’ data from 19 facilities in a second country. Outpatient confirmed malaria, inpatient malaria admissions.	Linear regression on case count data to describe trend over time.	Nonmalaria health facility attendance, timing of LLIN distributions, introduction of health insurance schemes, civil conflict resolution
		Additional pre–post intervention descriptive comparisons.
Interrupted time series
Aregawi and others 2011[47]	4 Years’ pre- and 1 year post-intervention data, six inpatient facilities. Confirmed malaria outpatients and admissions, malaria inpatient deaths.	Interrupted time series using log-linear regression model. No potential confounders included in model.	Climate, urbanization, and socioeconomic development
Aregawi and others 2014[48]	5 Years’ pre- and 6 years’ post-intervention data, outpatient and inpatient, from 41 hospitals. Clinical and confirmed outpatient malaria, confirmed malaria admissions and deaths.	Interrupted time series using ARIMA model, accounting for temporal autocorrelation. No potential confounders included in model.	Linear association examined between rainfall and case count and slide positivity. Rainfall compared between pre–post intervention periods. Nonmalaria OPD and IPD data discussed
Bukirwa and others 2009[51]	One health center, 8 months pre- and 16 months post-intervention data. Clinical malaria and slide positivity rate	Interrupted time series using ARIMA model, accounting for temporal autocorrelation. Covariates included age sex, rainfall.	Changes in proportion of suspected malaria cases sent for diagnostic test, ACT scale-up, ITN distribution interventions
Karema and others 2012[46]	11 Years’ data from 30 hospitals	Interrupted time series using log-linear regression model. No potential confounders included in model, but accounts for temporal autocorrelation.	Rainfall and temperature trends, laboratory quality assurance, initial targeting of ITNs to children under five
Kigozi and others 2012[49]	5 Years’ data, one health facility. Slide positivity rate	Interrupted time series, correcting for temporal autocorrelation. Include age and monthly seasonality as covariates	Change in insecticide used for IRS, changes in ITN coverage and use
Landoh and others 2012[50]	6 Years’ data, national scale. Clinical and confirmed malaria	Interrupted time series using ARIMA model, accounting for temporal autocorrelation. Includes rainfall and temperature as covariates.	Increased access to health care, roll out of RDTs.
Santelli and others 2012[52]	5 Years’ data from three districts. Confirmed malaria (active and passive surveillance)	Interrupted time series using log-linear regression model, including seasonality and month-intervention interaction term.	Epidemic prior to intervention, local malaria control management, vector control, reduced efficacy of standard treatment course
Teklehaimanot and others 2009[53]	7 Years’ data, national scale. Confirmed malaria, malaria admissions and deaths	Interrupted time series using spline regression model, including seasonality as covariate. Separate time series model assessing change in rainfall.	Health service coverage and access
Dose-response analysis
Bennett and others 2014[24]	3 Years’ routine HMIS data, national scale (1,693 facilities). Confirmed malaria.	National platform approach (district-level dose-response). Covariates included treatment seeking, climate, health care access, testing rate, and reporting rate	Regional population movement, insecticide resistance, potential endogenous relationships between outcome and explanatory variables
Graves and others 2008[54]	6 Years’ data, national scale. Clinical malaria.	National platform approach (district-level dose-response), accounting for temporal correlation. Covariates included rainfall and vegetation cover (NDVI) indicator	Change in first-line treatment, presumptive treatment during high transmission season in one district

ACT = artemisinin-based combination therapy; ARIMA = autoregressive integrated moving average; ARMAX = autoregressive moving average model with exogenous inputs; HMIS = health management information system; IPD = inpatient department; OPD = outpatient department; RDT = rapid diagnostic test.

HEALTH MANAGEMENT INFORMATION SYSTEM

The core function of an HMIS is to collect, transmit, and analyze indicators required for health system management.[21] The current review focusses on health facility-based HMIS data, but is also relevant to any parallel malaria-specific reporting systems operating in countries. In general, clinical data from individual patients are aggregated by age category (< 5 or ≥ 5 years) at facility-level each month, then reported to the supervising administrative unit (e.g., district) using a standardized format. Districts review and analyze data received from facilities, provide feedback, and may aggregate data into district totals to submit to the next level (e.g., region).

Some countries have begun reporting malaria diagnoses made by community health workers using RDTs, either to HMIS or through parallel reporting structures.[22] HMIS usually does not include private health facility data, but there is growing interest in the utility of private sector data in malaria surveillance, particularly in elimination settings.[23] Following expansions in mobile telephone network coverage and reducing cost of mobile internet, some countries are beginning to transition their HMIS systems from paper- to cloud-based systems, such as the DHIS2.[17,24]

HMIS indicators particularly relevant to malaria impact evaluation include all-cause outpatient attendance, clinical malaria diagnoses, number receiving parasitological test (either by microscopy or RDT), test positivity rate, diagnostically confirmed malaria cases, and malaria inpatient admissions. Monthly malaria case count or monthly malaria incidence per 1,000 catchment area population are the main outcome indicators for impact evaluations included in this review.

IMPACT EVALUATION DESIGNS USING HMIS DATA

Pre–post intervention comparison.

The simplest approach to use of HMIS data in impact evaluation is through pre–post ecological comparison, where malaria case count or incidence is estimated before and after the intervention and tested statistically for evidence of change.[25] However, malaria transmission is influenced by a range of factors (e.g., climate and environment) and the main limitation of pre–post comparison designs is an erroneous assumption that all changes in malaria case rate over time are attributable to the intervention.[26,27] These simple pre–post comparisons are consequently vulnerable to confounding from secular trends and other contextual changes over time.[28]

Pre–post comparison studies can be improved by describing advances or declines in relevant contextual factors and potential confounders or effect modifiers, and presenting a balanced interpretation of impact estimates considering the limitations of available data and potential for bias in impact estimate.[29,30]

When appropriate nonintervention areas (here termed “contrast” areas since they are not true controls) can be either observed or modeled, the difference in difference (DiD) estimator can be applied; this estimates the difference in pre–post estimators between intervention and contrast areas. The DiD approach therefore enables pre–post comparison designs to take into account underlying trends in outcome level over the intervention period. To avoid biased impact estimations, DiD requires a valid contrast area or counterfactual, representing the change in outcomes that would have been experienced by the treatment group in the absence of interventions.[31] In practice, DiD estimators can be generated through use of multivariate logistic regression models including dummy variables identifying intervention/contrast group and pre/post intervention period, together with an interaction term of intervention/contrast group and pre/post intervention period. DiD is most commonly used for population-based survey data.32

Descriptive analyses of trend.

Descriptive analysis of indicators over time can be a useful component of an impact evaluation, providing quantitative or qualitative interpretation of change over time in outcome indicators. Simple descriptive interpretation analysis of HMIS data can be strengthened by presenting plots of multiple indicators over time, conducting analysis at both national and subnational level, and description of contextual changes which may be influencing observed trends.[33] Descriptive analyses of HMIS trends can be supplemented by χ2 or t tests for trend, as well as linear regression of log-transformed case count data.[34]

Logistic regression approaches enable examination of trends in outcome indicators after accounting for contextual factors that varied over the evaluation period and may have confounded or interacted with associations between exposure and outcome. Linear regression of case count data should be avoided, since this could result in prediction of negative case counts.26,27 Multiple examples exist using linear regression of log-transformed case count data,[34,35] or Poisson or negative binomial regression of case count data.[36,37] Autoregressive integrated moving average (ARIMA) and other time-series model structures are appropriate for analysis of trends in HMIS indicators over time, since they account for temporal autocorrelation and allow adjustments for seasonal trends.[38-42]

Although considering contextual factors in analysis interpretation does strengthen pre–post comparisons, this approach continues to be limited by the lack of counterfactual for describing the change in malaria outcome over the evaluation period in the absence of an intervention.

Interrupted time series.

Interrupted time series (ITS) analysis involves comparison of level and mean trend in outcome indicators before and after a breakpoint.[43,44] Although more than one breakpoint can be considered in ITS models, the approach is most suited to single interventions that are rolled out over short periods with consistent intensity. In settings where interventions required more time to achieve full implementation (e.g., staggered long-lasting insecticide-treated net (LLIN) distributions across a country over a period of 6 months) after introduction, or where a lag is expected between introduction of the intervention and effect on outcome, ITS models can be adapted to test for the optimal breakpoint, or divided into preintervention, intervention rollout, and postintervention segments. However, this weakens the ITS approach since testing numerous breakpoints or having a long scale-up period increases the plausibility of changes in outcome being attributable to other potentially confounding factors.[45]

Some ITS analyses simply examined associations between timing of intervention scale-up and change in case incidence[46,47]; however, it is recommended that contextual factors are included in ITS models to avoid erroneous attribution of changes in outcome to the intervention. Climate data are commonly included as covariates in ITS analyses, both for evaluations comprising a small number of facilities and for national level analyses.[48-50] Many ITS models use ARIMA structures,[48,50,51] but other time series regression models can be used.[46,52,53] Regardless of modeling structure, ITS should incorporate terms to address temporal autocorrelation, and seasonality effects should be modeled and removed from a time series before assessing treatment impact.[45]

A further extension is to analyze a contrast group time series using ITS, to explore if other contextual changes occurred which may confound any observed association between intervention breakpoint and change in malaria burden indicators. For example, ITS could be applied to all-cause or nonmalaria outpatient visits, to investigate if there were changes in facility attendance over the evaluation period. Although not strictly a counterfactual scenario, examination of trends in other indicators available from HMIS can give insight to potential confounding or contextual changes occurring at the same time as the intervention.

Subnational dose response.

The national platform approach to impact evaluation explores a dose-response relationship between intervention and outcome at a subnational (e.g., district) level.[9,32] The method overcomes challenges in identifying a valid counterfactual in environments of universal scale-up, or where ethical concerns preclude withholding of interventions. Continuous monitoring of different levels of contextual indicators, and collection of additional data before, during and after the intervention are recommended for the national platform approach, together with use of multiple analytical techniques to address potential biases in the data.[9]

Key to this dose-response analysis approach is the availability of intervention and covariate data at the subnational level. Intervention data may be available in district health authorities’ records, such as number of households receiving indoor residual spray, or total LLINs distributed per capita. Alternatively, district-level data on intervention coverage can be extracted from population surveys such as the DHS or MIS.

Examples of use of the national platform approach for impact evaluation are available from Eritrea and Zambia.[24,54] Both evaluations used routinely collected HMIS data aggregated at the district-level to conduct a national evaluation of changes in malaria burden over a period of malaria control intervention scale-up. The Eritrea evaluation used program records of vector control coverage (including decay functions to represent LLIN losses and reduced effectiveness of indoor residual spray (IRS) insecticide over time) together with climate data, to assess the dose-response relationship between intervention and malaria case count.[54] The Zambian study used the same principles, but with a more complex approach to generation of complete outcome and intervention data; a Bayesian framework was used incorporating temporal and spatial autocorrelation, and a range of contextual factors considered in the model.[24]

FURTHER CONSIDERATIONS FOR IMPACT EVALUATION DESIGN USING HMIS DATA

Overcoming common sources of bias in HMIS data to generate outcome indicators.

HMIS data have been underutilized in impact evaluation due to concerns that malaria burden estimates generated from HMIS data are very prone to bias. Most sources of bias in case incidence data would only be expected to lead to a biased estimate of intervention or program impact if the factor was associated with the intervention under evaluation (as classically defined for confounding variable). The exception is nondifferential misclassification bias, which would be expected to lead to an underestimate of intervention or program impact. Nevertheless, this section discusses methods to minimize bias when generating outcome indicators used for impact evaluation from routine HMIS indicators.

Bias in estimates of malaria case incidence from HMIS data can result from a range of factors: challenges in estimating size of catchment populations, changes in treatment-seeking, variable access to and use of parasitological diagnosis, and incomplete recording and reporting of data (both missing monthly reports and incomplete registration of patients in registers).

Defining the catchment population for a health facility enables standardized estimates of case incidence to be used as the primary outcome measure in impact evaluation. Case counts are influenced by the size and composition of the population being served by the health facility; facilities serving larger populations are expected to have more cases than those serving smaller populations. However, catchment populations may not be static over time, and individuals may choose not to attend their closest health facility,[55] creating challenges in appropriate estimation of catchment populations. Approaches used to estimate catchment population include generating Euclidean buffers around facilities,41 Thiessen polygons,[42] estimating travel time using road networks, land use and altitude gradients,[24,56,57] a variable market approach,[58] the Huff competitive market model,[59,60] or calculation of cumulative case ratios.[61] Where population data are available by age (categorical or continuous), malaria incidence rates can be disaggregated. This is particularly recommended where age structures may vary over time or space, and can also be valuable in tracing changes in transmission intensity indicated by peak shift of symptomatic and severe malaria cases.[62] If no population data are available within the country, district-level population can be extracted from global geo-referenced population databases,[63] and analysis conducted at district-level using aggregate HMIS indicators.

In settings where not all who fall ill seek treatment, and not all seeking care go to a public health facility, HMIS data will systematically underestimate malaria case rates for the given community. Less commonly, where variations in health-seeking behavior are associated with the intervention of interest, this may result in biased estimates of intervention impact. Estimates of health-seeking behavior can be extracted and modeled from MIS or DHS data, where women are asked if and where they took their children for medical treatment if they were recently unwell.[24] Questions in these surveys can also be used to estimate the proportion of the population who seek treatment from the private sector, and are consequently underrepresented in routine HMIS data. A simple alternative is include nonmalaria outpatient attendance at the facility to adjust for changes in all-cause health-seeking behavior over the evaluation period, assuming that there have been no changes in total catchment population over the evaluation period.[40]

Where HMIS data report only clinical (not parasitologically confirmed) malaria, symptom-based diagnostic algorithms are known to have low specificity.[64] In the absence of confirmatory testing, a health worker’s decision of whether to diagnose malaria may be influenced by factors related to the health worker (e.g., level of training, workload) or the patient characteristics (e.g., age, treatment expectation), or by broader contextual factors.[65] Accounting for low specificity of clinical diagnosis in analysis is very challenging; it is recommended that if possible, only confirmed malaria cases be used to derive primary outcome indicators for impact evaluations. Fortunately, widespread scale-up of RDTs has increased the likelihood of diagnostic confirmation among suspected malaria cases, making data on confirmed malaria cases increasingly available. However, changes in access to parasitological diagnosis have the potential to bias assessment of temporal trends in confirmed malaria rates, unless malaria testing rates among febrile patients, and periods of diagnostic tool stock-out are incorporated into analysis.[66] Settings where RDTs were initially used for diagnosis in children under five, then subsequently expanded to all ages have additional complexity, since both testing rates and test positivity rates during the under-five targeted RDT period may be biased by children’s increased risk of clinical malaria episodes compared with older age groups. Use of test positivity rate as an outcome indicator is not advised for impact evaluations, since this indicator may be biased by a wide range of factors, including the type of diagnostic tool, availability of diagnostics, workload at the facility, season, and patient characteristics. Programmatic changes such as adoption of RDTs at facilities, stock-outs, changes in RDT brand or antigen combination, introduction of systematic quality assurance and quality control approaches for monitoring RDT users, or even timing of refresher training for health workers on malaria case management are additional contextual factors which could be incorporated into impact evaluation.[67]

Incomplete reporting of data to the HMIS leads to systematic underestimation of disease rates. The reporting completeness of any HMIS, together with many other surveillance system attributes, should be carefully evaluated on some periodic basis.[68] Where the level of underreporting is associated with the intervention under evaluation, impact estimates may be biased unless underreporting is addressed in analysis. Incomplete reporting may result from inconsistent submission of monthly reports by facilities, omission of specific indicators within submitted reports, or patient data being only partially recorded in outpatient and laboratory registers. To address missing reports or indicators, imputation methods can be used if relatively few data are missing from a time series.[69] Models with spatial and temporal correlation structures may also inform estimation of missing data, weighting estimates according to known values which are close in space and time.[70] Alternatively, estimates of reporting completeness can be included as a variable in analysis; a preferred method over direct adjustment of imperfect incidence data, particularly at small temporal and spatial scales.[71] Underreporting due to patients not being recorded in registers is substantially more challenging to address in analysis, and may require restriction of analysis to sentinel facilities or those known to have highly complete data.[34,54] However, restricting impact evaluations to the best-performing facilities has potential to bias impact estimates if intervention quality, coverage or use in these populations differs to that among populations served by underperforming facilities which are excluded from analysis. When possible, HMIS data from all health facilities should be included in impact evaluations.

The counterfactual.

Presenting an appropriate counterfactual scenario is recommended by multiple major donor agencies for impact evaluations.[2-4] However, appropriate counterfactuals can be very challenging to identify under operational contexts in most malaria-endemic countries.

When using a pre–post comparison design, the counterfactual assumes that no change would have occurred in the outcome without the intervention. A crude approach to estimate a counterfactual in pre–post evaluations is to compare changes in nonmalaria HMIS indicators, such as all-cause outpatient attendance or mean rates of another disease, before and after the intervention. However, this approach should be used with caution, since other the disease used as a counterfactual could have experienced a natural epidemic, or been subject to other interventions not directly considered in the evaluation.

It may be possible in some settings to use a comparator area as a counterfactual, where the assumption is that had the intervention has not been applied, trends in the outcome indicator would be the same in the intervention and comparator areas. This approach may be useful where interventions were targeted to specific areas, or rolled out as part of stepped-wedge trial designs.[32,66,72]

The counterfactual for most ITS approaches assumes that the mean trend for the preintervention period would have continued in the absence of the intervention. If a time series model was fitted to preintervention period data including climate and other temporally dynamic covariate data, the regression model could be extended to predict a counterfactual for the postintervention period using climate covariates for that period. Generating counterfactuals using ITS and regression extrapolation is challenging in environments with limited preintervention data due to challenges in fitting preintervention models, and where recent systematic changes in diagnosis or data collection bias preintervention incidence estimates.

Dose-response evaluation designs make use of subnational variations in intervention coverage or program intensity to estimate impact on the malaria morbidity outcome indicator. In these evaluation designs, a counterfactual may be represented by a “zero-dose” district, but are not explicitly required in analysis.

Contextual and confounding factors.

When designing an evaluation of interventions applied in complex operating environments, it is recommended to include data on relevant contextual factors and their variation over time.[73,74] In this paper, we have referred to contextual factors as a broad group of qualitative or quantitative factors that may potentially bias exposure, outcome, or impact estimates. We consider confounding factors and effect modifiers to be specific contextual factors with precise epidemiological definitions: confounders being variables that are both associated with exposure and outcome but not on the causal pathway, while effect modifiers are alter the effect of a putative causal factor being investigated.[75] The Bradford Hill considerations can assist in describing the strength of evidence for causal associations in quasi-experimental impact evaluation designs.[76,77] Logic models and theoretical frameworks are a useful approach to describe hypothesized causal pathways, contextual factors, confounders and effect modifiers, informing evaluation design, and interpretation in complex settings.[78-80]

Potentially relevant contextual factors for malaria impact evaluations include other health programs being implemented that may influence malaria outcomes (e.g., vaccination, deworming, nutrition programs), climatic conditions (specifically temperature, precipitation and relative humidity, or proxy measures such as vegetation indices), and any other events in the community (e.g., food insecurity, population movements, economic shocks, or political events) that occurred over the period of observation and which have the potential to alter any of the indicators being monitored. A number of studies have used a mixed approach to evaluation, incorporating some contextual factors directly into models as potential confounders or effect modifiers and considering additional contextual factors in models interpretation as part of a plausibility approach.[81-83]

Indicators such as access to health services, treatment-seeking behavior, and reporting completeness should be examined for association with the intervention under evaluation, and ideally be incorporated into analysis to strengthen internal validity of HMIS data. In settings with mobile or migratory populations, indicators of human population movement may be valuable to incorporate in large-scale evaluations, since changes in malaria susceptibility and risk of transmission may occur following population movements.[84,85]

Environmental variables are recommended for inclusion in time series models, since vector populations and therefore malaria transmission intensity are influenced by climatic conditions.[86] Failure to incorporate environmental covariates in models may lead to erroneous attribution of impact to interventions, if malaria outcomes are (partially or fully) due to a change in climatic conditions rather than the intervention. Although some small-scale impact evaluations have used local weather station data, the use of satellite-derived environmental data is becoming increasingly popular.[24,38,40,50] A systematic audit of an extensive library of environmental covariates led to development of a set of covariates which were found to reliably model malaria risk Plasmodium falciparum parasite rate (PfPR2–10) patterns and how these patterns change through time.[87] However, anomalies in rainfall, temperature, and vegetation indices are the most widely used environmental covariates in malaria models. Many climate variables are best considered as lagged variables, due to the time required for a change in climate to manifest as a change in a malaria outcome.

Where evaluations incorporate data from a range of malaria transmission settings, an estimate of transmission intensity such as modeled PfPR2–10 can be incorporated in analysis as a potential effect modifier.[15,88] Maps of modeled PfPR2–10 are publically available at 5-km resolution,[89,90] enabling extraction of PfPR2–10 estimates for specific sites.[91,92] However, since PfPR2–10 estimates are generated using models that include environmental variables, interventions, seasonality and other confounders, including PfPR2–10 in impact estimation models may bias effect estimates upward due to circularity.

Considering the range of contextual factors, confounders and effect modifiers which could be included in impact evaluations, theoretical frameworks are recommended to explore the hypothesized relationships between each factor and the exposure and outcome indicators, and begin prioritizing factors on which to gather data and incorporate in analysis. In malaria impact evaluations utilizing HMIS data, priority contextual factors include indicators of access to health care and treatment seeking for malaria, malaria diagnostic practices, and climate data. In many settings, indicators of access to health care and treatment seeking are available in periodic national surveys such as DHS, while diagnostic practices will be documented or known by national malaria program staff. Satellite-derived climate data are readily available for many countries from sources such as the U.S. Geological Society.

Autocorrelation.

Time series data are typically autocorrelated, meaning that an observation may be correlated (positively or negatively) with observations one or more period lags prior.[45] Failure to account for autocorrelation in analysis of time series can result in erroneous findings of statistical significance.[44,93] Similarly, due to the correlated nature of data across spatial units, spatial autocorrelation should be considered in analysis of HMIS data from large numbers of facilities with known locations. Use of methods such as ARIMA automatically adjusts for temporal autocorrelation.[94] A simpler approach is to incorporate a lagged outcome variable in logistic regression models to account for temporal correlation.[54] The lag may be 1 month or more, according to the autoregressive structure of the data.

Accounting for autocorrelation is also important for standard error and 95% confidence interval estimation in models assessing the significance of exposure variables on the outcome of HMIS-derived rates. Although missing data can be generated in models using spatial and temporal interpolation methods,[70] using a Bayesian framework for analysis allows uncertainty in outcome data and other indicators to be propagated forward through models into impact estimates.[24,95]

CONCLUSIONS

HMIS data are common across all malaria-endemic countries, yet have been underused in impact evaluation due to quality and bias concerns. Increasing investments in malaria surveillance, broad availability of RDTs, improved systems to record, transmit, and process these results, and the reduced power of population malariometric surveys in areas of low transmission prompt a necessary reconsideration of the utility of HMIS data in impact evaluation.

The impact evaluations reviewed fall into four main designs: pre–post intervention studies, descriptive analyses, ITS, and dose-response analyses. Pre–post designs and descriptive analysis are limited by a lack of counterfactual, but can be bolstered by presenting a theoretical framework of contextual, confounding, and interacting factors which could bias impact estimation. ITS and dose-response designs are more rigorous approaches to impact estimation using HMIS data. ITS is appropriate when interventions have been scaled up nationally and rapidly, relevant contextual variables are included in models, and sufficient preintervention data are available to generate a counterfactual. Where subnational data are available describing intervention coverage or intensity over time, a dose-response analysis strategy incorporating relevant subnational data on potential confounders is appropriate. The methods discussed for analysis of HMIS data in impact evaluation range from simple to complex, yet most can present meaningful information if presented using a plausibility approach.[5,6]

Multiple approaches have been presented to address some of the common biases in HMIS data, including methods to estimate the catchment population of facilities, incorporate estimates of health service access and health-seeking behavior, diagnostic test use and validity, as well as incomplete reporting. These variables should be either directly included in models or explored in result interpretation along with other contextual factors, considering if, and to what extent, these may have influenced the observed data or estimated impact. We recommend that confirmed rather than clinical malaria be used as the primary outcome indicator when using HMIS data in impact evaluation, due to low specificity of symptom-based diagnostic algorithms for malaria. A population denominator should also be included, either by use of malaria incidence as the primary outcome, or inclusion of health facility catchment populations in Poisson models using case count data.

As malaria programs continue to strengthen and expand to pursue malaria elimination, historical and prospective use of HMIS data for rigorous impact evaluations will be needed. Improvements in health system performance, access to health care, increasing availability of RDTs for parasitological confirmation of suspected malaria cases, combined with advances in electronic information systems (data capture and transmittal) will increase the validity of indicators derived from HMIS making these data increasingly useful and efficient for rigorous impact evaluations.