Re-analysis of health and educational impacts of a school-based deworming programme in western Kenya: a pure replication.

BACKGROUND: Helminth (worm) infections cause morbidity among poor communities worldwide. An influential study conducted in Kenya in 1998-99 reported that a school-based drug-and-educational intervention had benefits regarding worm infections and school attendance. Effects were seen among children treated with deworming drugs, untreated children in intervention schools and children in nearby non-intervention schools. Combining these effects, the intervention was reported to increase school attendance by 7.5% in treated children. Effects on other outcomes (worm infections, anaemia, nutritional status and examination performance) were also investigated.
METHODS: In this pure replication, we used data provided by the original authors to re-analyse the study according to their methods. We compared these results against those presented in the original paper.
RESULTS: Although most results were reproduced as originally reported, we identified discrepancies of several types between the original findings and re-analysis. For worm infections, re-analysis showed reductions similar to those originally reported. For anaemia prevalence, in contrast to the original findings, re-analysis found no evidence of benefit. For nutritional status, both original findings and re-analysis described modest evidence for a small improvement. For school attendance, re-analysis showed benefits similar to those originally found in intervention schools for both children who did and those who did not receive deworming drugs. However, after correction of coding errors, there was little evidence of an indirect effect on school attendance among children in schools close to intervention schools. Combining these effects gave a total increase in attendance of 3.9% among treated children, which was no longer statistically significant. As in the original results, re-analysis found no effect of the intervention on examination performance.
CONCLUSIONS: Re-applying analytical approaches originally used, but correcting various errors, we found little evidence for some previously-reported indirect effects of a deworming intervention. Effects on worm infections, nutritional status, examination performance and school attendance on children in intervention schools were largely unchanged.

Key Messages

It remains controversial whether or not deworming schoolchildren results in better school attendance—one study conducted in Kenya in 1998–99 remains central to the debate.

This original study looked at both direct and indirect effects of a complex deworming intervention on a range of outcomes including removal of worm infections, anaemia, nutritional status, school attendance and examination performance.

In this pure replication, we re-analysed the original data according to the methods used by the original authors and compared results with those presented in the original paper.

Although most results were reproduced as originally reported, there were discrepancies of several types between the original paper and re-analysis.

In contrast to the original study, re-analysis found little evidence for an effect on anaemia or for indirect effects on school attendance for children in non-intervention schools; other results were largely unchanged.

Introduction

Helminth infections cause substantial morbidity across the developing world., Opinions differ over whether deworming schoolchildren in such settings improves nutritional outcomes, school attendance or educational achievement. For some, deworming is among the most cost-effective investments in global health: in 2012, Nobel laureate economist Robert Mundell described deworming as a ‘simple, cheap investment [that] can mean a child is healthier and spends more time in school’. However, in the same year, the Cochrane Collaboration concluded ‘… it is probably misleading to justify contemporary deworming programmes based on evidence of consistent benefit on nutrition, haemoglobin, school attendance or school performance as there is simply insufficient reliable information to know whether this is so’.

Central to this debate is a study describing the impacts of a school-based deworming programme in Kenya on the health, school attendance and academic performance of school pupils. The study is highly regarded, winning the 2005 Kenneth J. Arrow Award for Health Economics. However, the Cochrane Collaboration judged the study to be at ‘high risk of bias’. The debate was recently sharpened by publication of a journalistic article titled ‘Deworming debunked’ in the British Medical Journal.

In this ‘pure replication’ of this influential study, we conduct an independent reconstruction of variables from the data and re-estimate intervention effects using the study’s original methods. In a companion paper we apply alternative statistical methods to the same data to see how the conclusions reached compare with those of the original study.

Methods of original study

Study design

Data were collected alongside a school-based deworming programme conducted in 75 schools in western Kenya in 1998–99. The original paper reports that schools were stratified by administrative area and involvement in other programmes, and allocated to three groups. However, in a correction (unpublished observation, Edward Miguel) the allocation was described as follows.For these groups (25 schools per group) the deworming intervention was introduced in stages over 2 years, in a ‘stepped-wedge’ design as shown in Figure 1. Schools therefore represent the unit of clustering in this study. The study design could be described as a ‘cluster quasi-randomized stepped-wedge trial’.

Figure 1.

Stepped-wedge design shown in schematic form. The intervention was rolled out in ‘steps’, with Group 1 receiving the intervention in year 1 (1998), Group 2 in year 2 (1999) and Group 3 in 2001.

Schools were stratified by division and zone and the zones were listed alphabetically within each division. Three schools that were originally excluded were included at the end of the list. Within each zone, the schools were listed in increasing order of pupil enrolment as of February 1997 for grades 3–8. The first school in the list was allocated to group 1, the second to group 2, the third to group 3, the fourth to group 1 and so forth to the end of the list.

Intervention

This complex intervention was in two parts: first, administration of anti-helminthic (deworming) drug treatments given in appropriate doses at spaced intervals and second, a package of educational interventions. Girls over 12 years were not intended to receive the drug intervention, although in practice some did. Different drug combinations were used based on the prevalence of different types of worm infections in each school before the intervention. Educational measures consisted of worm prevention education, including stressing the importance of hand washing, wearing shoes and not swimming in fresh water. The original paper also describes other school-based interventions occurring concurrently in 27 of 75 schools. No intervention was conducted for schools in the ‘control’ state.

Types of worm infection

The original analysis examined four different types of worm infection: hookworm, roundworm, whipworm (geohelminths) and schistosomiasis. All the schools received drug treatment against geohelminths, but only a minority [6/25 in 1998 (Group 1 only), 16/50 in 1999 (Groups 1+2)] were eligible for schistosomiasis treatment.

Outcome measures

The impact of the intervention was measured on worm infections, nutritional and haematological parameters, school attendance and examination performance (Table 1). School attendance was measured by fieldworkers performing multiple unannounced visits to schools during the academic year. Examination performance was measured through examinations. Worm infections were measured in treatment schools immediately before deworming, as it was felt unethical to test for worm infection without providing treatment. Therefore, worm infection rate was not measured in Group 2 in 1998 nor in either year in Group 3. Haemoglobin and nutritional status were measured in randomly selected subsets of children. The summary effects for all health outcomes were based on results collected at the start of 1999, before Group 2 received deworming treatment. In the original analysis, the direct benefit of the intervention in each of the three domains was estimated by comparing outcomes in schools that received the intervention with those that did not receive it in that year.

Table 1.

Summary of findings of original study

Effects felt to be beneficial and significant by the original authors are shaded.

SD, standard deviation; SE, standard error; Hb, haemoglobin; WAZ, weight-for-age z-score; HAZ, height-for-age z-score; mod, moderate; hvy, heavy; inf, infection; prop’n, proportion.

†Year 1 data only.

Categories of effect

The main effects of the original study fall into several different categories (Table 1). The direct effect was the effect of the intervention on pupils who received drug treatment. An indirect-within-school effect was the effect on all children in treatment schools arising from drug treatment of the children within those schools, including both treated and untreated children in treatment schools. The naïve effect (original authors’ terminology) was the effect found by comparing all children in treatment schools with all children in control schools, irrespective of whether or not the children were themselves treated. This naïve effect is the most similar to type of result that would typically be examined in a pragmatic evaluation of a cluster-randomized trial in biomedical literature. The original authors termed this effect as ‘naïve’ as they felt it would inappropriately underestimate the true total effect.

The original analysis also assessed indirect benefits (‘positive externalities’) resulting from reduced transmission of worm infections to nearby untreated schools using a spatial approach, illustrated in Figure 2. The schools (Treatment = T, Control = Cn) were each at different distances from treatment schools. As worm infections are transmitted by excretion of worm eggs in faeces, and as faecal contamination of the environment was known to be common, it was hypothesized that there would be a local reduction of transmission of worm infection around the intervention schools. Effects were calculated based on composites of results at 0–3 km and 3–6 km. It is not clear to us from the original paper how these intervals were decided on. The authors hypothesized that both treatment and control schools would have greater indirect reduction in worm infection if they were close to many treatment schools. An additional independent term was also used in their modelling process to account for variation in local population density. The variation in indirect benefit across a gradient of exposure created by the variation in spatial proximities was then used to estimate the indirect-between-school effect. Finally, a total effect on those treated (or total effect) was determined: this is the combination of the ‘direct effect’ (which applies to treated pupils only) with ‘indirect-within-school effect’ (which applies to treatment schools only) and the ‘indirect-between-school effect’ (which applies to all schools). In this pure replication, we did not evaluate the appropriateness of separating effects into the different categories as described above. Instead, we reproduced the analytical steps to re-determine the results as originally calculated.

Figure 2.

Schematic representation of approach for determining indirect-between-school effect.

The original paper reported direct and indirect benefits for health and school attendance arising from the intervention, but no effect on examination performance (Table 1) among eligible children (all boys and girls aged < 13 years). As the number of individuals to whom the effects apply varies, the total effect is not a simple addition of different component effects. The results are for both years of the study combined, unless otherwise stated.

Methods of pure replication

In line with a published analysis plan, we describe discrepancies between the published paper and a re-analysis conducted according to the authors’ original methods. No new calculations were performed or concepts introduced. The authors’ calculation steps were reproduced after making appropriate corrections, where necessary. The authors supplied us with their data, files for data processing (do-files) and explanatory notes, including a document describing issues that they were already aware of in the original manuscript, which was used for cross-checking against discrepancies that we identified. We checked individual values for the outputs in each of the tables and associated calculations described in the intervening text. Appendix tables from the original study were not included.

In line with the reporting format of the original paper, we describe results by giving the parameter value, accompanying standard error (SE) interval and a categorical description of the level of statistical significance at 90%, 95% and 99% confidence.

Results

We identified five types of discrepancy between the original reports and our re-analysis (summarized in Table 2 and below, details in Web appendix tables, available as Supplementary data at IJE online).

Table 2.

Summary of discrepancies detected in re-analysis

Coding error descriptions:

Error #1  =   Number of schools counted for calculating local population were truncated to 12, rather than allowing up to 75.

Error #2  =   three schools (numbers 108, 109 and 115) had local population density figures calculated incorrectly.

Error #3  =  wrong combinatorial logic used in creation of ‘any moderate/heavy geohelminth infection’ variable.

Error #4  =   wrong variable used for calculation of population of pupils within 3–6 km of schools.

Unclear labelling. For example, results were labelled as ‘Average infection intensity in eggs/gram’, when a clearer description of the calculation performed would have been ‘Average worm burden in whole population tested, in eggs/gram’.

Rounding errors. For example, a result of 0.745 was displayed as 0.74 rather than 0.75.

Inaccurately reported denominators.

Mislabelled levels of significance. For example, a result with P = 0.06 was annotated as being significant at a 95% confidence level, whereas it should have been labelled as being significant at a 90% confidence level.

Coding errors in STATA analysis (‘do’) files.

The results described and the discrepancies identified in the original tables were as follows (original titles reproduced in bold).

Table I: 1998 Average pupil and school characteristics, pre-treatment. These were descriptive statistics relating to pupils and schools that were measured in all 75 schools at baseline in 1998. Two coding errors (#1 and #2, see Table 2) affected calculations of local population densities and also affected tables VII to X. Although some results in this table were affected by these coding errors, this did not lead to meaningful differences in interpretation.

Table II: January 1998 Helminth Infections, Pre-treatment, Group 1 schools. These data were proportions of pupils with different types of worm infection. No major discrepancies were found.

Table III: Proportion of pupils receiving deworming treatment in PSDP. These data described proportions of different groups of pupils receiving the drug component of the intervention (PSDP = Primary School Deworming Program). No major discrepancies were found.

Table IV: Proportion of pupil transfers across schools. These data were analysed to identify if there were unbalanced movements between groups during the course of the study. No major discrepancies were found.

Table V: January to March 1999, Health and Health Behavior Differences between Group 1 (1998 Treatment) and Group 2 (1998 Comparison) schools. This table reported health-related outcomes for Groups 1 and 2, mainly with data from the beginning of 1999, before Group 2 had received treatment. There were no baseline parasitological data from Group 2 in 1998 to compare against as these were not collected. One important discrepancy was identified: the proportion of pupils who were anaemic was inaccurately labelled as being significantly lower for intervention pupils at 95% confidence—in fact, there was no significant difference (P = 0.194). We also noted that the difference in height-for-age z-score (HAZ) between intervention and control pupils in 1999 remained small (0.08 standard deviations in re-analysis; 0.09 standard deviations in original analysis) and had an associated P-value of 0.09, which was correctly labelled as being significant at the 90% confidence level.

Table VI: Deworming externalities within schools, January to March 1999. This table combined several results: baseline comparisons between Groups 1 and 2, comparison of parasite burden in Group 1 pupils in 1998 and 1999, comparison of parasite burden between Groups 1 and 2 in 1999 and comparison of school participation rates in Groups 1 and 2 in the first year of the study. No major discrepancies were found.

Table VII: Deworming health externalities within and across schools, January to March 1999. This table described the indirect-within-school and the indirect-between-school benefits of deworming treatment on worm infection rates. In text between tables VII and VIII, data from Table VII was used to calculate the indirect-between-school benefit and also the total effect of the study on deworming. A major discrepancy was identified that occurred as a result of coding errors #1 and #2 described above. A further coding error was also present (#3, relating to combining types of geohelminth infection) but had little effect on interpretation. Having corrected these errors, re-analysis found no statistically significant indirect-between-school effect on the worm infection outcome, according to the analysis methods originally used. However, among variables used to construct this effect, a parameter describing the effect of Group 1 living within 0–3 km did remain significant, albeit at a slightly smaller size (original −0.26, SE 0.09, significant at 95% confidence level; updated -0.21, SE 0.10, significant at 95% confidence). The corresponding parameter for the 3–6-km distances became much smaller and statistically insignificant (original -0.14, SE 0.06, significant at 90% confidence; updated -0.05, SE 0.08, not statistically significant).

Table VIII: School participation, school-level data. This table described average school attendance proportions for a wide range of different categories in 1998 and 1999. This table was affected by coding errors #1 and #2 and also the accidental use of a preliminary version of a data table, but these did not lead to any major discrepancies. Both Table VIII and Table IX included weighted regression analyses: these were inaccurately described as being weighted by number of pupils, whereas these were actually weighted by numbers of pupil-observations. We note that for the purposes of school attendance outcomes, year 1 is treated as May 1998 to March 1999, and year 2 as March to November 1999. The effect of the specifications of year 1 and 2, and the potential effect of weighting by pupil-observations, are explored in a companion paper.

Table IX: School participation, direct Effects and externalities. This table described the major results of the analysis on school attendance, in the form of parameter estimates from different regressions. This table included four results with inaccurately labelled levels of significance, though none of these influenced the overall interpretation. A further coding error (#4, see Table 2) led to use of the wrong variable being used in a calculation of local populations around schools. In regression (3), variables describing the effects of treatment school pupils at between 0–3 km and 3–6 km were reported, which were subsequently amongst the variables used for construction of the between-school-indirect effect on school attendance. For the former distance, the effect declined from 0.044 (SE 0.022, significant at 95% confidence level) to 0.040 (SE 0.022, significant at 90% confidence level). For the latter distance, the effect remained slightly negative and not statistically significant. In text between Tables IX and X, parameter estimates from Table IX were used for further calculations to determine effects on school attendance, both the indirect-between-school effect and the total effect on treated pupils. Correction of all coding errors in Table IX thus led to the major discrepancies shown in Table 3. The indirect-between-school effect was substantially reduced (from +2.0% to −1.7%) with an increased standard error (from 1.3% to 3.0%) making the result non-significant. The total effect on school attendance was also substantially reduced (from 7.5% to 3.9% absolute improvement), making it only slightly more than one standard error interval away zero, hence also non-significant.

Table 3.

Comparison of original and re-analysis results on school attendance from table IX

Effects on school attendance, 1998 + 99	Absolute improvement in school attendance (%)
	Original result	Standard error	Result from fully corrected re-analysis	Standard error
‘Naïve’ effect	5.1	2.2	5.7	1.4
Indirect-between-school effect	2.0	1.3	−1.7	3.0
Total effect on treated pupils	7.5	2.7	3.9	3.2

Table X: Academic examinations, individual-level data. This table described the major results of the analysis on examination performance, in the form of regression parameters. The effects of the intervention on performance were obtained from regression (2) in this table. There was no change in the interpretation of these findings arising from correction of coding errors, as shown in Table 4.

Table 4.

Comparison of original and re-analysis results on examination performance from Table X

Total effect on exam performance	Change in exam performance (standard deviations)
	Original result	Standard error	Result from fully corrected re-analysis	Standard error
1998, Year 1	−0.032	0.046	−0.035	0.047
1999, Year 2	0.001	0.073	−0.015	0.079

Two other notable issues were identified in this re-analysis that represented decisions on data analysis and presentation that could influence the interpretation of the results.

Randomization procedure

The method of allocation of schools to different treatment groups (alternating assignment from a sorted list) does not meet a strict definition of randomization—this would be more accurately described as systematic allocation resulting in ‘quasi-randomization’. Comparison of variables at baseline did suggest that this process achieved three broadly similar groups at baseline, though the differential extent of missing data makes this hard to confirm.

Presentation of missing data

Throughout the original paper, there is limited description of the extent of missing data, especially for baseline parameters presented in Table I. In this table, there is in fact a large amount of missing data for year of birth—this information is missing for 17% of children in Group 1, 19% of children in Group 2 and 31% of children in Group 3. The extent of these missing data is not described in the table or the accompanying text. From the children whose age is known, there are highly significant between-group differences in average age, suggesting a possible imbalance at baseline. It is impossible to determine retrospectively whether this is due to systematic differences in data collection, or due to an actual difference in the average age.

Discussion

The original study re-analysed here was a large, innovative and highly influential trial that focused attention on an important question in global health—what are the impacts of delivering deworming treatment to children through school-based programmes? In this pure replication, we re-analysed the trial data using the methods used by the original authors. We identified discrepancies with the original results. Many were trivial, others more substantial. Our most important finding was that after correction of coding errors in the original authors’ analysis files, there was little evidence for previously described ‘positive externalities’ (or indirect effects) from the deworming intervention on school attendance in untreated schools.

Our findings from a fully corrected re-analysis according to the authors’ original methods are as follows: we found beneficial effects on worm infections similar to or greater than those originally reported, although the indirect-between-school effect was more modest than previously described and was not statistically significant. In contrast to the original study, we found limited evidence of non-worm-related health benefits as the prevalence of anaemia was not significantly affected by the intervention. The improvement in height-for-age z-score (HAZ) remained of modest size (0.08 standard deviations) and significant at the 90% level. Given that anaemia and nutritional status are plausible biological mediating factors between reduced worm infections and improvements in school attendance, the weakening of evidence for these effects raises the question of causality in this study.

We found beneficial effects on school attendance similar to or greater than those originally reported for the direct, indirect-within-school and ‘naïve’ effects. However, after correcting coding errors in the original analysis, other measurements of effect size were substantially different. These results related to ‘externalities’, hypothesized indirect effects of deworming occurring in all schools, a key focus of the original paper. In corrected re-analysis, the indirect-between-school effect on school attendance had shifted in direction and was less precisely estimated—there was now little evidence for an effect of this kind in the format of analysis originally employed. We have not re-examined for evidence of indirect-between-school effect at a distance other than that used in original paper (up to 6 km from schools) as this would deviate from our stated pre-analytical plan. We do note that some parameters suggest effects may be present at distances of up to 3 km. It remains unclear how the distance intervals used for these spatial effects in the original paper were decided upon. The ‘total effect’ on school attendance resulting from the intervention described by the original authors, a combination of the ‘naïve’ and indirect-between-school effects, was more modest and less precisely estimated than previously reported and was also not statistically significant. This counter-intuitive finding—strong evidence for a ‘naïve’ effect but no evidence of a total effect-derives from the additive logic used by the original authors to calculate the total effect result and the reversal in direction of the indirect-between-school effect. As in the original study, our re-analysis showed no improvement in examination performance associated with the intervention. Other discrepancies were detected in the results originally presented, but these did not appear systematic and did not affect the major findings of the study.

The language, layout and various other features of the analysis and presentation in the original report represent disciplinary conventions within econometrics in 2004 rather than public health in 2015. Though unfamiliar to readers of epidemiological journals, these differences do not in themselves affect the interpretation of the results. However, evaluation of randomization processes and the handling of missing data are considered to be highly important in the analysis of randomized trials according to the CONSORT framework. In a separate companion paper, we applied alternative methodological approaches to dealing with these, and other, analytical issues, focusing on the ‘naïve’ results as described in this paper, and conclude that there is some evidence of a ‘naïve’ effect on school attendance, but with high risk of bias. Taken together, these re-analyses suggest only limited evidence for direct or indirect benefits of school-based deworming on school attendance from these data.

Supplementary Data

Supplementary data are available at IJE online. The pre-analysis plan for this reanalysis can be found at http://www.3ieimpact.org/media/filer/2013/05/14/aiken_replication_plan_final.pdf.

Funding

This work was supported by the International Initiative for Impact Evaluation (3ie) as part of the first round of their Replication Programme [http://www.3ieimpact.org/en/funding/replication-window/]. The aim of this programme is to improve the quality of evidence for development policy by re-appraising a wide range of influential studies in the development field, seeking to verify and examine the robustness of the original findings in these studies. The authors of this re-analysis report have conducted this work as consultants for 3ie. The funders have had no role in design, conduct or reporting of this re-analysis, other than facilitating contact with the original study authors. Open access publication fees were paid by the Centre for Evaluation in LSHTM.

Table 5.

Summary of findings from re-analysis, as based on authors’ original approaches

Effects that were beneficial and significant according to the approaches of the original authors are shaded.

SD, standard deviation; SE, standard error; Hb, haemoglobin; WAZ, weight-for-age z-score; HAZ, height-for-age z-score; mod, moderate; hvy, heavy; inf, infection; prop’n, proportion.

†Year 1 data only.