Commentary: Replication of influential trial helps international policy.

The World Bank, the Gates Foundation and the World Health Organization promote national child deworming programmes in developing countries. They assert these programmes will improve nutritional status, health and school performance, and hence contribute to economic growth. Indeed, the World Health Organization states that deworming contributed to Japan’s economic boom in the 1950s, and Nobel Laureates meeting in Copenhagen ranked deworming as the fourth most important intervention to solve the health problems of the whole world.

Surprisingly, the evidence base for these claims from controlled studies is limited. Critically, according to the Cochrane review which two of us author, there is quite good evidence of no effect for the main biomedical outcomes in deworming, making the broader societal benefits on economic development barely credible (Figure 1). Nevertheless, the advocates increasingly rely on a single large quasi-randomized trial carried out in Kenya, published in 2004 in Econometrica, which reports school attendance.

Figure 1.

Conceptual framework used to underpin the effects of deworming programmes on economic growth in developing countries. Contemporary evidence questions this model, as there is now quite good contemporary evidence, summarized in the Cochrane review, that these programmes have little or no effect on the main effects or mediating pathways (reproduced with permission).

This study has been highly influential. The International Initiative for Impact Evaluation commissioned the London School of Hygiene and Tropical Medicine (LSHTM) to replicate the analysis, as the original analysis is ‘based on econometric approaches and used a language and format that would be unfamiliar to many health care researchers’. The replication aimed to provide detail of the methods and reporting in line with the CONSORT statement. The team are internationally recognized, independent and meticulous in their approach. They agreed a protocol, carefully checked and corrected the raw data, and then re-ran their prespecified analysis.

Their first paper is a pure replication, exactly repeating the authors’ original analysis. This paper clarifies some methodological details not provided in the original paper, but it also uncovers a series of important coding and analysis errors. Some of the corrected results are consistent with the original findings, but others are quite different. Most notably, the much quoted ‘positive externalities’—where the benefits of treating children in one school ‘spill over’ to benefit children in adjacent schools—vanish in their corrected analysis.

Their second paper uses approaches more familiar to epidemiologists, and allows a more thorough exploration of the data. There are substantial amounts of missing information, and some unexpected patterns that are difficult to explain. For example, there is a correlation between the number of observations in each school and the reported attendance, with more observations associated with lower attendance reported—except in some of the intervention groups, where more frequent observation is associated with better attendance. This raises the possibility that the process of observation influenced outcome reporting and this was different in control and intervention groups.

The second paper confirms an association with higher attendance in the deworming schools when stratified by year, but found the combined estimate across years was much larger, and the authors caution the validity of the combined year estimate: this may be due to the stepped-wedge design where the combined year estimate includes a before and after comparison; or it may be due to the problems with secular and group effects of the number of observations of school attendance varying. To further complicate matters, there was also a concurrent ‘School Assistance Program’ (SAP) evaluating five other interventions in 27/75 of the study schools, which was not part of the randomized intervention. Attendance patterns differed on the basis of involvement in the SAP, which the authors suggest could well lead to bias.

One of the critical points the LSHTM papers make is that the administration of the deworming drugs was part of a ‘complex health education and drug treatment intervention’. The intervention schools also received regular public health lectures, wall charts and teacher training on worm prevention. Interestingly, the pure replication shows very similar impacts of the intervention on children in the intervention schools irrespective of whether they took the deworming medicine or not. For Miguel and colleagues, they term this as ‘indirect within-school effects’ of deworming, but equally the effects could be due to the non-specific effects of the health promotion interventions in the intervention schools.

So where does this take us? There is a small effect on school attendance for sure, but this should not be over-interpreted, as the LSHTM authors say, given the concerns raised regarding the risk of bias, and the independent effects on school attendance of the programme of health promotion. In addition, the study shows that allocation to the deworming group does not influence student progress, as the re-analysis shows clearly no evidence of a difference in examination performance between deworming and control groups.,

In the context of the global evidence base, there is only one other study that has examined school attendance with no obvious effect shown. Given all these uncertainties, we simply don’t know if there is truly an effect on school attendance from the data, in taking all these factors into account formally in a GRADE analysis. But what weakens the case of deworming still further is that there is now quite good evidence of no effect for most of the main outcomes (Figure 1), including nutritional status, haemoglobin, cognition and school performance. This is important because without these effects it seems implausible that deworming itself would have an independent effect on school attendance or economic development.

Obviously children infected with worms should be treated. Trials from over 20 years ago in an area of Kenya where all the children were heavily infected showed benefit. Another trial in India showed benefit, but subsequent trials in the same area failed to show an effect., So deworming may have helped in these exceptional, heavily infected, untreated populations from another decade, but this is scarcely a solid base for contemporary policy: public health nutrition has changed, worm burden has declined and this probably accounts for the lack of effect on biomedical outcomes in contemporary studies.

We have been perplexed by the unquestioning belief behind deworming in the advocates, and have found it is deep rooted in American history. The Rockefeller Sanitary Commission in 1909 sought to eradicate hookworm in the Southern USA population, where it was seen as a cause of ‘some of the proverbial laziness of the poorer classes’ and to improve worker productivity; these were extended with the Rockefeller International Health Commission established in 1913. The beliefs and assumptions appear to continue, with ‘Deworm the World’ aiming to deworm 220 million children in India during 2015.

What have we learned from this? Certainly that replication is a valuable process, not least for errors that have the potential to mislead, but also to provide a much better interpretation of the potential biases in studies. In terms of policies in deworming, it suggests donors and Nobel Prize winners need to be aware of their prior assumptions, and base decisions on the all the evidence from reliable studies within a systematic review, not just select a single study. They need to look at all relevant outcomes, not just one. They need to consider bias and confounding. Our view is that current promotion of community deworming is certainly a panacea: a single solution to multiple problems in low- and middle-income countries, and that the belief that deworming will impact substantially on economic development seems delusional when you look at the results of reliable controlled trials. This is a view, based on over 15 years of engagement with critically appraising the literature in this field.

We recommend anyone responsible for public or philanthropic money going to large scale deworming to read these replication studies, the original publication, the authors’ responses to the replication and the 2015 version of the Cochrane review that includes corrections from these replications plus data from five further studies, including the recently published DEVTA study of 2 million children., Then make up your own mind.

Conflict of interest: PG and DTR are authors of a Cochrane review of community deworming programmes. PG is funded by the Department for International Development to provide reliable independent systematic reviews for the benefit of developing countries; PG receives salary support from Countdown, a DFID-supported research grant in programmatic implementation of neglected tropical disease programmes.