When is systematic review replication useful, and when is it wasteful?

To the Editor,

During the past quarter of a century and following the global explosion in access to scientific literature, systematic reviews have become an excellent way to inform decisions with summaries of the effects of interventions, and to learn about their impact under different conditions or among populations. Systematic reviews play a key role in ensuring that policy decisions are informed by research. Local and national governments as well as international health organizations now regularly commission and request that systematic reviews inform policy decisions.

In recent years, there has been a proliferation of systematic reviews –with multiple systematic reviews addressing the same question, often without any justification. While there has been a strong call to control the unnecessary duplication of systematic reviews and the confusion and waste this generates, there is also a growing recognition that there can be valid reasons for replicating systematic reviews.

Replication can serve to increase certainty in findings. For example, given the high stakes of the topic of antibiotics for food animals, the World Health Organization (WHO) commissioned two independent systematic reviews to determine the certainty of the conclusions (1). Systematic review replication can also be conducted as a purposeful repetition to determine if the conclusions of an existing review hold when perceived methodological limitations are addressed. Controversies surrounding a Cochrane systematic review of prophylactic human papillomavirus (HPV) vaccines relating mainly to missing trials (2), for instance, led to a replication adding data from unpublished reports, and found no impact on the direction or effect for the reported outcomes (3). Replication can also serve to purposefully broaden or narrow the research question in existing reviews, particularly for questions that may address sizable benefits or harms. When a Cochrane review on mass deworming was criticized for not including long-term educational and labour outcomes from non-randomized studies, a Campbell replication review extended the scope of the original review and confirmed its findings (4, 5). These examples highlight the potential value of planned and purposeful systematic review replication.

On the other hand, it is impractical and wasteful to replicate every systematic review of a policy or practice relevant question and it is important to determine when such an effort should be prioritised. Under the circumstances of the SARS-CoV2 pandemic there have been a number of questions with multiple reviews that have no added value, and thus increase research waste (e.g., there are numerous systematic reviews on pharmaceutical therapies found to not be beneficial, to cause harm, or to not be cost-effective; for example several dozens of systematic reviews looking into the effects of hydroxychloroquine in people with SARS-CoV2 add little value to what is known), while some basic questions remain unaddressed (what protections work best for different populations and areas of work, physical distancing, effectiveness of building measures to reduce the incidence of SARS-CoV2; how best to reopen schools; strategies to enable teleworking and tele-schooling in high vulnerability populations, effectiveness of interventions in different groups when disaggregated by gender or ethnicity, equity) (6).

These examples highlight how important it is to have clear guidance on when it may be appropriate to replicate a systematic review. Members of this author team in collaboration with a global contributors recently published a checklist that provides guidance on when to replicate systematic reviews of interventions (7). The 4-item checklist was developed using consensus building methods and involving an international multidisciplinary group of over 50 methodologists and knowledge users. This checklist helps guide the decision to replicate a systematic review by addressing 4 main domains: if replication is a priority for decision makers; if the replication will likely reduce uncertainties; if the effects (balance between benefits and harms) of the replication are substantial; if the replication has an added value over conducting alternative studies. In addition to enhancing coordination and research governance, having a checklist with explicit criteria for when to replicate and when not to, can help increase the value of research and tackle research waste. The checklist can serve as an explicit prompt to consider the value of systematic review replication, while limiting waste resulting from unnecessary duplication.

Disclaimer.

Authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the RPSP/PAJPH and/or PAHO.