Jeffrey C Valentine1, Simon G Thompson2. 1. College of Education and Human Development, University of Louisville, Louisville, KY, U.S.A. 2. Department of Public Health, University of Cambridge, Cambridge, U.K.
Abstract
BACKGROUND: Confounding caused by selection bias is often a key difference between non-randomized studies (NRS) and randomized controlled trials (RCTs) of interventions. KEY METHODOLOGICAL ISSUES: In this third paper of the series, we consider issues relating to the inclusion of NRS in systematic reviews on the effects of interventions. We discuss whether potential biases from confounding in NRS can be accounted for, the limitations of current methods for attempting to do so, the different contexts of NRS and RCTs, the problems these issues create for reviewers, and a research agenda for the future. GUIDANCE: Reviewers who are considering whether or not to include NRS in meta-analyses must weigh a number of factors. Including NRS may allow a review to address outcomes or pragmatic implementations of an intervention not studied in RCTs, but it will also increase the workload for the review team, as well as their required technical repertoire. Furthermore, the results of a synthesis involving NRS will likely be more difficult to interpret, and less certain, relative to the results of a synthesis involving only randomized studies. When both randomized and non-randomized evidence are available, we favor a strategy of including NRS and RCTs in the same systematic review but synthesizing their results separately. CONCLUSION: Including NRS will often make the limitations of the evidence derived from RCTs more apparent, thereby guiding inferences about generalizability, and may help with the design of the next generation of RCTs.
BACKGROUND: Confounding caused by selection bias is often a key difference between non-randomized studies (NRS) and randomized controlled trials (RCTs) of interventions. KEY METHODOLOGICAL ISSUES: In this third paper of the series, we consider issues relating to the inclusion of NRS in systematic reviews on the effects of interventions. We discuss whether potential biases from confounding in NRS can be accounted for, the limitations of current methods for attempting to do so, the different contexts of NRS and RCTs, the problems these issues create for reviewers, and a research agenda for the future. GUIDANCE: Reviewers who are considering whether or not to include NRS in meta-analyses must weigh a number of factors. Including NRS may allow a review to address outcomes or pragmatic implementations of an intervention not studied in RCTs, but it will also increase the workload for the review team, as well as their required technical repertoire. Furthermore, the results of a synthesis involving NRS will likely be more difficult to interpret, and less certain, relative to the results of a synthesis involving only randomized studies. When both randomized and non-randomized evidence are available, we favor a strategy of including NRS and RCTs in the same systematic review but synthesizing their results separately. CONCLUSION: Including NRS will often make the limitations of the evidence derived from RCTs more apparent, thereby guiding inferences about generalizability, and may help with the design of the next generation of RCTs.
Authors: Sebastian E Baumeister; Michael F Leitzmann; Jakob Linseisen; Sabrina Schlesinger Journal: J Natl Cancer Inst Date: 2019-11-01 Impact factor: 13.506
Authors: David Burnes; Christine Sheppard; Charles R Henderson; Monica Wassel; Richenda Cope; Chantal Barber; Karl Pillemer Journal: Am J Public Health Date: 2019-06-20 Impact factor: 9.308
Authors: Julie M Petersen; Malcolm Barrett; Katherine A Ahrens; Eleanor J Murray; Allison S Bryant; Carol J Hogue; Sunni L Mumford; Salini Gadupudi; Matthew P Fox; Ludovic Trinquart Journal: Res Synth Methods Date: 2022-01-05 Impact factor: 9.308
Authors: Thomas P A Debray; Karel G M Moons; Gert van Valkenhoef; Orestis Efthimiou; Noemi Hummel; Rolf H H Groenwold; Johannes B Reitsma Journal: Res Synth Methods Date: 2015-08-19 Impact factor: 5.273