Matt Vassar1, Austin L Johnson2, Adriana Sharp3, Cole Wayant4. 1. matt.vassar@okstate.edu, Clinical Assistant Professor of Psychiatry and Behavioral Sciences, Department of Psychiatry and Behavioral Sciences, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK 74107. 2. austin.johnson14@okstate.edu, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK 74107. 3. adriana.sharp@okstate.edu, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK 74107. 4. cole.wayant@okstate.edu, Office of Medical Student Research and Department of Biomedical Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK 74107.
Abstract
OBJECTIVE: Reproducibility of systemic reviews (SRs) can be hindered by the presence of citation bias. Citation bias may occur when authors of SRs conduct hand-searches of included study reference lists to identify additional studies. Such a practice may lead to exaggerated SR summary effects. The purpose of this paper is to examine the prevalence of hand-searching reference lists in otolaryngology SRs. METHODS: The authors searched for systematic reviews published in eight clinical otolaryngology journals using the Cochrane Library and PubMed, with the date parameter of January 1, 2008, to December 31, 2017. Two independent authors worked separately to extract data from each SR for the following elements: whether reference lists were hand-searched, other kinds of supplemental searching, PRISMA adherence, and funding source. Following extraction, the investigators met to review discrepancies and achieve consensus. RESULTS: A total of 539 systemic reviews, 502 from clinical journals and 37 from the Cochrane library, were identified. Of those SRs, 72.4% (390/539) hand-searched reference lists, including 97.3% (36/37) of Cochrane reviews. For 228 (58.5%) of the SRs that hand-searched reference lists, no other supplemental search (e.g., search of trial registries) was conducted. CONCLUSIONS: These findings indicate that hand-searching reference lists is a common practice in otolaryngology SRs. Moreover, a majority of studies at risk of citation bias did not attempt to mitigate the bias by conducting additional supplemental searches. The implication is that summary effects in otolaryngology systematic reviews may be biased toward statistically significant findings.
OBJECTIVE: Reproducibility of systemic reviews (SRs) can be hindered by the presence of citation bias. Citation bias may occur when authors of SRs conduct hand-searches of included study reference lists to identify additional studies. Such a practice may lead to exaggerated SR summary effects. The purpose of this paper is to examine the prevalence of hand-searching reference lists in otolaryngology SRs. METHODS: The authors searched for systematic reviews published in eight clinical otolaryngology journals using the Cochrane Library and PubMed, with the date parameter of January 1, 2008, to December 31, 2017. Two independent authors worked separately to extract data from each SR for the following elements: whether reference lists were hand-searched, other kinds of supplemental searching, PRISMA adherence, and funding source. Following extraction, the investigators met to review discrepancies and achieve consensus. RESULTS: A total of 539 systemic reviews, 502 from clinical journals and 37 from the Cochrane library, were identified. Of those SRs, 72.4% (390/539) hand-searched reference lists, including 97.3% (36/37) of Cochrane reviews. For 228 (58.5%) of the SRs that hand-searched reference lists, no other supplemental search (e.g., search of trial registries) was conducted. CONCLUSIONS: These findings indicate that hand-searching reference lists is a common practice in otolaryngology SRs. Moreover, a majority of studies at risk of citation bias did not attempt to mitigate the bias by conducting additional supplemental searches. The implication is that summary effects in otolaryngology systematic reviews may be biased toward statistically significant findings.
Authors: Gordon Guyatt; Andrew D Oxman; Elie A Akl; Regina Kunz; Gunn Vist; Jan Brozek; Susan Norris; Yngve Falck-Ytter; Paul Glasziou; Hans DeBeer; Roman Jaeschke; David Rind; Joerg Meerpohl; Philipp Dahm; Holger J Schünemann Journal: J Clin Epidemiol Date: 2010-12-31 Impact factor: 6.437
Authors: Judith Ac Rietjens; Wichor M Bramer; Eric Ct Geijteman; Agnes van der Heide; Wendy H Oldenmenger Journal: Palliat Med Date: 2019-01-28 Impact factor: 4.762