Mark E Glickman1, Sowmya R Rao2, Mark R Schultz3. 1. Center for Health care Organization and Implementation Research, Bedford VA Medical Center, 200 Springs Road (152), Bedford, MA 01730, USA; Department of Health Policy and Management, Boston University School of Public Health, 715 Albany Street, Talbot Building, Boston, MA 02118, USA. Electronic address: mg@bu.edu. 2. Center for Health care Organization and Implementation Research, Bedford VA Medical Center, 200 Springs Road (152), Bedford, MA 01730, USA; Department of Quantitative Health Sciences, University of Massachusetts Medical School, 55 N. Lake Avenue, Worcester, MA 01655, USA. 3. Center for Health care Organization and Implementation Research, Bedford VA Medical Center, 200 Springs Road (152), Bedford, MA 01730, USA.
Abstract
OBJECTIVES: Procedures for controlling the false positive rate when performing many hypothesis tests are commonplace in health and medical studies. Such procedures, most notably the Bonferroni adjustment, suffer from the problem that error rate control cannot be localized to individual tests, and that these procedures do not distinguish between exploratory and/or data-driven testing vs. hypothesis-driven testing. Instead, procedures derived from limiting false discovery rates may be a more appealing method to control error rates in multiple tests. STUDY DESIGN AND SETTING: Controlling the false positive rate can lead to philosophical inconsistencies that can negatively impact the practice of reporting statistically significant findings. We demonstrate that the false discovery rate approach can overcome these inconsistencies and illustrate its benefit through an application to two recent health studies. RESULTS: The false discovery rate approach is more powerful than methods like the Bonferroni procedure that control false positive rates. Controlling the false discovery rate in a study that arguably consisted of scientifically driven hypotheses found nearly as many significant results as without any adjustment, whereas the Bonferroni procedure found no significant results. CONCLUSION: Although still unfamiliar to many health researchers, the use of false discovery rate control in the context of multiple testing can provide a solid basis for drawing conclusions about statistical significance.
OBJECTIVES: Procedures for controlling the false positive rate when performing many hypothesis tests are commonplace in health and medical studies. Such procedures, most notably the Bonferroni adjustment, suffer from the problem that error rate control cannot be localized to individual tests, and that these procedures do not distinguish between exploratory and/or data-driven testing vs. hypothesis-driven testing. Instead, procedures derived from limiting false discovery rates may be a more appealing method to control error rates in multiple tests. STUDY DESIGN AND SETTING: Controlling the false positive rate can lead to philosophical inconsistencies that can negatively impact the practice of reporting statistically significant findings. We demonstrate that the false discovery rate approach can overcome these inconsistencies and illustrate its benefit through an application to two recent health studies. RESULTS: The false discovery rate approach is more powerful than methods like the Bonferroni procedure that control false positive rates. Controlling the false discovery rate in a study that arguably consisted of scientifically driven hypotheses found nearly as many significant results as without any adjustment, whereas the Bonferroni procedure found no significant results. CONCLUSION: Although still unfamiliar to many health researchers, the use of false discovery rate control in the context of multiple testing can provide a solid basis for drawing conclusions about statistical significance.
Authors: Richa Gawande; My Ngoc To; Elizabeth Pine; Todd Griswold; Timothy B Creedon; Alexandra Brunel; Angela Lozada; Eric B Loucks; Zev Schuman-Olivier Journal: J Gen Intern Med Date: 2018-12-03 Impact factor: 5.128
Authors: Neal B Shah; Andrew S Allegretti; Sagar U Nigwekar; Sahir Kalim; Sophia Zhao; Benjamin Lelouvier; Florence Servant; Gloria Serena; Ravi Ishwar Thadhani; Dominic S Raj; Alessio Fasano Journal: Clin J Am Soc Nephrol Date: 2019-04-08 Impact factor: 8.237
Authors: Jonathan A Mitchell; Alessandra Chesi; Okan Elci; Shana E McCormack; Heidi J Kalkwarf; Joan M Lappe; Vicente Gilsanz; Sharon E Oberfield; John A Shepherd; Andrea Kelly; Babette S Zemel; Struan F A Grant Journal: J Bone Miner Res Date: 2015-05-26 Impact factor: 6.741
Authors: Devika R Jutagir; Bonnie B Blomberg; Charles S Carver; Suzanne C Lechner; Kiara R Timpano; Laura C Bouchard; Lisa M Gudenkauf; Jamie M Jacobs; Alain Diaz; Susan K Lutgendorf; Steve W Cole; Aaron S Heller; Michael H Antoni Journal: Breast Cancer Res Treat Date: 2017-05-30 Impact factor: 4.872
Authors: Eli Puterman; Alison Gemmill; Deborah Karasek; David Weir; Nancy E Adler; Aric A Prather; Elissa S Epel Journal: Proc Natl Acad Sci U S A Date: 2016-10-03 Impact factor: 11.205
Authors: Bradley J Hindman; Robert P From; Ricardo B Fontes; Vincent C Traynelis; Michael M Todd; M Bridget Zimmerman; Christian M Puttlitz; Brandon G Santoni Journal: Anesthesiology Date: 2015-11 Impact factor: 7.892