Matthew F Buas1, Christopher I Li2, Garnet L Anderson3, Margaret S Pepe4. 1. Department of Cancer Prevention and Control, Roswell Park Comprehensive Cancer Center, Buffalo, NY, United States; Epidemiology Program, Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States. Electronic address: matthew.buas@roswellpark.org. 2. Translational Research Program, Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States. 3. Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States. 4. Biostatiistics and Biomathematics Program, Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States.
Abstract
BACKGROUND: Biomarker candidates are often ranked using P-values. Standard P-value calculations use normal or logit-normal approximations, which may not be correct for small P-values and small sample sizes common in discovery research. METHODS: We compared exact P-values, correct by definition, with logit-normal approximations in a simulated study of 40 cases and 160 controls. The key measure of biomarker performance was sensitivity at 90% specificity. Data for 3000 uninformative false markers and 30 informative true markers were generated randomly. We also analyzed real data for 2371 plasma protein markers measured in 121 breast cancer cases and 121 controls. RESULTS: In our simulation, using the same discovery criterion, exact P-values led to discovery of 24 true and 82 false biomarkers, while logit-normal approximate P-values yielded 20 true and 106 false biomarkers. The estimated true discovery rate was substantially off for approximate P-values: logit-normal estimated 42 but found 20. The exact method estimated 22, very close to 24, which was the actual number of true discoveries. Although these results are based on one specific simulation, qualitatively similar results were obtained from 10 random repetitions. With real data, ranking candidate biomarkers by exact P-values, versus approximate P-values, resulted in a very different ordering of these markers. CONCLUSIONS: Exact P-values, which correspond to permutation tests with non-parametric rank statistics such as empirical ROC statistics, are preferred over approximate P-values. Approximate P-values can lead to inappropriate biomarker selection rules and incorrect conclusions. IMPACT: Exact P-values in place of approximate P-values in discovery research may improve the yield of biomarkers that validate clinically.
BACKGROUND: Biomarker candidates are often ranked using P-values. Standard P-value calculations use normal or logit-normal approximations, which may not be correct for small P-values and small sample sizes common in discovery research. METHODS: We compared exact P-values, correct by definition, with logit-normal approximations in a simulated study of 40 cases and 160 controls. The key measure of biomarker performance was sensitivity at 90% specificity. Data for 3000 uninformative false markers and 30 informative true markers were generated randomly. We also analyzed real data for 2371 plasma protein markers measured in 121 breast cancer cases and 121 controls. RESULTS: In our simulation, using the same discovery criterion, exact P-values led to discovery of 24 true and 82 false biomarkers, while logit-normal approximate P-values yielded 20 true and 106 false biomarkers. The estimated true discovery rate was substantially off for approximate P-values: logit-normal estimated 42 but found 20. The exact method estimated 22, very close to 24, which was the actual number of true discoveries. Although these results are based on one specific simulation, qualitatively similar results were obtained from 10 random repetitions. With real data, ranking candidate biomarkers by exact P-values, versus approximate P-values, resulted in a very different ordering of these markers. CONCLUSIONS: Exact P-values, which correspond to permutation tests with non-parametric rank statistics such as empirical ROC statistics, are preferred over approximate P-values. Approximate P-values can lead to inappropriate biomarker selection rules and incorrect conclusions. IMPACT: Exact P-values in place of approximate P-values in discovery research may improve the yield of biomarkers that validate clinically.
Authors: Matthew F Buas; Jung-hyun Rho; Xiaoyu Chai; Yuzheng Zhang; Paul D Lampe; Christopher I Li Journal: Breast Cancer Res Treat Date: 2015-08-30 Impact factor: 4.872
Authors: Zobeida Cruz-Monserrate; Kristyn Gumpper; Valentina Pita; Phil A Hart; Christopher Forsmark; David C Whitcomb; Dhiraj Yadav; Richard T Waldron; Stephen Pandol; Hanno Steen; Vincent Anani; Natasha Kanwar; Santhi Swaroop Vege; Savi Appana; Liang Li; Jose Serrano; Jo Ann S Rinaudo; Mark Topazian; Darwin L Conwell Journal: Pancreatology Date: 2021-01-22 Impact factor: 3.996