Ewout W Steyerberg1, Hajime Uno2, John P A Ioannidis3, Ben van Calster4. 1. Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands; Department of Public Health, Erasmus MC, Rotterdam, The Netherlands. Electronic address: e.w.steyerberg@lumc.nl. 2. Division of Population Sciences, Dana-Farber Cancer Institute, 02215 MA, Boston, USA. 3. Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA. 4. Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands; Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
Abstract
OBJECTIVE: To evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies. STUDY DESIGN AND SETTING: A previously published model predicted the likelihood of having a mutation in germline DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of over 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to bias and discriminative ability (concordance index, c) in independent data. RESULTS: We found over 50% bias for five of six originally selected predictors, unstable model specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model prespecification based on external knowledge and using continuous predictors led to better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively). CONCLUSION: Prediction models perform poorly if based on small numbers of events and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes.
OBJECTIVE: To evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies. STUDY DESIGN AND SETTING: A previously published model predicted the likelihood of having a mutation in germline DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of over 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to bias and discriminative ability (concordance index, c) in independent data. RESULTS: We found over 50% bias for five of six originally selected predictors, unstable model specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model prespecification based on external knowledge and using continuous predictors led to better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively). CONCLUSION: Prediction models perform poorly if based on small numbers of events and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes.
Authors: H G Ternavasio-de la Vega; F Castaño-Romero; S Ragozzino; R Sánchez González; M P Vaquero-Herrero; M Siller-Ruiz; G Spalter-Glicberg; C Ramírez-Baum; S Rodríguez-Rodríguez; J E García-Sánchez; I García-García; M Marcos Journal: Epidemiol Infect Date: 2018-09-03 Impact factor: 4.434
Authors: Nini H Jonkman; Marco Colpo; Jochen Klenk; Chris Todd; Trynke Hoekstra; Vieri Del Panta; Kilian Rapp; Natasja M van Schoor; Stefania Bandinelli; Martijn W Heymans; Dominique Mauger; Luca Cattelani; Michael D Denkinger; Dietrich Rothenbacher; Jorunn L Helbostad; Beatrix Vereijken; Andrea B Maier; Mirjam Pijnappels Journal: BMC Geriatr Date: 2019-06-27 Impact factor: 3.921