R Pajouheshnia1, M van Smeden2, L M Peelen3, R H H Groenwold2. 1. UMC Utrecht Julius Center, Utrecht University, Utrecht, The Netherlands. Electronic address: R.Pajouheshnia@umcutrecht.nl. 2. Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, The Netherlands. 3. UMC Utrecht Julius Center, Utrecht University, Utrecht, The Netherlands.
Abstract
BACKGROUND AND OBJECTIVE: Diagnostic and prognostic prediction models often perform poorly when externally validated. We investigate how differences in the measurement of predictors across settings affect the discriminative power and transportability of a prediction model. METHODS: Differences in predictor measurement between data sets can be described formally using a measurement error taxonomy. Using this taxonomy, we derive an expression relating variation in the measurement of a continuous predictor to the area under the receiver operating characteristic curve (AUC) of a logistic regression prediction model. This expression is used to demonstrate how variation in measurements across settings affects the out-of-sample discriminative ability of a prediction model. We illustrate these findings with a diagnostic prediction model using example data of patients suspected of having deep venous thrombosis. RESULTS: When a predictor, such as D-dimer, is measured with more noise in one setting compared to another, which we conceptualize as a difference in "classical" measurement error, the expected value of the AUC decreases. In contrast, constant, "structural" measurement error does not impact on the AUC of a logistic regression model, provided the magnitude of the error is the same among cases and noncases. As the differences in measurement methods between settings (and in turn differences in measurement error structures) become more complex, it becomes increasingly difficult to predict how the AUC will differ between settings. CONCLUSION: When a prediction model is applied to a different setting to the one in which it was developed, its discriminative ability can decrease or even increase if the magnitude or structure of the errors in predictor measurements differ between the two settings. This provides an important starting point for researchers to better understand how differences in measurement methods can affect the performance of a prediction model when externally validating or implementing it in practice.
BACKGROUND AND OBJECTIVE: Diagnostic and prognostic prediction models often perform poorly when externally validated. We investigate how differences in the measurement of predictors across settings affect the discriminative power and transportability of a prediction model. METHODS: Differences in predictor measurement between data sets can be described formally using a measurement error taxonomy. Using this taxonomy, we derive an expression relating variation in the measurement of a continuous predictor to the area under the receiver operating characteristic curve (AUC) of a logistic regression prediction model. This expression is used to demonstrate how variation in measurements across settings affects the out-of-sample discriminative ability of a prediction model. We illustrate these findings with a diagnostic prediction model using example data of patients suspected of having deep venous thrombosis. RESULTS: When a predictor, such as D-dimer, is measured with more noise in one setting compared to another, which we conceptualize as a difference in "classical" measurement error, the expected value of the AUC decreases. In contrast, constant, "structural" measurement error does not impact on the AUC of a logistic regression model, provided the magnitude of the error is the same among cases and noncases. As the differences in measurement methods between settings (and in turn differences in measurement error structures) become more complex, it becomes increasingly difficult to predict how the AUC will differ between settings. CONCLUSION: When a prediction model is applied to a different setting to the one in which it was developed, its discriminative ability can decrease or even increase if the magnitude or structure of the errors in predictor measurements differ between the two settings. This provides an important starting point for researchers to better understand how differences in measurement methods can affect the performance of a prediction model when externally validating or implementing it in practice.
Authors: Johanna A A G Damen; Thomas P A Debray; Romin Pajouheshnia; Johannes B Reitsma; Rob J P M Scholten; Karel G M Moons; Lotty Hooft Journal: BMJ Open Date: 2019-04-01 Impact factor: 2.692
Authors: Maarten van Smeden; Georg Heinze; Ben Van Calster; Folkert W Asselbergs; Panos E Vardas; Nico Bruining; Peter de Jaegere; Jason H Moore; Spiros Denaxas; Anne Laure Boulesteix; Karel G M Moons Journal: Eur Heart J Date: 2022-08-14 Impact factor: 35.855
Authors: Kimberley J Haines; Elizabeth Hibbert; Joanne McPeake; Brian J Anderson; Oscar Joseph Bienvenu; Adair Andrews; Nathan E Brummel; Lauren E Ferrante; Ramona O Hopkins; Catherine L Hough; James Jackson; Mark E Mikkelsen; Nina Leggett; Ashley Montgomery-Yates; Dale M Needham; Carla M Sevin; Becky Skidmore; Mary Still; Maarten van Smeden; Gary S Collins; Michael O Harhay Journal: Crit Care Med Date: 2020-12 Impact factor: 9.296